Search docs.molo17.com
    Operational guide Deploy and run

    Promote pipelines from QA to production

    This guide shows how to take a pipeline that works on a QA Core Hub and recreate it on a production Core Hub, where the hosts, users, passwords and schemas are different. You export it once, change only what differs, import it, start it and prove with SQL that the data arrives. The guide also covers cloning entities inside a pipeline and the errors that people meet most often when they import.

    In this guide

    You will learn which tools can export and import a pipeline, how to export a QA pipeline with Gluesync Automator, which lines of the export you must change for production, how to supply the passwords that an export never contains, how to validate and import the archive, how to start the pipeline and check the result in the databases, three ways to clone an entity inside a pipeline, what each known import error means and how to fix it, and a go-live checklist.

    Quick information

    Goal

    Recreate a tested pipeline (connections, entities, keys, filters and field functions) on a production Core Hub and start it

    Audience

    Gluesync administrators who run more than one Core Hub (DEV, QA, production)

    Level

    Intermediate

    Time required

    About 45 minutes for a pipeline with three entities

    Area

    Gluesync Automator → Backup & export, Import & clone; Control Plane → pipeline page

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You have a QA Core Hub with the pipeline working and a production Core Hub that is empty or ready for it. Both run the same Gluesync version. A backup made on an older Core Hub and restored on a newer one (or the other way round) is the most common cause of odd import errors;

    2. You can sign in to both Core Hubs as a user with the Super Admin role;

    3. The production databases exist, are reachable from the production Core Hub, and the target tables exist, or you let Automator create them (Settings tab, Create missing tables automatically, on by default);

    4. You have the production database host names, ports, user names and passwords from your password manager. The export never contains them;

    5. You have Gluesync Automator on a workstation that can reach both Core Hubs. Download it from molo17.com. The version number is shown under the title in the top bar. See Gluesync Automator;

    6. You have a plan for the first start in production: the target tables are empty, or you accept that rows with the same key are overwritten.

    This guide was tested with a QA Core Hub and a second, separate production Core Hub in Docker, each with its own PostgreSQL source and target. The production hosts, database names, users, passwords and schemas all differ from QA, and so does the Core Hub storage key. The pipeline reads three tables: one with a target filter and a field function, one plain table, and one table without a primary key that uses a logical key. Names in your installation will differ.
    What differs QA Production

    Source host and database

    pgsql-source, shop

    pg-ledger.prod.internal, ledger

    Target host and database

    pgsql-target, shop

    pg-replica.prod.internal, replica

    Users

    gluesync on both

    prod_reader (source), prod_writer (target)

    Passwords

    QA passwords

    Different passwords that you hold in your vault

    Schemas

    promote_src (source), promote_tgt (target)

    sales (source), sales_replica (target)

    Choose how to export and import

    Gluesync 2.3 has several ways to read a pipeline configuration. Only some can also write it back.

    Route What it does Use it for

    Gluesync Automator (web application you run on your workstation)

    Backup & export writes a ZIP (Pipeline backup) or a YAML file (Export entities). Import & clone validates and imports the ZIP (Import All), creates entities from YAML in an existing pipeline, imports connection settings only, or duplicates a pipeline.

    The normal way to promote a pipeline. This guide uses it.

    Core Hub REST API

    GET /pipelines/<id>/config returns the whole configuration as JSON. PUT /pipelines/<id>/config sends a complete configuration to a pipeline and answers 202.

    Scripts and CI jobs. You map agent IDs and schema names yourself. See Optional: promote with the REST API.

    Core Hub MCP tool export_pipeline_yaml

    Returns the same JSON as the REST call. Despite its name it is not YAML. Secrets are replaced by <hidden>.

    Reading a configuration from an AI client. See Core Hub MCP server.

    Control Plane

    The Control Plane has no menu to export or import a pipeline or to clone an entity.

    Checking the result, starting entities, editing a connection.

    Backups to Gluesync Connect

    Back up the whole Core Hub (databases, storage key, license) and restore it on the same host or a new machine. See Backups with Gluesync Connect.

    Disaster recovery. Do not use it to promote one pipeline: a restore replaces the target Core Hub.

    Guide steps

    Look at the QA pipeline you are going to promote

    UI

    1. Sign in to the QA Control Plane and open the pipeline;

    2. In the Features column, note which entities have a filter (funnel icon) or a field function (fx icon). You will check that they survive the move.

    The QA pipeline Promote demo with entities customers
    Figure 1. The QA pipeline. Entity customers has a target filter and a field function.
    Expected result

    You know what the production pipeline must contain: three entities, a filter on customers and a field function on its email column.

    Export the pipeline from QA

    UI

    1. Start Automator and sign in to the QA Core Hub: enter the CoreHub URL with the port (for example https://qa.example.com:1717), the Username and the Password. Keep Enable TLS on. Leave Skip certificate verification on only for self-signed certificates;

    2. Open the Backup & export tab;

    3. Leave Include secrets unchecked (see the note below);

    4. In Pipeline, choose the pipeline;

    5. Click Pipeline backup. Your browser downloads backup_<pipeline name>_<pipeline id>_<date>.zip. Export entities downloads only the YAML file of the entities; use it when the connections already exist in production.

    The Backup & export tab of Gluesync Automator with Include secrets unchecked (1)
    Figure 2. Automator, Backup & export. The numbers show Include secrets, the pipeline and Pipeline backup.
    Expected result

    Automator shows Exported backup_Promote demo_<id>_<date>.zip and the ZIP is in your downloads folder. It holds pipeline_<id>/backup_<name>_<id>_<date>.yaml (entities, keys, filters, field functions), agents-config.yaml (the two connections) and global-config.yaml (the alert settings of QA).

    Include secrets keeps the passwords in the file, and on Gluesync 2.3 they are written in clear text (the hint under the check box mentions AES256, but the file we exported held the plain password). They are QA passwords and production needs its own, so you gain nothing and you create a sensitive file. Export without secrets. A file without secrets shows password: '*' for each connection.
    Unpack the archive and change what differs

    Config

    1. Unpack the ZIP in an empty folder and delete the alert settings of QA, so that the import cannot send them to production:

      mkdir work && cd work
unzip "../backup_Promote demo_8aa2b0d9_20261028_210446.zip"
rm pipeline_8aa2b0d9/global-config.yaml

    2. Open pipeline_<id>/agents-config.yaml. For each agent, change the connection to the production values and type the production password. Do not change agentType, agentTag or agentId:

      agents:
- agentType: SOURCE
  agentTag: postgresql-cdc
  hostCredentials:
    connectionName: Promote demo-source
    host: pg-ledger.prod.internal      # was pgsql-source
    port: 5432
    databaseName: ledger               # was shop
    username: prod_reader              # was gluesync
    password: '<production source password>'   # was '*******'

    3. Do the same for the TARGET entry (pg-replica.prod.internal, replica, prod_writer);

    4. Open pipeline_<id>/backup_<name>_<id>_<date>.yaml. The first key below exportMetadata is the source schema, and its target: line is the target schema. Change both:

      sales:                 # was promote_src
  target: sales_replica    # was promote_tgt
  sourceType: SQL
  targetType: SQL

    5. If a production table has another name, change name: of that table (the target table) in the same file. The entity name is entityName:;

    6. Zip the folder again, with the pipeline_<id> folder at the top of the archive:

      zip -r ../promote-demo-prod.zip pipeline_8aa2b0d9
    Expected result

    promote-demo-prod.zip holds agents-config.yaml with production values only and a YAML file whose schema pair is sales to sales_replica.

    Change the schema names in the YAML file itself. The Automator check box Customize source/target schemas (override YAML) does not rename them: it only picks which YAML section is read, and the target: line of the file still wins. In the test, a different schema name in this box made Automator ignore every per-table setting (keys, filters, field functions) and then fail on tables without a primary key.

    Keep the production passwords out of shell history and shared folders: type them in an editor, and delete work and the ZIP when the import is done. Do not commit the ZIP to a repository.
    Validate and import the archive into production

    UI

    1. In Automator click Sign out, then sign in to the production Core Hub. Automator holds one session at a time;

    2. Open Import & clone. In Import a full backup click the file field and choose promote-demo-prod.zip;

    3. Click Validate. Validation checks the structure, that the passwords were replaced and that each YAML file has a schema pair. It creates nothing;

    4. Click Import All.

    The Automator sign-in form with the production CoreHub URL marked 1 and the Sign in button marked 2.
    Figure 3. Signing in to the production Core Hub.
    The Import a full backup card with full.zip chosen (1) and Validate clicked (2). The red message reads: Config contains masked passwords
    Figure 4. What Validate answers if the passwords are still masked.
    The Import a full backup card with promote-demo-prod.zip chosen (1)
    Figure 5. Validation of the archive with the production values.
    The Automator execution log: Validation OK
    Figure 6. The execution log after Validate and while Import All runs.
    Expected result

    Validation ends with Validation OK: archive is structurally consistent and ready to import. and discovered 1 schema pair(s): sales→sales_replica. Import All ends with Imported 1 pipeline(s): Promote demo (<new id>). The pipeline exists in production with all entities on hold. Nothing is started.

    If production already has a pipeline with the same name, the import names the new one Promote demo (restored), then (restored 2), and so on. Rename it in the Control Plane when you are sure that it is the right one.

    The Settings tab of Automator has Create missing tables automatically, on by default, so a missing target table is created. If your DBA creates the tables, create them first. With the option off, Automator assumes that all tables exist and creates the entities even if they do not (see the troubleshooting table).
    Check the production pipeline before you start it

    UI

    1. Sign in to the production Control Plane and open the pipeline. All entities are on hold;

    2. Check the Features column: the funnel and fx icons must be there, as in QA;

    3. Open the gear menu of the pipeline, choose Edit source and read the connection: the host, database and user must be the production ones. Choose Test and continue to open a connection with the production password. Repeat with Edit target.

    The Edit source page of the production pipeline with host pg-ledger.prod.internal (1)
    Figure 7. The production source connection. 1 host, 2 database, 3 user, 4 Test and continue.
    Expected result

    The connection test succeeds on both sides, and the pipeline has the same entities, filters and field functions as in QA.

    Start the pipeline in production and verify with SQL

    UI

    1. In the pipeline page start an entity with the Start sync button (the play icon). In Sync changes (CDC) choose Full copy ("Recommended for the first start"). Keep Keep and merge under Data already in the target, or choose Empty the table first if the target tables hold test rows. Click Start entity. Repeat for the other entities;

    The Sync changes (CDC) dialog for customers with Full copy chosen and marked 1 and the Start entity button marked 2.
    Figure 8. The first start of an entity. 1 Full copy, 2 Start entity.

    1. When the entities are active, compare the databases. On the production source:

      SELECT status, count(*) FROM sales.customers GROUP BY status;

    2. On the production target:

      SELECT 'customers' AS t, count(*) FROM sales_replica.customers
UNION ALL SELECT 'orders', count(*) FROM sales_replica.orders
UNION ALL SELECT 'order_lines', count(*) FROM sales_replica.order_lines;

SELECT id, status, email FROM sales_replica.customers ORDER BY id LIMIT 3;

    3. Change data on the source, then read the target again. This proves that change data capture (CDC) works with the production connection, the filter and the logical key:

      INSERT INTO sales.customers VALUES (31, 'New Active', 'new.active@prod.example.com', 'IT', 'gold', 'active', 500, now());
INSERT INTO sales.customers VALUES (32, 'New Inactive', 'new.inactive@prod.example.com', 'IT', 'gold', 'inactive', 500, now());
UPDATE sales.order_lines SET qty = 99 WHERE order_id = 1 AND line_no = 1;
DELETE FROM sales.order_lines WHERE order_id = 2 AND line_no = 2;
    Expected result

    In the test, the source had 30 customers (10 active, 10 inactive, 10 suspended), 60 orders and 40 order lines. After the first start the target held 20 customers (only active and suspended), 60 orders and 40 order lines, and the emails were masked (u****@prod.example.com). After the changes the target had customer 31 with the masked email n*********@prod.example.com, no customer 32, qty = 99 on order line (1, 1) and 39 order lines. The filter, the field function and the logical key work in production as they did in QA.

    Do not start the QA and the production pipelines against the same databases. If you copy QA settings and forget to change a host, both pipelines write to the same target.
    Clone an entity inside a pipeline

    Config

    Sometimes you need a second entity from the same source table: another target table, another filter, an archive copy. There are three ways. The Control Plane has no clone button.

    Way 1: Automator, YAML with a duplicate key. Use it when you want the filter, field functions and mapping of the original.

    1. Export the production pipeline with Export entities;

    2. Make a new file with only the schema header and one table block. Copy the whole block of the table you want to clone (with its columns), then change these: the key (add @@2), name (the new target table), entityName, and add sourceTableName and the filter you want. The whitelist must name the real source table, not the key with @@2:

      sales:
  target: sales_replica
  sourceType: SQL
  targetType: SQL
  tables:
    whitelist:
    - orders                       # the real source table
    custom:
      orders@@2:                   # unique key of the clone
        name: orders_eu            # target table
        entityName: orders_eu
        sourceTableName: orders
        keys:
        - id
        columns:                   # the same columns list as in the orders block
        - source: id
          target: id
          # ... one entry per column, copied unchanged
        filter:
          clauses:
          - column: currency
            type: string
            operation: In
            value: EUR

    3. In Automator, Import & clone, Create entities from YAML (single pipeline): choose the production pipeline, leave Customize source/target schemas unchecked, choose the file and start the run. Create tables is on, so Automator creates orders_eu.

    Way 2: Core Hub MCP or REST, one call. create_entity takes the source and target agent IDs of the pipeline (from get_pipeline), a second target table and the key. It creates the target table (target_table_created: true) and the entity, on hold. It gives you no filter or field function: add them afterwards in the Control Plane.

    { "pipeline_id": "<production pipeline id>",
  "source_agent_id": "<source agent id>", "target_agent_id": "<target agent id>",
  "source_schema": "sales", "source_table": "orders",
  "target_schema": "sales_replica", "target_table": "orders_copy",
  "key_columns": ["id"], "entity_name": "orders_copy" }

    Way 3: Duplicate pipeline. In Import & clone, Duplicate an existing pipeline copies a whole pipeline (agents, entities and schedules) into a new one in the same Core Hub. Give the new name and the source and target passwords. Use it for a second pipeline on the same databases, not to change schemas: in the test, Override source & target schemas was ignored and the copied entities still pointed at the original target schema, so two pipelines wrote to the same tables.

    The production pipeline page with entities customers
    Figure 9. Production pipeline after two clones. 1 orders_eu (YAML, filter on currency). 2 orders_copy (MCP, on hold).
    Expected result

    The clone appears as a separate entity. After a Full copy start, orders_eu held the 21 EUR orders of the source (21 of 60) and nothing else.

    Optional: promote with the REST API

    Use this route in a script. It does not need Automator, but you map the IDs yourself.

    1. On QA, read the configuration. A Super Admin gets the passwords with include_secrets=true; without it they are <hidden>:

      curl -sk -H "Authorization: Bearer $QA_TOKEN" \
  "https://qa.example.com:1717/pipelines/<qa pipeline id>/config" > promote-demo.json

    2. In production, create the pipeline and its two agents with their connections (the Control Plane, Automator Import pipeline settings (config only), the Duplicate an existing pipeline card with Clone entities off, or the MCP tools create_pipeline, add_agent, assign_agent, attach_connection), then read its agents block with GET /pipelines/<prod pipeline id>/config?include_secrets=true;

    3. In promote-demo.json replace the QA agent IDs with the production agent IDs (in agents and in every agentEntities), replace the agents block by the production one, and replace the QA schema names with the production ones;

    4. Send it:

      curl -sk -X PUT -H "Authorization: Bearer $PROD_TOKEN" -H "Content-Type: application/json" \
  --data @promote-demo.json \
  "https://prod.example.com:1717/pipelines/<prod pipeline id>/config"
    Expected result

    Core Hub answers 202 with Configuration dispatched to agents. The pipeline lists the three entities with their filter, field function and keys. In the test the filter, the MaskEmail function and the composite key all arrived.

    Known import errors and what to do

    Message or symptom Cause What to do

    Config contains masked passwords (*). Please replace them with real passwords before importing. (Validate or Import All)

    The export was made without Include secrets, so agents-config.yaml holds '*', and you did not replace them.

    Type the production passwords in agents-config.yaml (see "Unpack the archive and change what differs") and zip again.

    Failed to initialize pool: The connection attempt failed. with PUT /pipelines/<id>/agents/<id>/config/credentials and 500. The import takes a couple of minutes to fail.

    The production Core Hub cannot reach the host in the file. Typical cause: the QA host name is still in agents-config.yaml, or a firewall, DNS or container network blocks the production database.

    Fix host, port and the network (the Core Hub, not your workstation, connects to the database). Open Edit source and Edit target and use Test and continue. A failed import leaves a half-built pipeline with only a source agent in production: delete it before you import again, or the next one is named (restored). The Automator answer says success: true with No pipelines were imported; read the message, not the flag.

    Error: 'type', then create_all_entities execution failed after the tables of the target schema are listed

    The exported YAML had no data type for the columns (type: null). This came from an old Automator exporting from a Core Hub that did not match the one it was written for.

    Use the current Automator (the version under the title; the banner v… is available shows a newer one), make sure QA and production run the same Gluesync version, export again and import the new file. This was not reproducible with the current Automator: YAML with type: null imported without error.

    Undefined table Error: relation "<schema>.<table>" does not exist when you start an entity

    The target table is missing. With Create missing tables off, Automator logs assuming all tables exist and creates the entity anyway.

    Create the tables with the DDL of the source (same columns and key), or switch Create missing tables automatically on in the Settings tab and import again. If a table in the YAML does not exist in production and should not, delete its entry from the whitelist and custom sections.

    the source table of <table> has no key: it has no primary key and keys: names none. Name the key column(s) in keys: (older versions: HTTP 400 Missing entity primary keys). Tables with a primary key import, tables without one fail.

    The logical key (or the _RRN technical key on IBM i) lives in the keys: list of the table in the YAML. Automator did not read that table block, most often because the schema name typed in Customize source/target schemas differs from the first key of the YAML.

    Change the schema key and target: in the YAML (see "Unpack the archive and change what differs") and leave the check box off. Check that the table block still has keys:. For IBM i, switch on the _RRN technical field on the source agent in production first. See Choose keys for tables without a primary key.

    POST /api/run answers 422 Unprocessable Entity

    The browser sent a request that Automator cannot read. The body of the answer names the field, for example "loc": ["body","pipelineId"], "msg": "Field required", or chunkSize above 500.

    Select a pipeline, choose the file again, keep Chunk size between 1 and 500, and run again. If it persists, update Automator and send the answer body to support.

    Import & clone reports Entity creation started but no entity appears

    The whitelist does not name the source table. For a cloned entry the whitelist must hold the real table name (orders), not the key orders@@2.

    Fix the whitelist and run again. Read the Automator log: Skipping table … not in whitelist.

    Field functions are missing after the import

    An old Automator did not export field functions (they were not in the YAML).

    Use the current Automator and export again. In the test the function MaskEmail and the filter of customers arrived in production. Check the Features column.

    Impossible to upsert entities before database initialization, please verify the connection to the database when entities are created, while the tables are created

    Support has seen it when an agent was added twice to a pipeline: the first one stayed half-built inside Core Hub, and every entity creation tried to configure it.

    Restart the Core Hub container. The half-built agent is not saved and disappears. Clean the target tables and import again.

    Automator does not connect: PermissionError(13, 'Permission denied'), Connection refused, 401 Unauthorized, 404 Not Found

    PermissionError means that a local firewall, antivirus or proxy blocks Automator before it reaches Core Hub. Connection refused is a wrong port. 401 is a wrong user or password, or a Core Hub that restarted. 404 is the wrong address: another web server answers on the port.

    Allow Automator in the firewall and the antivirus, check any proxy, and give the CoreHub URL with the port where Core Hub listens (for example https://prod.example.com:1717). With Skip certificate verification off, a self-signed certificate gives CERTIFICATE_VERIFY_FAILED.

    The restore stops with a 500 on the production Core Hub, or entities look different

    QA and production run different Gluesync versions, or Automator is old. The Windows file properties of Automator can show an old number: read the version under the title in the web page.

    Bring both Core Hubs to the same version (see Upgrade Gluesync safely), update Automator, export again.

    Import All finishes, but production has the alert (SMTP) settings of QA, or the log says Failed to restore global notification settings

    The pipeline backup holds global-config.yaml. Import All tries to apply it to the production Core Hub.

    Delete global-config.yaml from the ZIP before you import, then set the production alert settings in the Control Plane.

    Go-live checklist

    Before you start the production pipeline

    • QA and production run the same Gluesync version, and Automator is current;

    • you exported without Include secrets, and the ZIP and the folder with production passwords are deleted after the import;

    • agents-config.yaml holds production hosts, ports, databases, users and passwords, and the YAML holds the production schema names;

    • global-config.yaml was removed, and the production alert, SMTP and webhook settings are set in the Control Plane;

    • Validate passed and Import All created the pipeline once (no (restored) copy left over);

    • Test and continue succeeds on the source and the target connection;

    • the target tables exist, with the same keys, and are empty or you chose Keep and merge on purpose;

    • the source is ready for CDC: for PostgreSQL, the user may replicate and tables without a primary key have REPLICA IDENTITY FULL; for other sources see the agent page;

    • filters, field functions and keys show in the Features column and in Edit mapping;

    • users, groups and roles for production exist (they are not part of a pipeline backup). See Manage users, groups and roles;

    • you started with Full copy, compared counts with SQL, and changed rows on the source to see an insert, an update and a delete arrive;

    • the QA pipeline does not point at production databases and the production pipeline does not point at QA ones.

    Related guides

    Contact MOLO17 Support if you need assistance with promoting pipelines between environments.