Search docs.molo17.com
    Operational guide Deploy and run

    Use Gluesync Automator

    This guide shows how to use Gluesync Automator, the tool with a web page that creates entities, exports and restores pipelines and duplicates them. You sign in to Core Hub, create entities for a whole schema without writing YAML, load a YAML file, export a pipeline, check the archive, restore it on the same or on a second Core Hub, and duplicate a pipeline.

    In this guide

    You will learn how to start Automator and sign in, how to read the Core hub overview, how to create entities in bulk from a source schema, how to create entities from a table-list YAML file and preview the plan first, how to export one pipeline or all pipelines with or without secrets, how to validate an archive, how to restore it on the same Core Hub or on a second one with the Customize panel, how to duplicate a pipeline and how to read the log.

    Quick information

    Goal

    Create, back up, restore and duplicate pipeline configuration with Automator instead of by hand

    Audience

    Gluesync administrators who can sign in to Core Hub with a user that manages pipelines

    Level

    Intermediate

    Time required

    About 45 minutes for all workflows. Each one takes a few minutes

    Area

    Gluesync Automator, in a browser on localhost:8080

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You downloaded the current Automator for your system from the Gluesync Automator page and started it. The version is under the title, for example v1.4.26;

    2. You can reach Core Hub from the computer where Automator runs, and you know its address, port, user name and password;

    3. For the bulk and YAML workflows, the pipeline exists and has a SOURCE and a TARGET connection that Core Hub can open;

    4. The schema that receives the tables exists on the target. Automator creates tables, but it does not create schemas;

    5. For the restore workflows, you have a pipeline to export and, for a second Core Hub, the address and a login of that Core Hub.

    This guide was tested with Automator 1.4.26 on macOS, on two Core Hubs of Gluesync 2.3, on a pipeline from PostgreSQL to PostgreSQL with three tables. Other releases can name buttons differently. The version under the title tells you which one you run.
    Automator keeps one Core Hub session at a time. To work on a second Core Hub, select Sign out and sign in again.

    Guide steps

    Start Automator and sign in

    UI

    1. Start Automator and open http://localhost:8080 in your browser;

    2. In CoreHub URL, type the address of Core Hub, for example https://corehub.example.com. Include the port when it is not 443;

    3. Enter Username and Password;

    4. Under Security, leave Enable TLS selected when Core Hub answers on HTTPS. Clear it when Core Hub answers on plain HTTP. Leave Skip certificate verification selected for a self-signed certificate;

    5. Select Sign in.

    Expected result

    The header shows Connected to: with the address, the number of pipelines, the badge READY and the Sign out button. The tabs Core hub, Backup & export, Import & clone, Bulk and Settings appear. The log lists Authenticating with CoreHub… and Authentication successful.

    Two things help when the sign in does not work:

    • A wrong password ends with 401 Client Error: Unauthorized for url: <address>/authentication/login;

    • Enable TLS against a Core Hub that answers on HTTP ends with an SSLError that contains WRONG_VERSION_NUMBER. Clear Enable TLS.

    The Automator sign in form with Enable TLS and Skip certificate verification selected and a red error under Sign in that ends with SSL WRONG_VERSION_NUMBER wrong version number

    Use Sign out to sign in to another Core Hub.

    Check the Core hub overview

    UI

    1. Select the Core hub tab;

    2. Read Environment: the address, TLS, Skip verification and Chronos;

    3. Read Totals: Pipelines, Agents, Entities, Schedules available and Running containers;

    4. Select Refresh under Pipelines when you changed something in Core Hub.

    Expected result

    The totals match what the Control Plane shows. Chronos and Schedules available show Unavailable when Automator cannot reach Chronos. Import and export still work.

    The Core hub tab of Automator with the Environment card listing the address
    Create entities in bulk from a source schema

    UI

    Use this to create one entity for every table of a schema without clicking each table in Core Hub and without a YAML file.

    1. Select the Bulk tab;

    2. In Pipeline, select the pipeline;

    3. Select Load source schemas. Automator asks the SOURCE agent of the pipeline for its schemas;

    4. In Source schema, select the schema, for example shop. The system schemas of the database are in the list too;

    5. In Target schema, type the name of the schema on the target;

    6. Read the table list. Every table is selected, and the line above it says how many tables are available and how many are selected. Clear the tables you do not want, or select Toggle/untoggle all and tick only the ones you want;

    7. Select Create all entities (N), where N is the number of selected tables.

    Expected result

    The message Bulk entity creation completed successfully appears under the buttons. The log shows Successfully created regular entities: 3 and Failed regular entities: 0. The tables exist on the target, and the pipeline in Core Hub lists the entities.

    The Bulk tab with the pipeline shop-demo
    The Bulk tab after the run with the target schema shop and the green message Bulk entity creation completed successfully

    Automator creates the entities. It does not start them. Start them in Core Hub, as in Start a snapshot and CDC.

    Get YAML template downloads a YAML file named template_<pipelineId>_<schema>_<time>.yaml with the selected tables and their columns. It needs a target schema too. Use it as the start of the YAML workflow below.

    Open the Settings tab to change how Automator creates entities: Chunk size, Skip errors, Create missing tables automatically and Enable scheduling for created entities. The settings apply to the next run.

    Create entities from a YAML file

    UI

    Use a YAML file when you need a different target table name, a column mapping or a filter. The template table-list-template.yaml shows every option. A small file is enough to start:

    shop:
      target: shop
      sourceType: SQL
      targetType: SQL
      tables:
        whitelist:
          - reviews
        custom:
          reviews:
            name: reviews_copy
            keys:
              - id
            columns:
              - id: "id"
              - rating: "stars"
              - body: "body"
            filter:
              clauses:
                - column: rating
                  type: int
                  operation: GreaterEqual
                  value: 3

    The first key (shop) is the source schema, target is the target schema, and every line under columns maps a source column to a target column.

    1. Select the Import & clone tab. The first card is Create entities from YAML (single pipeline);

    2. In Pipeline, select the pipeline;

    3. Under YAML configuration or pipeline backup (ZIP with UDFs), choose the file;

    4. Read the line 1 entities loaded from the export file. and the table under it. The Status column shows new table when the target table does not exist, and table exists when it does. ⚠︎ mismatch means that the existing table differs from the plan;

    5. Change the Target schema or table of a row when you need to;

    6. Under Customize entity creation, set If the entity is already in the pipeline when the entity may exist: keep current behaviour, skip (leave as-is), update in place or fail if it already exists. Clear Parts to apply you do not need (UDFs, Function collection, Schedules, Groups, Create missing tables);

    7. Select Preview. It only reads from Core Hub. A table shows each entity, its target and its Action, for example create if missing or use existing (no create);

    8. Select Start import.

    Expected result

    The run status changes to COMPLETED. The entity exists in the pipeline, and the target table reviews_copy was created. Dry run (preview writes only; nothing created) does the same without writing anything.

    The Create entities from YAML card after Preview
    Export a pipeline

    UI

    1. Select the Backup & export tab;

    2. Decide on Include secrets under Export options. Leave it cleared to write ******* in place of every agent password. Select it to write the real passwords into the archive;

    3. Under Export a single pipeline, select the pipeline;

    4. Select Export entities for one YAML file with the entities of that pipeline, or Pipeline backup for a ZIP archive with the pipeline, its agents and its UDFs;

    5. To export every pipeline of the Core Hub, select Full backup under Export all.

    Expected result

    The browser downloads the file, and Automator shows Exported backup_<pipelineId>_<time>.yaml or Full backup completed: backup_<name>_<pipelineId>_<time>.zip. The archive of Pipeline backup holds a folder pipeline_<pipelineId> with the pipeline YAML and agents-config.yaml. The archive of Full backup holds one YAML per pipeline and one agents-config.yaml at the top.

    The Backup & export tab with Include secrets cleared
    An archive made with Include secrets holds the passwords of your databases as text in agents-config.yaml. Store it like a password, and delete it when you no longer need it.

    An archive made without Include secrets holds password: '*******' for every agent. It cannot be imported until the passwords are supplied, as the next step shows.

    Check an archive before you import it

    UI

    1. On the Import & clone tab, find the card Import a full backup;

    2. Under Backup archive (ZIP) containing pipeline(s), agents, entities, schedules and UDFs, choose the archive;

    3. Open Advanced: Validate / Import All without Customize;

    4. Select Validate.

    Expected result

    For an archive with real passwords, Automator shows Validation OK. The log lists the agents, the plan (would create pipeline 'shop-demo (restored)' from original ID 79cbc326) and the schema pairs (discovered 1 schema pair(s): shop→shop).

    The Import a full backup card with pipe-secrets.zip chosen

    For an archive that still holds *******, Validate stops with This backup has masked passwords. Use Customize to enter SOURCE/TARGET passwords, then Preview and Import selected. Use the Customize panel in the next step.

    The Import a full backup card with pipe-nosecrets.zip chosen and the red message This backup has masked passwords. Use Customize to enter SOURCE/TARGET passwords
    Restore on the same Core Hub

    UI

    Use this to get a copy of a pipeline next to the original, or to test a restore. Use an archive with real passwords.

    1. In Import a full backup, choose the archive and open Advanced: Validate / Import All without Customize;

    2. Select Import All.

    Expected result

    The message Imported: shop-demo (restored) (<pipelineId>) - 3 of 3 entities created. appears. The new pipeline has the name of the original plus (restored), because the name was taken, and its own agents and entities. Nothing is started.

    The Import a full backup card with the green message Imported shop-demo restored c25b07d0 - 3 of 3 entities created
    Restore on a second Core Hub with Customize

    UI

    Use Customize to restore on another Core Hub, to type the passwords of an archive that has none, or to change names and tables before anything is written.

    1. Select Sign out, and sign in to the second Core Hub as in the first step;

    2. On Import & clone, choose the archive in Import a full backup, and select Customize…;

    3. Read the four steps of the panel: Analyze, Customize, Preview and Import;

    4. Under Global settings, read what the archive does not hold: Notifications (SMTP), Users, OIDC and user groups and Data access policies show (not in backup);

    5. Under Defaults for every pipeline, set If an entity already exists and Pipeline name already taken. The default for the second is to add (restored);

    6. For each pipeline, change Pipeline name if you want. Under Agents, change the host, port, database and username of the SOURCE and TARGET agent for the new environment. In the Password fields, type the password when the label says missing (masked in the backup). Leave them empty when it says from backup; leave empty to keep;

    7. Under Tables, change Target schema, Target table or Entity name of a row, or clear Import to leave a table out. (as in backup) means no change;

    8. Select Preview. Read the plan. Every agent line ends with password: override when you typed a password;

    9. Select Import selected.

    Expected result

    Automator names every pipeline with the number of entities it created, for example Imported: shop-demo (restored) (<pipelineId>) - 3 of 3 entities created. The second Core Hub lists the pipelines. Nothing is started.

    The Customize panel of a pipeline with the SOURCE agent postgresql-cdc and the TARGET agent postgresql-cdc
    The Tables list of the Customize panel with the text 11 tables will be created

    Check the connections before you start anything. In the lab, a pipeline whose SOURCE and TARGET agents were of the same type (PostgreSQL to PostgreSQL) was restored on the second Core Hub with the details of the target database on both agents. The source then showed the tables of the target database. Open each restored pipeline in Core Hub, select the SOURCE and TARGET connection and compare the database name with the one of the original pipeline. If a connection is wrong, fix it in Core Hub before you create or start entities. See Troubleshooting.

    Check the restored pipelines

    UI

    1. Open the second Core Hub in the Control Plane;

    2. Open each restored pipeline. Compare the SOURCE and TARGET connections, the entities and the target tables with the original;

    3. In Automator, open the Core hub tab and select Refresh. The totals include the new pipelines.

    Expected result

    Each pipeline has the entities of the archive, and each connection points to the database that you meant. The entities are not started.

    For the full flow from QA to production, see Promote pipelines from QA to production.

    Duplicate a pipeline

    UI

    Duplicate a pipeline when you want a copy that points to the same or to other databases, without an archive.

    1. On Import & clone, find the card Duplicate pipeline (w/ agents);

    2. In Source pipeline, select the pipeline;

    3. In New pipeline name, type the name of the copy;

    4. Type Source host password and Target host password. Both are required;

    5. Leave Clone entities, schemas, and schedules from the original pipeline selected to copy the entities. Clear it to get only the pipeline and its connections;

    6. To point the copy elsewhere, fill Source host address, Target host address, Source host port and Target host port, or select Override source & target schemas. Empty fields reuse the values of the original;

    7. Select Duplicate Pipeline.

    Expected result

    The message Pipeline duplicated successfully: <name> (<pipelineId>) appears. The new pipeline has its own agents and the entities of the original.

    The Duplicate pipeline card with the fields Source pipeline
    The Duplicate Pipeline button and the green message Pipeline duplicated successfully shop-demo-clone
    Customize new source/target agents (Danger zone) replaces the agent types of the copy. Use it only when the new agents exist or can be created.
    Read the log

    UI

    Every workflow writes to the Execution log at the bottom of the page.

    1. Select a level in Level (All, Debug+, Info+, Warning+ or Error) to filter the lines, and use the search box to find a table or a pipeline;

    2. Read the summary line above the lines, for example info 11 warn 0 error 0, the duration and entities ok 1 / skip 0 / fail 0;

    3. Select History to open an earlier run;

    4. Select Copy or Download to attach the log to a ticket, and Collapse to hide it.

    Expected result

    Each run is listed under History with the time and the number of lines. A failed run keeps its lines, so you can read the reason after the page shows FAILED.

    Troubleshooting

    Problem Possible cause What to do

    The sign in ends with SSLError and WRONG_VERSION_NUMBER

    Enable TLS is selected, and Core Hub answers on plain HTTP

    Clear Enable TLS, or use the HTTPS address and port of Core Hub

    The sign in ends with 401 Client Error: Unauthorized

    User name or password is wrong

    Sign in to the Control Plane with the same user to check it, then try again

    Bulk shows Please enter a target schema name

    Target schema is empty. Create all entities and Get YAML template need it

    Type the schema of the target and select the button again

    The log shows Create target table execution error: ERROR: schema "<name>" does not exist

    The target schema is missing, or its name has a typing error. Automator creates tables, not schemas

    Create the schema on the target, or correct the name, and run again with the same selection

    The log shows CoreHub request PUT /pipelines/<id>/config/entities failed: 500 and, after Response:, a message of the database

    Core Hub passed on an error of the source or the target database while it saved the entity

    Read the text after Response:. Fix the cause in the database, then run again.

    ERROR: all replication slots are in use during a bulk, YAML, restore or duplicate run on PostgreSQL

    Every entity of a PostgreSQL CDC source uses one replication slot, and the database reached max_replication_slots

    Raise max_replication_slots (and max_wal_senders) on the source database and restart it. Delete the partial pipeline that a failed Duplicate Pipeline left in Core Hub, then run the workflow again

    Validate or Import All says This backup has masked passwords

    The archive was exported without Include secrets

    Export again with Include secrets, or select Customize… and type the passwords

    In Customize, the red message A masked password is still present. Fill the SOURCE/TARGET password fields, then Preview again. stays after you filled both passwords

    The text covers more than masked passwords. In the lab it appeared when SOURCE and TARGET are of the same agent type, so both received the same plan key

    Export the pipeline again with Include secrets, choose that archive, and leave the Password fields empty

    The result starts with PARTIAL: and says 3 of 4 entities created (1 failed)

    One entity could not be created. The line after it names the table and the reason

    Fix the cause, delete the partial pipeline in Core Hub and import again

    An entity is refused with column id of the YAML columns is not a column of the source table discovery returns

    The SOURCE connection of the pipeline does not point to the database that has the table

    Open the SOURCE connection in Core Hub and check the database name. See the warning in Restore on a second Core Hub

    The Core hub tab shows Unavailable for Chronos

    Automator cannot reach Chronos at the address of Core Hub

    Nothing to fix for entities and archives. Core Hub and the entity workflows keep working