Bootstrapper

Gluesync Bootstrapper is a configuration tool for setting up database schema mappings between source and target systems in an automated way, bypassing the need to manually configure entities in the Gluesync Core Hub Web UI.

This project is Open Source

Gluesync Automator UI

For users who prefer a graphical interface over command-line tools, the Bootstrapper is also available as a standalone executable with a web-based UI. The Gluesync Automator provides a user-friendly interface accessible via localhost:8080, featuring drag-and-drop configuration uploads, live log monitoring, and simplified authentication without requiring Python dependencies.

Overview

The Bootstrapper allows you to define table structures, column mappings, and connection properties using YAML configuration files, enabling precise control over how data is transferred and transformed across different database systems. This automation tool integrates directly with the Core Hub API to streamline entity configuration.

Key Features

Real-time Data Synchronization
- Automated synchronization between source and target databases
- Support for multiple database systems (MS SQL Server, Couchbase, etc.)
- Configurable filtering and transformation rules
Schema Configuration
- Table structure mapping
- Column mapping with type safety
- Custom document key generation
- TTL (Time To Live) management for target documents
- Schedules for recurring jobs (via Chronos)
- …and much more
Pipeline Management
- Flexible polling intervals and batch processing
- Automatic agent configuration
- Direct Core Hub API integration

Configuration

The Bootstrapper works with two primary configuration files:

Schema configuration (table-list-template.yaml)
Agent configuration (config.json)

Schema Configuration

The schema configuration defines how tables should be synchronized and transformed:

You can download a ready-to-use schema template here:

table-list-template.yaml

dbo: # Source schema
  target: public # Target schema
  customProperties: # Custom properties for both source and target
    source: # Source custom properties
      maxItemsCountPerIteration: 1000
      pollingIntervalMilliseconds: 100
    target: # Target custom properties
      ttlValue: 10000000
  tables: # Tables to be synchronized
    whitelist: # Allowed tables to be synchronized, empty list means all tables
      - DRIVERS
      - VEHICLES
    blacklist: # Forbidden tables to be synchronized, empty list means no restrictions
      - TEST

Table Configuration

Each table can be configured with detailed mapping information:

DRIVERS:
  keys:
    - ID
  documentKey:
    prefix: "TESTPREFIX"
    suffix: "TESTSUFFIX"
    separator: "-"
    keys:
      - ID
      - LAST_NAME
  name: drivers
  columns:
    - FIRST_NAME: "FIRST_NAME"
    - LAST_NAME: "LAST_NAME"
    - AGE: "AGE"
    - EMAIL: "EMAIL"

Agent Configuration

The agent configuration defines the connection details for source and target databases:

{
  "agents": [
    {
      "agentType": "SOURCE",
      "agentTag": "mssql-ct",
      "hostCredentials": {
        "connectionName": "MS SQL Server source",
        "host": "localhost",
        "port": 1433,
        "databaseName": "demo"
      }
    }
  ]
}

Usage

Getting Started

Before using the Bootstrapper, you need:

A running Gluesync deployment with Core Hub accessible
Authentication credentials for the Core Hub API
Proper network connectivity between source and target systems

Authentication

To use the tool, you need to get a valid authentication token from the Core Hub API. This can be done in several ways:

By logging in to the Core Hub Web UI and copying the token from the browser’s cookies
By using an HTTP client to request a token

curl -X POST https://<corehub-url>/authentication/login -H "Content-Type: application/json" -d '{"username":"<username>","password":"<password>"}'

Main Script Usage

The primary entry point is the main.py script which creates a new pipeline and configures the agents:

python main.py --config <path_to_config> --token <auth_token> [--skip-errors] [--chunk-size <number>]

Alternatively, you can use environment variables:

# Set environment variables
export FILE_CONF_PATH="./my-config.json"
export CORE_HUB_URL="https://my-corehub:1717"
export DEFAULT_PASSWORD="my-secure-password"
export CREATE_ENTITIES_FROM_SCHEMA="true"
export TARGET_SCHEMA="public"
export SOURCE_TYPE="SQL"
export TARGET_TYPE="NoSQL"

# Run the script
python main.py

Creating Entities

To create entities in the Core Hub based on your YAML configuration:

python create_all_entities.py --pipeline <pipeline_id> --source-schema <source_schema> --target-schema <target_schema> --source-type <source_agent_type> --target-type <target_agent_type> --yaml-file <path_to_yaml_config> --token <auth_token> [--skip-errors] [--chunk-size <number>]

Parameters:

--pipeline: Required. The ID of the pipeline to create entities for
--source-schema: Required. Source schema name
--target-schema: Required. Target schema name
--source-type: Required. Source agent type (e.g., SQL for RDBMSs, NoSQL for NoSQL databases)
--target-type: Required. Target agent type (e.g., NoSQL for NoSQL databases, SQL for RDBMSs)
--yaml-file: Required. Path to the YAML configuration file
--token: Required. Authentication token for API access
--skip-errors: Optional. Continue execution even if errors occur
--chunk-size: Optional. Number of entities to process in each chunk (default: 50)

Accepted values for source-type and target-type: - SQL (for any RDBMS) - NoSQL (for NoSQL databases, Kafka, AWSS3, etc.)

Best Practices

Store configuration files in version control
Create different configuration files for development, staging, and production environments
Validate configurations before applying them to production systems
Use consistent naming conventions for schemas, tables, and columns
Document custom transformations and rules
Test configurations thoroughly before deployment

Troubleshooting

Common Issues

Issue	Possible Cause	Resolution
Authentication failures	Invalid or expired token	Generate a new authentication token
Entity creation errors	Incorrect schema configuration	Verify YAML configuration format and structure
Connection issues	Network connectivity problems	Check network settings and firewall rules
Type mapping errors	Incompatible data types between source and target	Verify column type definitions in the YAML configuration

Issue

Possible Cause

Resolution

Authentication failures

Invalid or expired token

Generate a new authentication token

Entity creation errors

Incorrect schema configuration

Verify YAML configuration format and structure

Connection issues

Network connectivity problems

Check network settings and firewall rules

Type mapping errors

Incompatible data types between source and target

Verify column type definitions in the YAML configuration

References

Release Notes

2.3.18

Released: February 26, 2026

Fixed an issue with target table creation and no keys at source;
Azure data lake data type mapping improvements;

2.3.17

Released: February 26, 2026

Create target table with declared keys when source tables has none;

2.3.16

Released: February 25, 2026

Enhaced data type mapping;

2.3.15

Released: February 25, 2026

Major refactor on data types management with the introduction of data type matrix;

2.3.14

Released: February 25, 2026

Fixed an issue with create table processing and columns ids exports;

2.3.13

Released: February 22, 2026

Support for new Chronos schedules types (redo api);

2.3.12

Released: February 19, 2026

Fixed an issue with case sensitiveness applied when attempting to perform table’s creation;

2.3.11

Released: February 15, 2026

Added support for pre & post snapshot queries as per Gluesync 2.1.11;

2.3.10

Released: February 2, 2026

Added support for SQL where clauses applied at source database level;

2.3.9

Released: January 19, 2026

Added support for bulk loads at entity level;

2.3.8

Released: January 16, 2026

Fixed import of document keys;
Fixed import of filters;
Minor fixes and improvements;

2.3.7

Released: January 14, 2026

Minor fixes;

2.3.6

Released: December 19, 2025

Better handling of target table type matching;

2.3.5

Released: December 10, 2025

It is not longer necessary to manually declare the source & target agents types, those will be derived from the embedded agents.json;

2.3.4

Released: December 8, 2025

Enriched metadata for full pipeline export;

2.3.3

Released: December 6, 2025

GSSD-442: when using "targetOnlyColumns" with "lockedSchema" set to true, you’ll prompted with an error;

2.3.2

Released: December 6, 2025

Support for source & target type within .yaml (both for import & export);

2.3.1

Released: December 5, 2025

Added support for pipeline export (also in bulk);

2.3.0

Released: November 28, 2025

Added support for Logical partitions;
Timeout is now float;

2.2.7

Released: November 19, 2025

Added support for Tables Id as per GS 2.1.8.1;

2.2.6

Released: November 7, 2025

Added support for table creation outside "targetOnlyColumns";

2.2.5

Released: October 15, 2025

Fixes around URL encoding during api calls;

2.2.4

Released: October 10, 2025

Suppress additional table discovery checks when not needed;

2.2.3

Released: October 9, 2025

API updates for table creation;

2.2.2

Released: October 3, 2025

Re-added back snapshot write method param;

2.2.1

Released: October 2, 2025

Improved handling of Chains;

2.2.0

Released: September 25, 2025

Support for unlocked schema APIs;

2.1.16

Released: September 23, 2025

Support for conditional deletion ahead of snapshot task;
Support for custom snaphost method property (e.g. INSERT / UPSERT).

2.1.15

Released: September 16, 2025

Added support for Conductor’s APIs;

2.1.14

Released: September 15, 2025

Improvements and fixes around the table creation APIs;

2.1.13

Released: September 3, 2025

Fixed table keys declaration;

2.1.12

Released: September 1, 2025

void tag

2.1.11

Released: August 28, 2025

Support for multiple schemas within the same given .yaml file;

2.1.10

Released: August 26, 2025

Fixed chains creation;
Readme updated;

2.1.9

Released: August 25, 2025

Avoid table’s discovery if CREATE_TABLES_IF_NOT_EXIST is set to false;

2.1.8

Released: August 25, 2025

Support for groups' scheduling;

2.1.7

Released: August 22, 2025

Handle multiple schemas within the same given .yaml file;

2.1.6

Released: August 20, 2025

Make the pipeline name configurable;