Bootstrapper Static Repository

Gluesync Bootstrapper is a configuration tool for setting up database schema mappings between source and target systems in an automated way, bypassing the need to manually configure entities in the Gluesync Core Hub Web UI.

Open Source This project is Open Source

Overview

The Bootstrapper allows you to define table structures, column mappings, and connection properties using YAML configuration files, enabling precise control over how data is transferred and transformed across different database systems. This automation tool integrates directly with the Core Hub API to streamline entity configuration.

Key Features

  • Real-time Data Synchronization

    • Automated synchronization between source and target databases

    • Support for multiple database systems (MS SQL Server, Couchbase, etc.)

    • Configurable filtering and transformation rules

  • Schema Configuration

    • Table structure mapping

    • Column mapping with type safety

    • Custom document key generation

    • TTL (Time To Live) management for target documents

    • Schedules for recurring jobs (via Chronos)

    • …​and much more

  • Pipeline Management

    • Flexible polling intervals and batch processing

    • Automatic agent configuration

    • Direct Core Hub API integration

Configuration

The Bootstrapper works with two primary configuration files:

  1. Schema configuration (table-list-template.yaml)

  2. Agent configuration (config.json)

Schema Configuration

The schema configuration defines how tables should be synchronized and transformed:

dbo: # Source schema
  target: public # Target schema
  customProperties: # Custom properties for both source and target
    source: # Source custom properties
      maxItemsCountPerIteration: 1000
      pollingIntervalMilliseconds: 100
    target: # Target custom properties
      ttlValue: 10000000
  tables: # Tables to be synchronized
    whitelist: # Allowed tables to be synchronized, empty list means all tables
      - DRIVERS
      - VEHICLES
    blacklist: # Forbidden tables to be synchronized, empty list means no restrictions
      - TEST

Table Configuration

Each table can be configured with detailed mapping information:

DRIVERS:
  keys:
    - ID
  documentKey:
    prefix: "TESTPREFIX"
    suffix: "TESTSUFFIX"
    separator: "-"
    keys:
      - ID
      - LAST_NAME
  name: drivers
  columns:
    - FIRST_NAME: "FIRST_NAME"
    - LAST_NAME: "LAST_NAME"
    - AGE: "AGE"
    - EMAIL: "EMAIL"

Agent Configuration

The agent configuration defines the connection details for source and target databases:

{
  "agents": [
    {
      "agentType": "SOURCE",
      "agentTag": "mssql-ct",
      "hostCredentials": {
        "connectionName": "MS SQL Server source",
        "host": "localhost",
        "port": 1433,
        "databaseName": "demo"
      }
    }
  ]
}

Usage

Getting Started

Before using the Bootstrapper, you need:

  1. A running Gluesync deployment with Core Hub accessible

  2. Authentication credentials for the Core Hub API

  3. Proper network connectivity between source and target systems

Authentication

To use the tool, you need to get a valid authentication token from the Core Hub API. This can be done in several ways:

  1. By logging in to the Core Hub Web UI and copying the token from the browser’s cookies

  2. By using an HTTP client to request a token

curl -X POST https://<corehub-url>/authentication/login -H "Content-Type: application/json" -d '{"username":"<username>","password":"<password>"}'

Main Script Usage

The primary entry point is the main.py script which creates a new pipeline and configures the agents:

python main.py --config <path_to_config> --token <auth_token> [--skip-errors] [--chunk-size <number>]

Alternatively, you can use environment variables:

# Set environment variables
export FILE_CONF_PATH="./my-config.json"
export CORE_HUB_URL="https://my-corehub:1717"
export DEFAULT_PASSWORD="my-secure-password"
export CREATE_ENTITIES_FROM_SCHEMA="true"
export TARGET_SCHEMA="public"
export SOURCE_TYPE="mssql"
export TARGET_TYPE="couchbase"

# Run the script
python main.py

Creating Entities

To create entities in the Core Hub based on your YAML configuration:

python create_all_entities.py --pipeline <pipeline_id> --source-schema <source_schema> --target-schema <target_schema> --source-type <source_agent_type> --target-type <target_agent_type> --yaml-file <path_to_yaml_config> --token <auth_token> [--skip-errors] [--chunk-size <number>]

Best Practices

  • Store configuration files in version control

  • Create different configuration files for development, staging, and production environments

  • Validate configurations before applying them to production systems

  • Use consistent naming conventions for schemas, tables, and columns

  • Document custom transformations and rules

  • Test configurations thoroughly before deployment

Troubleshooting

Common Issues

Issue Possible Cause Resolution

Authentication failures

Invalid or expired token

Generate a new authentication token

Entity creation errors

Incorrect schema configuration

Verify YAML configuration format and structure

Connection issues

Network connectivity problems

Check network settings and firewall rules

Type mapping errors

Incompatible data types between source and target

Verify column type definitions in the YAML configuration

References