Search docs.molo17.com
    A newer version of this documentation is available.
    View Latest (v2.3)

    Bootstrapper Static Repository

    Gluesync Bootstrapper is a configuration tool for setting up database schema mappings between source and target systems in an automated way, bypassing the need to manually configure entities in the Gluesync Core Hub Web UI.

    Open Source This project is Open Source

    Overview

    The Bootstrapper allows you to define table structures, column mappings, and connection properties using YAML configuration files, enabling precise control over how data is transferred and transformed across different database systems. This automation tool integrates directly with the Core Hub API to streamline entity configuration.

    Key Features

    • Real-time Data Synchronization

      • Automated synchronization between source and target databases

      • Support for multiple database systems (MS SQL Server, Couchbase, etc.)

      • Configurable filtering and transformation rules

    • Schema Configuration

      • Table structure mapping

      • Column mapping with type safety

      • Custom document key generation

      • TTL (Time To Live) management for target documents

      • Schedules for recurring jobs (via Chronos)

      • …​and much more

    • Pipeline Management

      • Flexible polling intervals and batch processing

      • Automatic agent configuration

      • Direct Core Hub API integration

    Configuration

    The Bootstrapper works with two primary configuration files:

    1. Schema configuration (table-list-template.yaml)

    2. Agent configuration (config.json)

    Schema Configuration

    The schema configuration defines how tables should be synchronized and transformed:

    dbo: # Source schema
  target: public # Target schema
  customProperties: # Custom properties for both source and target
    source: # Source custom properties
      maxItemsCountPerIteration: 1000
      pollingIntervalMilliseconds: 100
    target: # Target custom properties
      ttlValue: 10000000
  tables: # Tables to be synchronized
    whitelist: # Allowed tables to be synchronized, empty list means all tables
      - DRIVERS
      - VEHICLES
    blacklist: # Forbidden tables to be synchronized, empty list means no restrictions
      - TEST

    Table Configuration

    Each table can be configured with detailed mapping information:

    DRIVERS:
  keys:
    - ID
  documentKey:
    prefix: "TESTPREFIX"
    suffix: "TESTSUFFIX"
    separator: "-"
    keys:
      - ID
      - LAST_NAME
  name: drivers
  columns:
    - FIRST_NAME: "FIRST_NAME"
    - LAST_NAME: "LAST_NAME"
    - AGE: "AGE"
    - EMAIL: "EMAIL"

    Agent Configuration

    The agent configuration defines the connection details for source and target databases:

    {
  "agents": [
    {
      "agentType": "SOURCE",
      "agentTag": "mssql-ct",
      "hostCredentials": {
        "connectionName": "MS SQL Server source",
        "host": "localhost",
        "port": 1433,
        "databaseName": "demo"
      }
    }
  ]
}

    Usage

    Getting Started

    Before using the Bootstrapper, you need:

    1. A running Gluesync deployment with Core Hub accessible

    2. Authentication credentials for the Core Hub API

    3. Proper network connectivity between source and target systems

    Authentication

    To use the tool, you need to get a valid authentication token from the Core Hub API. This can be done in several ways:

    1. By logging in to the Core Hub Web UI and copying the token from the browser’s cookies

    2. By using an HTTP client to request a token

    curl -X POST https://<corehub-url>/authentication/login -H "Content-Type: application/json" -d '{"username":"<username>","password":"<password>"}'

    Main Script Usage

    The primary entry point is the main.py script which creates a new pipeline and configures the agents:

    python main.py --config <path_to_config> --token <auth_token> [--skip-errors] [--chunk-size <number>]

    Alternatively, you can use environment variables:

    # Set environment variables
export FILE_CONF_PATH="./my-config.json"
export CORE_HUB_URL="https://my-corehub:1717"
export DEFAULT_PASSWORD="my-secure-password"
export CREATE_ENTITIES_FROM_SCHEMA="true"
export TARGET_SCHEMA="public"
export SOURCE_TYPE="mssql"
export TARGET_TYPE="couchbase"

# Run the script
python main.py

    Creating Entities

    To create entities in the Core Hub based on your YAML configuration:

    python create_all_entities.py --pipeline <pipeline_id> --source-schema <source_schema> --target-schema <target_schema> --source-type <source_agent_type> --target-type <target_agent_type> --yaml-file <path_to_yaml_config> --token <auth_token> [--skip-errors] [--chunk-size <number>]

    Best Practices

    • Store configuration files in version control

    • Create different configuration files for development, staging, and production environments

    • Validate configurations before applying them to production systems

    • Use consistent naming conventions for schemas, tables, and columns

    • Document custom transformations and rules

    • Test configurations thoroughly before deployment

    Troubleshooting

    Common Issues

    Issue Possible Cause Resolution

    Authentication failures

    Invalid or expired token

    Generate a new authentication token

    Entity creation errors

    Incorrect schema configuration

    Verify YAML configuration format and structure

    Connection issues

    Network connectivity problems

    Check network settings and firewall rules

    Type mapping errors

    Incompatible data types between source and target

    Verify column type definitions in the YAML configuration

    References