Search docs.molo17.com
    A newer version of this documentation is available.
    View Latest (v2.3)

    Gluesync Architecture

    Gluesync’s architecture is built on a flexible, agent-based system centered around the Core Hub, with extensibility through SDKs and modules. This design enables real-time data integration across diverse platforms while ensuring scalability, reliability, and security.

    Architecture Overview

    Gluesync 2 Architecture

    The architecture consists of five main components:

    • Source Agents: Handle data extraction and change capture

    • Core Hub: Orchestrates data flow and system operations

    • Target Agents: Manage data loading and transformation

    • SDKs: Enable developer integration with Core Hub

    • Modules: Extend platform functionality through SDK-based components

    Core Components

    Core Hub

    The Core Hub is the central orchestrator, featuring:

    • High Availability

      • Core Hub #1 and #2 for redundancy

      • Automatic failover support

      • Load balancing capabilities

    • Data Processing

      • Aggregation engine

      • Denormalization support

      • Transaction coordination

    • Management

      • Administration interface

      • Monitoring dashboard

      • Security controls

    Source Agents

    Agent Type Capabilities

    NoSQL CDC

    * Natively integrated NoSQL CDC * Real-time change capture * Event-based replication

    RDBMS

    * Transaction-log based CDC * SQL database support * Real-time monitoring

    Data Lakes

    * AWS S3 & S3-like support * Dell ECS integration * Azure blob storage support

    Target Agents

    Agent Type Features

    NoSQL

    * Native integration * Load balancing * Multi-target support

    RDBMS

    * JDBC-oriented connections * Transaction pool management * Bulk loading capabilities

    Data Lakes & Streaming

    * S3 and blob storage support * Kafka integration * Solace messaging support

    Communication Flow

    Data Flow Diagram

    diagram

    • WebSocket connections enable real-time bidirectional communication

    • Core Hub processes and transforms data in-flight

    • Multi-target support allows for parallel data distribution

    Advanced Features

    Data Processing

    • Aggregation Engine

      • Real-time data aggregation

      • Custom aggregation rules

      • Performance optimization

    Denormalization

    • Smart Denormalization

      • Configurable strategies

      • Automatic schema mapping

      • Performance tuning

    Multi-Source Support

    • Event-based Integration

      • Multiple source connections

      • Parallel processing

      • Consistent ordering

    Deployment Options

    Flexible Deployment Models

    Model Description Best For

    On-Premises

    Full deployment within your infrastructure

    High security requirements

    Cloud

    Deployment on AWS, Azure, or GCP

    Scalability and flexibility

    Hybrid

    Mix of on-premises and cloud components

    Balanced approach

    Security Architecture

    Security Measures

    • Network Security

      • TLS encryption

      • Secure WebSocket connections

      • Network isolation options

    • Authentication

      • API key authentication

      • Role-based access control

      • Session management

    • Data Protection

      • In-transit encryption

      • Secure credential storage

      • Audit logging

    SDKs and Developer Integration

    Open Source SDKs

    Gluesync provides open source SDKs that enable developers to build custom integrations and extensions:

    SDK Features

    Python SDK

    * Core Hub handshake protocol implementation * Authentication and authorization * Event handling and processing * Available at GitLab

    Node.js SDK

    * JavaScript-based Core Hub integration * Real-time event processing * Promise-based API design * Available at GitLab

    Coming Soon

    * Java SDK * Kotlin SDK

    Developer Benefits

    • Open Platform

      • Build custom agents and modules

      • Extend Core Hub functionality

      • Access Gluesync APIs securely

    • Integration Support

      • Comprehensive documentation

      • Reference implementations

      • Community-supported examples

    Modules

    Platform Extensions

    Modules are platform extensions built on top of Gluesync SDKs that enhance system capabilities:

    Module Functionality

    Chronos

    * Advanced scheduling capabilities * Time-based job orchestration * Recurring task management * Available at GitLab

    Bootstrapper

    * System initialization * Configuration management * Deployment automation * Available at GitLab

    Module Architecture

    diagram

    • Modules connect to Core Hub through SDK interfaces

    • Custom modules can extend platform capabilities

    • Open architecture enables third-party development

    Monitoring & Administration

    Built-in Monitoring

    • Real-time metrics collection

    • Performance monitoring

    • Resource utilization tracking

    • Alert management

    Administration Tools

    • Web-based admin interface

    • REST API access

    • Configuration management

    • System health monitoring

    For detailed deployment instructions, see our Deployment Guide for Docker Compose and Kubernetes.