Search docs.molo17.com

    Gluesync Conductor Gluesync conductor screenshot Repository

    Gluesync Conductor is a container management platform that provides a comprehensive RESTful API for managing Docker containers, with specialized integration for Gluesync agent orchestration and lifecycle management within containerized environments.

    Open Source This project is Open Source

    Overview

    Gluesync Conductor serves as the orchestration layer for Gluesync deployments by providing a centralized API for container management. It enables seamless agent deployment, configuration, and monitoring through integration with the Gluesync Core Hub, offering a streamlined experience for managing complex data pipeline infrastructures in containerized environments.

    Built with Node.js and designed for flexibility, Conductor provides consistent RESTful API responses and supports advanced container operations including version management, persistence tracking, and automated updates.

    Key Features

    • Container Management API

      • Complete RESTful API for Docker container operations

      • Container listing with detailed configuration and status information

      • Add, update, and remove containers programmatically

      • Container restart and image pulling capabilities

    • Agent Orchestration

      • Seamless integration with Gluesync Core Hub Web UI

      • Automated agent deployment from pipeline wizard

      • Dynamic agent provisioning based on pipeline requirements

      • Version checking and update management for containers

    • Configuration Management

      • Docker label-based container identification and tracking

      • Persistent container configuration through Docker Compose

      • Environment variable management and port mapping

      • Volume mounting and network configuration

    • Gluesync SDK Integration

      • Direct connection to CoreHub for advanced orchestration

      • License and security configuration management

      • Automatic service discovery and connection handling

      • UDP port range configuration for CoreHub communication

    • Monitoring & Operations

      • Real-time container status and health monitoring

      • Detailed logging and error handling

      • Container lifecycle management and persistence tracking

      • Support for multiple container runtimes (Docker, Podman rootful and rootless)

    Installation & prerequisites

    System requirements

    • Node.js: Version 16 or higher

    • Container Runtime: Docker and Docker Compose, or Podman (Linux) with Docker Compose v2

    • Package Manager: Yarn

    Quick start

    # Install dependencies
    cd packages/gluesync-conductor
    yarn install
    
    # Start the service
    PORT=50015 yarn dev

    Environment configuration

    For Gluesync SDK integration, configure the following environment variables:

    • GLUESYNC_LICENSE_FILE: Path to license file (default: /opt/gluesync/data/gs-license.dat)

    • GLUESYNC_SECURITY_CONFIG: Path to security config (default: /opt/gluesync/data/security-config.json)

    • COREHUB_UDP_PORT_RANGE: Port range for discovery (default: 1718-1727, recommended: 1717-1727)

    The following variables tune orchestration and updates (Conductor 0.14.0 and later). The kits set them where needed; change them only for a specific reason:

    • CONDUCTOR_ORCHESTRATOR: Forces the orchestrator, compose or swarm. When unset, Conductor uses Swarm mode if its own container belongs to a Swarm service, and Docker Compose mode otherwise. The Swarm kit sets it to swarm

    • CONDUCTOR_UPDATE_MIN_FREE_BYTES: Minimum free space, in bytes, on the disk of the Docker Compose or stack file. Below it, Conductor refuses an update before it changes anything and reports DISK_FULL (default: 536870912, 512 MiB)

    • CONDUCTOR_DOCKER_WAIT_MS: How long, in milliseconds, an update waits for the Docker API to come back when it drops during the update, before the update fails (default: 60000; 0 disables the wait)

    • UPDATE_VERIFY_SECONDS: How long, in seconds, an updated service must keep running (and healthy, when it has a health check) before the update counts as successful; otherwise it is rolled back (default: 20; 0 disables the check)

    • UPDATE_VERIFY_MAX_WAIT_SECONDS: Maximum time, in seconds, to wait for an updated service to become stable (default: 180; the Windows Server Swarm kit sets 600). In Swarm mode, the time spent pulling the image is not counted

    • UPDATE_PULL_MAX_WAIT_SECONDS: Swarm mode only. Maximum time, in seconds, that a new task can spend pulling its image during an update (default: 1200)

    • DOCKER_CLI_PATH: Swarm mode only. Absolute path of the docker CLI, which Conductor runs for docker stack deploy and docker service logs. When unset, Conductor looks in /usr/local/bin/docker, /usr/bin/docker and /bin/docker on Linux, and in C:\Windows\System32\docker.exe and C:\Program Files\docker\docker.exe on Windows; it never searches the PATH

    API endpoints

    Conductor provides a comprehensive REST API for container management:

    Container operations

    • GET /containers: List all containers with detailed information

    • GET /containers/id: Get specific container configuration

    • POST /containers: Add new containers to the system

    • PUT /containers/id: Update existing container configuration

    • POST /containers/id/restart: Restart a container

    • POST /containers/id/pull: Pull latest container image

    Version management

    • GET /containers/id/version: Check for available container updates

    • Integration with MOLO17 backoffice API for version information

    Docker labels & identification

    Conductor uses Docker labels to identify and track containers:

    Container labels

    • com.molo17.conductor.unique_id: Stores container nickname as unique identifier

    • com.molo17.conductor.versiontag: Tracks container version tag

    • com.molo17.conductor.type: Identifies container as Gluesync agent

    Persistence tracking

    Containers added through the API are automatically persisted to the Docker Compose configuration file, ensuring they survive system restarts and can be managed as part of the infrastructure.

    Integration with Core Hub

    Conductor integrates deeply with the Gluesync Core Hub to provide:

    • Web UI Integration: Direct agent deployment from pipeline creation wizard

    • Status Synchronization: Real-time agent status updates in Core Hub interface

    • Configuration Management: Centralized agent configuration through Web UI

    • Security Coordination: SSL certificate management and validation

    • Service Discovery: Automatic agent registration and discovery

    Container platform support

    Docker & Docker Compose

    Primary support for Docker environments:

    • Direct container management through Docker API

    • Docker Compose stack integration

    • Network configuration and service linking

    • Volume management and persistent storage

    Docker Swarm

    From release 0.14.0, Conductor also manages a Gluesync kit deployed as a Docker Swarm stack, on Linux and on Windows Server 2022. The API is the same as with Docker Compose. Conductor must run on a Swarm manager, and works on the stack services instead of single containers:

    • Updates with automatic rollback when the new version does not become stable

    • Self-update only to a newer version, so switching the release channel from alpha to beta or GA keeps Swarm support

    • Image pinned to its digest after an update, so redeploying the stack does not restart the updated services

    • The stack file (docker-stack.yml) as the source of truth, updated with the new image tags

    See Deploy Gluesync on Docker Swarm for the installation and the full behaviour.

    Podman

    Conductor runs on Podman 4.9 and later, rootful and rootless, on Linux. It talks to Podman through its Docker-compatible API socket and drives the stack with Docker Compose v2, the same implementation it uses on Docker. podman-compose is not used. The API and the response shapes are the same on both engines. To install Gluesync on Podman, see Gluesync on Podman.

    • Engine detection. From the engine’s GET /version and GET /info, Conductor works out whether the engine is Docker or Podman, whether it is rootless, and its version, once per process. Conductor’s own GET /version and GET /info return it in a data.containerEngine field.

    • Socket. Helper containers (self-update, restart of the whole stack) mount the socket by its path on the host, so Conductor resolves that path in this order: GS_CONTAINER_SOCKET in its environment, the source of the bind mount of its own container on /var/run/docker.sock, then /var/run/docker.sock. The installer writes GS_CONTAINER_SOCKET (/run/podman/podman.sock rootful, /run/user/<uid>/podman/podman.sock rootless) to the .env of the kit, and Conductor gives that file to every Compose command it runs (--env-file).

    • Image names. Podman reports images fully qualified (docker.io/molo17/x:1). Conductor compares and reports them in their short form (molo17/x:1), so a rollback never mistakes a running service for one on another image.

    • SELinux. Conductor keeps the z and Z options of a volume, adds :z to the agent folders it creates, and starts its helper containers with label=disable. Missing bind folders are created before compose up, because Podman, unlike Docker, does not create them.

    • Registry lookups. Podman has no GET /distribution/{name}/json. When the engine answers 404 or 501, Conductor asks Docker Hub itself for the manifest of a Docker Hub image.

    • Docker Swarm is not supported. Podman has no Swarm mode. With CONDUCTOR_ORCHESTRATOR=swarm on Podman, every route except /health answers HTTP 501 with the diagnosis SWARM_UNSUPPORTED_ON_PODMAN and no action runs. Leave CONDUCTOR_ORCHESTRATOR unset, or set it to compose. See Conductor error codes.

    Kubernetes

    Conductor does not run on Kubernetes. There, the Gluesync Kubernetes Operator (GKO) serves the part of the Conductor API that Platform updates in the Core Hub UI uses, and connects to Core Hub as its Conductor module. Updates work from the same screen, and Core Hub’s running pipelines are put into maintenance mode while Core Hub changes.

    Best practices

    • Version Management: Regularly check for and apply container updates

    • Resource Allocation: Configure appropriate CPU and memory limits

    • Persistence: Ensure critical containers are properly persisted

    • Monitoring: Implement logging and monitoring for container health

    • Security: Use SSL/TLS for API communications in production

    • Backup: Maintain backups of Docker Compose configurations

    Troubleshooting

    Common issues

    • Connection Failures: Verify Docker daemon accessibility and API connectivity

    • Version Conflicts: Check container image versions and compatibility

    • Resource Constraints: Monitor system resources and container limits

    • Network Issues: Validate network configuration and port mappings

    • Persistence Problems: Ensure Docker Compose files are properly maintained

    When a request fails, Conductor returns a diagnosis code that says who has to act and what to do. See Conductor error codes.

    Diagnostic tools

    • Container logs: docker logs <container_id>

    • API health checks: Test endpoints with curl or API client

    • Docker Compose status: docker-compose ps

    • Network diagnostics: Container networking verification

    Development

    Conductor includes development tools for code quality and testing:

    # Code quality checks
    yarn lint
    yarn type-check
    
    # Testing
    yarn test
    
    # Documentation
    yarn docs

    Changelog

    See the Conductor changelog for the release history.