Gluesync Conductor

Gluesync Conductor is a container management platform that provides a comprehensive RESTful API for managing Docker containers, with specialized integration for Gluesync agent orchestration and lifecycle management within containerized environments.
This project is Open Source
Overview
Gluesync Conductor serves as the orchestration layer for Gluesync deployments by providing a centralized API for container management. It enables seamless agent deployment, configuration, and monitoring through integration with the Gluesync Core Hub, offering a streamlined experience for managing complex data pipeline infrastructures in containerized environments.
Built with Node.js and designed for flexibility, Conductor provides consistent RESTful API responses and supports advanced container operations including version management, persistence tracking, and automated updates.
Key Features
-
Container Management API
-
Complete RESTful API for Docker container operations
-
Container listing with detailed configuration and status information
-
Add, update, and remove containers programmatically
-
Container restart and image pulling capabilities
-
-
Agent Orchestration
-
Seamless integration with Gluesync Core Hub Web UI
-
Automated agent deployment from pipeline wizard
-
Dynamic agent provisioning based on pipeline requirements
-
Version checking and update management for containers
-
-
Configuration Management
-
Docker label-based container identification and tracking
-
Persistent container configuration through Docker Compose
-
Environment variable management and port mapping
-
Volume mounting and network configuration
-
-
Gluesync SDK Integration
-
Direct connection to CoreHub for advanced orchestration
-
License and security configuration management
-
Automatic service discovery and connection handling
-
UDP port range configuration for CoreHub communication
-
-
Monitoring & Operations
-
Real-time container status and health monitoring
-
Detailed logging and error handling
-
Container lifecycle management and persistence tracking
-
Support for multiple container runtimes (Docker, Podman rootful and rootless)
-
Installation & prerequisites
System requirements
-
Node.js: Version 16 or higher
-
Container Runtime: Docker and Docker Compose, or Podman (Linux) with Docker Compose v2
-
Package Manager: Yarn
Quick start
# Install dependencies
cd packages/gluesync-conductor
yarn install
# Start the service
PORT=50015 yarn dev
Environment configuration
For Gluesync SDK integration, configure the following environment variables:
-
GLUESYNC_LICENSE_FILE: Path to license file (default:/opt/gluesync/data/gs-license.dat) -
GLUESYNC_SECURITY_CONFIG: Path to security config (default:/opt/gluesync/data/security-config.json) -
COREHUB_UDP_PORT_RANGE: Port range for discovery (default:1718-1727, recommended:1717-1727)
The following variables tune orchestration and updates (Conductor 0.14.0 and later). The kits set them where needed; change them only for a specific reason:
-
CONDUCTOR_ORCHESTRATOR: Forces the orchestrator,composeorswarm. When unset, Conductor uses Swarm mode if its own container belongs to a Swarm service, and Docker Compose mode otherwise. The Swarm kit sets it toswarm -
CONDUCTOR_UPDATE_MIN_FREE_BYTES: Minimum free space, in bytes, on the disk of the Docker Compose or stack file. Below it, Conductor refuses an update before it changes anything and reportsDISK_FULL(default:536870912, 512 MiB) -
CONDUCTOR_DOCKER_WAIT_MS: How long, in milliseconds, an update waits for the Docker API to come back when it drops during the update, before the update fails (default:60000;0disables the wait) -
UPDATE_VERIFY_SECONDS: How long, in seconds, an updated service must keep running (and healthy, when it has a health check) before the update counts as successful; otherwise it is rolled back (default:20;0disables the check) -
UPDATE_VERIFY_MAX_WAIT_SECONDS: Maximum time, in seconds, to wait for an updated service to become stable (default:180; the Windows Server Swarm kit sets600). In Swarm mode, the time spent pulling the image is not counted -
UPDATE_PULL_MAX_WAIT_SECONDS: Swarm mode only. Maximum time, in seconds, that a new task can spend pulling its image during an update (default:1200) -
DOCKER_CLI_PATH: Swarm mode only. Absolute path of thedockerCLI, which Conductor runs fordocker stack deployanddocker service logs. When unset, Conductor looks in/usr/local/bin/docker,/usr/bin/dockerand/bin/dockeron Linux, and inC:\Windows\System32\docker.exeandC:\Program Files\docker\docker.exeon Windows; it never searches thePATH
API endpoints
Conductor provides a comprehensive REST API for container management:
Container operations
-
GET /containers: List all containers with detailed information
-
GET /containers/id: Get specific container configuration
-
POST /containers: Add new containers to the system
-
PUT /containers/id: Update existing container configuration
-
POST /containers/id/restart: Restart a container
-
POST /containers/id/pull: Pull latest container image
Docker labels & identification
Integration with Core Hub
Conductor integrates deeply with the Gluesync Core Hub to provide:
-
Web UI Integration: Direct agent deployment from pipeline creation wizard
-
Status Synchronization: Real-time agent status updates in Core Hub interface
-
Configuration Management: Centralized agent configuration through Web UI
-
Security Coordination: SSL certificate management and validation
-
Service Discovery: Automatic agent registration and discovery
Container platform support
Docker & Docker Compose
Primary support for Docker environments:
-
Direct container management through Docker API
-
Docker Compose stack integration
-
Network configuration and service linking
-
Volume management and persistent storage
Docker Swarm
From release 0.14.0, Conductor also manages a Gluesync kit deployed as a Docker Swarm stack, on Linux and on Windows Server 2022. The API is the same as with Docker Compose. Conductor must run on a Swarm manager, and works on the stack services instead of single containers:
-
Updates with automatic rollback when the new version does not become stable
-
Self-update only to a newer version, so switching the release channel from alpha to beta or GA keeps Swarm support
-
Image pinned to its digest after an update, so redeploying the stack does not restart the updated services
-
The stack file (
docker-stack.yml) as the source of truth, updated with the new image tags
See Deploy Gluesync on Docker Swarm for the installation and the full behaviour.
Podman
Conductor runs on Podman 4.9 and later, rootful and rootless, on Linux. It talks to Podman through its Docker-compatible API socket and drives the stack with Docker Compose v2, the same implementation it uses on Docker. podman-compose is not used. The API and the response shapes are the same on both engines. To install Gluesync on Podman, see Gluesync on Podman.
-
Engine detection. From the engine’s
GET /versionandGET /info, Conductor works out whether the engine is Docker or Podman, whether it is rootless, and its version, once per process. Conductor’s ownGET /versionandGET /inforeturn it in adata.containerEnginefield. -
Socket. Helper containers (self-update, restart of the whole stack) mount the socket by its path on the host, so Conductor resolves that path in this order:
GS_CONTAINER_SOCKETin its environment, the source of the bind mount of its own container on/var/run/docker.sock, then/var/run/docker.sock. The installer writesGS_CONTAINER_SOCKET(/run/podman/podman.sockrootful,/run/user/<uid>/podman/podman.sockrootless) to the.envof the kit, and Conductor gives that file to every Compose command it runs (--env-file). -
Image names. Podman reports images fully qualified (
docker.io/molo17/x:1). Conductor compares and reports them in their short form (molo17/x:1), so a rollback never mistakes a running service for one on another image. -
SELinux. Conductor keeps the
zandZoptions of a volume, adds:zto the agent folders it creates, and starts its helper containers withlabel=disable. Missing bind folders are created beforecompose up, because Podman, unlike Docker, does not create them. -
Registry lookups. Podman has no
GET /distribution/{name}/json. When the engine answers 404 or 501, Conductor asks Docker Hub itself for the manifest of a Docker Hub image. -
Docker Swarm is not supported. Podman has no Swarm mode. With
CONDUCTOR_ORCHESTRATOR=swarmon Podman, every route except/healthanswers HTTP501with the diagnosisSWARM_UNSUPPORTED_ON_PODMANand no action runs. LeaveCONDUCTOR_ORCHESTRATORunset, or set it tocompose. See Conductor error codes.
Kubernetes
Conductor does not run on Kubernetes. There, the Gluesync Kubernetes Operator (GKO) serves the part of the Conductor API that Platform updates in the Core Hub UI uses, and connects to Core Hub as its Conductor module. Updates work from the same screen, and Core Hub’s running pipelines are put into maintenance mode while Core Hub changes.
Best practices
-
Version Management: Regularly check for and apply container updates
-
Resource Allocation: Configure appropriate CPU and memory limits
-
Persistence: Ensure critical containers are properly persisted
-
Monitoring: Implement logging and monitoring for container health
-
Security: Use SSL/TLS for API communications in production
-
Backup: Maintain backups of Docker Compose configurations
Troubleshooting
Common issues
-
Connection Failures: Verify Docker daemon accessibility and API connectivity
-
Version Conflicts: Check container image versions and compatibility
-
Resource Constraints: Monitor system resources and container limits
-
Network Issues: Validate network configuration and port mappings
-
Persistence Problems: Ensure Docker Compose files are properly maintained
When a request fails, Conductor returns a diagnosis code that says who has to act and what to do. See Conductor error codes.
Development
Conductor includes development tools for code quality and testing:
# Code quality checks
yarn lint
yarn type-check
# Testing
yarn test
# Documentation
yarn docs
Changelog
See the Conductor changelog for the release history.