Search docs.molo17.com
    Operational guide Deploy and run

    Free disk space and manage cache and logs

    A Gluesync host that runs out of disk space stops replicating, and the Control Plane does not warn you before it happens. This guide shows what grows on the disk, how to see it, which folders are safe to clean and which are never safe, how to cap logs and the source cache, what to do when the disk is already full, and how to avoid a failed update.

    In this guide

    You will learn how to see what uses the disk, free space safely from the Control Plane and from the host, cap the Core Hub log, the source agent cache, notifications and webhook logs, check the Prometheus and Docker limits, recover from a full disk in the right order, understand the free-space check that runs before an update, and watch free space so that you are warned early.

    Quick information

    Goal

    Keep the Gluesync disk from filling up, and recover without losing changes when it has

    Audience

    Administrators who run Gluesync with Docker Compose on Linux or Windows Server

    Level

    Intermediate

    Time required

    About 30 minutes

    Area

    Control Plane → Settings → Support, Settings → Notifications, pipeline → Edit source; host shell

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You can sign in to the Gluesync Control Plane with a role that can change settings;

    2. You have shell access with administrator rights to the host that runs Gluesync (root or sudo on Linux, an elevated PowerShell on Windows Server);

    3. You know the Gluesync folder, the folder that contains docker-compose.yml (for example /opt/gluesync on Linux or C:\Program Files\Gluesync on Windows Server). The paths below are relative to it;

    4. You have at least one pipeline with a source agent that keeps a local cache (this guide uses a MySQL source agent).

    The Control Plane does not show how much disk space is free. In the tests behind this guide, a disk with less than 1 MB free raised no notification and no system condition, and the Core Hub metrics, the Core Hub Grafana dashboard and the Control Plane have no disk gauge. Use the host commands in this guide, and set up monitoring in Watch free space so that you are warned early.

    What grows on the disk

    The table lists every place where a Gluesync installation writes, with what is safe to do. Paths are relative to the Gluesync folder.

    Place What it holds and when it grows Limit you can set Clean by hand?

    data/core-hub/agents/<agent-id>/cdc_arena/

    The source agent cache: changes read from the source and not yet delivered to the target. Created by the agents that cache locally (in the tests: MySQL). The files grow in steps of 64 MiB when the target, Core Hub or the network cannot keep up with the source.

    Source change retention in hours on the source agent (Cap the source cache and keep it draining)

    No. Deleting it loses the undelivered changes without any error (see Cap the source cache and keep it draining).

    data/core-hub/agents/<agent-id>/ (other files, for example mysqlBinlog)

    Reader checkpoint and agent bootstrap file. Tiny.

    None needed

    No. The checkpoint tells the agent where to continue.

    data/core-hub/database/

    Core Hub’s own databases: pipelines and settings, notifications, metrics, AI and data comparison. About 5 MB after an hour with one busy pipeline.

    Notification and webhook retention (Trim notifications and webhook logs)

    No. It holds your configuration. Use a backup if you need a copy.

    data/core-hub/connections/

    Connection assets such as uploaded certificates.

    None needed

    No.

    logs/core-hub/gluesync-core-hub.log and logs/core-hub/archived/

    The Core Hub log. It rolls over at 100 MB per file, the rolled files are compressed (in the lab, a 105 MB log became archived/gluesync-core-hub.2026-10-28.0.log.gz of 9 MB), and old ones are deleted by age and by total size.

    Log level, Max history (Days) and Total size cap (GB) (Cap the Core Hub log and the Docker logs)

    Yes, with Clear Logs. Do not delete the active log with rm (Cap the Core Hub log and the Docker logs).

    logs/gluesync-conductor.log and logs/chronos/

    Logs of the Conductor and Chronos modules. The Conductor log grew by about 3 MB per hour in the lab.

    Check the size with du

    Yes, after you stop the module.

    Docker images

    About 8 GB for the default kit (Core Hub 3.3 GB, Chronos 2.4 GB, Grafana 1.4 GB, Conductor 0.4 GB, Prometheus 0.4 GB, Traefik 0.2 GB). Each update adds the new images next to the old ones.

    Remove images that no container uses (Free space safely right now)

    Yes, unused images only.

    Docker container logs

    Without a limit, Docker keeps every line the containers print. In a test, a container that printed 40 MB left a single 40 MB file.

    Docker log rotation (Cap the Core Hub log and the Docker logs)

    Cap them. Do not delete files inside Docker’s own folder by hand.

    Prometheus data

    The metrics database. It lives inside the Prometheus container, not in a folder of the kit. About 19 MB per hour with 6,000 series and a 5-second scrape in the lab.

    --storage.tsdb.retention.size and --storage.tsdb.retention.time (Check the Prometheus limits)

    Cap it. Do not delete it while Prometheus runs.

    Grafana

    About 86 MB inside the Grafana container, almost all of it static.

    None needed

    Leave it.

    Guide steps

    See how much space is free and what uses it

    CLI

    1. On the host, go to the Gluesync folder;

    2. Check the free space of the disk that holds it;

    3. Show the size of each Gluesync folder;

    4. Find the biggest files in the agent folders;

    5. Show what Docker uses.

    On Linux:

    cd /opt/gluesync
df -h .
du -sh data/core-hub/* logs
du -ah data/core-hub/agents | sort -h | tail -n 10
docker system df

    On Windows Server, use an elevated PowerShell. Get-PSDrive C shows the free space of the disk, and docker system df shows what Docker uses:

    cd "C:\Program Files\Gluesync"
Get-PSDrive C
docker system df

    Example output of the du commands in the lab:

    33M     data/core-hub/agents
6.0K    data/core-hub/connections
5.5M    data/core-hub/database
49K     logs
    Expected result

    You know the free space of the disk and which row of the table in What grows on the disk is the big one. A large cdc_arena folder means a cache backlog, a large logs folder means logs, and a large Docker figure means images or container logs.

    Optional, for advanced users

    Core Hub writes a hardware summary to its log once a minute, with a line such as Disk : 353 MB used / 364 MB total (11 MB usable). It measures the volume that holds the Core Hub data folder. You do not need it to follow this guide.
    Free space safely right now

    UI CLI

    Start with the actions that cannot lose data, and stop as soon as you have enough space.

    1. In the Control Plane, select the gear icon in the top bar and open Support;

    2. Scroll to Clear Logs and select it;

    3. In the Confirm Clear Logs dialog, tick Also remove previously generated support archives only if you have already uploaded or saved them;

    4. Select Confirm.

    Confirm Clear Logs dialog with the option to also remove previously generated support archives
    Figure 1. Confirm that the active log is emptied and the rolled archives are deleted.
    Clear Logs result with the Cleared message
    Figure 2. The result lists the files emptied, the archives deleted and the space freed.

    Then, on the host, remove the Docker images that no container uses:

    docker image prune -a
    Expected result

    Clear Logs shows Cleared! with the number of files emptied, archives deleted and the space freed. df -h . shows more free space. Gluesync keeps running.

    docker image prune -a deletes every image that no container uses, including the images of stopped containers you may want to start again. Do not add --volumes or run docker system prune --volumes on a host that runs other containers, because that deletes Docker volumes.

    Clear Logs works even when the disk is at 100 percent, because it truncates the active file in place. In a test with 0 bytes free, it still ran and gave the space back. Changing a setting at that point fails with Request failed with status code 500, so free space first and change settings afterwards.
    Cap the Core Hub log and the Docker logs

    UI Config

    1. Open Settings → Support and scroll to Log level settings;

    2. Select INFO in Log level. The change applies at once, without a Save;

    3. Scroll to Log retention settings;

    4. Enter Max history (Days), for example 3;

    5. Enter Total size cap (GB), for example 5;

    6. Select Save.

    Settings
    Figure 3. Set the log level and the log retention.

    Then cap the container logs of Docker itself. Follow Log rotation in the Docker installation guide, which merges max-size and max-file into the Docker daemon configuration. Docker applies the new limits only to containers that are created afterwards, so recreate the Gluesync containers (an update does it) for the change to take effect.

    Expected result

    Core Hub applies the retention values at once and again at every start (it logs Configuring log retention: maxHistory=3 days, totalSizeCap=5GB). A container created with max-size and max-file never keeps more than that: in a test with max-size=5m and max-file=2, a container that printed 40 MB kept a 5 MB file and a small current file, where a container without limits kept a single 40 MB file.

    The values in Log retention settings are the ones Core Hub applies, also at every start. They win over the maxHistory and totalSizeCap written in shared/logback.xml. In the lab kit the file says 5 days and 5 GB, and Core Hub still started with maxHistory=3 days, the value from the Control Plane.

    Do not delete the active log file with rm while Gluesync runs. In a test, the file disappeared, Core Hub kept the deleted file open ((deleted) in the list of its open files), and no new log file appeared until the next restart. Use Clear Logs instead.

    DEBUG is the default level in the kit. At idle, a Core Hub at DEBUG wrote about 14 KB of log per minute, and at INFO about 6.6 KB per minute. A busy pipeline writes much more. Keep DEBUG for the time you need it, and read Logging for the other options.
    Cap the source cache and keep it draining

    UI

    The source agent writes every change to its local cache before it sends the change to Core Hub and the target. The cache grows when the target, Core Hub or the network cannot keep up, and it shrinks again when the target catches up.

    1. Open the pipeline and select the gear icon above the entity table;

    2. Select Edit source;

    3. Open the Advanced settings tab;

    4. Set the key Source change retention in hours. If the key is not in the list, select Add property and choose it. The default is 24. Enter a lower value, for example 8;

    5. Select Test and continue.

    Edit source page for the MySQL agent
    Figure 4. Lower Source change retention in hours on the source agent.
    Expected result

    You return to the pipeline page and the entity stays Active. No restart is needed. The agent keeps changes for the number of hours you set and evicts older ones, as described in Source advanced settings.

    Lowering the value limits how much disk the cache can use, and it limits how long Core Hub or the target can be down before the agent drops changes. Choose a value longer than your longest expected outage, then plan a snapshot for anything older.

    What fills the cache, and how it drains. In the tests, with the target database stopped and a MySQL source that kept receiving rows (about 2.9 KB each):

    • The cache file of the entity grew from 16 MiB in steps of 64 MiB, to 400 MiB after 100,000 rows;

    • The Control Plane showed no warning while it grew;

    • After the target started again, the backlog drained on its own, the file went back to 16 MiB and about 340 MB of disk space was released, without any manual cleaning;

    • An entity that was stopped did not add to the cache of the MySQL agent. The changes stayed in the source database log until the entity was started again, and then reached the target.

    Never delete the cdc_arena folder to free space while it holds changes that are not delivered. In a test, with a backlog of 20,000 rows, Core Hub was stopped, cdc_arena was deleted and Core Hub was started again. The target stayed 20,000 rows short of the source. Core Hub showed no error and raised no notification, and new changes continued to arrive normally. The only way to get the lost rows is a new snapshot. Remove cache files only when you accept a new snapshot, and only while Core Hub is stopped.
    Trim notifications and webhook logs

    UI

    These are small, but they have no other limit, so set them to what you need.

    1. Open Settings → Notifications;

    2. Open the Notifications hub tab;

    3. Set Retention value and Retention unit. The default is 2 Day;

    4. Select Save.

    Notifications hub tab with annotation 1 on Retention value and Retention unit and annotation 2 on the Save button
    Figure 5. Set how long notifications are kept.

    1. Open the Webhooks tab and scroll to Retention;

    2. Set Max delivery logs, Max dead letters, Max retention days and Max dead letter days. The defaults are 10000, 2000, 7 and 30;

    3. Select Save.

    Webhooks tab with annotation 1 on the four Retention fields and annotation 2 on the Save button
    Figure 6. Set how many webhook delivery logs and dead letters are kept, and for how long.
    Expected result

    The values stay in the fields after you reopen the tab. Core Hub purges older notifications and webhook records on its own. Use Purge logs on the same tab to delete the delivery logs at once.

    Check the Prometheus limits

    CLI Config

    The default kit starts Prometheus with a size cap and a time cap. Prometheus deletes the oldest data when either one is reached.

    1. Open docker-compose.yml and find the prometheus service. The default kit sets:

      command:
  - --storage.tsdb.retention.size=5GB
  - --storage.tsdb.retention.time=5d

    2. Confirm what Prometheus runs with:

      curl -sk https://<host>/prometheus/api/v1/status/flags | tr ',' '\n' | grep retention

    3. To change the limits, edit the two values and recreate the container with docker compose up -d prometheus.

    Expected result

    The reply shows the active values, for example "storage.tsdb.retention.size":"5GiB" and "storage.tsdb.retention.time":"5d". In the lab, 5 days of metrics need about 2.3 GB (19 MB per hour), well below the 5 GB cap.

    In the default kit the Prometheus data folder is inside the container: the line ./data/prometheus:/prometheus is commented out. Read Prometheus integration before you move it to a volume.
    Recover when the disk is already full

    CLI

    What you see depends on what filled the disk.

    The source cache filled it. In the tests, Core Hub crashed: the Java process ended with a SIGBUS error (exit code 134) when the cache needed more space, Docker restarted the container again and again, and the Control Plane did not answer. No change was lost. Freeing 15 MB was not enough: Core Hub used it in seconds and crashed again. Freeing or adding a few hundred MB, with the target running, let it start and drain the cache.

    Something else filled it, and the cache did not need to grow. Core Hub kept running and you could sign in. Saving a setting failed with an error, as in the screenshot, and Clear Logs still worked.

    Log retention settings with the Save button shown as Failed and the message Request failed with status code 500
    Figure 7. A setting cannot be saved when the disk is full.

    Recover in this order:

    1. Do not delete anything inside data/core-hub yet;

    2. If Core Hub restarts again and again, stop it so that it does not fail repeatedly:

      docker compose stop gluesync-core-hub

    3. Make room without touching Gluesync state: delete the rolled logs in logs/core-hub/archived/ (Core Hub is stopped), remove unused images (Free space safely right now), delete files that are not Gluesync’s, or add disk space. Adding space was the cleanest recovery in the tests;

    4. Make sure the target database is running and reachable;

    5. Start Core Hub again:

      docker compose start gluesync-core-hub

    6. Sign in, open the pipeline and wait while the entities catch up. The cache drains and its files shrink by themselves;

    7. Only when there is no other way to get space, and you accept a new snapshot, stop Core Hub and remove the cache files of the entity (Cap the source cache and keep it draining).

    Expected result

    The Control Plane answers again, the entities are Active, the target row counts match the source, and df -h . shows free space. The cdc_arena files are back to their starting size.

    A crash can cut a batch in half. In a test, after the crash an entity showed On error and paused, and the target reported Key (id)=(90001) already exists because the rows were sent again. Check the entity error, read Insert conflict strategy, and start the entity again once the cause is clear.
    Understand the free-space check before an update

    API Config

    An update downloads several GB of images and rewrites docker-compose.yml. A disk that fills up halfway can leave that file empty (0 bytes), and then neither the update nor a restart can read it. Recent Conductor versions therefore check the free space first, before they download or change anything.

    In a test, with the Gluesync folder on a 64 MB disk, an update request was refused at once with HTTP 507 and the error DISK_FULL, and docker-compose.yml was byte-for-byte unchanged:

    {
  "success": false,
  "error": "Failed to do containers action",
  "details": "not enough free disk space to write /mnt/kit/docker-compose.yml: need 536870912 bytes, 55992320 available",
  "diagnosis": {
    "code": "DISK_FULL",
    "params": { "path": "/mnt/kit/docker-compose.yml", "neededBytes": 536870912, "availableBytes": 55992320 },
    "resolutionCodes": ["FREE_DISK_SPACE", "CONTACT_SUPPORT_WITH_DETAILS"]
  }
}

    The check works like this:

    • It measures the volume that holds docker-compose.yml. The path in the error names it;

    • The default is 512 MiB (536870912 bytes);

    • It does not measure Docker’s data folder, where the images are downloaded. Check that folder yourself with docker system df and df -h before you update.

    To change the threshold, set CONDUCTOR_UPDATE_MIN_FREE_BYTES (a number of bytes) in the Conductor service and recreate it:

      gluesync-conductor:
    environment:
      - CONDUCTOR_UPDATE_MIN_FREE_BYTES=2147483648
    docker compose up -d gluesync-conductor
    Expected result

    When the volume has less free space than the threshold, the update is refused with DISK_FULL and nothing is changed. After you free space, the same update runs. A lower value in CONDUCTOR_UPDATE_MIN_FREE_BYTES appears as the need figure in the message (in a test, need 1000 bytes).

    Free space before you update, and keep a copy of docker-compose.yml. The Conductor also keeps a last valid copy in .conductor/docker-compose.last-good.yml in the Gluesync folder. After a successful update, the platform removes the old images that nothing uses. The DISK_FULL codes are listed in Conductor error codes.
    Watch free space so that you are warned early

    Config

    Gluesync does not raise an alert for a full disk. Add one on the host.

    1. Use your server monitoring (for Windows Server, the monitoring you already use for the system disk), and alert at 20 percent free and again at 10 percent free, on the disk that holds the Gluesync folder and on the disk that holds Docker’s data;

    2. If you use Prometheus, add node_exporter to the scrape configuration and a rule such as the one below. In the lab, a rule of this form was reported as firing by Prometheus.

      groups:
  - name: gluesync-disk
    rules:
      - alert: GluesyncDiskLow
        expr: |
          node_filesystem_avail_bytes{mountpoint="/", fstype!~"tmpfs|overlay"}
            / node_filesystem_size_bytes{mountpoint="/", fstype!~"tmpfs|overlay"} < 0.20
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Less than 20% free on the Gluesync disk"

    3. Set the mountpoint to the mount point of the disk that holds the Gluesync folder, and send the alert with the Alertmanager or Grafana alerting that you already use.

    Expected result

    Your monitoring warns you while there is still room to act, instead of after the cache has stopped growing or Core Hub has crashed.

    The Core Hub Grafana dashboard has panels for CPU, JVM heap and system memory, but none for disk. Read Monitor pipelines in Grafana for the panels that exist.
    Verify the result

    You have successfully secured the disk if:

    • You can name the disk, the folders and the Docker figures that use space on your host;

    • The Core Hub log has a Max history (Days) and a Total size cap (GB), and you know where Clear Logs is;

    • The source agent has a Source change retention in hours value that is longer than your longest outage;

    • Notification and webhook retention are set, and the Prometheus flags show the limits you want;

    • Docker container logs are capped with max-size and max-file;

    • You know the order to recover from a full disk and you will not delete cdc_arena or data/core-hub/database;

    • A monitor warns you at 20 percent free.

    Troubleshooting

    Problem Possible cause What to do

    The Control Plane does not open and the Core Hub container keeps restarting

    The disk is full and the cache needed more space, so the Java process crashed (exit code 134)

    Stop Core Hub, add or free several hundred MB, make sure the target is running, and start Core Hub (Recover when the disk is already full)

    A setting fails with Request failed with status code 500

    The disk is full, so Core Hub cannot write its database

    Use Clear Logs, free more space, and save the setting again

    df shows no more space after you deleted the log file

    The active log was deleted while Core Hub kept it open

    Restart Core Hub, and use Clear Logs next time

    The cdc_arena folder is large and the pipeline looks healthy

    The target is slower than the source, or it was down

    Check the target and the entity status. The cache drains when the target catches up. Lower Source change retention in hours only if you accept dropping older changes

    The target is missing rows after you cleaned cdc_arena

    The cache held changes that were not delivered, and they were deleted

    Start the entity with a snapshot to copy the rows again, as in Start a snapshot and CDC replication

    docker system df shows hundreds of GB

    Container logs without a limit, or old images

    Check docker system df -v, set Docker log rotation, recreate the containers, and remove unused images (Cap the Core Hub log and the Docker logs, Free space safely right now)

    An update fails with DISK_FULL (HTTP 507)

    The volume that holds docker-compose.yml has less free space than the threshold

    Free space on that volume and try again. Nothing was changed (Understand the free-space check before an update)

    The update reports no error but the images do not download

    Docker’s data folder is full, which the check does not measure

    Check docker system df and df -h, free space there, and run the update again

    docker-compose.yml is empty (0 bytes) after a failed update

    An older Conductor was writing the file when the disk filled up

    Do not restart the stack. Look for .conductor/docker-compose.last-good.yml in the Gluesync folder, a copy of the last valid file, or restore your own copy. If you have neither, ask MOLO17 Support for a file that matches your versions

    Related guides

    Contact MOLO17 Support if you need assistance with disk space, caches or logs.