Universal file store agent

Target universal file store

For deeper context on how Parquet payloads are buffered, versioned, and stored across file-based targets, see Parquet files support.

Prerequisites

To have Gluesync working with your file storage endpoint you will need to have:

A reachable file storage endpoint (FTP/FTPS/SFTP server, WebDAV endpoint, SMB/CIFS share, or a locally mounted NFS path);
Valid credentials with read and write permissions on the target directory;
The destination directory (or share, or mount point) already created before starting the pipeline.

Protocol-specific requirements

Protocol	Default port	Notes
FTP	21	Standard unencrypted file transfer.
FTPS	990	FTP over implicit TLS. Requires a valid TLS certificate on the server.
SFTP	22	SSH-based file transfer. Requires SSH host key acceptance.
WebDAV	80	HTTP-based WebDAV access.
WebDAVS	443	WebDAV over HTTPS. Requires a valid TLS certificate.
SMB / CIFS	445	Windows/Samba file sharing. Ensure the share is accessible from the agent network.
NFS	2049	Network File System. The agent must have the remote mount available or use an NFS-aware library.

Protocol

Default port

Notes

FTP

Standard unencrypted file transfer.

FTPS

990

FTP over implicit TLS. Requires a valid TLS certificate on the server.

SFTP

SSH-based file transfer. Requires SSH host key acceptance.

WebDAV

HTTP-based WebDAV access.

WebDAVS

443

WebDAV over HTTPS. Requires a valid TLS certificate.

SMB / CIFS

445

Windows/Samba file sharing. Ensure the share is accessible from the agent network.

NFS

2049

Network File System. The agent must have the remote mount available or use an NFS-aware library.

Setup via Web UI

Hostname / server: Hostname or IP address of the remote server. For NFS, use localhost or the host where the mount is available;
Port: Port of the remote server. Typical defaults: 21 for FTP/FTPS, 22 for SFTP, 80 for WebDAV, 443 for WebDAVS, 445 for SMB/CIFS. Use 0 for NFS;
Base path: Base path on the remote server. For SMB/CIFS this is the share name (e.g. share). For NFS this is the local mount point (e.g. /mnt/nfs-target). For WebDAV use /;
Username: (optional) Remote server username. Leave empty for NFS;
Password: (optional) Remote server password. Leave empty for NFS;
Disable auth: (optional, defaults to false) Set to true to skip authentication. Required for NFS;
Enable TLS: (optional, defaults to false) Enable TLS encryption. Automatically upgrades FTP → FTPS and WebDAV → WebDAVS.

Custom properties

Transport protocol: (required, defaults to FTP) The file transfer protocol to use. Accepted values: FTP, FTPS, SFTP, WebDAV, WebDAVS, SMB, CIFS, NFS;
Use secure transport: (optional, defaults to false) Enable TLS on the transport layer. Equivalent to setting enableTls in hostCredentials. When true, FTP is promoted to FTPS and WebDAV is promoted to WebDAVS;
File type: (optional, defaults to Parquet) Output format for all entities unless overridden at entity level. Accepted values: Parquet, JSON, CSV;
Remote root path: (optional, defaults to /) Additional path appended to the base path after it is normalised. Use this to target a sub-directory within the base path without changing the databaseName field.

Specific configuration

Source change retention in hours: (optional, defaults to 72) Number of hours that processed change records are retained in the internal tracking store;
Use relative timestamp paths: (optional, defaults to false) When enabled, Parquet and CSV files are written under a relative timestamp-based folder structure instead of an absolute one;
Create sidecar JSON companion file: (optional, defaults to true) When enabled, a small JSON metadata file is created alongside each Parquet upload;
Max concurrent uploads: (optional, defaults to 4) Maximum number of parallel file uploads allowed at any given time. Increasing this value can improve throughput on high-bandwidth connections.

Entity custom properties

Maximum row count in Parquet file: (optional, defaults to 250000) Maximum number of rows written into a single Parquet file before a new file is started;
Maximum row count in CSV file: (optional, defaults to 250000) Maximum number of rows written into a single CSV file before a new file is started.

Setup via Rest APIs

The examples below show the minimum hostCredentials and customHostCredentials payload for each supported protocol. Replace placeholder values with your own.

FTP

curl -X POST 'http://<gluesync-core-address>/api/v1/connections' \
  -H 'Content-Type: application/json' \
  -d '{
        "hostCredentials": {
            "connectionName": "FTP target",
            "host": "ftp.example.com",
            "port": 21,
            "databaseName": "gluesync",
            "username": "gluesync",
            "password": "password",
            "disableAuth": false,
            "enableTls": false
        },
        "customHostCredentials": {
            "protocol": "FTP",
            "fileType": "Parquet",
            "customPath": "/"
        }
  }'

FTPS

curl -X POST 'http://<gluesync-core-address>/api/v1/connections' \
  -H 'Content-Type: application/json' \
  -d '{
        "hostCredentials": {
            "connectionName": "FTPS target",
            "host": "ftps.example.com",
            "port": 21,
            "databaseName": "gluesync",
            "username": "gluesync",
            "password": "password",
            "disableAuth": false,
            "enableTls": false
        },
        "customHostCredentials": {
            "protocol": "FTPS",
            "fileType": "Parquet",
            "customPath": "/"
        }
  }'

SFTP

curl -X POST 'http://<gluesync-core-address>/api/v1/connections' \
  -H 'Content-Type: application/json' \
  -d '{
        "hostCredentials": {
            "connectionName": "SFTP target",
            "host": "sftp.example.com",
            "port": 22,
            "databaseName": "gluesync",
            "username": "gluesync",
            "password": "password",
            "disableAuth": false,
            "enableTls": false
        },
        "customHostCredentials": {
            "protocol": "SFTP",
            "fileType": "Parquet",
            "customPath": "/"
        }
  }'

WebDAV

curl -X POST 'http://<gluesync-core-address>/api/v1/connections' \
  -H 'Content-Type: application/json' \
  -d '{
        "hostCredentials": {
            "connectionName": "WebDAV target",
            "host": "webdav.example.com",
            "port": 80,
            "databaseName": "/",
            "username": "gluesync",
            "password": "password",
            "disableAuth": false,
            "enableTls": false
        },
        "customHostCredentials": {
            "protocol": "WebDAV",
            "fileType": "Parquet",
            "customPath": "/"
        }
  }'

WebDAVS

curl -X POST 'http://<gluesync-core-address>/api/v1/connections' \
  -H 'Content-Type: application/json' \
  -d '{
        "hostCredentials": {
            "connectionName": "WebDAVS target",
            "host": "webdavs.example.com",
            "port": 443,
            "databaseName": "/",
            "username": "gluesync",
            "password": "password",
            "disableAuth": false,
            "enableTls": true,
            "trustServerCertificate": true
        },
        "customHostCredentials": {
            "protocol": "WebDAVS",
            "enableTls": true,
            "fileType": "Parquet",
            "customPath": "/"
        }
  }'

SMB

curl -X POST 'http://<gluesync-core-address>/api/v1/connections' \
  -H 'Content-Type: application/json' \
  -d '{
        "hostCredentials": {
            "connectionName": "SMB target",
            "host": "smb.example.com",
            "port": 445,
            "databaseName": "share",
            "username": "gluesync",
            "password": "password",
            "disableAuth": false,
            "enableTls": false
        },
        "customHostCredentials": {
            "protocol": "SMB",
            "fileType": "Parquet",
            "customPath": "/"
        }
  }'

CIFS

curl -X POST 'http://<gluesync-core-address>/api/v1/connections' \
  -H 'Content-Type: application/json' \
  -d '{
        "hostCredentials": {
            "connectionName": "CIFS target",
            "host": "cifs.example.com",
            "port": 445,
            "databaseName": "share",
            "username": "gluesync",
            "password": "password",
            "disableAuth": false,
            "enableTls": false
        },
        "customHostCredentials": {
            "protocol": "CIFS",
            "fileType": "Parquet",
            "customPath": "/"
        }
  }'

NFS

curl -X POST 'http://<gluesync-core-address>/api/v1/connections' \
  -H 'Content-Type: application/json' \
  -d '{
        "hostCredentials": {
            "connectionName": "NFS target",
            "host": "localhost",
            "port": 0,
            "databaseName": "/mnt/nfs-target",
            "username": "",
            "password": "",
            "disableAuth": true,
            "enableTls": false
        },
        "customHostCredentials": {
            "protocol": "NFS",
            "fileType": "Parquet",
            "customPath": "/"
        }
  }'

For NFS the agent resolves paths on the local filesystem. The NFS share must already be mounted at the path specified in databaseName before the pipeline starts.

File output structure

When replication is active, the agent produces files with the following naming convention:

{remote_root_path}/{entity_name}/{timestamp}/{entity_name}_{timestamp}_{sequence}.{ext}

Where:

{entity_name} is the name of the replicated table or collection.
{timestamp} is the generation timestamp (when useTimestampPath is enabled).
{sequence} is an incremental file sequence number.
{ext} is the file extension based on the selected format (parquet, json, or csv).

Best practices

Use Parquet for analytics workloads and downstream BigQuery/Snowflake ingestion.
Use JSON when consuming data with document-oriented tools or streaming pipelines.
Use CSV for simple interoperability with spreadsheets and legacy ETL tools.
Enable sidecar metadata files to preserve schema information and replication offsets for audit and recovery.
Set max concurrent uploads based on your network bandwidth and server capacity to avoid overwhelming the endpoint.