Universal file store agent for Gluesync: overview

Index

Core principles

Gluesync’s universal file store agent enables replication to any network-accessible file storage endpoint. It supports a broad range of protocols commonly used for file transfer and network-attached storage, making it ideal for data lake ingestion, archival, and interoperability with legacy or cloud-agnostic storage systems.

The agent writes data coming from any supported source into the remote filesystem in Parquet file format by default, with optional JSON and CSV outputs selectable per entity. Files are organised inside a folder hierarchy that reflects the transaction type (snapshot or changes), the source schema and table name, and a timestamp component.

Support for JSON or CSV files remains available as optional formats. When using CSV or JSON output, each file corresponds to a single row identified by its primary key, grouped under a folder named after the source schema and table.

Change data capture

This agent does not currently support reading incremental changes from file storage systems. It operates exclusively as a target connector.

Supported protocols

The following transport protocols are supported and can be selected via the protocol custom host credential:

Protocol Default port Notes

Protocol	Default port	Notes
`FTP`	21	Plain FTP. TLS can be enabled by setting `enableTls: true` or by choosing `FTPS` directly.
`FTPS`	21	FTP over TLS (explicit or implicit). Automatically promoted from `FTP` when `enableTls` is set to `true`.
`SFTP`	22	SSH File Transfer Protocol. Authentication via username and password.
`WebDAV`	80	HTTP-based WebDAV. TLS can be enabled by setting `enableTls: true` or by choosing `WebDAVS` directly.
`WebDAVS`	443	WebDAV over HTTPS. Automatically promoted from `WebDAV` when `enableTls` is set to `true`.
`SMB`	445	Server Message Block. The `databaseName` field must contain the share name.
`CIFS`	445	Alias for SMB. Shares the same underlying transport. The `databaseName` field must contain the share name.
`NFS`	N/A	Network File System accessed as a locally mounted path. The `host` and `port` fields are ignored; `databaseName` must point to the local mount point (e.g. `/mnt/nfs-target`). Authentication is not used.

FTP

Plain FTP. TLS can be enabled by setting enableTls: true or by choosing FTPS directly.

FTPS

FTP over TLS (explicit or implicit). Automatically promoted from FTP when enableTls is set to true.

SFTP

SSH File Transfer Protocol. Authentication via username and password.

WebDAV

HTTP-based WebDAV. TLS can be enabled by setting enableTls: true or by choosing WebDAVS directly.

WebDAVS

443

WebDAV over HTTPS. Automatically promoted from WebDAV when enableTls is set to true.

SMB

445

Server Message Block. The databaseName field must contain the share name.

CIFS

445

Alias for SMB. Shares the same underlying transport. The databaseName field must contain the share name.

NFS

N/A

Network File System accessed as a locally mounted path. The host and port fields are ignored; databaseName must point to the local mount point (e.g. /mnt/nfs-target). Authentication is not used.

Supported file formats

The agent can write replicated data in the following formats:

Format	Description
Parquet	Columnar storage format optimized for analytics workloads. Recommended for data lakes and BigQuery/Snowflake ingestion.
JSON	Line-delimited JSON records. Useful for document-oriented downstream consumers.
CSV	Comma-separated values. Compatible with spreadsheet tools and legacy ETL pipelines.

Format

Description

Parquet

Columnar storage format optimized for analytics workloads. Recommended for data lakes and BigQuery/Snowflake ingestion.

JSON

Line-delimited JSON records. Useful for document-oriented downstream consumers.

CSV

Comma-separated values. Compatible with spreadsheet tools and legacy ETL pipelines.

Supported data types

The universal file store agent maps source database types to the following internal type system:

Data type	Description
string	Text and character data.
boolean	True/false values.
binary	Raw binary or BLOB data.
int	32-bit signed integers.
long	64-bit signed integers.
float	Single-precision floating-point numbers.
double	Double-precision floating-point numbers.
bigdecimal	High-precision decimal values.
date	Calendar date values.
time	Time-of-day values. datetime
Combined date and time values.	offset_date_time
Timezone-aware date and time values.	offset_time
Timezone-aware time values.	array
Ordered lists of values.	map
Key-value structures.	timestamp
Epoch-based timestamps.	smallint

Data type

Description

string

Text and character data.

boolean

True/false values.

binary

Raw binary or BLOB data.

int

32-bit signed integers.

long

64-bit signed integers.

float

Single-precision floating-point numbers.

double

Double-precision floating-point numbers.

bigdecimal

High-precision decimal values.

date

Calendar date values.

time

Time-of-day values.

datetime

Combined date and time values.

offset_date_time

Timezone-aware date and time values.

offset_time

Timezone-aware time values.

array

Ordered lists of values.

map

Key-value structures.

timestamp

Epoch-based timestamps.

smallint

Platform supported features

Feature	Support	Notes
Target role	✅	Writes files to remote storage via FTP, FTPS, SFTP, WebDAV, SMB, or NFS.
Source role	❌	Not supported. This agent is target-only.
Snapshots	✅	Supports initial snapshot loads as file batches.
CDC	✅	Appends incremental changes to existing files or creates new file batches.
Bulk load	N/A	File-based writes are inherently batched; no staging-table flow required.
Database pooling	❌	Connection pooling is not applicable for file storage protocols.
Source persisted internal cache	N/A	Not applicable for target-only agents.
Target persisted internal cache	N/A	Not applicable for file storage agents.
UDF	✅	Supports User Defined Functions for custom data transformation.
Custom document keys	✅	Supports custom document keys for NoSQL entity types.
Before & after images	❌	Not supported for file storage targets.
Recursion protection	N/A	Not applicable for target-only agents.
Logical partitioning	❌	Not supported for file storage targets.
Allowed operations forwarding	✅	Supports selective operation forwarding.
Snapshot write method	✅	Supports UPSERT and INSERT snapshot write methods.
Data filtering	✅	Supports data filtering on source entities.

Feature

Support

Notes

Target role

✅

Writes files to remote storage via FTP, FTPS, SFTP, WebDAV, SMB, or NFS.

Source role

❌

Not supported. This agent is target-only.

Snapshots

✅

Supports initial snapshot loads as file batches.

CDC

✅

Appends incremental changes to existing files or creates new file batches.

Bulk load

N/A

File-based writes are inherently batched; no staging-table flow required.

Database pooling

❌

Connection pooling is not applicable for file storage protocols.

Source persisted internal cache

N/A

Not applicable for target-only agents.

Target persisted internal cache

N/A

Not applicable for file storage agents.

UDF

✅

Supports User Defined Functions for custom data transformation.

Custom document keys

✅

Supports custom document keys for NoSQL entity types.

Before & after images

❌

Not supported for file storage targets.

Recursion protection

N/A

Not applicable for target-only agents.

Logical partitioning

❌

Not supported for file storage targets.

Allowed operations forwarding

✅

Supports selective operation forwarding.

Snapshot write method

✅

Supports UPSERT and INSERT snapshot write methods.

Data filtering

✅

Supports data filtering on source entities.

Q&A

Can I export data as Parquet, JSON, or CSV?
Yes. Parquet is the default output. JSON and CSV can be toggled independently at the entity level via the Use JSON file and Use CSV file entity properties documented in the target configuration page.

Can I use FTPS instead of FTP without changing the protocol field?
Yes. Set enableTls: true in either hostCredentials or customHostCredentials while keeping protocol as FTP; the agent will automatically upgrade the connection to FTPS. The same mechanism applies to WebDAV → WebDAVS.

How does the agent resolve the remote root path?
The effective root is computed by joining databaseName (from hostCredentials) and customPath (from customHostCredentials). For SMB and CIFS the share name is taken from databaseName, and the remaining path is derived from customPath.