Search docs.molo17.com

    Universal file store agent for Gluesync: overview

    Universal File Storage logo

    Core principles

    Gluesync’s universal file store agent enables replication to any network-accessible file storage endpoint. It supports a broad range of protocols commonly used for file transfer and network-attached storage, making it ideal for data lake ingestion, archival, and interoperability with legacy or cloud-agnostic storage systems.

    The agent writes data coming from any supported source into the remote filesystem in Parquet file format by default, with optional JSON and CSV outputs selectable per entity. Files are organised inside a folder hierarchy that reflects the transaction type (snapshot or changes), the source schema and table name, and a timestamp component.

    Support for JSON or CSV files remains available as optional formats. When using CSV or JSON output, each file corresponds to a single row identified by its primary key, grouped under a folder named after the source schema and table.

    Change data capture

    This agent does not currently support reading incremental changes from file storage systems. It operates exclusively as a target connector.

    Supported protocols

    The following transport protocols are supported and can be selected via the protocol custom host credential:

    Protocol Default port Notes

    FTP

    21

    Plain FTP. TLS can be enabled by setting enableTls: true or by choosing FTPS directly.

    FTPS

    21

    FTP over TLS (explicit or implicit). Automatically promoted from FTP when enableTls is set to true.

    SFTP

    22

    SSH File Transfer Protocol. Authentication via username and password.

    WebDAV

    80

    HTTP-based WebDAV. TLS can be enabled by setting enableTls: true or by choosing WebDAVS directly.

    WebDAVS

    443

    WebDAV over HTTPS. Automatically promoted from WebDAV when enableTls is set to true.

    SMB

    445

    Server Message Block. The databaseName field must contain the share name.

    CIFS

    445

    Alias for SMB. Shares the same underlying transport. The databaseName field must contain the share name.

    NFS

    N/A

    Network File System accessed as a locally mounted path. The host and port fields are ignored; databaseName must point to the local mount point (e.g. /mnt/nfs-target). Authentication is not used.

    Supported file formats

    The agent can write replicated data in the following formats:

    Format Description

    Parquet

    Columnar storage format optimized for analytics workloads. Recommended for data lakes and BigQuery/Snowflake ingestion.

    JSON

    Line-delimited JSON records. Useful for document-oriented downstream consumers.

    CSV

    Comma-separated values. Compatible with spreadsheet tools and legacy ETL pipelines.

    Supported data types

    The universal file store agent maps source database types to the following internal type system:

    Data type Description

    string

    Text and character data.

    boolean

    True/false values.

    binary

    Raw binary or BLOB data.

    int

    32-bit signed integers.

    long

    64-bit signed integers.

    float

    Single-precision floating-point numbers.

    double

    Double-precision floating-point numbers.

    bigdecimal

    High-precision decimal values.

    date

    Calendar date values.

    time

    Time-of-day values.

    datetime

    Combined date and time values.

    offset_date_time

    Timezone-aware date and time values.

    offset_time

    Timezone-aware time values.

    array

    Ordered lists of values.

    map

    Key-value structures.

    timestamp

    Epoch-based timestamps.

    smallint

    Platform supported features

    Feature Support Notes

    Target role

    ✅

    Writes files to remote storage via FTP, FTPS, SFTP, WebDAV, SMB, or NFS.

    Source role

    ❌

    Not supported. This agent is target-only.

    Snapshots

    ✅

    Supports initial snapshot loads as file batches.

    CDC

    ✅

    Appends incremental changes to existing files or creates new file batches.

    Bulk load

    N/A

    File-based writes are inherently batched; no staging-table flow required.

    Database pooling

    ❌

    Connection pooling is not applicable for file storage protocols.

    Source persisted internal cache

    N/A

    Not applicable for target-only agents.

    Target persisted internal cache

    N/A

    Not applicable for file storage agents.

    UDF

    ✅

    Supports User Defined Functions for custom data transformation.

    Custom document keys

    ✅

    Supports custom document keys for NoSQL entity types.

    Before & after images

    ❌

    Not supported for file storage targets.

    Recursion protection

    N/A

    Not applicable for target-only agents.

    Logical partitioning

    ❌

    Not supported for file storage targets.

    Allowed operations forwarding

    ✅

    Supports selective operation forwarding.

    Snapshot write method

    ✅

    Supports UPSERT and INSERT snapshot write methods.

    Data filtering

    ✅

    Supports data filtering on source entities.

    Q&A

    Can I export data as Parquet, JSON, or CSV?
    Yes. Parquet is the default output. JSON and CSV can be toggled independently at the entity level via the Use JSON file and Use CSV file entity properties documented in the target configuration page.

    Can I use FTPS instead of FTP without changing the protocol field?
    Yes. Set enableTls: true in either hostCredentials or customHostCredentials while keeping protocol as FTP; the agent will automatically upgrade the connection to FTPS. The same mechanism applies to WebDAV → WebDAVS.

    How does the agent resolve the remote root path?
    The effective root is computed by joining databaseName (from hostCredentials) and customPath (from customHostCredentials). For SMB and CIFS the share name is taken from databaseName, and the remaining path is derived from customPath.