Search docs.molo17.com

    Parquet Files Support

    How-to guide

    For a step-by-step walkthrough tested on a running Gluesync, see the how-to guide Replicate to a data lake.

    Overview

    Gluesync supports storing replicated data in Apache Parquet file format at the target data store level, providing enhanced performance, compression, and compatibility with analytics tools. This file format comes by default on the support agents and joins the previous support for JSON files stored in a folder-to-table manner.

    Users can select the data storage format on a per-entity basis. Parquet is the default format, offering significant advantages over JSON for large-scale data processing.

    Parquet support is designed for scenarios where file-based storage in object stores is preferred over traditional database tables.

    Powered by MOLO17 ParquetKt

    Starting from version 2.1.10, Gluesync’s Parquet workflows are powered by the MOLO17 ParquetKt library, a pure Kotlin implementation derived from parquet-dotnet. Replacing the previous Apache Parquet Java dependency grants up to a 100x speed-up and unlocks the following capabilities for every supported agent:

    • Type-safe serialization: Native support for Kotlin data classes preserves the source field order and automatically handles nullable columns, aligning perfectly with Gluesync’s schema-first approach.

    • High-throughput readers and writers: Internal benchmarks show sustained performance above 287k rows/s on wide datasets, scaling to 1.9M rows/s on narrow datasets thanks to optimized Arrow-based batching and zero-copy buffers.

    • Dual APIs: Gluesync uses the library’s high-level API for standard tasks and falls back to the low-level API for fine-grained CDC batching, ensuring deterministic _operation metadata.

    • Coroutines and streaming support: Suspendable I/O and Flow-based streaming allow Gluesync agents to keep memory usage flat even when building very large snapshot chunks.

    • Codec flexibility: Native implementations for SNAPPY, ZSTD, GZIP, and uncompressed outputs make it easy to balance file size and CPU consumption per entity.

    Performance profile

    Real-world throughput measurements inherited from ParquetKt guide the Core Hub defaults:

    Operation Throughput Details

    Write (100 columns, 500k rows)

    287k rows/s

    Produces ~22 MB Parquet files with SNAPPY compression.

    Read (100 columns, 500k rows)

    310k rows/s

    Processes ~5 million cells per batch.

    Write (20 columns, 100k rows)

    1.9M rows/s

    Ideal for CDC bursts or entity-level fan-out.

    These figures translate into faster checkpoint flushes, fewer object store PUT operations, and overall lower replication latency.

    Compression guidance

    Use the following codec recommendations when tuning entity settings:

    Codec File size (100k rows, 20 columns) When to use

    UNCOMPRESSED

    ~15.6 MB

    Best for debugging or when downstream systems demand raw blocks.

    SNAPPY (default)

    ~8.9 MB

    Balanced compression and CPU usage; ideal for continuous CDC.

    ZSTD

    ~6.9 MB

    Great size reduction with ~1.2M rows/s throughput for large archives.

    GZIP

    ~7.0 MB

    Maximum compression at the cost of ~240k rows/s; reserve for cold storage.

    These numbers assume buffered batches of 100k records; actual values depend on column cardinality and data types.

    Supported data stores

    Gluesync supports Parquet file storage in the following data stores:

    • AWS S3 (and any S3-compatible storage)

    • Google Cloud Storage

    • Azure Data Lake Gen 2

    How it works

    When Parquet is selected as the storage format:

    • The Agent leverages Apache Arrow to perform batched data processing and compression to target Parquet files.

    • Incremental data is stored in Parquet files, with each file containing a batch of changes.

      • Batch of changes can be grouped by highering the agent’s polling interval.

    This buffering mechanism optimizes network usage and file sizes for better performance. In the following sections, the snapshot and CDC implementation details are described.

    Here below an example of the folder structure created by Gluesync when Parquet is selected as the storage format:

    Parquet files support

    Snapshot implementation

    Snapshot operations store data in a structured folder hierarchy with the following naming convention:

    /bucket_name/raw/snapshots/entity_name/year=2025/month=09/snapshot.parquet

    entity_name is the name of the entity being replicated, this is configured in the Core Hub from the Objects browser.

    By default, Gluesync now emits files using relative timestamp segments (/schema_table/timestamp.snap for snapshots and /schema_table/timestamp.chng for CDC). You can revert to the legacy absolute path style (raw/snapshot || changes/schema_table/timestamp.parquet) via the agent configuration flag Use relative timestamp paths (Advanced settings → default: false).

    File size optimization

    For further optimization, files are compressed based on a buffer of batched data cached in an Avro file with a maximum size of 250 MB. This file size is configurable through the target agent configuration.

    CDC implementation

    Change Data Capture (CDC) operations efficiently handle inserts, updates, and deletes events in Parquet format. Each change is stored in a single file that includes an additional column named _operation which contains a single uppercase character indicating the type of operation performed:

    • I Insert

    • U Update

    • D Delete

    Transaction identifiers

    Starting from version 2.1.10, transaction identifiers are now stored within the destination Parquet files. These are dynamic fields that are added to the Parquet file based on the source checkpoint schema, such as:

    • _timestamp

    • _transaction_id

    • Other source-specific checkpoint fields

    This enhancement provides better data lineage and traceability for each transaction.

    Technical fields

    All technical fields are now stored within the destination Parquet files prefixed with _ to distinguish them from business data. Examples include:

    • _transaction_id

    • _timestamp

    • Other Gluesync-managed metadata fields

    This naming convention ensures clear separation between user data and system metadata.

    Unified transaction storage

    As of version 2.1.10, all transactions are now stored within the same Parquet file irrespectively of the transaction type (Update, Delete, Insert). This represents a significant change from the previous implementation where each transaction type was stored in a separate file.

    This unified approach:

    • Simplifies file management and reduces the number of files

    • Improves query performance by reducing file scanning overhead

    • Maintains transaction type information through the _operation column

    Inserts

    New rows are appended to batch files, creating multiple files per batch for optimal parallel processing.

    Updates and Deletes

    Both Updates and Deletes include the full updated row with the additional operation type indicator, explained before.

    Direct file modifications are avoided; instead, merge logic is handled downstream (e.g., via Spark jobs).

    File storage

    CDC files are stored following the path structure defined in the agent configuration. Depending on the selected output format, files are generated with one of the following extensions:

    • .chng

    • .parquet

    Truncate Operation

    When a TRUNCATE operation is detected on the source database, a dedicated Parquet file is generated with the .trunc extension. This file contains a minimal schema composed of three columns:

    • _operation with value T (indicating a truncate event)

    • The columns that define the transaction identifier, as previously described

    No row-level data is included, as the truncate operation applies to the entire table.

    Polling interval

    Entities using Parquet support have a default polling interval of 100 milliseconds. This high interval encourages aggregation of more rows into fewer, larger Parquet files for better compression and performance. Users can adjust this value based on their throughput requirements.

    Metadata

    By default, each Parquet file includes a sidecar manifest JSON file containing:

    • Timestamp of the snapshot

    • Row count

    • Operation (Insert, Update, Delete)

    • Schema version

    • Originating transaction identifier

    • Transaction identifiers from the source checkpoint schema

    This behavior can be disabled via configuration. The relevant setting is available in the agent configuration under Create sidecar JSON companion file flag (Advanced settings → default: true).

    Starting from version 2.1.10, the sidecar JSON file has been updated to reflect the unified transaction storage model and includes enhanced transaction metadata.

    Configuration

    • Parquet file format are selected as default file compression in the Core Hub entity configuration.

    • Adjust polling intervals as needed (default: 100 milliseconds, consider to increase it to decrease the number of files).

    Best practices

    • Use the metadata sidecar files for data lineage and validation.

    • Leverage transaction identifiers and technical fields (prefixed with _) for advanced data tracking and auditing.

    • Implement downstream merge logic for handling updates and deletes in analytics workloads.

    • Take advantage of the unified transaction storage model to simplify query logic.

    • Test with sample data to validate compression ratios and read performance.

    Limitations and considerations

    • Parquet files are immutable; changes require new file versions.

    • Network bandwidth may impact upload performance for large batches.

    Troubleshooting

    • File size issues: Adjust polling intervals or batch sizes.

    • Read performance: Verify file sizes and compression settings.

    FAQ

    The answers below were checked with a PostgreSQL source and an S3-compatible bucket, and read back with DuckDB. The Google Cloud Storage and Azure Data Lake Gen 2 targets write the same files.

    Where do the files go and how are they named?

    With the default settings an entity writes into one folder named after the entity, directly under the bucket (or under the custom path of the connection):

    products/2026-10-28T14:06:26.072477675.snap     rows copied by a snapshot
    products/2026-10-28T14:06:26.072477675.json     sidecar file of the snapshot file
    products/2026-10-28T14:07:10.969927626.chng     a batch of changes (CDC)
    products/2026-10-28T14:08:39.978266542.trunc    a TRUNCATE of the source table

    The name is the UTC time at which the file was written, so sorting file names sorts the files in write order. If the entity name contains a dot, it is written with an underscore. The .snap, .chng and .trunc files are ordinary Parquet files with a different extension. The .json sidecar has the same name as its data file, and you can turn it off with Create sidecar JSON compation file.

    How do I get the older layout with raw/snapshot and year=/month= folders?

    Turn on Use relative timestamp paths in the agent settings. The name is confusing: with the option on (it is off by default) the agent writes the older layout, and with it off it writes the layout above.

    raw/changes/products/year=2026/month=10/2026-10-28T14:12:57.921891967.parquet
    raw/snapshot/products/year=2026/month=10/2026-10-28T14:12:57.921891967.parquet

    The month is not padded with a zero (month=9). Both layouts keep the files of an entity apart from those of other entities, and neither partitions by a column of your data: year and month are the time of the upload. Do not switch layouts on an entity that already has files, unless your readers look at both.

    Can I partition the files by a column, or by the date of the data?

    No. Gluesync does not partition by the values of a column. If your query engine needs partitions, create them downstream, for example with a job that reads the latest state (see [faq-query]) and writes it partitioned.

    What columns does a Parquet file contain?

    The columns of the entity, in the order of the entity, followed by technical columns whose names start with _:

    • _operation: I insert, U update, D delete, T truncate. The rows of a snapshot file are written with U.

    • The fields of the checkpoint of the source, each prefixed with _. A PostgreSQL source writes _value (the position in the WAL) and _timestamp; other sources write their own. They are empty in snapshot files.

    A delete row carries the key. The other columns hold what the source gave Gluesync: with PostgreSQL and the default replica identity they are NULL. A truncate file contains only _operation and the checkpoint fields.

    Types are kept: a smallint or integer becomes a Parquet INT32, a numeric(8,2) a DECIMAL(8,2), a text a string. The Parquet writer does not support arrays and maps.

    One CDC file holds inserts, updates and deletes together. Is that right?

    Yes. A file is a batch of changes in the order of the source, whatever the operation. Raise the polling interval of the entity for fewer and larger files.

    What happens when a column is added to, or dropped from, the source?

    Nothing, until you change the mapping of the entity: the files keep the columns of the entity as it was saved. To add the column, edit the mapping, which stops the entity, and start it again, see the schema change FAQ. On an object store target, select Add field on the target to create the new field and draw its arrow.

    From then on, new files contain the new column, and old files are not rewritten: a folder can hold files with different columns. Starting a full copy writes a complete set of new .snap files and does not delete the old files.

    Your reader has to cope with that. In DuckDB use union_by_name = true, which fills the missing columns with NULL. Without it DuckDB takes the columns of the first file, and the new column is silently left out.

    How do updates and deletes show up? Do I get the current state of a table?

    The files are a log, not a table: an update is a new row with the new values, and a delete is a row with _operation = 'D'. Gluesync does not merge them. To get the current state, keep the last row of each key and drop the keys whose last row is a delete:

    SELECT * EXCLUDE (_operation, _value, _timestamp, filename, file_row_number, rn)
    FROM (
      SELECT *, row_number() OVER (PARTITION BY id
                                   ORDER BY filename DESC, file_row_number DESC) AS rn
      FROM read_parquet(['s3://lake/products/*.snap', 's3://lake/products/*.chng'],
                        union_by_name = true, filename = true, file_row_number = true)
    )
    WHERE rn = 1 AND _operation <> 'D';

    Replace id with the key of your entity, and the path with yours. Two points when you adapt it:

    • Every full copy writes a new set of .snap files next to the old files. The query still gives the right result, because the newest row of each key wins.

    • A .trunc file means that the source table was emptied at that time. Keep only the files whose name sorts after the last .trunc file, by adding WHERE filename > 's3://lake/products/<name of the .trunc file>' to the inner query.

    How do I query the files?

    Any engine that reads Parquet can: DuckDB, Spark, Trino, Athena and others. Point it to the files, with these cautions:

    • Select the files by pattern: *.snap and *.chng. A tool that lists a folder and keeps only *.parquet files skips them. The older layout writes .parquet files.

    • Do not read the .json sidecar files as data.

    • Configure the engine for your store. For the test above, DuckDB needed its httpfs extension and the endpoint, keys and path style of the bucket.

    What is in the sidecar JSON file?

    The number of rows of the file, its timestamp, the count of rows per operation, a schema version, and the transaction identifiers of the source that the file covers. It lets you check a file without opening it. The sidecar of a snapshot file has no transaction identifiers.

    Does Gluesync delete or compact the files?

    No. Files are only added: a full copy, a truncate or a delete in the source does not remove anything from the bucket. Plan retention and compaction downstream, for example with a lifecycle rule of the bucket and a job that rewrites the latest state.

    See also