AWS S3 Agent for Gluesync: Object Storage Integration

Index

Core principles

Objects storage in Amazon Simple Storage Service (S3) is capable of providing a flexible, scalable and cost-effective solution for storing large amounts of data in a file format.

Gluesync offers the support to store data coming from supported data sources into AWS S3 buckets (and S3-like buckets as well) in Parquet file format by default, with optional JSON and CSV outputs per entity.

The files stored in the AWS S3 destination bucket follow AWS best practices and include keyspace support. This means that documents are organized within a folder path structure based on the transaction type (snapshot or changes), table name, year, month, and timestamp.

Support for JSON or CSV files remains available as optional formats, allowing users to choose based on their preference. In this case, each document is grouped by the source schema and table name, with individual files named according to their primary key.

Change data capture

This agent does not currently support reading incremental changes from AWSS3 buckets, this feature will soon be added.

Platform supported features

Feature Support Notes

Feature	Support	Notes
Target role	✅	Writes Parquet or JSON batches to AWS S3/S3-compatible buckets via the AWS SDK.
Source role	❌	Object storage is target-only in Gluesync today.
Snapshots	✅	Exports snapshot data into `/snapshots/…` folders following keyspace conventions.
CDC	❌	Object storage is target-only in Gluesync today.
CDC type	N/A	Object storage is target-only in Gluesync today.
Bulk load	N/A	Object-store uploads already implement the optimal batching pattern; no staging tables.
Database pooling	✅	Supports connection pooling for efficient resource utilization across multiple pipelines.
Source persisted internal cache	N/A	State tracking lives in Core Hub metadata; the agent retains no disk cache.
Target persisted internal cache	✅	Temporary files are locally cached on the agent before being uploaded.
Driver / connector	AWS SDK for Java/Kotlin	Works with AWS-native buckets as well as S3-compatible endpoints (MinIO, Dell ECS, etc.).
Chains	N/A	Target-only connector.
Truncate operation support (reading)	N/A	Target-only connector; does not ingest from S3.
Truncate operation support (writing)	N/A	The agent does not remove existing objects unless explicitly configured.

Target role

✅

Writes Parquet or JSON batches to AWS S3/S3-compatible buckets via the AWS SDK.

Source role

❌

Object storage is target-only in Gluesync today.

Snapshots

✅

Exports snapshot data into /snapshots/… folders following keyspace conventions.

CDC

❌

Object storage is target-only in Gluesync today.

CDC type

N/A

Object storage is target-only in Gluesync today.

Bulk load

N/A

Object-store uploads already implement the optimal batching pattern; no staging tables.

Database pooling

✅

Supports connection pooling for efficient resource utilization across multiple pipelines.

Source persisted internal cache

N/A

State tracking lives in Core Hub metadata; the agent retains no disk cache.

Target persisted internal cache

✅

Temporary files are locally cached on the agent before being uploaded.

Driver / connector

AWS SDK for Java/Kotlin

Works with AWS-native buckets as well as S3-compatible endpoints (MinIO, Dell ECS, etc.).

Chains

N/A

Target-only connector.

Truncate operation support (reading)

N/A

Target-only connector; does not ingest from S3.

Truncate operation support (writing)

N/A

The agent does not remove existing objects unless explicitly configured.

Supported version

With this agent, you can target any AWSS3 region as well as any S3-like storage provided by third parties.

Currently tested S3-like storages are:

Dell ECS;
Min.io.

Q&A

Can I export data as Parquet, JSON, or CSV? Yes. Parquet remains the default, while JSON and CSV can be toggled at the entity level from the Use JSON file or Use CSV file properties documented under the target configuration. CSV exports follow the same folder hierarchy as other formats, making it easier to feed downstream tools that prefer flat files.