Google Cloud Storage Agent for Gluesync: Overview

Index

Core principles

Objects storage in Google Cloud Storage is capable of providing a flexible, scalable and cost-effective solution for storing large amounts of data in a file format.

Gluesync offers the support to store data coming from supported data sources into Google Cloud Storage buckets in Parquet file format using the native Google Cloud SDK, with optional JSON and CSV exports per entity.

The files stored in the Google Cloud Storage destination bucket follow the best practices including keyspace support. This means that documents are organized within a folder path structure based on the transaction type (snapshot or changes), table name, year, month, and timestamp.

Support for JSON or CSV files remains available as optional formats, allowing users to choose based on their preference. In this case, each document is grouped by the source schema and table name, with individual files named according to their primary key.

Change data capture

This agent does not currently support reading incremental changes from Google Cloud Storage.

Platform supported features

Feature	Support	Notes
Target role	✅	Streams snapshots/CDC batches to GCS buckets via the native Google Cloud SDK.
Source role	❌	Object storage is target-only in Gluesync.
Snapshots	✅	Exports data as Parquet (default) or JSON, following the bucket folder hierarchy.
CDC	❌	Object storage is target-only in Gluesync.
CDC type	N/A	Object storage is target-only in Gluesync.
Bulk load	N/A	Snapshot/CDC batching already uses optimal object-store semantics; no staging tables.
Database pooling	✅	Supports connection pooling for efficient resource utilization across multiple pipelines.
Source persisted internal cache	N/A	State is maintained in Core Hub metadata; agent keeps no local cache.
Target persisted internal cache	✅	Temporary files are locally cached on the agent before being uploaded to GCS.
Driver / connector	Google Cloud SDK	Gluesync uses the official SDK with service accounts and TLS.
Chains	N/A	Target-only connector.
Truncate operation support (reading)	N/A	Target-only connector; does not ingest from GCS.
Truncate operation support (writing)	N/A	The agent doesn’t delete existing objects unless explicitly configured by users.

Feature

Support

Notes

Target role

✅

Streams snapshots/CDC batches to GCS buckets via the native Google Cloud SDK.

Source role

❌

Object storage is target-only in Gluesync.

Snapshots

✅

Exports data as Parquet (default) or JSON, following the bucket folder hierarchy.

CDC

❌

Object storage is target-only in Gluesync.

CDC type

N/A

Object storage is target-only in Gluesync.

Bulk load

N/A

Snapshot/CDC batching already uses optimal object-store semantics; no staging tables.

Database pooling

✅

Supports connection pooling for efficient resource utilization across multiple pipelines.

Source persisted internal cache

N/A

State is maintained in Core Hub metadata; agent keeps no local cache.

Target persisted internal cache

✅

Temporary files are locally cached on the agent before being uploaded to GCS.

Driver / connector

Google Cloud SDK

Gluesync uses the official SDK with service accounts and TLS.

Chains

N/A

Target-only connector.

Truncate operation support (reading)

N/A

Target-only connector; does not ingest from GCS.

Truncate operation support (writing)

N/A

The agent doesn’t delete existing objects unless explicitly configured by users.

Q&A

Can I export data as Parquet, JSON, or CSV? Yes. Parquet remains the default, while JSON and CSV can be toggled at the entity level from the Use JSON file or Use CSV file properties documented under the target configuration. CSV exports follow the same folder hierarchy as other formats, so downstream tools that prefer flat files can ingest them immediately.