AWS S3 Agent for Gluesync: Object Storage Integration
Core principles
Objects storage in Amazon Simple Storage Service (S3) is capable of providing a flexible, scalable and cost-effective solution for storing large amounts of data in a file format.
Gluesync offers the support to store data coming from supported data sources into AWS S3 buckets (and S3-like buckets as well) in JSON format using native AWS S3 SDK.
JSON stored in the AWS S3 destination bucket will be stored using the best practices suggested by AWS with keyspace support, this means that documents will be stored in folders grouped by source schema and table name, named by their primary key.
Supported version
With this agent, you can target any AWSS3 region as well as any S3-like storage provided by third parties.
Currently tested S3-like storages are:
-
Dell ECS;
-
Min.io.
Q&A
I am looking to store other file formats like Parquet, CSV, and XML files, is it supported? We’re open to supporting a wider amount of different use cases and that also means different file formats for your object storage needs. Please do not hesitate to reach out to us to let us know your need: we’re more than happy to accommodate your feature request.