A newer version of this documentation is available.
View Latest (v2.0)

Installation steps

Gluesync SQL to NoSQL for AWS S3

Prerequisites

In order to have Gluesync working on your AWS S3 instance you will need to have:

  • an S3 bucket;

  • valid user credentials with permissions: arn:aws:iam::aws:policy/AmazonS3FullAccess;

  • IAM access key obtained through AWS IAM console;

  • IAM secret key obtained through AWS IAM console;

  • Name of the region where the buckets belong to.

Basic configuration example

This module can be customized by using a configuration file, in JSON format. The file name to use must be specified as a parameter when launching the app, with the -f or --file tokens. The file should be composed of the union of the common configuration file (see here Installation steps) and source-specific configuration:

{
  ...
  // (source section omitted)
  ...
  "targetHost": "https://s3.eu-central-1.amazonaws.com",
  "targetPort": "",
  "targetName": "myBucketName",
  "targetUsername": "IAMCredentialAccessKeyID",
  "targetPassword": "IAMCredentialSecret",
  ...

  "awsS3": {
    "trustStorePath": "/path/to/certs/dir/cert.p12",
    "trustStorePassword": "change_me",
    "keyStorePath": "/path/to/certs/dir/cert.jks",
    "keyStorePassword": "change_me",

    "entitiesMetadata": {
      "articles": [ "id", "article_name" ]
    },

    "awsRegion": "eu-central-1",
    "allowObjectsDeletion": false
  }
}

The parameters that refer to a target are the parameters that describe the connection to the AWS S3 bucket.

  • targetHost: the name of the region where your bucket belongs to. In the case of AWS S3 buckets format has to comply with the following s3.your-regionname-X.amazonaws.com, for AWS S3-like services (min.io, …​) use either HTTP or HTTP specifying DNS/IP Address.

  • targetPort: defaults to ` (empty string). Specify a port (for example `9000) in case you are using AWS S3-like storages like Minio, DELL EMC ECS, …​

  • targetName: the name of the target bucket as represented inside your AWS S3 console (do not use the arn);

  • targetUsername: IAM Credential access key ID created under your AWS IAM console;

  • targetPassword: IAM Credential secret created under your AWS IAM console.

Other configuration parameters are the following:

AWS S3-specific configurations are listed under the awsS3 property:

  • timeoutSeconds (optional): defaults to 60. Number of seconds to set as timeout for operations involving communication with the destination;

  • allowObjectsDeletion (optional): defaults to true. If set to false tells Gluesync to skip any action that involves data deletion committed into the source database: Gluesync will propagate only events that involve Insert(s) and Update(s).

  • entitiesMetadata (optional): defaults to null. A key-value object that describes the binding of values per each table field that you would like to set as metadata values belonging to each stored object in the S3 bucket. Where key is the table (entity) name and as a value a list (array) of strings representing the column names;

  • awsRegion: defaults to aws-global. Use that field to specify your AWS Region name where your bucket belongs.

  • trustStorePath (optional): the path to the trustStore certificate;

  • trustStorePassword (optional): if a certificate path is specified, then this value is passed as a trustStore password;

  • keyStorePath (optional): the path to the keystore certificate;

  • keyStorePassword (optional): if a certificate path is specified, then this value is passed as a keystore password;