Search docs.molo17.com
    A newer version of this documentation is available.
    View Latest (v2.3)

    Installation steps

    Gluesync NoSQL to SQL for Aerospike

    Prerequisites

    To have Gluesync working on your Aerospike instance as a source connector you will need to have:

    • valid user credentials with permission to read, and write to the source database;

    • ESP connector installed in your system (bare metal, VM, Docker…​);

    Setting up gluesyncDC under Aerospike XDR

    First of all, we need to set up an XDR "datacenter" endpoint for Gluesync under your aerospike.conf file. This XDR endpoint will be responsible for streaming changes made at the selected namespace level right into the ESP connector.

    To set this XDR endpoint you can refer to the following example:

    {
  ...

  xdr {

    # For naming convention purposes we keep gluesyncDC
    dc gluesyncDC {
      # Explicitly tell the XDR that this endpoint is a streaming connector and not an Aerospike node
      connector true

      # 192.168.1.201 is your IP / DNS address of the Aerospike ESP server
      # Port 8080 is the default we suggest using, you can change it as you wish
      node-address-port 192.168.1.201 8080

      # Substitute test with your namespace, add as many of these objects as necessary
      namespace test {

          # We explicitly avoid replicating Gluesync's set to limit overhead
          ignore-set GLUESYNC
      }
    }
  }

  ...
}

    Where:

    • gluesyncDC: is the name of the XDR endpoint we suggest you take as per our naming convention;

    • connector: true, will explicitly tell the XDR that this endpoint is a streaming connector and not an Aerospike node;

    • node-address-port: as per our example, 192.168.1.201 is the IP Address assigned to the ESP server, change it accordingly to your deployment configuration. DNS is fine either. Port 8080 is the default port we are explicitly declaring to let XDR establish a connection with the ESP server; the port can change as per your requirement but it has to match on the ESP’s end as well;

    • namespace: is your namespace name;

    • ignore-set: please keep the ignore-set GLUESYNC configuration as this avoids the replication of Gluesync’s managed set across XDR to reduce overhead.

    Please consider reading Aerospike XDR documentation for any additional configuration by following this link.

    After you’ve completed this configuration step you will need to apply the configuration, to do so you can either restart your Aerospike server or you can dynamically apply the same configuration without having to restart the server by following this Aerospike documentation link.

    Repeat this step on each of your nodes belonging to the Aerospike Cluster since every node owns its partition, you will otherwise get just a fraction of the data.

    Setting up ESP connector yaml

    Once you have XDR set your Aerospike cluster will try to establish a connection with the given ESP endpoint. To let it be able to achieve we require deploying a working ESP connector by issuing it an aerospike-esp-outbound.yml file that will look like the following provided in this example below.

    service:
  port:
    - 8080
  manage:
    port: 8902
    # address & port that can be used to check ESP metrics endpoint (optionally)
    # http://localhost:8902/manage/rest/v1/metrics
  protocol: TCP

# tells ESP to perform console logging (optional)
logging:
  enable-console-logging: true

# converts output XDR message format in JSON (see https://aerospike.com/docs/connect/esp/configuring/format)
format:
  mode: json

# Destinations
destinations:
  gluesyncDC:
    urls:
    # endpoint that the ESP uses to connect to Gluesync Aerospike source connector, port 1717 is its web server's default port
      - http://YOUR_GLUESYNC_ADDRESS_HERE:1717/esp
    protocol: HTTP_1_1
    max-requests-queued-per-endpoint: 10240
    connection-ttl: 15000
    max-connections-per-endpoint: 100
    call-timeout: 10000
    connect-timeout: 2000
    health-check:
      call-timeout: 10000
    headers:
      authority-header-omit-default-port: true
      send-digest-header: true
    http-success-status-codes:
      - 200
      - 202-206
      - 208
      - 400
      - 405

record-ordering:
  enable: true
  lut-cache-ttl-seconds: 30

port-based-config:
# port 8080 is our default port here if you've declared another port at the aerospike.conf XDR level, please change it accordingly
  8080:
    routing:
      mode: static-multi-destination
      # This has to match the same declared DC endpoint that we have put in the aerospike.conf file
      destinations: gluesyncDC

    To learn more about the Aerospike ESP connector please refer to the following link.

    Once you’ve done configuring your ESP connector we are almost at the end of our Gluesync Aerospike source connector setup. We are now going to configure the Gluesync’s config.json file.

    Basic configuration example

    This module can be customized by using a configuration file, in JSON format. The file name to use must be specified as a parameter when launching the app, with the -f or --file tokens. The file should be composed by the union of common configuration files (see here Installation steps) and source/destination specific configuration:

    {
  "sourceHost": "aerospike",
  "sourcePort": "3000",
  "sourceName": "test",
  "sourceUsername": "admin",
  "sourcePassword": "admin",
  "sourceEntities": {
    "DRIVERS": {
      "schema": "public",
      "table": "drivers",
      "mapping": {
        "ID": "id",
        "NAME": "first_name",
        "SURNAME": "last_name",
        "EMAIL": "email"
      }
    },
    "CUSTOMERS": {
      "schema": "public",
      "table": "customers"
    }
  },

  ...

  "maxItemsCountPerTransaction": 100,
  ...
  "aerospike": {
    "trustStorePath": "/path/to/certs/dir/cert.p12",
    "trustStorePassword": "change_me",
    "keyStorePath": "/path/to/certs/dir/cert.jks",
    "keyStorePassword": "change_me",
    "timeoutSeconds": 60,
    "indexReplicaCount": 0,
    "disableAuth": false,
    "additionalHosts": ["IP_ADDRESS_HOST#1", "IP_ADDRESS_HOST#2", ...],
    "entitiesKeys": {
      "myEntity": ["ID", ...],
      "DRIVERS": ["ID"]
    }
  },
}

    Aerospike-specific configurations are listed under the aerospike property:

    • timeoutSeconds (optional): number of seconds to set as timeout for operations involving communication with Aerospike. The default value is 60 seconds.;

    • disableAuth (optional), defaults to false: you can set that parameter to true to disable the authentication mechanism (as per the Aerospike docs) when you’re connecting to a development instance of Aerospike.

    • additionalHosts (optional), defaults to null: tells Aerospike source connector to make use of a cluster map of additional hosts available at the bootstrap time by feeding those to the Aerospike client SDK. If you don’t provide additional hosts at the bootstrap time they will be then retrieved anyway when the connection to targetHost is established. This comes to help in case the targetHost is down at the bootstrap time;

    • enableTls (optional): defaults to false, enable or disable usage of TLS encryption;

    • tlsName (optional): defaults to null, TLS certificate name used for secure connections. Set null or remove the key entry if TLS is disabled;

    • trustStorePath (optional): the path to the trustStore certificate;

    • trustStorePassword (optional): if a certificate path is specified, then this value is passed as a trustStore password;

    • keyStorePath (optional): the path to the keystore certificate;

    • keyStorePassword (optional): if a certificate path is specified, then this value is passed as a keystore password;

    • entitiesKeys: represents the object that helps Gluesync understand which are the fields related in your JSON document that are concurring to build a unique Key in your RDBMS. This object has to contain the list of your entities (case sentitive) with as a corresponding value the array of key fields (case sentitive);

    copySourceEntitiesAtStartup feature is not currently supported with this initial release of the Aerospike source connector for Gluesync.

    Generating valid TLS certificates for your Aerospike database

    To create certificates for connection to the Aerospike Server you need:

    • CA Certificate: example.ca.crt

    • Client Certificate: example.client.crt

    • Client Private Key: example.client.key

    First, install the CA certificate into a new Java TrustStore with the following command:

    keytool -importcert -storetype jks -alias example.ca -keystore example.ca.jks -file example.ca.crt -storepass changeit

    Then use the following command to create a single chain certificate file named example.client.chain.crt:

    cat example.ca.crt example.client.crt example.client.key > example.client.chain.crt

    Finally, create a keystore in PKCS #12 format to pass to Gluesync with the following command:

    openssl pkcs12 -export -in example.client.chain.crt -out example.client.chain.p12 -password pass:"changeit" -name example.client -noiter -nomaciter

    In the Aerospike-specific configuration section it will now be possible to use certificates as follows:

      "aerospike": {
    ...
    "trustStore": "/opt/app/config/example.ca.jks",
    "keyStorePath": "/opt/app/config/example.client.chain.p12",
    "keyStorePassword": "changeit",
    "enableTls": true,
    "tlsName": "TLS_NAME",
    ...
  }

    For further details, please refer to the official Aerospike documentation at this link: aerospike.com/docs/server/operations/configure/network/tls/mtls_java#java-application

    Looking for data modeling features or other options?

    The data modeling feature under the Aerospike source connector is not currently supported as it will be added later with a future product release.