A newer version of this documentation is available.
View Latest (v2.0)

Installation steps

Gluesync SQL to NoSQL for Apache Kafka

Prerequisites

To have Gluesync working on your Kafka instance you will need to have:

  • as many topics created one per each of the entities you’re looking to source changes from;

  • valid user credentials with permission to post to the above-mentioned topics;

  • valid certificate in case of SASL authentication.

Basic configuration example

This module can be customized by using a configuration file, in JSON format. The file name to use must be specified as a parameter when launching the app, with the -f or --file tokens. The file should be composed of a union of the common configuration files (see here Installation steps) and source-specific configuration:

{
  ...
  // (source section omitted)
  ...
  "targetHost": "KAFKAIPADDRESS",
  "targetPort": "9092",
  "targetUsername": "KafkaUsername",
  "targetPassword": "KafkaPassword",
  ...

  "kafka": {
    "topicPrefix": "MYTOPICPREFIX_",
    "useTransactions": true,
    "saslMechanism": "SCRAM-SHA-256",
    "securityProtocol": "SASL_SSL",
    "certificatesType": "PEM",
    "certificatesPath": "/path/to/certificate.pem",
    "certificatesPassword": "certificatePassword",
    "disableAuth": false
  },
}

The following are the parameters used to configure your Kafka target:

  • targetHost: the host address of your Kafka cluster;

  • targetPort: the port exposed by your Kafka cluster;

  • targetUsername: username to use to connect against Kafka;

  • targetPassword: password to use to connect against Kafka.

Apache Kafka-specific configurations are listed under the kafka property:

  • topicPrefix (optional): default to NULL. String value that defines a prefix used in conjunction with the entity name to build the Kafka topic name;

  • useTransactions (optional): defaults to false. If set to true it enables the usage of Kafka transactions, look at this documentation in order to know more about it;

  • saslMechanism (required if auth in place): defaults to SCRAM-SHA-256. Accepts the following parameters:

    • GSSAPI,

    • OAUTHBEARER,

    • PLAIN,

    • SCRAM-SHA-256,

    • SCRAM-SHA-512.

  • securityProtocol (required if auth in place): defaults to SASL_SSL. Accepts the following parameters:

    • PLAINTEXT,

    • SASL_PLAINTEXT,

    • SASL_SSL,

    • SSL.

  • certificatesType (required if auth in place): defaults to PEM. Accepts the following parameters:

    • PEM,

    • JKS.

  • certificatesPath (required if auth in place): defaults to NULL. The path to your certificates;

  • certificatesPassword (required if auth in place): defaults to NULL. The password used to lock your certificate;

  • disableAuth (optional): defaults to false. Disable authentication over Kafka broker.