Gluesync - Real-time data replication for RDBMS, NoSQL & data lakes

Gluesync, a data replication suite by MOLO17, has been designed from the ground up to be a stateless service acting as a middleware tier application capable of connecting to source and target database systems without having to store information within itself.

Gluesync is entirely written in Kotlin language compiling Java binaries. It is usually shipped through containers but can work even in bare-metal environments installed as a Java application that runs as a system service in Linux, Mac and Windows operating systems.

Being a stateless middleware enables Gluesync with ease of deployment as well as off-the-shelf security measures we are covering in the following sections.

Speaking about what’s inside the middleware we can think of Gluesync as a multi-layered application: we can identify 4 main components: source connector, target connector, data ingestion engine and data modeling engine.

From the picture here below we can see their representation and better understand how the information flows between the different application layers.

One of the main considerations we took when designing the Gluesync architecture is its native ability to be upscaled and deployed with ease, just like you used to do with your container-based applications: pull-config-deploy-enjoy. That’s the motto. You gain full control of what happens under the hood without keeping you into playing with GUIs that wouldn’t have allowed you to harness the full potential of this data replication suite.

Gluesync is being shipped through containers, if you haven’t already read about containers you can have a look here at this link that points you directly to the official docker’s homepage for a reference about a common container environment.

Here the following diagram represents an architectural overview of a Gluesync environment.

The design concept that has been adopted has taken into consideration the purpose of each core functionality provided by the suite: it provides the ability to replicate data from a relational database to a non-relational database and vice versa. These two aforementioned functionalities are called "ways" or "directions".

So, rather than having a monolithic single piece of general-purpose software, we have decoupled its functionality into an auto-consistent and highly-resilient and specialized service per each "direction". Capable of replicating, logging, alerting and being monitored by itself without the need for a master central authority that could only have increased complexity and introduced a single point of failure into the overall architecture.

Being that said the result and outcome for our users is the ability to decide what to deploy per each use case: you have the control over the decision to deploy only the module to replicate data from MS SQL Server to MongoDB or just the vice versa due to your specific use case. In that way, you’re going to have the fine-graned control over permissions, security and performances that you deserve from a product made for real-world production use cases.

When talking about sourcing changes from a relational database there are just a few ways to accomplish the task of auditing writes, updates and deletions performed at field-row level. The most common approach, but also the most challenging, is reading from the database transaction logs that luckily nowadays are wrapped around an API layer called CDC - Change Data Capture - which provides a way for application developers and DBAs to read through it and understand the entire history of the changes that have been made from a certain time frame.

We used the term "challenging" because every database vendor has implemented its way to expose these logs and building a tool capable of being compatible with all of them is indeed a challenge by itself and sometimes specific vendors or database versions (especially older ones) don’t provide either CDC or low-level APIs to grab transaction logs from it.

To provide wider compatibility on capturing real-time change streams from the vast majority of relational databases out there in the field we decided to develop a fine-tuned subset of UDFs (user-defined functions) that together with a set of triggers helps Gluesync to enlarge the compatibility base with relational databases while maintaining a safe, fast and secure approach with which entire lifecycle is entirely managed by its engine itself. We called this feature GDC - Gluesync Data Capture - used for certain kinds of database brands | versions | editions that do not currently (or at all) the native CDC technique or for those whom we initially decided to make compatible first through that feature and then to provide CDC out-of-the-box in the upcoming future.

As naming conventions to nickname each type of Gluesync direction we adopted the following nomenclature per each replication component:

Before proceeding, for each Gluesync instance that you’re willing to deploy, please check if you have the following information:

If you’re involved in the implementation of Gluesync or just trying it out via our trial program, we highly suggest you have a visual query editor tool to bootstrap multiple data source connections, and easily import/edit / display data. The tested tools from the Gluesync product and QA teams are:

…​and also, do not miss our section dedicated to tutorials and use cases of Gluesync.