Data Filtering

Data filtering enables granular control over which data gets synchronized, allowing you to define specific criteria and rules to include or exclude data based on your business requirements.

On-the-fly data filtering feature

You can access this feature from the Objects Browser via the Settings dialog. The Settings dialog includes an on-the-fly data filtering feature that enables users to define filters for incoming data feeds. This feature allows you to filter out data that does not match your desired rules while the data is being processed. Filters do not consume resources at source datastore level, they are applied at the Gluesync platform level.

Filters are applied after the data is fetched from the source datastore and before it is processed by the Gluesync platform, therefore applied both for CDC tasks as well as while performing snapshots.

How to use

Here’s a step-by-step guide to setting up data filtering:

Setup entity

First, choose tables/collections and set up columns for a new entity to replicate

Setup entity

From the Objects Browser, open the Settings dialog and navigate to the dedicated section named Filter out incoming data. This section allows you to define filtering rules.

Filtering rules

Select columns for filtering

Multiple fields selection

Define filtering operators

For each selected column, you can choose one of the following operators:

Filtering operators

Operator

Description

=

Equal to

>

Greater than

<

Less than

>=

Greater than or equal to

⇐

Less than or equal to

!=

Not equal to

in

Value exists in a list

Not in

Value does not exist in a list

Is null

Value is null

Not null

Value is not null

Regex

Use regular expressions like Dan.* to filter data similarly to a SQL LIKE Dan% clause

Specify rule values

In the field values section, define the specific rule to apply. For example, if you choose the > operator, enter the value that the data must exceed.

Filters set

Example use case

Scenario: You want to filter out all records where the "ID" column is greater than 20.

Steps:

  1. Select the "ID" column from the dropdown.

  2. Choose the > operator.

  3. Enter 20 in the field values section.

This setup ensures that only records with an ID greater than 20 are included in your data feed.