Selective deletion ahead of snapshot

Selective deletion ahead of snapshot lets you define rules that remove rows from the target table right before a snapshot task starts. This is useful to prune stale or unwanted data on the target so that the upcoming snapshot loads a clean, policy-compliant dataset.

Overview

  • Scope: Applies only to the target table and only immediately before a snapshot task runs.

  • Controls: Uses the same filters and operators as Data Filtering.

  • Access: Configure this from the Objects Browser via the Settings dialog.

Selective deletion is a destructive operation on the target. Review your rules carefully and consider backups or a dry run on a staging environment before applying to production.

How it works

  • Before a snapshot begins, Gluesync evaluates the configured deletion filters against the target table.

  • Rows matching the rules are deleted on the target.

  • The snapshot task then proceeds to load data from the source into the target based on your entity configuration.

  • This feature does not affect the source system.

  • This feature does not apply to CDC streams; it only applies to snapshot executions.

How to use

Setup entity

Create or open the entity whose target table you want to prune before snapshot runs.

Access deletion rules

From the Objects Browser, open the Settings dialog and navigate to the section named Selective deletion ahead of snapshot.

Select columns for deletion rules

Use the dropdown to choose one or more columns from the target table to apply deletion filters to.

Define operators and values

Use the same operators available in Data Filtering:

Operator

Description

=

Equal to

>

Greater than

<

Less than

>=

Greater than or equal to

⇐

Less than or equal to

!=

Not equal to

in

Value exists in a list

Not in

Value does not exist in a list

Is null

Value is null

Not null

Value is not null

Regex

Use regular expressions like Dan.* to match values (similar to SQL LIKE Dan%)

Example rule values

Specify concrete values for the chosen operators. For example, with > enter the threshold value to delete older rows.

Example use cases

  • Remove target rows older than a retention threshold before reseeding with a fresh snapshot.

  • Delete target rows for specific tenants (e.g., tenant_id in (A, B)) prior to a scoped snapshot.

  • When consolidating multiple sources into one target, delete rows where source_table = 'X' and reload only the desired subset.

Best practices

  • Start in staging: validate deletion scope on non-production environments.

  • Keep rules simple and explicit to avoid accidental over-deletion.

  • Combine with Unlock Schema and UDFs if your snapshot logic also needs schema adjustments or transformations.

  • Monitor snapshot logs to confirm the number of rows deleted and reloaded.

Troubleshooting

  • No rows deleted: Verify your filter values and operators match the target’s actual data.

  • Too many rows deleted: Relax or refine your conditions; test in staging first.

  • Constraint errors: Ensure target-side foreign keys or constraints allow the intended deletions, or perform deletions in a safe order where necessary.

See also