Entity settings reference
The Settings for <entity> dialog holds everything you can change on one entity: where its rows go, which rows it copies, and how it reads and writes them. This page describes each setting on its own, in the order the dialog shows them. From Gluesync 2.3.3.0, the ? icon next to a setting in the dialog opens its section on this page.
To open the dialog, open Data pipelines, select the pipeline, open Options (the three-dot icon) on the row of the entity and select Edit settings. The same dialog opens from the Settings button of a table while you add entities.
How to read this page
Each setting says:
-
Applies to: the snapshot (the initial copy of the table), live changes (change data capture, CDC), or both;
-
Default: the value an entity gets when you leave the field empty;
-
Range: the values Core Hub accepts. Core Hub refuses to save an entity with a value outside the range, unless the setting says the value is clamped.
The dialog has four tabs:
| Tab | What it holds |
|---|---|
The entity nickname and the settings the source agent declares for the table, such as the snapshot |
|
Where the rows are written, the target filters, the selective deletion ahead of a snapshot and, on document targets, the document key. |
|
The settings the target agent declares for the entity, such as batch sizes, SQL commands around the snapshot and CDC, and time to live. |
|
How the entity reads and writes: message and page sizes, polling, concurrency, bulk load, and what to do on a duplicate key. Chained entities do not have this tab. |
Your changes apply to the entity as you make them, and are saved with the entity. A tab with an invalid value is marked, and the dialog cannot be closed until the value is fixed.
Source tab
The settings of the entity on the source side.
Entity nickname
The name the Control Plane shows for the entity in lists, metrics, notifications and logs. It does not change the source or target table. It can be up to 50 characters long.
Settings declared by the source agent
Under Involved tables, the dialog shows the settings the source agent declares for the table. On the Entity tab, it shows the settings the target agent declares for the entity. Which ones appear depends on the agent: the sections below and those of the Entity tab list them, with the agents that declare each one. An agent that declares none shows nothing.
SQL settings are edited in a code editor. Other settings show the name of the property and a Value field.
SQL Snapshot where clause filter (without initial WHERE)
Applies to: snapshot only. Default: empty, every row is read.
A condition, in the SQL dialect of the source database and with the source column names, that limits the rows the snapshot reads. Core Hub adds it after WHERE to the snapshot SELECT and to the SELECT COUNT that measures the snapshot progress, so the progress counts only the rows that match. Typing the keyword WHERE at the beginning is optional. Because the condition runs in the source database, it can use OR, LIKE, functions, ranges and subqueries.
In a partitioned snapshot, each partition reads (your condition) AND (the partition range).
|
Declared by: IBM i (AS400), Db2 LUW (CDC and triggers), Informix, MariaDB, MySQL, Microsoft SQL Server, Oracle (LogMiner, triggers, XStream), PostgreSQL, SAP HANA, SAP SQL Anywhere, SAP ASE (Sybase), Vertica, YugabyteDB.
For a walkthrough, see Filter rows with a snapshot WHERE clause and target filters.
SQL Snapshot order by clause (without initial ORDER BY)
| Available from Gluesync 2.3.3.0. |
Applies to: snapshot only. Default: empty.
The order in which the snapshot reads the rows, in the SQL dialect of the source. Typing ORDER BY at the beginning is optional. Core Hub appends the entity keys that the clause does not name, so that the order is always complete; keys you already named are not repeated. A bare column name sorts on the stored value of the column, not on a converted value.
|
A snapshot resumed after a stop skips, by count, the rows it had already copied. Use columns that do not change while the snapshot runs: a row that changes position in the meantime can make the snapshot skip another row, with no error. See Resumable snapshot. |
Changing the clause while a snapshot is not finished makes Core Hub start that snapshot over. On Oracle, any order turns off the key-order index hint.
Declared by the same agents as SQL Snapshot where clause filter (without initial WHERE).
Disable automatic triggers creation
Applies to: live changes, on trigger-based sources. Default: off.
When on, the agent still creates the change table, its index and its sequence when it enables change capture for the table, but it does not create the triggers. Create them yourself, for example when the database administrator must review every DDL. The agent writes the trigger DDL it would have run to its log at DEBUG level.
Declared by: Db2 LUW (triggers), Oracle (triggers and legacy triggers), SAP HANA.
Use Dedicated Journal Reader
Applies to: live changes. Default: off. Agent: IBM i (AS400).
When on, the entity reads its own copy of the journal instead of sharing the journal reader of the other entities on the same journal, and it is left out of the shared reader. Use it for a table whose changes must not wait behind the other tables of a busy journal. Each dedicated reader adds its own load on the IBM i system.
Source TTL after sync
Applies to: live changes, in DCP read mode only. Default: 0, the source document is left as it is. Unit: seconds. Agent: Couchbase as a source.
When above 0, after the target confirms a batch of live changes, the agent sets this expiry on the version of each source document it delivered, in the background. Snapshot batches and deletions are not touched. If setting the expiry fails, the failure is logged and the document stays in the source.
Target tab
Where and how the rows of the entity land on the target. When the entity has several targets, select the target in the list under Target on the left.
Target schema and table
The schema (or scope, namespace, database, depending on the target) and the table (or collection, topic, index) the entity writes to. The names of the two fields follow the target.
-
On a relational target, you choose an existing table, or type a new name and select Create table "<name>" to have the table created from the entity’s fields.
-
On other targets you can type a schema or a table that is not listed and select Add "<name>"; it is marked NEW.
Changing the schema clears the table. Changing either reloads the fields of the entity, so check the field mapping afterwards.
Filter out incoming data
Applies to: snapshot and live changes. Default: no filter.
Conditions a row must match to be written to the target. Add a condition with Add condition, choose the field, the operator and the value. A row must match all the conditions.
The conditions are checked by the target agent after the field mapping, the field functions and the user-defined function, so they use the target field names and the values as they will be written.
| Operator | Matches |
|---|---|
equals, does not equal, is greater than, is greater than or equal to, is less than, is less than or equal to |
Compares the value as the field type: numbers as numbers ( |
is one of, is not one of |
The value is (or is not) one of a comma-separated list. Blanks around each item are removed. |
matches regex |
The whole value matches the regular expression. Start it with |
is empty, is not empty |
The field has no value ( |
An empty value never matches a comparison, including does not equal and is not one of. A field whose type cannot be compared (for example binary or UUID values) never matches.
A condition that is likely not to match what you expect is flagged in the dialog; you can still save it. A condition on a field the entity no longer copies blocks the save until you remove it.
During the snapshot, a row that does not match is never written. For live changes, see When a change does not match the filter. On file and object-storage targets, a change that does not match is always dropped.
Chained entities do not support target filters yet.
For a walkthrough with SQL checks, see Filter rows with a snapshot WHERE clause and target filters.
When a change does not match the filter
| This choice is available from Gluesync 2.3.3.0. Up to Gluesync 2.3.2, a live change that does not match the target filters is always turned into a delete on the target, as with Delete the row on the target below; on file and object-storage targets it is dropped. |
Applies to: live changes. Default: Delete the row on the target.
What the target does with a live change whose new values do not match the target filters.
| Choice | What happens |
|---|---|
Delete the row on the target |
The change becomes a delete on the target, so that the target holds only the rows inside the filter. A row an update moves out of the filter is removed from the target. |
Leave the target as it is |
Inserts and updates that do not match are not written. A row an update moves out of the filter keeps its last version on the target. A delete is skipped only when the deleted row is known to be outside the filter; a delete that carries only the row key is still applied. |
|
With Delete the row on the target, a condition on a field whose value cannot be compared never matches, so every live change of the entity becomes a delete. Check the conditions with the preview in the dialog before you start the entity. |
Selective deletion ahead of snapshot
Applies to: snapshot. Default: no conditions.
Conditions that pick the target rows to delete before a snapshot copies the table, instead of emptying the whole table. The target runs one DELETE … WHERE with all the conditions joined by AND, on the first page of the snapshot, before writing it. Without conditions it runs a TRUNCATE instead.
The deletion runs only when the snapshot is started with Delete only some rows first (or Empty the table first, which truncates). A snapshot started with Keep and merge overwrites the rows with the same key and deletes nothing. You can also set or change the conditions in the start dialog. See Start a snapshot and CDC replication.
Not every target runs it, and the dialog says so:
-
Targets that skip it, and delete nothing: Aerospike, Apache Cassandra, Azure Cosmos DB, Couchbase, CouchDB, Elasticsearch, Microsoft Fabric, the file and object-storage targets, Google Pub/Sub, Apache Kafka, MongoDB, RavenDB, Redis Streams, ScyllaDB, Solace.
-
Google BigQuery cannot delete selectively: with conditions it skips the deletion, without conditions it empties the table.
-
Some operations cannot be used in the
DELETEof a given target; the dialog hides them. -
A relational target configured to never truncate on a snapshot skips the deletion too.
|
The deletion cannot be undone. Try it on a copy of the target first. |
Document key
Applies to: snapshot and live changes, on document, key-value, streaming and file targets. Default: Default key.
The identifier the target uses for each row, such as the document ID in Couchbase or MongoDB, the record key in Kafka, or the key in Redis.
-
Default key:
<scope>::<collection>::<key values>, or<collection>::<key values>when the target has no scope, with the values of the entity keys joined by::. -
User defined key: an optional Prefix, the Fields you choose in the Key order you set, and an optional Suffix, joined by the Separator (
::,-,_,/or none). The prefix and suffix are added only when they are not empty. An empty field value becomes an empty part of the key.
The Preview shows the key a row will get. A change that does not carry one of the key fields is refused. For more, see Document keys.
Entity tab
The settings the target agent declares for the entity. Which ones appear depends on the target; a target that declares none shows No entity settings available for this target. Settings marked per table appear under Tables settings.
Absent field policy (PATCH routes of an HTTP source)
Applies to: entities whose source is a PATCH route of the HTTP source. Default: write NULL.
A PATCH request carries only some fields of a row. The target must leave the other columns as they are, so Core Hub refuses to save such an entity until its absent field policy is Skip column. The dialog shows a warning with a Set to Skip column button when it is not. With Skip column, an update keeps the values the target already holds for the fields the request leaves out.
A target that cannot skip columns, or that cannot do it while using bulk load for live changes, refuses the policy.
SQL Pre-Snapshot start command
Applies to: snapshot. Default: empty, nothing runs.
SQL that the target runs on its own database before writing the first page of a snapshot, for example to disable an index or a constraint. A snapshot resumed after a stop does not run it again.
Applies to the four SQL commands of this group:
-
The command runs on a connection of the target agent, not on the source, and is committed when it succeeds.
-
The text is sent as one statement. MySQL and MariaDB accept several statements separated by
;; for the other databases, send one statement or a block the database accepts as one (for example a stored procedure call). -
If the command fails, the write it surrounds fails and the entity reports a target writing error.
-
The commands run only when the agent is the target of the pipeline.
-
Apache Cassandra and ScyllaDB run the command as CQL, and Databricks as Databricks SQL.
If a target filter removes rows from the first page of the snapshot, the command can be skipped. Check the target after the first snapshot.
Declared by: IBM i (AS400), Apache Cassandra, ClickHouse, CockroachDB, Databricks, Db2 LUW, GridGain, Informix, MariaDB, MySQL, Microsoft SQL Server, Oracle, PostgreSQL, ScyllaDB, SingleStore, Snowflake, SAP SQL Anywhere, SAP ASE (Sybase), Vertica, YugabyteDB.
SQL Post-Snapshot stop command
Applies to: snapshot. Default: empty.
SQL that the target runs once the snapshot of the entity is complete, for example to rebuild what the pre-snapshot command disabled or to refresh statistics. The rules of SQL Pre-Snapshot start command apply.
SQL Pre-CDC start command
Applies to: live changes. Default: empty.
SQL that the target runs before writing each batch of live changes, not once when CDC starts. Keep it short: it runs as often as the entity receives changes. The rules of SQL Pre-Snapshot start command apply.
SQL Post-CDC stop command
Applies to: live changes. Default: empty.
SQL that the target runs after writing each batch of live changes. It runs after the rows are written: if it fails, the rows are already on the target and the entity reports an error. The rules of SQL Pre-Snapshot start command apply.
Max Write Batch Size
Applies to: snapshot and live changes. Unit: rows.
The most rows the target sends to the database in one batch of inserts and updates. Larger batches mean fewer round trips but more memory and longer transactions.
| Target | Default |
|---|---|
Apache Cassandra, CockroachDB, GridGain, Microsoft SQL Server, PostgreSQL, Vertica, YugabyteDB |
1000 |
ClickHouse (shown as Bulk insert batch size), Amazon Redshift |
10000 |
Snowflake |
100000 |
-
On Apache Cassandra the batch is also limited by Max batch size in kb.
-
On ClickHouse it also sets the size of the delete markers written to
ReplacingMergeTreetables. -
On Snowflake it applies to multi-table entities only; a single-table entity always loads through a stage.
-
MariaDB, MySQL, SingleStore and Informix show the setting but do not use it in this release.
Max Delete Batch Size
Applies to: live changes. Unit: rows. Default: 1000, Informix 100000.
The most deletes the target sends to the database in one batch. On ClickHouse it is shown as Delete mutation batch size, and in append mode each batch is also limited by the size of its keys.
Declared by: ClickHouse, CockroachDB, GridGain, Informix, MariaDB, MySQL, Microsoft SQL Server, PostgreSQL, Amazon Redshift, SingleStore, Snowflake (multi-table entities only), Vertica, YugabyteDB. MariaDB, MySQL and SingleStore show the setting but do not use it in this release.
Max Write Bulk Size
Applies to: snapshot with bulk load. Unit: rows. Default: 250000, Vertica 5000.
When bulk load is on, the snapshot collects rows in a staging table or file and loads them into the target table when this many rows are collected, and once more at the end. During live changes each batch is loaded on its own, so this value has no effect there. On Amazon Redshift it is the most rows per file.
Declared by: IBM i (AS400), Informix, MariaDB, MySQL, Microsoft SQL Server, Oracle, PostgreSQL, Amazon Redshift, SingleStore, Snowflake, Vertica.
Write page size
Applies to: snapshot and live changes. Unit: rows. Default: 250.
The number of rows the target writes together, when Write page multiplier is 0:
-
Azure Cosmos DB: one bulk request per page;
-
ScyllaDB: one CQL batch per page;
-
Aerospike: rows per batch write, when Use batch writes is on;
-
Apache Kafka: rows prepared together; producer batching is set on the agent.
Couchbase shows it as Write page size (deprecated, ignored): the Couchbase target sets how many writes are in flight by itself.
Write page multiplier
Applies to: snapshot and live changes. Default: 0, off.
When above 0, the page size is computed from the size of the data instead of Write page size: rows per page = multiplier ÷ size in KB of the whole batch being written, at least 1. Bigger rows give smaller pages. Declared by the same targets as Write page size; ignored by Couchbase.
Maximum row count in CSV file
Applies to: snapshot. Unit: rows. Default: 250000.
During a snapshot, the target closes and uploads a CSV file when it holds this many rows, or earlier when the agent runs low on memory. During live changes every batch is uploaded as its own file.
Declared by: Amazon S3, Azure Data Lake Storage, Google Cloud Storage, FTP and file storage, Google BigQuery (for its load files).
Maximum row count in Parquet file
Applies to: snapshot and live changes. Unit: rows. Default: 250000, Microsoft Fabric 1000000.
A snapshot Parquet file is closed when it holds this many rows. During live changes a file is normally closed at the end of each batch. Microsoft Fabric Open Mirroring caps each change file at the lower of this value and its own batch size.
Declared by: Amazon S3, Azure Data Lake Storage, Google Cloud Storage, FTP and file storage, Microsoft Fabric.
Bulk batch size override
Default: 0, the agent’s value. Agent: Elasticsearch.
The most actions in one _bulk request for this entity. 0 uses the batch size set on the agent. Requests are also limited by the agent’s maximum request size.
Rows per bulk snapshot load
Applies to: snapshot. Unit: rows. Default: 1000000. Agent: Databricks.
The snapshot stages rows in a file and loads it into the table with one MERGE from a Unity Catalog volume when the file holds this many rows, and once more at the end. The entity’s progress moves forward when each load completes.
Maximum rows per MERGE statement
Unit: rows. Default: 1000. Agent: Databricks.
The most rows in one generated MERGE, for upserts and deletes. A statement is also kept under 3,000,000 characters.
Max batch size in kb
Unit: KB. Default: 5. Agent: Apache Cassandra.
Keeps each CQL batch under this size: the batch holds as many rows as fit in 70% of this size, given the average row size, and never more than Max Write Batch Size. Raise it only if the Cassandra cluster accepts larger batches.
Partition key field
Default: id. Agent: Azure Cosmos DB.
The document field whose value is the partition key of every write and delete. It must match the partition key path of the Cosmos DB container; the agent does not check it.
Database name override
Default: empty, the table name. Agent: CouchDB.
The database the entity writes to, instead of one named after the table. The agent’s database prefix is still added, the name is lowercased, characters CouchDB does not accept become _, and gs_ is added in front of a name that does not start with a letter.
Index name override
Default: empty. Agent: Elasticsearch.
The index the entity writes to. It takes precedence over the index name template of the agent and over the default name. The name is lowercased and characters Elasticsearch does not accept are replaced.
Ingest pipeline
Default: empty, none. Agent: Elasticsearch.
The Elasticsearch ingest pipeline that processes the inserted and updated documents of the entity. Deletes do not go through it.
Stream key override
Default: empty. Agent: Redis Streams.
The stream the entity writes to, used as it is, instead of the stream key template of the agent or the default key.
Pipeline batch size override
Default: 0, the agent’s value. Agent: Redis Streams.
The number of XADD commands sent in one round trip for this entity.
Strong consistency
Default: on. Agent: Snowflake, multi-table entities.
When on, an upsert first reads which keys exist, then updates those rows and inserts the others. When off, it runs a plain batched update and fails if the number of updated rows differs from the batch. Single-table entities do not use this setting.
TTL
Applies to: snapshot and live changes. Default: 0.
The time to live of each record the target writes. The unit and the meaning of 0 depend on the target:
| Target | Unit | 0 means |
|---|---|---|
Aerospike |
seconds |
The default TTL of the namespace |
Couchbase |
seconds |
No expiry |
Solace |
milliseconds |
No TTL. Delete messages never get one. |
Use batch writes
Default: off. Agent: Aerospike.
When on, rows are written with batch upserts, in pages set by Write page size and Write page multiplier, and the snapshot writes pages in parallel. Rows with skipped columns are always written one at a time.
Minimum batch size
Unit: rows. Default: 500. Agent: RavenDB.
A snapshot batch with at least this many rows is written with RavenDB bulk insert, compressed. Smaller batches and live changes are written as a session batch.
Use persisted messages
Default: off. Agent: Solace.
When on, messages are published as persistent (guaranteed) messages and each page waits for the broker to acknowledge them. When off, they are published as direct messages, faster but without acknowledgement.
Message priority
Default: 4. Agent: Solace.
The priority of the data messages of the entity. Delete messages do not carry it.
Max outstanding element count
Unit: messages. Default: 100. Agent: Google Pub/Sub.
The most messages the publisher keeps waiting for Pub/Sub at once; when it is reached, writing waits. It also caps how many rows are published together.
Fine tuning tab
How the entity reads from the source and writes to the target. The same settings appear in the group settings of the pipeline, to change several entities at once. Leave a field empty to use its default.
Allowed operations
Applies to: snapshot and live changes. Default: all: insert, update, delete, truncate.
The operations the target writes. A change of an operation that is not selected is dropped by the target. Use it, for example, to keep on the target the rows deleted on the source: leave out Delete.
|
Max number of items per iteration on fetch
Applies to: snapshot and live changes. Default: 10000. Range: greater than 0.
How many rows the source asks the database for in one round trip. On JDBC sources it is the fetch size of the snapshot read and of the change queries; on MongoDB the cursor batch size; on ScyllaDB the page size; on DynamoDB Streams the records per read (at most 1000). It does not limit how many rows go into one message: Max CDC message size (KB) and Max snapshot page size (KB) do.
During live changes, a read that returns this many rows means a backlog: the next read starts at once, without waiting for the Polling interval (ms).
Max CDC message size (KB)
Applies to: live changes. Default: 4096 KB (4 MB). Range: 1 to 1048576 KB.
The size at which the source closes a message of live changes and hands it to the target. A message is a page of changed rows that Core Hub passes from the source agent to the target agent in memory; the size is an estimate of that memory, not of the bytes on the network. A message closes as soon as it reaches the size, so it can be slightly larger.
-
A large source transaction is split into several messages, so it reaches the target as several writes.
-
Small transactions are grouped into one message until it reaches the size.
-
Lower it when the target or Core Hub is short on memory, or to get smaller target transactions; raise it to write fewer, larger batches.
A Snapshot only entity keeps the value but never uses it, so the field is disabled.
Max snapshot page size (KB)
Applies to: snapshot. Default: 8192 KB (8 MB). Range: 1 to 1048576 KB.
The size at which the snapshot closes a page of rows and hands it to the target, measured like Max CDC message size (KB). Larger pages mean fewer, bigger writes and more memory per page. Snapshot autotuning can try other values in its trial runs.
Polling interval (ms)
Applies to: live changes. Default: 500 ms. Range: greater than 0.
How long the entity waits before reading the source for new changes again, after a read that found no change or less than a full read (see Max number of items per iteration on fetch). Lower values reduce latency and add load on the source; higher values do the opposite. On Oracle triggers it is the delay between reads of the change table, using the lowest value among the entities of the table; on PostgreSQL it is the longest wait between attempts while no change arrives.
Unchanged data filter
Default: Disabled.
This setting is kept so that existing configurations still load, but it has no effect in this release, whatever its value.
To skip updates that change nothing, use the write policy changedColumnsPolicy of the target, set through the REST API or the entity import file: CHANGED_COLUMNS_ONLY writes only the changed columns, SKIP_UNCHANGED_ROWS writes the whole row but skips an update whose non-key columns are all unchanged. Both need a source that provides the before image of an update, and apply to live updates only.
Snapshot writing concurrency
Applies to: snapshot. Default: 2. Range: 1 to 10; a value outside is brought to the nearest bound.
How many parts of a snapshot page the target writes in parallel. Higher values speed up the snapshot of a large table on a target that can take parallel writes, and add load on it.
Parallel writes are not used when bulk load is on for the snapshot, and some targets always write one part at a time: Snowflake, Databricks, Microsoft Fabric, Couchbase, the HTTP target and the file targets (unless they write JSON files). On Couchbase and on Aerospike with batch writes, the value sets the number of write lanes instead.
On duplicate key
Applies to: rows written as inserts: live inserts and an Empty the table first or Delete only some rows first snapshot. Default: Upsert.
What to do when the target refuses an insert because a row with the same key is already there:
| Choice | What happens |
|---|---|
Upsert |
The row is written again as an update of the existing row. |
Skip |
The existing row is kept, and a warning notification lists the skipped keys. |
Fail |
The rows are logged and the entity stops with an error. |
Other write errors are reported as they are. While the entity replays changes after a restart from a point in time, duplicates are always upserted. Rows written with bulk load do not use this setting.
The field is shown only on targets where a duplicate key can happen. A warning icon means it depends on the target table: the choice applies only where the target enforces the key, for example on Elasticsearch, Greenplum, MongoDB, Snowflake and Vertica.
Snapshot only
Default: off.
The entity is copied by snapshots only and never captures live changes. The source creates no change-capture objects for it, such as triggers, logical replication slots or change tables. On Oracle triggers, Db2, SAP HANA, PostgreSQL and YugabyteDB, turning it on removes the ones the entity had. Starting the entity runs the snapshot and stops there.
Once its snapshot is complete, starting the entity again does nothing: run a one-shot snapshot to refresh its data, or schedule one with Chronos.
Resumable snapshot
Applies to: snapshot, on single-table SQL sources. Default: on.
When on, and the entity keys contain the primary key of the table, the snapshot reads the table in key order, so that an interrupted snapshot continues where it stopped: it skips, by count, the rows already copied.
Turn it off when sorting the whole table costs the source too much. The table is then read as it is stored, and an interrupted snapshot starts over; Continue snapshot is refused.
The field is shown only for entities that read in key order. Other entities are always treated as resumable.
Bulk load settings
Default: both off.
Writes rows by staging them in a table or a file of the target and loading them in bulk, instead of with batched statements. Bulk load is faster for large volumes. The dialog shows only the modes the target supports.
| Check box | Effect |
|---|---|
Bulk for Snapshot |
The snapshot is written in bulk. See also Max Write Bulk Size. |
Bulk for CDC |
Live changes are written in bulk. On most targets this also turns bulk load on for the snapshot. |
| Targets | Supported modes |
|---|---|
IBM i (AS400), Google BigQuery, Greenplum, Informix, MariaDB, MySQL, Microsoft SQL Server, Oracle, PostgreSQL, SAP HANA, SingleStore, SAP SQL Anywhere, Vertica |
Snapshot and CDC |
Databricks, Amazon Redshift |
Snapshot only |
-
If a bulk load fails, the target writes the same rows with batched statements.
-
PostgreSQL does not use bulk load on tables with identity columns.
-
Bulk load cannot be combined with target scalar functions, and bulk for CDC cannot be combined with column-skip write policies.
-
Rows written in bulk do not use On duplicate key, and the snapshot does not write in parallel (Snapshot writing concurrency).
Related pages
-
Source agent advanced settings and Target agent advanced settings, the settings of the agent rather than of one entity