Microsoft Fabric Open Mirroring glyph Microsoft Fabric Open Mirroring agent

Target Microsoft Fabric Open Mirroring

Prerequisites

To have Gluesync writing into a Fabric mirrored database you will need:

  • A Microsoft Fabric workspace with a mirrored database that uses Open Mirroring, reachable over HTTPS from Core Hub (onelake.dfs.fabric.microsoft.com);

  • A Microsoft Entra app registration (service principal) with a client secret. Note its Application (client) ID, its Directory (tenant) ID and the secret value;

  • That app registration added to the Fabric workspace as a member with the Contributor role or higher (Manage access on the workspace);

  • The Fabric tenant setting that lets service principals use Fabric APIs, enabled for that app registration (Admin portal → Tenant settings → Developer settings).

Setup via Web UI

  • Client ID (Application ID) (username): the Application (client) ID of the app registration;

  • Client Secret (password): the secret value of the app registration;

  • Tenant ID (Entra) (tenantId): the Directory (tenant) ID of the app registration;

  • Landing zone URL (landingZoneUrl): the URL shown on the Home page of the mirrored database, in the form https://onelake.dfs.fabric.microsoft.com/<workspace id>/<mirrored database id>/Files/LandingZone. Paste it as it is: a trailing slash, a query string, the abfss:// form and the blob host are all understood;

  • Workspace ID (workspace) and Mirrored database ID (mirroredDatabase): used only when the landing zone URL is empty. Both accept the GUID or the name;

  • OneLake DFS host (host): used only without a landing zone URL. Default https://onelake.dfs.fabric.microsoft.com;

  • Optional custom path prefix under landing zone (customPrefix): a folder below Files/LandingZone under which the table folders are written.

On agent setup Gluesync signs in to Entra, asks for a OneLake token and checks that the landing zone folder exists. A failure comes back as one error that names what to change, for example a wrong client secret, a service principal with no access to the workspace, or a landing zone that does not exist. These errors are not retried.

Specific configuration

  • Seconds a change file may collect commits before it is published (cdcBatchMaxSeconds): default 1. 0 writes one file per commit. See Change files;

  • Rows after which an open change file is published (cdcBatchRows): default 100000;

  • Megabytes after which an open change file is published (cdcBatchMaxMb): default 16. Also bounds the memory a table holds while it collects changes;

  • Max concurrent uploads (maxConcurrentUploads): default 0, automatic: the larger of 16 and twice the number of tables. Files in flight across all tables;

  • Memory in MB the files being uploaded may hold (uploadMemoryBudgetMb): default 256. Uploads wait while the files in flight would hold more;

  • Parquet compression (parquetCompression): default ZSTD. SNAPPY, ZSTD, GZIP or UNCOMPRESSED; Fabric reads all of them. ZSTD files are about 3 times smaller than SNAPPY ones;

  • Rows per Parquet row group (parquetRowGroupRows): default 10000. Rows of a batch written as one row group. A batch of Core Hub is never split into smaller ones than this;

  • Upload block size in MB (uploadBlockSizeMb): default 8. Size of the blocks a file is appended in;

  • Parallel block uploads per file (uploadBlockConcurrency): default 4. Blocks of one file in flight;

  • Use the upsert row marker (4) for inserts and updates (useUpsertRowMarker): default false. See Row markers;

  • Write _partnerEvents.json at landing-zone root (writePartnerEvents): default true;

  • Emit FlushWithClose event for Event Grid triggers (emitFlushWithClose): default true. Closes each uploaded file with the FlushWithClose storage event, the one Fabric and Event Grid listeners react to;

  • Request try timeout in seconds (tryTimeoutInSeconds): default 600. Time limit of one OneLake request.

Per entity:

  • Maximum row count in Parquet file (parquetMaxRowCount): default 1000000. Rows after which a snapshot file is closed and uploaded. Lower it for very wide rows.

What is written

For a table Orders of schema sales, relative to the workspace:

<mirrored database>/Files/LandingZone/_partnerEvents.json
<mirrored database>/Files/LandingZone/sales.schema/Orders/_metadata.json
<mirrored database>/Files/LandingZone/sales.schema/Orders/00000000000000000001.parquet
<mirrored database>/Files/LandingZone/sales.schema/Orders/00000000000000000002.parquet

A table with no schema lands in <mirrored database>/Files/LandingZone/Orders.

  • Files: every Parquet file has a 20-digit name, strictly increasing and without gaps. Gluesync does not store the sequence: the first time a table is written after a start, the table folder is listed and the sequence continues after the highest 20-digit name there, which Fabric keeps for this purpose. A name is used up only when its upload succeeded, so a retried upload takes the same name and leaves no gap;

  • _metadata.json: {"keyColumns": […​]} with the primary key columns, written once per table before the first file. Fabric does not allow the key columns to change afterwards, so an existing file is never rewritten;

  • _partnerEvents.json: written at the root of the landing zone with partner name MOLO17 and source type Gluesync. Switch it off with Write _partnerEvents.json at landing-zone root;

  • Atomic publication: Fabric reads a file as soon as it appears under its 20-digit name and never accepts it changing. A file is therefore uploaded under an underscore-prefixed temporary name (_00000000000000000042.parquet), which Open Mirroring ignores, closed with a single flush and renamed to its final name in one operation that refuses to replace an existing file. A retry first looks at the final name: a file of the right size there was published by an earlier attempt and is kept, and a different file there is reported as a conflict and never overwritten;

  • Rows: a change file keeps the order of the transactions. A delete row carries the key columns only.

Row markers

Change files end with an rowMarker column (int32):

Value Meaning

0

Insert.

1

Update. Updated rows hold the full row, as the source delivers it.

2

Delete.

4

Upsert: used for inserts and updates when Use the upsert row marker is enabled. It makes a change replayed after a restart harmless.

Snapshot files have no rowMarker column. Open Mirroring treats a file without it as a plain insert load and recommends leaving it out of the initial load.

An update that changes a key column is written as a delete of the old key followed by an insert of the new row.

Open Mirroring has no truncate. A truncate request is skipped, and a new snapshot does not clear the table folder: delete the table folder in the landing zone to start a table over.

Tables without a primary key

Open Mirroring allows them, but then Fabric cannot apply updates or deletes, and a _metadata.json cannot be changed once written. For such a table Gluesync writes no _metadata.json, appends inserts and updates as new rows (marker 0) and skips deletes with a warning. The key columns are declared at the first write after a start in which the table has a primary key.

Data types

Values are written as Parquet columns. Timestamps have microsecond precision.

Gluesync type Default target type Parquet

SHORT, INT

smallint, int

INT32

LONG

long

INT64

FLOAT, DOUBLE

float, double

FLOAT, DOUBLE

BIG_DECIMAL

bigdecimal

FIXED_LEN_BYTE_ARRAY, DECIMAL(precision, scale)

BOOLEAN

boolean

BOOLEAN

STRING

string

BYTE_ARRAY, STRING

BYTE_ARRAY

binary

BYTE_ARRAY

LOCAL_DATE

date

INT32, DATE

LOCAL_TIME, OFFSET_TIME

time, offset_time

INT64, TIME_MICROS. Offset times are written in UTC.

LOCAL_DATE_TIME

datetime

INT64, TIMESTAMP_MICROS

OFFSET_DATE_TIME

offset_date_time

INT64, TIMESTAMP_MICROS, the UTC instant. Alternative target type: timestamp.

ARRAY, MAP

string

BYTE_ARRAY, STRING holding the JSON text. Open Mirroring asks for complex values as a JSON string.

Schema evolution

Open Mirroring supports adding columns: a column that appears in later Parquet files is added to the Delta table by Fabric, and Gluesync recomputes the file schema when the entity gets a column. A column that disappears from the entity is written as NULL in new rows, while old rows keep their values.

These changes are not in-place evolutions. Delete the table folder in the landing zone and recreate it, by taking a new snapshot and letting the changes follow, so that Fabric rebuilds the Delta table:

  • dropping a column;

  • renaming a column;

  • changing the type of a column. Without recreating the folder, the replication of that table stops with an error;

  • renaming a table: the folder under the new name is a new table;

  • changing the key columns of a table, which _metadata.json fixes.

Throughput

The agent does not support bulk mode: the bulk operations settings of an entity do not apply to it. Every snapshot is written the way described below, with no setting to turn on.

A snapshot streams its pages straight into one Parquet file per table, closed and published when it reaches Maximum row count in Parquet file or when the snapshot ends. A file is uploaded by creating it, appending its blocks in parallel, flushing once and renaming it; a file that fits in one block takes a create, one append that flushes, and a rename. Files of different tables upload in parallel, while the files of one table are published one at a time, in sequence order.

Change files

The changes of a table are collected and written as one Parquet file when the first of three limits is reached: Seconds a change file may collect commits (default 1 second), Rows after which an open change file is published (default 100,000) or Megabytes after which an open change file is published (default 16 MB). A closed file is queued and uploaded while the next changes are already being collected, in the order the files closed.

Gluesync confirms a change to the source only when the file that holds it is on OneLake, so a restart or a crash loses nothing. The changes of a file that was not yet published are read again and applied again, which is why inserts and updates of a collected file carry the upsert marker (4): applying them twice leaves the table as it was.

The window is a trade-off. With the default of 1 second a change reaches Fabric up to a second later than with one file per commit, and a crash makes Gluesync send again at most the commits of the files that were not published. In return the number of files stays at about one per table per second whatever the load: one file per commit (0) lands about 1.3 commits per second and table, a one-second window lands about 500 commits per second on one table. Every file costs Fabric a Delta commit, so a longer window means fewer, larger files and less work for the replication; a shorter one only pays when the delay matters more than the rate. The rate of commits with a single row is bound by Gluesync’s reading of the source and is shared by all tables; commits that carry more rows land proportionally more rows.

These numbers were measured from a developer laptop to a real landing zone, with a snapshot of 1,000,000 rows into one table. They are not a guarantee: your rate depends on your network, the width of the rows and the settings above.

Table Rate Time

Narrow, 5 columns

155,000 rows/s

6.4 s

Wide, 50 columns of mixed types

41,000 rows/s

24 s

Changes, with the defaults, from the same laptop and a PostgreSQL source:

Load Rate

1 narrow table, 1 row per commit

about 500 rows/s

10 narrow tables, 25 rows per commit

9,800 rows/s

4 wide tables (50 columns), 25 rows per commit

8,490 rows/s

4 narrow tables, 1,000 or more rows per commit

12,000 rows/s

The time is mostly Parquet encoding and the upload of the finished file. The defaults are the fastest settings of the measurements that Fabric ingested without error.

Reading the mirrored tables

The SQL analytics endpoint of the mirrored database shows a table landed in the root of the landing zone in the dbo schema, and a table landed in a <schema>.schema folder in that schema.

Monitoring

  • In Fabric, open the mirrored database and use Monitor replication to see the status of each table, the rows replicated and the time of the last update. A table whose replication stops, for example after a type change, is reported there with the error;

  • Fabric also writes its own status in the Files/Monitoring folder of the mirrored database: replicator.json for the replicator and tables.json with the status of each table (Initial or Replicating) and an errorMessage when a table fails;

  • In the Gluesync logs, every upload reports its path, duration, and the number of uploads active and queued. Warnings tell when a delete or a truncate is skipped;

  • Max concurrent uploads bounds the files in flight, so a slow OneLake shows up as queued uploads rather than as failures.

Not covered in this release

  • Microsoft Fabric Open Mirroring as a source;

  • Delimited-text landing files: only Parquet is written.