Search docs.molo17.com
    Configuration guide Connectors

    Set up a Microsoft Fabric Open Mirroring target

    Open Mirroring lets a partner tool land change files in a folder of a Fabric mirrored database, and Fabric turns them into Delta tables. The Microsoft Fabric Open Mirroring agent is the Gluesync target that writes those files: one Parquet file per snapshot or per second of changes, with a row marker that says whether each row is an insert, an update or a delete. This guide prepares the Fabric side and the service principal, creates the connection and the pipeline, copies a PostgreSQL table with a snapshot and live changes, and shows how to read what the agent wrote and how to recognise the errors of a wrong secret, tenant or landing zone.

    In this guide

    You will give a Microsoft Entra service principal access to a Fabric workspace, copy the landing zone URL of an open mirrored database, and connect Gluesync to it. You will add a PostgreSQL table, check how its columns are typed, run a snapshot, follow live inserts, updates and deletes, and read the change file and the replication status that confirm the rows arrived. A table of errors that you can reproduce closes the guide.

    Quick information

    Goal

    Replicate a PostgreSQL table into a Fabric mirrored database through the landing zone, with a snapshot and CDC

    Audience

    Data engineers and Fabric workspace administrators

    Level

    Intermediate

    Time required

    About 45 minutes

    Area

    Fabric workspace and Microsoft Entra; Control Plane → Pipelines → Microsoft Fabric Open Mirroring target → Base settings and Advanced settings

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You have a Microsoft Fabric workspace with a mirrored database created as an open mirrored database, and Core Hub can reach onelake.dfs.fabric.microsoft.com over HTTPS. Microsoft Fabric Open Mirroring is a target only: you need another database as the source. This guide was tested with a PostgreSQL 16 source;

    2. You can register an application in Microsoft Entra, and you can add members to the Fabric workspace and change the Fabric tenant settings, or someone who can does it for you;

    3. You can sign in to the Control Plane with a role that can create pipelines;

    4. You have a source table with a primary key. This guide uses a PostgreSQL table shop.orders with 100,000 rows (id bigint primary key and ten more columns of the types in the table at the end of step 4).

    The Control Plane screens and the messages quoted here were captured against the Gluesync 2.3 release image and a real Fabric workspace. The Fabric and Entra portals change often, so follow the labels in your portal where they differ from the names below.

    Guide steps

    Give a service principal access to the mirrored database

    UI

    Gluesync signs in to Microsoft Entra as an application (a service principal) and writes to OneLake in its name. You prepare four things, and you note four values for step 3.

    1. In the Microsoft Entra admin center, register an application and add a client secret. Note the Application (client) ID, the Directory (tenant) ID and the secret value. Copy the value when it is shown: Entra shows it only once;

    2. In the Fabric workspace, select Manage access and add the application as a member with the Contributor role or higher;

    3. In the Fabric Admin portal, under Tenant settings → Developer settings, allow service principals to use Fabric APIs, for this application;

    4. Open the mirrored database and, on its Home page, copy the Landing zone URL. It has this form:

      https://onelake.dfs.fabric.microsoft.com/<workspace id>/<mirrored database id>/Files/LandingZone
    Expected result

    You have four values: the client ID, the client secret value, the tenant ID and the landing zone URL. The landing zone URL contains the workspace ID and the mirrored database ID, which you need again in step 3.

    Use the secret value, not the secret ID. Entra lists both, and the secret ID is the usual cause of a failed sign-in. Step 7 shows the message.

    Prepare the PostgreSQL source

    SQL

    Any Gluesync source can feed this target. This guide uses PostgreSQL with the WAL change data capture agent, so connect to the source database as an administrator and run the statements below. Replace the password.

    1. Turn on logical decoding and make room for one replication slot per entity. Restart PostgreSQL afterwards:

      ALTER SYSTEM SET wal_level = logical;
      ALTER SYSTEM SET max_replication_slots = 20;
      ALTER SYSTEM SET max_wal_senders = 20;
    2. Create the user that Gluesync reads with:

      CREATE USER gluesync WITH REPLICATION LOGIN PASSWORD '<a strong password>';
      GRANT USAGE ON SCHEMA shop TO gluesync;
      GRANT SELECT ON ALL TABLES IN SCHEMA shop TO gluesync;
    Expected result

    SHOW wal_level; returns logical, and the user can connect. The PostgreSQL CDC agent reference lists the other grants and settings.

    Create the pipeline and the Fabric target connection

    UI

    1. In the Control Plane, select Create your first pipeline, or the + button of the pipeline list if you already have pipelines. The Agents setup wizard opens;

    2. In 1. Setup source, choose the PostgreSQL CDC agent tile and fill Base settings with the Host name / Address, Database name, Username and Password of the source. Select Test and continue. The source appears with a green check mark. Select Next;

    3. In 2. Setup target(s), choose the Microsoft Fabric Open Mirroring agent tile and fill Base settings:

      • Client ID (Application ID): the application (client) ID of step 1;

      • Client Secret: the secret value of step 1;

      • Tenant ID (Entra): the directory (tenant) ID of step 1;

      • Landing zone URL (from the Fabric mirrored database Home page): the URL of step 1. Paste it as it is. A trailing slash, a query string and the abfss:// form are accepted too;

      • Workspace ID (used when no landing zone URL is set) and Mirrored database ID (used when no landing zone URL is set): the two IDs from the landing zone URL;

      • Optional custom path prefix under landing zone: enter /;

    4. Leave OneLake DFS host, Write _partnerEvents.json at landing-zone root and Request try timeout in seconds at their defaults. Select Test and continue. The target appears with a green check mark. Select Next, enter a Pipeline name and select Finish.

    Setup target step of the Agents setup wizard with the Microsoft Fabric Open Mirroring agent: Client ID
    Figure 1. The Microsoft Fabric Open Mirroring target connection. The values in angle brackets are placeholders.
    Expected result

    The pipeline appears in the HEALTHY group of the pipeline list, with the PostgreSQL and the Fabric icons. On Test and continue Gluesync signs in to Entra, asks OneLake for a token and checks that the landing zone folder exists, so a green check mark means that the credentials and the URL work.

    The form marks Workspace ID, Mirrored database ID and Optional custom path prefix under landing zone as required, although the landing zone URL alone is enough: when it is filled in, Gluesync ignores the two IDs. Paste the IDs, and enter / as the prefix. Gluesync trims slashes from the prefix, so the files land directly in the landing zone and not in a subfolder. Enter a folder name instead if you want the tables under Files/LandingZone/<folder>.

    See the Microsoft Fabric Open Mirroring target reference for every connection setting.

    Add the entity and review how it is mapped

    UI

    An open mirrored database is an object store, so there is no table to create: each source table becomes a folder of Parquet files in the landing zone.

    1. In the pipeline, select Entities. The Add entities page opens with the Object browser. Expand the source schema shop and select the table orders;

    2. Select the field mapping icon of the entity in Selected entities. The Field mapping dialog opens. Under Target, Target Schema (optional — maps to <Schema>.schema folder) is shop and Target Table is orders. Read the target types, and select Confirm;

    3. Select Confirm and continue.

    Field mapping dialog with source table shop.orders and target schema shop
    Figure 2. The Field mapping dialog of a table that lands in the shop.schema/orders folder, with the proposed target types.

    The target schema becomes a folder name, and an empty target schema writes the table in the root of the landing zone. For the table of this guide the agent writes in Files/LandingZone/shop.schema/orders. Fabric shows a table of a <schema>.schema folder in that schema of the SQL analytics endpoint.

    Expected result

    The entity appears in the pipeline on Hold as shop.orders. The proposed target types, with a PostgreSQL source, are:

    Source type Target type Parquet type written

    bigint

    long

    int64

    integer

    int

    int32

    varchar(40), text

    string (40), string

    string

    numeric(12,2)

    bigdecimal (12, 2)

    decimal128(12, 2)

    boolean

    boolean

    bool

    date

    date

    date32

    timestamp

    datetime (6)

    Timestamp in microseconds, marked as UTC

    timestamptz

    offset_date_time (6)

    Timestamp in microseconds, the UTC instant

    text[]

    string

    string

    A table without a primary key shows Error in field mapping. Please review under its name in the object browser, and you cannot confirm it. Choose the keys first, as Choose keys for tables without a primary key explains. The full type matrix, including the alternative target types, is in the agent reference.

    Run the snapshot

    UI

    The agent has no bulk mode. The snapshot streams its pages into one Parquet file per table, closes the file when it reaches the Maximum row count in Parquet file of the entity (1,000,000 rows by default) or when the snapshot ends, and uploads it.

    1. In the pipeline, find the entity. On Hold, select the start options button, the one next to the play button. The Snapshot (copy-only) dialog opens;

    2. Keep Keep and merge under Data already in the target and Standard under Speed;

    3. Select Start snapshot.

    The snapshot ends and the entity stays on Hold. Read the Core Hub log for the upload:

    docker logs <core hub container> 2>&1 | grep -E "_metadata.json|Parquet saved"
    Expected result

    The Last rows column shows the rows of the last batch, and the log has two lines like these, with the IDs of your workspace:

    Wrote immutable Open Mirroring _metadata.json for <mirrored database id>/Files/LandingZone/shop.schema/orders with keyColumns=[id]

    Parquet saved for entity shop.orders at <mirrored database id>/Files/LandingZone/shop.schema/orders/00000000000000000001.parquet (Open Mirroring landing zone; not Delta)

    The 100,000 rows of the test table became one Parquet file of about 650 KB, uploaded in under a second. In Fabric, open the mirrored database and select Monitor replication: the table orders is listed as replicating, with 100,000 replicated rows. Fabric needs some seconds after the upload, and up to a minute or two for the first table, to report it.

    Do not start the snapshot again to "refresh" the table. Open Mirroring has no truncate and a new snapshot does not clear the table folder, so the files add to the ones in the folder. To start a table over, delete its folder in the landing zone first. The agent reference explains what the agent skips and why.

    Capture live changes and read the change file

    UI + SQL

    1. Select the play button of the entity. The Sync changes (CDC) dialog opens. Keep Where it stopped, and notice Write method: Standard with the note This target has no bulk load. Select Start entity. The entity becomes Active;

    2. On the source, insert, update and delete rows, and set a column to NULL:

      INSERT INTO shop.orders VALUES (100001, 'new-customer', 'new', 12.50, 3, false, 'inserted', ARRAY['x'], now(), current_date, now());
      UPDATE shop.orders SET status = 'shipped', amount = 99.99 WHERE id = 5;
      DELETE FROM shop.orders WHERE id = 6;
      UPDATE shop.orders SET note = NULL WHERE id = 8;
    3. After a few seconds, read the Last rows column of the entity and the Core Hub log. Then wait for Fabric to report the new rows under Monitor replication.

    Pipeline page with the entity shop.orders in the Active group and 4 in the Last rows column
    Figure 3. The entity is Active and the last change file carried 4 rows.
    Expected result

    The Last rows column shows 4, and the log shows a second file, 00000000000000000002.parquet, in the same table folder. The file name is the next number of the sequence, without a gap. The file holds the four changes in the order of the transactions, with a __rowMarker__ column at the end:

    • the insert of row 100001 and the two updates (rows 5 and 8) carry the marker 4, the upsert. Each updated row holds the full row, so row 8 has note empty and every other column filled;

    • the delete of row 6 carries the marker 2 and fills only the key column id.

    Fabric then reports Monitor replication with 100,004 replicated rows: the 100,000 of the snapshot and the four changes.

    Inserts and updates carry the upsert marker, because the agent collects the changes of a table for Seconds a change file may collect commits before it is published (1 by default) and sends them as one file. A change that is sent twice after a restart is therefore harmless. Gluesync confirms a change to the source only when the file that holds it is on OneLake. Set the window to 0 to write one file per commit, with the insert marker 0 and the update marker 1: it lowers the delay to Fabric but also the rate, as the reference explains in Change files.

    Tune the file settings

    UI

    The defaults suit most loads. To change one, edit the target and add it as an advanced property.

    1. In the pipeline list, select the Fabric icon of the pipeline. The Edit target(s) page opens. Select Advanced settings;

    2. Select Add property, open the Key list and choose the setting, for example Seconds a change file may collect commits before it is published (0 = one file per commit). Enter the value;

    3. Select Test and continue.

    Edit target page on Advanced settings with the key Seconds a change file may collect commits before it is published and the value 1
    Figure 4. The advanced properties of the target. The key list also holds the upload, Parquet and row marker settings.
    Expected result

    The key list offers these settings: Max concurrent uploads, Emit FlushWithClose event for Event Grid triggers, Parquet compression, Rows per Parquet row group, Upload block size in MB, Memory in MB the files being uploaded may hold, Parallel block uploads per file, Rows after which an open change file is published, Megabytes after which an open change file is published, Seconds a change file may collect commits before it is published, Seconds to wait for Fabric to drop a truncated table and Use the upsert row marker (4) for inserts and updates. The reference lists the default of each.

    Fix the errors you are most likely to meet

    UI

    Each message below was reproduced on the target connection of step 3 by changing one value and selecting Test and continue. The notification Host Credentials Update Failed carries the text.

    1. Use the secret ID instead of the secret value in Client Secret. The notification reads Entra refused the service principal credentials (AADSTS7000215): the Client Secret is wrong: use the secret value, not the secret ID. Fix: copy the secret value. If it was not saved, add a new secret in Entra;

    2. Change Tenant ID (Entra) to a GUID that is not your tenant. The notification reads Entra refused the service principal credentials (AADSTS900021): AADSTS900021: Requested tenant identifier '<id>' is not valid. Fix: copy the Directory (tenant) ID of the application;

    3. Use a mirrored database ID that does not exist in the landing zone URL. The notification reads Check connection failed. Could not connect to host "https://onelake.dfs.fabric.microsoft.com" as user "<client id>". The landing zone <url> does not exist: check the workspace and the mirrored database item, and that the open mirrored database was created. The same text appears for an unknown workspace ID and for a custom path prefix whose folder does not exist. Fix: copy the URL from the Home page of the mirrored database again;

    4. Cut the URL after the database ID, or end it with another folder. The notification reads '<url>' is not a Fabric Open Mirroring landing zone URL: the mirrored database item is missing. or …​ it must end with /Files/LandingZone. and adds Copy it from the Home page of the mirrored database. Fix: paste the whole URL;

    5. Put an ID that OneLake does not accept in the URL, such as the all-zero GUID. The notification Data Lake Storage reads Status code 400, InvalidRequestUri. Fix: check both IDs in the URL.

    Expected result

    After each fix, Test and continue shows a green check mark again. These errors come from the check at connection time and are not retried.

    Data types

    The agent reads the type of each source column and writes one Parquet column of the matching type. The rules that matter most in practice:

    Source type Parquet type Note

    Integers, floating point numbers, boolean

    int32, int64, float, double, bool

    Written unchanged

    numeric(p,s)

    decimal(p,s)

    Keep the precision and the scale of the source

    varchar(n), text

    string

    No length limit in Parquet

    date

    date

    A date, without a time

    timestamp, timestamptz

    Timestamp in microseconds

    Microsecond precision. The value is the UTC instant

    Arrays such as text[]

    string

    Open Mirroring asks for complex values as a JSON string

    The reference page lists every Gluesync type, the alternative target type of OFFSET_DATE_TIME and how times with an offset are written.

    Bulk and throughput settings

    The agent has no bulk mode, so the entity dialogs offer no bulk load for this target. The settings that matter are these:

    Setting Default When to change it

    Seconds a change file may collect commits before it is published

    1

    Raise it for fewer, larger files and less work for Fabric, lower it to 0 only when the delay to Fabric matters more than the rate

    Rows after which an open change file is published

    100000

    Lower it to publish large transactions earlier

    Megabytes after which an open change file is published

    16

    Lower it to limit the memory a table holds while it collects changes

    Maximum row count in Parquet file (entity)

    1000000

    Lower it for very wide rows

    Parquet compression

    ZSTD

    SNAPPY writes bigger files and uses less CPU

    Read Tune replication throughput for the settings that apply to every target.

    Verify the result

    You have set up the Fabric target if:

    • The service principal is a Contributor of the workspace, and Test and continue on the target shows a green check mark;

    • The log shows _metadata.json written with keyColumns=[id] and one Parquet file for the snapshot;

    • Monitor replication in Fabric lists the table with the rows of the source;

    • A CDC insert, update and delete produce a second file with the markers 4 and 2, and Fabric counts the new rows;

    • No snapshot was started a second time on a table that already holds rows.

    Troubleshooting

    Problem Possible cause What to do

    Required under Workspace ID, Mirrored database ID or Optional custom path prefix under landing zone, and Test and continue does nothing

    The form marks these three fields as required, even when the landing zone URL is filled in

    Enter the two IDs from the URL and / as the prefix

    Entra refused the service principal credentials (AADSTS7000215): the Client Secret is wrong: use the secret value, not the secret ID.

    The secret ID, an expired secret or a mistyped secret is in Client Secret

    Create a secret in Entra and copy its value at once

    Entra refused the service principal credentials (AADSTS900021) …​ Requested tenant identifier '<id>' is not valid

    Tenant ID (Entra) is empty or is not the directory ID of the application

    Copy the Directory (tenant) ID from the app registration

    Check connection failed …​ The landing zone <url> does not exist

    A wrong workspace, mirrored database or prefix, or an item that is not an open mirrored database

    Copy the landing zone URL from the Home page of the mirrored database, and clear the prefix or use /

    '<url>' is not a Fabric Open Mirroring landing zone URL …​

    The URL is cut or ends with another folder

    Paste the URL up to /Files/LandingZone

    Data Lake Storage notification with Status code 400, InvalidRequestUri

    An ID in the URL is not a valid GUID or name for OneLake

    Check both IDs in the URL

    Error in field mapping. Please review in the object browser

    The source table has no primary key

    Choose the keys, see Choose keys for tables without a primary key

    The connection works, but a second snapshot adds rows to the files already in Fabric

    Open Mirroring has no truncate: a snapshot does not clear the table folder

    Delete the table folder in the landing zone before the snapshot to start a table over

    Contact MOLO17 Support if you need assistance with a Microsoft Fabric Open Mirroring target.