Search docs.molo17.com
    Integration guide Connectors

    Replicate a database to a data lake on S3 or Azure Data Lake Storage

    A data lake target does not hold a copy of your table: it holds files. This guide builds a pipeline from PostgreSQL to an S3 bucket, then reads the files that Gluesync writes to see exactly how a snapshot, an insert, an update and a delete appear, how column types are decided, and how to get the current state of the table back with DuckDB.

    In this guide

    You will learn how to configure an S3 or S3-compatible target, choose the file format, add a table, check the Parquet column types, run a full copy, read the change files, query the latest state of each row, and change the folder layout and the file size. It also lists what is different for Azure Data Lake Storage Gen2 and for JSON files.

    The guide builds on the Parquet files support FAQ, which explains file names, the _operation column and merging. It does not repeat those answers: it shows the steps and the files that result.

    Quick information

    Goal

    Replicate a table to a bucket as Parquet files and read the current state back

    Audience

    Data engineers and Gluesync administrators

    Level

    Intermediate

    Time required

    About 45 minutes

    Area

    Control Plane → Data pipelines → Agents setup wizard

    Applies to

    Gluesync 2.3

    Before you begin

    Make sure that:

    1. You can sign in to the Gluesync Control Plane with a user that can create pipelines;

    2. A PostgreSQL source with logical replication enabled is reachable from Core Hub, and you have a table with a primary key to replicate;

    3. An S3 bucket, or an S3-compatible store such as MinIO, exists, and you have an access key and a secret key that can write to it;

    4. Core Hub can reach the object store from its own network. Use the address that Core Hub sees, not the one that your browser sees;

    5. DuckDB, or another engine that reads Parquet, can reach the bucket, so that you can check the files.

    This guide was tested with a MinIO bucket named lake and the table data_lake_demo.invoices, in a pipeline called Data lake demo. Replace these names with yours. The table has one column of each common type (integer, smallint, bigint, numeric(10,2), numeric(12,4), double precision, boolean, date, timestamp, timestamptz, varchar, text) so that the types in the files can be checked one by one.

    Guide steps

    Optional: prepare a test bucket and a source table

    SQL

    Skip this step if you already have a bucket and a table.

    1. Create the bucket in your object store. With MinIO, mc mb local/lake creates a bucket named lake;

    2. In the source database, create the schema and the table, and insert a few rows:

    CREATE SCHEMA data_lake_demo;
CREATE TABLE data_lake_demo.invoices (
  id           integer PRIMARY KEY,
  customer     varchar(40) NOT NULL,
  quantity     smallint,
  units        bigint,
  amount       numeric(10,2),
  discount     numeric(12,4),
  rate         double precision,
  paid         boolean,
  invoice_date date,
  created_at   timestamp,
  updated_at   timestamptz,
  note         text);

INSERT INTO data_lake_demo.invoices
SELECT g, 'Customer ' || g, (g % 50)::smallint, g * 100000000::bigint, g * 10.25,
       round(g / 7.0, 4), g * 0.5, g % 2 = 0, DATE '2026-10-01' + g,
       TIMESTAMP '2026-10-01 08:30:15.123456' + g * interval '1 hour',
       TIMESTAMPTZ '2026-10-01 08:30:15.123456+00' + g * interval '1 hour',
       'note ' || g
FROM generate_series(1, 5) g;
    Expected result

    The bucket exists and is empty, and the table holds five rows.

    Choose the S3 agent as the target

    UI

    1. In the Data pipelines page, select the + button above the list of pipelines to open the Agents setup wizard;

    2. In 1. Setup source, select PostgreSQL CDC agent, enter the host, database, user and password of the source, and select Test and continue. Then select Next;

    3. In 2. Setup target(s), scroll to the Object Store group and select AWS S3 & S3-like agent.

    The target agent list with the Object Store group
    Figure 1. Select AWS S3 & S3-like agent (1) in the Object Store group.
    Expected result

    A connection form titled AWS S3 & S3-like (Parquet & JSON) opens. The Object Store group also lists Azure Data Lake Storage gen 2 agent and Google Cloud Storage agent.

    Enter the connection and choose the file type

    UI

    1. In Hostname (prepend HTTP:// or HTTPS://), enter the address of the store, with its scheme. For MinIO in the same Docker network this is http://data-lake-minio;

    2. In Port, enter the port. The default is 443, the HTTPS port of AWS S3. MinIO uses 9000;

    3. In Bucket name, enter the bucket;

    4. In Access Key and Secret Key, enter the credentials;

    5. Leave Region (S3-like may leave it empty, defaults to Global) as GLOBAL for an S3-compatible store. For AWS S3, enter the region of the bucket;

    6. In File type, keep Parquet;

    7. Leave Custom root path as / to write at the top of the bucket, or enter a folder to write under it;

    8. Select Test and continue, then Next.

    The AWS S3 connection form with Hostname
    Figure 2. Fill in the connection (1 to 6), choose the File type (7), and set the Custom root path (8).
    Expected result

    The connection test succeeds and the wizard shows the target with a green check mark and the text <access key>@<hostname>:<port>.

    File type What the agent writes

    Parquet (default)

    One file per batch, named after the time it was written. Snapshot rows go to .snap files and changes to .chng files, each with a .json sidecar. Inserts, updates and deletes all become rows of a log, with the operation in the _operation column. Use it for analytics engines.

    JSON

    One small object per row, named after the key of the row. There is no log: an update replaces the object and a delete removes it, so the bucket always shows the current rows. See Use JSON files instead of Parquet.

    CSV

    Files like the Parquet ones, written as text. The entity setting Maximum row count in CSV file limits their size. CSV was not tested for this guide.

    The file type belongs to the connection, not to the entity. Every table that uses this target writes the same format. To get Parquet for some tables and JSON for others, create two target connections.
    Name the pipeline

    UI

    1. Enter the name of the pipeline in Pipeline name. This guide uses Data lake demo;

    2. Select Finish.

    The Ready step of the wizard with the Pipeline name field marked by annotation 1 and the Finish button marked by annotation 2
    Figure 3. Enter the name (1) and select Finish (2).
    Expected result

    The pipeline opens with an empty list of entities and a message No entities found.

    Add the table to the pipeline

    UI

    1. Select Entities +;

    2. In the Object browser, open the schema of your table;

    3. Select the check box of the table. This guide selects invoices;

    4. Select Confirm and continue.

    The Object browser with the invoices table selected by annotation 1 and the Confirm and continue button marked by annotation 2
    Figure 4. Select the table (1), then Confirm and continue (2).
    Expected result

    The entity data_lake_demo.invoices appears in the pipeline with the status On hold, mapped field by field to an object with the same name. The folder in the bucket is named after the entity, with the dot replaced by an underscore: data_lake_demo_invoices.

    Check the column types of the files

    UI

    The Parquet column types come from the Target type of each field. Gluesync fills it from the source column, so a table that you add with the default mapping gets typed columns.

    1. Select the entity, then Edit;

    2. In Selected entities, select the mapping icon next to the table to open Field mapping;

    3. Read the Target type column (1). Select a type to see the types that you can choose;

    4. Select Cancel if you do not want to change anything.

    The Field mapping dialog with the Target type column marked by annotation 1
    Figure 5. The Target type column (1) decides the Parquet type. A size, as in bigdecimal (10, 2) (2), becomes the precision and scale of the column. The fx button (3) applies a field function.
    Expected result

    Every field has a target type. Gluesync warns Data type mismatch detected for quantity, because a smallint is written as int: this is expected and the value is kept.

    PostgreSQL column Target type Parquet column read back by DuckDB

    integer, smallint

    int

    INTEGER (both are 32-bit)

    bigint

    long

    BIGINT

    numeric(10,2)

    bigdecimal (10, 2)

    DECIMAL(10,2)

    double precision

    double

    DOUBLE

    boolean

    boolean

    BOOLEAN

    date

    date

    DATE

    timestamp, timestamptz

    datetime (6), offset_date_time (6)

    TIMESTAMP WITH TIME ZONE (both, microseconds)

    varchar, text

    string

    VARCHAR

    timestamp without time zone is written as a UTC timestamp: the clock value of the source is kept as it is and the reader shows it with a time zone. Cast it back in your query, for example created_at::TIMESTAMP, if you need a timestamp without a zone.
    Choose the types before the first start

    • Check every numeric column: a column declared without precision (numeric alone) is written with a wrong scale, see Decimal columns without precision;

    • Change a target type before you start the entity for the first time, never after: see the troubleshooting table.

    Start the entity with a full copy

    UI

    1. Select the Start button of the entity;

    2. In Start from, select Full copy;

    3. In Data already in the target, keep Keep and merge;

    4. Select Start entity.

    The Sync changes dialog with Full copy marked by annotation 1
    Figure 6. Select Full copy (1), keep Keep and merge (2), then select Start entity (3).
    Expected result

    The entity status changes to Active, Snapshot Progress reaches 100.0% and Snapshot Row Count shows the rows of the table. The bucket holds the first two objects, a .snap file and its .json sidecar:

    data_lake_demo_invoices/2026-10-28T20:12:29.264147137.snap
data_lake_demo_invoices/2026-10-28T20:12:29.264147137.json
    A second full copy of a table writes a new set of .snap files next to the first one. In the test, the two snapshot files of open_numbers were both still in the bucket after the second full copy. See the FAQ for retention.
    Read the snapshot file

    CLI

    Read the file with the engine that your analysts use. With DuckDB, configure the httpfs extension for your store first. For the MinIO bucket of this guide:

    INSTALL httpfs; LOAD httpfs;
CREATE SECRET lake (TYPE s3, KEY_ID '<access key>', SECRET '<secret key>',
                    ENDPOINT 'localhost:9000', USE_SSL false, URL_STYLE 'path');

DESCRIBE SELECT * FROM read_parquet('s3://lake/data_lake_demo_invoices/*.snap');
SELECT * FROM read_parquet('s3://lake/data_lake_demo_invoices/*.snap') ORDER BY id;
    Expected result

    DESCRIBE lists the typed columns of the table, followed by the three technical columns, which are always text:

    id INTEGER          customer VARCHAR     quantity INTEGER       units BIGINT
amount DECIMAL(10,2)   discount DECIMAL(12,4)   rate DOUBLE      paid BOOLEAN
invoice_date DATE   created_at TIMESTAMP WITH TIME ZONE   updated_at TIMESTAMP WITH TIME ZONE
note VARCHAR        _operation VARCHAR   _value VARCHAR   _timestamp VARCHAR

    The five rows have _operation set to U (a snapshot row is written like an update, not as an insert) and empty _value and _timestamp. The .json sidecar of the file reads {"rowCount":5, …​ "operations":[{"type":"Update"},5], "schemaVersion":2, "transactionIds":[]}.

    The columns are in the order of the entity, which is the order of the table. The technical columns come last. Parquet itself stores the key id as a required column and every other column as optional.

    Change data in the source and read the change files

    SQL

    Run a transaction that inserts, updates and deletes one row each, then read the change file:

    BEGIN;
INSERT INTO data_lake_demo.invoices VALUES (6, 'Customer 6', 6, 600000000, 61.50, 0.8571,
  3.0, true, '2026-10-07', '2026-10-01 14:30:15.123456', '2026-10-01 14:30:15.123456+00', 'note 6');
UPDATE data_lake_demo.invoices SET discount = 0.5, note = 'note 2 updated' WHERE id = 2;
DELETE FROM data_lake_demo.invoices WHERE id = 3;
COMMIT;
    SELECT regexp_extract(filename, '([0-9T:.-]+)\.chng', 1) AS file, file_row_number AS row,
       id, customer, quantity, discount, note, _operation, _value, _timestamp
FROM read_parquet('s3://lake/data_lake_demo_invoices/*.chng',
                  filename = true, file_row_number = true)
ORDER BY filename, file_row_number;
    Expected result

    One .chng file with three rows, in the order of the transaction:

    row  id  customer    quantity discount note            _operation _value     _timestamp
 0    6  Customer 6     6      0.8571   note 6              I     0/4A487F8  1793218383603
 1    2  Customer 2     2      0.5000   note 2 updated      U     0/4A48C38  1793218383609
 2    3  (empty)     (empty)  (empty)    (empty)             D     0/4A48CE0  1793218383610

    _value is the position in the PostgreSQL WAL and _timestamp is the time in milliseconds since 1970. Both are text, like _operation. The new row of the update holds all the columns and the old values are not written: an update has no before image. The delete holds only the key.

    Keep the old values of deleted rows

    SQL

    By default, PostgreSQL sends only the key of a row that is deleted. To get the whole old row in the D row, set the replica identity of the source table to FULL. Then delete a row, change the key of another, and update a third:

    ALTER TABLE data_lake_demo.invoices REPLICA IDENTITY FULL;
DELETE FROM data_lake_demo.invoices WHERE id = 4;
UPDATE data_lake_demo.invoices SET id = 50, note = 'note 5 new key' WHERE id = 5;
UPDATE data_lake_demo.invoices SET quantity = 22 WHERE id = 1;

    Run the statements one at a time, with a few seconds between them, so that each one lands in its own file. Read the files with the query of the previous step.

    Expected result

    Three more .chng files:

    file  row  id  customer    quantity discount note             _operation _value
 1     0    4  Customer 4     4      0.5714   note 4               D     0/4A4B5B8
 2     0    5  Customer 5     5      0.7143   note 5               D     0/4A4BA60
 2     1   50  Customer 5     5      0.7143   note 5 new key       I     0/4A4BA60
 3     0    1  Customer 1    22      0.1429   note 1               U     0/4A4BE38
    Source change What the file holds

    INSERT

    One row with all the columns and _operation = I.

    UPDATE (key unchanged)

    One row with the new values of all the columns and _operation = U, even with REPLICA IDENTITY FULL. There is no before image of an update.

    UPDATE that changes the key

    Two rows in the same file with the same _value: a D row for the old key, then an I row for the new key. The D row carries the old values of the row with REPLICA IDENTITY FULL. Reading the rows in file order gives the right result.

    DELETE

    One row with _operation = D and the key. With the default replica identity the other columns are empty. With REPLICA IDENTITY FULL they hold the values of the deleted row.

    REPLICA IDENTITY FULL makes PostgreSQL write the whole old row in its log for every update and delete of the table. Use it only on the tables whose before values you need. The behavior was verified with a PostgreSQL source; other sources send different old values.
    Get the current state of the table

    SQL

    The files are a log. Rebuild the table by keeping the last row of each key and dropping the keys whose last row is a delete. This is the query of the FAQ applied to the files of this guide:

    SELECT * EXCLUDE (_operation, _value, _timestamp, filename, file_row_number, rn)
FROM (
  SELECT *, row_number() OVER (PARTITION BY id
                               ORDER BY filename DESC, file_row_number DESC) AS rn
  FROM read_parquet(['s3://lake/data_lake_demo_invoices/*.snap',
                     's3://lake/data_lake_demo_invoices/*.chng'],
                    union_by_name = true, filename = true, file_row_number = true)
)
WHERE rn = 1 AND _operation <> 'D'
ORDER BY id;
    Expected result

    The same four rows as SELECT * FROM data_lake_demo.invoices ORDER BY id in the source: ids 1, 2, 6 and 50, with quantity 22 for id 1, discount 0.5000 for id 2, and no row for ids 3, 4 and 5.

    Two details of the query matter in practice:

    • ORDER BY filename DESC puts the newest file first: the names are UTC times, so they sort in write order. Do not order by _value: snapshot rows have none.

    • PARTITION BY must list every column of the key of the entity.

    Change the folder layout

    UI

    The default layout is one folder per entity at the top of the bucket (data_lake_demo_invoices/<time>.snap). To write the layout with raw/, changes or snapshot, year= and month= folders instead, turn on Use relative timestamp paths. The name is the opposite of what it does: with the option on, the agent writes the older layout described in the FAQ.

    1. Open the pipeline, select the Settings gear above the entity list, then Edit target;

    2. Select the Advanced settings tab, then Add property;

    3. In Key, select Use relative timestamp paths. The other keys in the list are Create sidecar JSON companion file and Max concurrent uploads;

    4. Turn the value on;

    5. Select Test and continue.

    The Advanced settings tab of the target with the key Use relative timestamp paths marked by annotation 1
    Figure 7. Select the key (1), turn on the value (2), then save with Test and continue (3).
    Expected result

    Saving stops the entities that use the target and starts them again. The next change file is written to the older layout, and a snapshot goes to a snapshot folder in the same way:

    raw/changes/data_lake_demo_invoices/year=2026/month=10/2026-10-28T20:17:06.654125626.parquet
raw/changes/data_lake_demo_invoices/year=2026/month=10/2026-10-28T20:17:06.654125626.json
raw/snapshot/data_lake_demo_open_numbers/year=2026/month=10/2026-10-28T21:05:46.283383717.parquet

    Choose the layout before the first start and keep it. Files that are already written stay where they are, so a folder that changes layout holds files in two places. If DuckDB reads the year= and month= folders, it adds year and month columns to the result. They are the upload time of the file, not data of your table.

    The Custom root path of the connection puts all these folders under a folder of the bucket instead of at its top.

    Choose the size of the snapshot files

    UI

    1. Open the entity, select Edit, then the gear icon next to the table;

    2. Open the Entity tab. Maximum row count in Parquet file is 250000 by default;

    3. Change the value if you need smaller or larger snapshot files, then select Confirm and continue.

    The entity settings dialog on the Entity tab with Maximum row count in Parquet file set to 2
    Figure 8. The entity settings for the snapshot files. The value was lowered to 2 for the test.
    Expected result

    The value appears under Advanced settings in the Entity Settings of the entity page. The agent checks it after each batch of rows read from the source: with the value 2, a snapshot of three rows was still written as one file with three rows. A table smaller than one batch (Max items per iteration, 10000 by default) always produces one snapshot file.

    Change files are not affected: each batch of changes is its own file. In the test, ten inserts in ten transactions, committed a few hundred milliseconds apart with the default Polling interval of 500, gave ten change files with one row each. See the FAQ for how to get fewer, larger files.

    Decimal columns without precision

    A PostgreSQL numeric column declared without precision and scale (numeric alone) maps to a bigdecimal target type with no size. In the Parquet file the column is declared as DECIMAL(38,18), but the values are not scaled to 18 decimals. They come out wrong, silently:

    Value in the source Value read back from Parquet Why

    10.25

    0.000000000000001025

    The digits 1025 are stored without the scale and read back with 18 decimals.

    0.5

    0.000000000000000005

    Same.

    0.14285714285714285714 (20 decimals)

    14.285714285714285714

    The value has more than 18 decimals and is 100 times too large.

    A column with a declared size, such as numeric(10,2) or numeric(12,4), is exact: the values in the tables above are all correct. There are two ways to avoid the wrong values:

    • Declare precision and scale on the source column, for example numeric(12,4), and add the table again;

    • Write the column as text. The target type cannot simply be changed to string: Gluesync refuses it with Incompatible data types and disables Confirm. Apply the built-in field function Convert Decimal to String to the field and the type is accepted.

    The Field mapping dialog for the open_numbers table with the price field mapped from numeric to bigdecimal
    Figure 9. A numeric column (2) has the target type bigdecimal (1) with no size. The fx button (3) opens the field functions.
    The Field mapping dialog with the banner Incompatible data types marked by annotation 1
    Figure 10. Choosing string alone (1) gives Incompatible data types (2). Apply Convert Decimal to String with the fx button (3).
    The Field mapping dialog with the Convert Decimal to String function applied to the price field
    Figure 11. With the function selected, the mismatch becomes a warning and Confirm is enabled.

    With the function applied before the first start, the Parquet column is text and the values are exact:

    id  price         label           (price is VARCHAR in the file)
 1  10.25         two decimals
 2  0.5           one decimal
 3  1234567.891   three decimals

    Set the target types and field functions before the first start of the entity. In the test, changing the target type of a column of an entity that had already written files made the next write fail with the entity error Target write failed class java.lang.String cannot be cast to class [B (notification title Class Cast). Exiting and entering maintenance mode did not clear it. Deleting the entity and adding the table again, with the new mapping chosen before the start, gives an entity that works.

    Use JSON files instead of Parquet

    Create a second target connection, as in the steps above, and select JSON in File type. The test used a second pipeline, Data lake demo JSON, on a bucket named lake-json, with the same table. After a full copy the bucket holds one object per row of the table, and nothing else: no .snap or .chng files and no sidecar:

    data_lake_demo/invoices/data_lake_demo::invoices::1
data_lake_demo/invoices/data_lake_demo::invoices::2
data_lake_demo/invoices/data_lake_demo::invoices::6
...
    The object name is <scope>/<collection>/<scope>::<collection>::<key>. The scope and the collection are the Target Scope and Target Collection of the Field mapping dialog, and the entity page shows the pattern in Document Key Preview (`data_lake_demo

    invoices :: id`). The object has no extension. Its content is the row:

    {"id":2,"customer":"Customer 2","quantity":2,"units":200000000,"amount":"20.50","discount":"0.5000",
 "rate":1.0,"paid":true,"invoice_date":"2026-10-03","created_at":"2026-10-01T10:30:15.123456",
 "updated_at":"2026-10-01T10:30:15.123456Z","note":"json update"}

    Numbers and booleans are JSON numbers and booleans. Decimals, dates and timestamps are text: ISO 8601 for dates and timestamps, with a Z for a timestamp with a time zone. The fields are in the order of the entity.

    The bucket follows the table, so there is no history:

    Source change Objects in the bucket

    INSERT

    A new object.

    UPDATE

    The object of the row is replaced by the new row. There is no old version and no operation marker.

    UPDATE of the key

    The object of the old key is deleted and an object of the new key is created. Updating id 8 to 80 left …​::80 and removed …​::8.

    DELETE

    The object is deleted from the bucket. Deleting id 7 removed …​::7.

    After the changes of the test (an update of 2, a delete of 7, an insert of 17 and a key change from 8 to 80), the bucket held the objects 1, 2, 6, 9 to 16, 17, 50 and 80, which are the rows of the source table.

    JSON is the only file type where Gluesync deletes objects. Use it when the consumer needs the current state of each row and not the changes. Use Parquet when you need the history, or when the table is large: a JSON bucket has one object per row and the agent sends one request per row.

    What is different for Azure Data Lake Storage Gen2

    The Azure Data Lake Storage gen 2 agent is in the same Object Store group of the wizard and writes the same Parquet, JSON and CSV files with the same names and layout. It was not run for this guide. The notes below come from the settings of the agent and from its reference page, and were not tested against a storage account.

    Setting Value

    Host

    https://<storage-account>.dfs.core.windows.net, the Data Lake endpoint of the account, not the blob endpoint.

    Container name

    The name of the container (the file system) that receives the files.

    User name and password

    The client ID and the client secret of an Azure application (service principal). Gluesync signs in with them to Microsoft Entra ID.

    Tenant Id

    The directory (tenant) of the application.

    File type, Custom root path

    As for S3.

    Request try timeout in seconds

    How long one request may run before the client abandons it and retries. The default is 600. Keep it high: a short value can leave an empty file behind after a slow upload.

    Emit FlushWithClose event for Event Grid triggers

    Agent setting, on by default. It lets Event Grid triggers see the file as complete.

    The service principal needs write access to the container. See Azure Data Lake Gen2 as a target for the roles and the Event Grid notes.

    You cannot try this target with Azurite, the Azure storage emulator. Azurite answers the blob API, but it rejects the Data Lake (dfs) calls that the agent makes and it does not issue Microsoft Entra tokens. Test against a real storage account.

    Troubleshooting

    Problem Possible cause What to do

    The entity stops with Target write failed class java.lang.String cannot be cast to class [B

    A target type or a field function was changed on an entity that already wrote Parquet files

    Delete the entity, add the table again and choose the types before the first start. See Decimal columns without precision.

    Decimal values in the Parquet file are far too small or too large

    The source column is numeric without precision and scale

    Declare precision and scale on the source column, or apply Convert Decimal to String. See Decimal columns without precision.

    Confirm is disabled with Incompatible data types when you pick string for a number

    Gluesync converts a number to text only through a field function

    Apply Convert Decimal to String to the field, or keep the numeric target type.

    A timestamp column shows a time zone in DuckDB

    timestamp without time zone is written as a UTC timestamp

    Cast it in the query: created_at::TIMESTAMP.

    The query result has extra year and month columns

    The older layout is on, and DuckDB reads the year= and month= folders as partitions

    Ignore the columns or select the columns by name. They are the upload time.

    The same row appears several times in the files

    Every update, every delete and every full copy adds rows. Files are never merged or removed

    Use the current-state query, which keeps the newest row of each key.

    A delete is missing from the query result

    The row came back with a later I or U, or the D row is in a file that the pattern does not match

    Select both *.snap and *.chng, or *.parquet for the older layout, and order by file name.

    You expected the deleted row to be gone from the bucket

    Parquet files are never deleted. Only the JSON file type deletes objects

    Query the current state, or use JSON files. See Use JSON files instead of Parquet.
    Verify the result

    You have successfully replicated the table to a data lake if:

    • The entity is Active and the bucket holds a .snap file with the rows of the table, and a .json sidecar;

    • DESCRIBE on the file shows typed columns, with DECIMAL(p,s) for sized decimals, and the technical columns _operation, _value and _timestamp last;

    • An insert, an update and a delete in the source appear as I, U and D rows in a .chng file;

    • The current-state query returns the same rows as the source table;

    • No numeric column without precision is written as a decimal.

    Related guides

    Contact MOLO17 Support if you need assistance with data lake targets.