Search docs.molo17.com

    Document models

    A document model is the reverse of a Query entity. It splits one Couchbase document into rows of several SQL Server tables: the header fields go to the root table, each array of objects goes to a table of its own, and every row carries the key of its document.

    Available from Gluesync 2.3.2. For the overview of data modeling, see Data modeling.

    What you need

    Item Requirement

    Source agent

    The Couchbase agent, reading the changes through DCP (see Change data capture (DCP)). A change must carry the complete document, which DCP delivers. A source that sends only the changed fields is refused (SOURCE_PARTIAL_IMAGES).

    Target agent

    A SQL Server target agent. It advertises the DocumentModel capability, and an agent without it is refused.

    Target tables

    One SQL Server table per level, created before you save the entity, with their primary keys and foreign keys. Gluesync reads their catalog to find the key of each table, to order the writes and to check unique keys.

    Rights

    The target login needs INSERT, UPDATE and DELETE on the level tables. When a level reports its deletes (see Deletes), it also needs the right to create the table gluesync_docmodel_reported_deletes in the schema of the root table.

    Target entity

    The write operations INSERT and UPDATE allowed (TARGET_OPTION_NOT_SUPPORTED otherwise). Bulk options and a snapshot concurrency above 1 are ignored, with the warning TARGET_OPTION_IGNORED.

    How the rows are written

    Gluesync writes all the statements of one batch of documents in one transaction, with the rows of every level of a document applied together. For each document, the rows of every level are brought to exactly what the document holds now:

    • A new document inserts its root row and the rows of every level, in the order of the foreign keys.

    • A changed document rewrites its rows, adds the items that are new, updates the ones that changed, and removes the items that left the document.

    • An empty array removes every row of that level for the document.

    • A deleted document removes its rows from every table it owns, children first.

    • If several events of one document arrive in one batch, the last one wins.

    • Replaying a batch, after a restart for example, gives the same result.

    A snapshot empties the level tables first, children first, and then loads the documents. A resync does the same. To adopt rows that already exist, switch on Disable truncate on target table before snapshot inserts on the target agent: the snapshot then upserts over what is there.

    Describe the shape of your documents

    Before you write the model, sample the collection to see its shape.

    curl -X POST "https://your-gluesync.com/pipelines/$PIPELINE/agents/$SOURCE_AGENT/discovery/document-shape" \
      -H "Authorization: Bearer <your-token>" -H "Content-Type: application/json" \
      -d '{ "entityObject": { "id": 0, "scope": "bank", "collection": "holders" }, "sampleSize": 300 }'

    The id of the entity object is the id of <scope>.<collection>, computed as explained in Query entities. The MCP tool describe_document_shape takes pipeline_id, agent_id, scope, collection and sample_size and computes it for you. It samples up to 10,000 documents (300 by default), writes nothing, and reads values only to type them.

    The answer lists every path, with top-level fields by name, nested fields as address.city and the fields of array items after [], as in Cards[].CardType. For each path you get its kind (SCALAR, OBJECT, ARRAY_OF_OBJECTS, ARRAY_OF_SCALARS, or MIXED when documents disagree), the share of documents that hold it and hold null there, the types seen, the most items an array held, and the item fields that look like a key.

    A document model is a tree of levels. The root is the document. Every other level reads one place of its parent and writes to one table.

    Take this document, with the id holder::H-1001:

    {
      "HolderId": "H-1001",
      "Name": "Anna Rossi",
      "Profile": { "Email": "anna@example.com", "Country": "IT" },
      "Cards": [
        {
          "CardId": "C-1",
          "CardType": "debit",
          "Tokens": [ { "TokenId": "T-1", "Status": "active" } ]
        }
      ]
    }

    It becomes one row in HOLDERS, one in HOLDER_PROFILE, one in CARDS and one in TOKENS. The model has four levels.

    Level path cardinality key parentLink (child to parent)

    holders (root)

    HolderId

    profile

    Profile

    ONE

    HolderId

    HolderId to HolderId

    cards

    Cards

    MANY

    HolderId, CardId

    HolderId to HolderId

    tokens, under cards

    Tokens

    MANY

    HolderId, CardId, TokenId

    HolderId to HolderId, CardId to CardId

    • path is the field of the parent that holds the level, exact and case-sensitive. The first part of the path must be a column of the source entity that the changes carry (CONTAINER_PATH_NOT_AN_ENTITY_COLUMN).

    • cardinality is MANY for an array of objects and ONE for an object.

    • columns are the fields of the level, as paths relative to the item. The root declares no columns: its columns are those of the source entity.

    • key is the full key of the level: the key of its parent followed by the item key. The root key identifies the document.

    • parentLink says which column of the child takes the value of which column of the parent. The child row receives these values from its parent row, so you do not need to repeat them in every item, and every table down the tree carries the root key.

    The SQL Server tables for this model:

    CREATE TABLE dbo.HOLDERS (
      HolderId varchar(40) NOT NULL PRIMARY KEY, Name nvarchar(200) NULL);
    CREATE TABLE dbo.HOLDER_PROFILE (
      HolderId varchar(40) NOT NULL PRIMARY KEY REFERENCES dbo.HOLDERS (HolderId),
      Email nvarchar(200) NULL, Country char(2) NULL);
    CREATE TABLE dbo.CARDS (
      HolderId varchar(40) NOT NULL REFERENCES dbo.HOLDERS (HolderId),
      CardId varchar(40) NOT NULL, CardType varchar(20) NULL,
      PRIMARY KEY (HolderId, CardId));
    CREATE TABLE dbo.TOKENS (
      HolderId varchar(40) NOT NULL, CardId varchar(40) NOT NULL,
      TokenId varchar(40) NOT NULL, Status varchar(20) NULL,
      PRIMARY KEY (HolderId, CardId, TokenId),
      FOREIGN KEY (HolderId, CardId) REFERENCES dbo.CARDS (HolderId, CardId));

    Where the root key comes from

    Gluesync must be able to find the rows of a document, also when the document is deleted and only its id is left. The root key comes from one of two places.

    • The document id. The keyMapping of the source entity describes how the id is composed (holder::H-1001 is the prefix holder, the separator ::, and the key HolderId). Gluesync takes the root key out of the id. This is the default, and the key columns of the entity are flagged as primary key.

    • A column that stores the id. Set documentIdColumn in the write policy of the root: the id is written to that column of the root table, which needs a unique index (DOCUMENT_ID_COLUMN_NOT_UNIQUE otherwise). A delete finds the rows through it, and a document keeps its root key. Use it when the key lives in the body of the document and not in its id. Without it, a model whose root key is read from the body is refused (ROOT_KEY_MUTABLE).

    When the root key of a document changes in its body with the same id, the document fails with ROOT_KEY_CHANGED_IN_BODY. Two documents that claim one root key fail with ROOT_KEY_COLLISION. Move such rows with a resync.

    The key of every table holds the root key

    The primary key of each owned child table must contain the root key columns, directly or through its parent link (CHILD_PK_WITHOUT_ROOT_KEY). That is what keeps the rows of two documents apart, and what lets a delete scope itself to one document.

    If a child table is keyed by an id of its own, set deleteScope to CHAIN on that level. Its deletes are then scoped through the parent chain, and the problem becomes a warning. An item that moves to another document can be removed by a delayed event of the document it left. Such moves are counted in gluesync_docmodel_item_moved_across_documents_total.

    Other kinds of level

    Need How

    An embedded object with its own table

    A ONE level with embedding NESTED and its path, keyed exactly by the key of its parent. The 1:1 table is written after its parent row and deleted before it.

    Fields of a sub-object as columns of the parent row

    A ONE level with embedding MERGED. Its fields are written as dotted names (Limits.Daily) in the parent’s table. A collision with a field of the parent is refused (FLATTEN_COLLISION).

    An object split over several tables

    One ONE level for each extra table, with an empty path: it reads the object of its parent itself, and its key is exactly the key of its parent (EXTENSION_LEVEL_KEY_MISMATCH).

    An array of scalars

    A MANY level whose scalarValueColumn names the column that holds the element. Add an ordinalColumn when the values have no natural key.

    Items with no natural key

    The ordinalColumn of the level stores the array position and is part of the item key. Reordering the array then updates the rows.

    A sub-tree kept whole in one column

    List the column in jsonColumns: the value is written as canonical JSON text, with the keys sorted, no blanks, and numbers in plain notation.

    A shared lookup, such as an address used by several documents

    ownership REFERENCE: its rows are upserted and never deleted or compared.

    A model goes as deep as the root and two nested levels. A deeper level is refused (DEPTH_EXCEEDED).

    Write policy per level

    The writePolicy of a level controls what Gluesync removes.

    Setting Values Meaning

    deletes

    ON (default), REPORT_ONLY, OFF

    On the root: what happens when a document is deleted. On a child: what happens to the rows of items that are no longer in the document. See Deletes.

    absentArray

    DELETE_CHILDREN (default), KEEP_CHILDREN

    What happens to the child rows when the array is absent or null in the document. An array that is present and empty always removes the rows (subject to deletes). A container that a change does not carry at all fails the document with PATH_SHAPE_MISMATCH, and is never read as zero children.

    deleteScope

    ROOT_KEY (default), CHAIN

    How a delete finds its rows. CHAIN is for child tables whose key does not hold the root key.

    ordinalColumn, jsonColumns

    As described above.

    documentIdColumn

    Root only. The column that stores the document id.

    Deletes

    deletes Effect

    ON

    Gluesync deletes the rows. Deleting a document removes its rows from every table the entity owns, in the reverse order of the foreign keys. A batch that deletes more documents than maxDeletesPerWindow (10,000 by default) writes nothing and fails with DATA_MODEL_MASS_DELETE_HELD.

    REPORT_ONLY

    Gluesync works out what ON would delete, counts it, and deletes nothing. It records the rows per document in the table gluesync_docmodel_reported_deletes, in the schema of the root table, in the same transaction as the other writes. The count is visible in that table and in gluesync_docmodel_children_deleted_total{mode="report_only"} (documents: gluesync_docmodel_rows_written_total{op="delete_reported"}). Switch to ON after you have reviewed the count.

    OFF

    Gluesync keeps every row. Nothing is worked out and nothing is recorded.

    The target entity must allow the DELETE operation too. Without it nothing is deleted, whatever the level says.

    When you change a level from REPORT_ONLY to ON and save, Gluesync restarts the entity and deletes the recorded rows first, in one go. They are the rows whose count you signed off, so the guard maxDeletesPerWindow does not hold them back. Changing to OFF discards the record and keeps the rows. A level that stays REPORT_ONLY keeps its record.

    A TTL expiry of a document counts as a delete under onExpiry DELETE (the default). With IGNORE, the rows of documents that expire stay, and the number is counted in gluesync_docmodel_expiries_ignored_total.

    Duplicate item keys

    Two items of one array may carry the same key. The option duplicateItemKeys of the model says what happens.

    Value Effect

    FAIL_DOCUMENT (default)

    The document is a data error (DUPLICATE_ITEM_KEY) and none of it is written. The comparison is case-insensitive on case-insensitive key columns.

    LAST_WINS

    One row is written for each key, the one of the last item, and the rest of the document is written. A level keyed by the key of its parent alone also becomes possible: one row per parent, the last item’s. Gluesync reports that as the warning ITEM_KEY_MISSING. With FAIL_DOCUMENT the same level is refused.

    An item with a null key fails with NULL_ITEM_KEY, whatever the policy.

    Arrays and maps in a text column

    When a map or a list lands in a text column, structuredValueText says how it is written.

    Value Effect

    AS_DELIVERED (default)

    The target writes what it receives. A list of scalars is joined with `, `, and a list of objects is the JSON of each object joined the same way. That is not valid JSON, and a number can arrive in exponent form.

    GS1_JSON

    One JSON array or one JSON object, compact, as Gluesync 1.x wrote it on every SQL target. Keys stay in document order, strings are escaped as Gson does, dates are ISO-8601 text, integers stay integers, and a number with a fraction is written in plain decimal notation.

    For the array [{"Tier":"basic","Rate":0.0000125},{"Tier":"plus","Amount":20500000.25}]:

    structuredValueText Text in the column

    AS_DELIVERED

    {"Tier":"basic","Rate":1.25E-5}, {"Tier":"plus","Amount":2.050000025E7}

    GS1_JSON

    [{"Tier":"basic","Rate":0.0000125},{"Tier":"plus","Amount":20500000.25}]

    The plain notation matters when SQL Server reads the text back, for example with CONVERT(decimal(18,6), JSON_VALUE(col, '$.Amount')), which cannot convert an exponent. The setting applies at every level: a column of the root, of an item, and of a MERGED sub-object.

    To keep one sub-tree whole with sorted keys instead, use jsonColumns on the level.

    Field value types and date formats

    A document field can be converted before it is written. fieldValueTypes maps a field name to a type, and dateFormat sets how date fields are read.

    "options": {
      "fieldValueTypes": { "IssueDate": "LOCAL_DATE", "Amount": "BIG_DECIMAL", "ZipCode": "STRING" },
      "dateFormat": "dd/MM/yyyy"
    }
    • The types are STRING, INT, FLOAT, DOUBLE, LONG, BIG_DECIMAL, BOOLEAN, LOCAL_DATE, LOCAL_DATE_TIME, LOCAL_TIME, OFFSET_DATE_TIME and BINARY.

    • A type applies to every field of that name in three places: a field of the document, a field of an object field, and a field of each object of an array field. It does not go deeper.

    • STRING writes the text of the value. The numbers parse that text. BOOLEAN reads the text true, in any case, as true and anything else as false. BINARY decodes Base64.

    • A value that does not convert stays as it is. A field without a type is left alone.

    • dateFormat is a Java DateTimeFormatter pattern. It applies to the fields typed LOCAL_DATE, LOCAL_DATE_TIME, LOCAL_TIME and OFFSET_DATE_TIME. With the pattern dd/MM/yyyy, the text 15/03/2021 is read as 15 March 2021 and 01/01/0001 as 1 January 0001. Without a pattern, the date shapes of ISO-8601 are read.

    • A field that is null in every sampled document is typed as a nullable string and writes NULL.

    Limits on a document model

    Option Default and meaning

    maxItemsPerLevel

    10,000. An array with more items fails the document with DOCUMENT_TOO_LARGE.

    maxDocumentBytes

    None. A document above this size fails with DOCUMENT_TOO_LARGE.

    maxDeletesPerWindow

    10,000. See Deletes.

    onExpiry

    DELETE or IGNORE, see above.

    Create a document model

    A document model is saved as part of an entity, with the REST API or the MCP server. There is no wizard for it in the Control Plane.

    1. Create the SQL Server tables, one for each level.

    2. Describe the shape of the documents, as above.

    3. Write the entity. Its two sides are in agentEntities:

      • A source NoSqlEntity on the Couchbase agent, with its entityObject (scope and collection), its keyMapping, its columns (the root fields, with the container fields typed array or object) and the documentModel.

      • A target MultiTable entity on the SQL Server agent, with one table for each level and the columnsMappingMatrix that maps each field to a column.

    4. Validate it with POST /pipelines/{pipelineId}/config/entities/validate, fix the issues, and save it with PUT /pipelines/{pipelineId}/config/entities. Validation checks the model against the real tables of the target.

    5. Start the entity with a snapshot.

    The documentModel of the example above:

    {
      "documentModel": {
        "root": {
          "name": "holders",
          "key": ["HolderId"],
          "columns": [],
          "writePolicy": { "deletes": "ON" },
          "children": [
            {
              "name": "profile", "path": "Profile", "cardinality": "ONE", "embedding": "NESTED",
              "key": ["HolderId"],
              "parentLink": [{ "childColumn": "HolderId", "parentColumn": "HolderId" }],
              "columns": [{ "name": "Email", "dataType": "string" }, { "name": "Country", "dataType": "string" }]
            },
            {
              "name": "cards", "path": "Cards", "cardinality": "MANY",
              "key": ["HolderId", "CardId"],
              "parentLink": [{ "childColumn": "HolderId", "parentColumn": "HolderId" }],
              "columns": [{ "name": "CardId", "dataType": "string" }, { "name": "CardType", "dataType": "string" }],
              "children": [
                {
                  "name": "tokens", "path": "Tokens", "cardinality": "MANY",
                  "key": ["HolderId", "CardId", "TokenId"],
                  "parentLink": [
                    { "childColumn": "HolderId", "parentColumn": "HolderId" },
                    { "childColumn": "CardId", "parentColumn": "CardId" }
                  ],
                  "columns": [{ "name": "TokenId", "dataType": "string" }, { "name": "Status", "dataType": "string" }]
                }
              ]
            }
          ]
        },
        "options": { "duplicateItemKeys": "FAIL_DOCUMENT", "onExpiry": "DELETE" }
      }
    }

    In the real payload every column is a complete column object, with its id, tableId, position and the other fields of a column. The example shows only the name and the type. The matrix maps the root fields from the id of the entity object, and the fields of each other level from the id of its virtual table, which is the id of #gs_model.<level name>, with the ids of columns and labels computed as in Query entities.

    Data errors

    A document that cannot be split is a data error of that document, and the target entity’s write strategy decides what happens: FAIL stops the entity, SKIP and UPSERT skip the document and retry it on its next change.

    Code Cause

    PATH_SHAPE_MISMATCH

    A value at a declared path has another shape than the level says, for example an object where an array is declared.

    NULL_ITEM_KEY

    An item has a null key.

    DUPLICATE_ITEM_KEY

    Two items of one array share a key and duplicateItemKeys is FAIL_DOCUMENT.

    ITEM_KEY_OUT_OF_DOMAIN

    A key value does not fit its target column.

    DOCUMENT_TOO_LARGE

    The document exceeds maxItemsPerLevel or maxDocumentBytes.

    ROOT_KEY_CHANGED_IN_BODY

    The root key changed inside the document body while the id stayed the same.

    ROOT_KEY_COLLISION

    Another document already holds the root key.

    The checks of the model against its tables report these issues at save time.

    Code What it means

    ROOT_KEY_MUTABLE

    The root key is read from the body, and no documentIdColumn stores the id.

    CHILD_PK_WITHOUT_ROOT_KEY

    The primary key of a child table lacks the root key. An error, or a warning under deleteScope CHAIN.

    TARGET_KEY_MISMATCH

    The primary key of a level table is not exactly its mapped full key.

    SHARED_TARGET_TABLE

    Two entities write one owned table.

    FK_CYCLE

    The foreign keys of the level tables form a cycle, so there is no order to write them in.

    UNIQUE_KEY_EXCLUDES_ROOT_KEY

    A unique key of a level table leaves out the root key. A value that moves between documents fails until the removal lands.

    ROOT_KEY_COLLISION_POSSIBLE

    Root keys taken from ids land in case-insensitive or numeric columns, where two ids can become one key.

    MAPPING_FUNCTION_NOT_SUPPORTED

    A field function writes a key or parent link column.

    TECHNICAL_FIELD_NOT_SUPPORTED

    A technical field is mapped on a child level.

    Monitor a document model

    The target agent publishes, per entity and level, gluesync_docmodel_rows_written_total, gluesync_docmodel_children_deleted_total, gluesync_docmodel_issues_total (by issue code), gluesync_docmodel_array_absent_total, gluesync_docmodel_reported_deletes_discarded_total, gluesync_docmodel_item_moved_across_documents_total and gluesync_docmodel_expiries_ignored_total. See Metrics and insights.

    Limits

    • Source: Couchbase through DCP. Target: SQL Server.

    • DCP deletions carry only the id of the document, so deletes cannot be filtered by the category of the document.

    • Rows of documents deleted while deletes was not ON are not reconciled afterwards. Use REPORT_ONLY, which records them.

    • The tables are emptied when a snapshot starts, unless you disable the truncate on the target.