Azure Data Lake Storage Gen2 logo Azure Data Lake Storage Gen2 agent

Target Azure Data Lake Storage Gen2

Before configuring Azure Data Lake Storage Gen2 as a target, make sure you have completed the setup described in the Azure Data Lake Storage Gen2 introduction.

For a deep dive into how Parquet files are structured, buffered, and named on object-storage targets, see Parquet files support.

Prerequisites

To have Gluesync working with your Azure Data Lake Storage Gen2, you will need to have:

  • A properly configured Azure Data Lake Storage Gen2 account

  • Service principal (application) with appropriate permissions

  • Storage account name and container name

Setup via Web UI

  • Host: <storage-account-name>.dfs.core.windows.net (this will be used as the host)

  • Client ID: The Application (client) ID

  • Client Secret: The client secret for your registered application

  • Database Name: The name of your container (e.g., "datalake")

Custom host credentials

This can be also set via Rest API by specifying customHostCredentials param.

  • Client ID: The Application (client) ID (used as username in Web UI)

  • Tenant ID: The Application (tenant) ID

  • Client Secret: The client secret (used as password in Web UI)

Custom properties

  • Parquet file size threshold: (default to 250) Specifies the maximum file size (in megabytes) of the cached data before it is uploaded in Parquet file format.

  • Use relative timestamp paths: (optional, defaults to false) Use relative timestamp paths for Parquet files.

  • Create sidecar JSON compation file: (default to true) Enable or disable the creation of the metadata Json file for parquet files.

Setup via Rest APIs

Here’s an example of calling the Core Hub’s Rest API via curl to set up the connection for this Agent:

curl -X POST 'http://<gluesync-core-address>/api/v1/connections' \
  -H 'Content-Type: application/json' \
  -d '{
        "hostCredentials": {
            "connectionName": "myAgentNickName",
            "host": "https://<storage-account-name>.dfs.core.windows.net",
            "port": 443,
            "databaseName": "<your-container-name>",
            "username": "<your-client-id>",
            "password": "<your-client-secret>",
            "customHostCredentials": {
                "tenantId": "<your-tenant-id>"
            }
        }
  }'

Entity custom configuration

  • useJsonFile: (defaults false) Specifies the output file format for stored data. When set to true, data is written in JSON format. When set to false (default), data is written in Parquet format.

  • useCsvFile: (defaults false) When enabled, the agent writes entity data in CSV format. When set to false, data follows the default Parquet layout (or JSON if that option is enabled).

Azure Data Factory Integration

Gluesync is fully compatible with Azure Data Factory triggers that rely on Microsoft.Storage.BlobCreated events. When Gluesync writes files to Azure Data Lake Storage Gen2, it generates the appropriate event types that can be filtered by your Event Grid triggers.

Event Generation

Gluesync generates the following API operations when writing files:

  • PutBlob - Generated when uploading Parquet files, CSV files, and JSON documents

  • PutBlob - Generated when creating metadata files (if enabled)

These operations are fully compatible with Azure Data Factory triggers that filter for: * CopyBlob * PutBlob ← Generated by Gluesync * PutBlockList * FlushWithClose

Trigger Configuration

For optimal integration with Azure Data Factory, configure your Event Grid trigger to filter Microsoft.Storage.BlobCreated events with the following API calls:

// Example filter for Azure Data Factory trigger
if (event.eventType === 'Microsoft.Storage.BlobCreated') {
    if (['CopyBlob', 'PutBlob', 'PutBlockList', 'FlushWithClose'].includes(event.data.api)) {
        // Process Gluesync-generated files
        console.log('✓ Blob accettato:', event.data.url);
        console.log('✓ API:', event.data.api);
    }
}

Gluesync’s use of the Azure Blob Storage SDK ensures that all file creation events are properly committed and trigger the Microsoft.Storage.BlobCreated event with the correct API operation type.