A newer version of this documentation is available.
View Latest (v2.2)

Azure Data Lake Storage Gen2 agent

Static

Index

Target Azure Data Lake Storage Gen2

Agent changelog

Azure Data Lake Storage Gen2

Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics, built on Azure Blob Storage. It combines the power of a high-performance file system with massive scale and economy to help you speed your transition to the cloud.

Gluesync offers the support to store data coming from supported data sources into Azure Data Lake Storage buckets in Parquet file format using native SDK.

The files stored in the Azure Data Lake Storage destination bucket follow the best practices including keyspace support. This means that documents are organized within a folder path structure based on the transaction type (snapshot or changes), table name, year, month, and timestamp.

Support for JSON files remains available as an optional format, allowing users to choose based on their preference. In this case, each document is grouped by the source schema and table name, with individual files named according to their primary key.

Change data capture

This agent does not currently support reading incremental changes from Google Cloud Storage.

Platform supported features

Feature Support Notes

Target role

✅

Writes Parquet (default) or JSON batches to ADLS Gen2 via the native Azure Storage SDK.

Source role

❌

Azure Data Lake Storage Gen2 currently acts only as a target connector.

Snapshots

✅

Exports snapshot datasets into the hierarchical namespace using Gluesync’s folder structure.

CDC

❌

Azure Data Lake Storage Gen2 currently acts only as a target connector.

CDC type

N/A

Azure Data Lake Storage Gen2 currently acts only as a target connector.

Bulk load

N/A

Object storage uploads already use optimized batching; no staging-table flow required.

Source persisted internal cache

N/A

State tracking resides in Core Hub metadata; the agent keeps no disk cache.

Target persisted internal cache

✅

Files are temporarily cached on the agent before being uploaded to ADLS Gen2.

Driver / connector

Azure Storage SDK + Apache Arrow

Gluesync uses Microsoft’s SDK with hierarchical namespace, plus Arrow for Parquet writes.

Chains

N/A

Target-only connector.

Truncate operation support (reading)

N/A

Target-only connector; does not ingest from ADLS Gen2.

Truncate operation support (writing)

N/A

The agent does not delete existing files unless explicitly configured.

Prerequisites

To have Gluesync working with your Azure Data Lake Storage Gen2, you will need:

  • An active Microsoft Azure subscription

  • Sufficient permissions to create and manage Azure resources

  • Azure CLI or Azure Portal access

Setup Instructions

1. Create an Application (Service Principal) and Secret

  1. Go to Microsoft Entra ID (formerly Azure AD) → App registrations → New registration

  2. Enter a name for your application

  3. Select "Accounts in this organizational directory only" (for single-tenant)

  4. Click "Register"

  5. In the app’s overview page, copy the following:

    • Application (client) ID

    • Directory (tenant) ID

  6. Navigate to "Certificates & secrets" → "New client secret"

  7. Add a description and select an expiration period

  8. Click "Add" and immediately copy the client secret value (it will be hidden afterward)

You’ll need these values later: * Tenant ID * Client ID * Client Secret

2. Create a Storage Account (ADLS Gen2)

  1. In the Azure Portal, go to "Storage accounts" → "+ Create"

  2. Select your subscription and resource group

  3. Enter a unique name (lowercase only)

  4. Select a region

  5. Performance: Standard (typical)

  6. Redundancy: Choose between LRS or ZRS based on your needs

  7. In the "Advanced" tab, enable "Hierarchical namespace" (required for ADLS Gen2)

  8. (Optional) In the "Networking" tab, configure network settings (public endpoint is sufficient for initial setup)

  9. Review and create the storage account

3. Assign RBAC Permissions to the Application

  1. Open your newly created storage account

  2. Go to "Access Control (IAM)" → "Add" → "Add role assignment"

  3. Select the appropriate role:

    • For initial setup: "Storage Blob Data Owner" (temporary, for ACL management)

    • For production: "Storage Blob Data Contributor" (read/write access)

  4. Under "Members", select "User, group, or service principal"

  5. Click "Select members" and search for your application

  6. Click "Select" and then "Review + assign"

Note: RBAC alone is not sufficient for ADLS Gen2 - POSIX ACLs must also be configured on the filesystem.

4. Create a Container and Configure ACLs

  1. In your storage account, go to "Storage browser" → "Containers" → "+ Container"

  2. Enter a name (e.g., "datalake") and click "Create"

Credentials to save

  • URL: <storage-account-name>.dfs.core.windows.net

  • Client ID: The Application (client) ID created in the setup

  • Tenant ID: The Application (tenant) ID created in the setup

  • Client Secret: The secret value created for your application

  • Container: The name of your container (e.g., "datalake")

Private Link Configuration

If you intend to use Private Links, you must define both blob and dfs private links. This is necessary because the underlying SDK utilizes both endpoints for its operations.

For more information on how to configure private endpoints for Azure Data Lake Storage Gen2, please refer to the learn.microsoft.com/en-us/azure/storage/common/storage-private-endpoints#private-endpoints-for-azure-data-lake-storage-gen2 [official Microsoft documentation].

Next Steps

Now that you’ve set up your Azure Data Lake Storage Gen2 account, you can proceed to configure it as a target in Gluesync: