Azure Data Lake Storage Gen2 agent
Target Azure Data Lake Storage Gen2
Before configuring Azure Data Lake Storage Gen2 as a target, make sure you have completed the setup described in the Azure Data Lake Storage Gen2 introduction.
For a deep dive into how Parquet files are structured, buffered, and named on object-storage targets, see Parquet files support.
Prerequisites
To have Gluesync working with your Azure Data Lake Storage Gen2, you will need to have:
-
A properly configured Azure Data Lake Storage Gen2 account
-
Service principal (application) with appropriate permissions
-
Storage account name and container name
Setup via Web UI
-
Host:
<storage-account-name>.dfs.core.windows.net(this will be used as the host) -
Client ID: The Application (client) ID
-
Client Secret: The client secret for your registered application
-
Database Name: The name of your container (e.g., "datalake")
Custom host credentials
This can be also set via Rest API by specifying customHostCredentials param.
-
Client ID: The Application (client) ID (used as username in Web UI)
-
Tenant ID: The Application (tenant) ID
-
Client Secret: The client secret (used as password in Web UI)
Custom properties
-
Parquet file size threshold: (default to
250) Specifies the maximum file size (in megabytes) of the cached data before it is uploaded in Parquet file format. -
Use relative timestamp paths: (optional, defaults to
false) Use relative timestamp paths for Parquet files. -
Create sidecar JSON compation file: (default to
true) Enable or disable the creation of the metadata Json file for parquet files.
Setup via Rest APIs
Here’s an example of calling the Core Hub’s Rest API via curl to set up the connection for this Agent:
curl -X POST 'http://<gluesync-core-address>/api/v1/connections' \
-H 'Content-Type: application/json' \
-d '{
"hostCredentials": {
"connectionName": "myAgentNickName",
"host": "https://<storage-account-name>.dfs.core.windows.net",
"port": 443,
"databaseName": "<your-container-name>",
"username": "<your-client-id>",
"password": "<your-client-secret>",
"customHostCredentials": {
"tenantId": "<your-tenant-id>"
}
}
}'