Google Cloud Storage agent

Target Google Cloud Storage

For deeper context on how Parquet payloads are buffered, versioned, and stored across cloud object stores, see Parquet files support.

Prerequisites

To have Gluesync working on your Google Cloud Storage instance you will need to have:

A Google Cloud Storage bucket;
A configured Service account with:
- Permission as Storage Object Admin, Storage Object Creator, Storage Object Viewer;
- A key (in JSON format) associated with this account created via your GCP console;

Setup via Web UI

Project Id: Your Google Cloud Project Id;
Port: Optional, defaults to 443;
Service Agent (.json) file: (required, defaults to NULL) The Service Agent JSON file; The certificate must be uploaded to the agent, to do it, you need to either upload it via the UI or mount the certificate as a volume.

To learn more about how to mount a certificate as a volume, check out this documentation.

Looking for more information about Google Service Account? Check out the [Service Account documentation](cloud.google.com/iam/docs/service-accounts) from Google.

Custom host credentials

This agent does not implemented any custom host credentials.

Specific configuration

Parquet file size threshold: (default to 250) Specifies the maximum file size (in megabytes) of the cached data before it is uploaded in Parquet file format.
Use relative timestamp paths: (optional, defaults to false) Use relative timestamp paths for Parquet files.
Create sidecar JSON compation file: (default to true) Enable or disable the creation of the metadata Json file for parquet files.

Entity custom configuration

Use JSON file: (defaults false) Specifies the output file format for stored data. When set to true, data is written in JSON format. When set to false (default), data is written in Parquet format.

Setup via Rest APIs

Here following an example of calling the Core Hub’s Rest API via curl to setup the connection for this Agent.

Connect the agent

curl --location --request PUT 'http://core-hub-ip-address/pipelines/{pipelineId}/agents/{agentId}/config/credentials' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
        "hostCredentials": {
        "connectionName": "myAgentNickName",
        "host": "project-id",
        "port": 443,
        "username": "",
        "password": "",
        "disableAuth": false,
        "certificatePath": "/myPath/service-agent.json"
    }
}'