Static Google Cloud Storage agent

Target Google Cloud Storage

Prerequisites

To have Gluesync working on your Google Cloud Storage instance you will need to have:

  • A Google Cloud Storage bucket;

  • A configured Service account with:

    • Permission as Storage Object Admin, Storage Object Creator, Storage Object Viewer;

    • A key (in JSON format) associated with this account created via your GCP console;

Setup via Web UI

  • Project Id: Your Google Cloud Project Id;

  • Port: Optional, defaults to 443;

  • Service Agent (.json) file: (required, defaults to NULL) The Service Agent JSON file; The certificate must be uploaded to the agent, to do it, you need to either upload it via the UI or mount the certificate as a volume.

To learn more about how to mount a certificate as a volume, check out this documentation.

Looking for more information about Google Service Account? Check out the [Service Account documentation](cloud.google.com/iam/docs/service-accounts) from Google.

Custom host credentials

This agent does not implemented any custom host credentials.

Specific configuration

  • parquetFileSizeThreshold: (default to 250) Specifies the maximum file size (in megabytes) of the cached data before it is uploaded in Parquet file format.

Entity custom configuration

  • useJsonFile: (defaults false) Specifies the output file format for stored data. When set to true, data is written in JSON format. When set to false (default), data is written in Parquet format.

Setup via Rest APIs

Here following an example of calling the Core Hub’s Rest API via curl to setup the connection for this Agent.

Connect the agent

curl --location --request PUT 'http://core-hub-ip-address:1717/pipelines/{pipelineId}/agents/{agentId}/config/credentials' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
        "hostCredentials": {
        "connectionName": "myAgentNickName",
        "host": "project-id",
        "port": 443,
        "username": "",
        "password": "",
        "disableAuth": false,
        "certificatePath": "/myPath/service-agent.json"
    }
}'