Google BigQuery agent

Target Google BigQuery

Prerequisites

To have Gluesync working on your Google BigQuery instance you will need to have:

A Google Cloud Project ID;
Target tables required to be created and having a primary key defined;
A configured Service account with:
- Permission to write to the target BigQuery dataset;
- A key (in JSON format) associated with this account created via your GCP console;

Creating a table with Primary key

To create a table with a primary key in Google BigQuery you can take the following example as a reference:

create table myschema.CUSTOMERS
(
    ID      int not null,
    NAME    string(1000),
    SURNAME string(1000),
    ADDRESS string(1000),
    GENDER  string(1000),
    PHONE   string(1000),
    EMAIL   string(1000),
    PRIMARY KEY (ID) NOT ENFORCED
)

Setup via Web UI

Project Id: Your Google Cloud Project Id;
Service Agent (.json) file: (required, defaults to NULL) The Service Agent JSON file; The certificate must be uploaded to the agent, to do it, you need to either upload it via the UI or mount the certificate as a volume.
Dataset location: (defaults to US) The location of the dataset to sync, please refer to the following documentation for more details.

Looking for more information about Google Service Account? Check out the Service Account documentation from Google.

Custom host credentials

Dataset location: (defaults to null, meaning US) The location of the dataset to sync.

Specific configuration

This agent does not implemented any specific configuration.

Setup via Rest APIs

Here following an example of calling the CoreHub’s Rest API via curl to setup the connection for this Agent.

Connect the agent

curl --location --request PUT 'http://core-hub-ip-address:1717/pipelines/{pipelineId}/agents/{agentId}/config/credentials' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
        "hostCredentials": {
        "connectionName": "myAgentNickName",
        "host": "project-id",
        "port": 443,
        "username": "",
        "password": "",
        "disableAuth": false,
        "certificatePath": "/myPath/service-agent.json"
    },
    "customHostCredentials": {
        "datasetLocation": "EU"
    }
}'