A newer version of this documentation is available.
View Latest (v2.2)

Google BigQuery Agent for Gluesync: Overview

Static

Index

Target Google BigQuery

Agent changelog

Core principles

Google BigQuery is capable of providing a flexible, scalable and cost-effective solution for analytics, data warehouse and machine learning at scale.

Gluesync offers the support to store data coming from supported data sources into Google BigQuery tables via its native BigQuery SDK, giving speed to any data integration task.

All Google Cloud regions are supported.

Change data capture

This agent does not currently support reading incremental changes from Google BigQuery.

Platform supported features

Feature Support Notes

Target role

✅

Uses the native Google BigQuery SDK to write into datasets across all regions.

Source role

❌

BigQuery acts only as a target in Gluesync.

Snapshots

✅

Supports snapshot jobs that upload Parquet/CSV files through GCS staging.

CDC

❌

BigQuery acts only as a target in Gluesync.

CDC type

N/A

BigQuery acts only as a target in Gluesync.

Bulk load

✅

Optimized for staging files in GCS and launching WRITE_TRUNCATE load jobs.

Source persisted internal cache

N/A

No disk cache—state is handled upstream in Core Hub.

Target persisted internal cache

N/A

Data moves directly through the BigQuery SDK; temporary files live only in GCS buckets.

Driver / connector

Google BigQuery SDK

Gluesync embeds the official SDK—no extra Simba licenses required.

Chains

N/A

Target-only connector.

Truncate operation support (reading)

N/A

Target-only connector; does not read from BigQuery tables.

Truncate operation support (writing)

✅

Supports the Core Hub “TRUNCATE before snapshot” behavior via WRITE_TRUNCATE load jobs.

Setup requirements

1. Prepare a Google Cloud Storage (GCS) staging bucket

Gluesync stages snapshot and CDC batches as Parquet files inside a dedicated GCS bucket before launching the WRITE_TRUNCATE load jobs toward BigQuery. Create (or reuse) a bucket that satisfies the following:

  • Located in the same Google Cloud project (recommended) and near the target datasets to minimize egress costs.

  • Versioning optional; lifecycle policies can automatically purge files once they age out.

  • Bucket name and path must be supplied in the agent configuration (stagingBucket and optional folderPrefix).

If your organization enforces VPC Service Controls, ensure the bucket is part of the same perimeter as BigQuery so the transient files can move freely.

2. Grant permissions to the agent’s identity

The service account (or user) configured in the Google BigQuery agent must have the ability to read and write objects within the staging bucket, as well as manage temporary folders created by the bulk loader. At minimum assign:

  • roles/storage.objectAdmin on the bucket (or roles/storage.legacyBucketWriter + roles/storage.legacyObjectReader if you prefer legacy ACLs)

  • roles/storage.objectViewer if you split read and write responsibilities

These privileges allow the agent to:

  1. Upload Parquet/CSV batches to gs://<bucket>/<prefix>/snapshots|changes/…​

  2. Read sidecar manifests while verifying load-job status

  3. Delete or overwrite staging files after BigQuery load jobs complete

Without these permissions, bulk load operations will fail before reaching BigQuery.

3. Provide the credentials to Gluesync

When registering the target agent in Core Hub, set:

  • Service account JSON or workload identity binding used to authenticate against both BigQuery and GCS

  • Staging bucket name and optional folder prefix

Ensure the credentials also include BigQuery dataset permissions (e.g., roles/bigquery.dataEditor, roles/bigquery.jobUser). The agent reuses the same identity for both BigQuery and GCS operations to keep audit trails consistent.

Usage considerations

Bulk load quotas

Google BigQuery enforces quotas and limits for load jobs (per table, per project, and per region). When Gluesync pushes batched Parquet files via the bulk loader, it consumes the “load job” quota and the “metadata operations” quota documented at BigQuery quotas and limits. Keep in mind:

  • Each CDC cycle typically issues one load job per table; large deployments can hit the per-table load frequency limit.

  • Concurrent loads per project are capped; Gluesync automatically staggers batches, but misconfigured schedules may still trigger “Quota exceeded” errors.

  • If you retain historical snapshots inside the same dataset, they also count toward load quotas.

Monitor BigQuery’s quota metrics in Cloud Monitoring and adjust Gluesync batch sizes or schedules to stay below the documented thresholds.

Automatic fallback to streaming

Whenever a load job returns quota-related errors, Gluesync can temporarily fall back to streaming inserts (using the BigQuery Storage Write API) until the quota cool-down window expires. Streaming has its own rate limits, but it lets replication continue at reduced throughput instead of failing the pipeline. Once the next load window opens, the agent resumes bulk loads automatically.

If you frequently hit load quotas, consider splitting datasets across projects or staggering bulk cycles via Chronos so that each table loads less often. Streaming fallback should remain a safety net, not the primary ingestion mode.

Q&A

Do I require the purchase of additional licenses for using this agent? Unlike other competitors, making use of Simba’s Google BigQuery JDBC driver, this agent does not require additional licenses to be used. Gluesync’s Google BigQuery agent works leveraging the native Google BigQuery SDK, which is provided as part of the Google Cloud SDK.