MovingLake Docs
Connectors Destinations API Documentation Security movinglake.com
MovingLake Docs

BigQuery Destination

BigQuery Destination Documentation

bigquery logo

How to use the Bigquery Destination.

This destination syncs your data to your BigQuery instance in real time and in Relational Table format. Following the ELT way of loading data MovingLake will sync raw JSONs into tables with _raw suffixes. JSON tables are also appended with the time it the row was loaded as well as appending a unique identifier and a SHA-256 HASH of the raw JSON payload. These last two fields are quite important not just for locating rows but also when doing joins across related tables as it is not common for these to natively have an ID which relates them.

After syncing the raw JSONs, MovingLake automatically transforms the data into relational tables using our Automatic Transformer. These tables are named the same as the JSON tables but without the _raw suffix. Also one raw table might generate more than one transformed table.

Connector guide.

Follow one of these next steps to allow MovingLake to connect to your instance:

Using MovingLake’s managed Service Account

  1. Log-in to your Google Cloud Console Account.
  2. Navigate to the project where your data warehouse lives.
  3. Go to IAM view and click on the +ADD button.
  4. Copy paste the address movinglake-etl-service@movinglake.iam.gserviceaccount.com in the New principals text field.
  5. Add the role Bigquery User and hit Save. Also add the Bigquery Data Editor role in the case you already have a destination dataset in mind.
  6. That’s it, now go back to your MovingLake account and create the destination.

Using a self-provisioned Service Account.

  1. Log-in to your Google Cloud Console Account.
  2. Navigate to the project where your data warehouse lives.
  3. Go to IAM but this time to the Service Account page.
  4. Create a new Service Account. Naming is not important. Be sure to add the Bigquery User role
  5. Once the Service Account is created, hit the three dots under Actions on your new Service Account row.
  6. Select Add Key, then Create new key.
  7. Select JSON if not preselected, and click on CREATE. A JSON file will be downloaded to your machine.
  8. That’s it! Now go back to your MovingLake account and create the destination. Be sure to upload the JSON file as security key when creating the destination.