BigQuery

ibis.memtable Support

The Google BigQuery backend supports memtables by constructing a string with the contents of the in-memory object. This will be very inefficient for medium to large in-memory tables. Please file an issue if you observe performance issues when using in-memory tables.

Install

Install ibis and dependencies for the BigQuery backend:

pipcondamamba

pip install 'ibis-framework[bigquery]'

conda install -c conda-forge ibis-bigquery

mamba install -c conda-forge ibis-bigquery

Connect

ibis.bigquery.connect

con = ibis.bigquery.connect(
    project_id="ibis-bq-project",
    dataset_id="testing",
)

ibis.bigquery.connect is a thin wrapper around ibis.backends.bigquery.Backend.do_connect.

Connection Parameters

do_connect(project_id=None, dataset_id='', credentials=None, application_name=None, auth_local_webserver=True, auth_external_data=False, auth_cache='default', partition_column='PARTITIONTIME', client=None, storage_client=None)

Create a Backend for use with Ibis.

Parameters:

Name	Type	Description	Default
project_id	str | None	A BigQuery project id.	None
dataset_id	str	A dataset id that lives inside of the project indicated by project_id.	''
credentials	google.auth.credentials.Credentials | None	Optional credentials.	None
application_name	str | None	A string identifying your application to Google API endpoints.	None
auth_local_webserver	bool	Use a local webserver for the user authentication. Binds a webserver to an open port on localhost between 8080 and 8089, inclusive, to receive authentication token. If not set, defaults to False, which requests a token via the console.	True
auth_external_data	bool	Authenticate using additional scopes required to query external data sources <https://cloud.google.com/bigquery/external-data-sources>_, such as Google Sheets, files in Google Cloud Storage, or files in Google Drive. If not set, defaults to False, which requests the default BigQuery scopes.	False
auth_cache	str	Selects the behavior of the credentials cache. 'default' Reads credentials from disk if available, otherwise authenticates and caches credentials to disk. 'reauth' Authenticates and caches credentials to disk. 'none' Authenticates and does not cache credentials. Defaults to 'default'.	'default'
partition_column	str | None	Identifier to use instead of default _PARTITIONTIME partition column. Defaults to 'PARTITIONTIME'.	'PARTITIONTIME'
client	bq.Client | None	A Client from the google.cloud.bigquery package. If not set, one is created using the project_id and credentials.	None
storage_client	bqstorage.BigQueryReadClient | None	A BigQueryReadClient from the google.cloud.bigquery_storage_v1 package. If not set, one is created using the project_id and credentials.	None

Returns:

Type	Description
Backend	An instance of the BigQuery backend.

ibis.connect URL format

In addition to ibis.bigquery.connect, you can also connect to BigQuery by passing a properly formatted BigQuery connection URL to ibis.connect

con = ibis.connect(f"bigquery://{project_id}/{dataset_id}")

This assumes you have already authenticated via the gcloud CLI

Finding your project_id and dataset_id

Log in to the Google Cloud Console to see which project_ids and dataset_ids are available to use.

BigQuery Authentication

The simplest way to authenticate with the BigQuery backend is to use Google's gcloud CLI tool.

Once you have gcloud installed, you can authenticate to BigQuery (and other Google Cloud services) by running

gcloud auth login

For any authentication problems, or information on other ways of authenticating, see the gcloud CLI authorization guide.

---

Last update: August 1, 2023