Dask

https://www.dask.org

Install

Install Ibis and dependencies for the Dask backend:

Install with the dask extra:

pip install 'ibis-framework[dask]'

And connect:

import ibis

con = ibis.dask.connect()
1
Adjust connection parameters as needed.

Install for Dask:

conda install -c conda-forge ibis-dask

And connect:

import ibis

con = ibis.dask.connect()
1
Adjust connection parameters as needed.

Install for Dask:

mamba install -c conda-forge ibis-dask

And connect:

import ibis

con = ibis.dask.connect()
1
Adjust connection parameters as needed.

dask.Backend

add_operation

add_operation(self, operation)

Add a translation function to the backend for a specific operation.

Operations are defined in ibis.expr.operations, and a translation function receives the translator object and an expression as parameters, and returns a value depending on the backend.

compile

compile(self, query, params=None, **kwargs)

Compile expr.

Returns

Type Description
dask.dask.dataframe.dask.dataframe.core.dask.dataframe.core.DataFrame | dask.dask.dataframe.dask.dataframe.core.dask.dataframe.core.Series | dask.dask.dataframe.dask.dataframe.core.dask.dataframe.core.Scalar Dask graph.

connect

connect(self, *args, **kwargs)

Connect to the database.

Parameters

Name Type Description Default
*args Mandatory connection parameters, see the docstring of do_connect for details. ()
**kwargs Extra connection parameters, see the docstring of do_connect for details. {}

Notes

This creates a new backend instance with saved args and kwargs, then calls reconnect and finally returns the newly created and connected backend instance.

Returns

Type Description
ibis.backends.base.BaseBackend An instance of the backend

create_table

create_table(self, name, obj=None, *, schema=None, database=None, temp=None, overwrite=False)

Create a table.

create_view

create_view(self, name, obj, *, database=None, overwrite=False)

database

database(self, name=None)

Return a Database object for the name database.

Parameters

Name Type Description Default
name str | None Name of the database to return the object for. None

Returns

Type Description
ibis.backends.base.Database A database object for the specified database.

drop_table

drop_table(self, name, *, force=False)

drop_view

drop_view(self, name, *, force=False)

execute

execute(self, query, params=None, limit='default', **kwargs)

Execute an expression.

from_dataframe

from_dataframe(self, df, name='df', client=None)

Construct an ibis table from a pandas DataFrame.

Parameters

Name Type Description Default
df pandas.pandas.DataFrame A pandas DataFrame required
name str The name of the pandas DataFrame 'df'
client ibis.backends.pandas.BasePandasBackend | None Client dictionary will be mutated with the name of the DataFrame, if not provided a new client is created None

Returns

Type Description
Table A table expression

get_schema

get_schema(self, table_name, database=None)

has_operation

has_operation(cls, operation)

list_tables

list_tables(self, like=None, database=None)

read_csv

read_csv(self, source, table_name=None, **kwargs)

Register a CSV file as a table in the current session.

Parameters

Name Type Description Default
source str | pathlib.pathlib.Path The data source. Can be a local or remote file, pathlike objects also accepted. required
table_name str | None An optional name to use for the created table. This defaults to a generated name. None
**kwargs typing.Any Additional keyword arguments passed to Pandas loading function. See https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html for more information. {}

Returns

Type Description
ibis.ibis.Table The just-registered table

read_delta

read_delta(self, source, table_name=None, **kwargs)

Register a Delta Lake table in the current database.

Parameters

Name Type Description Default
source str | pathlib.Path The data source. Must be a directory containing a Delta Lake table. required
table_name str | None An optional name to use for the created table. This defaults to a sequentially generated name. None
**kwargs typing.Any Additional keyword arguments passed to the underlying backend or library. {}

Returns

Type Description
ibis.ibis.Table The just-registered table.

read_json

read_json(self, path, table_name=None, **kwargs)

Register a JSON file as a table in the current backend.

Parameters

Name Type Description Default
path str | pathlib.Path The data source. A string or Path to the JSON file. required
table_name str | None An optional name to use for the created table. This defaults to a sequentially generated name. None
**kwargs typing.Any Additional keyword arguments passed to the backend loading function. {}

Returns

Type Description
ibis.ibis.Table The just-registered table

read_parquet

read_parquet(self, source, table_name=None, **kwargs)

Register a parquet file as a table in the current session.

Parameters

Name Type Description Default
source str | pathlib.pathlib.Path The data source(s). May be a path to a file, an iterable of files, or directory of parquet files. required
table_name str | None An optional name to use for the created table. This defaults to a generated name. None
**kwargs typing.Any Additional keyword arguments passed to Pandas loading function. See https://pandas.pydata.org/docs/reference/api/pandas.read_parquet.html for more information. {}

Returns

Type Description
ibis.ibis.Table The just-registered table

reconnect

reconnect(self)

Reconnect to the database already configured with connect.

register_options

register_options(cls)

Register custom backend options.

rename_table

rename_table(self, old_name, new_name)

Rename an existing table.

Parameters

Name Type Description Default
old_name str The old name of the table. required
new_name str The new name of the table. required

table

table(self, name, schema=None)

to_csv

to_csv(self, expr, path, *, params=None, **kwargs)

Write the results of executing the given expression to a CSV file.

This method is eager and will execute the associated expression immediately.

Parameters

Name Type Description Default
expr ibis.ibis.Table The ibis expression to execute and persist to CSV. required
path str | pathlib.Path The data source. A string or Path to the CSV file. required
params collections.abc.Mapping[ibis.ibis.Scalar, typing.Any] | None Mapping of scalar parameter expressions to value. None
kwargs typing.Any Additional keyword arguments passed to pyarrow.csv.CSVWriter {}
https required

to_delta

to_delta(self, expr, path, *, params=None, **kwargs)

Write the results of executing the given expression to a Delta Lake table.

This method is eager and will execute the associated expression immediately.

Parameters

Name Type Description Default
expr ibis.ibis.Table The ibis expression to execute and persist to Delta Lake table. required
path str | pathlib.Path The data source. A string or Path to the Delta Lake table. required
params collections.abc.Mapping[ibis.ibis.Scalar, typing.Any] | None Mapping of scalar parameter expressions to value. None
kwargs typing.Any Additional keyword arguments passed to deltalake.writer.write_deltalake method {}

to_pandas

to_pandas(self, expr, *, params=None, limit=None, **kwargs)

Execute an Ibis expression and return a pandas DataFrame, Series, or scalar.

Note

This method is a wrapper around execute.

Parameters

Name Type Description Default
expr ibis.ibis.Expr Ibis expression to execute. required
params collections.abc.Mapping[ibis.ibis.Scalar, typing.Any] | None Mapping of scalar parameter expressions to value. None
limit int | str | None An integer to effect a specific row limit. A value of None means “no limit”. The default is in ibis/config.py. None
kwargs typing.Any Keyword arguments {}

to_pandas_batches

to_pandas_batches(self, expr, *, params=None, limit=None, chunk_size=1000000, **kwargs)

Execute an Ibis expression and return an iterator of pandas DataFrames.

Parameters

Name Type Description Default
expr ibis.ibis.Expr Ibis expression to execute. required
params collections.abc.Mapping[ibis.ibis.Scalar, typing.Any] | None Mapping of scalar parameter expressions to value. None
limit int | str | None An integer to effect a specific row limit. A value of None means “no limit”. The default is in ibis/config.py. None
chunk_size int Maximum number of rows in each returned DataFrame batch. This may have no effect depending on the backend. 1000000
kwargs typing.Any Keyword arguments {}

Returns

Type Description
collections.abc.Iterator[pandas.pandas.DataFrame] An iterator of pandas DataFrames.

to_parquet

to_parquet(self, expr, path, *, params=None, **kwargs)

Write the results of executing the given expression to a parquet file.

This method is eager and will execute the associated expression immediately.

Parameters

Name Type Description Default
expr ibis.ibis.Table The ibis expression to execute and persist to parquet. required
path str | pathlib.Path The data source. A string or Path to the parquet file. required
params collections.abc.Mapping[ibis.ibis.Scalar, typing.Any] | None Mapping of scalar parameter expressions to value. None
**kwargs typing.Any Additional keyword arguments passed to pyarrow.parquet.ParquetWriter {}
https required

to_pyarrow

to_pyarrow(self, expr, params=None, limit=None, **kwargs)

to_pyarrow_batches

to_pyarrow_batches(self, expr, *, params=None, limit=None, chunk_size=1000000, **kwargs)

to_torch

to_torch(self, expr, *, params=None, limit=None, **kwargs)

Execute an expression and return results as a dictionary of torch tensors.

Parameters

Name Type Description Default
expr ibis.ibis.Expr Ibis expression to execute. required
params collections.abc.Mapping[ibis.ibis.Scalar, typing.Any] | None Parameters to substitute into the expression. None
limit int | str | None An integer to effect a specific row limit. A value of None means no limit. None
kwargs typing.Any Keyword arguments passed into the backend’s to_torch implementation. {}

Returns

Type Description
dict[str, torch.torch.Tensor] A dictionary of torch tensors, keyed by column name.
Back to top