Why Ibis?

Ibis is the portable Python dataframe library.

If you’ve had issues with scaling data transformation code in Python, need to work with data in multiple data platforms, find yourself translating between other Python dataframe APIs, or just want a great Python dataframe experience, Ibis is for you.

Portability

You can reuse the same code across different backends.

Most Python dataframes are tightly coupled to their execution engine. And many databases only support SQL, with no Python API. Ibis solves this problem by providing a common API for data manipulation in Python, and compiling that API into the backend’s native language. This means you can learn a single API and use it across any supported backend (execution engine).

While portability with Ibis isn’t perfect, commonalities across backends and SQL dialects combined with years of engineering effort produce a full-featured and robust framework for data manipulation in Python.

In the long-term, we aim for a standard query plan Intermediate Representation (IR) like Substrait to simplify this further.

Ecosystem

Ibis is part of a larger ecosystem of Python data tools.

It is designed to work well with other tools in this ecosystem, and we continue to make it easier to use Ibis with other tools over time.

Local experience

Out of the box, Ibis offers a great local experience for working with many file formats.

DuckDB is the default backend, with Polars and DataFusion as two other great local options. Many of the backends can run locally but require more setup than a pip installation.

Scaling up and out

After prototyping on a local backend, directly scale in the cloud.

You can prototype on DuckDB and deploy with MotherDuck. You can scale from any Python client with Ibis installed to whatever your backend supports.

Use cases

You can use Ibis at any stage of your data workflow.

Use the same framework for local exploration on a few files or production workloads on the most advanced data platforms.

Ibis helps with:

data catalog exploration
exploratory data analysis
transforming data
visualizing data
data science and machine learning

Supported backends

You can install Ibis and a supported backend with pip, conda, mamba, or pixi.

Install with the bigquery extra:

pip install 'ibis-framework[bigquery]'

Connect using ibis.bigquery.connect.

Install with the clickhouse extra:

pip install 'ibis-framework[clickhouse]'

Connect using ibis.clickhouse.connect.

Install with the dask extra:

pip install 'ibis-framework[dask]'

Connect using ibis.dask.connect.

Install with the datafusion extra:

pip install 'ibis-framework[datafusion]'

Connect using ibis.datafusion.connect.

Install with the druid extra:

pip install 'ibis-framework[druid]'

Connect using ibis.druid.connect.

Install with the duckdb extra:

pip install 'ibis-framework[duckdb]'

Connect using ibis.duckdb.connect.

Install with the exasol extra:

pip install 'ibis-framework[exasol]'

Connect using ibis.exasol.connect.

Install alongside the apache-flink package:

pip install ibis-framework apache-flink

Connect using ibis.flink.connect.

Install with the impala extra:

pip install 'ibis-framework[impala]'

Connect using ibis.impala.connect.

Install with the mssql extra:

pip install 'ibis-framework[mssql]'

Connect using ibis.mssql.connect.

Install with the mysql extra:

pip install 'ibis-framework[mysql]'

Connect using ibis.mysql.connect.

Install with the oracle extra:

pip install 'ibis-framework[oracle]'

Connect using ibis.oracle.connect.

Install with the pandas extra:

pip install 'ibis-framework[pandas]'

Connect using ibis.pandas.connect.

Install with the polars extra:

pip install 'ibis-framework[polars]'

Connect using ibis.polars.connect.

Install with the postgres extra:

pip install 'ibis-framework[postgres]'

Connect using ibis.postgres.connect.

Install with the pyspark extra:

pip install 'ibis-framework[pyspark]'

Connect using ibis.pyspark.connect.

Install with the snowflake extra:

pip install 'ibis-framework[snowflake]'

Connect using ibis.snowflake.connect.

Install with the sqlite extra:

pip install 'ibis-framework[sqlite]'

Connect using ibis.sqlite.connect.

Install with the trino extra:

pip install 'ibis-framework[trino]'

Connect using ibis.trino.connect.

Warning

Note that the ibis-framework package is not the same as the ibis package in PyPI. These two libraries cannot coexist in the same Python environment, as they are both imported with the ibis module name.

Install the ibis-bigquery package:

conda install -c conda-forge ibis-bigquery

Connect using ibis.bigquery.connect.

Install the ibis-clickhouse package:

conda install -c conda-forge ibis-clickhouse

Connect using ibis.clickhouse.connect.

Install the ibis-dask package:

conda install -c conda-forge ibis-dask

Connect using ibis.dask.connect.

Install the ibis-datafusion package:

conda install -c conda-forge ibis-datafusion

Connect using ibis.datafusion.connect.

Install the ibis-druid package:

conda install -c conda-forge ibis-druid

Connect using ibis.druid.connect.

Install the ibis-duckdb package:

conda install -c conda-forge ibis-duckdb

Connect using ibis.duckdb.connect.

Install the ibis-exasol package:

conda install -c conda-forge ibis-exasol

Connect using ibis.exasol.connect.

PyFlink is not available on conda-forge; please

use pip to install the PyFlink backend instead.

Install the ibis-impala package:

conda install -c conda-forge ibis-impala

Connect using ibis.impala.connect.

Install the ibis-mssql package:

conda install -c conda-forge ibis-mssql

Connect using ibis.mssql.connect.

Install the ibis-mysql package:

conda install -c conda-forge ibis-mysql

Connect using ibis.mysql.connect.

Install the ibis-oracle package:

conda install -c conda-forge ibis-oracle

Connect using ibis.oracle.connect.

Install the ibis-pandas package:

conda install -c conda-forge ibis-pandas

Connect using ibis.pandas.connect.

Install the ibis-polars package:

conda install -c conda-forge ibis-polars

Connect using ibis.polars.connect.

Install the ibis-postgres package:

conda install -c conda-forge ibis-postgres

Connect using ibis.postgres.connect.

Install the ibis-pyspark package:

conda install -c conda-forge ibis-pyspark

Connect using ibis.pyspark.connect.

Install the ibis-snowflake package:

conda install -c conda-forge ibis-snowflake

Connect using ibis.snowflake.connect.

Install the ibis-sqlite package:

conda install -c conda-forge ibis-sqlite

Connect using ibis.sqlite.connect.

Install the ibis-trino package:

conda install -c conda-forge ibis-trino

Connect using ibis.trino.connect.

Install the ibis-bigquery package:

mamba install -c conda-forge ibis-bigquery

Connect using ibis.bigquery.connect.

Install the ibis-clickhouse package:

mamba install -c conda-forge ibis-clickhouse

Connect using ibis.clickhouse.connect.

Install the ibis-dask package:

mamba install -c conda-forge ibis-dask

Connect using ibis.dask.connect.

Install the ibis-datafusion package:

mamba install -c conda-forge ibis-datafusion

Connect using ibis.datafusion.connect.

Install the ibis-druid package:

mamba install -c conda-forge ibis-druid

Connect using ibis.druid.connect.

Install the ibis-duckdb package:

mamba install -c conda-forge ibis-duckdb

Connect using ibis.duckdb.connect.

Install the ibis-exasol package:

mamba install -c conda-forge ibis-exasol

Connect using ibis.exasol.connect.

PyFlink is not available on conda-forge; please

use pip to install the PyFlink backend instead.

Install the ibis-impala package:

mamba install -c conda-forge ibis-impala

Connect using ibis.impala.connect.

Install the ibis-mssql package:

mamba install -c conda-forge ibis-mssql

Connect using ibis.mssql.connect.

Install the ibis-mysql package:

mamba install -c conda-forge ibis-mysql

Connect using ibis.mysql.connect.

Install the ibis-oracle package:

mamba install -c conda-forge ibis-oracle

Connect using ibis.oracle.connect.

Install the ibis-pandas package:

mamba install -c conda-forge ibis-pandas

Connect using ibis.pandas.connect.

Install the ibis-polars package:

mamba install -c conda-forge ibis-polars

Connect using ibis.polars.connect.

Install the ibis-postgres package:

mamba install -c conda-forge ibis-postgres

Connect using ibis.postgres.connect.

Install the ibis-pyspark package:

mamba install -c conda-forge ibis-pyspark

Connect using ibis.pyspark.connect.

Install the ibis-snowflake package:

mamba install -c conda-forge ibis-snowflake

Connect using ibis.snowflake.connect.

Install the ibis-sqlite package:

mamba install -c conda-forge ibis-sqlite

Connect using ibis.sqlite.connect.

Install the ibis-trino package:

mamba install -c conda-forge ibis-trino

Connect using ibis.trino.connect.

Add the ibis-bigquery package:

pixi add ibis-bigquery

Connect using ibis.bigquery.connect.

Add the ibis-clickhouse package:

pixi add ibis-clickhouse

Connect using ibis.clickhouse.connect.

Add the ibis-dask package:

pixi add ibis-dask

Connect using ibis.dask.connect.

Add the ibis-datafusion package:

pixi add ibis-datafusion

Connect using ibis.datafusion.connect.

Add the ibis-druid package:

pixi add ibis-druid

Connect using ibis.druid.connect.

Add the ibis-duckdb package:

pixi add ibis-duckdb

Connect using ibis.duckdb.connect.

Add the ibis-exasol package:

pixi add ibis-exasol

Connect using ibis.exasol.connect.

PyFlink is not available on conda-forge; please

use pip to install the PyFlink backend instead.

Add the ibis-impala package:

pixi add ibis-impala

Connect using ibis.impala.connect.

Add the ibis-mssql package:

pixi add ibis-mssql

Connect using ibis.mssql.connect.

Add the ibis-mysql package:

pixi add ibis-mysql

Connect using ibis.mysql.connect.

Add the ibis-oracle package:

pixi add ibis-oracle

Connect using ibis.oracle.connect.

Add the ibis-pandas package:

pixi add ibis-pandas

Connect using ibis.pandas.connect.

Add the ibis-polars package:

pixi add ibis-polars

Connect using ibis.polars.connect.

Add the ibis-postgres package:

pixi add ibis-postgres

Connect using ibis.postgres.connect.

Add the ibis-pyspark package:

pixi add ibis-pyspark

Connect using ibis.pyspark.connect.

Add the ibis-snowflake package:

pixi add ibis-snowflake

Connect using ibis.snowflake.connect.

Add the ibis-sqlite package:

pixi add ibis-sqlite

Connect using ibis.sqlite.connect.

Add the ibis-trino package:

pixi add ibis-trino

Connect using ibis.trino.connect.

See the backend support matrix for details on operations supported. Open a feature request if you’d like to see support for an operation in a given backend. If the backend supports it, we’ll do our best to add it quickly!

Community

Community discussions primarily take place on GitHub.

Getting started

If you’re interested in trying Ibis we recommend the getting started tutorial.