Apache Arrow DataFusion

What Apache Arrow DataFusion is

Apache Arrow DataFusion is a free, extensible query engine written in Rust for developers building database and analytics systems around particular workloads. It uses Apache Arrow as its in-memory format and runs queries inside the application process, with threads for parallel execution. Developers can work through SQL and DataFrame APIs; related projects provide Python and Java interfaces. Documented query features include SQL parsing and planning, parallel and streaming execution, and optimization. Built-in inputs include CSV, Parquet, JSON, Avro, and Arrow, and developers can add sources with the TableProvider trait or customize functions, operators, and other components. Runtime controls include enforced memory limits and disk spilling for sorts, grouping, and joins. The core project is for in-process use; Ballista is a related distributed-processing extension. Rust users commonly add DataFusion from crates.io, while official Apache releases are source artifacts. The project recommends verifying releases with an OpenPGP signature or SHA-512 checksum.

Who it suits

It suits developers building customized database or analytics systems who want an in-process query engine and can work with Rust libraries or a CLI. Teams needing distributed processing can consider the related Ballista extension.

Limits

The core DataFusion project is designed for in-process use. Ballista is a related extension for distributed processing.

Getting started

1. Visit the Apache DataFusion website.
2. Rust users can add DataFusion from crates.io.
3. Alternatively, obtain an official Apache source release.
4. Verify a release artifact with its OpenPGP signature or SHA-512 checksum.

Verdict

DataFusion offers a customizable query engine with a broad set of built-in formats and runtime controls. Its core is intended for in-process use, while distributed processing is handled by the related Ballista project.