Daft

What Daft is

Daft is a free, self-hosted data engine for processing multimodal and structured data in AI workloads. It uses a Python API to work with images, audio, video, text, and structured records, and includes AI operations for prompts, text and image embeddings, and zero-shot text and image classification. Processing runs locally by default and can extend across multiple machines with Ray or Kubernetes.

Supported formats include CSV, JSON, line-delimited JSON, Parquet, Hudi, Iceberg, Delta Lake, Paimon, Lance, PDF documents, and media files. Documented object storage connectors include AWS S3, Azure Blob Storage, Google Cloud Storage, Tencent Cloud COS, and GooseFS. It also documents table format support for Hudi, Iceberg, and Delta Lake, and catalog support for AWS Glue and Unity Catalog. Named AI providers include OpenAI, Hugging Face Transformers, Google, and LM Studio.

Daft is code-driven and supports Linux and macOS. Installation requires Python 3.10 or higher, and the documented installation command is `pip install -U daft`. Usage telemetry collects non-identifiable system metadata and can be disabled with `DO_NOT_TRACK=true`. The Dashboard has no built-in authentication, and its documentation advises against exposing it directly to the public internet.

Who it suits

Daft suits engineers building AI workloads that process large volumes of multimodal and structured data through Python. It is a fit for teams that want local execution with an option to distribute processing across machines, and can work with its documented storage, table, catalog, and AI provider connections.

Limits

Daft is free to use. Installation requires Python 3.10 or higher, and the Dashboard has no built-in authentication.

Getting started

1. Use a Linux or macOS environment with Python 3.10 or higher.
2. Install Daft with `pip install -U daft`.
3. Use the Python API to process supported structured or multimodal data.
4. Connect documented storage, table, catalog, or AI provider integrations as needed.
5. Run locally by default or scale across machines with Ray or Kubernetes.

Verdict

Choose Daft if you need a free, code-driven engine for AI workloads across structured data and media, with local execution and distributed scaling options. Consider the Dashboard security limit when deciding how to deploy it, especially if the Dashboard could be reachable from the public internet.