Apache Iceberg

Apache Iceberg is an open table format for large analytic datasets, designed to let different engines work with the same tables. It is best suited to teams building self-hosted analytics around shared data; its snapshot history and flexible schema changes are compelling, provided the engines in use support the chosen release.

Overview

Iceberg supplies a table layer for storage you bring and operate, rather than a hosted analytics service. Spark, Trino, Flink, Presto, Hive, and Impala can work against the same tables, which makes it a practical foundation for teams that need engine choice without maintaining separate table formats for each one. Its self-hosted model puts deployment and storage decisions with the team.

Table snapshots provide useful operational control: query a prior state by version or timestamp, then roll back when a change needs to be undone. Atomic updates, serializable isolation, and consistent snapshot reads strengthen that history, while optimistic concurrency supports multiple writers and retries failed commits against the latest table state.

Key features

• SQL row changes: Merge new data, update existing rows, and issue targeted deletes. These operations suit ongoing data maintenance where replacing an entire table would be too blunt.
• Schema evolution: Add, rename, or reorder columns without rewriting the table, helping teams adapt datasets with less disruption.
• Hidden partitioning: Iceberg derives partition values and skips irrelevant partitions and files automatically. That reduces the need for queries and users to manage partition details themselves.
• Snapshots and rollback: Querying a table at a particular version or timestamp and restoring an earlier state make it easier to inspect changes and recover from unwanted updates.
• Compaction: Rewrite data files through bin-packing or sorting strategies to reorganize table data.
• Encryption: Tables can use predefined AWS, Azure, or GCP key management clients, or a custom KMS client. The catalog must preserve the table's encryption key ID for the table's lifetime, so catalog behavior is part of the security design.
• Integrations: Documented options include Spark, Flink, Kafka Connect, Hive, and catalogs such as AWS Glue, JDBC, and Nessie. This gives teams several routes into existing data stacks.

Pricing

Apache Iceberg is free and open source under the Apache License 2.0. The Apache Iceberg plan costs nothing and includes table-level data scope, dataset branching, and point-in-time rollback, with table-level snapshots. Users bring their own storage and host the deployment themselves. There is no free trial because the software is already free; the trade-off is that teams operate the environment and choose compatible engines and storage.

Platforms

Iceberg supports API, Linux, macOS, self-hosted, and Windows environments. Downloads include source archives, Spark and Flink runtime JARs, and AWS, GCP, and Azure bundles. Its deployment model suits teams comfortable integrating libraries and managing their own infrastructure.

Who it's for

Iceberg is a strong fit for data teams managing large analytic tables that need to be shared across engines, updated by multiple writers, and recoverable through table history. It also suits organizations that want control over storage, deployment, and encryption key management. Teams seeking a managed service or a graphical, turnkey data workspace should look elsewhere.

Pros and cons

• Pro: Shared tables across Spark, Trino, Flink, Presto, Hive, and Impala support mixed-engine analytics without tying a dataset to one engine.
• Pro: Snapshot queries, rollback, and concurrent-write handling give operators concrete ways to inspect and recover table changes.
• Pro: Schema changes and hidden partitioning reduce routine table maintenance as datasets evolve.
• Con: Self-hosting and bring-your-own storage require teams to own deployment and infrastructure decisions.
• Con: Version 1.12.0 removes support for Spark 3.4 and Flink 2.0, which rules out that release for workloads still depending on either version.
• Con: Encryption depends on the catalog retaining the key ID property throughout the table's lifetime, adding a requirement to catalog selection and operation.

Alternatives

For a wider comparison, see Data Version Control Tools. Choose Apache Paimon when a free, open-source data lake platform with connector and storage plugin choices is a better fit. git-annex is a free option for managing large files without storing their contents in Git, a different focus from Iceberg's analytic tables. Weights & Biases offers a free tier with five model seats, 5 GB per month of storage, and 1 GB per month of Weave ingestion, with a free trial also available.

ClearML is another freemium option; its self-hosted version is open source. Dolt pairs free open-source software with a paid DoltHub Pro plan at $5 a month, while Neon offers a free tier with project, compute, storage, branch, and egress caps. Nile has a free local option that runs on the user's machine without a cloud account. DataChain offers free local compute for a single developer across millions of records and S3, GCS, or Azure storage, alongside a Teams plan on request.

Verdict

Choose Apache Iceberg for self-hosted analytic tables that need cross-engine access, controlled schema evolution, and recoverable snapshots. Its open format and operational safeguards make it a capable foundation for substantial data workloads, while teams relying on Spark 3.4 or Flink 2.0 should avoid version 1.12.0 and verify compatibility before upgrading.