Compacting Apache Iceberg data files can reduce metadata overhead and the cost of opening many small files, but a reduction from 1,000 files to 6 does not by itself prove that queries became faster. The specific 1,000-to-6 result has not been verified here, and compaction can help some workloads while slowing others. To judge the outcome, compare the same queries before and after and report what files were rewritten, not just the new file count.
What compaction changes—and what the file count does not tell you
Iceberg’s Spark rewriteDataFiles action combines small data files into larger ones. The intended benefits are less manifest metadata to manage and lower runtime file-open costs, as Apache Iceberg explains in its maintenance documentation. Fewer files may therefore help, but the change in file count is not a query benchmark.
As an Amazon Associate I earn from qualifying purchases.
“1,000 files into 6” could describe every file in a table, files in one partition, or only the files selected for rewriting. It could also mean six files were added by the rewrite, rather than that the entire table now contains six files. Iceberg’s Spark procedure reports rewritten and added data-file counts and rewritten bytes; those values should not be conflated with the total table file count. The Spark procedure reference documents the procedure’s outputs.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Why performance may improve—or get worse
Metadata- and file-open-heavy queries
Queries that spend a substantial share of their time planning around metadata or opening many small files may benefit from a more consolidated layout. That is the mechanism Iceberg describes; the size of any improvement depends on the table, query, engine, and environment.
#1 Best Overall
Broad scans and aggregations
Compaction can change how work is divided among tasks. A 2026 study by Jannic Cutura and Subash Prakash reports that, in its benchmark, metadata-heavy queries benefited while full-scan aggregations could slow as compaction reduced task parallelism. This is a result from the authors’ setup, not a universal rule for Iceberg tables. The paper simulated 2,376 Iceberg tables and used 96 TPC-H tables for cross-schema validation; its reported model metrics do not establish the outcome of a particular compaction run. See the 2026 study.
Selective queries and broad queries are different tests
A selective query may read only a small part of a table, while a broad aggregation may need to scan most of it. Evaluate those patterns separately: a rewrite that changes planning overhead or task parallelism can affect them differently. A single query or one elapsed-time number cannot establish a general performance result.
How to verify a 1,000-to-6 result
A credible comparison needs the same workload before and after compaction, plus enough detail to tell whether the physical work changed. Record:
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minute- What the counts mean: total table files, files in a named partition, or files selected and added by the rewrite.
- The table’s size, format, partitioning or layout, and data-file size distribution before and after.
- The query text and workload type, including whether it is metadata-heavy, selective, or a full-scan aggregation.
- The query engine and exact version, storage environment, cache state, and concurrency.
- The compaction strategy and options, including target file size and the scope of the rewrite.
- Repeated before-and-after timings, with planning time separated from execution time when possible.
- Files read, bytes scanned, and task counts, alongside elapsed time, to show whether the query did less work or ran with different parallelism.
Without those measurements, the defensible conclusion is limited: the file layout became more compact if the rewrite counts confirm that, but the effect on query performance remains unestablished.
Rank #3
Choose and configure the rewrite for the deployed version
Spark
The current Iceberg Spark procedure reference documents rewrite_data_files with binpack as the default strategy and sort as an option for specified orderings, including Z-order and Hilbert. It documents a 512 MB default target when the value is inherited from the table’s write.target-file-size-bytes property, a minimum rewrite size of 75% of target, and a maximum of 180%. These are documented defaults, not a universal target recommendation; check the Iceberg release and table properties actually in use.
Iceberg’s maintenance page illustrates a Spark rewrite using a 500 MB target and describes the files metadata table as a way to inspect data-file sizes and decide whether partitions need compaction. That example is guidance, not a guarantee that 500 MB—or any other size—is right for a particular workload.
Flink
Iceberg’s Flink maintenance documentation also describes a RewriteDataFiles action and shows an example using a 256 MiB target and 32 MiB minimum. Confirm that the API and options are available in the installed Iceberg release before relying on them: Flink maintenance documentation.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWhen manifest maintenance is the separate issue
Data-file compaction and manifest rewriting address related but distinct parts of performance. Manifests and manifest lists help Iceberg plan scans and prune data; rewriting manifests can reorganize metadata when write patterns do not align with query filters. If planning is the bottleneck, compacting data files alone may not address it. Iceberg documents manifest rewriting as a separate maintenance path in its maintenance guide and Spark procedures reference.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
