What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Which is faster for CSV benchmarking: pandas, Polars, or DuckDB? In Polars’ May 2025 PDS-H benchmark at scale factor 10, Polars’ streaming engine was fastest among the tested versions, followed by DuckDB, Polars in-memory, and then pandas. That is a result for one analytical query suite—not a universal ranking of CSV readers. The right choice depends on your files, transformations, memory limits, and preferred API.
What the published speed comparison shows
Polars’ May 2025 PDS-H publication reports total times for its scale-factor-10 suite, where one scale-factor unit is roughly equivalent to 1 GB of CSV data. The figures below are project-published benchmark results for the named versions; they are suite totals, not the time required simply to open an arbitrary CSV.
As an Amazon Associate I earn from qualifying purchases.
| Tool and mode | Version in the May 2025 PDS-H run | Scale-factor-10 total |
|---|---|---|
| Polars streaming engine | 1.30.0 | 3.89 seconds |
| DuckDB | 1.3.0 | 5.87 seconds |
| Polars in-memory engine | 1.30.0 | 9.68 seconds |
| pandas | 2.2.3 | 365.71 seconds |
Polars reports that its streaming and DuckDB had similar results at scale factor 100, and that Polars streaming fell behind on query 21. pandas was included only at scale factor 10: Polars says its performance there was poor and it encountered out-of-memory failures at higher scale. These results favor streaming Polars for this suite and make clear that Polars’ execution mode matters; they do not predict every application’s outcome. Polars’ PDS-H benchmark publication provides the project’s methodology and results.
Speed, parsing accuracy, and reader throughput are different questions
A benchmark that measures analytical query execution does not tell you whether a tool will correctly interpret every CSV dialect, and a parsing-robustness score does not measure throughput. Keep those questions separate when comparing tools.
#1 Best Overall
Parsing robustness: DuckDB’s Pollock results
DuckDB’s April 2025 Pollock article reports DuckDB 1.2 at 9.961/10 simple and 9.599/10 weighted score in the configured benchmark mode. The same table lists pandas 1.4.3 at 9.895/10 simple and 9.431/10 weighted. DuckDB’s auto-detect-only mode scored 9.075/10 simple and 8.439/10 weighted. The configured DuckDB run was given known dialect and schema options, while auto-detect-only was not given the custom configuration file; those scores therefore do not represent identical prior information. The article does not list Polars in its score table. These are robustness scores, not speed measurements. DuckDB’s Pollock write-up describes the benchmark.
DuckDB’s reader history is not a head-to-head ranking
In a June 26, 2024 post about its own reader, the DuckDB project said: “DuckDB has improved CSV reader performance by nearly 3×, while adding the ability to handle many more CSV dialects automatically.” This is the project’s account of DuckDB’s progress over time, not a controlled comparison among DuckDB, pandas, and Polars. Read the DuckDB benchmark-history post.
How the tools differ in a CSV workflow
| Tool | When it fits | What to control in a comparison |
|---|---|---|
| pandas | A familiar Python DataFrame workflow with extensive CSV options. | Name the parser engine and relevant options. pandas’ stable 3.0.5 documentation describes the C and PyArrow engines as faster and the Python engine as more feature-complete; only PyArrow supports multithreading on that documentation page. pandas CSV and text-file documentation. |
| Polars | A multithreaded, single-machine DataFrame library with in-memory and streaming execution modes. | Report the mode and version rather than giving a single undifferentiated “Polars” number. Polars’ comparison guide is project guidance. Polars’ benchmark guide. |
| DuckDB | SQL-shaped filtering and aggregation, including queries directly over CSV files; its Python interface can also query Pandas and Polars DataFrames. | Record thread count, sniffing and schema options, and whether your workflow materializes a table. DuckDB CSV documentation and DuckDB Python API documentation. |
Chunking can let pandas process a read in pieces, but operations such as groupby are harder to implement chunkwise. For DuckDB, repeated CSV sniffing can add overhead when reading many small files with a shared dialect and schema; turning it off is appropriate only when you can reliably supply those common settings. DuckDB’s documentation gives one compressed-file example: loading a GZIP CSV took 107.1 seconds, compared with 121.3 seconds to decompress it separately in parallel and load the uncompressed CSV. Those timings describe DuckDB’s documented setup, not a general performance ratio. pandas chunk-reading guidance and DuckDB’s CSV guide.
Recommended Free Tools
Benchmark the job you actually need to do
A useful result answers this question: how long and how much memory does each tool need to read the same data, perform the same work, and produce an equivalent result on the same machine? Record the configuration alongside every measurement.
Quick Recap
Best Value
- Choose representative files. Preserve the real delimiter, quoting, missing-value conventions, compression, column types, and number of files. Record input size and row count.
- Define the same output. Specify the columns, types, rows, and final result to produce. Check that results agree across tools. If measuring ingestion, include parsing and type inference; if measuring analysis, apply the same filter, aggregation, join, or sort.
- Make configuration visible. Record software versions; pandas parser engine; Polars in-memory or streaming mode; DuckDB thread count and CSV auto-detection or schema options; and whether data is materialized. Do not give one tool helpful schema information while leaving another to infer it unless that difference reflects your real use case.
- Separate first reads from repeats. Measure cold and repeated reads if both occur in production. For many small files, measure CSV-sniffer overhead separately; test an explicit shared schema or dialect only if the application can provide it.
- Measure more than elapsed time. Report wall-clock time, peak memory, and result correctness. Run more than once under recorded hardware and software conditions, and avoid unrelated machine load during the comparison. pandas cautions that benchmark results can change with hardware and system stress. pandas benchmark guidance.
- Choose on workflow fit as well as speed. Consider existing pandas dependencies and syntax, Polars’ expression and streaming model, or DuckDB’s SQL and direct file querying. A hybrid workflow is also possible: DuckDB documents querying Pandas and Polars DataFrames through its Python interface.
Which one should you use?
- Choose pandas when your application already relies on pandas and its CSV controls, or when the familiar DataFrame workflow is more important than a benchmark advantage on a different workload.
- Choose Polars when its DataFrame and execution model suits the job; test streaming and in-memory modes separately because their performance can differ substantially.
- Choose DuckDB when the work is naturally expressed in SQL or you want to query CSV files directly. Include sniffing, schema choices, and any materialization in the end-to-end test.
- Do not pick solely from a ranking unless your workload, configuration, hardware, and correctness requirements match the published benchmark closely.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
