Use lazy execution as the starting point for most multi-step Polars workflows, especially when reading files: it gives Polars the whole query to optimize before it runs. Choose eager execution when you want results immediately, such as while exploring data or inspecting each intermediate step. Lazy execution can improve efficiency, but it does not guarantee a faster query, streaming, or automatic reuse of results.
What’s the difference between lazy and eager execution?
With eager execution, Polars runs each operation as you call it and returns a DataFrame you can inspect or use immediately. With lazy execution, operations build a plan represented by a LazyFrame; Polars runs that plan when you request a result, usually with .collect(). The Polars user guide recommends lazy execution in general unless you need intermediate results or are still exploring what the query should do: Polars Lazy API.
As an Amazon Associate I earn from qualifying purchases.
When should you use each mode?
| Situation | Good starting point | Why |
|---|---|---|
| Several transformations over CSV, Parquet, IPC, or JSON files | Lazy scan, transformations, then collect | Polars can optimize the query as a whole and may push eligible work into the file reader. |
| Exploring data and checking the result after each step | Eager | Each operation produces an immediate DataFrame for inspection. |
| Data is already in a DataFrame, but you want whole-query optimization for subsequent work | Convert it with .lazy(), compose operations, then collect |
You can build a lazy plan from an existing in-memory DataFrame. |
| Input may exceed available memory | Try lazy execution with the streaming engine and inspect the plan | Eligible operations may run in batches, but some work still needs in-memory execution. |
| One expensive plan branches into several outputs | Consider collect_all |
Combined execution can apply common-subplan elimination to diverging queries. |
These are starting points, not performance guarantees. The best choice depends on the query, input source, and operations the execution engine supports.
Why lazy execution often suits multi-step pipelines
A lazy scan keeps file input part of the plan. By contrast, a read_* call loads data eagerly before later transformations are added. Starting with scan_* lets Polars consider the source and downstream operations together. Its documented optimizations include:
#1 Best Overall
- Predicate pushdown: apply eligible filters earlier, potentially reducing the rows processed.
- Projection pushdown: read only the columns needed by the query.
- Slice pushdown: move eligible row limits earlier in the plan.
- Common-subplan elimination: identify repeated work within a combined plan.
- Expression simplification, join ordering, type coercion, and cardinality estimation: transform or plan parts of a query more effectively.
These are opportunities for the optimizer, not a promise that every query will use every optimization. See the Polars guides to optimizations and lazy usage.
Example: filter and select a CSV
In eager code, reading the file materializes a DataFrame before filtering and selecting:
import polars as pl
df = pl.read_csv("events.csv")
result = df.filter(pl.col("status") == "ok").select("event_id")In lazy code, scan the file, add operations to the plan, and collect the result:
import polars as pl
result = (
pl.scan_csv("events.csv")
.filter(pl.col("status") == "ok")
.select("event_id")
.collect()
)The lazy form lets Polars optimize across the scan, filter, and selection before producing the result. A lazy plan is not inherently faster for a tiny task, so choose it for its planning advantages rather than assuming a speedup. Polars API and engine behavior can change; check the documentation for the version you use.
What does .collect() do?
.collect() is the point at which a LazyFrame plan executes and yields a result. Treat the LazyFrame as a plan, not as a cached DataFrame: separate downstream queries that each call .collect() do not automatically share the work they have in common.
If a costly plan branches into multiple results, the Polars query-execution guide recommends considering collect_all so diverging queries can be executed together and common subplans can be eliminated. See query execution for collection and query reuse details.
Rank #4
Does lazy execution mean streaming or lower memory use?
No. Lazy execution describes when Polars builds and runs a query plan; streaming is a separate execution option. You can request the streaming engine with .collect(engine="streaming"). For eligible queries, it processes data in batches and can reduce memory pressure.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchStreaming has limits: some operations are inherently non-streaming or unsupported by the streaming engine, so Polars may fall back to in-memory execution. Do not assume that a lazy query, or even a query requested with the streaming engine, will keep the entire workload out of memory. The streaming guide describes the engine and its limitations.
How can you check whether the plan is optimized?
For performance-sensitive work, inspect the plan instead of inferring execution from the order of operations in your code. Call .explain() on the LazyFrame to view its query plan and check whether expected pushdowns or other optimizations appear. Polars also documents plan visualization for more detailed inspection. See query plan inspection.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
