Polars: The DataFrame That Replaced pandas

pandas is not slow because its authors made bad choices. It is slow because of the choices that made it useful in 2008: a mutable, row-oriented data model built on NumPy, a single-threaded execution model that predates multi-core as a first-class concern, and a Python object layer that the GIL can never fully relinquish. Those choices were correct for their era. By 2025 they are a structural liability, and no amount of optimization inside the existing architecture can fix them.

Polars is the coherent alternative. Written in Rust, built directly on Apache Arrow’s columnar memory layout, parallelized across all cores by default, and designed around a query optimizer rather than imperative mutation, Polars is not pandas with a coat of paint. It is a different model of computation. For analytical workloads — group-by aggregations, multi-column filters, Parquet scans, joins at scale — the performance difference is not incremental. In real benchmarks, group-by operations run 5 to 10x faster; sorting up to 11x; joins 3 to 8x depending on key cardinality. More importantly, Polars enables workloads that pandas simply cannot do on a single machine: processing datasets larger than available RAM through its streaming engine without a Spark cluster or any distributed infrastructure. The rest of this post explains why, shows you what the API looks like in practice, and is honest about where Polars falls short.

Why Polars Is Fast

Columnar memory via Apache Arrow

Every DataFrame in Polars is backed by Apache Arrow’s columnar memory format. In practice this means the values for each column are stored in a contiguous block of typed memory with no Python object overhead. A column of 10 million integers is 80 MB of raw int64 memory. The equivalent pandas Series backed by a NumPy array is similar, but pandas object columns — strings, mixed types, nullable numerics before pandas 2.x — become arrays of Python object pointers, each of which is a heap-allocated Python object with 28+ bytes of overhead and a reference count that the GIL must protect.

The columnar layout enables two further wins. First, SIMD (Single Instruction, Multiple Data) CPU instructions can operate on 4, 8, or 16 values in a single clock cycle when they sit in a contiguous typed buffer. Polars’ Rust core exploits AVX2/AVX-512 SIMD automatically through the arrow2 / arrow-rs primitives. Second, when you only need three columns from a 50-column dataset, only those 3 column buffers need to be read from disk or touched in memory. This is projection pushdown made physically real.

Apache Arrow Columnar Layout (3 columns, 5 rows)
------------------------------------------------
Column "user_id" (Int64):
  [101, 204, 307, 104, 501]  <- contiguous 40 bytes

Column "revenue" (Float64):
  [12.5, 98.0, 3.1, 55.0, 7.8]  <- contiguous 40 bytes

Column "region" (Utf8 / LargeUtf8):
  offsets: [0, 2, 4, 6, 8, 10]
  data:    "useueuuseu"  <- packed byte buffer, no Python objects

pandas object column equivalent:
  [<PyObject id=0x7f1a>, <PyObject id=0x7f2b>, ...]  <- pointer array
   each pointer → heap-allocated str with 49-73 bytes overhead

Rust, multi-threading, and the GIL

Polars is implemented in Rust with Python bindings generated via PyO3. The Python layer is thin: you call a method, it invokes Rust, and the GIL is released for the duration of the computation. Polars’ internal thread pool (built on Rayon) fans work across all available cores simultaneously. Group-by hashing, sort, join, and scan all parallelize at the Rust level without any Python thread coordination. pandas, executing NumPy or Cython kernels, also releases the GIL for individual operations, but the chained operation model — df.groupby("x")["y"].sum() — calls back into Python between each step, reacquiring the GIL repeatedly and serializing what could otherwise be pipelined.

Two Execution Models: Eager vs. Lazy

Polars supports two modes of operation that are architecturally distinct, not just a performance toggle.

Eager execution (DataFrame)

The DataFrame API runs immediately. You call a method, it executes, it returns a result. This is pandas’ model and Polars supports it directly for interactive work and small datasets.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

import polars as pl

# Eager: executes immediately, result is a DataFrame
df = pl.read_csv("sales.csv")

result = (
    df
    .filter(pl.col("amount") > 100)
    .group_by("region")
    .agg(
        pl.col("amount").sum().alias("total"),
        pl.col("amount").mean().alias("avg"),
        pl.len().alias("count"),
    )
    .sort("total", descending=True)
)
print(result)

Lazy execution (LazyFrame)

LazyFrame is the point of Polars. Instead of executing each operation immediately, LazyFrame builds a logical query plan — a tree of operations that Polars can analyze and rewrite before executing. You trigger execution explicitly with .collect().

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import polars as pl

# Lazy: scan_parquet reads metadata only, builds query plan
result = (
    pl.scan_parquet("sales/*.parquet")          # no data read yet
    .filter(pl.col("amount") > 100)             # predicate pushed to reader
    .select(["region", "amount", "product_id"]) # projection pushed to reader
    .group_by("region")
    .agg(
        pl.col("amount").sum().alias("total"),
        pl.col("amount").mean().alias("avg"),
        pl.len().alias("count"),
    )
    .sort("total", descending=True)
    .collect()                                   # execute with optimizer
)

The optimizer pass happens between plan construction and .collect(). You can inspect what it does:

1
2
3
4
5
6
7
8

lf = (
    pl.scan_parquet("sales/*.parquet")
    .filter(pl.col("amount") > 100)
    .select(["region", "amount"])
    .group_by("region")
    .agg(pl.col("amount").sum())
)
print(lf.explain(optimized=True))

What the optimizer actually does

User-written logical plan:
  SORT ["total"]
    AGGREGATE [sum(amount), mean(amount), count] BY [region]
      FILTER [amount > 100]
        PROJECTION [region, amount, product_id]
          SCAN sales/*.parquet [all columns]

After optimizer (simplified):
  SORT ["total"]
    AGGREGATE [sum(amount), mean(amount), count] BY [region]
      SCAN sales/*.parquet
        predicate_pushdown: amount > 100      <- applied at reader level
        projection_pushdown: [region, amount] <- only these columns decoded

Predicate pushdown pushes filter conditions as close to the data source as possible. For Parquet files, row groups whose statistics (min/max per column) prove they contain no rows satisfying the predicate are skipped entirely without decompression. For a filter like amount > 100 on a dataset where 70% of row groups have max(amount) < 100, the optimizer skips 70% of the file reads.

Projection pushdown eliminates unused columns at the reader. If your pipeline selects 3 of 40 columns, Polars decodes only those 3 columns from each Parquet row group. Combined with Parquet’s columnar layout, this is a substantial I/O and memory reduction.

Common subexpression elimination detects when the same expression (e.g., pl.col("amount") * 0.9) appears multiple times in a query and computes it once, reusing the result buffer.

Join reordering and cardinality-based heuristics attempt to order joins so smaller tables are on the probe side.

The Expression API

The expression system is Polars’ most important conceptual departure from pandas. In pandas, you manipulate DataFrames through a mix of column assignment, .apply() with Python lambdas, chained indexing, and method calls on Series objects. In Polars, you construct expressions — composable, lazy descriptions of computations — and pass them to a small set of DataFrame verbs.

The core verbs:

• select([exprs]) — choose or compute columns, returning only those columns
• with_columns([exprs]) — add or replace columns, keeping all others
• filter(expr) — keep rows where expression is true
• group_by([exprs]).agg([exprs]) — aggregate
• join(other, on=..., how=...) — join two frames

Expressions are built with pl.col("name") as the entry point, then chained:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

import polars as pl

df = pl.read_parquet("events.parquet")

result = df.with_columns([
    # String operations
    pl.col("email").str.to_lowercase().alias("email_lower"),
    # Conditional with when/then/otherwise
    pl.when(pl.col("status") == "active")
      .then(pl.col("revenue"))
      .otherwise(0.0)
      .alias("active_revenue"),
    # Date extraction
    pl.col("created_at").dt.year().alias("year"),
    # Rolling computation
    pl.col("daily_sales")
      .rolling_mean(window_size=7)
      .over("store_id")
      .alias("rolling_7d_avg"),
])

Window functions with .over()

.over() is the expression-level equivalent of SQL window functions. It computes a group-level aggregation but broadcasts the result back to every row of the original frame — no merge required, no index alignment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# For each row, compute the group-level aggregate and attach it
result = df.with_columns([
    # Rank within each region by revenue, descending
    pl.col("revenue")
      .rank(method="dense", descending=True)
      .over("region")
      .alias("revenue_rank"),
    # Sum of revenue for the row's region (broadcast to all rows)
    pl.col("revenue")
      .sum()
      .over("region")
      .alias("region_total"),
    # Share of region total
    (pl.col("revenue") / pl.col("revenue").sum().over("region"))
      .alias("region_share"),
])

The key distinction: group_by().agg() produces one row per group. .over() produces the same number of rows as the input, with each row annotated with its group’s aggregate. This maps directly to SQL’s PARTITION BY window clause.

Streaming: Larger-Than-RAM Data

The streaming engine is where Polars separates from every single-machine DataFrame library. The premise: if your data does not fit in RAM, you need either a distributed system (Spark, Trino) or a streaming execution model that processes data in batches. Polars provides the latter.

As of Polars 1.31+, a new streaming engine — a complete rewrite informed by the lessons of the original implementation and the Morsel-Driven Parallelism paper — is shipping as the default path for sink_* operations and available via collect(engine="streaming"). The rewrite is generally faster and uses less memory than the in-memory engine for the operations it supports natively.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

import polars as pl

# Process a multi-hundred-GB Parquet directory without loading it all into RAM
(
    pl.scan_parquet("s3://data-lake/events/year=2025/**/*.parquet")
    .filter(pl.col("event_type") == "purchase")
    .group_by(["user_id", "product_category"])
    .agg(
        pl.col("amount").sum().alias("total_spend"),
        pl.len().alias("purchase_count"),
    )
    .sort("total_spend", descending=True)
    .sink_parquet(
        "output/user_category_spend.parquet",
        row_group_size=100_000,
    )
)
# sink_parquet defaults to streaming engine; no .collect() needed

sink_parquet (and sink_csv, sink_ipc) trigger streaming execution automatically. The frame is never materialized in full. For workloads that are primarily aggregation and filtering — which most analytical batch jobs are — this works well.

Honest caveats about streaming maturity: not every Polars operation has a native streaming implementation. When the engine encounters an unsupported operation, it falls back to in-memory processing for that subplan. This means you need to test that your specific pipeline actually stays in streaming mode for its hot path. The new engine introduced in 1.31 is more complete than its predecessor but is still under active development; check the GitHub tracking issue (#20947) if you hit unexpected memory spikes. Some complex join patterns and certain window functions do not yet stream natively.

Real Speedups vs. Overstated Ones

The benchmark sites consistently publish 5–15x numbers. Those numbers are real for the right workloads. They are misleading for others.

Speedups that are real and consistent:

• Group-by aggregations on large frames — Polars’ parallel hash aggregation across all cores is the most consistently large win. Pandas processes groups one at a time on one thread.
• Multi-column filters + Parquet scans — predicate and projection pushdown eliminate work at the reader level in ways pandas cannot.
• Joins — Polars uses parallel hash joins. Pandas .merge() is single-threaded.
• String operations on large columns — the Arrow Utf8 buffer layout enables vectorized string operations that outperform Python object arrays significantly.
• Sort — Polars implements parallel radix sort; benchmarks consistently show 8–11x over pandas.

Speedups that are overstated or reversed:

• Tiny DataFrames — on frames under ~100K rows, Polars’ startup overhead (thread pool initialization, plan compilation) can make it slower than pandas for simple operations. The crossover point depends on operation complexity, but below 10K rows, reach for pandas without guilt.
• Single-row lookups — Polars is not a key-value store. Indexed .loc[] access in pandas has no direct equivalent and is not Polars’ design target.
• IO-bound workloads — if your bottleneck is network bandwidth reading from S3, multi-threaded compute does not help. The CPU is not the constraint.
• apply() with Python lambdas — Polars has map_elements() for row-wise Python function application. It is slow for the same reason pandas’ apply() is slow: it calls a Python function per row. The fix is to rewrite the logic using Polars expressions, which eliminates the Python call overhead entirely.

Interoperability: Arrow, pandas, and DuckDB

pandas bridge

Polars DataFrames convert to and from pandas via the Arrow IPC bridge:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

import polars as pl
import pandas as pd

# Polars -> pandas
pl_df = pl.read_parquet("data.parquet")
pd_df = pl_df.to_pandas()               # copies data by default
pd_df_arrow = pl_df.to_pandas(use_pyarrow_extension_array=True)  # zero-copy for most types

# pandas -> Polars
pl_from_pd = pl.from_pandas(pd_df)      # copies
pl_zero_copy = pl.from_pandas(
    pd_df,
    schema_overrides=None,
)

Zero-copy is achievable when the pandas DataFrame uses PyArrow-backed dtypes (introduced in pandas 2.0 via pd.ArrowDtype). When pandas columns are backed by NumPy arrays, Polars must copy on conversion. In practice, the Arrow round-trip cost is acceptable for interop with scikit-learn, matplotlib, or other pandas-dependent libraries — you pay the copy once, not on every operation.

DuckDB integration

Polars and DuckDB are natural companions, not competitors. DuckDB operates on Arrow memory directly via zero-copy replacement scans — it can query a Polars DataFrame without copying data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

import polars as pl
import duckdb

pl_df = pl.read_parquet("transactions.parquet")

# DuckDB queries the Polars DataFrame over Arrow with zero copy
result = duckdb.sql("""
    SELECT
        region,
        SUM(amount) AS total,
        COUNT(*) AS txn_count
    FROM pl_df
    WHERE amount > 50
    GROUP BY region
    ORDER BY total DESC
""").pl()  # .pl() returns a Polars DataFrame

# Alternatively, go the other direction: Polars reads DuckDB output
conn = duckdb.connect("analytics.duckdb")
pl_result = pl.from_arrow(
    conn.execute("SELECT * FROM monthly_summary").arrow()
)

The typical pattern in a data pipeline is DuckDB for SQL-heavy data prep and joins across heterogeneous sources, Polars for subsequent in-memory transformations or streaming aggregation, and Arrow as the zero-copy handoff format between them. When you are reading from an Apache Iceberg lakehouse and scanning Parquet directly, Polars’ scan_parquet and Polars Cloud’s native Iceberg support make the two systems first-class partners.

GPU engine

As of 2025, Polars ships an optional NVIDIA GPU engine powered by RAPIDS cuDF. Install polars[gpu] and pass engine="gpu" to .collect():

1
2
3
4
5
6
7

result = (
    pl.scan_parquet("large_dataset/*.parquet")
    .filter(pl.col("score") > 0.5)
    .group_by("category")
    .agg(pl.col("score").mean())
    .collect(engine="gpu")   # falls back to CPU if op unsupported
)

On compute-bound queries against 100M+ row datasets, the GPU engine delivers 2–5x additional speedup over the CPU engine. This is still in open beta as of mid-2026 and limited to single-GPU execution, but it requires zero code changes to existing Polars logic.

Migrating from pandas

The conceptual shifts

Three ideas in pandas have no equivalent in Polars because Polars deliberately removed them:

No index. pandas DataFrames have an index — a row label that enables .loc[] alignment. Polars has no index. Rows are just positions. This removes an entire class of subtle bugs (implicit index alignment, SettingWithCopyWarning, index drift after merge) and simplifies the memory model. The adjustment is mostly unlearning; you use filter() and join on explicit columns instead.

No in-place mutation. pandas encourages df["new_col"] = ... assignment. Polars DataFrames are immutable. Every operation returns a new frame (or modifies in place only via the specialized set_sorted hint). Use with_columns() to add or replace columns.

Expressions, not chained indexing. pandas idioms like df[df["x"] > 5]["y"].mean() chain Python attribute access and item access. Polars expresses the same thing as df.filter(pl.col("x") > 5).select(pl.col("y").mean()) — a description of the computation that the optimizer can inspect and rewrite.

Common operation mapping

A side-by-side example

pandas:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

import pandas as pd

df = pd.read_csv("orders.csv")
df["order_date"] = pd.to_datetime(df["order_date"])
df["year"] = df["order_date"].dt.year
df = df[df["status"] == "complete"]
result = (
    df.groupby(["year", "category"])["amount"]
    .agg(["sum", "mean", "count"])
    .reset_index()
    .rename(columns={"sum": "total", "mean": "avg", "count": "n"})
    .sort_values("total", ascending=False)
)

Polars:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

import polars as pl

result = (
    pl.scan_csv("orders.csv")
    .with_columns(
        pl.col("order_date").str.to_date().alias("order_date"),
    )
    .with_columns(
        pl.col("order_date").dt.year().alias("year"),
    )
    .filter(pl.col("status") == "complete")
    .group_by(["year", "category"])
    .agg(
        pl.col("amount").sum().alias("total"),
        pl.col("amount").mean().alias("avg"),
        pl.len().alias("n"),
    )
    .sort("total", descending=True)
    .collect()
)

Both produce the same result. The Polars version uses the lazy API, so the CSV is scanned in parallel, the filter is applied while reading, and only the three needed columns are tracked through the aggregation.

What does not translate cleanly

Polars’ expression system is powerful but not a superset of pandas’ API surface. Some things that pandas users rely on require genuine rethinking:

MultiIndex DataFrames — Polars has no hierarchical index. Pivot-style multi-level column frames must be kept as long format and reshaped explicitly.

In-place assign chaining — pandas’ .assign() chaining works because pandas allows arbitrary callables referencing the intermediate frame. Polars’ with_columns() takes expressions but not callables that reference the current frame mid-chain; complex dependent column additions may require multiple with_columns() calls.

crosstab / complex pivot — Polars’ pivot() API is functional but less featureful than pandas’ pivot_table. Complex multi-value pivots sometimes require custom aggregation + unnest patterns.

Ecosystem integrations — geopandas, statsmodels, seaborn, and the long tail of libraries that accept a pd.DataFrame do not accept pl.DataFrame. You will call .to_pandas() at the boundary, which means the ecosystem constraint is a conversion cost, not a hard wall, but it is real. For building internal data platforms, the team readiness and library compatibility question is often more important than the raw performance argument.

Honest Downsides

The ecosystem is smaller. Not marginally — substantially. The PyData ecosystem (scikit-learn, statsmodels, seaborn, geopandas, plotly express’s DataFrame interface, and thousands of domain libraries) assumes pandas. Polars works around this by converting at the boundary, but that conversion is a real cost and a real source of friction.

The API has changed significantly across versions. Users who adopted Polars before 1.0 encountered breaking changes across minor releases — method renames, argument restructuring, groupby semantics changes. The 1.0 release in mid-2024 committed to a semantic versioning policy with documented breaking changes in major versions only, which improves the stability picture, but the pre-1.0 experience left a legitimate trust gap. Parts of the API marked unstable in the docs can still change without a major version bump.

The streaming engine has rough edges. Transparent fallback to in-memory processing is pragmatic engineering, but it means “I’m using streaming” is not a guarantee. A pipeline that processes 10GB fine in testing can silently fall back to in-memory on a 200GB dataset because one operation in the plan lacks a streaming implementation. You need to test explicitly and inspect plans.

map_elements() for row-wise Python function application is an escape hatch, not a feature. It breaks out of the Rust engine into Python, row by row, with the full GIL overhead. If you have a large apply() in a pandas pipeline and port it naively to map_elements(), your Polars code will be similarly slow. The right answer is to rewrite the logic as expressions. That is sometimes hard.

Verdict

Polars is the right choice for analytical data pipelines where performance and memory efficiency matter on a single machine. Group-by aggregations, large Parquet scans, multi-table joins, and streaming processing of datasets larger than RAM are its native habitat. The lazy API with its query optimizer is not a nice-to-have — it is the primary interface, and building habits around scan_* + .collect() rather than read_* is the most important shift.

The case for pandas is narrower than it was two years ago but still real: the ecosystem, the ML toolchain integration, and the sheer installed base of pandas-fluent developers all matter operationally. The practical answer for most teams is not a wholesale migration but a boundary: use Polars for data pipelines and batch processing, convert to pandas at the scikit-learn or plotting boundary, and let Arrow be the zero-copy bridge between them.

For greenfield analytical work — new pipelines, new data platforms, anything where you choose the tooling — the default should be Polars with DuckDB as its SQL companion. The combination covers the full space of single-machine analytical compute with a clean, well-reasoned API and performance that was impossible in pure Python two years ago.

Sources

• Polars User Guide — Streaming
• Polars User Guide — Lazy API
• Polars User Guide — Window Functions
• Polars User Guide — GPU Support
• Tracking Issue: New Streaming Engine — pola-rs/polars #20947
• Polars Blog — Streaming Sort-Merge Joins
• Polars Blog — GPU Acceleration with NVIDIA RAPIDS
• NVIDIA Technical Blog — Polars GPU Engine Powered by RAPIDS cuDF Open Beta
• Polars Blog — Polars in Aggregate: Polars Cloud, Streaming Engine, and New Data Types (Dec 2025)
• Polars Blog — The Power of Predicate Pushdown
• Polars vs Pandas in 2026: Performance Benchmarks — danilchenko.dev
• Pandas vs. Polars vs. DuckDB vs. PySpark Benchmarking — Pipeline2Insights
• Polars vs Pandas — Databricks
• 10 PyArrow Zero-Copy Recipes Across Pandas, Polars, DuckDB — Medium
• Polars Versioning Policy
• RAPIDS cuDF Polars Documentation