Watch the video, transcript and sources
You think a join hands your rows back in the order you wrote them. Polars 2.0 stopped promising that, by default, on purpose. Polars is an open-source DataFrame library: a table engine you call from Python, with its core written in Rust. It is free under the MIT license, and you install it with pip install polars. One detail first. The switch that brings the old order back is an argument to the join itself. I'll come back to it at the end.
Since 2.0, collect runs the streaming engine by default. Streaming splits the query into chunks that run in parallel, and the chunks finish when they finish. For join, group by and unpivot, the release does not promise an order. Who gets hit first? Whoever compares output to a saved file, row by row. That test can pass on your laptop and fail on a different machine. Next, the migration guide. It says the exact row order shown above is not guaranteed. So if you rely on the order, sort explicitly.
Out-of-core is on by default too. It starts spilling to disk at about eighty percent of RAM, with a sixty-four gigabyte budget. Sort, window functions and many expressions can spill now. Joins and group-bys are coming. Polars says it beats DataFusion and DuckDB on the TPC-H and TPC-DS benchmarks. The numbers come from their own runs, on derived data, and their footnote says they are not comparable to official results. Their own numbers say Polars gets about three point eight times faster from sixteen cores to one hundred ninety-two, on TPC-H. They also say the big machine's extra threads slow down small queries. We ran the same join twice on Polars 2.0.0. The default run shuffles the rows. With the flag, the input order survives.
Find the joins that feed a test. Any join without a sort after it is a row order decision you never made. Polars 2.0 is stricter about types. The schema check catches a missing column before any data is read. It cannot catch a cast that fails on one row, because that error waits for the data. Now the loop I opened. The switch lives on the join. Set maintain order on the join, or add a sort after it. The guide suggests the sort. The join flag kept the order in our run, with the left side's order kept as left, right.
Verdict, under the hood: needs review. I'd review every join before upgrading. Got a question about this? Put it in the comments.
Verdict: NEEDS REVIEW — I'd review every join before upgrading.
Sources
https://news.ycombinator.com/item?id=49977177
https://lobste.rs/s/ppzbqo/release_polars_2_0
https://pola.rs/posts/release-polars-2/
https://docs.pola.rs/releases/upgrade/2/
https://github.com/pola-rs/polars-2.0-benchmark
https://pola.rs/
And that's the diff for today. I'm Niko from Axrisi. Merge responsibly.
YouTube · Newsletter · thedailydiff.dev · forward this to the intern who deployed on Friday.