The author discusses the DataFrame scale gap, highlighting the differences between SQL, databases, and DataFrame implementations. They express surprise at the lack of query optimization and poor scaling in DataFrame solutions like pandas, compared to PySpark’s closer alignment with databases. The goal of Polars is to combine the best of both worlds in a flexible DataFrame API. They introduce Polars Cloud and a novel Streaming Engine design, aiming for scalable data processing with distributed capabilities. Polars offers various scaling strategies, including distributed, partitioned, and fault-tolerant queries. The author invites early access to Polars Cloud for interested users.
https://pola.rs/posts/polars-cloud-what-we-are-building/