Dataframes: what do you need?
Every now and then somebody asks if there’s a data frame library or pandas equivalent available. There’s been a few attempts that I know about, but they are a long way from matching the feature set of pandas.
A dataframe library in rust is something I keep thinking about. For my uses case, etl, I would love to have a library to make some basic tasks easy: melting, pivoting, splitting string columns, joins, concat, apply/map, etc. I’ve made some attempts at a basic api, but I think I need more type artistry to create an ergonomic interface.
There’s also a world of scientific computing in pandas that I don’t know much about.
It’s been suggested to just use pandas, for exploratory work, and then write something custom in Rust when needed. I’ve done this; but but basically I love programming in Rust and would love a dataframe library :)
So, if you have thoughts about what would be important for a Rust dataframe (api design, features, etc), how you might use a Rust dataframe, please leave them here. I might be ready to take another crack at writing it.
Some prior art and reading:
- https://github.com/hwchen/llamas2
- https://github.com/apache/arrow/tree/master/rust probable backend for any production-grade dataframe
- https://github.com/andygrove/datafusion spark-like
- https://github.com/kernelmachine/utah dataframe
- https://github.com/sinhrks/brassfibre dataframe
- https://paulkernfeld.com/2019/01/13/frunk-column.html
- My reddit post exploring dataframe api: link
- Searching for
dataframein this subreddit also brings up some discussion.
Posted Feb 12, 2019, 2:19 AM UTC · edited
Weld is a interesting Pandas-like library written in Rust. It also got a Python binding.
It looks like Weld is a runtime, which integrates with dataframe libraries like pandas. I'm looking to build a frontend for dataframe; perhaps it would be able to use Weld, but that's a consideration for the future.