r/rust•
hwchen

Dataframes: what do you need?

Every now and then somebody asks if there’s a data frame library or pandas equivalent available. There’s been a few attempts that I know about, but they are a long way from matching the feature set of pandas.

A dataframe library in rust is something I keep thinking about. For my uses case, etl, I would love to have a library to make some basic tasks easy: melting, pivoting, splitting string columns, joins, concat, apply/map, etc. I’ve made some attempts at a basic api, but I think I need more type artistry to create an ergonomic interface.

There’s also a world of scientific computing in pandas that I don’t know much about.

It’s been suggested to just use pandas, for exploratory work, and then write something custom in Rust when needed. I’ve done this; but but basically I love programming in Rust and would love a dataframe library :)

So, if you have thoughts about what would be important for a Rust dataframe (api design, features, etc), how you might use a Rust dataframe, please leave them here. I might be ready to take another crack at writing it.

Some prior art and reading:

Posted Feb 12, 2019, 2:19 AM UTC · edited

4320
weihanglo•

Weld is a interesting Pandas-like library written in Rust. It also got a Python binding.

8
hwchenOP•

It looks like Weld is a runtime, which integrates with dataframe libraries like pandas. I'm looking to build a frontend for dataframe; perhaps it would be able to use Weld, but that's a consideration for the future.

6
Programmurr•

Join a project. Don't try this on your own. Wes, the author of pandas, is very receptive to Rust for data science. Join his circle, if possible.

8
Aubudon•

I love that youre looking into this. Don't forget to check out spark, dask, and vaex as well. They are other good sets of apis with parallelism.

5
hwchenOP•

I haven't heard of vaex before! Thanks for the tip.

1
repoptrac•
  1. ​

I would recommend taking a look at R tidyverse for ideas in addition to Python pandas.

For instance, R has

​

* `dplyr`: to select/add/modify colume and/or row

* `tidyr`: for melting and pivoting with `gather` and `spread`

* `purrr`: for functional features such as `map` and `walk`

​

Python pandas library was developed based on `data.frame()` from R and `plyr` package by Hadley Wickham.

Now R has updated version of `plyr` in `dplyr` and `tidyr` and much more in `tidyverse`.

In my biased opinion, R tidyverse has better design and easier to extend than python pandas.

​

One thing that will be hard to translate to Rust directly is non-standard evaluation in R.

R uses non-standard evaluation heavily to make interactive analysis easier.

For instance, in R

​

library(tidyverse)

mtcars %>%

	select(cyl, disp)

​

will select two variables `cyl` and `disp` in data.frame `mtcars`.

In python, it will be something like this

​

import pandas as pd

# assuming mtcars is a data.frame

mtcars[:, ['cyl', 'disp']]

​

​

As jusskfulwood mentioned, it would be great if Rust compiler can check that the variables exist in the data.frame but not quite sure how.

​

4
short_sells_poo•

Compile time checking will always be a bit of a tricky area, because for exploratory analysis, I really don't want to bother with manual type annotations. So there absolutely has to be a possibility to leave type checking to run time and support effectively dynamically typed frames. Once you have such dynamically typed dataframes, their interaction with typed dataframes will also be (by definition) checked at run time.

Having said that, one can go pretty far with heavy usage of macros and tuples, but I'm not 100% certain that the current type system is powerful enough to go beyond the existing DataFrame implementations and isn't an unmaintainable monstrosity. I agree with other posters that rust will be much better placed for a truly outstanding DataFrame implementation once const generics and GATs are implemented.

4
genneth•

As someone who transitioned from R to Python, I still miss the syntactic sugars that made R just that little bit less friction-y to get things done. Sure it sometimes made you mad because things were unpredictably failing due to overlapping syntax, but that's the correct trade-off for that space. Unfortunately, it's not obvious that's the cultural values that Rust really has.

2
athrowaway3z•

I would love to have a macro for Einstein notation.

http://ajcr.net/Basic-guide-to-einsum/

3
mamcx•

I have look at several ndarrays crates for use for my relational backend/lang. I'm in the camp of data manipulation, not science per se. So, I see this alike kdb+... I wanna to cover much more than ndarray but wish to have it as my array/vector.

​

Some stuff:

- Be "just" a data frame library. Put all the other stuff in another crate (not need to be another repo). I see too much effort in try to implement scientific stuff intermixed with the data frame. This make code confusing.

- Being overly focused with scientific calculation make some restricted to operate on numbers.

- Nice to be able to have row-major and column-major

- Put focus in how manipulate the data. Filter, select (columns, rows), mapping, ie: the functional interface. In my design, I don't create specific sum, avg, etc implementation, instead I make "binary ops, unary ops" by ref or cloning and just pass functions for the rest. This allow to be more than "make math". For example, I wanna do equally:

array |> sum
array |> String.Upper

Other way to look at it: Be a good foundation like vectors and b-trees.

-If enum based, then put also decimal, dates, some way to emulate structs, and a way to put raw binary data so is kind of possible to build a custom object (ie: Like Json but a bit more)

- Super-nice if allow to pack heterogenous data using ANY or other way that allow to put any data there. Not yet see how can this be possible.

- Loading from data sources as CVS not need to be hardcode. Instead, give a trait or similar so is possible to plug any we want, alike serde.

- I think a nice, ergonomic interface can be done first before to be super-concerned with performance. After several attempts trying to build a relational engine on rust I have become more understanding in how hard is to build a heterogenous, 2d structure. Is hard. I'm under the impression that a generic way to solve all is kind of impossible to do with rust now. Need to do specialization, code generation or similar

- Work together. I think a small group of people could do much more than just one. The problem, like always, are the competing ideas and each personal agenda. That is why I think to get focused in just "be a ndarray" is key. Anyone, could extend from there.

3
fuasthma•

I like the idea of having a trait for the data loading part. Assuming, it's not something too crazy it shouldn't be too hard for those of us working on these types of crates to implement. For example, I'm currently working on the next version of my data_reader crate that should allow for heterogeneous data files to be read in, and I wouldn't have a problem adding such a trait if it looks like it'll be a standard one among dataframe type libraries.

1
[deleted]•

[deleted]

1
NukaColaBear•

This is crazy I was just thinking about possibly implementing dataframes and rust.

I'll make sure to check out those links.

Thanks.

1
hwchenOP•

Cool! Let me know if you have any further thoughts.

1
nevi-me•

I'm still prototyping something https://github.com/nevi-me/rust-dataframe, I could do with help and ideas.

1
[deleted]•

The thing I want is type-safety. I want my joins and maps to fails at compile-time if they are invalid. This is hard to do - at the moment Rust's type-system makes this possible but not very ergonomic.

I have a WIP library for this but it's on hold until a few more advanced features land in Rust (const-generics, GATs).

1
mamcx•

I'm building a relational engine/lang. I need dynamic behavior, but I wonder how much possible is to encode joins in type safe way?

​

I think is not possible. I ask here:

​

http://lambda-the-ultimate.org/node/5353

​

The problem is that JOINs create (or remove!) new columns depending in several factors. For my lang I think I need to create a typing syntax like:

fun queryOrders(...) = Customers LEFTJ Orders

but do the same on rust? I think is possible to encode "I will do a CROSS JOIN" but not "and this fields will be produced"

2
[deleted]•

It is possible - here is an example. df.join(df2, leftcol, rightcol) will compile iff the columns exist in their respective frames and have the same type, else it will fail, with a pretty confusing compiler error.

As you can see, the definition is complicated. Because essentially it requires emulating dependent-typing using traits.

1
nevi-me•

I'm working on a dataframe library https://github.com/nevi-me/rust-dataframe

I'm planning on supporting as much of Spark's functions as possible, and have made good progress so far.

The difficult thing for me so far, is how I'd use this library for exploratory analysis. I have some use-cases where I can do ETL on files whose structure doesn't change (e.g. read JSON log files, transform and aggregate, write to other format), but I haven't a clue on how I'd create bindings for other languages.

​

1
vadixidav•

There was an old attempt at making something like arrayfire (which does have a Rust crate) in pure Rust called collenchyma. You could use that to develop a high performance backend. It would need to be updated and maintained again though.

1
hwchenOP•

In case you’re not in the other thread on forming a dataframe workgroup, I’ve set up a repo for initial discussion.

https://github.com/rust-dataframe/discussion

1