Source
r/rust•
socratesTwo

cffi vs cpython vs pyo3, what should I use?

Hi all, I've gotten to the stage in a project where I need to connect Python to my Rust. After some Googling, PyO3, cpython, and cffi seem to be the most mature ways of doing it, but I lack the experience in this area to make a sensible decision about which to use.

PyO3 seems pretty cool, but it requires the nightly toolchain and I'm not so sure I want to introduce a dependency on that to the company. Dev-ops is already trying to reel me in a bit and I'd rather not push it. cpython seems a bit less fleshed out, but at least (AFAICT) it'll build on stable... and I suspect both of these are using cffi under the hood, so one question is are they a good value or would I be better off skipping the middle layer?

Are there other gotchas I should be aware of? The answer to "what should I do" questions is typically "it depends.", and if that's the answer, what does it depend on?

Posted Apr 8, 2020, 8:33 PM UTC · edited

1214
Alpharemercurial•

Spoiler: I use cpython a lot (I'm a Mercurial core dev that does most of the Rust work) and one of my colleagues is a maintainer.

I like cpython: it's well built, does not use any more unsafe that is needed, is getting frequent updates and might get a nicer interface (read: proc macros) once my colleague is a little bit less busy. It's used in production by Facebook for Mononoke : building on rust-stable is a must for any production-grade code IMO. Specialization is still a ways off from being stable, and - here comes more anecdotal evidence - I haven't heard as many people vouch for the soundness of PyO3 as cpython's.

9
simukis•

I found a number of soundness problems in pyo3, and I think at least some of those might be relevant to cpython, but I didn’t look deep enough into cpython to tell.

A quick look tells that at least one of them is not a problem in cpython though – cpython is properly catches panics that are invoked in rust code (instead of propagating them into python).

8
Alpharemercurial•

I would be very interested to know if you find soundness issues in cpython! Good to know

2
ra_kete•

Since you have experience with rust-cpython, maybe you know this: Is there a way to create a Python subclass in Rust? Specifically, I want to wrap a Rust enum into a Python enum with rust-cpython, but I cannot find a way to subclass Python's Enum class. Now I suspect that might just not be possible, but maybe I'm missing something?

1
Alpharemercurial•

I'm almost sure this is not possible at this time. Your best best is to use composition, i.e. encapsulate your Rust logic in a Python class. You could go post an issue and get an answer from the maintainers.

1
ra_kete•

Alright, thanks!

1
OkRaise6•

cpython has never done me wrong, and is what PyOxidizer is built on top of.

3
HeroicKatoraimage · oxide-auth•

Program interaction with Python is defined in terms of its c interface so usage of cffi is mandatory. In my opinion, the stable option cpython is pretty good and fulfilled all requirements I had.

3
quasilinux•

I don't think this is completely accurate. cpython and PyO3 work by binding to libpython, and any Rust binaries or Python libraries created with them must also link to libpython. cffi (the python module) isn't necessarily part of the equation.

2
Alpharemercurial•

That is correct, both rust-cpython and PyO3 are bindings the the Cpython ABI and do not use cffi.

1
HeroicKatoraimage · oxide-auth•

Oh, cffi is the name of a particular python module? I wasn't aware of that.. I thought it just referred to generic C foreign function interface.

1
quasilinux•

Just to provide a dissenting opinion, since most of the consensus here seems to be for cpython.

I used cffi in a simple Python-calling-into-Rust scenario at my work. Using cffi has a couple advantages depending on your needs: the Rust libraries created for cffi can be used with any other language that can call a C FFI, and the Rust libraries also don't link directly against libpython. As a result, you get a very portable Rust library that just needs to be built for the specific platform (Linux, Windows, OSX). Libraries created with cpython or PyO3 must use the same exact Python version and even Linux distro, since each may build Python differently.

Steps involved are roughly the following:

  • Create the Rust library to solve your problem.

  • Export an external C FFI from your Rust code (The FFI Omnibus was really valuable here).

  • Write a Python module with cffi to call your external Rust library.

  • Package this as a python wheel (I recommend looking into milksnake for this).

This also tends to push your Rust code towards being the simplest to solve the problem at hand, since you're limited by the complexity of calling between Python and Rust. However, our result was a clean interface that was easy to audit and test for correctness, another big benefit.

Happy to help if you have any questions about getting started with this path!

3
Zethra•

Imo not working on stable is disqualify for serious use. Experimenting with stuff on nightly is great, but I wouldn't feel comfortable putting that code into production.

1
urschrei•

It really depends on the complexity of the interaction between your Python code and your Rust code. YMMV (massively), but I try not to overcomplicate my Python with heavy use of classes, and I try to exchange the simplest possible data structures (arrays, or arrays-of-arrays) between the two codebases. That allows me to write simple ctypes or cython wrappers around my Rust API. That approach will work equally well for CFFI (I never really got to grips with it because it was a little too magical, but that's very much a "me" problem – I know it works well).

1