Source
r/Python•
Liquid_Fire

Python safe serialization?

Are there any libraries for serialization and safe deserialization of Python objects?

By "safe" I mean that the objects will come from an untrusted source (over the network). I know Python comes with pickle and marshal, but neither of those provides any safety for deserializing untrusted objects.

I have a set of classes that I would like to serialize/deserialize, and they all inherit from a common base class. Ideally the library would be able to deserialize an instance to its original subclass (and refuse to deserialize objects that are not instances of the base class). All of the member variables of the classes contain either instances of Python built-in types, or instances of another serializable class from the same set (but there are no reference cycles or other funny business). Finally, it would be nice if the library serializes to JSON but it is not a hard requirement.

Are there any libraries that can do the above securely, and if so, what would you recommend?

Posted Jun 27, 2011, 5:23 PM UTC

813
Lexarius•

Use YAML (a superset of JSON) as your serialization format with PyYAML.

Features of YAML include object type tagging, support for references (it handles reference cycles just fine), and when using PyYAML's "safe" dumper/loader, the ability to define which of your classes should be considered safe.

5
Liquid_FireOP•

Thanks, that looks like exactly what I need! I'll be sure to check it out.

1
[deleted]•

PyYAML, since it is so flexible and human readable, is easy to debug. Coding for custom classes is quite straightforward.

1
voidspace•

Not builtin to the standard library, but it shouldn't be too hard to roll your own on top of json.

If the instances are of a known set of types all you need is the type name and the json serializion of the instance members.

So something like: json.dumps([type(obj).__name__, obj.__dict__])

Deserializing: name, _dict = json.loads(the_json); instance = object.__new__(getattr(module, name)); instance.__dict__.update(_dict)

4
Tommah•

One problem with this approach is that set and frozenset are not serializable to JSON.

1
voidspace•

That may or may not be an issue for the OP, and is anyway solveable with the json module. It seems the OP likes YAML as a solution anyway.

1
jabwork•

http://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2011-hidden-treasures-in-the-standard-library-4901130

There's a fantastic demonstration of the hmac module shortly in (nearish the 7 minute mark)

http://www.doughellmann.com/PyMOTW/hmac/index.html

By the same guy (Doug Hellmann) non-video explanation. Recommend perusing PyMOTW if you haven't already

3
tarekziadeRetired Packaging Dude•

what about crypting the result of a pickle dumps, and decrypting at load time ? You can use a lib like PyCrypto and let pickle handle all the hard work.

Also, I would not bother and just use SSL if it's an option

2
kingkilrPyPy/Cryptography/Computers•

You can do this entirely with the stdlib I think, using the hmac module.

3
Liquid_FireOP•

The source of the serialized objects is untrusted (a client connecting to my server). Encryption does not help. I need something that will ensure that deserialization produces a valid object (of the original type, from a restricted subset of types), and will not execute any untrusted code from the serialized data.

Of course I could easily write something like this using e.g. the json module, but I thought it might exist already as a library.

3
tarekziadeRetired Packaging Dude•

sorry I misunderstood the untrusted source part. I get it now

1
nirs•

Encryption does not give you any safety. What you need is a way to authenticate a serialized object string before you de-serialize it - a MAC. The standard library includes a good one - HMAC.

2
bryancole•

You can achieve what you want with the cPickle module. You need to use the Unpickler() factory-function and give it a "find_globals" function (passed as a kw-arg, probably). This function is called with a module name and class name and returns a class object. Use this to control which classes the Unpickler can load. I've not tested this myself recently; see the pickle docs.

PyYAML also gets my recommendation, but it's much slower than pickle. If you want speed, use pickle; if you want readability, use YAML; if you want JSON, stdlib.json ought to be fine

1