Pitfalls and bumps in Clojure’s Extensible Data Notation (EDN)
nitor.com/fi/uutiset-ja-blogi/pitfalls-and-bu...Posted Sep 25, 2019, 5:04 PM UTC
4619
nitor.com
Very important topics. I hope the community can settle on fixes.
10
Here is another one: I have some large edn files that I was trying to parse, there is apparently some error in it, or else it is too large, or there is some other problem.
I have no idea what it is, because parsing it results in a return value of a space character turned into a symbol.
I *love* edn as a data format but there are a lot of rough edges using the current tooling.
5
I have no idea what it is, because parsing it results in a return value of a space character turned into a symbol.
Sounds like some zero-width character tomfoolery. Wonder if there's an event-based EDN parser you can use to see the results as they're parsed piecemeal. Might be a useful error case to document too.
6
That was my thought as well, I just encountered this two days ago so I'm going to manually chop that file up into smaller pieces and try to narrow down what is causing it. A quick search for non-standard characters didn't turn anything up, but that was a pretty simple editor regex search so I could easily have missed something.
I couldn't find an event based edn parser or similar, the edn ecosystem seems to be shockingly sparse.
1
Looks like you could directly use clojure.lang.EdnReader, which appears to have the underlying methods for implementing a lazy/partial read (although like most clojure core code, it's undocumented).
3
Ah, nice find! Thanks.
1
1A
This doesn't seem an issue to me. Forward compatibility is way to hard to maintain. I actually think the Spec wasn't modified because they're even trying to maintain forward compatibility. So I guess ideally, your EDN shouldn't have namespaced maps, even though it seemed the Clojure edn reader can read them. Personaly, I think it be fine to update the Spec to support is as well, since they allow to save space.
1B
I think Cider fixed this, and no longer relies on that. Not 100% sure, but I remember seeing something to that regard in the latest release notes.
1C
Again, same as 1A form my perspective. Also, the article says it is not backwards compatible, but this is incorrect. Old EDN can still be read by the new EDN reader without issue. New EDN cannot be read by the old EDN reader, and this is called forward compatibility, and I've rarely seen anything that goes to the extent of being forward compatible.
1D
This is just true of Java serialization as a whole. You need to special handle all of them and that's where EDN shines, because it allows for extensions, as opposed to JSON which is lossy in most these scenarios.
1E
Ya, that one is valid. Good pitfall. I 100% agree EDN should get a dedicated writer.
1F
Well, I think the EDN spec actually doesn't allow them. But you can extend it using its extensions features to support them. I did that in one of our app. Again, I think it's a valid caveat and a good reason to have a dedicated EDN writer, which could throw on these scenarios.
I think this was a great post, but I think it's misleading to say these are issues with EDN, that being the normal format. Most issues are related to the current Clojure EDN printer, which is that there isn't one. I'd love to see one personally. Would be nice to have a very optimized writer and reader for it, or at least a dedicated writer which doesn't require you to use a highly configured pr-str.
3
Yes. What I really wish for right now is that there was some basic method in Clojure(Script) that guaranteed to print a valid EDN string.
For one of my applications, serializing and deserializing Clojure(Script) data across a wire is core to it. I currently use EDN for this, by calling pr-str on the value and then sending it along. The biggest pain is that many values don't print even valid tagged literals.
For instance, if you (pr-str some-var) you'll get out the string #'some-var, which can't be read by an EDN parser. Same thing for some popular libraries like Reagent, which print RAtoms like #<RAtom "foo">, which also can't be read.
What I'd like is there to be some function I could call, similar to pr-str, that would guarantee to output a valid EDN string - even if it was just #object [clojure.core.Var ...] or whatever, as long as it's a tagged literal I can handle it.
Perhaps I should switch to transit, but I think then I would have to declare every type I want to be able to roundtrip, which isn't tenable for this. EDN's tagged literals were perfect because a lot of custom types would simply print as #object or some other tag that I could simply pass through both sides.
1
I think you're basically asking to disallow reference types in Clojure. If there were only value types, all values are trivially serializable. Other than that, I can't think of any way to do what you want. Except maybe spec your maps and only serialize keys you understand. Javascript->JSON shares the same problem. Read-str on an atom doesn't make sense – you can't reconstruct the registered watcher callbacks for example (and even printing atom state will fire side effects). IMO the problem with ratoms everywhere in reagent is a huge problem with reagent/react (literally not functional programming) and has nothing to do with clojure.
Perhaps a helpful idea would be to inspect specs when printing to automate the process of selecting keys to print, though it is not going to help get inside ref types.
1
I’m apparently not very good at explaining this, because the issue I opened on Reagent also devolved into the same discussion.
For my specific use case I mean literally, I need it to print valid EDN. That’s it. It doesn’t matter if I can actually do anything with the value. The problem is that:
#<RAtom “foo”>
Cannot be read in by the EDN parser because #<...> is not valid EDN syntax. If it was:
#reagent/RAtom [“foo”]
It would work fine for my use case, which is essentially just a pretty printer that lives over a wire.
3
I see I am confused but help me understand – if you don't care if the representation is useful, what's wrong with "#'some-var" ?
edit: Oh, you just want it to pass through a reader without crashing by default so you can just ignore it
1
Core reference types, like atom print as #object [clojure.core.Atom ...], which is fine, because it’s valid EDN.
1
Transit solves all this stuff. https://github.com/cognitect/transit-format
There are additional issues in EDN not mentioned here that transit solves, for example send (java.lang.Long. 65332980922449989) to javascript – Javascript lacks the numeric primitive to represent that, and it is definitely not EDN's fault.
Another huge issue is performance. Transit is wicked fast (e.g. in javascript it delegates to JSON.parse which is implemented in C++). EDN is a hog, we're talking multiple-seconds to serialize a megabyte of data for a http response that transit can handle in milliseconds.
Did Rich actually say that EDN is a serialization format? No mention of that here: https://github.com/edn-format/edn (2014). Maybe I am splitting hairs, certainly it could be more clearly explained. Transit is awesome. Use transit. There are bindings for Python, Ruby, Javascript, ClojureScript, Java and a clear path to extending it to support your new platform.
I've personally never used EDN for serialization. I've only used it for handwritten config files.
NOTE: Transit is intended primarily as a wire protocol for transferring data between applications. If storing Transit data durably, readers and writers are expected to use the same version of Transit and you are responsible for migrating/transforming/re-storing that data when and if the transit format changes.
Keep that in mind. For persistent data, EDN, or Nippy https://github.com/ptaoussanis/nippy , or Fressian https://github.com/clojure/data.fressian might all be more apt, as they maintain backwards compatibility.
Nothing on the OPs blog post is an issue with EDN that I could tell, but more with pr-str. Though the asymmetry of the spec and the edn reader is interesting. Seems there's room for a EDN writer library that takes care of those concerns. I wish the blog posts described how to address the issues, I believe they are all addressable, in that the printer can be customized to solve all these problems.
Probably also good to note that the Transit spec and format have been stable since first release, so the backwards compatibility concerns have so far been minimal. :)
In retrospective sure, but the repo does state that warning, I'd like to know my persistent data will remain readable for the forseable future, and so I've always avoided it. Maybe now I could trust it more, but honestly, that warning bothers me immensely. If it made a commitment to backwards compatibility I'd feel okay about it, but it makes the opposite statement instead.
I guess not that EDN, Nippy or Fressian ever explicitly stated a commitment to be backwards compatible, but at least they didn't explicitly state against it either.
I think "data transfer format" is pretty close to "serialization format".
You are right! wikipedia says:
It is unfortunate that Rich's talk introducing Transit was lost.
More at http://blog.cognitect.com/blog/2014/7/22/transit