question for (mega|atto)parsec experts!
I'm trying to build a custom JSON parser that creates objects in a new data format (the format's irrelevant to my question, but it's called "Concrete", a thrift-based spec for NLP research). The idea is the input will be lazy Text representing a JSON array, where each object in the array represents a document of some sort. Now, I had a working version using Megaparsec that adopted some existing code to parse the JSON, transform it to a list of Concrete Communication objects, and serialize them. Problem was, it was too memory-hungry, and I had trouble refactoring it, and I concluded that something I was doing forced it to keep the entire Text object in memory. So, I stripped it down to just parsing the JSON, and made versions for both Megaparsec and Attoparsec. These are working, but have the same memory usage issue: give it a 50M file and it uses ~1.5G peak memory, give it a 100M file and it uses ~3G peak memory. I was hoping someone could take a quick look at the grammars, and maybe clue me into why this is happening:
https://github.com/TomLippincott/concrete-haskell (src/Data/Concrete/Parsers/JSON/(Attoparsec.hs|Megaparsec.hs)
I understand the notion of backtracking and why it can be needed, but I don't think the JSON spec has this sort of ambiguity at the global level. I've even switched to explicitly parsing "a json array of json objects", and get the same memory footprint. My hope, since each object in the top-level array is to be independently transformed into a Concrete Communication, is to get an approximate upper bound on memory of holding one such object. When parsing the top-level array, it shouldn't need to know anything about the array entries its already processed, just the current one, or that it's expecting a comma/next object. Or, I could be completely wrong about this! Any help, or general advice, is much appreciated.
EDIT: figured out a major part of it. Representing the whole JSON array is what keeps the individual array entries from being freed as parsing proceeds, so by creating a special top-level rule that's identical to that for a JSON array except it doesn't actually return a JArray the memory is vastly reduced.
Posted Jun 3, 2017, 5:03 PM UTC · edited