par2 file size limitations

Question

It appears that this subreddit isn't really developer centric, so if there's a more appropriate place to post this, please let me know.

I've been reading up on the par2 algorithm regarding how it reconstructs missing parts of files and found out that it has a limitation where the maximum number of blocks it can handle is 32,768. Given that a lot of usenet providers allow a maximum article size of roughly 1 MB, that means that if you want to keep the block size 1 MB or less, then the maximum total size of the archive is about 32 GB.

These days, there are archives that exceed 32 GB in size and, as a result, the block size ends up exceeding the maximum size of an article. That means that even if a few articles making up a par2 file are missing, the entire par2 file ends up being unusable since there are no intact blocks.

If the block size is 1 MB or less, then, even with missing articles, it was possible to still get most of the blocks out of a damaged par2 file.

Does anyone know if uploaders have started generating par2 files on individual parts of an archive. That is, have a set of par2 files for the first part, and another set for the second part, and so on rather than generating them on the entire set of files? Also, has anyone heard of any updates to the par2 specification to allow for more than 32767 blocks? I know the paper describing the implementation shows that it is possible to have it handle up to 2147483648 blocks, but, based on my understanding, the implementation would be a bit slower than the one used by the par2 archivers today.

Posted Feb 17, 2017, 2:59 AM UTC

2319
Loading comments…