[D] Why have the standard data formats in object detection remained as COCO/PASCAL VOC/YOLO as opposed to switching over to a nested columnar format?

Discussion

In image classification, early standards generally had users divide image files into different training and validation folders, as well as (sometimes) requiring separate folders for each class label. While researchers seem to have held on to these standards, the APIs of deep learning libraries eventually evolved to allow more flexible usage which didn't necessarily require users to shuffle around image files in folders should they want to adjust their training and validation setups.

Keeping training and validation files in different folders probably has certain benefits for the reproducibility of research, namely in making it abundantly clear how the train/val split was performed. However, this convention -- when enforced -- does introduce something of a "barrier" for beginners just looking to train a model on their existing labeled data with minimal data wrangling.

Thus, existing library APIs for image classification seem to have converged to loading references to filepaths, either as a list or in a tabular/columnar format, while encouraging users to perform their own train/val splits on these references to the data. This allows for maximum flexibility, as users can have their data organized however they wish, and easily perform whatever augmentations they wish to perform on the splits.

Most approachable tutorials for adapting a model to custom datasets now follow the standard of having users define their own dataloaders. This new standard, however, does not seem to have proliferated to user facing object detection and semantic/instance segmentation libraries.

Here, some of you might interject and point out that

  1. The data is hierarchical in nature. Several bounding boxes, polygon coordinates or RLEs may exist for each image. COCO/PASCAL VOC/YOLO are the natural way to store such data.
  2. Storing hierarchical data in a (nested) columnar format will introduce memory overhead.
  3. Data formats such as .csv, .tsv don't work well with nested data.

None of these, in my opinion, are convincing arguments for why so many object detection libraries need to break with the conventions that have evolved for image classification. Hierarchical data can easily be stored as lists nested in cells of a dataframe. We don't necessarily have to repeat a row for however many objects exist in a given image. The arrow (https://arrow.apache.org/install/) library exists for storing and loading nested columnar data.

Am I insane in thinking that object detection library APIs will eventually -- but inevitably -- converge to the standard of image classification APIs? Why haven't they already? Are the established standards too entrenched? Torchvision already seems to be heading there: https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html. I can't say the same for most other libraries, where the standard remains "Please organize your data after these very specific instructions and execute this script with 20 poorly documented (optional) args".

Posted Apr 15, 2021, 8:12 AM UTC · edited

21
SolidAsparagus•

I think you explained exactly why the object detection format has developed into the way it has. You can't easily use CSV or TSV files to store hierarchical data. The next simplest file format is JSON and that is what we use. Everyone knows how JSON works, it's human readable, and it is easy to explore in a notebook.

Hierarchical data can easily be stored as lists nested in cells of a dataframe.

Hierarchical data cannot be easily stored in a columnar format. It can be stored in columnar format if you really want, but it's far from easy. What file format do you store nested columnar data with? Arrow is a powerful technology for in-memory data storage. How do you store that data on disk? Are you expecting data scientists to work with Parquet or another type of binary blob where the schema needs to be known to read the data? Binary blobs suck and I see no reason to believe that they will be adopted in an industry where so much of the work is data exploration by data scientists whose software engineering skills vary a lot.

If you look at major PyTorch CV toolboxes like MMDetection or Detectron2, they already support multiple ways of using your data with their toolboxes (MMDection, Detectron2). You can convert your data to COCO/PASCAL VOC or you can implement your own custom dataset, similar to how that torchvision example works. Those libraries need to have their own internal representation anyway so they don't really care how your data is formatted on disk.

So to summarize - I don't see a reason to believe that object detection data will be stored in anything other than JSON on disk, probably in the COCO format as often as not since it's well-defined and good enough. But the leading PyTorch user-facing libraries do not require that format today. Tensorflow might be a different story since (A) tf.datasets as an input might combine loading from disk with training-specific transformation and (B) some good CV people moved to PyTorch so the PT toolboxes are newer/better designed to handle the "how do I use this with my data" problem.

5
[deleted]•

[deleted]

1