[D] How to deal with adding new data and new labels to existing models

Discussion

Say you work at a company that identifies humans. You start off with a dataset with pictures of humans and bounding boxes. The company becomes a huge success and you want to develop your product further. Now you also want to identify eyes and ears, so you make a dataset for that. You have really struck gold, the market is going wild for your product, so you decide to add a new category, indicators of whether or not the human is ill.

My question is, how do you deal with this kind of growth in ML products?

For every category you add, you have to add annotations to the dataset. This can be a tremendous amount of work and might not be feasible to backfill the data you already have with the new labels. I have two suggestions for how to deal with this, you only annotate new data and then train the model in two phases. First phase you train the model on only detecting humans, next phase you add outputs for eyes and ears and fine tune on the rest of the data.

An other way you could do it is to train two separate models, one for humans and one for eyes/ears. Depending on your domain, you might want to have everything in one model if you have real time constraints, so multiple models might not be favorable.

Is there anywhere I can read more about how to deal with these kinds of issues? Do you guys have any experience in dealing with issues like this?

Posted Jun 27, 2019, 12:26 PM UTC

44
diffgram-anthony•

Transfer Learning. http://cs231n.github.io/transfer-learning/

Basically means you start training from where the previous model left off.

It's fairly standard to start training from large datasets like imagenet or mscoco. Here's some examples

https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md

​

If this is NOT what you mean, then please explain! :)

3
mlawayOP•

Transfer learning is definitely one answer to this issue. I was thinking more about how people deal with the technical debt? As you keep adding new data and new classes, your model training procedure is going to end up becoming extremely complicated and cumbersome, making future developments harder.

1
diffgram-anthony•

There are challenges for sure! There are commercial systems that help with it as one option

(Self promotion) https://diffgram.com/ Also https://www.wandb.com/ is doing some stuff in this area

1
roemil•

Sounds a lot like transfer learning. You could look up that.

1