Source
r/cs231n•
fjdkf

What's a good accuracy on the last part of assignment 2?

IIRC in lecture, Andrej said his inefficient for-loop implementation got 80% accuracy overnight, so I'm wondering what accuracy I should be aiming for.

Currently, I'm only matching the 80%(7conv+1fc).

Validation set accuracy: 0.811
Test set accuracy: 0.799

Loss graph
Accuracy graph

I was also thinking of building an ensemble, but I'd like to know what accuracy to gun for on the base models.

Posted Jul 3, 2016, 2:03 PM UTC

16
kingtaurus•

If you are curious you can check out the scores that various models get for CIFAR-10. A test time accuracy of 80% is very good start for a model (although it depends on the number of filters, size of the filters and various other hyper-parameters - for example changing how much the learning rate decays can provide a significant increase in test accuracy after a few more epochs.).

Couple of tips:

  • make sure you do Xavier initialization (check that you use the correct parameters depending upon your activation function);
  • make sure you are using ReLU (but try Leaky ReLU as well) as your activation function - or possibly ELU;
  • implement batch normalization prior to fully connected layers (its often useful in other layers);
  • make sure you've implemented dropout;
  • test out a different regularization and dropout parameters;
  • building an ensemble might net you an additional 1-2% (do this at the end).
2
fjdkfOP•

Thanks for the feedback. Fixed a major bug with my Xavier init, and accuracy shot up 4%, which was nice.

It was mentioned in the lecture that dropout may not be as critical if batch norm is used. So, I was hesitant to do it. Good idea though, I should test it instead of making assumptions.

I tried doing a strategy of dropping the learning rate if the accuracy was plateauing. I think my implementation is too simplistic though, so I should probably work on that.

I only train on my cpu, but after looking at the links, it looks like I should be able to hit 90% before doing anything with ensembles. Should be interesting.

1
kingtaurus•

In addition you can also do some data augmentation - which can often help improve things. This can include:

  • Randomly crop a [height, width] section of the image. ** You will need to train a new model (if you change the spatial size.**
  • Randomly flip the image horizontally.
  • Randomly adjust the contrast a small amount.
  • Randomly tint the image a small amount.
  • Randomly adjust the brightness of the image.
  • Introduce a small amount of per-pixel noise.
  • Whitening the images.

Finally if you do decide to train an ensemble consider doing transfer learning can improve how quickly you can build new models (but be careful, because it can cause features to be identical between models and thus trying to improve inference by using an ensemble will not generate a better model).

If you look at the tensorflow model, it achieves 86% test time accuracy (after 256 epochs) with a shallow model.

EDIT: reworked a sentence.

2
fjdkfOP•

Thanks, I think I need to pursue some of those data augmentation techniques, because I'm starting to get super high accuracy on the training data, and l2 regularization doesn't seem to bring up the validation accuracy that much when I crank it up.
i.e. (Epoch 17 / 20) train acc: 0.985000; val_acc: 0.845000

The data augmentation could help address some of the overfitting I've been getting.

1
Abdul_Muqeet•

What batch size are you using ? I am wonder to see that you have trained for 100 epochs in about 2000 iterations.

1
fjdkfOP•

I was using a batch size of 512. I cut the size of an epoch by 80% though, so in reality it was only 20 'real' epochs. I didn't realize an epoch has a real definition at the time.

1