Source
r/MLQuestions•
scrdest

Training Autoencoders on subsets of features

I'm building a deep autoencoder-based model in Keras that analyzes biological data, and I've ran into a problem. I'm hoping someone here could provide me with some guidance.

The number of feature vectors in datasets I have to work with can be absolutely huge, and if I try to train my model - either on CPU or GPU - memory allocation fails. Now, I suppose I could brute-force it by getting better hardware, but I came up with another approach, and I'm wondering if it is feasible.

If I were to simply sample the features with replacement between training batches and train the model on only the subset of features, would the resultant NN weights still 'make sense', or will they become a completely scrambled mess? This approach seems to resemble the various ensemble learning techniques, especially bagging, but I couldn't find much about using those for training AEs.

Posted Feb 27, 2018, 1:09 PM UTC

38
TotesMessenger•

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

 ^(If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads.) ^(Info ^/ ^Contact)

1
jmmcd•

Are you talking about the "number of feature vectors" or the number of features? ie, a subset of the vectors, or a subset of each vector? Phrasing unclear.

1
scrdestOP•

Ah, sorry. Allow me to clarify: the feature vectors are 1D, just numeric value + label pairs, not much to trim off here.

I would like to only consider a random sample of features per epoch.

1
jmmcd•

I still can't understand.

When you say 1D, do you mean 1D or just one value? "numeric value + label pairs" suggests the latter.

When you say a random sample of features, that doesn't make sense, since there's only one feature.

1
kikko686•

Do you mean you have N samples and M features, where you want to perform training on a batch of samples each with a subset of the M features? This sounds very interesting and I would love to know what you results you find. Please post here when you figure it out. Also, how large of a feature space are we talking?

1
ddofer•

The solution you're talking about will be tricky to debug and test. Here's a pair of easier alternatives:

  1. Try a number of bagging autoencoder models. i.e train multiple, seperate autoencoders and each autoencoder model will only get a subset of the total features. (Suitability depends what your overall goal is with this).

  2. Try some feature selection in advance on your data. e.g. use decision stumps/adaboost (depth 2-3 decision forest ensembles with bagging for diversity, then keep the features that were used more than K times; or use a simple linear model and use that for feature importance).

More solutions possible, but require more background on what you want to do. (Some are purely from the Bioinformatics POV, I have friends from the M.Linial lab working on something relevant). Let me guess, GWAS or SNPs?

1
ddofer•

There are also methods for regularizing / feature selection layers (including from some Israeli and intel peeps as I recall). That likely won't help you here, if the issue is pure batch size. Is your input data floats or booleans? Consider using lower numeric precision, e.g. float 16/ 8 / 32 , that'll give you an easy upgrade.

1
talksaboutthings•

I think this is essentially putting a dropout layer immediately after your input. I haven't read much on dropout for autoencoders, but I would suggest you take a look in that direction. My intuition would be that as long as you rescale input properly as in regular dropout (i.e. double the values of your inputs if you drop half of them), you should be able to train such a network on sparsified input. I don't know how large a dropout parameter you can get away with, though.

I am a little confused at why you are having memory overflow, however, training a deep neural net. Unless I've tried to load the entire dataset into memory or train absolutely monstrous architectures, memory constraints have typically not been a huge concern in my experience. In vision applications, I've simply dropped my batch size by a factor of 1/4 or so to run on low-memory hardware and had no major issues. I guess if your input has absolutely massive dimensionality (10e5+), then you might encounter difficulties throwing it into a deep neural net.

If I were in your shoes, I would also consider encoding different pieces of massive input separately (i.e. autoencode the first 10k inputs, the next 10k inputs, the next 10k etc. and then concatenate the encodings together. You could even try autoencoding those autoencodings if you want).

1