Source
r/computervision•
brates09

Incorporating additional inputs into fully convolutional networks?

Could anyone point me to a good method for incorporating non-image inputs into a fully-convolutional network?

For regular CNNs the traditional way would be to encode the input into some feature vector and concatenate this with the dense section of the CNN, but for FCNs there is no dense section. The obvious extension to that would be to duplicate the representation of the additional inputs to be 'image-sized' and concatenate this constant valued image in the feature channels but maybe there is a better approach that I am not aware of?

Posted Jun 15, 2017, 9:43 AM UTC

411
SphericalSpiral•

Use the NN as a feature extractor and extract a feature vector for the image data. Then concatenate your non-image feature vector to the result, and input the result into a classifier (e.g. SVM).

4
IntelligentVaporeon•

Help us a little bit with the problem statement; why would you use a CNN for non-image inputs? The very nature of CNNs/FCNs is based on the idea of analysing images.

1
brates09OP•

Right, but I mean to augment your image with some auxilliary information. For example, a categorical vector. An example in my specific application is that I am working with 3D microscopy slides and I would like to augment my input images with a 'depth' value.

If I was using a regular CNN e.g. VGG, I could perform late fusion with this data and concatenate at the first FC layer, but that doesn't make so much sense with a fully convolutional network.

1
IntelligentVaporeon•

Okay, I don't fully understand what you're trying to do because I've never worked with such data before (I thought working with RGB-D data means you just have a 3D input), but I will try to work with you.

You want to concatenate before the FC layer. The data in such deep layers is basically a feature representation learned by the net, so for me it doesn't make sense to perform the concatenation there, because your augmented info is considered raw data. I'm not sure if you get my point, I'm not very articulate.

But if you know what you're doing, we can skip that part. I'd say the FC layer equivalent in FCNs is the convolution layer with kernel size 1 - just before the upsampling, so does it make sense to perform the concatenation there?

1
EpicSolo•

CNNs are also used for text and other data where space ties to semantics.

1
tdgros•

yes, when words are converted to fixed size rows by word embeddings, so sentences look like images :)

1
tdgros•

FCN are convolutional all the way because they're typically made to classify pixels. They are implicitely classifiers ran in a sliding window fashion. So they do have the same densely connected layers at the end, but flattened into 1x1 convolutions so the (implicit) sliding window calculations are made efficient no matter the size of the image. You can find this embedding at the same place it'd be if the underlying classification net hadn't been converted to a FCN.

If your additional data was another modality, then it'd make sense to incorporate it near the FC layers. Imho it makes more sense to avoid conv layers, because it doesn't make a lot of sense to convolve pixels and non-image data, but you can totally imagine a MLP classify multimodal data (even though nothing's intuitive about that)

But you're talking about using depth, is this a full depth image? if so, then some authors have proposed nice versions of incorporating this kind of data directly into a CNN (of the top of my head, there's a sort of decomposition of a depth image into disparity+normal angle + I don't remember what, the idea is that there is too little depth image datasets for a large network to learn stuff on, so they "expand" it into more images, not sure if it's that good). So if the external data is image-like, then try and feed it through a normal net, if not, aim for the embeddings, you can find them near the top (at least it was pretty clear in the seminal FCN for semantic segmentation paper)

1
brates09OP•

So the representation of depth is sort of implicit. I work with microscopy image stacks, and the image quality degrades as you descend into the sample. I would like to include the 'slice depth' somehow as I think that potentially different low level representations should be learnt dependent on the depth of the slice. This leads me to think that maybe the depth information needs to be inserted from the start, potentially as an additional channel that takes a constant value for the entire image, but I am not sure how much sense that makes.

An alternative approach would be to make this a multi-task learning problem and ask the network to predict the slice depth as well, which may also force it to confront this issue.

1
tdgros•

wouldn't it be possible to input the whole stack to the network? the learned filters would adapt to each stack's proprieties, for instance, you'd find broader edge filters if the worse stacks were blurrier for instance...

Inserting a constant value is rather useless as it would only have the effect of a constant bias. It could be smarter to fine-tune one net per stack.

1
brates09OP•

Yes, currently I use a CNN + ConvLSTM hybrid architecture (https://arxiv.org/abs/1705.09597 ;) ). So to some degree, this can be learnt in the weights of the ConvLSTM, which ties together the slices. But this uses a shared CNN network, I am thinking about ways in which this CNN network could be trained to be explicitly 'conditioned' on the slice depth.

1
tdgros•

If the complexity is the issue, then you can probably just fine-tune the upper layers, with one net per slice. Or alternatively: you fine-tune the upperlayers for all the slices, using the slice depth as the additional data. It still seems quite valid to me.

1