1
$\begingroup$

I want to train a convolutional neural network on multiple input images. My image is 240x360 and is in RGB. Therefore my input image has a shape of 3x240x360. Now I want to use multiple images of the same size. Do I stack the images and increase the input channel size of my CNN or do I have to create another CNN and then concatenate the tensors later on? I assume both could work but would there be a difference between the two implementations? You can imagine that my problem is like with a self driving Tesla. A Tesla has 8 video streams that all have the same dimensions. The images will probably share the same features e.g. a car or a pedestrian, but in which image it occurs in will make a difference in meaning. What would be the best architecture for this?

$\endgroup$
1
  • $\begingroup$ What is your input layer size? If your input layer is 360 x 240 and if you give larger than that, you object in the image would be too small to learn for neural network. Have you consider utilizing GPU more to speed up training? $\endgroup$
    – Cloud Cho
    Commented Oct 30, 2023 at 20:37

1 Answer 1

2
$\begingroup$

You can:

  1. Just concatenate the channels of each input image, and then have the usual CNN , or
  2. Define one feature extractor network (aka backbone), and apply each backbone on each input image such that to share weights among the inputs, then you have the "head" part of the network with the various layers (e.g., classification head, detection head, etc.)
  3. Alternatively, if the content of each image is quite different you can train $N$ backbones (so no more weight sharing, and so saving parameters, as in approach 2), followed by the head part.

Also, in both approach 2 and 3 you need a way to combine the last feature maps of the backbone(s). To do so you can just: sum, multiply, or concatenate them.

To recap, the architecture in approach 2 or 3 would be: images -> [backbone(img) for img in images] -> combine (e.g., concat) -> output-head.

Indeed, approach 1 is the simplest but also likely to be less performant, whereas the approach 3 is expected to have best accuracy at the cost of increased training cost and slower inference speed. Therefore, approach 2 is the compromise between the two.

$\endgroup$
2
  • $\begingroup$ So solution 2 is like creating an auto encoder trained on all images and then concatenating the latent vectors as input to the head? $\endgroup$
    – Erik Storm
    Commented Oct 28, 2023 at 13:09
  • $\begingroup$ @ErikStorm Well, kind of. I prefer to call it a "feature extractor" which yields a feature vector (similar to a large latent space) or low-dimensional feature maps. Yes, then you can concatenate the vectors for the head part of the network. $\endgroup$ Commented Oct 29, 2023 at 14:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .