How do neural networks weigh multiple inputs/features of different dimensionality?

Question

I am confused about how neural networks weigh different features or inputs.

Consider this example. I have 3 features/inputs: an image, a dollar amount, and a rating. However, since one feature is an image, I need to represent it with very high dimensionality, for example with $128 \times 128 = 16384$ pixel values. (I am just using 'image' as an example, my question holds for any feature that needs high dimensional representation: word counts, one-hot encodings, etc.)

Will the $16384$ 'features' representing the image completely overwhelm the other 2 features that are the dollar amount and rating? Ideally, I would think the network would consider each of the three true features relatively equally. Would this issue naturally resolve itself in the training process? Would training become much more difficult of a task?

Answer 1

As stated in your example, the three features are: an image, a price, a rating. Now, you want to build a model that uses all of these features and the simplest way to do is to feed them directly into the neural network, but it's inefficient and fundamentally flawed, due to the following reasons:

• In the first dense layer, the neural network will try to combine raw pixel values linearly with price and rating, which will produce features that are meaningless for inference.

• It could perform well just by optimizing the cost function, but the model performance will be nowhere as good as it could perform with a good architecture.

So, the neural network doesn't care if the data is a raw pixel value, price, or rating: it would just optimize it to produce the desired output. That is why it is necessary to design a suitable architecture for the given problem.

Possible architecture for your given example :

1. Separate your raw features, i.e. pixel value, and high-level data, i.e. price and rating

2. Stack 2-3 dense layers for raw features (to find complex patterns in images)

3. Stack 1-2 dense layers for high-level features

4. Combine them together in a final dense layer

If you want to de-emphasize the importance of the image, just connect the first dense layer 16,384 to another layer having fewer connections, say 1024, and have more connections from the high-level data, say 2048.

So, again, here's the possible architecture

1. Raw features -> dense layer (16384) -> dense layer (1024)
2. High-level features -> dense layer (2048)
3. Combine 1 and 2 with another dense layer

Answer 2

The answer by ssh is correct. Your results could be further improved i) by extracting image features by a convolutional (instead of fully connected) architecture, and ii) by exploiting transfer learning.

To exploit transfer learning you i) pick some widely used model, eg. ResNet-18, ii) initialize it with ImageNet pretrained parameters, iii) replace its fully connected layer (the one that produces 1000-D softmax input) with your own randomly initialized fully connected layer. If you are interested, have a look at detailed instructions.