Which neuron represents which part of the input?

Question

In a neural network, each neuron represents some part of the input. For example, in the case of a MNIST digit, consider the stem of the number 9. Each neuron in the NN represents some part of this digit.

1. What determines which neuron will represent which part of the digit?

2. Is it possible that if we pass in the same input multiple times, each neuron can represent different parts of the digit?

3. How is this related to the back-propagation algorithm and chain rule? Is it the case that, before training the neural network, each neuron doesn't really represent anything of the input, and, as training proceeds, neurons start to represent some part of the input?

Answer 1

The determination of what each neuron represents is dictated by the initial weights of the network. As you may know, the common practice is to initialize the weights randomly. This means that given the same input data, the network may switch what each neuron means. It may also function differently, taking longer or shorter times to train. However, if you set the random seed to some constant, you can make the network deterministic.

Answer 2

It is the combination of the output of all neurons that determines the output of the neural network. In the case of convolutional neural networks (CNNs), the term "feature" is used because it is associated with the feature maps (or activation maps) and filters (or kernels) of the CNN. However, this terminology might not be accurate (because these might not be features in the intuitive sense) and it is used more to interpret the inner workings of the CNN.

After training, the weights of the neural network are fixed (unless you perform online and continual training), so the output of each neuron will be the same given the same input, thus the output of the neural network will also be the same (unless there is some random operation being performed). During training, the weights and thus the output of each neuron (and of the neural network) often change.

The contribution of each neuron to the output of the neural network is determined by the weights of the connections between the neurons, which change during training and can be initialised in different ways, which might affect differently the final weights (after training).

There several ways of visualising the contribution of each neuron to the output of the neural network. See also this article Visualize Features of a Convolutional Neural Network and the paper Visualizing and Understanding Convolutional Networks (2013).