Disclaimer: I'm not a student in computer science and most of my knowledge about ML/NN comes from YouTube, so please bear with me!
Let's say we have a classification neural network, that takes some input data $w, x, y, z$, and has some number of output neurons. I like to think about a classifier that decides how expensive a house would be, so its output neurons are bins of the approximate price of the house.
Determining house prices is something humans have done for a while, so let's say we know a priori that data $x, y, z$ are important to the price of the house (square footage, number of bedrooms, number of bathrooms, for example), and datum $w$ has no strong effect on the price of the house (color of the front door, for example). As an experimentalist, I might determine this by finding sets of houses with the same $x, y, z$ and varying $w$, and show that the house prices do not differ significantly.
Now, let's say our neural network has been trained for a little while on some random houses. Later on in the data set. it will encounter sets of houses whose $x, y, z$ and price are all the same, but whose $w$ are different. I would naively expect that at the end of the training session, the weights from $w$ to the first layer of neurons would go to zero, effectively decoupling the input datum $w$ from the output neuron. I have two questions:
- Is it certain, or even likely, that $w$ will become decoupled from the layer of output neurons?
- Where, mathematically, would this happen? What in the backpropagation step would govern this effect happening, and how quickly would it happen?
For a classical neural network, the network has no "memory," so it might be very difficult for the network to realize that $w$ is a worthless input parameter.
Any information is much appreciated, and if there are any papers that might give me insight into this topic, I'd be happy to read them.