How can I find what does an specific neuron do in neural network?

Question

How can I know what each neuron does in NN? Consider the Playground from Tensorflow, there are some hidden layers with some neurons in each. Each of them shows a line(horizontal or vertical or ...). Where these shapes come from. I think they are understandable for nn not a person!

Answer 1

In TensorFlow Playground, the horizontal line show where each class is separated for each neuron. What happens when you take any intermediate neuron to make the decision? You can see the answer by the line provided by that neuron. And this decision is a result of the weighted sum from the decisions of the previous neurons (up to activation).

Take the middle-top neuron in the link you share, which is an almost horizontal line - slightly tilted to the right. This neuron classifies everything above it as a blue, and everything below it as an orange. Hover over the neuron to see a larger picture on the output.

You can also see how this is actually calculated by hovering over the line coming from the neurons in the previous layer to the neuron you are looking at. For the case of the same neuron (center-top), the weight coming from the first input ($x_1$) is 0.091, while from the second one ($x_2$), it is 0.49. The neuron ends up being almost horizontal because the contribution from the horizontal input ($x_2$) is so much larger compared to the vertical one ($x_1$).

Of course you need to take into account the nonlinearity coming from the activation function but the idea presented above is the essence of it. The example uses tanh activation, which behaves very linear in its intermediate region so one can ignore this issue to some extend for this particular case.

Edit: It appears that the values for the weights change at every browser session, so the neuron I describe might look a little different to you. To get the same configuration, simply click on the colored lines between neurons to edit them and use the values above for the connections.

Answer 2

I think serali answered this question well, though I wanted to give some extra reading for those interested.

There are many ways of deciphering what a neuron in a NN is doing. This lecture does a fantastic job at covering some of these methods and is an incredibly interesting watch. This covers more advanced methods of visualising what a model is doing.