I made my own Neural Network from scratch in unity with C# and I am using it as DQN. I set up my network which has 4 layers: 9 input values, 20 nodes in the second layer, 15 nodes in the third layer, and 8 outputs. I set all the layers' activation functions as Identity (which has no impact on the value), and the loss function as MSE I scaled my inputs and rewards, so my input values are between 0-1, and targets are between -1 and 1.
But after the first training, the weights become NaN. On another project I have a much simpler network which is 5-4-3-2 nodes in layers. And there it works perfectly. But when I increase the nodes for layers it immediately makes the weights NaN. If I add a coefficient factor to Identity something like 0.1 it doesn't give NaN but this time the 8 output values become very similar to each other.
I debugged the delta values and after 6 or 7 epochs later the weights go to very large numbers like 2345221. Than after that they become NaN, I think due to large values.
With other activation function(TanH, sigmoid) they don't give that NaN values but the outputs are become very close to 1(I understand that because the network has very nodes and the outputs are larger than 3 or 4 and this functions converge to 1 with larger values than 2). But this NaN thing happens with the ReLU functions too.
My question is: Is it normal to have this numbers or something is wrong with my code? If it is normal what can I do because the other functions also give bad results, I am trying to predict the cumulative reward here.
