0
$\begingroup$

I have a fully connected neural network with the following number of neurons in each layer [4, 20, 20, 20, ..., 1]. I am using TensorFlow and the 4 real-valued inputs correspond to a particular point in space and time, i.e. (x, y, z, t), and the 1 real-valued output corresponds to the temperature at that point. The loss function is just the mean square error between my predicted temperature and the actual temperature at that point in (x, y, z, t). I have a set of training data points with the following structure for their inputs:

(x,y,z,t):

(0.11,0.12,1.00,0.41)

(0.34,0.43,1.00,0.92)

(0.01,0.25,1.00,0.65)

...

(0.71,0.32,1.00,0.49)

(0.31,0.22,1.00,0.01)

(0.21,0.13,1.00,0.71)

Namely, what you will notice is that the training data all have the same redundant value in z, but x, y, and t are generally not redundant. Yet what I find is my neural network cannot train on this data due to the redundancy. In particular, every time I start training the neural network, it appears to fail and the loss function becomes nan. But, if I change the structure of the neural network such that the number of neurons in each layer is [3, 20, 20, 20, ..., 1], i.e. now data points only correspond to an input of (x, y, t), everything works perfectly and training is all right. But is there any way to overcome this problem? (Note: it occurs whether any of the variables are identical, e.g. either x, y, or t could be redundant and cause this error.)

My question: is there any way to still train the neural network while keeping the redundant z as an input? It just so happens the particular training data set I am considering at the moment has all z redundant, but in general, I will have data coming from different z in the future. Therefore, a way to ensure the neural network can robustly handle inputs at the present moment is sought.

| improve this question | |
$\endgroup$
  • $\begingroup$ Hi. Try to reduce the number of layers. Btw, how many layers and parameters do you have in your neural network, and which activation functions are you using? Maybe you're facing an "exploding" or "vanishing" gradient problem". $\endgroup$ – nbro Apr 25 at 4:07
  • $\begingroup$ i would suspect that there is some error in the configuration of your nn, normally if an input is "redundant", it would learn a very low weight $\endgroup$ – k.c. sayz 'k.c sayz' Apr 25 at 4:23
  • $\begingroup$ @nbro I have attempted to vary the size and number of layers, but it appears to not improve the situation as it still outputs nan. For example, I have tried the following combinations of [# layers, #neurons per layer]: [10,50], [8,20], [4,20], [1,10]. I am currently using tanh activation functions with Xavier initialization. As you mention, I believe it is related to the gradients being unable to update weights for z. For example, if I add a few training data points with z != 1.0, everything works again, which indicates this is the origin. But uncertain of a good general fix. $\endgroup$ – Mathews24 Apr 25 at 14:58
  • $\begingroup$ @Mathews24 Hyperbolic tangent (tanh) is a so-called "saturating" activation function. Try to use ReLU, which is non-saturating. Also, if the problem persists, try to use even fewer layers. $\endgroup$ – nbro Apr 26 at 1:01
  • $\begingroup$ @nbro I just attempted using ReLU and even fewer layers (e.g. ranging from 1 to 8 layers with 20 neurons each) but it still does not resolve the problem. $\endgroup$ – Mathews24 Apr 26 at 23:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.