0
$\begingroup$

I'm testing my own implementation of a neural network on recognising the type of a function. I generate sine, linear and quadratic functions with random parameters, compute their values for a linspace of size 100 and pass the y-values as input into the network, expecting a vector size 3 as the output.

I've already checked my gradient with the one returned by the numdiff library and it's spot on. The input is normalized.

The structure of the neural network is 100 nodes in the input layer, 10 nodes in the hidden layer and 3 nodes in the output layer. These are the results I got for 200 epochs with batch size 20, learning rate of 0.001 and 400 training samples:

enter image description here

The cost (E_mean) is decreasing but the accuracy on training data isn't. What happened too was that the accuracy has sky-rocketed but then dropped immediately.

I'd be grateful for any help!

Improve this question
$\endgroup$
0

3 Answers 3

Reset to default
0
$\begingroup$

One cause I can think of is that your network is too simple for the task. That would explain why the cost stops decreasing at around 0.33. Your network might be outputting the same prediction independent of the input, by using the bias values and setting the weights to low values. That would also be the reason why the accuracy is 0.33, because one third of the time the network is lucky and its guess matches the input.

Another problem could be your that you are using a bad model architecture, for example using the wrong activation function. I suggest using the relu or the sigmoid function.

I hope you are able to solve your problem.

Improve this answer
$\endgroup$
0
$\begingroup$

If the accuracy is sky-rocketing unexpectedly during training (and then dropping immediately), my first hunch would be to experiment with the learning rate - it seems that the learning rate may be too high. Please remember that the learning rate is just a parameter that magnifies your step in the direction of the gradient calculated at point (t). It means that in (the case of first-order optimization) there is nothing that prohibits it from overshooting the local or global minimum.

The second thing that I would like to inspect is the architecture of the network. Given that you have 100 inputs and then only 10 neurons, it is very likely that the architecture is just too simple to approximate the target function. However, this seems strange, as you are trying to approximate the sine, linear and quadratic functions. This should be feasible by using a rather simple architecture. I omit here a detailed explanation of the approximation of common functions. If you are interested, jump straight into an approximation of common polynomials and the minimum number of parameters required to do so.

Finally, I wonder about your samples. You say that you generate some common functions and then sample points that serve as training data. Please keep in mind, that your sample should be reflective of the data-generating function. If it not (e.g. you sample only around some small interval) - you will not approximate the function correctly, even if your acc is 100% ;)

Improve this answer
$\endgroup$
0
$\begingroup$

Your network is always guessing the same output. I would recommend using cross-entropy loss rather than mean-squared error.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .