I'm testing my own implementation of a neural network on recognising the type of a function. I generate sine, linear and quadratic functions with random parameters, compute their values for a linspace of size 100 and pass the y-values as input into the network, expecting a vector size 3 as the output.
I've already checked my gradient with the one returned by the numdiff library and it's spot on. The input is normalized.
The structure of the neural network is 100 nodes in the input layer, 10 nodes in the hidden layer and 3 nodes in the output layer. These are the results I got for 200 epochs with batch size 20, learning rate of 0.001 and 400 training samples:
The cost (E_mean) is decreasing but the accuracy on training data isn't. What happened too was that the accuracy has sky-rocketed but then dropped immediately.
I'd be grateful for any help!