1
$\begingroup$

Background:
This is for a simulated robot with four legs, walking on a flat terrain. The ANN (an MLP) is given inputs as the robot's body angle, positions and angle of each leg with respect to the body and two points of contact with the terrain, on each leg (if there's no contact, the value is zero). The outputs are the four motor rates for each leg.
I'm using Keras with the CNTK backend to train the network. The network has 30 input nodes (ReLU), one hidden layer with 64 nodes (ReLU) and an output layer with 4 nodes (Sigmoid). Optimizer is Adam.

The training data has 2459 datapoints. Running model.validate with parameters testDataPercentage = 0.25, epochs = 50 and batchSize = 10 gave me: loss: 2.9509 - accuracy: 0.3283 - val_loss: 2.8592 - val_accuracy: 0.3213.

But running model.evaluate multiple times gave me: 

['loss', 'accuracy'] [3.10, 0.50]
['loss', 'accuracy'] [3.04, 0.23]
['loss', 'accuracy'] [3.01, 0.11]
['loss', 'accuracy'] [3.45, 0.02]
['loss', 'accuracy'] [3.17, 0.40]
['loss', 'accuracy'] [3.03, 0.27]
['loss', 'accuracy'] [3.012, 0.46]

Loss doesn't decrease much over 50 epochs. It reduces from maybe 3 to 2.8. That's it.

Question:
I don't understand why the accuracy varies so much for each run.
If I add a hidden layer or even add a dropout of 0.2, the results are similar: loss: 2.9253 - accuracy: 0.2978 - val_loss: 2.9350 - val_accuracy: 0.3148. Reducing the number of hidden nodes to 15 gives the same results: loss: 2.9253 - accuracy: 0.2978 - val_loss: 2.9350 - val_accuracy: 0.3148. Hidden layers with 64 nodes gives the same results. Training data with just 500 data points also gives the same results. Using sigmoid instead of ReLU gives slightly worse results.
I've been through many tutorials and guides on how to debug or check why the neural network is not working, but they don't teach properly, what these values mean and how to adjust the network.
Does the loss not decreasing, mean the network is not learning?
Does the fact that increasing or decreasing the layers or the number of training data mean that the network is not learning?

Improve this question
$\endgroup$
3
  • $\begingroup$ The inputs are quite complicated for an MLP, you might find better results if you reduce the number of inputs being fed in. As for loss not decreasing, yes that generally means the network has stopped learning. Quite often a neural network will not learn a problem not matter how many neurons or layers you throw at it, and then it's likely an issue of using a better suited model. This uses pytorch, but you might find some help here: github.com/juanto121/qwop-ai $\endgroup$
    – Recessive
    Dec 10, 2019 at 6:12
  • $\begingroup$ Thank you, but reducing it down to the most important 9 input parameters also didn't help. The losses are still an almost flat horizontal line and the accuracy fell loss: 2.9470 - accuracy: 0.3576 - val_loss: 2.9396 - val_accuracy: 0.2443. For these types of inputs and outputs I don't see how an RNN or LSTM or CNN would help. MLP is the only one possible. $\endgroup$
    – John David
    Dec 10, 2019 at 11:01
  • $\begingroup$ There is a chance that there is an error in your structure, just verify first that the network is able to learn a known value - say how to count up in binary (ie, input of 010 results in 011). How are you training this network? What are you using as training data? $\endgroup$
    – Recessive
    Dec 11, 2019 at 8:18

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .