Background:
This is for a simulated robot with four legs, walking on a flat terrain. The ANN (an MLP) is given inputs as the robot's body angle, positions and angle of each leg with respect to the body and two points of contact with the terrain, on each leg (if there's no contact, the value is zero). The outputs are the four motor rates for each leg.
I'm using Keras with the CNTK backend to train the network. The network has 30 input nodes (ReLU), one hidden layer with 64 nodes (ReLU) and an output layer with 4 nodes (Sigmoid). Optimizer is Adam
.
The training data has 2459 datapoints. Running model.validate
with parameters testDataPercentage = 0.25, epochs = 50 and batchSize = 10 gave me: loss: 2.9509 - accuracy: 0.3283 - val_loss: 2.8592 - val_accuracy: 0.3213
.
But running model.evaluate multiple times gave me:
['loss', 'accuracy'] [3.10, 0.50]
['loss', 'accuracy'] [3.04, 0.23]
['loss', 'accuracy'] [3.01, 0.11]
['loss', 'accuracy'] [3.45, 0.02]
['loss', 'accuracy'] [3.17, 0.40]
['loss', 'accuracy'] [3.03, 0.27]
['loss', 'accuracy'] [3.012, 0.46]
Loss doesn't decrease much over 50 epochs. It reduces from maybe 3 to 2.8. That's it.
Question:
I don't understand why the accuracy varies so much for each run.
If I add a hidden layer or even add a dropout of 0.2, the results are similar: loss: 2.9253 - accuracy: 0.2978 - val_loss: 2.9350 - val_accuracy: 0.3148
.
Reducing the number of hidden nodes to 15 gives the same results: loss: 2.9253 - accuracy: 0.2978 - val_loss: 2.9350 - val_accuracy: 0.3148
. Hidden layers with 64 nodes gives the same results. Training data with just 500 data points also gives the same results. Using sigmoid instead of ReLU gives slightly worse results.
I've been through many tutorials and guides on how to debug or check why the neural network is not working, but they don't teach properly, what these values mean and how to adjust the network.
Does the loss not decreasing, mean the network is not learning?
Does the fact that increasing or decreasing the layers or the number of training data mean that the network is not learning?
loss: 2.9470 - accuracy: 0.3576 - val_loss: 2.9396 - val_accuracy: 0.2443
. For these types of inputs and outputs I don't see how an RNN or LSTM or CNN would help. MLP is the only one possible. $\endgroup$