What explains the apparent 'mirroring' of the graphs on the RHS,
The model starts untrained and no better than random guessing (the baseline). As the training progresses, the model does better than random guessing on the training data, but does worse than initially on the validation data.
The decrease in performance is because the data it is being trained on is now deliberately labelled differently to the validation set. The task has been made impossible to generalise on, For example if there is a handwritten "8" labelled as 4 in the training database, then a very similar looking "8" in the validation database may be labelled 1 - if the trained model correctly matches up features, it will guess 4 consistently. This issue will occur frequently over the dataset, and similar issues will affect validation examples that are not close to training samples (in the feature space they will be "in-between" many training examples, most of which will be labelled differently to the validation label example).
The degree to which the curves on the random labels graph diverge is roughly affected by:
- Size of the training data. A larger training dataset will cause less divergence. A near-infinite dataset would result in a flat line close to the baseline for both training and validation results.
- Capacity of the model. A model with greater capacity to learn complex functions can better approximate the training data. The better it does this, the closer the training error will get to zero, and the larger the validation error will be as a result.
The mirroring effect between training and validation curves is not perfect, either in practice or "ideally". With a large dataset and very high capacity model capable of overfitting that dataset, then validation scores should be similar to random guessing and thus close to the baseline, whilst the training curve would tend towards zero error.
and the fact that training error on the RHS is approx. equal to validation error on the LHS?
That is a coincidence. Howver you can read it as roughly "despite the scrambled labels, the model does learn to predict labels on the training dataset, about as well as it generalised before on the unscrambled data".
It's evident from the graphs that the neural network is not learning noise - so what exactly is it learning?
It is learning a kind of noise - it is doing its best to approximate a function that returns a fixed set of incorrect, randomly assigned labels. When the random values are frozen in the dataset, they define a complex function that can be learned, even though it is not meaningful or useful for any other purpose.
The problem with validation error is that this noise function is not coherent, so there is no approximation that will ever do better than random chance when you look at ability to generalise. In fact it will usually do worse in rough proportion to how well it learns the training data.
If labels were somehow assigned using coherent noise e.g. Perlin noise in feature space (I'm not sure how you would do this, but it is feasible), then the model may perform better on the validation set.
If the labels were instead randomised each epoch, so that they were not fixed, then the training would learn nothing, and you would end up with a roughly flat line for both training and validation error.
