It is my understanding that when training a Deep NN in Tensorflow/PyTorch/... we only keep the current state of the network in memory, except perhaps when we manually decide to save the current weights to the HDD/SSD.
Now, naively speaking it may seem reasonable to not only remember the current state (i.e. the current values of the trained weights), i.e. "where we are", but also the "best" weights "so far" by some metric, such as the validation error. Immediately, this approach doubles our memory requirements, especially if we want to keep everything in the GPU memory.
Is this done in practice? If not what are arguments against it?
Now there are certainly details that make this question more tricky. If you saved only each epoch it's probably fine to just save your weights to the HDD without significant loss in speed. If you remember the best weights for each step of SGD than you would have to compute the validation error over and over, which is costly. Or you use perhaps an estimate of the training error based on its gradient, which however could be leading to overfitting if you are not careful. You could also mix and match, or you could compute the validation error only on a subset of the data, or only remember the best every $k$ steps, etc. etc. You might also want to go back and restart from a "better" set of weights in the hopes of getting a better trajectory.
I also don't know whether in practice at the of training people typically, deliberately, go back to a previous "checkpoint" they might have saved somewhere.