Let's quickly get out our copies of Deep Learning by Goodfellow et al. (2016). More specifically, I'm referring to page 276.
On this page, the authors argue for a relatively small minibatch size, since there are less than linear returns for estimating the gradient when increasing the minibatch size. Returns here refer to the reduction of the standard error of the mean (gradient per weight) computed over a minibatch.
So, yes. In theory, having unlimited resources, you will get the best performance when averaging the loss over all samples in your dataset. In practice, however, the larger the size of minibatches, the slower the training procedure, and consequently the less the total number of weight updates that can be afforded. Reversely, in practice, the cheaper the weight updates, the quicker the training procedure can converge to a (subjectively) satisfactory result.
Eventually, also Goodfellow et al. state that rapidly computing gradients leads to much faster convergence (in terms of total computations) for most optimization algorithms than when training them more slowly on exact gradients.
So, to summarize: If the main concern is to get to a specific level of accuracy at all, go for rather low minibatch sizes, whereas you could go up to a few hundreds (as the Goodfellow et al. state as a reasonable upper bound on page 148) if you are interested in more accurate gradients for your weight updates.