Run the algorithm on a chunk of the data for a large number of iterations and then do the same with another chunk and so on?
(Such approach will not produce the same result as mini-batch gradient descent, as I am not including all data in one iteration, but rather learning from some data and then proceeding to learn on more data, beginning with the updated weights may still converge to a reasonably trained network.)
This might work if each of the chunks of data individually is still large enough and sufficiently representative of the distribution of the complete population, but probably not the best way to go. In fact, I don't expect it to perform much better than simply using only the very last chunk, and training only on that one. This is because of the following reason. Suppose you first train for a while on chunk A, then for a long time on chunk B, then for a long time on chunk C, etc. While learning on chunk B, there is a significant risk that your model will "forget" everything it learned from chunk A. When learning on chunk C afterwards, it can also "forget" everything learned from chunk B again.
In pseudocode, the approach you proposed here looks as follows:
for each chunk:
for large number of iterations:
learn on chunk()
An easy way to improve on that would be to swap the loops around:
for large number of iterations:
for each chunk:
learn on chunk()
What I just described there is actually how I interpret "mini-batch gradient descent" though, the chunks would be minibatches (and the minibatches / chunks would be randomly re-selected from the complete population in every iteration of the outer loop, you wouldn't always use the same chunks). Note that this wouldn't be effective if your dataset is so large that it doesn't fit inside your RAM all at the same time, because then you'll have to deal with excessive I/O.
- Run the same algorithm (the same model also with only data varying) on different PC's (each PC using a chunk of the data) and then see the performance on a test set and take the final decision as a weighted average of all the different models' outputs with the weight being high for the model which did the best on test sets?
Yes, this can definitely be effective. This kind of idea (training different models on different subsets of data) is generally referred to as "ensemble" methods. You can even vary the models you use (e.g., have an ensemble with some Random Forests, some SVMs, some Neural Networks, etc.).
