Depending on the quantity of features you have you might want to try reducing features in order to reduce overfitting, and speed up your model. I assume you are using proper regularization as well. PCA may be an option to help if time is the main issue, medium PCA. As the medium article states PCA can be used to reduce the dimensionality of the data while keeping retaining variance. If you run a random forrest on the different sets and the feature importance ranking is at all different that is a big issue and the amount of features should be reduced. Ideally the features should be cropped till they have the same (structure/feature importance) but slightly different weights associated with those features. Here is a article demonstrating feature selection.
Here are some publications I believe are relevant to your problem. More in line with your desire to try to validate after one sweep. You may want to look up applications of LSTM or Yolo to your model I have a feeling these technologies will help direct your final decisions on what to do.
Subspace regularization method for the single-trial estimation of evoked potentials (1999) by P.A. Karjalainen, J.P. Kaipio, A.S. Koistinen, M. Vauhkonen
On Quadratic Penalties in Elastic Weight Consolidation (2017) by Ferenc Huszár
Parameter space exploration with Gaussian process trees (2014) by Robert B. Gramacy University of California, Santa Cruz, CA
Herbert K. H. Lee University of California, Santa Cruz, CA
William G. Macready
Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks (2017) by Nils Reimers, Iryna Gurevych
Evaluating Hospital Case Cost Prediction Models Using Azure Machine Learning Studio (2018) by Alexei Botchkarev -evaluation of regression performance