9
$\begingroup$

Let's call our dataset splits train/test/evaluate. We're in a situation where we require months of data so we prefer to use the evaluation dataset as infrequently as possible to avoid polluting our results. Instead we do 10 fold cross validation (CV) to estimate how well the model might generalize.

We're training deep learning models that take between 24-48 hours each and the process of parameter sweeping is obviously very slow when doing 10-fold cross validation.

Does anyone have any experience or citations for how well parameter sweeping on one split of the data followed by using cross validation to estimate how well it generalizes works?

I suspect it's highly dependent on the distribution of data and local minima & maxima of the hyper parameters but wanted to ask.

$\endgroup$
  • $\begingroup$ One problem: you might slightly overfit when testing on the set that the parameter sweep was tested on. A more rigorous approach is to test the parameter sweep on a validation set, which is held out during cross validation. But you would still end up testing on the set that you trained the parameter sweep on, which has the potential to overfit. $\endgroup$ – Reinstate Monica Sep 25 at 13:41
0
$\begingroup$

Depending on the quantity of features you have you might want to try reducing features in order to reduce overfitting, and speed up your model. I assume you are using proper regularization as well. PCA may be an option to help if time is the main issue, medium PCA. As the medium article states PCA can be used to reduce the dimensionality of the data while keeping retaining variance. If you run a random forrest on the different sets and the feature importance ranking is at all different that is a big issue and the amount of features should be reduced. Ideally the features should be cropped till they have the same (structure/feature importance) but slightly different weights associated with those features. Here is a article demonstrating feature selection.

Here are some publications I believe are relevant to your problem. More in line with your desire to try to validate after one sweep. You may want to look up applications of LSTM or Yolo to your model I have a feeling these technologies will help direct your final decisions on what to do.

Subspace regularization method for the single-trial estimation of evoked potentials (1999) by P.A. Karjalainen, J.P. Kaipio, A.S. Koistinen, M. Vauhkonen

On Quadratic Penalties in Elastic Weight Consolidation (2017) by Ferenc Huszár

Parameter space exploration with Gaussian process trees (2014) by Robert B. Gramacy University of California, Santa Cruz, CA Herbert K. H. Lee University of California, Santa Cruz, CA William G. Macready

Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks (2017) by Nils Reimers, Iryna Gurevych

Evaluating Hospital Case Cost Prediction Models Using Azure Machine Learning Studio (2018) by Alexei Botchkarev -evaluation of regression performance

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.