16
$\begingroup$

Let's call our dataset splits train/test/evaluate. We're in a situation where we require months of data. So we prefer to use the evaluation dataset as infrequently as possible to avoid polluting our results. Instead, we do 10 fold cross validation (CV) to estimate how well the model might generalize.

We're training deep learning models that take between 24-48 hours each, and the process of parameter sweeping is obviously very slow when performing 10-fold cross validation.

Does anyone have any experience or citations for how well parameter sweeping on one split of the data followed by cross validation (used to estimate how well it generalizes) works?

I suspect it's highly dependent on the distribution of data and local minima & maxima of the hyper parameters, but I wanted to ask.

Improve this question
$\endgroup$
1
  • $\begingroup$ One problem: you might slightly overfit when testing on the set that the parameter sweep was tested on. A more rigorous approach is to test the parameter sweep on a validation set, which is held out during cross validation. But you would still end up testing on the set that you trained the parameter sweep on, which has the potential to overfit. $\endgroup$
    – Reinstate Monica
    Commented Sep 25, 2019 at 13:41

1 Answer 1

Reset to default
0
$\begingroup$

Depending on the number of features you have you might want to try reducing features in order to reduce overfitting and speed up your model. I assume you are using proper regularization as well. PCA may be an option to help if time is the main issue, medium PCA. As the medium article states PCA can be used to reduce the dimensionality of the data while keeping retaining variance. If you run a random forest on the different sets and the feature importance ranking is at all different that is a big issue and the number of features should be reduced. Ideally, the features should be cropped till they have the same (structure/feature importance) but slightly different weights associated with those features. Here is a article demonstrating feature selection.

Here are some publications I believe are relevant to your problem. More in line with your desire to try to validate after one sweep. You may want to look up applications of LSTM or Yolo to your model I have a feeling these technologies will help direct your final decisions on what to do.

Subspace regularization method for the single-trial estimation of evoked potentials (1999) by P.A. Karjalainen, J.P. Kaipio, A.S. Koistinen, M. Vauhkonen

On Quadratic Penalties in Elastic Weight Consolidation (2017) by Ferenc Huszár

Parameter space exploration with Gaussian process trees (2014) by Robert B. Gramacy University of California, Santa Cruz, CA Herbert K. H. Lee University of California, Santa Cruz, CA William G. Macready

Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks (2017) by Nils Reimers, Iryna Gurevych

Evaluating Hospital Case Cost Prediction Models Using Azure Machine Learning Studio (2018) by Alexei Botchkarev -evaluation of regression performance

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .