6

You could say that NAS fits into the domain of Meta Learning or Meta Machine learning. I've pulled the NAS papers from my notes, this is a collection of papers/lectures that I personally found very interesting. It's sorted in rough chronological descending order, and *** means influential / must read. Quoc V. Le and Barret Zoph are to good authors on the ...


4

A surrogate model is a simplified model. It is a mapping $y_S=f_S(x)$ that approximates the original model $y=f(x)$, in a given domain, reasonably well. Source: Engineering Design via Surrogate Modelling: A Practical Guide In the context of Bayesian optimization, one wants to optimize a function $y=f(x)$ which is expensive (very time consuming) to evaluate, ...


3

Yes. Usually you would use cross validation to avoid overfitting during parameter tuning. If your dataset is large enough, and you don't try too many parameter combinations, this will work well, because to "get lucky" and overfit, a parameter combination will need to work very well on many variations of the problem, which is less likely than working well on ...


2

Such a system can and does solve the problem of hyperparameter tuning. Google's AutoML does this. Here is another example that uses a Genetic Algorithm to breed new neural network structures. AutoML has been shown to outperform humans in the rate that it improves network designs. It seems to favour Residual Network style topologies.


1

What is Bayesian optimization? Introduction Bayesian optimization (BO) is an optimization technique used to model an unknown (usually continuous) function $f: \mathbb{R}^d \rightarrow Y$, where typically $d \leq 20$, so it can be used to solve regression and classification problems, where you want to find an approximation of $f$. In this sense, BO is ...


1

Here are two review articles: Elsken, Metzen, Hutter: Neural Architecture Search: A Survey (2019), Journal of Machine Learning Research 20, 1-21 He, Zhao, Chu: AutoML: A Survey of the State-of-the-Art (2019)


1

See here for a potential way to do it: http://infinity77.net/global_optimization/#motivation-motivation http://infinity77.net/global_optimization/#rules-the-rules You basically test the two (or more) optimization algorithms against known objective functions, with several random (but repeatable) starting points and then analyze the outcome.


1

tl;dr The safest method I've found is to use cross-validation for hyperparameter selection and a hold-out test set for a final evaluation. Why this isn't working for you... In your case, I suspect you're either running a large number of iterations during for hyperparameter selection or you have a fairly small dataset (or even a combination of both). If ...


1

The theory behind hyper-parameter optimization (HPO) is not well developed. Nonetheless, there are several hyper-parameter optimization approaches, such as Bayesian optimization (using Gaussian processes), random search, grid search, genetic algorithms, etc. See, for example, the paper Hyperparameter Search in Machine Learning (2015), which attempts to ...


1

After you've computed $h^{1}_{optimal}$ the only thing you can be sure is that this is the best (assuming constrained case) value of $h^1$ (with respect to some model performance metric) given your initial values for $h^2, ..., h^n$. If you change a bit any of $h^2, ..., h^n$ you're no longer certain that the value $h^1$ you found is the optimal one. So yes, ...


1

Yes, you can also automate the choice of certain hyperparameters of the evolutionary algorithm. In this context, this process is called self-adaptation. There are different ways of performing self-adaptation (depending on the hyper-parameter that needs to self-adapt). See e.g. the chapter Self-Adaptation in Evolutionary Algorithms (by Silja Meyer-Nieberg and ...


Only top voted, non community-wiki answers of a minimum length are eligible