# Tag Info

## Hot answers tagged hyperparameter-optimization

18

There is no direct way to find the optimal number of them: people empirically try and see (e.g., using cross-validation). The most common search techniques are random, manual, and grid searches. There exist more advanced techniques such as Gaussian processes, e.g. Optimizing Neural Network Hyperparameters with Gaussian Processes for Dialog Act ...

15

It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU). You may want to look at ELU (exponential linear units) and SELU (self-normalising version of ELU). Under some mild assumptions, the latter has the nice property of self-normalisation, which mitigates the problem of vanishing and ...

14

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. I would quite like to see a reference to this suggestion, in case it has been misunderstood. As far as I know, there is no such effect in normal neural networks. In ...

11

I think in this case, you'll probably want to use a genetic algorithm to generate a topology rather than working on your own. I personally like NEAT (NeuroEvolution of Augmenting Topologies). The original NEAT paper involves evolving weights for connections, but if you only want a topology, you can use a weighting algorithm instead. You can also mix ...

7

For a more intelligent approach than random or exhaustive searches, you could try a genetic algorithm such as NEAT http://nn.cs.utexas.edu/?neat. However, this has no guarantee to find a global optima, it is simply an optimization algorithm based on performance and is therefore vulnerable to getting stuck in a local optima.

7

I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is. It's a complete non-starter, and there is a no such calculation possible in the general case (real-valued inputs to a neural network). Even with one input neuron it is not possible. That is because even with one input, the output ...

6

You could say that NAS fits into the domain of Meta Learning or Meta Machine learning. I've pulled the NAS papers from my notes, this is a collection of papers/lectures that I personally found very interesting. It's sorted in rough chronological descending order, and *** means influential / must read. Quoc V. Le and Barret Zoph are to good authors on the ...

6

There is a technique called Pruning in neural networks, which is used just for this same purpose. The pruning is done on the number of hidden layers. The process is very similar to the pruning process of decision trees. The pruning process is done as follows: Train a large, densely connected, network with a standard training algorithm Examine the trained ...

6

Unfortunately, even with large amounts of training data, hyperparameter choices can strongly influence the performance of a trained model. What you can usually drop when you have large amounts of training data is regularisation. If your training examples cover the function space you are learning really well, then it is harder to overfit the training data. ...

5

Paper Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[J]. arXiv preprint arXiv:1512.00567, 2015. gives some general design principles: Avoid representational bottlenecks, especially early in the network; Balance the width and depth of the network. Optimal performance of the network can be reached ...

5

Your scenario is common. The most straightforward approach is to subsample your data randomly. Unless your data or your model has strong bias, your performance to the smaller data set should be comparable. The accuracy might be lower, but the purpose is to do quick sanity check.

5

The other answer mentions NEAT to generate network weights or topologies. The paper NeuroEvolution: The Importance of Transfer Function Evolution and Heterogeneous Networks, which also gives a short summary of neuroevolution techniques, provides an alternative approach to NEAT. It uses Cartesian Genetic Programming to evolve multiple activation functions.

5

A surrogate model is a simplified model. It is a mapping $y_S=f_S(x)$ that approximates the original model $y=f(x)$, in a given domain, reasonably well. Source: Engineering Design via Surrogate Modelling: A Practical Guide In the context of Bayesian optimization, one wants to optimize a function $y=f(x)$ which is expensive (very time consuming) to evaluate, ...

5

The difference between the validation and test set in my opinion should be explained in this way: the validation set is meant to be used multiple times. the test set is meant to be used only once. I think that the misunderstanding here arise because machine learning is mostly taught focusing only on a specific part of a large pipeline, which is the model ...

5

In reinforcement learning, exploration has a specific meaning, which is in contrast with the meaning of exploitation, hence the so-called exploration-exploitation dilemma (or trade-off). You explore when you decide to visit states that you have not yet visited or to take actions you have not yet taken. On the other hand, you exploit when you decide to take ...

4

Yes. Usually you would use cross validation to avoid overfitting during parameter tuning. If your dataset is large enough, and you don't try too many parameter combinations, this will work well, because to "get lucky" and overfit, a parameter combination will need to work very well on many variations of the problem, which is less likely than working well on ...

4

What is Bayesian optimization? Introduction Bayesian optimization (BO) is an optimization technique used to model an unknown (usually continuous) function $f: \mathbb{R}^d \rightarrow Y$, where typically $d \leq 20$, so it can be used to solve regression and classification problems, where you want to find an approximation of $f$. In this sense, BO is ...

4

The book Deep Learning by Goodfellow, Bengio, and Courville says (Sec 8.3.3, p 292 in my copy) states that Unfortunately, in the stochastic gradient case, Nesterov momentum does not improve the rate of convergence. I'm not sure why this is, but the theoretical advantage depends on a convex problem, and from this, it sounds like the practical advantage ...

3

Yes you can decay the learning rate in Q-learning, and yes this should result in more accurate Q-values in the long term for many environments. However, this is something that is harder to manage than in supervised learning, and might not be as useful or necessary. The issues you need to be concerned about are: Non-stationarity The target values in value-...

3

The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so it should be small enough to be trainable in a reasonable time.

3

The correct number of child processes will depend on the hardware available to you. Simplifying a bit, child processes can be in one of two states: waiting for memory or disk access, or running. If your problem fits nicely in your computers' memory, then processes will spend almost all of their time running. If it's too big for memory, they will ...

3

***Take my answer as a side note to that given by cantordust: If one can verify that an activation function perform well in some cases, that good behavior often extrapolates to other problems. Thus, by testing activation functions on a few different problems, one can often infer how well (or badly) it will perform on most problems. The following video shows ...

3

In principle, yes, you can also evolve the genetic algorithm (or, in general, evolutionary algorithm), i.e. you can evolve its operations (such as the mutation and cross-over) and hyper-parameters (such as the size of the population or mutation rate). For example, you could use genetic programming to evolve the cross-over operation of a genetic algorithm. ...

3

In general, it is definitely very computationally expensive, so an exhaustive search is not performed in practice. However, there are some recent approaches for determining whether the architecture is "fine" without training the neural network first - by looking at the covariance matrix after forwarding the data, for example, in a recent paper ...

3

The crossover rate, $p_c \in [0, 1]$, is a hyper-parameter that controls the rate at which solutions are subjected to crossover. So, the higher $p_c$, the more crossovers you perform, so the more diversity (in terms of solutions/chromosomes) you may introduce in the population. Typical values of $p_c$ are in the range $[0.5, 1.0]$. For example, in this ...

3

Personally, I would choose the following two as the most important: epsilon: When using an epsilon-greedy policy, epsilon determines how often the agent should explore and how often it should exploit. Balancing exploration and exploitation is crucial for the success of the learning agent. Too little exploration might not teach anything to the agent and too ...

3

You don't NEED a hyperparameter tuner, but it can help in various situations. For example, if your model is not training well, perhaps using a tuner can help. It's hard to say in which hyperparameters you would be turning over in your specific model, but for some specific hyperparameters if you choose a bad value your model won't learn or diverge. Take for ...

2

Take a look at this article. It give tools to actually understand what your filters have learn and show what you can do next to optimize your hyper-parameters. Also check more recent articles that seek to provide interpretations of what NN learn.

2

First I need to note that there is no prescribed/best way to choose the shape of membership function in fuzzy systems, that's the fuzziness in it. One could argue that the best way is to ask an expert in the field where you are going to apply your fuzzy solution but those are not always available. With that said, fuzzy membership functions are used to ...

Only top voted, non community-wiki answers of a minimum length are eligible