# Tag Info

18

There is no direct way to find the optimal number of them: people empirically try and see (e.g., using cross-validation). The most common search techniques are random, manual, and grid searches. There exist more advanced techniques such as Gaussian processes, e.g. Optimizing Neural Network Hyperparameters with Gaussian Processes for Dialog Act ...

15

It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU). You may want to look at ELU (exponential linear units) and SELU (self-normalising version of ELU). Under some mild assumptions, the latter has the nice property of self-normalisation, which mitigates the problem of vanishing and ...

13

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. I would quite like to see a reference to this suggestion, in case it has been misunderstood. As far as I know, there is no such effect in normal neural networks. In ...

7

For a more intelligent approach than random or exhaustive searches, you could try a genetic algorithm such as NEAT http://nn.cs.utexas.edu/?neat. However, this has no guarantee to find a global optima, it is simply an optimization algorithm based on performance and is therefore vulnerable to getting stuck in a local optima.

7

I have an idea to find the optimal number of hidden neurons required in a neural network but I'm not sure how accurate it is. It's a complete non-starter, and there is a no such calculation possible in the general case (real-valued inputs to a neural network). Even with one input neuron it is not possible. That is because even with one input, the output ...

5

Gradient Descent is a method to find the optimum parameter of the hypothesis or minimize the cost function. where alpha is learning rate If the learning rate is high then it can overshoot the minimum and can fail to minimize the cost function. hence result in a higher loss. Since Gradient descent can only find local minimum so, the lower learning rate ...

5

Paper Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[J]. arXiv preprint arXiv:1512.00567, 2015. gives some general design principles: Avoid representational bottlenecks, especially early in the network; Balance the width and depth of the network. Optimal performance of the network can be reached ...

4

The number of dimensions is a hyperparameter of your model, and you should do a hyperparameter search, like with any other parameters. There's also a tradeoff between dimension and training speed, so it should be small enough to be trainable in a reasonable time.

4

Usually, when talking about regularization for neural networks there are 3 main types: L1, L2 and dropout. All affect the gradient descent procedure. L1 and L2 regularization is implemented in the loss function, and therefore are part of gradient descent directly by altering the derivatives of the loss function thereby altering the weight update rules of ...

3

In general, it is definitely very computationally expensive, so an exhaustive search is not performed in practice. However, there are some recent approaches for determining whether the architecture is "fine" without training the neural network first - by looking at the covariance matrix after forwarding the data, for example, in a recent paper ...

3

As you stated, it's popular to have some form of a rectified linear unit (ReLU) activation in hidden layers and the output layer is often a softmax or sigmoid (depending also on the problem: multi-class or binary classification, respectively), which provides an output that can be viewed as a probability distribution. You could generalize this further to ...

3

Here is my take. The larger the $\lambda$, the more the corresponding regularization term for a coefficient will be big, so when minimizing the cost function, the coefficient will be reduced by a bigger factor, you can see this effect in the derivation of the update rule for gradient descent for example: \begin{align*} \theta_j := \theta_j - \alpha\ \left[ \...

3

Has this been done? Difficult to prove a negative, but I suspect although plenty of research has been done into finding ideal learning rate values (the need for learning rate at all is an annoyance), it has not been done to the level of suggesting a global function worth approximating. The problem is that learning rate tuning, like other hyperparameter ...

3

The correct number of child processes will depend on the hardware available to you. Simplifying a bit, child processes can be in one of two states: waiting for memory or disk access, or running. If your problem fits nicely in your computers' memory, then processes will spend almost all of their time running. If it's too big for memory, they will ...

3

***Take my answer as a side note to that given by cantordust: If one can verify that an activation function perform well in some cases, that good behavior often extrapolates to other problems. Thus, by testing activation functions on a few different problems, one can often infer how well (or badly) it will perform on most problems. The following video shows ...

3

In general, many of the parameters you mentioned are called hyperparameters. All hyperparameters are user-adjusted (or user-programmed) in training phase. Some hyperparameters are: learning rate, batch size, epochs, optimizer, layers, activation functions etc. To answer your (a) part of your question, there are obsiously many frameworks and libraries, for ...

3

$T = \infty$ and $\gamma = 1$ cannot be both true at the same time because the return defined in equation 3.11 is supposed to be a unified definition of the return for both continuing and episodic tasks. In the case of continuing tasks, $T = \infty$ and $\gamma = 1$ cannot be true at the same time, because the return may not be finite in that case (as I ...

3

Personally, I would choose the following two as the most important: epsilon: When using an epsilon-greedy policy, epsilon determines how often the agent should explore and how often it should exploit. Balancing exploration and exploitation is crucial for the success of the learning agent. Too little exploration might not teach anything to the agent and too ...

2

Take a look at this article. It give tools to actually understand what your filters have learn and show what you can do next to optimize your hyper-parameters. Also check more recent articles that seek to provide interpretations of what NN learn.

2

I've summarized the key ideas of SVMs. So this is how $\gamma$ is used with a gaussian Kernel: $$K_{\text{Gauss}}(\mathbf{x}_i, \mathbf{x}_j) = e^{\frac{-\gamma\|\mathbf{x}_i - \mathbf{x}_j\|^2}{2 \sigma^2}}$$ The bigger the $\gamma$, the more "linear" the decision boundary will be. The closer to 0, the more support vectors you have/the more non-linear the ...

2

The theory behind hyper-parameter optimization (HPO) is not well developed. Nonetheless, there are several hyper-parameter optimization approaches, such as Bayesian optimization (using Gaussian processes), random search, grid search, genetic algorithms, etc. See, for example, the paper Hyperparameter Search in Machine Learning (2015), which attempts to ...

2

After you've computed $h^{1}_{optimal}$ the only thing you can be sure is that this is the best (assuming constrained case) value of $h^1$ (with respect to some model performance metric) given your initial values for $h^2, ..., h^n$. If you change a bit any of $h^2, ..., h^n$ you're no longer certain that the value $h^1$ you found is the optimal one. So yes, ...

2

is it common to deal with weights and biases in everyday tasks or in most of the cases existing algorithms do it well? No; and it is no coincidence that you will not be able to find any reference to such a practice in any course or tutorial about neural networks. Such a practice would require a whole additional level of (business/SME) know-how in order to ...

2

It depends on your application. In case of text recognition, non-uniform kernels are used since the information about text is less on the horizontal axis and more on the vertical axis. If in your case it is applicable then, it will be good idea. But, if it is not you are better off using a smaller uniform kernel (2x2, maybe). You can also zero-pad your image ...

2

You should read this study https://arxiv.org/abs/2006.05990 which does some empirical study on this question, specifically for on-policy, continuous action space DRL. It suggests that discount factor and learning rate are the two most important parameters to tune, followed by the width of the policy/value functions. That study also reports that it's very ...

2

In most cases, seems that embedding dim is chosen empirically, by trial and error. Older papers in NLP used 300 conventionally https://petuum.medium.com/embeddings-a-matrix-of-meaning-4de877c9aa27. More recent papers used 512, 768, 1024. One of the factors, influencing the choice of embedding is the way you would like different vectors to correlate with each ...

1

If I understand your question correctly, you are asking whether you could load the saved weights of a trained model with the Xception architecture on a a Resnet50 architecture. Short answer: No Long answer: Xception and Resnet50 have very different architectures. Here is a paper comparing multilple CNNs including Xception and Resnet50: https://www....

1

"Selecting the model" in this case refers to selecting the hyperparameters of the model. The reason to use a nested CV is simply to avoid overfitting training data. Consider the example in the link. First you like to select the best hyperparameters of your svm model by GridSearchCV(). This is done by 4-fold CV. Now the clf.best_score_ will be the ...

1

Below are some tweaks that helped me accelerate the training of DDPG on a Reacher-like environment: Reducing the neural network size, compared to the original paper. Instead of: 2 hidden layers with 400 and 300 units respectively I used 128 units for both hidden layers. I see in your implementation that you used 256, maybe you could try reducing this. ...

Only top voted, non community-wiki answers of a minimum length are eligible