Hot answers tagged

14 votes
Accepted

Why do you not see dropout layers on reinforcement learning examples?

Dropout essentially introduces a bit more variance. In supervised learning settings, this indeed often helps to reduce overfitting (although I believe there dropout is also already becoming less.. ...
user avatar
  • 9,379
7 votes
Accepted

Is pooling a kind of dropout?

Dropout and Max-pooling are performed for different reasons. Dropout is a regularization technique, which affects only the training process (during evaluation, it is not active). The goal of dropout ...
user avatar
  • 426
5 votes

Is some kind of dropout used in the human brain?

The human brain works by having neurons constantly fire at different rates. So, if the firing rate increases, the neuron is transmitting overly exciting or calming information to further neurons ...
user avatar
  • 705
5 votes
Accepted

Why is dropout favoured compared to reducing the number of units in hidden layers?

Dropout only ignores a portion of units during a single training batch update. Each training batch will use a different combination of units which gives them the best chance of that portion of them ...
user avatar
5 votes
Accepted

Why is my test error lower than the training error?

You use dropout during traing to reduce overfitting, but this reduces the training accuracy. The dropout will not be used during testing, therefore the accuracy will be higher. That's normal behavior ...
user avatar
  • 1,644
4 votes

Why is my test error lower than the training error?

I will just add to all the good answers already here. Like I said on my comment earlier, this is not a bad this(provided you have a split your data correctly). Other reasons could be: High ...
user avatar
4 votes
Accepted

Is the dropout technique specific only to neural networks?

I'm sure you can use dropout in any parameterized model, but I suspect it'll only really be helpful if you have enough parameters/nodes. Also dropout in neural nets has a Bayesian meaning, Yarin Gal ...
user avatar
  • 961
3 votes

Does the performance of a model increase if dropout is disabled at evaluation time?

Dropout is a technique that helps to avoid overfitting during training. That is, dropout is usually used for training. units may change in a way that they fix up the mistakes of the other units. This ...
user avatar
3 votes

Price Movement Forecasting Issue

Given a model that takes in a price and a second value, such as a moving average of the price, the following configuration is my recommendation. This is based on training on a history of 45 input ...
user avatar
3 votes

Does a bias also have a chance to be dropped out in Dropout layer?

Only the non-bias ones, It is discouraged to include the bias weights under norm penalty regularization for example, so why should it be included in the drop-out regularization scheme? Drop out can ...
user avatar
3 votes

5 years later, are maxout networks dead, and why?

Basically, if you read the full paper (especially, the abstract and the section 7), you find that the main accomplishment remains a marginal contribution on top of dropout. If you see the empirical ...
user avatar
2 votes

What to do if CNN cannot overfit a training set on adding dropout?

Sorry if this is a bad use of answer to add comment but since my reputation is not high enough this is only way to leave a comment to OP's question. I think some of the answers misunderstood the OP's ...
user avatar
2 votes

Is pooling a kind of dropout?

I think we would consider regularization and downsampling better in this way: dropout it puts some input value (neuron) for the next layer as 0, which makes the current layer a sparse one. So it ...
user avatar
  • 121
2 votes

Price Movement Forecasting Issue

Remember that any machine learning model works good only when there is a "rule" or a correlation between modeling data and modeled data. When there is not, even the best algorithm will not predict/...
user avatar
  • 1,039
2 votes

Does the performance of a model increase if dropout is disabled at evaluation time?

Dropout is usually disabled at test (or evaluation) time. For example, in Keras, dropout is disabled at evaluation time by default, although you can enable it, if you need to (see below). The purpose ...
user avatar
  • 33.8k
2 votes

Can dropout layers not influence LSTM training?

A couple of points: Have you firstly scaled your data, e.g. using MinMaxScaler? This could be one reason why your loss readings remain high. Additionally, consider that while Dropout can be useful ...
user avatar
2 votes

Why is dropout favoured compared to reducing the number of units in hidden layers?

The idea of dropout is that, at training time, with a certain probability $p_i \in [0, 1]$, the unit (or neuron) $i$ is dropped, $\forall i$, that is, the output of unit $i$ is set to zero so that $i$ ...
user avatar
  • 33.8k
1 vote

Is it mandatory to multiply every activation of a layer by droupout factor during testing?

There are two types of dropout, depending on whether a scaling correction is applied during: testing - without dropout applied, to decrease logits by a factor $1-p$ to match the expected magnitude ...
user avatar
  • 23.8k
1 vote

Is it possible that the model is overfitting when the training and validation accuracy increase?

I'll try to answer on more general questions Is it ok that model performs better on validation, then on train? It's certainly fine if you use techniques like dropout or data augmentation and the ...
user avatar
1 vote

How should we regularize an LSTM model?

If what is mentioned above, that is probably in the context of lstm networks. I would suggest using the keras tuner bayesian optimizer and making the l1 or l2 number a parameter of the kernel space. ...
user avatar
  • 11
1 vote

How should we regularize an LSTM model?

One LSTM layer should be enough unless you have lots of data. The same thing goes for the number of nodes in the layer. Start small first so 5 to 10 nodes and increment it until the performance is ...
user avatar
  • 216
1 vote
Accepted

Can Google's patented ML algorithms be used commercially?

Can you use them commercially? Yes. Is Google able to sue you any time they want? Yes. Will they do that... Probably not. Google isn't a known patent bully, I would give them the benefit of ...
user avatar
1 vote

Should I remove the units of a neural network or increase dropout?

There is no incentive to increase the size of the model for not reason. If a model of size x gives the best possible performance, there is no reason to use a model of size 2*x with 0.5 dropout during ...
user avatar
  • 426
1 vote
Accepted

Can the addition of dropout in a non-overfitting neural network increase accuracy?

Can the addition of dropout, in a non-overfitting neural network, increase accuracy? Yes, maybe. Even if I increase the complexity of the neural network? Yes, maybe. As always when making ...
user avatar
  • 23.8k
1 vote
Accepted

How to compute the action probabilities with Thompson sampling in deep Q-learning?

So, how can I calculate $\mu(a)$ when using Thompson Sampling based on dropout? The only way I could see this being calculated is if you iterate over all possible dropout combinations, or as an ...
user avatar
  • 23.8k
1 vote

What to do if CNN cannot overfit a training set on adding dropout?

Let's start with understanding what over-fitting means. Your model is over-fitting if during training your training loss continues to decrease but (in the later epochs) your validation loss begins to ...
user avatar
  • 624
1 vote

What are the counterparts of non-linearities and dropout in fully convolutional networks?

All-convolutional neural network is a more general concept which can be (and is often) used without deconvolutional and unpolling layers, e.g. for an ordinary classification task. The idea is to ...
user avatar
  • 1,877

Only top scored, non community-wiki answers of a minimum length are eligible