Hot answers tagged

18 votes
Accepted

Why do you not see dropout layers on reinforcement learning examples?

Dropout essentially introduces a bit more variance. In supervised learning settings, this indeed often helps to reduce overfitting (although I believe there dropout is also already becoming less.. ...
Dennis Soemers's user avatar
  • 10.2k
8 votes
Accepted

Is pooling a kind of dropout?

Dropout and Max-pooling are performed for different reasons. Dropout is a regularization technique, which affects only the training process (during evaluation, it is not active). The goal of dropout ...
Mark.F's user avatar
  • 446
7 votes
Accepted

Why is dropout favoured compared to reducing the number of units in hidden layers?

Dropout only ignores a portion of units during a single training batch update. Each training batch will use a different combination of units which gives them the best chance of that portion of them ...
user1269942's user avatar
5 votes

Is some kind of dropout used in the human brain?

The human brain works by having neurons constantly fire at different rates. So, if the firing rate increases, the neuron is transmitting overly exciting or calming information to further neurons ...
Daniel B.'s user avatar
  • 815
5 votes
Accepted

Why is my test error lower than the training error?

You use dropout during traing to reduce overfitting, but this reduces the training accuracy. The dropout will not be used during testing, therefore the accuracy will be higher. That's normal behavior ...
Demento's user avatar
  • 1,684
4 votes

Why is my test error lower than the training error?

I will just add to all the good answers already here. Like I said on my comment earlier, this is not a bad this(provided you have a split your data correctly). Other reasons could be: High ...
Tshilidzi Mudau's user avatar
4 votes
Accepted

Is the dropout technique specific only to neural networks?

I'm sure you can use dropout in any parameterized model, but I suspect it'll only really be helpful if you have enough parameters/nodes. Also dropout in neural nets has a Bayesian meaning, Yarin Gal ...
harwiltz's user avatar
  • 1,126
3 votes
Accepted

How does dropout work during backpropagation?

Backpropagation on a network with dropout works just as it does normally, it calculates the gradients and updates the weights. Longer explanation Dropout is a regularization technique which drops ...
Robin van Hoorn's user avatar
3 votes

Does the performance of a model increase if dropout is disabled at evaluation time?

Dropout is a technique that helps to avoid overfitting during training. That is, dropout is usually used for training. units may change in a way that they fix up the mistakes of the other units. This ...
Aray Karjauv's user avatar
3 votes

Price Movement Forecasting Issue

Given a model that takes in a price and a second value, such as a moving average of the price, the following configuration is my recommendation. This is based on training on a history of 45 input ...
Brian O'Donnell's user avatar
3 votes

Does a bias also have a chance to be dropped out in Dropout layer?

Only the non-bias ones, It is discouraged to include the bias weights under norm penalty regularization for example, so why should it be included in the drop-out regularization scheme? Drop out can ...
Ricardicus's user avatar
3 votes

5 years later, are maxout networks dead, and why?

Basically, if you read the full paper (especially, the abstract and the section 7), you find that the main accomplishment remains a marginal contribution on top of dropout. If you see the empirical ...
Amrinder Arora's user avatar
2 votes

What to do if CNN cannot overfit a training set on adding dropout?

Sorry if this is a bad use of answer to add comment but since my reputation is not high enough this is only way to leave a comment to OP's question. I think some of the answers misunderstood the OP's ...
Tom Charles Zhang's user avatar
2 votes

Should I remove the units of a neural network or increase dropout?

There is no incentive to increase the size of the model for not reason. If a model of size x gives the best possible performance, there is no reason to use a model of size 2*x with 0.5 dropout during ...
Mark.F's user avatar
  • 446
2 votes

Price Movement Forecasting Issue

Remember that any machine learning model works good only when there is a "rule" or a correlation between modeling data and modeled data. When there is not, even the best algorithm will not predict/...
GKozinski's user avatar
  • 1,250
2 votes

Can dropout layers not influence LSTM training?

A couple of points: Have you firstly scaled your data, e.g. using MinMaxScaler? This could be one reason why your loss readings remain high. Additionally, consider that while Dropout can be useful ...
Michael Grogan's user avatar
2 votes

Why is dropout favoured compared to reducing the number of units in hidden layers?

The idea of dropout is that, at training time, with a certain probability $p_i \in [0, 1]$, the unit (or neuron) $i$ is dropped, $\forall i$, that is, the output of unit $i$ is set to zero so that $i$ ...
nbro's user avatar
  • 40.1k
2 votes

How should we regularize an LSTM model?

One LSTM layer should be enough unless you have lots of data. The same thing goes for the number of nodes in the layer. Start small first so 5 to 10 nodes and increment it until the performance is ...
Marcus's user avatar
  • 226
2 votes

Is pooling a kind of dropout?

I think we would consider regularization and downsampling better in this way: dropout it puts some input value (neuron) for the next layer as 0, which makes the current layer a sparse one. So it ...
Bs He's user avatar
  • 121
2 votes

Does the performance of a model increase if dropout is disabled at evaluation time?

Dropout is usually disabled at test (or evaluation) time. For example, in Keras, dropout is disabled at evaluation time by default, although you can enable it, if you need to (see below). The purpose ...
nbro's user avatar
  • 40.1k
1 vote

In the Dropout paper, why would increasing the dropout increase the error rate if the capacity is constant?

In the images below (figure 9 from section 7.3) $n$ refers to a unit in the network and $p$ refers to the probability of retaining a unit in the network. The combination of both, $pn$ thus refers to ...
Mariusmarten's user avatar
1 vote

Is it mandatory to multiply every activation of a layer by droupout factor during testing?

There are two types of dropout, depending on whether a scaling correction is applied during: testing - without dropout applied, to decrease logits by a factor $1-p$ to match the expected magnitude ...
Neil Slater's user avatar
  • 31.5k
1 vote
Accepted

Why doesn't dropout mislead results during evaluation?

How/why do we achieve the same/similar results though we are skipping a layer altogether Dropout is not a layer, even tough deep learning libraries implement it as a layer module for convenience. Why ...
Edoardo Guerriero's user avatar
1 vote

Is it possible that the model is overfitting when the training and validation accuracy increase?

I'll try to answer on more general questions Is it ok that model performs better on validation, then on train? It's certainly fine if you use techniques like dropout or data augmentation and the ...
Kirill Fedyanin's user avatar
1 vote

How should we regularize an LSTM model?

If what is mentioned above, that is probably in the context of lstm networks. I would suggest using the keras tuner bayesian optimizer and making the l1 or l2 number a parameter of the kernel space. ...
aimonk's user avatar
  • 11
1 vote
Accepted

Can Google's patented ML algorithms be used commercially?

Can you use them commercially? Yes. Is Google able to sue you any time they want? Yes. Will they do that... Probably not. Google isn't a known patent bully, I would give them the benefit of ...
Michael Hearn's user avatar
1 vote
Accepted

Can the addition of dropout in a non-overfitting neural network increase accuracy?

Can the addition of dropout, in a non-overfitting neural network, increase accuracy? Yes, maybe. Even if I increase the complexity of the neural network? Yes, maybe. As always when making ...
Neil Slater's user avatar
  • 31.5k
1 vote
Accepted

How to compute the action probabilities with Thompson sampling in deep Q-learning?

So, how can I calculate $\mu(a)$ when using Thompson Sampling based on dropout? The only way I could see this being calculated is if you iterate over all possible dropout combinations, or as an ...
Neil Slater's user avatar
  • 31.5k
1 vote

What to do if CNN cannot overfit a training set on adding dropout?

Let's start with understanding what over-fitting means. Your model is over-fitting if during training your training loss continues to decrease but (in the later epochs) your validation loss begins to ...
Gerry P's user avatar
  • 714
1 vote

What are the counterparts of non-linearities and dropout in fully convolutional networks?

All-convolutional neural network is a more general concept which can be (and is often) used without deconvolutional and unpolling layers, e.g. for an ordinary classification task. The idea is to ...
Maxim's user avatar
  • 1,947

Only top scored, non community-wiki answers of a minimum length are eligible