Tag Info

14

Dropout essentially introduces a bit more variance. In supervised learning settings, this indeed often helps to reduce overfitting (although I believe there dropout is also already becoming less.. fashionable in recent years than in the few years before that; I'm not 100% sure though, it's not my primary area of expertise). In Reinforcement Learning, ...

6

Dropout and Max-pooling are performed for different reasons. Dropout is a regularization technique, which affects only the training process (during evaluation, it is not active). The goal of dropout is reduce unnecessary feature dependencies in the network, allowing it to be simpler and improves its generalization abilities (reduces overfitting). In simple ...

5

The human brain works by having neurons constantly fire at different rates. So, if the firing rate increases, the neuron is transmitting overly exciting or calming information to further neurons connected to it. How other neurons connected to the former neuron respond on the messages sent by it, depends on the strength of the connection between the connected ...

5

You use dropout during traing to reduce overfitting, but this reduces the training accuracy. The dropout will not be used during testing, therefore the accuracy will be higher. That's normal behavior if you work with dropout.

5

Dropout only ignores a portion of units during a single training batch update. Each training batch will use a different combination of units which gives them the best chance of that portion of them working together to generalize. Note the weights for each unit are kept and will be updated during the next batch in which that unit is selected. During ...

4

I'm sure you can use dropout in any parameterized model, but I suspect it'll only really be helpful if you have enough parameters/nodes. Also dropout in neural nets has a Bayesian meaning, Yarin Gal for example has done lots of work on this. In your decision tree example, I believe you're talking about pruning, which is different. In that context you're ...

3

Basically, if you read the full paper (especially, the abstract and the section 7), you find that the main accomplishment remains a marginal contribution on top of dropout. If you see the empirical results on Table 5 (of the page 5) of the maxout's original paper, you find that the misclassification rate is only very, very slightly lower than that of ...

3

I will just add to all the good answers already here. Like I said on my comment earlier, this is not a bad this(provided you have a split your data correctly). Other reasons could be: High dropout rate or excessive data augmentation could be one of the reason. This can cause the training accuracy to appear low whist in reality the model is in fact ...

2

Only the non-bias ones, It is discouraged to include the bias weights under norm penalty regularization for example, so why should it be included in the drop-out regularization scheme? Drop out can be implemented by multiplying units with zero and the bias term is rather special. The bias term determines the distance from the origin the linear decision ...

2

Sorry if this is a bad use of answer to add comment but since my reputation is not high enough this is only way to leave a comment to OP's question. I think some of the answers misunderstood the OP's intention. Over fitting is used as a means to test the complexity of the model - if a model cannot overfit a small dataset then it's likely not able to ...

2

Dropout is usually disabled at test (or evaluation) time. For example, in Keras, dropout is disabled at evaluation time by default, although you can enable it, if you need to (see below). The purpose of dropout is to decorrelate the units (or feature detectors) so that they learn more robust representations of the data (i.e. a form of regularisation). ...

2

Dropout is a technique that helps to avoid overfitting during training. That is, one can use dropout only for training. units may change in a way that they fix up the mistakes of the other units. This may lead to complex co-adaptations. This, in turn, leads to overfitting because these co-adaptations do not generalize to unseen data. If you want to ...

2

A couple of points: Have you firstly scaled your data, e.g. using MinMaxScaler? This could be one reason why your loss readings remain high. Additionally, consider that while Dropout can be useful for reducing overfitting, it is not necessarily a panacea. Let's take an example of using LSTM to forecast fluctuations in weekly hotel cancellations. Model ...

2

The idea of dropout is that, at training time, with a certain probability $p_i \in [0, 1]$, the unit (or neuron) $i$ is dropped, $\forall i$, that is, the output of unit $i$ is set to zero so that $i$ does not affect the other units it is connected to, both during the forward and backward (or back-propagation) passes (or steps). At every mini-batch, you ...

2

Given a model that takes in a price and a second value, such as a moving average of the price, the following configuration is my recommendation. This is based on training on a history of 45 input time steps and forecasting out 10 steps in the future. I have assumed that your 'num_steps' is a stride through the training data. Note that I have not tested ...

2

I think we would consider regularization and downsampling better in this way: dropout it puts some input value (neuron) for the next layer as 0, which makes the current layer a sparse one. So it reduces the dependence of each feature in this layer. pooling layer the downsampling directly remove some input, and that makes the layer "smaller" rather than "...

1

Can you use them commercially? Yes. Is Google able to sue you any time they want? Yes. Will they do that... Probably not. Google isn't a known patent bully, I would give them the benefit of the doubt in this kind of situation and say, unless you start really giving them real trouble, they wouldn't do anything. Some companies/people know an idea can ...

1

I'll try to answer on more general questions Is it ok that model performs better on validation, then on train? It's certainly fine if you use techniques like dropout or data augmentation and the difference is not that big. Because in case of dropout for train you use part of the network, and for validation the whole. I'm suspicious my model is too good. ...

1

If what is mentioned above, that is probably in the context of lstm networks. I would suggest using the keras tuner bayesian optimizer and making the l1 or l2 number a parameter of the kernel space. This way you find the optimal values, and its a great way to hypertune. Just keep in mind, the greater the range of parameters, or kernel if i am not wrong, the ...

1

One LSTM layer should be enough unless you have lots of data. The same thing goes for the number of nodes in the layer. Start small first so 5 to 10 nodes and increment it until the performance is reasonable. Once you have a model working you can apply regularization if you think it will improve performance by reducing overfitting of the training data. You ...

1

There is no incentive to increase the size of the model for not reason. If a model of size x gives the best possible performance, there is no reason to use a model of size 2*x with 0.5 dropout during training. Usually we want to find the smallest possible model with the best performance. Inflating the model just results in higher computational requirements. ...

1

Remember that any machine learning model works good only when there is a "rule" or a correlation between modeling data and modeled data. When there is not, even the best algorithm will not predict/classify correctly. I am not saying that this must be the case, but probably you have come pretty close to the physical limit of what can be achieved using this ...

1

Can the addition of dropout, in a non-overfitting neural network, increase accuracy? Yes, maybe. Even if I increase the complexity of the neural network? Yes, maybe. As always when making changes to ML algorithms, you need to test carefully to see if your changes have made an improvement. There are very few theories in non-linear machine learning ...

1

So, how can I calculate $\mu(a)$ when using Thompson Sampling based on dropout? The only way I could see this being calculated is if you iterate over all possible dropout combinations, or as an approximation sample say 100 or 1000 actions with different dropout, to get a rough distribution. I don't think this is feasible for practical reasons (the agent ...

1

Let's start with understanding what over-fitting means. Your model is over-fitting if during training your training loss continues to decrease but (in the later epochs) your validation loss begins to increase. That means the model can not generalize well to images it has not previously encountered. Naturally, you do not want this situation. What you want is ...

1

All-convolutional neural network is a more general concept which can be (and is often) used without deconvolutional and unpolling layers, e.g. for an ordinary classification task. The idea is to replace the pooling and fully-connected layer with particular convolutional layers that do the same. Note that activation functions and dropout are not affected by ...

Only top voted, non community-wiki answers of a minimum length are eligible