Questions tagged [dropout]

For questions related to the concept of dropout, which refers to the dropping out units in a neural network (NN), during the training of the NN, so that to avoid overfitting. The dropout method is a regularisation technique, which was introduced in "Dropout: A Simple Way to Prevent Neural Networks from Overfitting" (2014) by Nitish Srivastava et al.

Filter by
Sorted by
Tagged with
1
vote
0answers
24 views

How to make binary neural networks resilient to flipped activation values?

Assume I am given a binary neural network where the activation values are constrained to be 0 or 1 (by clipping the ReLU function). Additionally, assume the neural network is supposed to work in a ...
1
vote
1answer
59 views

Why is my validation/test accuracy higher than my training accuracy

Is this due to my dropout layers being disabled during evaluation? I'm classifying the CIFAR-10 dataset with a CNN using the Keras library. There are 50000 samples in the training set; I'm using a ...
1
vote
3answers
72 views

Does the performance of a model increase if dropout is disabled at evaluation time?

I know dropout layers are used in neural networks during training to provide a form of regularisation in an attempt to mitigate over-fitting. Would you not get an increased fitness if you disabled ...
0
votes
0answers
14 views

Tensorflow neural network - inherent overfitting in certain X and Y distributions?

Really open ended question here... I'm working on a classifier model that is used to predict whether some amount is > x, ...
3
votes
1answer
68 views

Can dropout layers not influence LSTM training?

I am working on a project that requires time-series prediction (regression) and I use LSTM network with first 1D conv layer in Keras/TF-gpu as follows: ...
6
votes
2answers
141 views

Why is dropout favoured compared to reducing the number of units in hidden layers?

Why is dropout favored compared to reducing the number of units in hidden layers for the convolutional networks? If a large set of units leads to overfitting and dropping out "averages" the response ...
3
votes
1answer
64 views

Can Google's patented ML algorithms be used commercially?

I just find that Google patents some of the widely used machine learning algorithms. For example: System and method for addressing overfitting in a neural network (Dropout?) Processing images using ...
1
vote
1answer
77 views

Dropout causes too much noise for network to train

I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly ...
3
votes
2answers
1k views

Is pooling a kind of DropOut

If I got well the global idea of DropOut it allows to improve the sparsity of the information that come from one layer to another by setting some weights to zero. In another hand, pooling, let's say ...
2
votes
1answer
42 views

Should I remove the units of a neural network or increase dropout?

When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is ...
3
votes
2answers
126 views

Price Movement Forecasting Issue

I am working on a project for price movement forecasting and I am stuck with poor quality predictions. At every time-step I am using an LSTM to predict the next 10 time-steps. The input is the ...
2
votes
1answer
90 views

Can the addition of dropout in a non-overfitting neural network increase accuracy?

According to Wikipedia Dropout is a regularization technique for reducing overfitting in neural networks My neural network is simple enough and does not overfit. Can the addition of dropout, in a ...
13
votes
1answer
2k views

Why do you not see dropout layers on reinforcement learning examples?

I've been looking at reinforcement learning, and specifically playing around with creating my own environments to use with the OpenAI Gym AI. I am using agents from the stable_baselines project to ...
8
votes
1answer
126 views

5 years later, are maxout networks dead, and why?

Maxout networks were a simple yet brilliant idea of Goodfellow et al. from 2013 to max feature maps to get a universal approximator of convex activations. The design was tailored for use in ...
3
votes
1answer
222 views

Action Probability with Thompson Sampling in Deep Reinforcement Learning

In some implementations of off-policy Q learning we need to know the action probabilities given by the behavior policy mu(a) (e.g., if we want to use importance ...
5
votes
2answers
743 views

What to do if CNN cannot overfit a training set on adding dropout?

I have been trying to use CNN for a regression problem. I followed the standard recommendation of disabling dropout and overfitting a small training set prior to trying for generalization. With a 10 ...
2
votes
1answer
733 views

What are the counterparts of non-linearities and dropout in fully convolutional networks?

I am trying to replicate the fully convolutional networks (FCN) concept described here for semantic segmentation. It seems people have successfully trained such models by removing fully connected ...
4
votes
1answer
690 views

Does a bias also have a chance to be dropped out in Dropout layer?

Suppose that you have 80 neurons in a layer, where one neuron is bias. Then you add a dropout layer after the activation function of this layer. In this case, does it have a chance to drop out the ...