Questions tagged [regularization]
For questions about application of regularization techniques.
61
questions
2
votes
0
answers
38
views
Does this learning scenario have a name? If so, can someone point me to relevant literature?
I am faced with a problem which I bet was already solved before, but that I had never seen. Perhaps by discussing it abstractly, someone can point me to relevant literature.
It goes like this: I have ...
- 121
0
votes
0
answers
8
views
Understanding Test Output Calculation in DropConnect
I've been studying the DropConnect regularization technique for neural networks and I'm trying to understand how the test output is calculated. I understand that during training, DropConnect randomly ...
0
votes
1
answer
31
views
How to handle BatchNorm in the last layers of Neural Networks?
I am creating a neural network using batchnorm as a regularization method to enable deep models and prevent overfitting.
I understand that batchnorming supresses the internal covariance shift ...
- 121
4
votes
1
answer
167
views
Combine multiple losses with gradient descent
I am optimizing a neural network with Adam using 3 different losses. Their scale is very different, and the current method is to either sum the losses and clip the gradient or to manually weight them ...
- 153
1
vote
1
answer
64
views
Do different models using early stopping have the same validation set to check model training performance?
I, i have a doubt about making validation using early stopping given two NN models.
Suppose I have two models M1 and M2 and a Training set TS and Test set TS.
Take the TS and consider TS_80% and TS_20%...
- 113
0
votes
1
answer
43
views
Should weight decay regularization be divided by the number of samples?
I was watching a video by Andrew Ng about regularization in logistic regression and neural network models.
He uses the following term for regularization to (the sum is over the weights in the network)....
- 245
1
vote
1
answer
53
views
In the Dropout paper, why would increasing the dropout increase the error rate if the capacity is constant?
In the original paper on dropout, in section 7.3.2, we see that while keeping $pn$ constant, we get a (test) error increase by decreasing retainment below 0.6. Why would that happen? If $pn$ is ...
- 11
0
votes
0
answers
27
views
Is there merit in sampling dropout from a more complex distribution?
In practice, Dropout is typically applied uniformly over hidden neurons in a network. Is there merit in sampling dropout from a more complex distribution? For example, would learning a data-...
- 11
1
vote
1
answer
751
views
How does dropout work during backpropagation?
I've searched for an answer to this, and read several scientific articles on the subject, but I can't find a practical explanation of how Dropout actually drops nodes in an algorithm.
I've read that ...
- 113
0
votes
0
answers
110
views
Optimal weight decay value in Adam
Is there any rule of thumb while assigning the weight_decay parameter in Adam optimizer?
As in, is it somehow related to (smaller or larger than) the learning rate ...
- 65
1
vote
0
answers
140
views
Higher validation loss after using Dropout
I’m working on a classification problem (500 classes). My NN has 3 fully connected layers, followed by an LSTM layer. I use nn.CrossEntropyLoss() as my loss ...
- 65
0
votes
1
answer
121
views
Dummy variable trap in neural networks and class visualization
Let's say I have data records looking like that: (x1, x2, x3, x4, ..., x100), where each x can be either ...
- 101
1
vote
1
answer
40
views
Is it mandatory to multiply every activation of a layer by droupout factor during testing?
Dropout is a regularization technique used in neural networks. It is useful in preventing overfitting by making a neural network as good as an ensemble system.
In dropout, we switch off $p$ percent of ...
- 3,581
0
votes
1
answer
103
views
Can we Consider Regularization as a "Constraint"?
I have the following question on "Regularization vs. Constrained Optimization" :
In the context of statistical modelling, we are often taught about "Regularization" as a method of ...
- 269
4
votes
2
answers
263
views
How does Regularization Reduce Overfitting?
As I understand, this is the general summary of the Regularization-Overfitting Problem:
The classical "Bias-Variance Tradeoff" suggests that complicated models (i.e. models with more ...
- 269
2
votes
0
answers
55
views
What determines when Dropout, BatchNorm & other Regularization will be effective?
I just had a very strange experience where I was training an 8 layer deep & pretty wide (max: 512 neurons) neural network for a regression task. I had assumed since it was big enough that it would ...
- 360
1
vote
1
answer
102
views
What does it mean when accuracy of regularized model is higher for training set than for validation set?
Accuracy of my regularized model is higher for training set than for validation set.
The situation improves when regularization coeefficient is reduced:
What does this really imply?
From my ...
- 111
3
votes
2
answers
4k
views
Should I apply normalization to the observations in deep reinforcement learning?
I am new to DRL and trying to implement my custom environment. I want to know if normalization and regularization techniques are as important in RL as in Deep Learning.
In my custom environment, the ...
- 31
0
votes
0
answers
476
views
Why is BatchNormalization causing severe overfitting to my data?
So I've been making a mini version of VGGNet, trying to tweak the hyperparameters to match the CIFAR-100 dataset.
It was running slow at first but I was able to get decent accuracy after 60 epochs or ...
2
votes
2
answers
482
views
Does regularization just mean using an augmented loss function?
We need to use a loss function for training the neural networks.
In general, the loss function depends only on the desired output $y$ and actual output $\hat{y}$ and is represented as $L(y, \hat{y})$.
...
- 3,581
2
votes
1
answer
280
views
Is the dropout technique specific only to neural networks?
In one Udemy course was mentioned that "dropout is unique to neural networks". However, I remember an example of decision trees where nodes that are not participating in the overall result ...
- 25
1
vote
0
answers
42
views
How to prove that a regularisation method simplified a neural network?
There are a few ways to regularise a neural network, for example dropout or the L1. Now, both these methods, and possibly most other regularisation methods, tend to remove from, or simplify the neural ...
- 206
3
votes
1
answer
117
views
Does adding a model complexity penalty to the loss function allow you to skip cross-validation?
It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of ...
- 33
1
vote
0
answers
58
views
Are there regularisation methods related only to architecture of the CNNs?
Are there any methods of regularisation of deep neural networks, particularly CNNs (or generally ANN but that will also work on CNNs) that are related only to the network's architecture and not the ...
- 1,240
1
vote
0
answers
49
views
If one of the inputs to a neural network (that represents a policy) is noisy and degrades the performance, would this architecture solve the issue?
I'm using genetic algorithms to train deep reinforcement learning (DRL) agents, similarly to what was done in this paper. DRL policies are therefore represented by deep neural networks, which map ...
- 229
7
votes
3
answers
8k
views
How should we regularize an LSTM model?
There are five parameters from an LSTM layer for regularization if I am correct.
To deal with overfitting, I would start with
reducing the layers
reducing the hidden units
Applying dropout or ...
- 123
10
votes
1
answer
4k
views
Can someone explain R1 regularization function in simple terms?
I'm trying to understand the R1 regularization function, both the abstract concept and every symbol in the formula.
According to the article, the definition of R1 is:
It penalizes the discriminator ...
- 211
4
votes
0
answers
62
views
When is using weight regularization bad?
Regularization of weights (e.g. L1 or L2) keeps them small and standardized, which can help reduce data overfitting. From this article, regularization sounds favorable in many cases, but is it always ...
- 753
1
vote
0
answers
30
views
How significant is the decoder part of the capsule network?
Capsule Networks use an encoder-decoder structure, where the encoder part consists of the capsule layers (PrimiaryCaps and DigitCaps) and is also the part of the capsule network which performs the ...
- 111
1
vote
1
answer
634
views
What are the conceptual differences between regularisation and optimisation in deep neural nets?
I'm trying to wrap my mind around the concepts of regularisation and optimisation in neural nets, especially around their differences.
In my current understanding, regularisation is intended to tackle ...
3
votes
0
answers
57
views
Enforcing sparsity constraints that make use of spatial contiguity
I have a deep learning network that outputs grayscale image reconstructions. In addition to good reconstruction performance (measured through mean squared error or some other measure like psnr), I ...
- 143
2
votes
1
answer
705
views
Why L2 loss is more commonly used in Neural Networks than other loss functions?
Why L2 loss is more commonly used in Neural Networks than other loss functions?
What is the reason to L2 being a default choice in Neural Networks?
- 69
2
votes
0
answers
86
views
Forcing a neural network to be close to a previous model - Regularization through given model
I'm wondering, has anyone seen any paper where one trains a network but biases it to produce similar outputs to a given model (such as one given from expert opinion or it being a previously trained ...
- 535
2
votes
1
answer
87
views
When would bias regularisation and activation regularisation be necessary?
For Keras on TensorFlow, a layer class constructor comes with these:
kernel_regularizer=...
bias_regularizer=...
...
- 1,283
0
votes
1
answer
85
views
Where is L2-regularization term applied
I have a confusion on where exactly is the L2 regularization (weight decay) is added.
In various resources I have come across, I find two equations where L2 regularization is applied.
Adding R(W) to ...
- 159
4
votes
1
answer
139
views
Why does L1 regularization yield sparse features?
In contrast to L2 regularization, L1 regularization usually yields sparse feature vectors and most feature weights are zero.
What's the reason for the above statement - could someone explain it ...
- 1,121
2
votes
1
answer
108
views
Does L1/L2 Regularization help reach an optimum result faster?
I understand that L1 and L2 regularization helps to prevent overfitting. My question is then, does that mean they also help a neural network learn faster as a result?
The way I'm thinking is that ...
- 233
0
votes
0
answers
37
views
Cannot fine-tune L2-regularization parameter
I have a data set of 1600 examples. I am using 1280 (80%) for training, 160 (10%) for testing, and 160 (10%) for validation. The training goes one of two ways no matter how I fine-tune the L2 ...
- 133
1
vote
1
answer
157
views
What is relation between gradient descent and regularization in deep learning?
Gradient descent is used to reduce the loss and regularization is used to fight over-fitting.
Is there any relation between gradient descent and regularization, or both are independent of each other?...
- 1,583
0
votes
0
answers
268
views
Derivation of regularized cost function w.r.t activation and bias
In regularzied cost function a L2 regularization cost has been added.
Here we have already calculated cross entropy cost w.r.t $A, W$.
As mentioned in the regularization notebook (see below) in ...
- 151
3
votes
0
answers
93
views
Are there principled ways of tuning a neural network in case of overfitting and underfitting?
Whenever I tune my neural network, I usually take the common approach of defining some layers with some neurons.
If it overfits, I reduce the layers, neurons, add dropout, utilize regularisation.
...
- 151
2
votes
0
answers
86
views
How is the gradient with respect to weights derived in batch normalization?
At the bottom of page 2 of the paper L2 Regularization versus Batch and Weight Normalization, the equation for the gradient of the output with respect to the weights is given as:
$$
\triangledown y_{...
- 21
4
votes
1
answer
87
views
Is there a way to ensure that my model is able to recognize an unseen example?
My question is more theoretical than practical. Let's say that I am training my cat classifier with a dataset that I feel is pretty representative of cat images in general. But then a new breed of cat ...
- 41
10
votes
3
answers
482
views
Are there any rules of thumb for having some idea of what capacity a neural network needs to have for a given problem?
To give an example. Let's just consider the MNIST dataset of handwritten digits. Here are some things which might have an impact on the optimum model capacity:
There are 10 output classes
The inputs ...
- 1,319
1
vote
1
answer
262
views
What is the difference between TensorFlow's callbacks and early stopping?
What is the difference between TensorFlow's callbacks and early stopping?
- 47
3
votes
1
answer
152
views
How do I poison an SVM with manifold regularization?
I'm working on Adversarial Machine Learning, and have read multiple papers on this topic, some of them are mentioned as follows:
Poisoning Attacks on SVMs: https://arxiv.org/pdf/1206.6389.pdf
...
- 125
3
votes
1
answer
123
views
Which is a better form of regularization: lasso (L1) or ridge (L2)?
Given a ridge and a lasso regularizer, which one should be chosen for better performance?
An intuitive graphical explanation (intersection of the elliptical contours of the loss function with the ...
- 308
1
vote
0
answers
174
views
Do L2 regularization and input normalization depend on sigmoid activation functions?
Following the online courses with Andrew Ng, he talks about L2 regularization (a.k.a. weight decay) and input normalization. Now, the argument is that L2 regularization make the weights smaller, ...
- 224
2
votes
0
answers
34
views
Regularization of non-linear parameters?
I was wondering whether it is possible to regularize (L1 or L2) non-linear parameters in a general regression model. Say, I have the following non-linear least squares cost function, where $p$ is a $...
- 21
3
votes
1
answer
407
views
Can dropout layers not influence LSTM training?
I am working on a project that requires time-series prediction (regression) and I use LSTM network with first 1D conv layer in Keras/TF-gpu as follows:
...
- 1,240