Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [regularization]

For questions about application of regularization techniques.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

Why does loss function and constraint touches at corner point in lasso regression?

As you can see in the picture, for two co-efficient w1 and w2, our loss function, f(w1, w2) should be minimized under constraint function g(w1, w2). For lasso regularization minimum point always lies ...
user avatar
0 votes
1 answer
38 views

Can we Consider Regularization as a "Constraint"?

I have the following question on "Regularization vs. Constrained Optimization" : In the context of statistical modelling, we are often taught about "Regularization" as a method of ...
user avatar
4 votes
2 answers
155 views

How does Regularization Reduce Overfitting?

As I understand, this is the general summary of the Regularization-Overfitting Problem: The classical "Bias-Variance Tradeoff" suggests that complicated models (i.e. models with more ...
user avatar
2 votes
0 answers
50 views

What determines when Dropout, BatchNorm & other Regularization will be effective?

I just had a very strange experience where I was training an 8 layer deep & pretty wide (max: 512 neurons) neural network for a regression task. I had assumed since it was big enough that it would ...
user avatar
  • 266
0 votes
0 answers
17 views

Why does conditioning neural network function on adjacency matrix of graph allow for distribution of gradient information from the supervised loss?

I was reading the following paper here and had a question about the paragraph on page 1 (in the introduction). The equation being referred to is: $$ \mathcal{L} = \mathcal{L}_0 + \lambda \mathcal{L}_{\...
user avatar
1 vote
1 answer
72 views

What does it mean when accuracy of regularized model is higher for training set than for validation set?

Accuracy of my regularized model is higher for training set than for validation set. The situation improves when regularization coeefficient is reduced: What does this really imply? From my ...
user avatar
1 vote
2 answers
872 views

Should I apply normalization to the observations in deep reinforcement learning?

I am new to DRL and trying to implement my custom environment. I want to know if normalization and regularization techniques are as important in RL as in Deep Learning. In my custom environment, the ...
user avatar
  • 11
0 votes
0 answers
121 views

Why is BatchNormalization causing severe overfitting to my data?

So I've been making a mini version of VGGNet, trying to tweak the hyperparameters to match the CIFAR-100 dataset. It was running slow at first but I was able to get decent accuracy after 60 epochs or ...
user avatar
2 votes
2 answers
178 views

Does regularization just mean using an augmented loss function?

We need to use a loss function for training the neural networks. In general, the loss function depends only on the desired output $y$ and actual output $\hat{y}$ and is represented as $L(y, \hat{y})$. ...
user avatar
  • 3,059
2 votes
1 answer
241 views

Is the dropout technique specific only to neural networks?

In one Udemy course was mentioned that "dropout is unique to neural networks". However, I remember an example of decision trees where nodes that are not participating in the overall result ...
user avatar
0 votes
0 answers
31 views

Incorporating regularization for the kernel perceptron

To my understanding, the following is how the kernel perceptron works.    Kernel perceptron algorithm       The parameters to be calculated are $\alpha = \begin{pmatrix} \alpha_1 &\ldots &\...
user avatar
1 vote
0 answers
36 views

How to prove that a regularisation method simplified a neural network?

There are a few ways to regularise a neural network, for example dropout or the L1. Now, both these methods, and possibly most other regularisation methods, tend to remove from, or simplify the neural ...
user avatar
  • 216
3 votes
1 answer
48 views

Does adding a model complexity penalty to the loss function allow you to skip cross-validation?

It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of ...
user avatar
  • 33
1 vote
0 answers
47 views

Are there regularisation methods related only to architecture of the CNNs?

Are there any methods of regularisation of deep neural networks, particularly CNNs (or generally ANN but that will also work on CNNs) that are related only to the network's architecture and not the ...
user avatar
  • 1,027
1 vote
0 answers
33 views

If one of the inputs to a neural network (that represents a policy) is noisy and degrades the performance, would this architecture solve the issue?

I'm using genetic algorithms to train deep reinforcement learning (DRL) agents, similarly to what was done in this paper. DRL policies are therefore represented by deep neural networks, which map ...
user avatar
  • 129
7 votes
3 answers
3k views

How should we regularize an LSTM model?

There are five parameters from an LSTM layer for regularization if I am correct. To deal with overfitting, I would start with reducing the layers reducing the hidden units Applying dropout or ...
user avatar
  • 123
5 votes
1 answer
2k views

Can someone explain R1 regularization function in simple terms?

I'm trying to understand the R1 regularization function, both the abstract concept and every symbol in the formula. According to the article, the definition of R1 is: It penalizes the discriminator ...
user avatar
4 votes
0 answers
52 views

When is using weight regularization bad?

Regularization of weights (e.g. L1 or L2) keeps them small and standardized, which can help reduce data overfitting. From this article, regularization sounds favorable in many cases, but is it always ...
user avatar
  • 663
1 vote
0 answers
20 views

How significant is the decoder part of the capsule network?

Capsule Networks use an encoder-decoder structure, where the encoder part consists of the capsule layers (PrimiaryCaps and DigitCaps) and is also the part of the capsule network which performs the ...
user avatar
1 vote
1 answer
60 views

What are the conceptual differences between regularisation and optimisation in deep neural nets?

I'm trying to wrap my mind around the concepts of regularisation and optimisation in neural nets, especially around their differences. In my current understanding, regularisation is intended to tackle ...
user avatar
3 votes
0 answers
45 views

Enforcing sparsity constraints that make use of spatial contiguity

I have a deep learning network that outputs grayscale image reconstructions. In addition to good reconstruction performance (measured through mean squared error or some other measure like psnr), I ...
user avatar
2 votes
1 answer
198 views

Why L2 loss is more commonly used in Neural Networks than other loss functions?

Why L2 loss is more commonly used in Neural Networks than other loss functions? What is the reason to L2 being a default choice in Neural Networks?
user avatar
2 votes
0 answers
85 views

Forcing a neural network to be close to a previous model - Regularization through given model

I'm wondering, has anyone seen any paper where one trains a network but biases it to produce similar outputs to a given model (such as one given from expert opinion or it being a previously trained ...
user avatar
2 votes
1 answer
57 views

When would bias regularisation and activation regularisation be necessary?

For Keras on TensorFlow, a layer class constructor comes with these: kernel_regularizer=... bias_regularizer=... ...
user avatar
  • 1,153
0 votes
1 answer
64 views

Where is L2-regularization term applied

I have a confusion on where exactly is the L2 regularization (weight decay) is added. In various resources I have come across, I find two equations where L2 regularization is applied. Adding R(W) to ...
user avatar
  • 159
4 votes
1 answer
103 views

Why does L1 regularization yield sparse features?

In contrast to L2 regularization, L1 regularization usually yields sparse feature vectors and most feature weights are zero. What's the reason for the above statement - could someone explain it ...
user avatar
2 votes
1 answer
76 views

Does L1/L2 Regularization help reach an optimum result faster?

I understand that L1 and L2 regularization helps to prevent overfitting. My question is then, does that mean they also help a neural network learn faster as a result? The way I'm thinking is that ...
user avatar
0 votes
0 answers
30 views

Cannot fine-tune L2-regularization parameter

I have a data set of 1600 examples. I am using 1280 (80%) for training, 160 (10%) for testing, and 160 (10%) for validation. The training goes one of two ways no matter how I fine-tune the L2 ...
user avatar
  • 133
1 vote
1 answer
66 views

What is relation between gradient descent and regularization in deep learning?

Gradient descent is used to reduce the loss and regularization is used to fight over-fitting. Is there any relation between gradient descent and regularization, or both are independent of each other?...
user avatar
  • 1,223
0 votes
0 answers
123 views

Derivation of regularized cost function w.r.t activation and bias

In regularzied cost function a L2 regularization cost has been added. Here we have already calculated cross entropy cost w.r.t $A, W$. As mentioned in the regularization notebook (see below) in ...
user avatar
  • 151
3 votes
0 answers
87 views

Are there principled ways of tuning a neural network in case of overfitting and underfitting?

Whenever I tune my neural network, I usually take the common approach of defining some layers with some neurons. If it overfits, I reduce the layers, neurons, add dropout, utilize regularisation. ...
user avatar
  • 151
2 votes
0 answers
49 views

How is the gradient with respect to weights derived in batch normalization?

At the bottom of page 2 of the paper L2 Regularization versus Batch and Weight Normalization, the equation for the gradient of the output with respect to the weights is given as: $$ \triangledown y_{...
user avatar
4 votes
1 answer
69 views

Is there a way to ensure that my model is able to recognize an unseen example?

My question is more theoretical than practical. Let's say that I am training my cat classifier with a dataset that I feel is pretty representative of cat images in general. But then a new breed of cat ...
user avatar
10 votes
3 answers
426 views

Are there any rules of thumb for having some idea of what capacity a neural network needs to have for a given problem?

To give an example. Let's just consider the MNIST dataset of handwritten digits. Here are some things which might have an impact on the optimum model capacity: There are 10 output classes The inputs ...
user avatar
1 vote
1 answer
112 views

What is the difference between TensorFlow's callbacks and early stopping?

What is the difference between TensorFlow's callbacks and early stopping?
user avatar
  • 47
3 votes
1 answer
124 views

How do I poison an SVM with manifold regularization?

I'm working on Adversarial Machine Learning, and have read multiple papers on this topic, some of them are mentioned as follows: Poisoning Attacks on SVMs: https://arxiv.org/pdf/1206.6389.pdf ...
user avatar
3 votes
1 answer
99 views

Which is a better form of regularization: lasso (L1) or ridge (L2)?

Given a ridge and a lasso regularizer, which one should be chosen for better performance? An intuitive graphical explanation (intersection of the elliptical contours of the loss function with the ...
user avatar
  • 188
1 vote
0 answers
83 views

Do L2 regularization and input normalization depend on sigmoid activation functions?

Following the online courses with Andrew Ng, he talks about L2 regularization (a.k.a. weight decay) and input normalization. Now, the argument is that L2 regularization make the weights smaller, ...
user avatar
2 votes
0 answers
26 views

Regularization of non-linear parameters?

I was wondering whether it is possible to regularize (L1 or L2) non-linear parameters in a general regression model. Say, I have the following non-linear least squares cost function, where $p$ is a $...
user avatar
  • 21
3 votes
1 answer
216 views

Can dropout layers not influence LSTM training?

I am working on a project that requires time-series prediction (regression) and I use LSTM network with first 1D conv layer in Keras/TF-gpu as follows: ...
user avatar
  • 1,027
3 votes
1 answer
1k views

What is the $\ell_{2, 1}$ norm?

I'm reading this paper and it says: In this paper, we present a multi-class embedded feature selection method called as sparse optimal scoring with adjustment (SOSA), which is capable of addressing ...
user avatar
  • 133
6 votes
2 answers
391 views

Why is dropout favoured compared to reducing the number of units in hidden layers?

Why is dropout favored compared to reducing the number of units in hidden layers for the convolutional networks? If a large set of units leads to overfitting and dropping out "averages" the response ...
user avatar
1 vote
1 answer
174 views

Dropout causes too much noise for network to train

I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly ...
user avatar
  • 1,324
1 vote
0 answers
103 views

What is the intuition behind the Label Smoothing?

I was learning about GAN when the term "Label Smoothing" appears. From the video tutorial that I watch, they use the term "label smoothing" to change the binary labels when calculating the loss of ...
user avatar
  • 2,511
2 votes
0 answers
33 views

What is the benefit of scaling the hyperparameter C of an SVM?

Please read the following page of the Sklearn documentation. The figure shown there (see below) illustrates why C should be scaled when using a SVM with 'l1' penalty, whereas it shouldn't be scaled ...
user avatar
  • 141
1 vote
0 answers
32 views

How can I model regularity?

I have data that are a result of rules that are exceptionless. I want to my program to 'look' at my data and figure out those rules. However, the data might contain what might look like an exception (...
user avatar
  • 111
2 votes
1 answer
81 views

Should I remove the units of a neural network or increase dropout?

When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is ...
user avatar
  • 121
2 votes
0 answers
76 views

Regarding L0 sparsification of DNNs proposed by Louizos, Kingma and Welling

I am reading the paper on $\ell_0$ regularization of DNNs by Louizos, Welling and Kingma (2017) (Link to arxiv). In Section 2.1 the authors define the cost function as follows: $$ \mathcal{R}\left( \...
user avatar
  • 21
4 votes
2 answers
7k views

Why did the L1/L2 regularization technique not improve my accuracy?

I am training a multilayer neural network with 146 samples (97 for the training set, 20 for the validation set, and 29 for the testing set). I am using: automatic differentiation, SGD method, fixed ...
user avatar
  • 121
4 votes
1 answer
814 views

How does L2 regularization make weights smaller?

I'm learning logistic regression and $L_2$ regularization. The cost function looks like below. $$J(w) = -\displaystyle\sum_{i=1}^{n} (y^{(i)}\log(\phi(z^{(i)})+(1-y^{(i)})\log(1-\phi(z^{(i)})))$$ And ...
user avatar