Questions tagged [regularization]

For questions about application of regularization techniques.

Filter by
Sorted by
Tagged with
2
votes
1answer
83 views

Why L2 loss is more commonly used in Neural Networks than other loss functions?

Why L2 loss is more commonly used in Neural Networks than other loss functions? What is the reason to L2 being a default choice in Neural Networks?
0
votes
0answers
44 views

Non-trainable regularizer in loss function

I train a fully convoluted network for semantic segmentation. To each convolution blocks, I associate a module pruning feature maps to reduce the quantity of information generated by the network. From ...
4
votes
0answers
75 views

Forcing a neural network to be close to a previous model - Regularization through given model

I'm wondering, has anyone seen any paper where one trains a network but biases it to produce similar outputs to a given model (such as one given from expert opinion or it being a previously trained ...
1
vote
1answer
45 views

When would bias regularisation and activation regularisation be necessary?

For Keras on TensorFlow, a layer class constructor comes with these: kernel_regularizer=... bias_regularizer=... ...
0
votes
1answer
49 views

Where is L2-regularization term applied

I have a confusion on where exactly is the L2 regularization (weight decay) is added. In various resources I have come across, I find two equations where L2 regularization is applied. Adding R(W) to ...
1
vote
1answer
49 views

Why does L1 regularization yield sparse features?

In contrast to L2 regularization, L1 regularization usually yields sparse feature vectors and most feature weights are zero. What's the reason for the above statement - could someone explain it ...
2
votes
0answers
32 views

Does L1/L2 Regularization help reach an optimum result faster?

So I understand that L1 and L2 regularization helps to prevent overfitting. My question is then, does that mean they also help a neural network learn faster as a result? The way I'm thinking is that ...
0
votes
0answers
26 views

Cannot fine-tune L2-regularization parameter

I have a data set of 1600 examples. I am using 1280 (80%) for training, 160 (10%) for testing, and 160 (10%) for validation. The training goes one of two ways no matter how I fine-tune the L2 ...
1
vote
1answer
69 views

If the i.i.d. assumption holds, shouldn't the training and validation trends be exactly the same?

If the i.i.d. (independent and identically distributed) assumption holds for a training-validation set pair, shouldn't their loss trends be exactly the same, since every batch from the validation set ...
1
vote
1answer
50 views

What is relation between gradient descent and regularization in deep learning?

Gradient descent is used to reduce the loss and regularization is used to fight over-fitting. Is there any relation between gradient descent and regularization, or both are independent of each other?...
0
votes
0answers
53 views

Derivation of regularized cost function w.r.t activation and bias

In regularzied cost function a L2 regularization cost has been added. Here we have already calculated cross entropy cost w.r.t $A, W$. As mentioned in the regularization notebook (see below) in ...
3
votes
0answers
79 views

Are there principled ways of tuning a neural network in case of overfitting and underfitting?

Whenever I tune my neural network, I usually take the common approach of defining some layers with some neurons. If it overfits, I reduce the layers, neurons, add dropout, utilize regularisation. ...
2
votes
0answers
30 views

How is the gradient with respect to weights derived in batch normalization?

This is a cross-post, as I didn't get any answers on Stats SE and I am hoping that it gets more attention here. At the bottom of page 2 of the paper L2 Regularization versus Batch and Weight ...
4
votes
1answer
63 views

Is there a way to ensure that my model is able to recognize an unseen example?

My question is more theoretical than practical. Let's say that I am training my cat classifier with a dataset that I feel is pretty representative of cat images in general. But then a new breed of cat ...
0
votes
1answer
34 views

What is the difference between TensorFlow's callbacks and early stopping?

What is the difference between TensorFlow's callbacks and early stopping?
3
votes
1answer
96 views

How do I poison an SVM with manifold regularization?

I'm working on Adversarial Machine Learning, and have read multiple papers on this topic, some of them are mentioned as follows: Poisoning Attacks on SVMs: https://arxiv.org/pdf/1206.6389.pdf ...
0
votes
0answers
19 views

Tensorflow neural network - inherent overfitting in certain X and Y distributions?

Really open ended question here... I'm working on a classifier model that is used to predict whether some amount is > x, ...
3
votes
1answer
74 views

Which is a better form of regularization: lasso (L1) or ridge (L2)?

Given a ridge and a lasso regularizer, which one should be chosen for better performance? An intuitive graphical explanation (intersection of the elliptical contours of the loss function with the ...
1
vote
0answers
24 views

Do L2 regularization and input normalization depend on sigmoid activation functions?

Following the online courses with Andrew Ng, he talks about L2 regularization (a.k.a. weight decay) and input normalization. Now, the argument is that L2 regularization make the weights smaller, ...
2
votes
0answers
23 views

Regularization of non-linear parameters?

I was wondering whether it is possible to regularize (L1 or L2) non-linear parameters in a general regression model. Say, I have the following non-linear least squares cost function, where $p$ is a $...
3
votes
1answer
82 views

Can dropout layers not influence LSTM training?

I am working on a project that requires time-series prediction (regression) and I use LSTM network with first 1D conv layer in Keras/TF-gpu as follows: ...
3
votes
1answer
85 views

What is the $\ell_{2, 1}$ norm?

I'm reading this paper and it says: In this paper, we present a multi-class embedded feature selection method called as sparse optimal scoring with adjustment (SOSA), which is capable of addressing ...
6
votes
2answers
145 views

Why is dropout favoured compared to reducing the number of units in hidden layers?

Why is dropout favored compared to reducing the number of units in hidden layers for the convolutional networks? If a large set of units leads to overfitting and dropping out "averages" the response ...
3
votes
0answers
19 views

L1 Reguarizer in Keras model throwing weight matrix dimension error

Was just experimenting with something when i ran into this error : I am getting matrix dimension errors only when i use L1 Regularizer. I have checked and the regularizer itself doesn't change the ...
1
vote
1answer
80 views

Dropout causes too much noise for network to train

I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly ...
1
vote
0answers
50 views

What is the intuition behind the Label Smoothing?

I was learning about GAN when the term "Label Smoothing" appears. From the video tutorial that I watch, they use the term "label smoothing" to change the binary labels when calculating the loss of ...
2
votes
0answers
28 views

What is the benefit of scaling the hyperparameter C of an SVM?

Please read the following page of the Sklearn documentation. The figure shown there (see below) illustrates why C should be scaled when using a SVM with 'l1' penalty, whereas it shouldn't be scaled ...
1
vote
0answers
27 views

How can I model regularity?

I have data that are a result of rules that are exceptionless. I want to my program to 'look' at my data and figure out those rules. However, the data might contain what might look like an exception (...
2
votes
1answer
47 views

Should I remove the units of a neural network or increase dropout?

When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is ...
2
votes
0answers
66 views

Regarding L0 sparsification of DNNs proposed by Louizos, Kingma and Welling

I am reading the paper on $\ell_0$ regularization of DNNs by Louizos, Welling and Kingma (2017) (Link to arxiv). In Section 2.1 the authors define the cost function as follows: $$ \mathcal{R}\left( \...
6
votes
3answers
3k views

Why L1/L2 regularization technique did not improve my accuracy?

I am training a Multilayer Neural Nets with 146 samples (97 for training set, 20 for validation set and 29 for testing set). I am using: automatic differentiation, SGD method, fixed learning rate + ...
4
votes
1answer
317 views

How does L2 regularization make weights smaller?

I'm learning the Logistic Regression and L2 Regularization. The cost function looks like below. $$J(w) = -\displaystyle\sum_{i=1}^{n} (y^{(i)}\log(\phi(z^{(i)})+(1-y^{(i)})\log(1-\phi(z^{(i)})))$$ ...
8
votes
1answer
435 views

What is “early stopping” in machine learning?

What is early stopping in machine learning and, in general, artificial intelligence? What are the advantages of using this method? How does it help exactly? I'd be interested in perspectives and ...