Questions tagged [weight-normalization]
The weight-normalization tag has no usage guidance.
6
questions
0
votes
0
answers
21
views
What is the right way to normalize the initial weights of a fully connected layer using a SiLu (Sigmoid-weighted Linear Unit) activation function?
I've been writing a deep learning Java framework as a way for myself to learn how it all works and I have had a decent amount of success so far. Best performance is just over 90% accuracy with three ...
0
votes
1
answer
87
views
Why and when do we need to normalize weights in Reinforcement Learning?
I recently came across this SO question, wherein the poster was asked to normalize their weights while using a function approximator with SARSA. I don't remember having to normalize any weights while ...
- 416
3
votes
0
answers
162
views
Are there neural networks with (hard) constraints on the weights?
I don't know too much about Deep Learning, so my question might be silly. However, I was wondering whether there are NN architectures with some hard constraints on the weights of some layers. For ...
- 173
3
votes
1
answer
224
views
Can you help me understand how weight normalization works?
I am trying to dissect the paper Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks.
Unfortunately, because my math is a little bit rusty, I got a little ...
- 43
7
votes
1
answer
287
views
How to solve the problem of too big activations when using genetic algorithms to train neural networks?
I am trying to create a fixed-topology MLP from scratch (with C#), which can solve some simple problems, such as the XOR problem and MNIST classification. The network will be trained purely with ...
- 73
9
votes
1
answer
5k
views
How does weight normalization work?
I was reading the paper Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks about improving the learning of an ANN using weight normalization.
They ...
- 145