Questions tagged [weight-normalization]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
21 views

What is the right way to normalize the initial weights of a fully connected layer using a SiLu (Sigmoid-weighted Linear Unit) activation function?

I've been writing a deep learning Java framework as a way for myself to learn how it all works and I have had a decent amount of success so far. Best performance is just over 90% accuracy with three ...
0 votes
1 answer
87 views

Why and when do we need to normalize weights in Reinforcement Learning?

I recently came across this SO question, wherein the poster was asked to normalize their weights while using a function approximator with SARSA. I don't remember having to normalize any weights while ...
3 votes
0 answers
162 views

Are there neural networks with (hard) constraints on the weights?

I don't know too much about Deep Learning, so my question might be silly. However, I was wondering whether there are NN architectures with some hard constraints on the weights of some layers. For ...
  • 173
3 votes
1 answer
224 views

Can you help me understand how weight normalization works?

I am trying to dissect the paper Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. Unfortunately, because my math is a little bit rusty, I got a little ...
7 votes
1 answer
287 views

How to solve the problem of too big activations when using genetic algorithms to train neural networks?

I am trying to create a fixed-topology MLP from scratch (with C#), which can solve some simple problems, such as the XOR problem and MNIST classification. The network will be trained purely with ...
9 votes
1 answer
5k views

How does weight normalization work?

I was reading the paper Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks about improving the learning of an ANN using weight normalization. They ...
  • 145