# Neural network: Initial weights for layer with non-negative constraint

I wonder how to initialize the weights of a layer with non-negative weight constraints and sigmoid activation afterwards. I did not find any guidelines. I thought about taking inspiration of glorotUniform initialization and using $$\text{Uniform}[0,b]$$ with $$b=2 \sqrt{6 / (n_{in} + n_{out})}$$ where $$n_{in}$$ and $$n_{out}$$ are the number of input and output units respectively. Another idea I had was choosing $$a=\frac{n-12}{2n}$$, $$b=1-\frac{n-12}{2n}$$ where $$n$$ is the number of input units and sampling out of $$\text{Uniform}[a,b]$$ such that the variance of each output variable is equal to $$1$$.

My question: Are there any guidelines how to initialize non-negative weights? If no, do you think my suggestions run in problems?

• Can you provide a bit more detail on why your weights have a non-negative constraint? Is that constraint enforced during optimization? Or is it only during initialization? What is your reason for wanting this? Jan 13 at 14:19
• The input and output lives in a cone, i.e. multiplication with positive values stays in the cone as well as addition of two cone elements. In some layers I use specific cone-operations which require that the input is a cone element. Non-negative weight constraints can be easily implemented using e.g. constraints in tensorflow. Problem is that I don't know about initialization guidelines. Jan 13 at 14:28
• I'm also curious: Do you have a reason to believe that it will benefit training to keep the weights $\gt 0$? To me it would make more sense to only constrain the network output like this. Regarding the question, I don't know about any guidelines, but I think your idea to augment the glorot uniform like this makes sense. Except that I'd choose $b = 0.5 \sqrt{6 / (n_{in} + n_{out})}$ to keep the layer output at a similar magnitude. Multiplying by two would yield outputs that are four times larger compared to the standard glorot-uniform-initialized layer, which might harm training. Jan 14 at 19:58
• @Chillston Thanks for the input, yes, positivity was more or less theoretically motivated because I'm working in the cone of positive definite symmetrical matrices and I thought about the values staying in it all the time but I also might try with negative values Jan 15 at 14:06