1
$\begingroup$

I wonder how to initialize the weights of a layer with non-negative weight constraints and sigmoid activation afterwards. I did not find any guidelines. I thought about taking inspiration of glorotUniform initialization and using $\text{Uniform}[0,b]$ with $b=2 \sqrt{6 / (n_{in} + n_{out})}$ where $n_{in}$ and $n_{out}$ are the number of input and output units respectively. Another idea I had was choosing $a=\frac{n-12}{2n}$, $b=1-\frac{n-12}{2n}$ where $n$ is the number of input units and sampling out of $\text{Uniform}[a,b]$ such that the variance of each output variable is equal to $1$.

My question: Are there any guidelines how to initialize non-negative weights? If no, do you think my suggestions run in problems?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Can you provide a bit more detail on why your weights have a non-negative constraint? Is that constraint enforced during optimization? Or is it only during initialization? What is your reason for wanting this? $\endgroup$
    – Robin van Hoorn
    Jan 13 at 14:19
  • $\begingroup$ The input and output lives in a cone, i.e. multiplication with positive values stays in the cone as well as addition of two cone elements. In some layers I use specific cone-operations which require that the input is a cone element. Non-negative weight constraints can be easily implemented using e.g. constraints in tensorflow. Problem is that I don't know about initialization guidelines. $\endgroup$
    – Philipp123
    Jan 13 at 14:28
  • 1
    $\begingroup$ I'm also curious: Do you have a reason to believe that it will benefit training to keep the weights $\gt 0$? To me it would make more sense to only constrain the network output like this. Regarding the question, I don't know about any guidelines, but I think your idea to augment the glorot uniform like this makes sense. Except that I'd choose $b = 0.5 \sqrt{6 / (n_{in} + n_{out})}$ to keep the layer output at a similar magnitude. Multiplying by two would yield outputs that are four times larger compared to the standard glorot-uniform-initialized layer, which might harm training. $\endgroup$
    – Chillston
    Jan 14 at 19:58
  • $\begingroup$ @Chillston Thanks for the input, yes, positivity was more or less theoretically motivated because I'm working in the cone of positive definite symmetrical matrices and I thought about the values staying in it all the time but I also might try with negative values $\endgroup$
    – Philipp123
    Jan 15 at 14:06

1 Answer 1

Reset to default
1
$\begingroup$

I, unfortunately, cannot provide you with a scientifically based answer, so I'll try to logic my way to an answer.

I know that there are things called 'nonnegativity-constrained autoencoders'. I do not know the details of their initialization but you could look into those for possible guidelines. There are possibly also other networks which have papers published talking about a 'nonnegativity-constrained' variant which I am unaware of.

Furthermore, I would have some concerns/thoughts about nonnegative neural nets:

  1. I would assume that having only positive network weights could lead to an exploding gradient. If anything, I would scale the input in a similar range as your weight initialization and have the upper bound of your weight initialization not exceed 1.
  2. Logically, but also building further on point 1, I would not use (variants of) ReLu activations. With non-negative constrained weights and non-negative inputs, ReLu becomes a simple linear activation. I would advise against any activation function without an upper limit, as it might again result in exploding weights/gradients.
  3. I would start testing the model with sigmoid activations in all layers (except possibly the last, of course).
  4. Xavier/glorot initialization is usually best for layers that also use negative weights and a tanh activation function. I do not know how a nonnegative weight constraint changes its effectiveness. However, if you centre your input around (i.e.) 0.5, taking a uniform distribution around 0.5 and vary its range depending on the layer size might work.

Again these are just my insights, not necessarily based on scientific research.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .