How do we design a neural network such that the $L_1$ norm of the outputs is less than or equal to 1?

Question

What are some ways to design a neural network with the restriction that the $L_1$ norm of the output values must be less than or equal to 1? In particular, how would I go about performing back-propagation for this net?

I was thinking there must be some "penalty" method just like how in the mathematical optimization problem, you can introduce a log barrier function as the "penalty function"

Answer 1

Penalty (barrier function) is perfectly valid and simplest method for simplex type constraint (L1 norm is simplex constraint on absolute values). Any type of barrier function may work, logarithmic, reciprocal or quadratic. All of them supported by any major framework(pytorch, tensorflow), just add them to loss function. You would need some hyperparameter tuning for the scale factor of penalty.

There is more efficient, though more complex way to do it. Instead of putting constraint you can automatically output value wich satisfy simplex constraint:

Assume that L1 norm constraint is $\left \|v\right \|_1 \leq 1$, $v \in \mathbb{R}^n$

1. put $sigmoid(v_i)$ activation on output to norm elements to [-1, 1]
2. add slack (fake) variable element $v_{n+1} = 1 - \sum_{1}^{n} v_i $
3. project new $v{}'\in \mathbb{R}^{n+1}$, $v{}'_i = |v_i|,1\leq i \leq n+1$ onto unit simplex with standard algorithm (also here)

Backpropagation of last step may require differentiable sorting, which is missing in most of frameworks, you may have to look for open sourced implementation, for example extract it from here or use some automatic differentiation package. Both require some substantial code reading/debugging. However in my experience assuming constant $\Delta$ also works in many cases, in that case differentiable sorting is not needed. Intuition behind constant $\Delta$ is that $\Delta$ could be chosen such way that there is some interval on which it's value doesn't affect sorting order.

Answer 2

The term 'size' isn't applicable to the tensor output of a network or network. These are the qualities.

• Rank $N$ that defines the rank of each tensor instance in $\mathbb{R}^N$
• Ranges of the indices to the dimensions from $1$ to $N$
• Ranges of the scalar values that comprise the tensor instance — If they are discrete rather than real (approximated by floating point or fixed point numbers), then the range is the description of the permissible ordinal values.

The question may be referring to this last quality.

The imposition of a penalty for values that are in the range of the numeric type used as the output of the last activation function but not in the allowable range of output for the desired trained function works in a limited way. It skews the output distribution with respect to the natural distribution of possible learning states and therefore can easily interfere with convergence quality or speed or both.

There are a number of techniques that map natural output distribution with constrained ranges, but it must be done without skewing the distribution upstream from the technique used, to avoid negatively impacting favorable convergence properties of the artificial network.

One simple case that can be described here is when the number of possible output states is in the set of $2^i$ where $i \in$ { positive integers }. In such a case, the final layer of the network can be $i$ threshold activation functions with 1 or -1 as possible output values.

In that case, the ordinal then becomes

$$o = \sum_{n=0}^i \frac {y_n + 1} {2},$$

where $y_n$ is the output from activation function index $n$.