2
$\begingroup$

Imagine a "simple" feedforward, fully connected neural network, with some input size, some number of hidden layers, and some # of neurons....etc BUT with a fixed number of output size (that is saying, even if we change and optimize the whole structure of the neural network through through cross validation, the output size must be constant),

Then what are some ways of performing neural network given that the L1 norm of the output values is less than or equal to 1? Then, given such methodology, how would I go about performing backpropagation?

I was thinking there must be some "penalty" method just like how in mathematical optimization problem, you can introduce log barrier function as the "penalty function"

$\endgroup$
0
$\begingroup$

The term 'size' isn't applicable to the tensor output of a network or network. These are the qualities.

  • Rank $N$ that defines the rank of each tensor instance in $\mathbb{R}^N$
  • Ranges of the indices to the dimensions from $1$ to $N$
  • Ranges of the scalar values that comprise the tensor instance — If they are discrete rather than real (approximated by floating point or fixed point numbers), then the range is the description of the permissible ordinal values.

The question may be referring to this last quality.

The imposition of a penalty for values that are in the range of the numeric type used as the output of the last activation function but not in the allowable range of output for the desired trained function works in a limited way. It skews the output distribution with respect to the natural distribution of possible learning states and therefore can easily interfere with convergence quality or speed or both.

There are a number of techniques that map natural output distribution with constrained ranges, but it must be done without skewing the distribution upstream from the technique used, to avoid negatively impacting favorable convergence properties of the artificial network.

One simple case that can be described here is when the number of possible output states is in the set of $2^i$ where $i \in$ { positive integers }. In such a case, the final layer of the network can be $i$ threshold activation functions with 1 or -1 as possible output values.

In that case, the ordinal then becomes

$$o = \sum_{n=0}^i \frac {y_n + 1} {2},$$

where $y_n$ is the output from activation function index $n$.

$\endgroup$
0
$\begingroup$

Penalty (barrier function) is perfectly valid and simplest method for simplex type constraint (L1 norm is simplex constraint on absolute values). Any type of barrier function may work, logarithmic, reciprocal or quadratic. All of them supported by any major framework(pytorch, tensorflow), just add them to loss function. You would need some hyperparameter tuning for the scale factor of penalty.

There is more efficient, though more complex way to do it. Instead of putting constraint you can automatically output value wich satisfy simplex constraint:

Assume that L1 norm constraint is $\left \|v\right \|_1 \leq 1$, $v \in \mathbb{R}^n$

  1. put $sigmoid(v_i)$ activation on output to norm elements to [-1, 1]
  2. add slack (fake) variable element $v_{n+1} = 1 - \sum_{1}^{n} v_i $
  3. project new $v{}'\in \mathbb{R}^{n+1}$, $v{}'_i = |v_i|,1\leq i \leq n+1$ onto unit simplex with standard algorithm (also here)

Backpropagation of last step may require differentiable sorting, which is missing in most of frameworks, you may have to look for open sourced implementation, for example extract it from here or use some automatic differentiation package. Both require some substantial code reading/debugging. However in my experience assuming constant $\Delta$ also works in many cases, in that case differentiable sorting is not needed. Intuition behind constant $\Delta$ is that $\Delta$ could be chosen such way that there is some interval on which it's value doesn't affect sorting order.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.