Is it suitable to find inverse of last layer's activation function and apply it on the target output?

Question

I have a neural network with the following structure:

I am expecting specific outputs from the neural network which are the target values for my training. Let's say the target values are 0.8 for the upper output node and -0.3 for the lower output node.

The activations function used for the first 2 layers are ReLu or LeakyReLu while the last layer uses atan as an activation function.

For back propogation, instead of adjusting values to make the network's output approach 0.8,-0.3. is it suitable if I use the inverse function for atan -which is tan itself- to get "the ideal input to the output layer multiplied by weights and adjusted by biases".

The tan of 0.8 and -0.3 is 0.01396 and -0.00524 approximately.

My algorithm would then adjust weights and biases of the network so that the "pre-activated output" of the output layer -which is basically (sum(output_layer_weight*output_layer's inputs)+output_layer_biases)- approaches 0.01396 and -0.00524.

Is this suitable

Answer 1

For the above stated artificial network, these two training scenarios are similar.

• Training to converge to the ideal output vector at the point after the last layer's activation functions are applied
• Training to converge to the vector formed by applying $tan$ functions to each component of the ideal output vector, when convergence occurs at the point just after the vector-matrix multiplication with the last layer's parameters, prior to the last layer's $atan$ activation functions

Distinctions between them include these.

• The applications of gradients and associated code must be adapted to the modification of the starting point of back propagation to before the final $atan$ activation functions.
• The slope and curvature of the loss function will differ if the same loss function is used for the two scenarios, so the accuracy, speed, and reliability of convergence will also be different.

Answer 2

The main difference your change would have is to allow you to apply a loss function to a different part of the network. This may affect training.

If you keep the same loss function (e.g. MSE), but apply it to the pre-transformed values, then you will have changed the objective of the network, perhaps significantly. Whether or not this is a good thing depends on how much you needed the original loss function. However, the fact that it would result in a different training target is usually going to be a bad thing if your original training target was correct. This will also be true if you pick a new arbitrary loss function that seems to fit the pre-transform representation better.

If you engineer a "correct" loss function such that the objective of the network remains unchanged, then the behaviour of the network will not change much - probably not at all. However, in some cases this can lead to more stable and/or faster training - it is often used for classifiers to avoid need to use exponentiation, see tf.nn.softmax_cross_entropy_with_logits in TensorFlow, which does exactly this.

Answer 3

I think your idea would work... fine, but I don't necessarily see any advantages to it. I haven't actually tried it (that'd be the best way for you to also see whether it works!), so I'm mostly going by first thoughts and intuition here.

Anyway, what you are essentially doing with your idea is "cutting off" the last layer of the Neural Network from the perspective of your learning algorithm (typically backpropagation). Whatever weights you have between the last hidden layer and the output layer will be fixed to their initial values. The last hidden layer can actually be viewed as an "output" layer, since you also have fixed targets that you want to converge towards for them.

Whether this makes your learning process better/faster/easier, or worse/slower/harder seems to be very much dependent on how the weights between your last hidden layer and your output layer are initialized. For example:

• If those weights are initialized to all-zero, your "real" output layer is doomed to always predict zeros, so your problem becomes impossible to solve.
• If those weights are initialized to implement the identity function, this becomes 100% equivalent to the case you would have if you'd simply cut off the last layer and train that in the traditional sense (i.e. you effectively have one layer less than you really do).
• If those weights are initialized randomly, it looks to me like you have a post-processing step consisting of a random projection. Such a random projection may be beneficial for training (random projections can be useful for dimensionality reduction, or for, in combination with the subsequent non-linear function, turning an otherwise linear function into a non-linear function).

I don't think it'd very often be better than actually have an extra "real", trainable layer with a non-linear activation function though. I suspect such a "non-trainable" extra layer can sometimes be better than not having anything there, but I don't think it'd often be better than having a real, trainable layer.

Answer 4

Be careful to specify exactly what you mean by adjust weights and biases of the network so that the "pre-activated output" ... approaches ....

When training a neural network, one minimizes a loss function. This loss function is determines how important the deviation from 0.01396 is compared to the deviation of the other node from -0.00524. By transforming the target labels backwards you should also express the original loss function in terms of the back-transformed labels.

---

What one can do in some cases is to combine the input to the last layer's activation function with the loss function and algebraically simplify the resulting expression.

This concept is for example implemented in Tensorflow's tf.nn.sigmoid_cross_entropy_with_logits. This function can be used for the case of a single output with sigmoid activation function and binary cross-entropy loss (a similar function also exists for the case of multiple output nodes with a softmax activation function).

Instead of first passing the values through a sigmoid activation and then calculating the binary cross-entropy with respect to the target label, it combines the two expressions and uses an equivalent but simpler expression.

If you look at the documentation of this function, you'll see that the number of transcendental functions (which are computationally expensive to calculate) can be reduced from

loss(x,z) = z * -log(1 / (1 + exp(-x))) + (1 - z) * -log(exp(-x) / (1 + exp(-x)))

to

loss(x,z) = max(x, 0) - x * z + log(1 + exp(-abs(x)))

where x is the input to the (sigmoid) nonlinearity (corresponding to the output of the green nodes in your diagram), also called 'logits' and z is the target label.