0
$\begingroup$

I just came accros the "Selu" function. It looks promising, especially with the header "self normalizing neural network", since I have some serious with exploding gradient within my network with (leaky) Relu. Personally I wasn't able to implement Batch Norm, so I hope this Selu could help me.

So I have two main questions:

  1. Does Selu work like Batch Norm, as in normalizing the network, thus avoiding exploding gradients

  2. How do I implement Selu in my Network?

So I am talking about this from the tensorflow documentation: https://www.tensorflow.org/api_docs/python/tf/nn/selu

This is how my Network currently looks like:

net = tf.layers.dense(s, 64, activation=tf.nn.leaky_relu, name='l1', trainable=trainable)
mid = tf.layers.dense(net, 33, activation=tf.nn.tanh, name='l2', trainable=trainable)
a = tf.layers.dense(mid, 3, activation=tf.nn.softmax, name='la', trainable=trainable)

So with Selu I am expecting it to look more like:

net = tf.layers.dense(s, 64, activation=tf.nn.selu, name='l1', trainable=trainable)
mid = tf.layers.dense(net, 33, activation=tf.nn.tanh, name='l2', trainable=trainable)
a = tf.layers.dense(mid, 3, activation=tf.nn.softmax, name='la', trainable=trainable)

Simple, right? however I know I am missing something as stated on their documentation:

To be used together with initializer = tf.variance_scaling_initializer(factor=1.0, mode='FAN_IN')

Here is where my problem occurs, because I have no idea what they mean with this neither if i need to give more information to the network besides only this line? I also saw you had to do something with the n_states (assuming dimension of state?).

If anyone could help me out, that would be great!

Jan

$\endgroup$
  • $\begingroup$ Implementation questions belong on Data Science. $\endgroup$ – Philip Raeisghasem Mar 13 at 8:43
  • $\begingroup$ Welcome to SE:AI. This seems like 2 questions. Q1: "Does Selu work like Batch Norm, as in normalizing the network, thus avoiding exploding gradients" should be suitable for this stack, but Q2: "How do I implement Selu in my Network?" is better suited toward Data Science or Overflow. Leaving as-is for now. My recommendation is to ask the first question here, and the implementation question on the other recommended stacks. $\endgroup$ – DukeZhou Mar 13 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.