# Selu and how to implement it?

I just came accros the "Selu" function. It looks promising, especially with the header "self normalizing neural network", since I have some serious with exploding gradient within my network with (leaky) Relu. Personally I wasn't able to implement Batch Norm, so I hope this Selu could help me.

So I have two main questions:

1. Does Selu work like Batch Norm, as in normalizing the network, thus avoiding exploding gradients

2. How do I implement Selu in my Network?

This is how my Network currently looks like:

net = tf.layers.dense(s, 64, activation=tf.nn.leaky_relu, name='l1', trainable=trainable)
mid = tf.layers.dense(net, 33, activation=tf.nn.tanh, name='l2', trainable=trainable)
a = tf.layers.dense(mid, 3, activation=tf.nn.softmax, name='la', trainable=trainable)


So with Selu I am expecting it to look more like:

net = tf.layers.dense(s, 64, activation=tf.nn.selu, name='l1', trainable=trainable)
mid = tf.layers.dense(net, 33, activation=tf.nn.tanh, name='l2', trainable=trainable)
a = tf.layers.dense(mid, 3, activation=tf.nn.softmax, name='la', trainable=trainable)


Simple, right? however I know I am missing something as stated on their documentation:

To be used together with initializer = tf.variance_scaling_initializer(factor=1.0, mode='FAN_IN')


Here is where my problem occurs, because I have no idea what they mean with this neither if i need to give more information to the network besides only this line? I also saw you had to do something with the n_states (assuming dimension of state?).

If anyone could help me out, that would be great!

Jan

• Implementation questions belong on Data Science. – Philip Raeisghasem Mar 13 at 8:43
• Welcome to SE:AI. This seems like 2 questions. Q1: "Does Selu work like Batch Norm, as in normalizing the network, thus avoiding exploding gradients" should be suitable for this stack, but Q2: "How do I implement Selu in my Network?" is better suited toward Data Science or Overflow. Leaving as-is for now. My recommendation is to ask the first question here, and the implementation question on the other recommended stacks. – DukeZhou Mar 13 at 21:04