I have an actor-critic network. The state space contains continuous variables with different ranges like (0,1.57) and (-0.70, 0.70). And it also contain absolute 6D pose in the form (x,y,z,roll,pitch,yaw). The action space is continuous too in the range (0, 1.57). I apply scaling at the input layer, and scale things back before applying the action received from network. Irrespective of learning for 100 or 1000 episodes, the actor model always gives either -1 or 1. Eg: [1,1,-1,-1] which gets scaled to [1.57,1.57,-1.57,-1.57] as an action vector. Could someone give me suggestion on what's happening with the network. The learning follows DDPG algorithm.

actor_lr = 0.0001
critic_lr = 0.001
def actor(state_size, action_size):

    inputs = Input(shape=(state_size,), name="state_space")
    layer_one = Dense(300, activation="relu")(inputs)
    layer_two = Dense(400, activation="relu")(layer_one)

    outputs = Dense(action_size, activation="tanh", name="action_space")(layer_two)

    model = Model(inputs = inputs, outputs = outputs)
    model.compile(loss="mse", optimizer=Adam(lr=actor_lr), metrics=["accuracy"])
    return model

def get_critic(state_size, action_size):

    state_input = Input(shape=(state_size), name="state_space")
    state_out = Dense(64, activation="relu")(state_input)

    action_input = Input(shape=(action_size), name="actions_sapce")
    action_out = Dense(18, activation="relu")(action_input)
    concat = Concatenate()([state_out, action_out])
    critic_layer_one = Dense(300, activation="relu", kernel_regularizer=regularizers.l2(0.01))(concat)
    critic_layer_two = Dense(400, activation="relu", kernel_regularizer=regularizers.l2(0.01))(critic_layer_one)

    outputs = Dense(1, activation="linear", kernel_regularizer=regularizers.l2(0.01))(critic_layer_two)

    model = Model([state_input, action_input], outputs, name = name)
    model.compile(loss="mse", optimizer=Adam(lr=critic_lr), metrics=["accuracy"])
    return model

1 Answer 1


Most probably your network is underfitted. In that case, the network outputs values randomly. Hyperbolic tangent tanh converges very quickly towards $-1$ or $1$, so that is why you always find $-1$ and $1$ in the output.

Let us execute the following code to get a better idea:

import tensorflow as tf
tanh_x = tf.keras.activations.tanh(-8.0).numpy()
print(tanh_x, type(tanh_x))

At least in my machine, the output will be exactly $1.0$ for tanh_x variable of type numpy.float32, which has a precision of up to $7$ decimal digits.

The value of tanh_x is in fact $-0.99999977493$, but Python only saves the first $7$ decimal digits and decides to round it up to $-1.0$. Every output value from the neural network outside the range of $[-8, 8]$ will be exactly $-1$ or $1$ after tanh activation and float32 precision. As you can see, if the network randomly outputs values they will almost surely be outside $[-8, 8]$ as this is a very short range.

  • $\begingroup$ Thank you for the comment and example. Does it also mean the network is not well configured(not enough layers and neurons)? $\endgroup$
    – goldfinch
    Feb 21, 2022 at 20:14
  • $\begingroup$ @Goldfinch Honestly, I think your network is too large with 120k parameters. You could begin by reducing dense layer sizes to 30 and 40 instead of 300 and 400. $\endgroup$
    – devidduma
    Feb 21, 2022 at 21:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .