Why does my actor-critic network always give either -1 or 1 at the output layer?

Question

I have an actor-critic network. The state space contains continuous variables with different ranges like (0,1.57) and (-0.70, 0.70). And it also contain absolute 6D pose in the form (x,y,z,roll,pitch,yaw). The action space is continuous too in the range (0, 1.57). I apply scaling at the input layer, and scale things back before applying the action received from network. Irrespective of learning for 100 or 1000 episodes, the actor model always gives either -1 or 1. Eg: [1,1,-1,-1] which gets scaled to [1.57,1.57,-1.57,-1.57] as an action vector. Could someone give me suggestion on what's happening with the network. The learning follows DDPG algorithm.

actor_lr = 0.0001
critic_lr = 0.001
def actor(state_size, action_size):

    inputs = Input(shape=(state_size,), name="state_space")
    layer_one = Dense(300, activation="relu")(inputs)
    layer_two = Dense(400, activation="relu")(layer_one)

    outputs = Dense(action_size, activation="tanh", name="action_space")(layer_two)

    model = Model(inputs = inputs, outputs = outputs)
    model.compile(loss="mse", optimizer=Adam(lr=actor_lr), metrics=["accuracy"])
    return model

def get_critic(state_size, action_size):

    state_input = Input(shape=(state_size), name="state_space")
    state_out = Dense(64, activation="relu")(state_input)

    action_input = Input(shape=(action_size), name="actions_sapce")
    action_out = Dense(18, activation="relu")(action_input)
    concat = Concatenate()([state_out, action_out])
    
    critic_layer_one = Dense(300, activation="relu", kernel_regularizer=regularizers.l2(0.01))(concat)
    critic_layer_two = Dense(400, activation="relu", kernel_regularizer=regularizers.l2(0.01))(critic_layer_one)

    outputs = Dense(1, activation="linear", kernel_regularizer=regularizers.l2(0.01))(critic_layer_two)

    model = Model([state_input, action_input], outputs, name = name)
    model.compile(loss="mse", optimizer=Adam(lr=critic_lr), metrics=["accuracy"])
    return model

Answer 1

Most probably your network is underfitted. In that case, the network outputs values randomly. Hyperbolic tangent tanh converges very quickly towards $-1$ or $1$, so that is why you always find $-1$ and $1$ in the output.

Let us execute the following code to get a better idea:

import tensorflow as tf
tanh_x = tf.keras.activations.tanh(-8.0).numpy()
print(tanh_x, type(tanh_x))

At least in my machine, the output will be exactly $1.0$ for tanh_x variable of type numpy.float32, which has a precision of up to $7$ decimal digits.

The value of tanh_x is in fact $-0.99999977493$, but Python only saves the first $7$ decimal digits and decides to round it up to $-1.0$. Every output value from the neural network outside the range of $[-8, 8]$ will be exactly $-1$ or $1$ after tanh activation and float32 precision. As you can see, if the network randomly outputs values they will almost surely be outside $[-8, 8]$ as this is a very short range.