Why I got the same action when testing the A2C?

Question

I'm working on an advantage actor-critic (A2C) reinforcement learning model, but when I test the model after I trained for 3500 episodes, I start to get almost the same action for all testing episodes. While if I trained the system for less than 850 episodes, I got different actions. The value of state is always different, and around 850 episodes, the loss becomes zero.

Here is the Actor and critic Network

        with g.as_default():
            #==============================actor==============================#
            actorstate = tf.placeholder(dtype=tf.float32, shape=n_input, name='state')
            actoraction = tf.placeholder(dtype=tf.int32, name='action')
            actortarget = tf.placeholder(dtype=tf.float32, name='target')

            hidden_layer1 = tf.layers.dense(inputs=tf.expand_dims(actorstate, 0), units=500, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
            hidden_layer2 = tf.layers.dense(inputs=hidden_layer1, units=250, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
            hidden_layer3 = tf.layers.dense(inputs=hidden_layer2, units=120, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
            output_layer = tf.layers.dense(inputs=hidden_layer3, units=n_output, kernel_initializer=tf.zeros_initializer())
            action_probs = tf.squeeze(tf.nn.softmax(output_layer))
            picked_action_prob = tf.gather(action_probs, actoraction)

            actorloss = -tf.log(picked_action_prob) * actortarget
            # actorloss = tf.reduce_mean(tf.losses.huber_loss(picked_action_prob, actortarget, delta=1.0), name='actorloss')

            actoroptimizer1 = tf.train.AdamOptimizer(learning_rate=var.learning_rate)

            if var.opt == 2:
                actoroptimizer1 = tf.train.RMSPropOptimizer(learning_rate=var.learning_rate, momentum=0.95,
                                                            epsilon=0.01)
            elif var.opt == 0:
                actoroptimizer1 = tf.train.GradientDescentOptimizer(learning_rate=var.learning_rate)

            actortrain_op = actoroptimizer1.minimize(actorloss)

            init = tf.global_variables_initializer()
            saver = tf.train.Saver(max_to_keep=var.n)

        p = tf.Graph()
        with p.as_default():
            #==============================critic==============================#
            criticstate = tf.placeholder(dtype=tf.float32, shape=n_input, name='state')
            critictarget = tf.placeholder(dtype=tf.float32, name='target')

            hidden_layer4 = tf.layers.dense(inputs=tf.expand_dims(criticstate, 0), units=500, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
            hidden_layer5 = tf.layers.dense(inputs=hidden_layer4, units=250, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
            hidden_layer6 = tf.layers.dense(inputs=hidden_layer5, units=120, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
            output_layer2 = tf.layers.dense(inputs=hidden_layer6, units=1, kernel_initializer=tf.zeros_initializer())
            value_estimate = tf.squeeze(output_layer2)

            criticloss= tf.reduce_mean(tf.losses.huber_loss(output_layer2, critictarget,delta = 0.5), name='criticloss')
            optimizer2 = tf.train.AdamOptimizer(learning_rate=var.learning_rateMADDPG_c)
            if var.opt == 2:
                optimizer2 = tf.train.RMSPropOptimizer(learning_rate=var.learning_rate_c, momentum=0.95,
                                                            epsilon=0.01)
            elif var.opt == 0:
                optimizer2 = tf.train.GradientDescentOptimizer(learning_rate=var.learning_rateMADDPG_c)

            update_step2 = optimizer2.minimize(criticloss)

            init2 = tf.global_variables_initializer()
            saver2 = tf.train.Saver(max_to_keep=var.n)

 

This is the choice of action.

def take_action(self, state):
                """Take the action"""
                action_probs = self.actor.predict(state)
                action = np.random.choice(np.arange(len(action_probs)), p=action_probs)
                return action

This is the actor.predict function.

def predict(self, s):
        return self._sess.run(self._action_probs, {self._state: s})

Any Idea what causing this?

Update

Change the learning rate, state, and the reward solve the problem where I reduce the size of the state and also added switching cost to the reward.

Answer 1

Disclaimer: Without the full code, we can only speculate. I encourage you to post the full code on Google Colab or something like this. In the meanwhile, here is my point of view:

---

The Problem

Looks like your model has found some "master action" that always leads to zero loss, no matter what the state is. So it's not necessarily bad, it's just unexpected according to your point of view.

An example for that would be pausing the game - so you never loose.

You might not like it, but in de model's point of view, it's absolutely nailing it!

The Solution

So how to convince the actor not to pause the game?

Not by changing the model, or tuning hyper-parameters, but by reformulating the problem. In this example, instead of just penalizing the model for failing, you should reward if for winning, so pausing is no longer the best option.

Conclusion

It might not be a problem in the Machine Learning model, but in your environment and reward models. As we don't have access to that, it's hard to provide an answer.

---

Edit:

You are the CartPole-v0 environment:

A reward of +1 is provided for every timestep that the pole remains upright. The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center.

Source: https://gym.openai.com/envs/CartPole-v0/

It is a solvable problem. Probably your model has just learned how to solve it after a few hundreds generations. (The link shows a table with "Episodes before solve" for each algorithm, showing numbers consistent to yours).

TL;DR: It's not a bug, it's a feature!

Answer 2

If the loss is zero, all gradients should be zero as well, so you should take a look at the computed gradients. There might be a problem with momentum or the scheduled lr which might still apply very little updates which eventually lead to this policy colapse.

On a side note I would also call reduce_mean on the actor loss since you're optimizing the expected value.