0
$\begingroup$

I'm working on an advantage actor-critic (A2C) reinforcement learning model, but when I test the model after I trained for 3500 episodes, I start to get almost the same action for all testing episodes. While if I trained the system for less than 850 episodes, I got different actions. The value of state is always different, and around 850 episodes, the loss becomes zero.

Here is the Actor and critic Network

        with g.as_default():
            #==============================actor==============================#
            actorstate = tf.placeholder(dtype=tf.float32, shape=n_input, name='state')
            actoraction = tf.placeholder(dtype=tf.int32, name='action')
            actortarget = tf.placeholder(dtype=tf.float32, name='target')

            hidden_layer1 = tf.layers.dense(inputs=tf.expand_dims(actorstate, 0), units=500, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
            hidden_layer2 = tf.layers.dense(inputs=hidden_layer1, units=250, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
            hidden_layer3 = tf.layers.dense(inputs=hidden_layer2, units=120, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
            output_layer = tf.layers.dense(inputs=hidden_layer3, units=n_output, kernel_initializer=tf.zeros_initializer())
            action_probs = tf.squeeze(tf.nn.softmax(output_layer))
            picked_action_prob = tf.gather(action_probs, actoraction)

            actorloss = -tf.log(picked_action_prob) * actortarget
            # actorloss = tf.reduce_mean(tf.losses.huber_loss(picked_action_prob, actortarget, delta=1.0), name='actorloss')

            actoroptimizer1 = tf.train.AdamOptimizer(learning_rate=var.learning_rate)

            if var.opt == 2:
                actoroptimizer1 = tf.train.RMSPropOptimizer(learning_rate=var.learning_rate, momentum=0.95,
                                                            epsilon=0.01)
            elif var.opt == 0:
                actoroptimizer1 = tf.train.GradientDescentOptimizer(learning_rate=var.learning_rate)

            actortrain_op = actoroptimizer1.minimize(actorloss)

            init = tf.global_variables_initializer()
            saver = tf.train.Saver(max_to_keep=var.n)

        p = tf.Graph()
        with p.as_default():
            #==============================critic==============================#
            criticstate = tf.placeholder(dtype=tf.float32, shape=n_input, name='state')
            critictarget = tf.placeholder(dtype=tf.float32, name='target')

            hidden_layer4 = tf.layers.dense(inputs=tf.expand_dims(criticstate, 0), units=500, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
            hidden_layer5 = tf.layers.dense(inputs=hidden_layer4, units=250, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
            hidden_layer6 = tf.layers.dense(inputs=hidden_layer5, units=120, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
            output_layer2 = tf.layers.dense(inputs=hidden_layer6, units=1, kernel_initializer=tf.zeros_initializer())
            value_estimate = tf.squeeze(output_layer2)

            criticloss= tf.reduce_mean(tf.losses.huber_loss(output_layer2, critictarget,delta = 0.5), name='criticloss')
            optimizer2 = tf.train.AdamOptimizer(learning_rate=var.learning_rateMADDPG_c)
            if var.opt == 2:
                optimizer2 = tf.train.RMSPropOptimizer(learning_rate=var.learning_rate_c, momentum=0.95,
                                                            epsilon=0.01)
            elif var.opt == 0:
                optimizer2 = tf.train.GradientDescentOptimizer(learning_rate=var.learning_rateMADDPG_c)

            update_step2 = optimizer2.minimize(criticloss)

            init2 = tf.global_variables_initializer()
            saver2 = tf.train.Saver(max_to_keep=var.n)

 

This is the choice of action.

def take_action(self, state):
                """Take the action"""
                action_probs = self.actor.predict(state)
                action = np.random.choice(np.arange(len(action_probs)), p=action_probs)
                return action

This is the actor.predict function.

def predict(self, s):
        return self._sess.run(self._action_probs, {self._state: s})

Any Idea what causing this?

Update

Change the learning rate, state, and the reward solve the problem where I reduce the size of the state and also added switching cost to the reward.

$\endgroup$
4
  • 1
    $\begingroup$ Not really possible to answer without seeing the full code. Can you post a colab notebook with a MWE? At the least we need more details such as a) what is the MDP, b) how are actions parametrized. You say you're getting the same action (I assume you mean over all timesteps in the episode) - is it possible that that is the optimal policy? $\endgroup$
    – Taw
    Aug 22 at 21:12
  • $\begingroup$ I got different actions when I train the model and it converges perfectly, but it is not changing during the testing for all the episodes. $\endgroup$ Aug 22 at 21:19
  • $\begingroup$ The code is taken from datahubbs.com/policy-gradients-and-advantage-actor-critic and stackoverflow.com/questions/45428574/… $\endgroup$ Aug 22 at 21:44
  • $\begingroup$ @Taw, I have added the code. $\endgroup$ Aug 23 at 19:50
1
$\begingroup$

Disclaimer: Without the full code, we can only speculate. I encourage you to post the full code on Google Colab or something like this. In the meanwhile, here is my point of view:


The Problem

Looks like your model has found some "master action" that always leads to zero loss, no matter what the state is. So it's not necessarily bad, it's just unexpected according to your point of view.

An example for that would be pausing the game - so you never loose.

You might not like it, but in de model's point of view, it's absolutely nailing it!

The Solution

So how to convince the actor not to pause the game?

Not by changing the model, or tuning hyper-parameters, but by reformulating the problem. In this example, instead of just penalizing the model for failing, you should reward if for winning, so pausing is no longer the best option.

Conclusion

It might not be a problem in the Machine Learning model, but in your environment and reward models. As we don't have access to that, it's hard to provide an answer.


Edit:

You are the CartPole-v0 environment:

A reward of +1 is provided for every timestep that the pole remains upright. The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center.

Source: https://gym.openai.com/envs/CartPole-v0/

It is a solvable problem. Probably your model has just learned how to solve it after a few hundreds generations. (The link shows a table with "Episodes before solve" for each algorithm, showing numbers consistent to yours).

TL;DR: It's not a bug, it's a feature!

$\endgroup$
2
0
$\begingroup$

If the loss is zero, all gradients should be zero as well, so you should take a look at the computed gradients. There might be a problem with momentum or the scheduled lr which might still apply very little updates which eventually lead to this policy colapse.

On a side note I would also call reduce_mean on the actor loss since you're optimizing the expected value.

$\endgroup$
6
  • $\begingroup$ My code was like this actorloss = tf.reduce_mean(tf.losses.huber_loss(picked_action_prob, actortarget, delta=1.0), name='actorloss') but nothing change. $\endgroup$ Aug 27 at 14:19
  • $\begingroup$ Are the gradients zero? $\endgroup$
    – tnfru
    Aug 28 at 11:42
  • $\begingroup$ No, it is not zero. $\endgroup$ Aug 28 at 14:07
  • $\begingroup$ But your loss is? Because if it is there might be some kind of momentum from your optimizer applying steps $\endgroup$
    – tnfru
    Aug 28 at 14:11
  • 1
    $\begingroup$ You need to find out the source of the non zero gradient. Your policy keeps moving (probably in the same direction) while your loss is zero. This eventually leads to selecting the same action almost every time and the policy colapse by this. You could also try lr decay, but I feel like it only minimizes the problem instead of solving it. $\endgroup$
    – tnfru
    Aug 29 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.