2
$\begingroup$

I'm trying to use TensorFlow and this how I compute the loss

    learning_rate=0.001
    g_target_q_t = tf.placeholder(tf.float32, None, name="target_value")
    g_action = tf.placeholder(tf.int32, None, name='g_action')
    action_one_hot = tf.one_hot(g_action, n_output, 1.0, 0.0, name='action_one_hot')
    q_acted = tf.reduce_sum(y * action_one_hot, reduction_indices=1, name='q_acted')

    g_loss = tf.reduce_mean(tf.square(g_target_q_t - q_acted), name='g_loss')
    optim = tf.train.RMSPropOptimizer(learning_rate=var.learning_rate, momentum=0.95, epsilon=0.01).minimize(g_loss)

I already changed the Optimizer to:

optim = tf.train.AdamOptimizer(learning_rate=var.learning_rate).minimize(
        g_loss)

and this is a sample training

def q_learning_mini_batch(current_agent, current_sess):
    """ Training a sampled mini-batch """

    batch_s_t, batch_s_t_plus_1, batch_action, batch_reward = current_agent.memory.sample()

    if current_agent.double_q:  # double q-learning
        pred_action = current_sess.run(g_q_action, feed_dict={x: batch_s_t_plus_1})
        q_t_plus_1 = current_sess.run(target_q_with_idx, {x_p: batch_s_t_plus_1, g_target_q_idx: [[idx, pred_a] for idx, pred_a in enumerate(pred_action)]})
        batch_target_q_t = current_agent.discount * q_t_plus_1 + batch_reward
    else:
        q_t_plus_1 = current_sess.run(y_p, {x_p: batch_s_t_plus_1})
        max_q_t_plus_1 = np.max(q_t_plus_1, axis=1)
        batch_target_q_t = current_agent.discount * max_q_t_plus_1 + batch_reward

    _, loss_val = current_sess.run([optim, g_loss], {g_target_q_t: batch_target_q_t, g_action: batch_action, x: batch_s_t})
    return loss_val

But the loss is very large

Episode: 884
step: 88499 agent 0 loss 6979308.5
-------------------------
Episode: 885
step: 88599 agent 0 loss 7371267.5
-------------------------
Episode: 886
step: 88699 agent 0 loss 8239375.5
-------------------------
Episode: 887
step: 88799 agent 0 loss 7476634.0
Update target Q network...
-------------------------
Episode: 888
step: 88899 agent 0 loss 6917692.5
-------------------------
Episode: 889
step: 88999 agent 0 loss 6011293.0
-------------------------
Episode: 890
step: 89099 agent 0 loss 5676079.0
-------------------------
Episode: 891
step: 89199 agent 0 loss 6374160.5

What are the explanations for this increasing?

$\endgroup$
  • $\begingroup$ Most likely because you're using MSE loss for classification - try using cross entropy with a final softmax layer $\endgroup$ – Recessive Oct 3 at 2:55
  • $\begingroup$ @Recessive it increases g_loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits =q_acted , labels=g_target_q_t), name='g_loss') $\endgroup$ – i_th Oct 3 at 5:07
  • $\begingroup$ @Recessive I use g_loss = tf.reduce_mean(tf.nn.sigmoid( q_acted), name='g_loss') and it works fine but I'm not sure if it is the right way. $\endgroup$ – i_th Oct 3 at 5:29
  • $\begingroup$ I don't think you can just chuck a cross entropy error function on the end, you'll probably need to change the last layer of your network to be softmax. I haven't used tensorflow that much before though, so I'm not sure. It could be a number of things, given how massive that error is it sounds like you have a massive data set - it could be possible there's an error in the data. It could also be that what you're trying to do simply isn't possible for your current architecture. $\endgroup$ – Recessive Oct 3 at 5:37
  • 1
    $\begingroup$ I had a similar picture with random data $\endgroup$ – Stepan Novikov Oct 24 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.