Why the loss keep increasing during training?

I'm trying to use TensorFlow and this how I compute the loss

    learning_rate=0.001
g_target_q_t = tf.placeholder(tf.float32, None, name="target_value")
g_action = tf.placeholder(tf.int32, None, name='g_action')
action_one_hot = tf.one_hot(g_action, n_output, 1.0, 0.0, name='action_one_hot')
q_acted = tf.reduce_sum(y * action_one_hot, reduction_indices=1, name='q_acted')

g_loss = tf.reduce_mean(tf.square(g_target_q_t - q_acted), name='g_loss')
optim = tf.train.RMSPropOptimizer(learning_rate=var.learning_rate, momentum=0.95, epsilon=0.01).minimize(g_loss)


I already changed the Optimizer to:

optim = tf.train.AdamOptimizer(learning_rate=var.learning_rate).minimize(
g_loss)


and this is a sample training

def q_learning_mini_batch(current_agent, current_sess):
""" Training a sampled mini-batch """

batch_s_t, batch_s_t_plus_1, batch_action, batch_reward = current_agent.memory.sample()

if current_agent.double_q:  # double q-learning
pred_action = current_sess.run(g_q_action, feed_dict={x: batch_s_t_plus_1})
q_t_plus_1 = current_sess.run(target_q_with_idx, {x_p: batch_s_t_plus_1, g_target_q_idx: [[idx, pred_a] for idx, pred_a in enumerate(pred_action)]})
batch_target_q_t = current_agent.discount * q_t_plus_1 + batch_reward
else:
q_t_plus_1 = current_sess.run(y_p, {x_p: batch_s_t_plus_1})
max_q_t_plus_1 = np.max(q_t_plus_1, axis=1)
batch_target_q_t = current_agent.discount * max_q_t_plus_1 + batch_reward

_, loss_val = current_sess.run([optim, g_loss], {g_target_q_t: batch_target_q_t, g_action: batch_action, x: batch_s_t})
return loss_val


But the loss is very large

Episode: 884
step: 88499 agent 0 loss 6979308.5
-------------------------
Episode: 885
step: 88599 agent 0 loss 7371267.5
-------------------------
Episode: 886
step: 88699 agent 0 loss 8239375.5
-------------------------
Episode: 887
step: 88799 agent 0 loss 7476634.0
Update target Q network...
-------------------------
Episode: 888
step: 88899 agent 0 loss 6917692.5
-------------------------
Episode: 889
step: 88999 agent 0 loss 6011293.0
-------------------------
Episode: 890
step: 89099 agent 0 loss 5676079.0
-------------------------
Episode: 891
step: 89199 agent 0 loss 6374160.5


What are the explanations for this increasing?

• Most likely because you're using MSE loss for classification - try using cross entropy with a final softmax layer – Recessive Oct 3 at 2:55
• @Recessive it increases g_loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits =q_acted , labels=g_target_q_t), name='g_loss') – i_th Oct 3 at 5:07
• @Recessive I use g_loss = tf.reduce_mean(tf.nn.sigmoid( q_acted), name='g_loss') and it works fine but I'm not sure if it is the right way. – i_th Oct 3 at 5:29
• I don't think you can just chuck a cross entropy error function on the end, you'll probably need to change the last layer of your network to be softmax. I haven't used tensorflow that much before though, so I'm not sure. It could be a number of things, given how massive that error is it sounds like you have a massive data set - it could be possible there's an error in the data. It could also be that what you're trying to do simply isn't possible for your current architecture. – Recessive Oct 3 at 5:37
• I had a similar picture with random data – Stepan Novikov Oct 24 at 12:17