Skip to main content
added 2 characters in body; edited tags
Source Link
nbro
  • 41.4k
  • 12
  • 115
  • 205

I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space.

The algorithm for table-based Q-Learning updates a single entry of the Q table - i.e. a single Q(s, a)$Q(s, a)$. However, a neural network outputs an entire row of the table - i.e. the Q-values for every possible action for a given state. So, what should the target output vector be for the network?

I've been trying to get it to work with something like the following:

q_values = model(state)
action = argmax(q_values)
next_state = env.step(state, action)
next_q_values = model(next_state)
max_next_q = max(next_q_values)

target_q_values = q_values
target_q_values[action] = reward(next_state) + gamma * max_next_q

The result is that my model tends to converge on some set of fixed values for every possible action - in other words, I get the same Q-values no matter what the input state is. (My guess is that this is because, since only 1 Q-value is updated, the training is teaching my model that most of its output is already fine.)

What should I be using for the target output vector for training? Should I calculate the target Q value for every action, instead of just one?

I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space.

The algorithm for table-based Q-Learning updates a single entry of the Q table - i.e. a single Q(s, a). However, a neural network outputs an entire row of the table - i.e. the Q-values for every possible action for a given state. So, what should the target output vector be for the network?

I've been trying to get it to work with something like the following:

q_values = model(state)
action = argmax(q_values)
next_state = env.step(state, action)
next_q_values = model(next_state)
max_next_q = max(next_q_values)

target_q_values = q_values
target_q_values[action] = reward(next_state) + gamma * max_next_q

The result is that my model tends to converge on some set of fixed values for every possible action - in other words, I get the same Q-values no matter what the input state is. (My guess is that this is because, since only 1 Q-value is updated, the training is teaching my model that most of its output is already fine.)

What should I be using for the target output vector for training? Should I calculate the target Q value for every action, instead of just one?

I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space.

The algorithm for table-based Q-Learning updates a single entry of the Q table - i.e. a single $Q(s, a)$. However, a neural network outputs an entire row of the table - i.e. the Q-values for every possible action for a given state. So, what should the target output vector be for the network?

I've been trying to get it to work with something like the following:

q_values = model(state)
action = argmax(q_values)
next_state = env.step(state, action)
next_q_values = model(next_state)
max_next_q = max(next_q_values)

target_q_values = q_values
target_q_values[action] = reward(next_state) + gamma * max_next_q

The result is that my model tends to converge on some set of fixed values for every possible action - in other words, I get the same Q-values no matter what the input state is. (My guess is that this is because, since only 1 Q-value is updated, the training is teaching my model that most of its output is already fine.)

What should I be using for the target output vector for training? Should I calculate the target Q value for every action, instead of just one?

edited title
Link
Kricket
  • 197
  • 4

Target What is the target output for updating a Deep Q Network

Source Link
Kricket
  • 197
  • 4

Target output for updating Deep Q Network

I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space.

The algorithm for table-based Q-Learning updates a single entry of the Q table - i.e. a single Q(s, a). However, a neural network outputs an entire row of the table - i.e. the Q-values for every possible action for a given state. So, what should the target output vector be for the network?

I've been trying to get it to work with something like the following:

q_values = model(state)
action = argmax(q_values)
next_state = env.step(state, action)
next_q_values = model(next_state)
max_next_q = max(next_q_values)

target_q_values = q_values
target_q_values[action] = reward(next_state) + gamma * max_next_q

The result is that my model tends to converge on some set of fixed values for every possible action - in other words, I get the same Q-values no matter what the input state is. (My guess is that this is because, since only 1 Q-value is updated, the training is teaching my model that most of its output is already fine.)

What should I be using for the target output vector for training? Should I calculate the target Q value for every action, instead of just one?