# Did I understand deep Q leaning right? (Implementation)

Gday guys,

so I tried to implement my own enviroment and agent in order to fully understand DQNs. The enviroment is a dungeon with five states.

actionspace = 2

statespace = 5

!!!Action a0 is actually jump to s0!!!

The implementation of the enviroment is trivial so I will left that part out.

env = enviroment.Dungeon()
episodes = 1000
dqAgent = agent.DeepQAgent(5, 2, 0.5, episodes)

for episode in range(episodes):
action = dqAgent.nextAction(env.state)
state, reward, newState = env.takeAktion(action)
dqAgent.update(state, action, reward, newState)


This is the academy or orchestration in which the agent will choose, which is then passed to the enviroment. (You get the idea)

The agents brain is a pytorch NN with the following architecture:

self.fc1 = nn.Linear(5, 5)
self.fc2 = nn.Linear(5, 2)
self.criterion = nn.MSELoss()
self.optim = optim.SGD(self.parameters(), self.lr)


The training procedure is the following:

1. choose an action based on the state (exploration vs. exploitation)
2. observe the feedback (oldState, reward, newState)
3. then update the agents "mind" based on the obervtion

The critical part is the following. When I got it right. Iam trying to minimize the loss between the predicted qValues for a given state AND the actual qValues for a state. The actual Values can be calculated with this equation: (please tell me if the quation is wrong)

oldQValues = self.predictQ(state)
newQValues = self.predictQ(newState)
bestNewAction = self.nextAction(newState)
targetQValues = oldQVals

targetQValues[0][action] = reward + self.discount * newQVal[0][nextAction]

self.optim.zero_grad()
loss = self.criterion(oldQVals, targetQ)
self.optim.step()


This is what I came up with. And this is kinda working. E.g when I let the agent predict the action for every given state:

tensor([[0.0000, 0.4230]], grad_fn=<ReluBackward0>)
tensor([[0.0000, 0.3108]], grad_fn=<ReluBackward0>)
tensor([[0.0371, 0.4376]], grad_fn=<ReluBackward0>)
tensor([[0.0000, 0.4763]], grad_fn=<ReluBackward0>)
tensor([[0.0137, 0.5945]], grad_fn=<ReluBackward0>)


The agant will always choose to go right. (a1) This is truly the right action to take. But as I alter the rewards the actions appear suboptimal. After tweaking the number of episodes and the learning rate the agent will also learn the new circumstances. But this whole situation seems strange to me. I would love it if someone can tell me if I got thie whole process right in my head. (Please ask if you want to check my knowlage) If I miss something or messed something up. Please also tell me.

thank you

• Technically this stack doesn't cover implementation. Can you try to make this question about theory specifically? – DukeZhou Oct 17 '19 at 20:32
• Well I understand the theory, but to get sure I have to implement it. – OleVoß Oct 17 '19 at 20:53
• "But this whole situation seems strange to me. I would love it if someone can tell me if I got this whole process right in my head" definitely suggests the conceptual nature of your question, just that it's buried at the very end! – DukeZhou Oct 17 '19 at 20:55
• So can this stay? ^^ – OleVoß Oct 17 '19 at 21:01
• I'm leaving it open, pending response--just wanted to advise, as specifically referencing implementation may lead to downvotes or close votes. – DukeZhou Oct 17 '19 at 21:03