# Did I understand deep Q leaning right? (Implementation)

Gday guys,

so I tried to implement my own enviroment and agent in order to fully understand DQNs. The enviroment is a dungeon with five states.

actionspace = 2

statespace = 5

The implementation of the enviroment is trivial so I will left that part out.

env = enviroment.Dungeon()
episodes = 1000
dqAgent = agent.DeepQAgent(5, 2, 0.5, episodes)

for episode in range(episodes):
action = dqAgent.nextAction(env.state)
state, reward, newState = env.takeAktion(action)
dqAgent.update(state, action, reward, newState)


This is the academy or orchestration in which the agent will choose, which is then passed to the enviroment. (You get the idea)

The agents brain is a pytorch NN with the following architecture:

self.fc1 = nn.Linear(5, 5)
self.fc2 = nn.Linear(5, 2)
self.criterion = nn.MSELoss()
self.optim = optim.SGD(self.parameters(), self.lr)


The training procedure is the following:

1. choose an action based on the state (exploration vs. exploitation)
2. observe the feedback (oldState, reward, newState)
3. then update the agents "mind" based on the obervtion

The critical part is the following. When I got it right. Iam trying to minimize the loss between the predicted qValues for a given state AND the actual qValues for a state. The actual Values can be calculated with this equation: (please tell me if the quation is wrong)

oldQValues = self.predictQ(state)
newQValues = self.predictQ(newState)
bestNewAction = self.nextAction(newState)
targetQValues = oldQVals

targetQValues[0][action] = reward + self.discount * newQVal[0][nextAction]

loss = self.criterion(oldQVals, targetQ)
self.optim.step()


This is what I came up with. And this is kinda working. E.g when I let the agent predict the action for every given state:

tensor([[0.0000, 0.4230]], grad_fn=<ReluBackward0>)