so I tried to implement my own enviroment and agent in order to fully understand DQNs. The enviroment is a dungeon with five states.
actionspace = 2
statespace = 5
The implementation of the enviroment is trivial so I will left that part out.
env = enviroment.Dungeon() episodes = 1000 dqAgent = agent.DeepQAgent(5, 2, 0.5, episodes) for episode in range(episodes): action = dqAgent.nextAction(env.state) state, reward, newState = env.takeAktion(action) dqAgent.update(state, action, reward, newState)
This is the academy or orchestration in which the agent will choose, which is then passed to the enviroment. (You get the idea)
The agents brain is a pytorch NN with the following architecture:
self.fc1 = nn.Linear(5, 5) self.fc2 = nn.Linear(5, 2) self.criterion = nn.MSELoss() self.optim = optim.SGD(self.parameters(), self.lr)
The training procedure is the following:
- choose an action based on the state (exploration vs. exploitation)
- observe the feedback (oldState, reward, newState)
- then update the agents "mind" based on the obervtion
The critical part is the following. When I got it right. Iam trying to minimize the loss between the predicted qValues for a given state AND the actual qValues for a state. The actual Values can be calculated with this equation: (please tell me if the quation is wrong)
oldQValues = self.predictQ(state) newQValues = self.predictQ(newState) bestNewAction = self.nextAction(newState) targetQValues = oldQVals targetQValues[action] = reward + self.discount * newQVal[nextAction] self.optim.zero_grad() loss = self.criterion(oldQVals, targetQ) self.optim.step()
This is what I came up with. And this is kinda working. E.g when I let the agent predict the action for every given state:
tensor([[0.0000, 0.4230]], grad_fn=<ReluBackward0>) tensor([[0.0000, 0.3108]], grad_fn=<ReluBackward0>) tensor([[0.0371, 0.4376]], grad_fn=<ReluBackward0>) tensor([[0.0000, 0.4763]], grad_fn=<ReluBackward0>) tensor([[0.0137, 0.5945]], grad_fn=<ReluBackward0>)
The agant will always choose to go right. (a1) This is truly the right action to take. But as I alter the rewards the actions appear suboptimal. After tweaking the number of episodes and the learning rate the agent will also learn the new circumstances. But this whole situation seems strange to me. I would love it if someone can tell me if I got thie whole process right in my head. (Please ask if you want to check my knowlage) If I miss something or messed something up. Please also tell me.