This is how they describe their infrastructure in https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf. I want to implement the game of Atari Breakout.
import torch import torch.nn as nn import torch.optim as optim import torch.nn.functional as F class DQN(nn.Module): def __init__(self, height, width): super(DQN, self).__init__() self.height = height self.width = width self.conv1 = nn.Conv2d(in_channels=4, out_channels=16, kernel_size=8, stride=4) self.conv2 = nn.Conv2d(in_channels=6, out_channels=32, kernel_size=4, stride=2) self.fc = nn.Linear(in_features=????, out_features=256) self.out = nn.Linear(in_features=256, out_features=4) def forward(self, state): # (1) Hidden Conv. Layer self.layer1 = F.relu(self.conv1(state)) #(2) Hidden Conv. Layer self.layer2 = F.relu(self.conv2(self.layer1)) #(3) Hidden Linear Layer self.layer3 = self.fc(self.layer2) #(4) Output actions = self.out(self.layer3) return actions
I will probably instantiate my policy network and my target network the following way :
policy_net = DQN(envmanager.get_height(), envmanager.get_width()).to(device) target_net = DQN(envmanager.get_height(), envmanager.get_width()).to(device)
I am very new in the world of Reinforcement Learning. I would like to implement their infrastructure in the
DQN(), but I think I am wrong in several places. Am I good here? If not, how can I fix it so that it reflect the infrastructure from the above picture?
I know that the formula to calculate the output size is equal to
where $O$ is the output height/length, $W$ is the input height/length, $K$ is the filter size, $P$ is the padding, and $S$ is the stride.
I obtained for
self.fc = nn.Linear(in_features=????, out_features=256) that
in_features must be equal to $32*9*9$