Here is my code
Recently, I solved the game of Atari Breakout using a classic DQN model. The convergence of the mean reward slowly improved during three days. I was interested in learning a method which may help me improving the convergence speed. I found the following article : https://arxiv.org/pdf/1706.10295v3.pdf. It says I can use an Independent Gaussian Noise to outperform an standard DQN.
Here is my Noisy DQN model :
import math import torch import torch.nn as nn import torch.nn.functional as F import numpy as np class NoisyLinear(nn.Linear): #Independent Gaussian Noise used with NoisyDQN model def __init__(self, in_features, out_features, sigma_init=0.017, bias=True): super(NoisyLinear, self).__init__(in_features, out_features, bias=bias) self.sigma_weight = nn.Parameter(torch.full((out_features, in_features), sigma_init)) self.register_buffer("epsilon_weight", torch.zeros(out_features, in_features)) if bias: self.sigma_bias = nn.Parameter(torch.full((out_features,), sigma_init)) self.register_buffer("epsilon_bias", torch.zeros(out_features)) self.reset_parameters() def reset_parameters(self): std = math.sqrt(3/self.in_features) self.weight.data.uniform_(-std, std) self.bias.data.uniform_(-std, std) def forward(self, input): self.epsilon_weight.normal_() bias = self.bias if bias is not None: self.epsilon_bias.normal_() bias = bias + self.sigma_bias * self.epsilon_bias.data return F.linear(input, self.weight + self.sigma_weight * self.epsilon_weight.data, bias) class NoisyDQN(nn.Module): """ Look at https://arxiv.org/pdf/1706.10295v3.pdf """ def __init__(self, input_shape, num_actions): super(NoisyDQN, self).__init__() self.conv = nn.Sequential( nn.Conv2d(in_channels=input_shape, out_channels=32, kernel_size=8, stride=4), nn.ReLU(), nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2), nn.ReLU(), nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1), nn.ReLU(), ) self.conv_output_size = self.get_output_size(input_shape) self.linear = nn.Sequential( NoisyLinear(in_features=self.conv_output_size, out_features=512), nn.ReLU(), NoisyLinear(in_features=512, out_features=num_actions) ) def get_output_size(self, input_shape): output = self.conv(torch.zeros((1, *input_shape))) return int(np.prod(output.shape)) def forward(self, input): self.layer1 = self.conv(input) self.layer1 = self.layer1.reshape(-1, self.conv_output_size) return self.linear(self.layer1)
The idea is to replace the epsilon greedy action selection and my standard DQN model by the Noisy Network you can see just above.
The code run successfully, but it doesn't improve even a bit. How can I fix that?
After nearly 200k episodes, I am still between
2 reward. The maximum reward I can get on the Atari Breakout game is about
500. With a standard DQN, after 100k episodes, I am near
11 as reward.
On the above picture the X-axis is the number of episodes and the Y-axis is the mean reward over the last 100 rewards. The Y-axis is describe with
mean_reward = np.mean(self.total_rewards[-100:])
After about 8 hours of training, I got this
As you can see, it is not working as good as in the paper. I worked a lot with the hyperparameters, but not changed.