1
$\begingroup$

Here is my code

Recently, I solved the game of Atari Breakout using a classic DQN model. The convergence of the mean reward slowly improved during three days. I was interested in learning a method which may help me improving the convergence speed. I found the following article : https://arxiv.org/pdf/1706.10295v3.pdf. It says I can use an Independent Gaussian Noise to outperform an standard DQN.

Here is my Noisy DQN model :

import math
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np


class NoisyLinear(nn.Linear): #Independent Gaussian Noise used with NoisyDQN model
    def __init__(self, in_features, out_features, sigma_init=0.017, bias=True):
        super(NoisyLinear, self).__init__(in_features, out_features, bias=bias)

        self.sigma_weight = nn.Parameter(torch.full((out_features, in_features), sigma_init))
        self.register_buffer("epsilon_weight", torch.zeros(out_features, in_features))

        if bias: 
            self.sigma_bias = nn.Parameter(torch.full((out_features,), sigma_init))
            self.register_buffer("epsilon_bias", torch.zeros(out_features))

        self.reset_parameters()

    def reset_parameters(self):
        std = math.sqrt(3/self.in_features)
        self.weight.data.uniform_(-std, std)
        self.bias.data.uniform_(-std, std)

    def forward(self, input):
        self.epsilon_weight.normal_()
        bias = self.bias
        if bias is not None:
            self.epsilon_bias.normal_()
            bias = bias + self.sigma_bias * self.epsilon_bias.data
        return F.linear(input, self.weight + self.sigma_weight * self.epsilon_weight.data, bias)


class NoisyDQN(nn.Module):
    """
    Look at https://arxiv.org/pdf/1706.10295v3.pdf
    """

    def __init__(self, input_shape, num_actions):
        super(NoisyDQN, self).__init__()

        self.conv = nn.Sequential(
            nn.Conv2d(in_channels=input_shape[0], out_channels=32, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2),
            nn.ReLU(),
            nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1),
            nn.ReLU(),
        )

        self.conv_output_size = self.get_output_size(input_shape)

        self.linear = nn.Sequential(
           NoisyLinear(in_features=self.conv_output_size, out_features=512),
           nn.ReLU(),
           NoisyLinear(in_features=512, out_features=num_actions)
        )

    def get_output_size(self, input_shape):
        output = self.conv(torch.zeros((1, *input_shape)))
        return int(np.prod(output.shape))

    def forward(self, input):
        self.layer1 = self.conv(input)
        self.layer1 = self.layer1.reshape(-1, self.conv_output_size)

        return self.linear(self.layer1)

The idea is to replace the epsilon greedy action selection and my standard DQN model by the Noisy Network you can see just above.

The code run successfully, but it doesn't improve even a bit. How can I fix that?

UPDATE

enter image description here

After nearly 200k episodes, I am still between 1.5 and 2 reward. The maximum reward I can get on the Atari Breakout game is about 500. With a standard DQN, after 100k episodes, I am near 11 as reward.

On the above picture the X-axis is the number of episodes and the Y-axis is the mean reward over the last 100 rewards. The Y-axis is describe with mean_reward = np.mean(self.total_rewards[-100:])

UPDATE

After about 8 hours of training, I got this

enter image description here

As you can see, it is not working as good as in the paper. I worked a lot with the hyperparameters, but not changed.

$\endgroup$
  • 1
    $\begingroup$ Can you clarify what you mean by "it doesn't improve even a bit"? Do you mean that by using this Noisy Network you don't get rewards? Maybe provide a plot of the behaviour of your agent. $\endgroup$ – nbro Apr 17 at 18:23
  • $\begingroup$ @nbro I will add more info very soon. I am waiting to have a clear plot. $\endgroup$ – jgauth Apr 17 at 18:37
  • $\begingroup$ Without even knowing how noisy network work, I suggest you reduce the variance (where you use it) or, in general, noisy that you use during your training process. For example, I see you're doing std = math.sqrt(3/self.in_features). Isn't std maybe too high? $\endgroup$ – nbro Apr 17 at 20:34
  • $\begingroup$ Hm, that seems like a reasonable range to initialize the parameters. $\endgroup$ – nbro Apr 17 at 20:42
  • 1
    $\begingroup$ Anyway, the fact that self.in_features is 3136 will restrict the range of self.weight and self.bias. Is that ok? Honestly, I would need to look at the paper that introduced this noisy networks in order to understand better what's going on. $\endgroup$ – nbro Apr 17 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.