0
$\begingroup$

I have a deep-Q-network-type reinforcement learner in a minigrid-type environment. After learning I can put the agent in a series of contrived situations and measure its Q values, and then infer its effective discount rate from these Q values (e.g. infer the discount factor based on how the value for moving forward changes with proximity to the goal).

When I measure the effective discount factor this way, it matches the explicit discount factor (𝛾) setting I used.

But if I add a very strong L2 regularization (weight decay) to the network, the inferred discount factor decreases, even though I didn't change the agent's 𝛾 setting.

Could someone help me think through why this happens? Thanks!

$\endgroup$

1 Answer 1

0
$\begingroup$

L2 regularization pushes weights to be more normally distributed (see here), so your output will get smoothed out. However, reward functions are pretty discrete (e.g. did the mouse get the cheese or not?). Maybe you could try other priors for the weights? The Cauchy distribution is sharper, and has loss $$\frac{\gamma^2}{2}\ln\left(1 + \left(\frac{w}{\gamma}\right)^2\right)$$

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .