0
$\begingroup$

I understand that gamma is an important factor in determining the rewards for a deep Q agent, however during testing of my network I am noticing that the agent is outputting more actions to "do nothing" as it learns more about it's given data set (financial stock data).

I have tried tweaking the gamma at different levels ranging from 0 - 1 and everywhere in between, however as the agent continues to learn, the times between actions is getting longer, and longer. This behaviour is undesirable, and it is preferable that the agent be making more often, short-term actions even if they result in reduced reward.

Does anyone have any tips on how to achieve this? Would a minus gamma have adverse effects on the network?

TLDR: Time between actions becoming increasingly long over time, would prefer an agent that makes many actions over long-term ones.

|improve this question
$\endgroup$
  • 1
    $\begingroup$ The value of gamma would determine which actions an agent would take It wouldn't have a significant impact on the time between taking actions. It sounds like something else in your implementation is eating up time. $\endgroup$ – Philip Raeisghasem Apr 21 '19 at 19:11
  • $\begingroup$ Agreed with Philip. There is nothing inherently part of Q learning, DQN, A2C etc that makes them slow down with more experience. Are you perhaps storing experience and using it directly to search for the probable best action? $\endgroup$ – Neil Slater Apr 21 '19 at 19:25
  • $\begingroup$ Thanks for the response - i am using experience replay with a memory length of 1000 with an action space of 3 (either 0,1 or 2) an action of 0 would mean "doing nothing" and my observations show that as the length of training time increases, the amount of time between getting a 1 or 2 action increases. I initially thought this was caused by my epsilon function, however the slowdown continues after epsilon has fully decayed $\endgroup$ – Ryan Hodgson Apr 21 '19 at 22:01
  • $\begingroup$ So, to clarify, the amount of time between decisions is not wall clock time (which would imply inefficient code), but time step intervals? The agent is in fact making action choices at normal speed, but choosing a specific "do nothing" action more frequently that you would like? I think in order to help with this, you need to explain more about your environment and reward scheme. $\endgroup$ – Neil Slater Apr 22 '19 at 12:05
  • $\begingroup$ You should note that it may be entirely realistic for the agent to make few "active" decisions if you have put it into a high risk environment, such as gambling or financial markets, where it has determined that the expected reward from acting is negative in most scenarios (including short-term ones) $\endgroup$ – Neil Slater Apr 22 '19 at 12:08
0
$\begingroup$

Thanks for the responses - after taking into account the 'risk' the network experiences when working with unstable financial data I modified the activation layers from RelU to Sigmoid - this has lead to a considerable improvement to both results (profit!) and additionally the rate at which actions are carried out.

I will also edit the above question to provide further details.

|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.