-1
$\begingroup$

I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over many years of data, the reward tends to become large. I have realized that after some training the agent becomes lazy, it just keeps the same position.

Is it a bad practice to use cumulative rewards as rewards? Would the daily revenue be a better reward for the agent?

$\endgroup$
1
  • $\begingroup$ Can you please describe more in detail your environment or problem? "prediction on the position on an asset in a stock trading-like environment" is not very clear to me. It's also not clear what you mean by "cumulative rewards as rewards". Maybe you mean "cumulative cash/credit/money as reward"? $\endgroup$
    – nbro
    Nov 21 '21 at 18:11
1
$\begingroup$

Reinforcement learning already has the objective of maximising the sum of future expected reward.

By making each reward the sum of all previous rewards, you will make the the difference between good and bad next choices low, relative to the overall reward guaranteed on each step.

The best reward for the agent should be as direct measure of what you want it to achieve as possible. Sometimes you need to compromise because you cannot easily measure what you really want. Sometimes you may need to assist the bot, giving it a more frequent reward signal (e.g. change to predicted cash-out value of all the current holdings) - but you should note this is also a compromise, because an agent that maximises this "reward shaping" reward scheme may not actually maximise what you really want.

You could view your attempt to use cumulative values as a failed experiment in reward shaping. I don't know if daily revenue would be a good for your bot, because a trading bot might have several different end goals for its creator.

First thing you should do is figure out what your success criteria is for the bot, and find the measure for it that best captures that goal as directly as possible. E.g. if the goal is for it to create a reliable income after 5 years of trading, then the reward might be the monthly income that it able to generate in the long term. Perhaps simulating or measuring this is too hard, in which case you could fall back to a proxy, such as cash value of any sell orders with a penalty if it is not maintaining a certain amount of total of holdings.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.