1
$\begingroup$

Problem setting

Consider a game like trading a stock

  • At each step, the agent can buy / sell a stock.
  • Trade is a pair of buy and sell actions.
  • We can set the reward for the sell action as the profit made in this trade. Although this might be misleading, since whether a trade is good or not depends on both buying and selling timing.
  • We don't know the reward for buy, whether a buy is good or not depends on when you sell.

Questions

  1. How should the reward scheme be for a game like this? i.e., whether one action is good or not depends on other actions taken before it?
  2. Can we set a scheme like this: set the reward to be zero at each step, no matter it is a buy or a sell. Only give the reward at the end of the game, say after 1000 steps, and the reward, in this case, is the total money made?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

how should the reward scheme be for a game like this? i.e., whether one action is good or not depends on other actions taken before it?

The reward scheme should always be a "natural" representation of the goals of the agent. If the goal of your agent is to make a profit, then the reward should be the amount of profit*.

It is important to RL, and becomes critical for rewards that may be delayed, that the state representation captures all the information that impacts those rewards. That means representing the direct effects of actions, so that they can be tracked. In the case of a trading bot, it will be important at a minimum that the state includes the current portfolio. If the agent can access data about the amount and value of stock it currently holds, it has a chance to predict likely profits later.

can we set a scheme like this: set the reward to be zero at each step, no matter it is a buy or a sell. Only give the reward at the end of the game, say after 1000 steps, and the reward, in this case, is the total money made?

You can, but that looks like it is making the problem harder than it needs to be.

I would naturally set the reward to be the profit on each transaction (including negative reward for buying stock). The agent should be able to use that to figure out long term rewards, and will learn faster if it is given this more direct feedback. RL algorithms are well-suited to figuring out the need to invest at an earlier time in order to make profit later. If there is also a time horizon, that should be factored into the state - the agent should be made aware that the game will end in so many time steps otherwise it may end up holding stock instead of selling it in time to meet the final evaluation.

One important caveat is that real trading problems are hard, because the real environment is highly variable and competitive. The competition includes intelligent people with advanced qualifications in statistics. The chances of making a trading bot that is successful in a real trading environment are low.

* Sometimes this "natural" rerpresentation is too sparse, and you may want to look into reward shaping. That is adding interim rewards or spreading out an existing reward, to assist the learning agent. For a stock trading game for instance, it may help to allocate some reward for gaining ownership of a stock, perhaps the nominal resale value minus the fees for selling it.

However, where it is possible to stick with the simplest direct interpretation of the agent's goals, then you should do that. Reward shaping comes with the potential risk that the changes you have made will impact what it means for the agent to perform optimally.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .