2
$\begingroup$

I am currently implementing this paper in Python. While reading about the reward scheme I came across the following:

Finally, the proposed reward scheme implicitly considers the number of steps as a cost because of the way in which Q-learning models the discount of future rewards (positive and negative).

How would you implement this "Number of Steps" cost? I am keeping track of the number of steps that have been taken, therefore would it be best to use an exponential functions to discount the reward at the current time step?

If anyone has a good idea or knows the standard in regard to this I would love to hear your thoughts.

$\endgroup$
3
$\begingroup$

How would you implement this "Number of Steps" cost?

What the paper is referring to is the reward discounting process which is a standard way of formulating RL problems, either continuous ones, or episodic ones where the goal is to complete a task in the least time (in the episodic version, a fixed cost per time step will also achieve this).

As such, this usually is implemented in the formulation of the value function calculations. The discount factor is usually represented as gamma, 𝛾.

For Q-learning, the factor should be in the TD error calculation:

𝛿 = R + 𝛾Q(s', a') - Q(s, a)

For Monte Carlo control, the factor appears more like this:

Gt = Σk 𝛾k Rt + k + 1

would it be best to use an exponential functions to discount the reward at the current time step?

Essentially that is what normal discounting is - an exponential decay of future reward. But if you have implemented "normal" Q-learning from equations like those above, it should already be there.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.