1
$\begingroup$

Paper link : Prioritized Experience Replay

About the blind cliffwalk setup:

  1. Why is the number of possible action sequences equal to 2^N? I cant think of sequences more than (N + 1) where one sequence is the sequence of all right actions and the other N sequences are due to wrong actions at each state.

Generally for prioritized experience replay:

  1. The replay memory consists of some transitions which are repeated.In the priority queue I feel that there should only be a single priority for each transition to speed up learning. Is there any advantage of having priority values for each repeated instance of the transition?

Edit for 2nd question:

Consider algorithm 1 on page 5 of the article.

enter image description here

Lets consider one of the transitions to be repeated in the replay memory. If one of them is sampled (line 9) and the priority updated (line 12). Will the priority update on the other instance of the same transition?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

for 1) i think your confusing elements touched vs sequences. At each point for N turns you have 2 possible options, therefore you have $\prod_{i=1}^N 2$ or $2^N$ possible sequences.

for 2) The priorities are updated based on the expected rewards. They do not add new elements each time, they update

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the answer! I was wondering what implications the 2^N has, given the setup of blind cliffwalk. Lets say there are just 3 states = (0,1 and 2). According to me the possible sequences of actions until termination would be the following: 0>0 (via the wrong (W) action in the first state) 0>1>0 0>1>2>0 (via W action in the last state) 0>1>2>0 (via R action in the last state) So this gives me 4 possible sequences of actions. $\endgroup$
    – human_ai
    Jun 27, 2019 at 3:19
  • $\begingroup$ According to the formula there should be 2 x 2 x 2 possible sequences of actions. But in each state one action leads to termination which does not propagate further. So does 2 x 2 x 2 make sense here? $\endgroup$
    – human_ai
    Jun 27, 2019 at 3:26
  • $\begingroup$ Priorities are updated based not the expected rewards but on current TD error. That way worst predictions updated more often. $\endgroup$
    – mirror2image
    Jun 27, 2019 at 5:23

Not the answer you're looking for? Browse other questions tagged .