2
$\begingroup$

I'm using Experience Replay based on the original Prioritized Experience Replay (PER) paper. In the paper authors show ~ an order of magnitude increase in data efficiency from prioritized sampling. There is space for further improvement, since PER remembers all experiences, regardless of their importance.

I'd like to extend PER so it remembers selectively based on some metric, which would determine whether the experience is worth remembering or not. The time of sampling and re-adjusting the importance of the experiences increases with the number of samples remembered, so being smart about remembering should at the very least speed-up the replay, and hopefully also show some increase in data efficiency.

Important design constrains for this remembering metric:

  • compatibility with Q-Learning, such as DQN
  • computation time, to speed up the process of learning and not trade off one type of computation for another
  • simplicity

My questions:

  1. What considerations would you make for designing such a metric?
  2. Do you know about any articles addressing the prioritized experience memorization for Q-Learning?
$\endgroup$
  • $\begingroup$ I recently saw a paper on arXiv which, based on a very quick scan of title/abstract/introduction, seems to be very closely related to what you're asking about: "Double Prioritized State Recycled Experience Replay" (arxiv.org/abs/2007.03961). Unforutnately I didn't get around to reading it in detail yet, so I can't effectively summarise it in a proper answer to your question. If you do indeed find useful stuff in there, it would be nice if you could write an answer yourself to your own question (for future visitors of the site). $\endgroup$ – Dennis Soemers Jul 18 at 11:54
  • $\begingroup$ Thanks Dennis, appreciated. I read the article, but I don't fully trust its results. There are quite a few spelling errors in the writing and ambiguities in reporting results. $\endgroup$ – MightestDuck Jul 21 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.