3
$\begingroup$

Can anyone point me in the direction of a nice graph that depicts the "family tree", or hierarchy, of RL algorithms (or models)? For example, it splits the learning into TD and Monte Carlo methods, under which is listed all of the algorithms with their respective umbrella terms. Beneath each algorithm is shown modifications to those algorithms, etc. I'm having difficulty picturing where everything lies within the RL landscape.

$\endgroup$
4
$\begingroup$

I highly recommend looking at Reinforcement Learning: An Introduction by Richard Sutton and Andrew Barto.

In it they write:

Reinforcement learning, like many topics whose names end with “ing,” such as machine learning and mountaineering, is simultaneously a problem, a class of solution methods that work well on the problem, and the field that studies this problem and its solution methods. It is convenient to use a single name for all three things, but at the same time essential to keep the three conceptually separate. In particular, the distinction between problems and solution methods is very important in reinforcement learning; failing to make this distinction is the source of many confusions.

On the solution side of things, rather than a family tree of reinforcement learning, there is a spectrum of different approaches to the problem.

A slice through the space of RL methods, highlighting the two of the most important dimensions explored in the Introduction of Reinforcement Learning

Many of the points within the spectrum above are TD($\lambda$) methods. When the state space is large, one might use a neural network to help generalize across similar states, such as DQN for instance.

It should be noted that there are many applications of reinforcement learning algorithms that do not involve training an agent to perform a task. Instead one might want to evaluate an existing agent or use a general value function to predict an event using an agent's policy.

Nevertheless, many popular problems consist of training an agent to perform a task. These algorithms can be categorized into a "family tree" based either on what values they are updating or on what problem they are tackling. (Source : https://github.com/NervanaSystems/coach)

  • On the left are Value Based methods which update the value function of a policy and find policies that optimize said value function.

  • To the right of those are policy gradient methods which optimize for the policies directly without explicitly finding a value function.

  • To the far right is Imitation Learning whereby one wants to copy a policy given demonstrations of an existing but unacceptable policy.

  • And in the middle is Direct Future Prediction which is its own weird beast that I found out while finding that picture. Here is the arxiv paper that describes DFP.

enter image description here

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.