Are A2C or A3C suitable for episodic tasks where the reward is delivered only at the end of the episode?

Question

My understanding of the main idea behind A2C / A3C is that we run small segments of an episode to estimate the return using a trainable value function to compensate for the unseen final steps of the episode.

While I can see how this could work in continuing tasks with relatively dense rewards, where you can still get some useful immediate rewards from a small experience segment, does this approach work for episodic tasks where the reward is only delivered at the end? For example, in a game where you only know if you win or lose at the end of the game, does it still make sense to use the A2C / A3C approach?

It's not clear to me how the algorithm could get any useful signal to learn anything if almost every experience segment has zero reward, except for the last one. This would not be a problem in a pure MC approach for example, except for the fact that we might need a lot of samples. However, it's not clear to me that arbitrarily truncating episode segments like in A2C / A3C is a good idea in this case.

Answer 1

My understanding of the main idea behind A2C / A3C is that we run small segments of an episode to estimate the return using a trainable value function to compensate for the unseen final steps of the episode.

This seems fairly accurate. The important thing to note is that the trainable value function is trained to predict values (specifically, advantage values of state-action pairs in the case of A2C / A3C, where the first A stands for "advantage"). These value estimates can intuitively be understood as estimates of long-term (discounted) rewards, they're not just short-term rewards.

So yes, initially when the agent only observes a reward at the end of a long trajectory, only state-action pairs close to the end will receive credit for that reward. For example, when using $n$-step returns, approximately only the last $n$ state-action pairs receive credit. However, in the next episode, that longer-term reward will already become "visible" in the form of an advantage value prediction when you're still $n$ steps away from the end, and then that update can again get propagated back $n$ steps further into the history of state-action pairs.

My explanation above is very informal... there are all kinds of nuances that I skipped over. Use of function approximation is likely to speed up the propagation of reward observations through the space of state-action pairs even more, and of course in reality things won't be as "clean" as getting the propagation to get $n$ steps further in the next episode in comparison to the previous episode, since selected actions and random state transitions can be different... but hopefully it gets the idea across.