Consider the actor-critic reinforcement learning setting (actor and critic parameterized by a neural network). The reward is given only at the end of the episode (or when there is a timeout there is no reward).
How could we learn the value function? Do you recommend computing intermediate rewards?
The reward is given only at the end of the episode (or when there is timeout there is no reward)
This is a common case. E.g. winning a board game, or reaching a goal state.
How could we learn the value function?
All RL algorithms are designed to cope with this scenario. Actor-Critic is not an exception. Value-based algorithms (including the critic in Actor-Critic) learn through time step backup updates. The simplest backup is to copy data about experience at time $t+1$ into an update for the state or state/action experienced at time $t$. That is what single-step temporal difference algorithms do. Other value-based algorithms can be more sophisticated and more efficient with assigning the update signal back in time.
A very sparse reward can be difficult for an agent to find or learn from. So some methods may work better than others. Without knowing the environment, and what the specific difficulties might be, it is not possible to recommend an approach, or to even suggest whether your algorithm needs any help.
Do you recommend computing intermediate rewards?
In general, no. However, these can help when the "natural" rewards in a problem are both sparse and hard for the agent to discover through exploration. Constructing extra reward signals to guide a learning agent is called reward shaping.
Reward shaping needs to be done with care because it can inadvertently change what the optimal solution is. But if done well, it can make a problem much easier to solve for an agent.
A starting rule of thumb for whether to add some kind of reward shaping is based on how often a random agent might accidentally obtain a difference in end rewards. Once the agent has experienced a difference between rewards, it can start to refine its predictions and begin to prefer the higher reward, which then usually leads to exploration nearer more valuable states. It may do this even if the higher reward only occurs one time in a thousand initially, say. However, if initial random actions gain no useful signal even for millions of trial-and-error episodes, then you will need to do something to assist the agent.
