For discrete action spaces, what is the purpose of the actor in Actor-Critic algorithms?
In brief, it is the policy function $\pi(a|s)$. The critic (a state action function $v_{\pi}(s)$) is not used to derive a policy, and in "vanilla" Actor-Critic cannot be used in this way at all unless you have the full distribution model of the MDP.
It just seems to me that if you can accurately estimate the value function, then you have solved the RL challenge.
Often this can be the case, and this is how e.g. Q-learning works, where the value function is more precisely the action value function, $q(s,a)$.
Continuous or very large action spaces can cause a problem here, in that maximising over them is impractical. If you have a problem like that to solve, it is often an indicator that you should use a policy gradient method such as Actor-Critic. However, you have explicitly asked about "discrete action spaces" here.
The main issue with your statement "if you can accurately estimate the value function" is that you have implicitly assumed the the learning is complete. You are looking at the final output of the algorithm, after it has converged, and asking why not use just half of its output. Whilst choosing between RL algorithms is more often related to how they learn, how efficiently they learn, and how they behave during the learning process.
My current understanding is that the critic estimates the future reward given an action, so why not just take the action that maximizes the estimated return?
The critic is $v_{\pi}(s)$ so does not take actions into account. You can of course learn $q_{\pi}(s,a)$ instead or as well - and some policy gradient methods also estimate the action value, such as Advantage Actor-Critic.
The main difference is in how data gathered from experience is applied to changing the policy. Recall at any step during reinforcement learning, that all estimates are based around a best guess at values of a current target policy. Policy gradient methods directly improve the policy, whilst value-based methods have an implied policy based on the same learned value functions. The $\text{argmax}_a q(s,a)$ approach used in Q-learning or SARSA gives a simple but crude mapping from values to policy, and means that the trajectory through policy space taken whilst learning is different when comparing policy gradient methods and value-based methods.
The following are notable differences affecting performance between policy-based methods and value-based methods:
Policy-gradient methods tend to make small changes to policy on each step, the policy changes smoothly, adjusting probabilities between choice of different actions by small amounts. This in turn means that the targets for estimated value functions change slowly. In comparison, when a value based method finds a new maximising action, it can make a big change to the policy, making the updates required less smooth.
Policy-gradient methods can learn to balance stochastic policies, whilst value-based methods assume deterministic policies. So if you have a situation like paper, scissor stone game where the ideal policy is random choice between actions, you should use policy gradients. This is a bit niche, but is worth noting.
Function approximators (typically neural networks) are easier to train on simpler target functions. The policy function $\pi(a|s)$ and action value function $q(s,a)$ are quite different views of the same problem, and it can be the case that one has a simple form whilst the other is more complex. This can make a big practical difference on which algorithms learn fastest.
For smaller, discrete action spaces, there is not always a clear choice between policy gradient and value-based methods. You could use either and it may be worth doing an experiment to find the most efficient algorithm for certain classes of problem.
