Can a large discrete action space be represented using Gaussian distributions?

Question

I have a large 1D action space, e.g. dim(A)=2000-10000. Can I use continuous action space where I could learn the mean and std of the Gaussian distributions that I would use to sample action from and round the value to the nearest integer? If yes, can I extend this idea to multi-dimensional large action space?

Answer 1

The answer is "it depends". Once you have arranged the actions into order, a key trait is whether the action value function has a simple enough shape that sampling from a Gaussian policy function would give consistent expected returns, enough that learning can occur. If the underlying "true" value function has a lot of high frequency noise then learning would be slow. In the worst case, if the action value $Q(s,a_{n})$ and $Q(s,a_{n+1})$ is not correlated for any $n$, then it will not be possible to learn with the approximation at all.

You may have some sense of how similar actions $a_{n}$ and $a_{n+1}$ are. If the actions represent different ordinal choices, such as selecting an integer number of items to perform some task with such as buy/sell or transport, then in many environments there will often be a strong correlation between outcome of choosing e.g. $a_{900}$ and $a_{901}$. If this holds in general, then that is a good indicator that you can treat $n$ as being continuous, and use learned parameters of a simple distribution function to find optimal policies (and in addition this could be far more efficient than using a discrete representation).

It might not matter if for a small fraction of cases the difference in outcomes between $a_{n}$ and $a_{n+1}$ is large, provided that successive approximations to the optimal policy can improve towards optimal through adjusting mean $\mu(s)$ and standard deviation $\sigma(s)$.

There may still be difficult cases that cannot be learned by the approximation - for instance if a specific action $a_n$ is optimal for a given state $s$, but $a_{n-1}$ and $a_{n+1}$ are a lot worse, then the training process of a typical policy gradient approach may never settle upon $\mu(s) = n$ and $\sigma(s) \approx 0$, because any intermediate values between the starting policy and the optimal one will perform badly.

For expanding into more dimensions, the same ideas apply to each dimension separately. You may want to use different distributions, or even have one dimension that uses a continuous model with a few parameters whilst another remains discrete with a free parameter for each choice.