I'm building a deep neural network to serve as the policy estimator in an actor-critic reinforcement learning algorithm for a continuing (not episodic) case. I'm trying to determine how to explore the action space. I have read through this text book by Sutton, and, in section 13.7, he gives one way to explore a continuous action space. In essence, you train the policy model to give a mean and standard deviation as an output, so you can sample a value from that Gaussian distribution to pick an action. This just seems like the continuous action-space equivalent of an $\epsilon$-greedy policy.
Are there other continuous action space exploration strategies I should consider?
Are these, or other, algorithms "better" (obviously depends on the problem being solved) alternatives to what is listed in the textbook for continuous action-space problems?
I know that this question could have many answers, but I'm just looking for a reasonably short list of options that people have used in real applications that work.