I'm building a deep neural network to serve as the policy estimator in an Actor-Critic reinforcement learning algorithm for a continuing (not episodic) case. I'm trying to determine how to explore the action space. I have read through this text book by Sutton and in section 13.7 he gives one way to explore a continuous action space. In essence you train the policy model to give a mean and standard deviation as an output so you can sample a value from that Gaussian distribution to pick an action. This just seems like the continuous action-space equivalent of an epsilon greedy policy.
Also, are there other continuous action space exploration strategies I should consider? I've been doing some research online and found some articles related to RL in robotics and found that the PoWER and PI^2 algorithms do something similar to what is in the textbook. Are these, or other, algorithms "better" (obviously depends on the problem being solved) alternatives to what is listed in the textbook for continuous action-space problems?
I know that this question could have many answers, but I'm just looking for a reasonably short list of options that people have used in real applications that work.