# How to modify the Actor-Critic policy gradient algorithm to perform Safe exploration in Reinforcement Learning

I am trying to implement safe exploration technique in [Ref.1]. I am using Soft Actor-Critic algorithm to teach an agent to introduce a bias between 0 and 1 to a specific state of interest in my environment.

I would like to ask for your help in order to modify the critic update equation -which is originally based on the return or the RL cost function- from this:

which is based on the return functions:

=

to be based on the following cost function to make the RL objective risk-sensitive and avoid large-valued actions (bias values) at the start of the agent's learning,

How can I include the second part of the cost function - in which the variance of the reward is evaluated- in the update equation?

[Ref.1] Heger, M., Consideration of risk in reinforcement learning. 11th International Machine Learning Conference (1994)

• what is $c_t$ here? – David Ireland Sep 3 '20 at 19:17
• It’s the return at instant t. When compared to the $R_t$ equation, $c_t$ ~ \$r_{t+k+1} – Ayomi Al-noor Sep 3 '20 at 19:33