I am trying to implement safe exploration technique in [Ref.1]. I am using Soft Actor-Critic algorithm to teach an agent to introduce a bias between 0 and 1 to a specific state of interest in my environment.
I would like to ask for your help in order to modify the critic update equation -which is originally based on the return or the RL cost function- from this:
which is based on the return functions:
to be based on the following cost function to make the RL objective risk-sensitive and avoid large-valued actions (bias values) at the start of the agent's learning,
How can I include the second part of the cost function - in which the variance of the reward is evaluated- in the update equation?
[Ref.1] Heger, M., Consideration of risk in reinforcement learning. 11th International Machine Learning Conference (1994)