I applied for a Ph.D. in AI, my advisor told me that my thesis is about safe applications of deep RL algorithms in healthcare. So I decided to do as the first paper, a comparison of Deep RL algorithms in terms of their inherent safety. However, after lots of research, I could not find an answer to my question, that is: How to measure Deep RL algorithms in terms of safety?
I think you should first start with definition of software safety in health domain. For example, you should start with Therac-25 accident. Then look at the current scientific articles and standards about software safety in medical domain. Then think about how your algorithm will be tested.
You are thinking Deep RL algorithms as a blackbox but they are software in the end. If Deep RL algorithms will be used in hospitals, they will have to be tested. The benchmarks, conditions and restrictions of normal software must apply to RL algorithms too.
There is this paper talking specifically about the safety of ML/DL Algorithm but in industrial application. Since the algorithm is purely learning function you have to define the safety in terms of the application you are using it for example diagnosis or surgery.