Reinforcement learning: How to deal with illegal actions? [duplicate]

Question

I'm a beginner of RL and currently trying to make DQN agent that can act optimally in a simple situation.

In the situation agent should decide at what rate to charge or discharge the electrical battery, which is equivalent to buying the electrical energy or selling it, for making money by means of arbitrage. So the action space is for example [-6, -4, -2, 0, 2, 4, 6]kW. The negative numbers mean discharging, and the positive numbers mean charging.

In a case that battery is empty, discharging actions(-6, -4, -2) should be forbidden. Otherwise in a case that battery is fully charged, charging actions(2, 4, 6) should be forbidden.

To deal with this issue, I tried two approaches:

• In every step, renewing the action space, which means masking the forbidden action.
• Give extreme penalties for selecting forbidden actions (in my case the penalty was -9999)

But none of them worked.

For the first approach, the training curve (the cumulative rewards) didn't converge.

For the second approach, the training curve converged, but the charging/discharging results are not reasonable (almost random results). I think in second approach, a lot of forbidden actions are selected randomly by the epsilon-greedy policy, and these samples are stored in experience memory, which negatively affect the result.

for example:

The state is defined as [p_t, e_t] where p_t is the market price for selling (discharging) the battery, and e_t is the amount of energy left in the battery.

When state = [p_t, e_t = 0], and discharging action (-6), which is forbidden action in this state, is selected, the next state is [p_t, e_t = -6]. And then the next action (2) is selected, then the next state is [p_t, e_t = -4] and so on.

In this case the < s, a, r, s' > samples are:

< [p_t, 0], -6, -9999, [p_t+1, -6] >

< [p_t, -6], 2, -9999, [p_t+1, -4] > ...

These are not expected to be stored in the experience memory because they are not desired samples (e_t should be more than zero). I think this is why desired results didn't come out.

So what should I do? Please help.

Answer 1

In my project I also had the problem that the action space is not the same for every state of the environment. I do not like the approach to penalty forbidden actions with a high negative reward since it feels a bit like cheating. However it might work, I just haven't tried it.

The approach I used, which you could apply as well, is to integrate an additional function into your action space. This function would map an action to a specific amount of kW. Thereby, depending on the current state, the function maps the action to the amount of kW to charge or discharge your battery with. This has the advantage that you do not have to deal with illegal actions.

This could be applied as follows: Instead of defining for every action the amount to charge / discharge your battery with, you create a set of functions that defines the respective amount. Here an example with five actions:

1. Action: Discharge the battery entirely
2. Action. Discharge the battery so that half of its capacity remains, otherwise do nothing
3. Action: do nothing
4. Action: charge the batter to half its capacity, otherwise do nothing
5. Action: charge the battery to its maximum capacity

Answer 2

You can set the number of output nodes to the number of all actions, then choose the highest output value, try to do that action, if it can't, move to the next highest output value and so on. The only problem with this is that you have to know how many possible actions there are