Reinforcement learning: How to deal with illegal actions? [duplicate]

I'm a beginner of RL and currently trying to make DQN agent that can act optimally in a simple situation.

In the situation agent should decide at what rate to charge or discharge the electrical battery, which is equivalent to buying the electrical energy or selling it, for making money by means of arbitrage. So the action space is for example [-6, -4, -2, 0, 2, 4, 6]kW. The negative numbers mean discharging, and the positive numbers mean charging.

In a case that battery is empty, discharging actions(-6, -4, -2) should be forbidden. Otherwise in a case that battery is fully charged, charging actions(2, 4, 6) should be forbidden.

To deal with this issue, I tried two approaches:

• In every step, renewing the action space, which means masking the forbidden action.
• Give extreme penalties for selecting forbidden actions (in my case the penalty was -9999)

But none of them worked.

For the first approach, the training curve (the cumulative rewards) didn't converge.

For the second approach, the training curve converged, but the charging/discharging results are not reasonable (almost random results). I think in second approach, a lot of forbidden actions are selected randomly by the epsilon-greedy policy, and these samples are stored in experience memory, which negatively affect the result.

for example:

The state is defined as [p_t, e_t] where p_t is the market price for selling (discharging) the battery, and e_t is the amount of energy left in the battery.

When state = [p_t, e_t = 0], and discharging action (-6), which is forbidden action in this state, is selected, the next state is [p_t, e_t = -6]. And then the next action (2) is selected, then the next state is [p_t, e_t = -4] and so on.

In this case the < s, a, r, s' > samples are:

< [p_t, 0], -6, -9999, [p_t+1, -6] >

< [p_t, -6], 2, -9999, [p_t+1, -4] > ...

These are not expected to be stored in the experience memory because they are not desired samples (e_t should be more than zero). I think this is why desired results didn't come out.

• ignoring forbidden actions is the right approach, only consider the allowed actions at specific timestep. – Brale Aug 14 '19 at 11:34
• I would ignore forbidding actions (e.g. act like 0) in your environment (simulator), since - for as far I understood - they cause the system to be in an invalid state. Nevertheless, you probably also should put a (small) negative reward for those actions. – agold Aug 14 '19 at 13:17

In my project I also had the problem that the action space is not the same for every state of the environment. I do not like the approach to penalty forbidden actions with a high negative reward since it feels a bit like cheating. However it might work, I just haven't tried it.

The approach I used, which you could apply as well, is to integrate an additional function into your action space. This function would map an action to a specific amount of kW. Thereby, depending on the current state, the function maps the action to the amount of kW to charge or discharge your battery with. This has the advantage that you do not have to deal with illegal actions.

This could be applied as follows: Instead of defining for every action the amount to charge / discharge your battery with, you create a set of functions that defines the respective amount. Here an example with five actions:

1. Action: Discharge the battery entirely
2. Action. Discharge the battery so that half of its capacity remains, otherwise do nothing
3. Action: do nothing
4. Action: charge the batter to half its capacity, otherwise do nothing
5. Action: charge the battery to its maximum capacity

You can set the number of output nodes to the number of all actions, then choose the highest output value, try to do that action, if it can't, move to the next highest output value and so on. The only problem with this is that you have to know how many possible actions there are

﻿Reinforcement learning: How to deal with illegal actions?

It may be useful to you to consider the action space and the state space using the mathematical constructs of sets, regions, and boundary conditions.

If the variable type used for actions allows the representation of actions that are not in the action space, then the type will lead to the inefficiency of validation. This is also true for the variable type used for states. In the current state of the technology, this is often the case, leaving the responsibility of execution efficiency with how those variables are assigned.

Which is faster?

• Assign a variable to a possibly invalid value.
• Validate it.
• Deal with invalid values.

... or, applying the concepts of sets, regions, and boundary conditions to the assignment mechanism, ...

• Assign a variable to a value within constraints that represent the actor and its environment.

Applying this to the problem of this question, what is outside the boundary conditions?

• Is it the negative current (relative to the polarity of the storage device) when the storage device's voltage is zero?
• Is it a negative voltage that could potentially result from a negative current applied to a zero voltage storage device?
• Is it the cost of the physical disintegration of the storage device under this condition?

These questions help illuminate whether this is a question of what is legal or forbidden or whether it is a question of return?

• What is the cost of wasting electrical energy?
• Wasting materials?
• Wasting time?

Applying a negative current to a dead lithium, alkaline, or lead acid battery wastes all three. The wasteful actions that lead to wasted states exist, but the curve of return on the investment of energy in resale or reuse value of the battery should not be represented by a first degree polynomial. It should not be represented by -9999 either, as that doesn't nearly represent the degree of waste, and the knee of the curve may not be hard.

If using discrete states and actions rather than floating point representations of current and voltage, then a lookup table (perhaps tuned by experience using other AI techniques and indexed on the make and model of the storage device) must represent the actual curve approximating return on investment. If the goal is making money, as the question suggests, the goal of the learning is to maximize ROI. That must be embodied in your definition of optimality and then implemented to remain faithful to that definition in the DQN code.

Lastly, in real application, the initial internal resistance in combination with the initial voltage is key to determining whether a charge can ever produce a positive return for the given battery, so if voltage and current are continuously available, your test suite (which you should implement first if you are adhering to test driven development best practices) should include a case that tests the abandonment of charging altogether as a cost mitigation necessity.

The test of the algorithm regarding its fitness for the project is whether it behaves when the representations are real world. If not, there are other AI strategies to test. Not every algorithm is fit for every purpose, which is why job openings titled "AI Architect" and "AI Designer" are beginning to appear. It is likely that DQN will work if the value function is either set correctly by you or learned correctly by the appropriate learning mechanism that converges on good value approximations. The later method is suggested as best practice in this case.