I'm a beginner of RL and currently trying to make DQN agent that can act optimally in a simple situation.
In the situation agent should decide at what rate to charge or discharge the electrical battery, which is equivalent to buying the electrical energy or selling it, for making money by means of arbitrage. So the action space is for example [-6, -4, -2, 0, 2, 4, 6]kW. The negative numbers mean discharging, and the positive numbers mean charging.
In a case that battery is empty, discharging actions(-6, -4, -2) should be forbidden. Otherwise in a case that battery is fully charged, charging actions(2, 4, 6) should be forbidden.
To deal with this issue, I tried two approaches:
- In every step, renewing the action space, which means masking the forbidden action.
- Give extreme penalties for selecting forbidden actions (in my case the penalty was -9999)
But none of them worked.
For the first approach, the training curve (the cumulative rewards) didn't converge.
For the second approach, the training curve converged, but the charging/discharging results are not reasonable (almost random results). I think in second approach, a lot of forbidden actions are selected randomly by the epsilon-greedy policy, and these samples are stored in experience memory, which negatively affect the result.
The state is defined as [p_t, e_t] where p_t is the market price for selling (discharging) the battery, and e_t is the amount of energy left in the battery.
When state = [p_t, e_t = 0], and discharging action (-6), which is forbidden action in this state, is selected, the next state is [p_t, e_t = -6]. And then the next action (2) is selected, then the next state is [p_t, e_t = -4] and so on.
In this case the < s, a, r, s' > samples are:
< [p_t, 0], -6, -9999, [p_t+1, -6] >
< [p_t, -6], 2, -9999, [p_t+1, -4] > ...
These are not expected to be stored in the experience memory because they are not desired samples (e_t should be more than zero). I think this is why desired results didn't come out.
So what should I do? Please help.