I'm currently working on a project to make an DQN agent that decides whether to charge or discharge an electric vehicle according to hourly changing price to sell or buy. The price pattern also varies from day to day. The goal of this work is to schedule the optimal charging/discharging actions so that it can save money.

The state contains past n-step price records, current energy level in battery, hour, etc., like below:

$$ s_t = \{ p_{t-5}, p_{t-4}, p_{t-3}, p_{t-2}, p_{t-1}, E_t, t \} $$

What I'm wondering is whether this is a partial observable situation or not, because the agent can only observe past n-step prices rather than knowing every price at every time step.

Can anyone comment on this issue?

If this is the partial observable situation, is there any simple way to deal with it?


An environment is partially observable if the agent cannot fully observe the current state but it only partially observes it. More specifically, in fully observable MDPs (FOMDPs), the agent knows the current state of the environment, which can or not (for example, depending on whether the state is Markov or not) contain all the information required to theoretically take the optimal action. In a partially observable MDPs (POMDPs), the state is not available or observable, so the agent might not possess all the required information to theoretically take the optimal action.

For example, in the game of poker, there are hidden and observable cards. In this game, the state of the environment could be defined as all the hands of all players, the cards on the table and the next cards that will be drawn from the deck until the end of the round. If a player had access to all this information, then the environment would be fully observable, but, usually, this information is not available, so poker is usually considered a partially observable game.

You defined the state as follows

$$s_t = \{ p_{t:t-n}, E_t\}$$

So, if you have access to the past $n$ price records, $p_{1:n}$, and the energy level, $E_t$, then your environment is fully observable. However, this does not mean that it is a Markov environment, that is, intuitively, that your current state is sufficient to theoretically take the optimal action.

To conclude, partial observability depends on the definition of state in a specific problem. The concept of Markov property is also related to the concept of partial observability. However, in the case of MDPs, the concept of Markov property has a specific definition: given the current state and an action, the probability of the next state is conditionally independent of all previous states and actions or, more formally

$$ P(S_{t+1} \mid S_t, A_t) = P(S_{t+1} \mid S_t, S_{t-1}, \dots, S_1, A_t, A_{t-1}, \dots, A_1) $$

where $S_{t+1}$ is the next state, $S_t$ the current state, $A_t$ the current action, $S_{t-1}, \dots, S_1$ the previous states and $A_{t-1}, \dots, A_1$ the previous actions.

  • $\begingroup$ Thank you for clear explanationI have an additional question. As you said in your answer, to find optimal action it will be needed to give sufficient state information (Markov environment). If that is true, my state representation in the problem explained in my question is not providing sufficient information because it provides 'past n-step price' rather than 'the price on the hour it trades'. I intended to give helpful information to infer the price on the hour because practically the price on that time is not known. $\endgroup$ – JH Lee Sep 1 '19 at 5:32
  • $\begingroup$ In this case, am I wrongly defining the problem or is it something that reinforcement learning cannot solve? $\endgroup$ – JH Lee Sep 1 '19 at 5:32
  • $\begingroup$ @JHLee I would ask this as a new question on the website. I would also add details related to your definition of the reward function and possible actions. $\endgroup$ – nbro Sep 1 '19 at 10:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.