# Is it required that taking an action updates the state?

For some environments taking an action may not update the environment state. For example, a trading RL agent may take an action to buy shares s. The state at time t which is the time of investing is represented as the interval of 5 previous prices of s. At t+1 the share price has changed but it may not be as a result of the action taken. Does this affect RL learning, if so how ? Is it required that state is updated as a result of taking actions for agent learning to occur ?

In gaming environments it is clear how actions affect the environment. Can some rules of RL breakdown if no "noticeable" environment change takes place as a result of actions ?

Update:

"actions influence the state transitions", is my understanding correct: If transitioning to a new state is governed by epsilon greedy and epsilon is set to .1 then with .1 probability the agent will choose an action from the q table which has max reward reward for the given state. Otherwise the agent randomly chooses and performs an action then updates the q table with discounted reward received from the environment for the given action.

I've not explicitly modeled an MDP and just defined the environment and let the agent determine best actions over multiple episodes of choosing either a random action or the best action for the given state, the selection is governed by epsilon greedy.

But perhaps I've not understood something fundamental in RL. I'm ignoring MDP in large part as I'm not modeling the environment explicitly. I don't set the probabilities of moving from each state to other states.

## 1 Answer

A very vague question. What's the objective?

Reinforcement Learning (RL) typically uses the Markov Decision Process framework, which is a sequential decision making framework. In this framework, actions influence the state transitions. In other words, RL deals with controlling (via actions) a Markov chain. The objective in RL is figure out how to take actions in an optimal (in some sense) way!

If, in the application you mentioned, the actions don't influence the state transitions and the objective is to predict states, RL is not required. It's just a regression/ time-series problem.

• I guess one thing that also might be useful to OP is that you could have an MDP where you take an action that does 'nothing' - i.e. taking action $a$ in state $s$ will just transition the agent trivially back into state $s$ with probability 1 and receiving reward 0. May 12, 2020 at 9:41
• @math_pile thanks, please see question update. May 12, 2020 at 10:55
• @blue-sky in your edit your understanding of $\epsilon$-greedy is the wrong way around, with probability 0.9 the agent would take the greedy option (the action that maxes the Q value) and with probability 0.1 act randomly. May 12, 2020 at 11:09
• @blue-sky: $\epsilon-$ greedy is as David pointed out. Coming back to your question, it is still not clear what your objective is. From what I understand, you don't need RL. If there are actions, and the actions might or might not affect the state, then perhaps Bandits framework might be useful (en.wikipedia.org/wiki/Multi-armed_bandit) May 12, 2020 at 15:34