0
$\begingroup$

For some environments taking an action may not update the environment state. For example, a trading RL agent may take an action to buy shares s. The state at time t which is the time of investing is represented as the interval of 5 previous prices of s. At t+1 the share price has changed but it may not be as a result of the action taken. Does this affect RL learning, if so how ? Is it required that state is updated as a result of taking actions for agent learning to occur ?

In gaming environments it is clear how actions affect the environment. Can some rules of RL breakdown if no "noticeable" environment change takes place as a result of actions ?

Update:

"actions influence the state transitions", is my understanding correct: If transitioning to a new state is governed by epsilon greedy and epsilon is set to .1 then with .1 probability the agent will choose an action from the q table which has max reward reward for the given state. Otherwise the agent randomly chooses and performs an action then updates the q table with discounted reward received from the environment for the given action.

I've not explicitly modeled an MDP and just defined the environment and let the agent determine best actions over multiple episodes of choosing either a random action or the best action for the given state, the selection is governed by epsilon greedy.

But perhaps I've not understood something fundamental in RL. I'm ignoring MDP in large part as I'm not modeling the environment explicitly. I don't set the probabilities of moving from each state to other states.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

A very vague question. What's the objective?

Reinforcement Learning (RL) typically uses the Markov Decision Process framework, which is a sequential decision making framework. In this framework, actions influence the state transitions. In other words, RL deals with controlling (via actions) a Markov chain. The objective in RL is figure out how to take actions in an optimal (in some sense) way!

If, in the application you mentioned, the actions don't influence the state transitions and the objective is to predict states, RL is not required. It's just a regression/ time-series problem.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I guess one thing that also might be useful to OP is that you could have an MDP where you take an action that does 'nothing' - i.e. taking action $a$ in state $s$ will just transition the agent trivially back into state $s$ with probability 1 and receiving reward 0. $\endgroup$
    – David
    May 12, 2020 at 9:41
  • $\begingroup$ @math_pile thanks, please see question update. $\endgroup$
    – blue-sky
    May 12, 2020 at 10:55
  • $\begingroup$ @blue-sky in your edit your understanding of $\epsilon$-greedy is the wrong way around, with probability 0.9 the agent would take the greedy option (the action that maxes the Q value) and with probability 0.1 act randomly. $\endgroup$
    – David
    May 12, 2020 at 11:09
  • $\begingroup$ @blue-sky: $\epsilon-$ greedy is as David pointed out. Coming back to your question, it is still not clear what your objective is. From what I understand, you don't need RL. If there are actions, and the actions might or might not affect the state, then perhaps Bandits framework might be useful (en.wikipedia.org/wiki/Multi-armed_bandit) $\endgroup$
    – math_phile
    May 12, 2020 at 15:34
0
$\begingroup$

It seems to me that you are confusing two things, State of the agent and State of the environment.

Think about a robot learning to walk on a rugged terrain. The actions of the robots don't change the terrain at all ! The robot can include in his own state, part of the environment state, the topology of the terrain in front of him for instance.

In an MDP, an agent may or may not modify the environment state, in your case, if we make the assumption that you don't manipulate the market, you can consider that you don't modify the environment state.

Your issue might be in the way you designed your agent, relying only on a part of the state environment, and not really having any state of his own, such as a portfolio for instance.

If you're only interested in the way the state environment changes, then the above answer is right, the RL framework may not be a good fit.

Finally, you don't have to model explicitly the environment, that's the beauty of model free RL ! Q-learning, that you mentioned, is model-free.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .