2
$\begingroup$

I am trying to implement reinforcement learning into my real-world problem. One thing making me hesitant to apply RL is that this real-world problem of mine is unique in a way how every state is independent of one another. The action taken by the agent at timestep t is the only thing that affects the state at the next timestep. (For example, in the cycle of "state-action-reward-next state", the "next state" is solely dependent on the "action" but not the "state".)

I am wondering if the RL could still be able to learn through this scenario. If not, what other methods could be an option?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ I don't think it would cause problems. The fundamental probability distributions that drive the MDP are $p(s_{t+1}|s_t,a_t)$ and $p(r_{t+1}|s_t,a_t)$. It does not require how these probabilities should look like. In your case, $p(s_{t+1}|s_t,a_t)=p(s_{t+1}|a_t)$ and $p(r_{t+1}|s_t,a_t)=p(r_{t+1}|a_t)$. $\endgroup$
    – user50121
    Oct 16, 2021 at 1:01
  • 1
    $\begingroup$ Welcome to AI stack exchange. A clarifying question: Does the available choice of action depend on the current state in any way? Also, what does the reward depend on? Some variations of your setup, e.g. where all actions are always available, should lead to optimisation of always choosing a single best action. $\endgroup$
    – Neil Slater
    Oct 16, 2021 at 8:21
  • $\begingroup$ @NeilSlater 1. action does not depend on the current state 2. The reward depends on the result/consequence of the action (which is the next state). Observation of this specific state will give the reward at that timestep. 3. (not too sure what to answer for this one) All actions within the action space is always available, hopefully, the policy gets optimized to choosing a best action. $\endgroup$
    – Ykwk
    Oct 19, 2021 at 19:51
  • $\begingroup$ @Ykwk That's what I needed to know - that there is one single best action, always available. I'm guessing the action space is non-trivial and/or very expensive to sample? $\endgroup$
    – Neil Slater
    Oct 19, 2021 at 19:54

1 Answer 1

Reset to default
1
$\begingroup$

You don't have a full reinforcement learning problem, but appear to have a context-free k-armed bandit problem:

  • The start state at time $t$ is essentially irrelevant to the problem. It does not impact available actions, reward or next state.

  • The next state at time $t+1$ is only of interest because it determines the reward.

  • All actions are effectively independent events, unaffected by prior history of the system.

As far as the agent is concerned, you can ignore the state. It may be occurring mechanically within the environment, but the agent can be optimised by observing reward values following each action. It does not need to observe the state, because there is nothing to learn from it, and there is no point in having a policy function with state as an input argument.

If your action space is small, you can use any one of a number of optimisers for k-armed bandits.

If your action space is large, you may need to use a gradient bandit of some kind (very similar to policy gradient methods used in RL, except there is no input layer, since using the state value as input to the function would be counter-productive).

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .