0
$\begingroup$

I am trying to get my head around a problem where the action by the agent can not change the environment. Without going into details, my problem is about error correction in an stochastic environment.

So, here the action by agent can not change the environment that causes these error and all we can do is to smartly correct as the errors happen. I am currently thinking about using Reinforcement learning for this agent who could correct the errors.

Now my questions are:

  1. Would reinforcing learning be an overkill since the agent can not influence the environment?
  2. How do RL, LSTM, and even random forest compare in such scenarios?

Thank you.

Improve this question
$\endgroup$
7
  • $\begingroup$ Could you clarify what the action is to "smartly correct"? Is it a numerical value (or vector of values) that adds to some other prediction, or is it a selection from different types of correction (e.g. choose between method "A" vs method "B" for correction)? $\endgroup$
    – Neil Slater
    Commented Aug 31, 2021 at 6:53
  • $\begingroup$ @NeilSlater. Here, by "smartly correct" I mean, if my machine learning approach (whatever it is which is not implemented yet) is successful and it comes up with a strategy then it can "smartly correct" the detrimental effect of the noise. And it is an action. for example, if noise flips a binary bit, the agent should be able to fix that by flipping it again. $\endgroup$
    – user101464
    Commented Aug 31, 2021 at 14:58
  • $\begingroup$ Sorry I don't understand that. Could you please state what form your actions take? Are the actions numerical (e.g. "correct by +1.345") or categorical (e.g. "correct using choice 5 out of 10")? Or is the action literally a vector of bits to flip? I may have further questions to clarify details of your reward/loss function and how you are establishing ground truth, but best to start with details of action. What I am trying to understand is what the supervised learning version of this would look like, in order to figure out the asnwer to your second question. $\endgroup$
    – Neil Slater
    Commented Aug 31, 2021 at 15:12
  • $\begingroup$ Thanks@NeilSlater The action is one of few operations the agent can take, let's say, integer from 1 to 4. For example, 1 is for turning on switch 1, and so on. To be more specific, I few sensors that give me some values, like temperature, how crowded the area is , and so on. Now I would like to find the best strategy, which is turning on and off some switches based on the information I got from sensors to correct the disturbances caused by the environment. As it is obvious, I can not do anything to change the environment that causing these errors, I can not even reduce them. $\endgroup$
    – user101464
    Commented Aug 31, 2021 at 15:53
  • $\begingroup$ OK, so your error correction choice is discrete, one of a few options. How do you then know whether the action is correct? Do you have some ground truth for historical data? Do you have any way to assess an action taken in either simulation or in reality as being the correct action? $\endgroup$
    – Neil Slater
    Commented Aug 31, 2021 at 17:12

1 Answer 1

Reset to default
2
$\begingroup$

Short Intro

It's very common for people to think that Deep Learning is a "superior form" of Neural Network, a "smarter model". And then they try to use DL for solving simple tasks and they'll find more problems than solutions.

We might think of plane as superior to a bike. But when we need to buy some bread for breakfast, taking a plane is not even an overkill. It's just nonsense.

Your problem

In a similar way, I've seen people thinking Reinforcement Learning as somehow superior to Supervised or Unsupervised Learning. So let's establish a baseline:

  • The choice between Reinforcement Learning or Supervised Learning is not about superiority, but rather the nature of the task and your available dataset.

Reinforcement Learning

Great when your problem can be modeled as an agent interacting with an environment. The agent will sense (input), process (policy) and act (output). The policy is learned based on the reward of each action.

In most RL tasks (like strategy games), you can't objectively rate each individual action. Instead, it takes multiple actions before the outcome is obtained. But you can't train your model when you can't measure it's performance. So how to train a RL model?

Policy:

A policy is like a function that maps states into actions. When you can't measure the performance of each individual action, you create a policy.

You let the agent interact with the environment (play the game) until you have a score. If the environment is stochastic (the game depends on luck), you might want several rollouts (play lots of times) before evaluating the policy.

So, for acquiring a single data sample, you might need to let an agent to play a game several times.

But if you can directly measure the agent's perform every each action, congratulations! You can probably save lots of computational power by modeling the problem as a:

Supervised Learning

The model is learned based on the input-output pairs. A training dataset.

LSTM

Great when your input is sequential and your output is a function of the previous state.

Random Forest

Great when your input is not sequential, but you have a lot of features.

Unsupervised Learning

Great when your input is not sequential and you want to discover hidden structure in your data.

I hope this helps you with both questions. :)

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for the insightful categorization. Since my data is sequential, is it correct to say that if I know the best policy for every predicted future ( Let's say coming from LSTM), that would be the same as RL predicting the future and finding the best policy? How important is it that the agent in RL is able to affect the environment? $\endgroup$
    – user101464
    Commented Aug 31, 2021 at 16:08
  • $\begingroup$ I've edited the answer, adding brief description about policy. I hope it answers your question. $\endgroup$
    – Andre Goulart
    Commented Aug 31, 2021 at 18:33
  • $\begingroup$ Thank you, Andre Goulart. It was a very informative answer. $\endgroup$
    – user101464
    Commented Aug 31, 2021 at 20:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .