Why is Reinforcement Learning viable compared to just only using Neural Network? [duplicate]

Question

I am confused as to how RL is viable compared to just using a simple NN.

I have data such as the following:

x1 | x2 | x3 | y
3  | 5  | 7  | 6
4  | 3  | 2  | 15
... and so on

where x are the input and y is the output. Let's say I am given x1, x2 and I need to find x3 such that y is as close as 10. I can just simply do neural network on the dataset above, set the goal of y to be 10 and do a 'solver' to find the optimal x3.

My dilemma with reinforcement learning is that if you just restate this problem such that x1 x2 are now called states, x3 is now action and -abs(y-10) as the reward function, this problem seems like it can be solved using RL as well. Maybe it is just like an off-line RL? This leads to my question of how could RL be useful if just a neural network can used to handle this kind of problem, especially when RL and NN both are identically solving for the best action or the x3, respectivley?

I am sure I am overlooking some fundamental knowledge here.

Adding on: I realized that I am missing the state transition or the 'next state' part of RL. But even for the next state, wouldn't NN be able to solve for what the best action/x3 is for that given state or the x1,x2?

Answer 1

Your analysis seems correct, that both approaches - searching using a neural network approximator for all three params, and using a trained reinforcement learning (RL) agent that selects the third param as an "action" - should work. The quality of the solution, given a fixed training set, is likely to be very similar if not identical. The main difference would be that training the RL agent would take longer since it is driven indirectly via the reward function and exploring alternative actions as opposed to directly from loss function on the approximation for $y$.

You note that there is no state transition (or time step) in the problem, and that does mean RL is over-specified. A related approach, called contextual bandits, would be a closer match. Again in your example, it will be slower because of trial-and-error search for solutions compared to directly approximating the function.

In general, if you have a fixed dataset and simple-to-express goal in terms of optimising a function that you know, then it can be posed as an RL problem. RL solvers will then usually work, but will most often be inefficient compared to other optimisation approaches.

RL, and learning through trial-and-error, is a very general learning mechanism. It can often be adapted and applied to scenarios where other algorithms also work. As you have done in the question, the basic approach is to map state, actions and rewards from the original problem description to frame it as RL. In your specific case, you must train the agent both offline and off-policy, because the agent cannot actively make new guesses for x3.

If this problem framing is done accurately, then RL can be applied. However, this is not always a good idea.

The more "natural" the fit of a problem is to the concepts of agent, environment, state, action and reward, then the more likely it is that RL will be a good fit to the problem. Unlike your example, it can be applied in situations where you do not have a dataset and the agent must actively collect experience.

Also unlike your example, RL can be applied in scenarios without target variables to learn, using a reward signal directly as feedback. Although sometimes explicitly learning target variables from the environment other than expected return or optimal action could help an agent (e.g. by regularising internal feature representations in its neural network if this is done using a multi-headed NN).