15
$\begingroup$

Is there a way to teach reinforcement learning in applications other than games?

The only examples I can find on the Internet are of game agents. I understand that VNC's control the input to the games via the reinforcement network. Is it possible to set this up with say a CAD software?

$\endgroup$
3
$\begingroup$

You will see a lot of game examples in reinforcement learning literature, because game environments can often be coded efficiently, and run fast on a single computer that can then contain the environment and the agent. For classic games, such as backgammon, checkers, chess, go, then there are human experts that we can compare results with. Certain games or simplified game-like environments are commonly used to compare different approaches, much like MNIST handwritten digits are used for comparing supervised learning approaches.

Is there a way to teach reinforcement learning in applications other than games?

Yes. Informally you could apply reinforcement learning approaches whenever you can frame a problem as an agent acting within an environment where it can be informed of the state and a goal-influencing reward value. More formally, reinforcement learning theory is based upon solutions to Markov Decision Processes, so if you can fit your problem description to a MDP then the various techniques used in RL - such as Q-learning, SARSA, REINFORCE - can be applied. This fit to theory does not need to be perfect for the resulting system to work, for instance you can often treat unknown or imperfectly observed state as effectively random to the agent, and consider this part of a stochastic environment.

Here are some examples of possible uses for reinforcement learning outside of recreational games:

  • Control logic for motorised robot, such as learning to flip pancakes and other examples. Here the environment measurements are made by physical sensors on the robot. The rewards are given for completing a goal, but may also be adjusted for smoothness, economic use of energy etc. The agent chooses low-level actions such as motor torque or relay position. In theory there can be nested agents where higher level ones choose the goals for the lower-level ones - e.g. the robot might decide at a high level between doing one of three tasks that require moving to different locations, and at a lower level might be decisions on how to control motors to move the robot to its chosen goal.

  • Self-driving cars. Although a lot of focus on sensor interpretation - seeing road markings, pedestrians etc, a control system is required in order to select accelerator, brake and steering.

  • Automated financial trading. Perhaps a game to some, there are clear real-world consequences. The reward signal is simple enough though, and RL can be adjusted to prefer long or short term gains.

is it possible to set this up with say a CAD software?

In theory yes, but I do not know what might be available to do this in practice. Also you need one or more goals in mind that you code into the agent (as reward values that it can observe) before giving it a virtual mouse and setting a task to draw something. Computer games come with a reward scheme built in as their scoring system, and provide frequent feedback, so an agent can gain knowledge of good vs bad decisions quickly. You would need to replace this scoring component with something that represents your goals for the CAD-based system.

CAD does not have anything suitable built-in, although CAD tools with simulations, such as various physics engines or finite element analysis, could allow you to score designs based on a simulated physical measure. Other possibilities include analysis of strain, non-wasteful use of material, whatever metrics the CAD/CAM system can provide for a partial or completed design. The tricky part is constraining a design to its goal or purpose and either arranging for that to be rewarded, or building the constraints into the environment; giving an RL agent full unconstrained control of CAD process and rewarding on lowest strain will likely result in something very uninteresting such as a small cube.

$\endgroup$
2
$\begingroup$

One of the cool examples of reinforcement learning is an autonomous flying helicopter. I had a chance to learn some of the stuff done by Andrew Ng and others recently. Here is the research article paper. There are other similar papers too. You can google them if you want to learn more.

You can also see it in action in in this youtube video.

Here is another completely different application in finance apparently.

$\endgroup$
1
$\begingroup$

Reinforcement learning (and, in particular, bandit) algorithms have been and can be used to solve problems other than games, such as

In general, any problem that can be modelled as the maximization of some notion of reward, where you need to interact with some environment (with some states) by taking some actions, can, in principle, be solved by reinforcement learning. Take a look at this pre-print paper (2019) for other applications.

However, note that there are several obstacles that prevent RL algorithms from being widely adopted to solve real-world problems, starting from poor sample complexity (i.e. they require many samples to reach a good performance) or the partial inability to evaluate their performance online without affecting the users.

$\endgroup$
  • $\begingroup$ Your answer doesn't reflect reality when it comes to systems in the real world. Even the papers you link are based on experiments with recorded data not real time systems. I have worked with professors in RL and even they say it's pretty much a disaster in real systems which is why the robotics community largely rejects it even though you see its cherry picked performance in youtube videos. $\endgroup$ – FourierFlux Nov 1 '20 at 14:39
  • $\begingroup$ @FourierFlux Bandits are used by Netflix, so RL is used in the real-world too. Check this netflixtechblog.com/artwork-personalization-c589f074ad76. However, note that I am not saying that they are extensively used. You're right that there are many obstacles that prevent RL from being more widely used. $\endgroup$ – nbro Nov 1 '20 at 14:41
  • $\begingroup$ Bandit problems aren't really what people think about when talking about RL. It's a very simplified case. $\endgroup$ – FourierFlux Nov 1 '20 at 14:42
  • $\begingroup$ @FourierFlux Well, Bandit problems are just special cases of the full RL problem, so they are RL problems too. $\endgroup$ – nbro Nov 1 '20 at 14:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.