12
$\begingroup$

Is there a way to teach reinforcement learning in applications other than games?

The only examples I can find on the Internet are of game agents. I understand that VNC's control the input to the games via the reinforcement network. Is it possible to set this up with say a CAD software?

$\endgroup$
  • 3
    $\begingroup$ Yes, it is possible but the real question should be whether it's more efficient than other algorithms or not. If you have a specific goal you can think of it and model it as a game (even life is a game ;) ). In fact, many successful AI's taking advantage of more than one technique. Take a look at this one: datascience.stackexchange.com/questions/11126/… $\endgroup$ – TasosGlrs Jun 16 '17 at 9:22
  • 2
    $\begingroup$ The point about AlphaGo is that a certain type of game (non-trivial, non-chance, perfect information) in particular provides an excellent proving ground for AI because these games have very simple parameters, but complexity akin to nature. Games, from a certain perspective, are the most useful kinds of artifacts because they teach abstract and strategic thinking. To understand the importance of games, look at the impact of Game Theory on computer science. $\endgroup$ – DukeZhou Jun 16 '17 at 17:00
  • $\begingroup$ That may be because of NDAs. The best learning apps are proprietary, as are the best cell tower software or the best voice recognition or the best online shopping centers. That's engineering in general. $\endgroup$ – FauChristian Jun 18 '18 at 1:19
  • $\begingroup$ Right. If all one has is game theory, the world becomes a game. $\endgroup$ – FelicityC Nov 7 '18 at 18:48
2
$\begingroup$

One of the cool examples of reinforcement learning is an autonomous flying helicopter. I had a chance to learn some of the stuff done by Andrew Ng and others recently. Here is the research article paper. There are other similar papers too. You can google them if you want to learn more.

You can also see it in action in in this youtube video.

Here is another completely different application in finance apparently.

$\endgroup$
2
$\begingroup$

You will see a lot of game examples in reinforcement learning literature, because game environments can often be coded efficiently, and run fast on a single computer that can then contain the environment and the agent. For classic games, such as backgammon, checkers, chess, go, then there are human experts that we can compare results with. Certain games or simplified game-like environments are commonly used to compare different approaches, much like MNIST handwritten digits are used for comparing supervised learning approaches.

Is there a way to teach reinforcement learning in applications other than games?

Yes. Informally you could apply reinforcement learning approaches whenever you can frame a problem as an agent acting within an environment where it can be informed of the state and a goal-influencing reward value. More formally, reinforcement learning theory is based upon solutions to Markov Decision Processes, so if you can fit your problem description to a MDP then the various techniques used in RL - such as Q-learning, SARSA, REINFORCE - can be applied. This fit to theory does not need to be perfect for the resulting system to work, for instance you can often treat unknown or imperfectly observed state as effectively random to the agent, and consider this part of a stochastic environment.

Here are some examples of possible uses for reinforcement learning outside of recreational games:

  • Control logic for motorised robot, such as learning to flip pancakes and other examples. Here the environment measurements are made by physical sensors on the robot. The rewards are given for completing a goal, but may also be adjusted for smoothness, economic use of energy etc. The agent chooses low-level actions such as motor torque or relay position. In theory there can be nested agents where higher level ones choose the goals for the lower-level ones - e.g. the robot might decide at a high level between doing one of three tasks that require moving to different locations, and at a lower level might be decisions on how to control motors to move the robot to its chosen goal.

  • Self-driving cars. Although a lot of focus on sensor interpretation - seeing road markings, pedestrians etc, a control system is required in order to select accelerator, brake and steering.

  • Automated financial trading. Perhaps a game to some, there are clear real-world consequences. The reward signal is simple enough though, and RL can be adjusted to prefer long or short term gains.

is it possible to set this up with say a CAD software?

In theory yes, but I do not know what might be available to do this in practice. Also you need one or more goals in mind that you code into the agent (as reward values that it can observe) before giving it a virtual mouse and setting a task to draw something. Computer games come with a reward scheme built in as their scoring system, and provide frequent feedback, so an agent can gain knowledge of good vs bad decisions quickly. You would need to replace this scoring component with something that represents your goals for the CAD-based system.

CAD does not have anything suitable built-in, although CAD tools with simulations, such as various physics engines or finite element analysis, could allow you to score designs based on a simulated physical measure. Other possibilities include analysis of strain, non-wasteful use of material, whatever metrics the CAD/CAM system can provide for a partial or completed design. The tricky part is constraining a design to its goal or purpose and either arranging for that to be rewarded, or building the constraints into the environment; giving an RL agent full unconstrained control of CAD process and rewarding on lowest strain will likely result in something very uninteresting such as a small cube.

$\endgroup$
  • $\begingroup$ Plenty of suitable things: Auto-dimensioning for maximal visual clarity, interference searches for moving parts under strain (without FEA), optimizing CAM output, rendering acceleration using approximate z-depth information, etc, etc $\endgroup$ – FauChristian Jun 18 '18 at 1:24
  • $\begingroup$ @FauChristian: I'm not sure all of those could be fed in as a reward signal as simply as a game score. Optimising a design for reduced strain for instance implies your design has a purpose - you have to also add some constraints/rules to track that purpose, and that could be harder unless you also have some ergonomic analysis. However, the comment adds some things I could mention, and will add to the answer. $\endgroup$ – Neil Slater Jun 18 '18 at 5:29
  • $\begingroup$ Yes. True. I'll try. Please excuse the string of comments I'll need to do so. In each case above my goal will be to aggregate the concerns of a mechanical designer into a single wellness signal that could guide iterations, with the truth in mind that there may be multiple organs of NN each of which may be fed a different aggregation. But for simplicity's sake, I'll aggregate each case to a scalar. Having a stochastic element will be needed for most of these because most cases in CAD have multiple critical points in the aggregated value's surface. $\endgroup$ – FauChristian Jun 25 '18 at 8:51
  • $\begingroup$ Auto Dimensioning — w = sqrt(Σ min(s_clear, s_nice)) + k n_jumps, ... where ... w is the quality aggregate of the state of the drawing's dimensioning, from which a normalized feedback signal can be derived, s_clear is the distance between a dimensioning line and the nearest other line excluding jump lines, s_nice is a metaparameter representing a nice distance of clearance between lines for the type of drawing being dimensioned, k is a constant, and n_jumps is the number of jump lines (where lines would cross but one of the two has a gap to indicate it is jumping behind the other line). $\endgroup$ – FauChristian Jun 25 '18 at 9:21
  • $\begingroup$ Interferance Searches — w = n, ... where ... w is the quality aggregate of the interferences search and n is the number of interferences found after feeding the iterations guesses into a dynamic simulation of the guess. This one is just like a game in that the more correct interference guesses the higher the score. $\endgroup$ – FauChristian Jun 25 '18 at 9:24
1
$\begingroup$

There is definitely a way to introduce what many are calling reinforced learning into real web, mobile, and workstation applications.

Military organizations do it, the movie industry does it, software centered companies are doing it, and I've done it for Fortune 500 businesses and small businesses alike. There are adaptive learning components in all kinds of system components embedded into larger systems, ranging from FaceBook's facial recognition robots to Google Translate to USPS zip code recognition systems to autonomous flight and traffic control systems. Computer aided design software (CAD) is certainly a viable target.

The Basis for Reinforcement

Consider a series of vectors describing events. Imagine they are divided into two sub-series A and B. A neural net (artificial or biological) could be trained using A.

The training could be supervised, meaning one of the dimensions of the vector is considered the label and therefore the dependent variable to optimally predict. The other dimensions then become the facts or input signals and therefore the independent variables to use for prediction. The training could be unsupervised using feature extraction.

Either way, when provided with A prior to B and expected to perform in production (real use) before B arrives, the later arrival of B presents a choice.

  1. Erase the weights and any meta-parameter adjustments made during the training with A and rerun the training with the concatenated series of A and B.
  2. Continue the training with B, in which case the network would be biased with A and the result would differ from the result obtained by training with B then A.
  3. Find a way to limit the bias of having first trained with A while avoiding the resource consumption required for choice #1 above.

Choice #3 is the best choice in many cases because it contains the benefits of choices #1 and #2. Mathematically, #3 is done by facilitating the preempting of what was learned from series A in some way. The neural net weights and meta-parameter adjustments must be made susceptible to correction as new experience indicates the need to do so. One naive approach can be formulated mathematically the inverse exponential function, which models natural decay in many phenomena in physics, chemistry, and social science.

P = e-nt, where P is the probability the fact is still efficacious, n is the rate of decay of past learned information, and t is some measure of forward progress, such as time stamp, sub-sequence (batch) number, fact sequence number, or event number.

In the case of A and B sub-series, when the above formula is implemented in some way in the learning mechanism, the training of A will place less bias on the final result after the continued training using B because the t for A is less than the t for B, telling the mechanism that B is more probably pertinent.

If we recursively divide A and B in halves, creating more and more granular sub-series, the above idea of letting previous information gradually decay remains both valid and valuable. The biasing of the network to the first information used for training is the equivalent of the psychological concepts of narrow-mindedness. Learning systems that have evolved into the brains of mammals seem to forget or lose interest in past things to encourage open-mindedness, which is nothing more than letting new learning sometimes preempt previous learning if the new information contains stronger patterns to learn.

There are TWO reasons for allowing newer example data to progressively outweigh older example data.

  • The above removal of the bias of earlier learning to adequately weigh more recent events in further learning makes sense if all the events experienced (trained upon) represent reasonable facts about the external world the system is attempting to learn.
  • The external world may be changing and the older learning may actually become irrelevant or even misleading.

This need to let the importance of prior information decay gradually as learning continues is one of the two major aspects of reinforcement. The second aspect is a set of corrective concepts built on the idea of feedback signaling.

Feedback and Reinforcement

A feedback signal in reinforced learning is the machine learning equivalent to familiar psychological concepts like pain, pleasure, contentment, and wellness. The learning system is given information to guide training beyond the goal of feature extraction, independence of groupings, or finding a neural net weight matrix that approximates the relationship between input event features and their labels.

The information provided may originate internally from pre-programmed pattern recognition or externally from reward and punishment, as is the case with mammals. The techniques and algorithms being developed in reinforced machine learning use these additional signals frequently (using time slicing in the processing) or continuously using the independence of processing units of parallel processing architectures.

This work was pioneered at MIT by Norbert Wiener and set forth in his book Cybernetics (MIT Press 1948). The word Cybernetics comes from an older word that means steering of Ships. The automatic movement of a rudder to stay on course may have been the first mechanical feedback system. Your lawn mower engine probably has one.

Adaptive Applications and Learning

Simple adaptation in real time for a rudder position or a lawnmower throttle is not learning. Such adaptation is usually some form of linear PID control. Machine learning technology being expanded today embraces the assessment and control of complex, nonlinear systems that mathematicians call chaotic.

By chaotic, they do not mean that the processes described are in a frenzy or are disorganized. Chaoticians discovered decades ago that simple non-linear equations can lead to highly organized behavior. What they mean is that the phenomenon is too sensitive to slight changes to find some fixed algorithm or formula to predict them.

Language is like that. The same statement said with a dozen different vocal inflections can mean a dozen different things. The English sentence, "Really," is an example. It is likely that reinforcement techniques will allow future machines to distinguish with high probabilities of success between the various meanings of that statement.

Why Games First?

Games have a very simple and easily defined set of possible scenarios. One of the major contributors to the advent of the computer, John von Neumann, argued in Theory of Games and Economic Behavior, a book he co-authored with Oskar Morgenstern, that all planning and decision making is actually game playing of various complexities.

Consider games the training example set of the collection of brains that will, in time, create systems that can determine the meaning of a statement like educated people can from three sources of hints.

  1. Context within a conversation or social scenario
  2. The vocal inflections of the speaker
  3. The facial expressions and body language of the speaker

Beyond Chess and The Game of Go

Along the path from games to language systems with accurate comprehension and deeper listening capabilities there are several applications of reinforced learning that are of much greater importance to the earth and the human experience.

  • Systems that learn how to shut off or attenuate lights, appliances, digital systems, HVAC, and other energy consuming devices — Energy is perhaps the most geo-politically influential commodity in human history because of fossil fuel resource depletion over time.)
  • Autonomous vehicle development — The dangerous trend of the operation of heavy equipment, like aircraft, RVs, trucks, buses, and tractor trailers by people in unknown states of mind on open roads will likely be looked back upon by future people as insanity.
  • The rating of information reliability — Information is everywhere and over 99% of it is in error, either partially or completely. Very little is authenticated by real research, either properly designed and interpreted double-blind randomized studies or confirmable laboratory testing and analysis.
  • Health care applications that better diagnose, tailor remedies to the individual, and assist with continued care to avert recurrence.

These four and many others are far more important than wealth accumulation via automated high speed trading or winning game competitions, two self centered machine learning interests that merely impact one or two generations of a single person's family.

Wealth and fame are what in game theory is called a zero sum game. They produce as many losses as there are winnings if you consider the higher Golden Rule philosophy that others and their families are of equal importance to us.

Reinforced Learning for CAD (Computer Aided Design) Software

Computer aided design is the naturally forerunner of computer design (without aid from humans), just as anti-lock breaks naturally leads to fully autonomous vehicles.

Consider the command, "Create me a soap dish for my shower that maximizes the likelihood my family can grab the soap on the first try without opening their eyes and minimizes the difficulty in keeping the soap and the shower surfaces clean. Here are the heights of my family members and some pictures of the shower space." Then a 3D printer would pop out the device, ready to attach, along with the installation instructions.

Of course, such a CD system (CAD without the A) would need to be trained in housekeeping, human behavior without vision, ways of attaching items to tile, the tools and home maintenance capabilities of the average consumer, the capabilities of the 3D printer, and several other things.

Such developments in manufacturing automation would probably start with reinforced learning of simpler commands like, "Attach these two parts using mass produced fasteners and best practices." The CAD program would then pick hardware from among screws, rivets, adhesives, and other options, perhaps asking the designer questions about operating temperature and vibration ranges. The choice, position, and angle would then be added to the appropriate set of CAD parts and assembly drawings and bills of materials.

$\endgroup$
  • $\begingroup$ I would not normally make a critical comment on answer this well written, but since you asked for critique: I think that the sections The Basis for Reinforcement, Feedback and Reinforcement and Adaptive Applications and Learning whilst interesting, could be cut down considerably, as the OP's question already implies some knowledge of RL, and you spend a lot of time setting up this related background discussion before addressing the original question. $\endgroup$ – Neil Slater Jun 13 '18 at 7:34
  • $\begingroup$ That the OP asked the question indicates holes either the current state of research indicative of someone new to the field and may need the primer in the earlier sections. The last section more directly answers the question. $\endgroup$ – FauChristian Jun 13 '18 at 8:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.