There is definitely a way to introduce what many are calling reinforced learning into real web, mobile, and workstation applications.
Military organizations do it, the movie industry does it, software centered companies are doing it, and I've done it for Fortune 500 businesses and small businesses alike. There are adaptive learning components in all kinds of system components embedded into larger systems, ranging from FaceBook's facial recognition robots to Google Translate to USPS zip code recognition systems to autonomous flight and traffic control systems. Computer aided design software (CAD) is certainly a viable target.
The Basis for Reinforcement
Consider a series of vectors describing events. Imagine they are divided into two sub-series A and B. A neural net (artificial or biological) could be trained using A.
The training could be supervised, meaning one of the dimensions of the vector is considered the label and therefore the dependent variable to optimally predict. The other dimensions then become the facts or input signals and therefore the independent variables to use for prediction. The training could be unsupervised using feature extraction.
Either way, when provided with A prior to B and expected to perform in production (real use) before B arrives, the later arrival of B presents a choice.
- Erase the weights and any meta-parameter adjustments made during the training with A and rerun the training with the concatenated series of A and B.
- Continue the training with B, in which case the network would be biased with A and the result would differ from the result obtained by training with B then A.
- Find a way to limit the bias of having first trained with A while avoiding the resource consumption required for choice #1 above.
Choice #3 is the best choice in many cases because it contains the benefits of choices #1 and #2. Mathematically, #3 is done by facilitating the preempting of what was learned from series A in some way. The neural net weights and meta-parameter adjustments must be made susceptible to correction as new experience indicates the need to do so. One naive approach can be formulated mathematically the inverse exponential function, which models natural decay in many phenomena in physics, chemistry, and social science.
P = e-nt, where P is the probability the fact is still efficacious, n is the rate of decay of past learned information, and t is some measure of forward progress, such as time stamp, sub-sequence (batch) number, fact sequence number, or event number.
In the case of A and B sub-series, when the above formula is implemented in some way in the learning mechanism, the training of A will place less bias on the final result after the continued training using B because the t for A is less than the t for B, telling the mechanism that B is more probably pertinent.
If we recursively divide A and B in halves, creating more and more granular sub-series, the above idea of letting previous information gradually decay remains both valid and valuable. The biasing of the network to the first information used for training is the equivalent of the psychological concepts of narrow-mindedness. Learning systems that have evolved into the brains of mammals seem to forget or lose interest in past things to encourage open-mindedness, which is nothing more than letting new learning sometimes preempt previous learning if the new information contains stronger patterns to learn.
There are TWO reasons for allowing newer example data to progressively outweigh older example data.
- The above removal of the bias of earlier learning to adequately weigh more recent events in further learning makes sense if all the events experienced (trained upon) represent reasonable facts about the external world the system is attempting to learn.
- The external world may be changing and the older learning may actually become irrelevant or even misleading.
This need to let the importance of prior information decay gradually as learning continues is one of the two major aspects of reinforcement. The second aspect is a set of corrective concepts built on the idea of feedback signaling.
Feedback and Reinforcement
A feedback signal in reinforced learning is the machine learning equivalent to familiar psychological concepts like pain, pleasure, contentment, and wellness. The learning system is given information to guide training beyond the goal of feature extraction, independence of groupings, or finding a neural net weight matrix that approximates the relationship between input event features and their labels.
The information provided may originate internally from pre-programmed pattern recognition or externally from reward and punishment, as is the case with mammals. The techniques and algorithms being developed in reinforced machine learning use these additional signals frequently (using time slicing in the processing) or continuously using the independence of processing units of parallel processing architectures.
This work was pioneered at MIT by Norbert Wiener and set forth in his book Cybernetics (MIT Press 1948). The word Cybernetics comes from an older word that means steering of Ships. The automatic movement of a rudder to stay on course may have been the first mechanical feedback system. Your lawn mower engine probably has one.
Adaptive Applications and Learning
Simple adaptation in real time for a rudder position or a lawnmower throttle is not learning. Such adaptation is usually some form of linear PID control. Machine learning technology being expanded today embraces the assessment and control of complex, nonlinear systems that mathematicians call chaotic.
By chaotic, they do not mean that the processes described are in a frenzy or are disorganized. Chaoticians discovered decades ago that simple non-linear equations can lead to highly organized behavior. What they mean is that the phenomenon is too sensitive to slight changes to find some fixed algorithm or formula to predict them.
Language is like that. The same statement said with a dozen different vocal inflections can mean a dozen different things. The English sentence, "Really," is an example. It is likely that reinforcement techniques will allow future machines to distinguish with high probabilities of success between the various meanings of that statement.
Why Games First?
Games have a very simple and easily defined set of possible scenarios. One of the major contributors to the advent of the computer, John von Neumann, argued in Theory of Games and Economic Behavior, a book he co-authored with Oskar Morgenstern, that all planning and decision making is actually game playing of various complexities.
Consider games the training example set of the collection of brains that will, in time, create systems that can determine the meaning of a statement like educated people can from three sources of hints.
- Context within a conversation or social scenario
- The vocal inflections of the speaker
- The facial expressions and body language of the speaker
Beyond Chess and The Game of Go
Along the path from games to language systems with accurate comprehension and deeper listening capabilities there are several applications of reinforced learning that are of much greater importance to the earth and the human experience.
- Systems that learn how to shut off or attenuate lights, appliances, digital systems, HVAC, and other energy consuming devices — Energy is perhaps the most geo-politically influential commodity in human history because of fossil fuel resource depletion over time.)
- Autonomous vehicle development — The dangerous trend of the operation of heavy equipment, like aircraft, RVs, trucks, buses, and tractor trailers by people in unknown states of mind on open roads will likely be looked back upon by future people as insanity.
- The rating of information reliability — Information is everywhere and over 99% of it is in error, either partially or completely. Very little is authenticated by real research, either properly designed and interpreted double-blind randomized studies or confirmable laboratory testing and analysis.
- Health care applications that better diagnose, tailor remedies to the individual, and assist with continued care to avert recurrence.
These four and many others are far more important than wealth accumulation via automated high speed trading or winning game competitions, two self centered machine learning interests that merely impact one or two generations of a single person's family.
Wealth and fame are what in game theory is called a zero sum game. They produce as many losses as there are winnings if you consider the higher Golden Rule philosophy that others and their families are of equal importance to us.
Reinforced Learning for CAD (Computer Aided Design) Software
Computer aided design is the naturally forerunner of computer design (without aid from humans), just as anti-lock breaks naturally leads to fully autonomous vehicles.
Consider the command, "Create me a soap dish for my shower that maximizes the likelihood my family can grab the soap on the first try without opening their eyes and minimizes the difficulty in keeping the soap and the shower surfaces clean. Here are the heights of my family members and some pictures of the shower space." Then a 3D printer would pop out the device, ready to attach, along with the installation instructions.
Of course, such a CD system (CAD without the A) would need to be trained in housekeeping, human behavior without vision, ways of attaching items to tile, the tools and home maintenance capabilities of the average consumer, the capabilities of the 3D printer, and several other things.
Such developments in manufacturing automation would probably start with reinforced learning of simpler commands like, "Attach these two parts using mass produced fasteners and best practices." The CAD program would then pick hardware from among screws, rivets, adhesives, and other options, perhaps asking the designer questions about operating temperature and vibration ranges. The choice, position, and angle would then be added to the appropriate set of CAD parts and assembly drawings and bills of materials.