I'm here to ask you for a solution on this problem which is: how to use Reinforcement Learning in Immersive Virtual Reality to make a person move to a specific location in a virtual environment. As you know reinforcement Learning is a sub-area of Machine Learning in which an active entity called an agent interacts with its environment and learns how to act in order to achieve a pre-determined goal. The Reinforcement Learning had no prior model of behaviour and the participants no prior knowledge that their task was to move to and stay in a specific place. The participants were placed in a virtual environment where they had to avoid collisions with virtual projectiles. Following each projectile the agent analysed the movement made by the participant to determine paths of future projectiles in order to increase the chance of driving participants to the goal position and make them stay there as long as possible.
Selecting a Scenario for Comprehension's Sake
Movement through a modeled reality is an area that was under development when I entered the research community. It is not a problem though. It is a nearly infinite set of problems, an area of research of interest to robotics engineering and gaming.
The solution cannot be very specific when so many details are left out of the scenario definition. Although I am fine with specifying a general solution, pure math and system architecture is not met with much enthusiasm by most of those interested in answers in this exchange, so I will make some assumptions.
Those with good mathematical and systems design backgrounds will be able to extrapolate the general approach from the more specific scenario in this answer. I'll place pertinent general theory inline to facilitate that generalization.
Narrowing the Specification Incompletely
- The vehicle for movement is not specified, so I will assume the participants are knights on horses. Linear movements in Cartesian coordinates as in a CAM move command for a CNC machine head is an unlikely use of reinforcement, and aircraft lift can stall, complicating the problem. Horses were picked from among the remaining list of typical vehicles (cars, bicycles, on foot, and horses).
- Time constraints were not specified, so it will be assumed there are none except the time to reach the goal cannot be infinite unless there is no available path at all to the destination.
- The nature of the obstacles and their movements were not specified, so I will assume they are solid and that only other knights move on their own horses.
- Risks associated with making contact with an obstacle was not specified, so I will assume the horses will not run into one another or stationary objects and will step over small ground objects or jump over slightly larger ones if approaching at sufficient speed.
- What will happen when a collision is about to occur was not specified, so I will assume that the horse will decelerate.
- What will happen if a horse passes under an obstacle will be assumed the hospitalization or death of the knight, since they cannot easily duck in their armor, leaving the objective unmet.
- The capabilities of the participants is not specified, so I will assume that the knights can see stereoscopically, are equipped with compasses, can turn their head and eyes, can control the reigns in the usual horseback riding way, can urge the horse forward with the vocal sound, "Ya!" and/or a light poke with both feet, hear and detect the volume of an auditory beacon, and sense acceleration in 3D to detect horse behavior.
- The specs of the other participants is not given, so I will assume the knights along with their horses are of equal volume.
- The obstacle statistics are not given, so I will assume the objects to be chaotically sized, shaped, colored, and placed such that their total volumetric space consumption is 1% and the mean object volume is the same as the volume of a knight and her or his associated horse.
- The scene is lit from one distant source so that shadows and shadowing are present and the entire scene is bounded by constants in altitude, longitude, and latitude dimensions.
- The coordinate system is not specified, the system is modeling equatorial terrestrial space, so latitude and longitude coordinates are not substantially ansiotropic.
- The start and end positions are of specific latitude and longitude and are sufficiently distant from one another so that line of sight between the start and end positions is extremely unlikely.
- How the end position will be known is not specified, so I will assume the end position is equipped with an auditory beacon.
- We will assume a zero sum game, where the achievement of the goal is not mutually exclusive, and we will assume collaboration linguistically is not possible, although collaborative strategy may emerge out of learned behavior organically. (That's an entire other topic.)
Mathematical precision is missing in the above definition, with several parameters only roughly defined, and the chaotic sizing, shaping, and positioning becomes a realistic challenge for reinforcement software engineering for the general scenario.
Conversely, engineering a solution for this specific case within the field of VR motion with the application of reinforcement concepts can proceed without mathematical abstractions that require much advance (and advanced) study and laboratory experience.
The above adequately defines system E (environment) in combination with system V (vehicles) of which there are N, one for each participant. Discrete changes to command signals C are multidimensional and A (acquired samples) are also multidimensional.
- Sampled audio vectors of spectral spectral distributions in musical half tones (frequency ratio of 21/24), sampled in frames of constant period.
- Sampled visual matrices in yuv420p form, sample
- Sampled tactile vectors of 3D acceleration force
The audio and vectors are maintained at constant levels as inputs to the base neural net until the next vector is acquired. The video matrices are fed into a convolutional neural net as is customary. (See the work of Google researchers Sergey Levin, Peter Pastor, Alex Krizhevsky, and Deirdre Quillen in August 2016 for their approach and references for background research that guided them.)
- Head position relative to the vehicle direction
- Left reign position
- Right reign position
- Activation of, "Ya!" vocalization
- Activation of the light poke
The output layer of the base neural net must be a real number with the appropriate range for all five, since the volume of the vocalization and the lightness of the poke have informational meaning to the motor control of the vehicle.
Real Time Learning
Real time learning requires at least one model of wellness to provide the reinforcement signal to the base behavioral network. In many cases, as suggested in early cybernetics work prior to the advent of digital systems, convergence requires more than one wellness variable.
Vector control of reinforcement is not well developed in open software yet, however the concepts of multidimensional gradients (Jacobian and Hessian matrices) are standard elements in artificial network theory and can be extended from matrices to cubes. Any intermediate calculus text will provide the theory applicable to gradient descent with a curved surface.
Performing the back-propagation effectively when more than one degrees of freedom are present is an interesting problem that is certainly researched and has been deployed to production in commercial and military applications.
Such cannot be described here because the mechanics of back-propagation with vector reinforcement signaling of which I am aware is currently either company confidential or classified. As with much technology, it may be released for publication in the future as open source code emerges independently over time.
These are probably the best choices for the first and second channels of learning reinforcement signaling to implement and tune. I doubt (but have only intuition to offer as reasoning) that only one channel will produce a very effective reinforced learning. The base network necessary for this scenario will be too deep to train with out both types of proximity estimations (1) beacon and (2) nearest obstacle.
Wellness of Behavior Modelling
The simplest and first cut at modelling participant behavioral wellness (proficiency in searching for the beacon) is the rate of change in beacon volume. It is a distorted estimate of proximity to the beacon, but systematically so. The distortions in the correlation between the differential of volume with respect to position (not time) is related to the effects of obstacles on sound.
Wellness of Position Modelling
Further development could add filtering out of transient attenuation created by close proximity to an object, which would require two additional neural nets, (a) to detect closest object proximity from patterns in audio and visual inputs and (b) to use acceleration to approximate change in position in the latitude-longitude plane and correlate filtering of audio volume changes with movement to better filter out transient changes not related to proximity of the beacon.
The tone of the beacon may be added as a third to improve rate of learning.
Crash level accelerations can be added over multiple game plays as a fourth.
The determination of initial state and meta-parameters for the base neural network and the connectivity between the models and the reinforcement signal to the base network is beyond the scope of this answer, requiring experimentation and possibly months of intensive analysis for this (or any) semi-specific case.
The only known systems that handle general cases without defining environment E, participant P, and their quantity N are DNA based systems that have developed such general adaptive capabilities over billions of years.