I wonder if Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR) use any machine learning or deep learning?

For example in AR, the virtual objects are brought into the real world, does this process involve any object detection and localization?


2 Answers 2



Yes for all the 3 cases



Even if the environment is virtual there is another thing to perceive which or better who is real: the user

Deep Learning can be used for things like: eye tracking, gesture recognition, voice recognition, ...

Furthermore, in games, you could think about more complex applications like understanding the way the user plays the game and react accordingly

AR and MR

In addition to what has been said for VR, in these cases you also have to integrate the environment in the final user experience so you ultimately have to perceive it and there multiple specific applications: object detection, planes detection, depth estimation, Visual Odometry (e.g. as feature detection module), SLAM, ...

These days Deep Learning is achieving Starte-Of-The-Art (SOTA) in many different tasks far beyond the classification and detection, but one things is research so published papers and so on and another thing is a product so to sum up every VR, MR, AR vendor is probably relying on an hybrid solution where traditional algorithms and deep learning coexist but so far it seems the trend is deep learning is eating the traditional algos up day after day


The rising force of deep learning in VR and AR

Virtual Reality And Machine Learning Go Hand In Hand

  • $\begingroup$ Thanks for the answer. What are modyle and SOTA? $\endgroup$
    – Lei Xun
    Jun 1, 2019 at 7:29
  • $\begingroup$ Hi @LeiXun, thanks for spotting typos and not clear parts: I have edited my answer accordingly (SOTA = State of the Art which means the best performance it is possible to achieve today with science and technology, so not cheating :)) $\endgroup$ Jun 1, 2019 at 8:53

Certainly in Virtual Reality the new Deep Convolutional Neural Networks algorithms applied in image processing are playing a role! Object detection, segmentation and semantic imaging are all features of these new ways of image processing. Augmented Reality then uses some advanced Optics features to accomplish its job. And so on...

  • 1
    $\begingroup$ Welcome to SE:AI! This answer could use some links to support it. $\endgroup$
    – DukeZhou
    Jun 3, 2019 at 17:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .