The immersive technologies of virtual reality (VR) and augmented reality (AR) meld virtual and physical experiences. Both VR and AR rely on well-timed motion tracking, visual rendering and display to ensure a seamless experience. In both virtual reality and augmented reality experiences, users feel as though they are moving around and interacting with objects the same way they would in the real, physical world.
In Virtual Reality, we replace a user’s sensory experience in the physical world with stimuli that represent a virtual world. In Augmented Reality, we digitally insert virtual objects alongside physical objects. In both of these, we’re fusing virtual experiences with physical experiences, bridging these virtual and physical realities together.
Game Engines let developers create content, and when running the applications, the game engine will render virtual content, especially the visuals and audio that a moving virtual camera sees in the virtual scene. The crux behind VR and AR is in estimating the device’s pose: where it’s located and where it’s pointed. VR and AR frameworks use that pose to update the game engine’s virtual camera so that movement in the physical world creates corresponding motion in the virtual world.
The devices then output the rendered game engine visuals through screen displays and lenses that properly feed the image into the user’s eyes. So as the user’s head or phone moves around, the virtual camera moves with it, and the visuals update. However, it’s very important that the motion tracking, visual rendering, and display is done as swiftly as possible. If the user moves their head and the display is slow to update, moving later, then this can cause motion sickness, especially in virtual reality. Researchers measure “motion to photon latency”, counting the time between when the head moves and when the displayed light photons account for the updated position. Device manufacturers do everything they can to bring motion-to-photon latency as low as possible, aiming to be less than 10 ms from head movement to device update.
However, when done properly, VR systems create the illusion that VR users are moving around in a virtual world the same way they would be moving around in a physical world. AR systems likewise provide the illusion that AR virtual objects belong right there among your physical world’s objects. Through some clever coordination of tracking and rendering, our systems can fuse virtual and physical worlds together.