Virtual reality

Virtual reality is a computer-simulated environment simulating physical presence in real or imagined worlds. Users wear hardware devices such as goggles, headphones, gloves, or use controllers to allow them to interact with the virtual world.

Most virtual reality systems work to track a user's motions and eye movements to correspondingly adjust the user's display to reflect the perspective changes. This is to create or simulate immersion in the virtual world. With their movements, users should experience a shift in environment as they move through it, a corresponding shift in any three-dimensional sound-scaping, and, if the hardware permits, haptic feedback. Any latency, also known as lag, in the virtual reality system reacting to the user's real-time movements can negatively impact the immersion of the virtual reality experience and has caused motion sickness in some users.

Developed in 1957 by Morton Heilig, a filmmaker, the Sensorama was a large booth-like machine intended to combine multiple technologies to give up to four people the illusion of being in a fully immersive world. This included smell, stereo sound, vibrations, and even atmospheric effects. In 1960, Morton Heilig refined the Sensorama into a headset that promised stereoscopic 3D images, wide vision, and stereo sound. This was the earliest patent for a head-mounted display.

Image of the Sensorama by Morton Heilig.

Despite Morton Heilig's patent for a head-mounted Sensorama, Ivan Sutherland created the first head-mounted display (HMD) for virtual reality in 1968. The early HMD design connected a stereoscopic display depicting simple virtual wireframe shapes that changed perspectives as the user moved their head. The HMD was suspended from the ceiling on a mechanical arm, and a user could only stand in one spot while wearing it.

From this project, Thomas Furness developed a flight simulator program known as the "Super Cockpit" in 1986. This program gave a trainee pilot a head-mounted display capable of projecting computer-generated 3D maps, infrared, radar imagery, and avionics data. The Super Cockpit featured integration between movement tracking and aircraft control.

The 1980s also saw the founding of VPL Research, headed by Jaron Lanier. They began developing a range of virtual reality gear, including the Dataglove and the EyePhone head-mounted display. These offered a development in virtual reality haptics. Similar to this, Nintendo released the Nintendo Power Glove, which used technology similar to VPL's Dataglove and a simulation glove developed by NASA.

In 1991, head-mounted displays made their way into arcade machines, where users would play a game with immersive stereoscopic 3D visuals but with limited haptic feedback. SEGA intended to release a virtual reality headset for the Sega Genesis console in 1993, and in 1995 Nintendo released the Nintendo Virtual Boy, a 3D game console. Both of SEGA and Nintendo's ventures into virtual reality failed commercially, but they represented some of the first attempts to bring virtual reality into people's homes.

In 2010, Oculus VR began developing their Rift head-mounted display. This became and influenced the popular commercial application of virtual reality, which uses a HMD and controllers to allow a user to immerse themselves in a virtual world. Users can do this at their computer desk or in their living room. Most of these systems, including the Oculus Rift and the HTC Vive, offer stereoscopic 3D imaging, stereo sound, two handheld controllers (some with limited haptic feedback), and the option of cameras to offer greater movement tracking. Many of these headsets also use accelerometers, gyroscopes, or both, to allow a headset to track a user's head movements and offer a virtual reality experience free from cameras to track movement.

In 2018, Oculus released their Oculus Go and Oculus Quest headsets, which were two standalone virtual reality systems that did not need a computer, games console, or mobile phone to work. These systems also worked to increase the affordability of virtual reality headsets.

Head-mounted displays have been paired with various input devices:

• joysticks
• force balls and tracking balls
• controller wands
• datagloves
• voice recognition
• motion trackers and bodysuits
• treadmills

Other virtual reality systems include virtual environments, or Cave Automatic Virtual Environments (CAVE). These systems project images on the walls, floor, and ceiling of a room. Users wear glasses to help the immersion of the virtual environment. With CAVE systems, groups of people can share the virtual reality experience, and some systems allow them to play as part of a team. The CAVE works through projectors that are positioned above or outside of a room and are controlled by a user's physical movement. The glasses worn by users are synchronized with the projectors and convey separate images per eye so the user sees the correct image at the correct time to maintain immersion.

Image of a person in a virtual reality CAVE

Despite Myron Kruegere's "artificial reality" systems and subsequent VIDEOPLACE interactive VR system, it was not until 1994 that the Virtual Reality Modelling Language (VRML) was introduced. VRML is a standard file format for representing 3D interactive vector graphics and was designed with the World Wide Web in mind.

After the 1997 founding of the Web3D consortium, an industry standard for web-based 3D graphics was developed. Subsequent to this was the development of X3D from the VRML framework. X3D is a royalty-free ISO/IEC standard for representing 3D computer graphics. The file format support includes XML, ClassicVRML, Compressed Binary Encoding and a draft JSON encoding. X3D is the successor to VRML and features extensions to VRML and the ability to encode a scene using XML syntax and Open Inventor syntax of VRML97.

Following X3D, WebVR was introduced. WebVR was an experimental JavaScript application programming interface that enabled applications to interact with virtual reality devices (such as the HTC Vive, Oculus Rift, and Open Source Virtual Reality). WebVR was implemented in Firefox and Chromium-based browsers before being superseded by the more powerful WebXR API capable of representing virtual reality and augmented reality devices simultaneously.

Virtual Reality is used for entertainment, which may be the most popular application of the technology. In entertainment, virtual reality can be used for gaming, through a virtual reality head-mounted display or in virtual environments. Cinemas using 3D can also be considered virtual reality, although they do not offer the same immersion level as headsets.

In gaming, head-mounted displays (or virtual reality headsets), often aided by computer systems or game consoles, allow players to play games in a higher level of immersion than traditional gaming. This can include sport games such as tennis or first-person perspective games in which users can engage with a virtual environment and virtual enemies. The control or sensor wands work to determine the speed and accuracy of the users' movements to create immersive experiences and real-to-life reactions within the game experience.

Music videos, live music events, short films, live sporting events, and theme parks have developed virtual reality applications or products where users can immerse in a film, feel like they are at a sporting event, or ride a rollercoaster they might otherwise not have a chance to visit. YouTube offers videos in VR that users can watch using head-mounted displays as simple as Google's cardboard head-mounted display, which uses a cardboard structure around a mobile phone to create a low-immersion virtual reality experience.

There are art galleries, both public and private, that offer virtual reality experiences. These experiences can include visiting famous art galleries, visiting curated collections that do not otherwise exist, viewing art that users might otherwise not have a chance to visit, experiencing private galleries otherwise closed to the public, and allowing users to partake in art sales virtually.

Since Thomas Furness' 1986 "Super Cockpit," virtual reality has been used for training, simulation, and education. Military and space programs have used virtual reality to recreate and train users in scenarios that are too dangerous or too difficult to replicate safely. This can include vehicle simulations and squad combat simulations. NASA used virtual reality in 1989 to develop a virtual reality simulation trainer for astronauts. Their simulation included the use of gloves to help the astronauts learn how to use tools in otherwise difficult environments.

Similar to military applications, medicine has used virtual reality to train medical students in surgical procedures and to help students learn how to diagnose a patient. Students are researchers at The University of Toronto and other research universities have used virtual reality to explore microbiology. VR learning has allowed students and researchers to be inside a small cell, bacteria, or virus to better understand how they are structured and how they interact within themselves and within a larger organism.

Architects and engineers have used virtual reality to show students complex structures and also hard-to-reach structures, such as bridges. Students can use VR to interact with models at a deep level of immersion to understand complex topics and structures and to build structures without any real-world consequences of failure.

The education use for virtual reality can also be as simple as simulating real work spaces for workplace occupational safety and health training. Users can develop skills without real-world consequences, in the case of a workplace hazard or injury. This workplace training can include driver training, miner training, and commercial flight training.

Similar to education uses, virtual reality is used to simulate environments or structures. For architects, this can mean creating virtual models of building plans to allow people to walk around and through the structure. In this case, the use of virtual reality can give people a better idea of how moving through a building will feel over a miniature model, with a more realistic feel of the building's features and scale. For automotive engineers, virtual reality is used to build virtual prototypes of new vehicles, test them thoroughly, and increase the overall efficiency of developing an automobile.

The real estate industry has also begun to use virtual reality to allow potential buyers or renters to virtually walk through a property. Real estate agents can also use virtual reality to show prospective buyers a furnished space, despite a space being empty in real life. These tours can increase the amount of potential buyers a real estate agent can reach, and they can offer out-of-market prospective buyers the option to virtually experience a space without travel expenses.

Virtual reality has also been used for marketing and e-commerce simulations. OnePlus, a mobile phone manufacturer, used virtual reality to launch one of their phones in 2015. Though limited, the virtual reality offered a 360 degree view of a launch, building a more immersive experience than traditional advertisements. During the 2020 COVID-19 pandemic, similar virtual reality experiences have been offered for product launches and previews. A lot of these advertising campaigns use 360 degree video for its greater accessibility over a headset-enabled virtual reality experience.

The military sector has been at the front of emerging technology advancements, including virtual reality applications for training and combat enhancements. Due to the constantly evolving nature of warfare, military is constantly looking for opportunities to build advantages through technological advancement. VR technologies can save money and help soldiers develop niche cognitive skills without risking safety for trainees. According to the Official Naval Research (ONR), people who do not play video games have a less developed field of vision, are worse at memorizing visual objects, and process new information slower than gamers.

Synthetic Training Environments (STE) is a key virtual reality application being developed by the US military, which trains soldiers for more physically and mentally stressful operational environments. STE is being developed by the Army Research Laboratory (ARL), USC Institute for Creative Technologies, Combined Arms Center-Training (CAC-T), and Program Executive Office for Simulation, Training and Instrumentation (PEO STRI). The application is projected to be used for training a combat team to establish adaptive units with higher readiness levels. For example, if there is a need for a hybrid team with specialties from different military branches, the STE could train them to work cohesively for the designated mission.

ClassVR is another military application used to teach soldiers and military operators via VR. It allows trainers to create customized training scenarios for trainees and upload real world footage to deliver controlled exercises for individuals and groups. A large benefit of VR applications is the scalability of training exercises. The technology allows high level scaling, ensuring consistency across large amounts of military personnel while allowing evaluation of individual performances.

Equipment training is another avenue where virtual reality shows advancements over traditional training, allowing trainees to gain skill with a wide range of equipment ranging from fire arms to armored vehicles to missile systems. The Pilot Training Next Program highlights how VR can train personnel at scale by bridging the gaps between expensive traditional simulations and classroom learning, streamlining how the US Air Force trains new pilots with virtual cockpits.

Besides its use in training and educating for healthcare, virtual reality has also found a use in psychological and physical therapy. Dr. Barbara Rothbaum of Emory University and Dr. Larry Hodges of the Georgia Institute of Technology researched the use of virtual environments for treating people with phobias and other psychological conditions. The application of virtual reality for exposure therapy has made available a less expensive and more convenient way for patients to be treated for phobias; patients have been shown to be more willing to try virtual exposure therapy because it does not occur in the real world. This research has been extended to include the use of virtual reality for the treatment of post traumatic stress disorder in war veterans.

Virtual reality programs are being used for rehabilitation with elderly patients diagnosed with Alzheimer's disease. This has provided these patients the opportunity to simulate real experiences they would have otherwise not experienced.

Researchers at the University of Toronto began using virtual reality to increase the quality of life for palliative care patients. Researchers Cosmin Munteanu and Sho Conte offered patients the chance to use virtual reality to explore places and have new experiences. Further, the use of virtual reality offered a break from the social isolation caused by the nature of palliative care. Both Muneanu and Conte have explored the possibility of using virtual reality to simulate difficult end-of-life discussions with patients and their families for caregivers. Through virtual reality, caregivers could have simulated conversations with various responses and offer a tool for self-reflection, learning, and training.

In 1998, at a hospital in Paris, virtual reality was used in conjunction with a robotic device to perform a remote surgery. The procedure proved challenging due to the delicate nature of surgery and the latency in the robot's movements compared to the surgeon's.

There is no exact date for the invention or creation of virtual reality. The dating depends on how an individual views the use of virtual reality. The head-mounted display often associated with virtual reality was developed in 1960 and some refer to this date for the invention of virtual reality. Others date the beginning to the work of Ivan Sutherland. And some suggest it was the popularization of the term "virtual reality" by Jaron Lanier that could be considered the beginning of what is considered virtual reality.

If the history of virtual reality is extended to the attempt to build a technology allowing a viewer to exist in a virtual or "non-real" situation, then the history of virtual reality could be extended further back to the idea of panoramic paintings or murals. Panoramic paintings were large paintings, often of historically significant events or moments, and were intended to fill the viewer's entire field of vision and create the illusion of being present at the scene.

Originally developed in 1838 by Charles Wheatstone, the stereoscope viewer used two side-by-side stereoscopic images or photos to give a user a sense of depth and immersion. These were later developed for use for "virtual tourism" and show the early design principles used by low-budget modern virtual reality headsets. Charles Wheatstone's stereoscope was later followed by David Brewster's lenticular stereoscope in 1849, and followed again by William Gruber's View-Master in 1939, where the popularity of "virtual tourism" grew.

image of an early View-Master

Similar early ventures into "virtual reality" include a 1929 link trainer developed by Edward Link. This device was patented in 1931 and was the first example of a commercial flight simulator that was entirely electromechanical. During World War II, over 10,000 of these link trainers were used by pilots for initial training and skills improvement.

In the 1930s, in a science fiction story called "Pygmalion's Spectacles" by Stanley G. Weinbaum, the shape of head-mounted display-based virtual reality was first described. The story contains the idea of a pair of goggles that let the wearer experience a fictional world through holographics, smell, taste and touch.

In 1957, Martin Heilig developed the earliest example of immersive, multi-sensory technology—the Sensorama. This is considered one of the earliest examples of a virtual reality system. The Sensorama had a stereoscopic color display, fans, odor emitters, stereo-sound, and a motional chair that simulated a motorcycle ride through New York.

In 1960, Martin Heilig patented his Telesphere Mask, which looks similar to modern head-mounted virtual reality displays. Through the Telesphere Mask, a user experienced three-dimensional images in color, with peripheral vision, binaural sound, scents, and air breezes. The Telesphere Mask was a commercial failure.

Developed in 1961 by Philco Corporation engineers, this head-mounted display included a video screen and tracking system linked to a closed circuit camera. The Headsight system was intended for use in dangerous situations and could display an environment remotely. It was similar to helmets used by helicopter pilots.

In 1965, Ivan Sutherland, a computer scientist, published a paper in which he described what he called the "Ultimate Display." His concept for the Ultimate Display would simulate reality to the point that a user could not tell the difference from reality. This included the use of a head-mounted display with 3D sound and tactile feedback that offered users the ability to interact with objects in a virtual world in a realistic way.

By 1968, following his paper, Ivan Sutherland created what is largely considered the first head-mounted display for virtual reality applications. It was called the Sword of Damocles, for the large telescoping arm from which the head-mounted display hung. The stereoscopic display showed users virtual wireframe shapes that changed perspectives as the user moved their head. This project never moved beyond the lab Ivan Sutherland worked in.

In 1969, Myron Kruegere, a virtual reality computer artist, developed a series of experiences that he termed "artificial reality," in which computer-generated environments responded to the people in it. The technology enabled people to communicate in a responsive computer environment, despite their physical distance. VIDEOPLACE, the name for the resulting technology, is considered the first interactive virtual reality system.

As the haptic and optic technology involved in virtual reality developed through the 1970s and 1980s, NASA's Ames Research Center developed virtual reality systems. This included their Virtual Interface Environment Workstation (VIEW) system, which combined a head-mounted device with gloves offering haptic interaction to give users a chance to interact in a virtual environment.

As well as their technology developments, NASA, along with the Department of Defense and the National Science Foundation, funded a lot of the research and development in virtual reality projects. Many of the early applications included vehicle simulators and training exercises.

In 1987, Jaron Lanier coined and popularized the term "virtual reality" and is often considered one of the fathers of virtual reality. Two years earlier, Jaron Lanier and Thomas G. Zimmerman left Atari to found VPL Research, where they developed, manufactured, and sold virtual reality goggles and gloves with haptic feedback. The focus of VPL Research was to popularize virtual reality technologies. They filed for bankruptcy in 1990 and their patents were purchased by Sun Microsystems in 1999.

In 1992, the Computer Gaming World publication predicted affordable virtual reality by 1994. This came after the announcement by SEGA that they were producing a virtual reality headset as an accessory to their Sega Genesis console. The initial announcement came in 1991, and a second announcement of a refined design came in 1993. The product was never released. SEGA did release their Sega VR-1 in 1994. This was an arcade attraction capable of tracking head movement and featuring 3D polygon graphics in stereoscopic 3D.

Nintendo released their Virtual Boy portable video game console in 1995. The Virtual Boy was marketed as the first console capable of displaying stereoscopic 3D graphics. A player would use the console as a head-mounted display with a controller to allow a user to navigate a game. Although the system did not use strictly three-dimensional graphics, they did use a parallax effect to create the illusion of depth. Nintendo's Virtual Boy failed to meet targets and Nintendo ceased distribution and game development in 1996.

In 2007, Google introduced their Street View feature to their Google Maps. Through Street View, users are shown a panoramic view of a specific location, although largely limited to roads, indoor buildings, and rural areas. In 2010, they introduced a stereoscopic 3D mode capable of use in a virtual reality headset.

Palmer Lucky designed the first prototype of the Oculus Rift head-mounted display virtual reality system in 2010. The prototype was capable of rotational tracking and it boasted a 90 degree field of vision unseen before in the consumer virtual reality market. In 2012, the Oculus Rift was presented for the first time at the E3 video game trade show. Oculus started a Kickstarter campaign to fund the Rift's development in the same year. The campaign was successful and raised close to USD $2.5 million.

In 2014, Facebook purchased Oculus VR for $2 billion. They sent development kits of the models out through 2013 to 2014. The Oculus Rift was commercially released in March 2016. Production of the first model, the Rift CV1, was discontinued in March 2019 with the release of the Oculus Rift S.

In 2016, the release of the Oculus Rift CV1 was accompanied by the HTC's release of their Oculus Rift competitor, the VIVE StreamVR headset. The HTC Vive StreamVR headset, developed with participation from Valve, included a technology they called Lighthouse, which utilized a wall-mounted tracking system that gave users free movement within a defined space.

In the same year, there were nearly 230 companies developing virtual reality related products. Some of these companies included Amazon, Apple, Facebook, Google, Microsoft, Sony, and Samsung. Sony filed a patent for a virtual reality technology with location tracking technology similar to the HTC Vive for their Playstation VR system.

By 2017 and on to 2019, the release of virtual reality headsets continued, with the consumer-focused technology offering devices with different price ranges and with different capabilities. These included the Samsung Odyssey, the Oculus Go, Google Cardboard, Google Daydream View, Lenovo Explorer Windows Mixed Reality Headset, Powis Popup, Samsung Gear VR, LG 360, Zeiss VR One Plus, and OnePlus Loop VR.

In 2019, Facebook launched it first-all-in-one 6DOF VR gaming system, Oculus Quest gives user the freedom to move through virtual worlds like real.In the same year, Facebook also launched Rift S, it second-ever tethered VR headset. While Vive finally jumped into the VR headset space with its own first-party hardware - Vive. HTC launched Vive Cosmos, its first true successor to the original Vive headset.

The COVID-19 pandemic in 2020 saw an increase in the use of virtual reality in clinical applications and an increase in sales of virtual reality hardware for consumers. A team from Simon Fraser University's School of Interactive Arts and Technology worked with Virtro to develop virtual reality and artificial intelligence-based training simulations for healthcare workers impacted by the COVID-19 pandemic. These programs are intended to increase the availability of training for healthcare workers, medical staff, and students and include simulations on the proper application of personal protective equipment in long-term care facilities.

Due to many regional lockdowns during the COVID-19 pandemic, many music artists have used virtual reality as a way to hold concerts and live shows for audiences not allowed to congregate. The amount of people in lockdown situations at home increased the sales of virtual reality equipment. Virtual reality offered users not only the chance to join virtual music events or live sporting events, but also a chance to virtually leave their homes or play games, which can near-simulate leaving home.

Virtual reality headset sales fell in 2020 due to supply issues caused by the pandemic, although analysts believe this dip to be temporary. Despite these constraints, demand for virtual reality headsets was expected to increase, with many analysts believing the popularity of virtual reality will continue to grow.