The XR hardware landscape

1. Overview

Previously, you looked at the software landscape of VR; now it’s time to look at the hardware landscape.

The VR HMD landscape is expanding with more offerings all the time. It can be difficult to keep track of the differences between them. However, there are major advantages that come with being up to date on the latest hardware.

In this tutorial, you will learn about different types of XR hardware as well as new technologies and features on the horizon.

2. Recap: VR vs. AR vs. MR

You learned about the differences between VR, AR, and MR in the introductory tutorial. But as a recap, XR is split into three primary categories.

VR

VR creates a digital environment that replaces the real-world environment through a head-mounted display. Check out some of the amazing VR games made with Unity:

AR

AR overlays digitally created content into the user’s real-world environment on a mobile device or head-mounted display. Check out some of the applications for handheld AR in this demo reel:

MR

MR includes any experiences that fall along the reality-virtuality continuum, including some elements of AR and VR. Check out this demo of Unity Slices: a multiplayer mixed reality experience that allows people to locally and remotely simultaneously interact with the same surface:

You’ll see next that, as the hardware evolves, the lines between these technologies get blurrier.

3. Opportunities for work in XR

As you have already heard, this technology changes so quickly that it can be difficult to keep up. But with changes in technology come new opportunities.

Hear from our established creators on the wide range of opportunities in this fast-changing field:

4. Categories of XR hardware

There is so much to learn about XR hardware and how it works, but here is a high-level overview of the categories of XR hardware:

VR devices

Since VR replaces your entire visual field with digital content, it can only be done with a head-mounted display and usually includes two handheld controllers.

However, a couple of crucial features distinguish the most popular VR devices:

• Power: Tethered vs. Standalone

• Tethered headsets like the PSVR2 are connected to a console, which allows for higher quality graphics.

• Standalone headsets like the Meta Quest 2 are entirely cordless, requiring that all processing be done in the headset. These are less cumbersome because they don’t have any wires, but they aren’t capable of producing as high-quality graphics as tethered headsets.

• Tracking: Inside-out vs. Outside-in

• Inside-out tracking means all cameras and sensors are located inside the HMD and controllers, tracking the position and rotation of your head and hands from within (inside) the devices, outward to the room around you. The Meta Quest and PSVR2 use inside-out tracking.

• Outside-in tracking means you have additional external stationary sensors in the room, outside the HMD and controllers, tracking the position and rotation of your head and hands. Outside-in tracing requires more setup, but it allows for more advanced tracking. The HTC Vive Pro and Valve Index use outside-in tracking.

AR / MR devices

Since both AR and MR combine digital content with real-world content, they can share the same hardware.

• Type: HMD vs. handheld

• Head-mounted AR/MR headsets like the Magic Leap or Hololens 2 have transparent lenses that can have digital images displayed on top of the lenses.

• Handheld AR/MR devices are most commonly smartphones. Using the camera and screen, images and filters can be superimposed onto the live view of the real world. Common examples of AR/MR on smartphones include social media apps like Snapchat or dedicated AR apps like Pokémon Go and IKEA place.

• Display: Transparent lenses vs. passthrough

• Transparent lenses like the Magic Leap or Hololens 2 project digital images on top of the real world you are seeing through transparent lenses.

• Passthrough is when a live camera’s feed is projected onto a screen that you’re looking at, which can then have images projected onto it. That is what is happening with smartphones. You’ll learn more about passthrough in the next step.

5. HMD tech: passthrough

Most new models of VR head-mounted displays have external cameras mounted on the outside of them pointing outwards. Passthrough technology is when the view from those cameras is passed through to the display in the headset so that it’s viewable by the user.

In the demo video below, the black-and-white video is a demonstration of passthrough:

Passthrough lets users look at their physical environment like they would without a headset, but they’re actually looking at the live feed from the external cameras.

The recent improvement in these external cameras' quality allows traditional VR headsets to act more like AR or MR headsets. It allows you to see the world around you and easily superimpose digital content on top of that image.

Technology like this is beginning to blend the distinctions between AR, VR, and MR. Some people are starting to think of these media as existing along a sliding scale of blended physical and digital realities, rather than existing as distinct media types.

6. HMD tech: hand tracking

With the improvement of external cameras on VR HMDs, hand-tracking has become possible. With hand tracking, you can freely move your hands and use gestures like pointing, squeezing, and pinching to perform actions instead of using controllers.

For a vivid demonstration of hand-tracking technology, check out the Hand Physics Lab app trailer:

There are benefits and drawbacks of using hand tracking instead of controllers.

Benefits:

• It can feel more intuitive and natural.

• It promotes a sense of presence and immersion since your hands are not tied to controllers.

Drawbacks:

• Since hand tracking relies on hand and gesture recognition from cameras, hand tracking isn’t as accurate or responsive as pressing a button on a controller is.

• Your hands need to be in view of the cameras for the hand tracking to work, which could significantly limit the experience.

• There are not as many inputs (buttons and triggers) as there are with a controller, which might limit the complexity of experiences.

• The lack of haptic feedback makes it harder to determine whether you performed an action correctly.

7. HMD tech: body and eye-tracking

As tracking technologies improve, we may see advancements such as full-body monitoring. This would allow a user’s legs, arms, and upper body to be represented in VR, instead of just their head and hands.

Headsets that use outside-in tracking can track a user’s entire body, including the use of dedicated sensors for body tracking such as the Vive Tracker.

In addition to this, newer HMDs can track a user’s eye movements, allowing for more responsive experiences, advanced interactions, and improved comfort.

8. Next steps

We will not be developing for these features in this pathway, but understanding the pros and cons of these technologies will allow you to make informed decisions for your projects in the future. In a later tutorial, we will offer you some tips for keeping up with these changes.

In the next tutorials, you will continue working on projects with your current hardware, but it is good to be aware of the other technology as you do so.