How Qualcomm Designs Chips for the Mobile VR Revolution

Believing mobile virtual reality is the next revolution, Qualcomm is positioning itself to be the go-to chipmaker behind mobile VR, AR, and MR experiences.

Chris Wiltz

May 22, 2018

8 Min Read
How Qualcomm Designs Chips for the Mobile VR Revolution

The Snapdragon 845 incorporates SLAM technology to permit room-scale 6DoF tracking, allowing users to move around large spaces in XR. (Image source: Qualcomm)

“Just a few years ago, when you mentioned mobile VR, people thought about Google Cardboard, but [mobile VR] has evolved,” said Hugo Swart, Head of the XR (VR and AR) Business Group at Qualcomm. During a keynote at the recent VRLA conference in Los Angeles, Swart spoke about Qualcomm's strategy for “extended reality” (XR), the blanket term that encompasses virtual reality (VR), augmented reality (AR), and mixed reality (MR), and a future of sleeker devices, capable of delivering XR experiences that will be imperceptible to users. “AR and VR will merge,” he said. “It's a device you'll wear all day...”

But Swart said that such a device is still a ways off. Right now, there's not even a smartphone equivalent to VR. “VR will still be in its flip phone stage by 2020,” he said.

But Qualcomm believes VR will have the same societal and cultural impact that the smartphone has had. The chipmaker—best known as a supplier of processors for the same smartphones that have changed the way we all work, play, and communicate—has been looking at creating chips and other hardware optimized for XR for some time now.

The Road to More Immersion

“Immersion is key for consumer value,” Swart said. “Consumers need to have the connectivity as if the experience is real... If you try to visualize immersion, the next few years, most of us will think about [the movie] Ready Player One, but it's not going to take that long. It won't be 2045 before we see that.”

In 2016, Qualcomm released a reference platform for a VR head-mounted display (HMD) based on its Snapdragon 820 processor. At a time when VR was mainly a tethered experience full of wires and external sensors, the company was looking at supplying a processor for an all-in-one mobile HMD.

Hugo Swart, Head of the XR (VR and AR) Business Group at Qualcomm, speaks to an audience at VRLA. (Image source: Design News)

But the Snapdragon 820 only met what Swart called the “minimum requirements for immersion.” That is, it only offered three degree-of-freedom (3DoF) head tracking. Users can move their heads to look around a virtual environment, but the headset won't track their body position. The experience is somewhat immersive, but more akin to being strapped to a chair bolted to the floor.

The following year, at CES 2017, Qualcomm debuted the second generation of its chip: the Snapdragon 835. Using a 10-nanometer manufacturing process for the first time, Qualcomm developed a chip that was 35 percent smaller and consumed 25 percent less power than previous designs. More importantly, the 835 offered the next level of immersion: six degrees of freedom (6DoF). Now, users could both look and walk around a virtual environment.

Qualcomm's bet paid off and the Snapdragon 835 has become the processor behind the latest series of mobile VR headsets hitting the market. Headsets including the Oculus Go, Lenovo Mirage, Pico Neo, and the upcoming HTC Vive Focus all have the 835 under the hood. All of these headsets are offering consumer and enterprise users a mobile, all-in-one, 6DoF VR experience with no need for tethered wires, an external PC, or external sensors.


Enter The Snapdragon 845

Earlier this year, at the 2018 Game Developer's Conference (GDC), Qualcomm announced a new chip and corresponding VR development kit: the Snapdragon 845. The goal is to go one step beyond the 835, offering features such as room-scale 6DoF tracking (letting users move around large spaces, like factory floors and warehouses) and enabling hand tracking by allowing integration of technologies such as the Leap Motion controller.

In an interview with Design News, Hiren Bhinde, Director Product Management at Qualcomm, said the 845 is designed to offer much more powerful processing and power saving capabilities to deliver higher-quality immersion.

“On the Snapdragon 835, we introduced a visual-inertial odometry (VIO) algorithm,” Bhinde explained. “What that means is finding where you are in a room. Let's say you're in a dark room or blindfolded, there are two things you'd like to know: where you are in the room and what is the room like—where is the chair, table, where are the walls...Just knowing where you are is how VIO helps, and that's mostly tracking.” This comes in contrast to today's PC-based VR systems, which rely on external sensors to track a wearer's locations and movements.

Bhinde said that Qualcomm was able to expand on this tracking capability by integrating simultaneous localization and mapping (SLAM) technology into the 845 chip. Those that follow the autonomous vehicle space will recognize SLAM as the same technology that helps self-driving cars understand their position on the road in relation to other vehicles and objects by creating and updating a map of the vehicle's surroundings. Using external cameras on a headset, developers can take advantage of this same technology to track a wearer's position in a large space.

“From a standalone VR perspective, the way it usually works is you look at features in the room—things like the TV and chairs that aren't moving—and look at their relative position to triangulate the wearer's position,” Bhinde said.

“For SLAM, you also want mapping,” he added, explaining that Qualcomm developed its own proprietary mapping algorithms. “Why did we do that? Well, for VR today, when you map the room, you know where the boundaries of the room are. And virtual objects need to be registered or localized with the real world.”

Bhinde likes the example of two medical students being trained on a procedure in VR. Let's say both students are wearing augmented reality glasses and performing a procedure where virtual objects are overlayed onto the real world. If one student is holding a virtual scalpel, that object needs to appear in the proper place for both students without resetting or jumping around. The idea is to map these objects as well as the room itself to create an XR experience with perfect fidelity to the real world.

“For SLAM, there are three key aspects we looked at: the boundary system; the localization aspect; and then, of course, just being able to provide room-scale 6DoF,” Bhinde said.

Qualcomm's new VR reference platform based on the Snapdragon 845 will allow engineers to develop around next generation mobile features, such as hand tracking. (Image source: Qualcomm)

Both Bhinde and Swat said that Qualcomm looks at three pillars of immersion to provide the optimal XR experience: visual, audio, and interaction. Bhinde said the 845 is designed to excel specifically in these three areas.

“On the visual end, you have to make sure the GPUs in mobile devices can do complicated processing for things like shading, which means extreme detail as well as pixel quality and quantity,” he said. “Interaction means having computer vision capabilities like SLAM and VIO.”

It's one thing to get the algorithm right, but there's the other area of making sure you're able to sustain performance and that the device doesn't become hot. “Imagine running on a treadmill at high elevation with dumbbells in your hands and weights on your legs. That's what XR does to a processor,” Bhinde said, adding that Qualcomm has been able to overcome these power challenges with a combination of hardware optimization and algorithms.

XR Needs 5G and AI

But the proliferation of mobile VR isn't going to depend solely on high-performance hardware. High-end applications—particularly those on the enterprise end, such as collaborative design and virtual training—are going to require powerful network connectivity—especially for complex, multiuser applications.

Perhaps this is where the long-awaited 5G technology can come in. “XR is, if not the most important, a very crucial use case for 5G,” Bhinde said. “At the end of the day, if you're in London and I'm in San Diego and we're having a social VR/AR experience, it's going to require extremely high bandwidth with ultra-low latency.”

Bhinde added that Qualcomm feels artificial intelligence will form the foundation to any merger between XR and 5G. Smarter processors with the ability to learn over time and make predictions can help ensure the low latency needs for XR use cases in 5G.

“One of the key challenges is latency in VR/AR devices. So if I'm able to predict where my hands are going to be or where my head is going to move next, that prediction could help me display and render the next visual even better,” Bhinde said. “And those sort of predictive algorithms could be enhanced using AI. We do think that AI will help the devices learn what's happening next even better.”

While AI is used in XR applications today for things such as virtual training, Bhinde said the next phase of processors optimized for XR will incorporate this sort of functionality directly into the hardware. “In the times to come, we'll be seeing processors with the ability to learn and think smarter and more intelligently, and we'll be applying those capabilities for XR use cases and for low latency in our 5G pipeline.”

Chris Wiltz is a Senior Editor at Design News covering emerging technologies including AI, VR/AR, and robotics.

Sign up for the Design News Daily newsletter.

You May Also Like