How Qualcomm wants to arm the movement for open XR | Hugo Swart interview

Apple finally entered the market for mixed reality this week as it showed off its Apple Vision Pro headset at its spaceship headquarters in Silicon Valley.

But it’s easy to forget that last week’s Augmented World Expo celebrated the same kind of technology and it was full of companies that wanted to pursue something very different. While Apple CEO Tim Cook didn’t mention the word “metaverse” at the event, he also didn’t make any case for openness.

Yet Qualcomm, the big designer of semiconductor chips for mobile and XR devices, is one of the companies that want the open side of the metaverse industry to prosper. For instance, Mark Zuckerberg’s new Meta Quest 3 VR headset coming this fall for $500 will use Qualcomm technology. I spoke with Hugo Swart, head of XR at Qualcomm, about this during the AWE event last week.

Here’s an edited transcript of our interview.

Hugo Swart, head of XR at Qualcomm, at AWE USA.

GamesBeat: Tell me about what you see coming.

Hugo Swart: On the hardware side, we showed DigiLens and DCL. These are all-in-one AR glasses with XR2. They’re both slated to have Snapdragon Spaces. That means applications that were built using our APIs – essentially OpenXR APIs – are going to be compatible with both devices. Then we went a bit on our vision for AR glasses.

We expect distributed compute to be the norm, where you don’t have everything in the glasses. You have some functions running in the glasses, like head tracking or hand tracking. Some of the perception algorithms will be here. You’ll have a companion host, whether a phone or a PC. Then here’s where the applications are running, where you do the rendering, and send it back. Image processing, display processing, late-stage reprojection and so forth. Voila, you have a great immersive experience in less than a watt on the glasses.

Today we already have the Lenovo A3, which you’re going to see demos on. These are still cabled. You have the Xiaomi AR glass that is wireless. We have the Snapdragon AR2, which is going to be all wireless. The AR2 was built for this distributed processing architecture.

That’s the AR hardware update. On the VR and MR hardware, it’s the Lenovo VRX. They have their VR headset that they announced today. That’s now commercially available. They’re using Snapdragon Spaces as the platform for developers. On VR and MR that was the main thing. And then Opal showed their device using XR2+. That’s available in the second half of this year.

On the software side we expanded a bit on our previous announcements of collaboration with operators. We signed a new operator working with us. In the past you’ve probably seen T-Mobile, Deutsche Telekom, KDDI doing their own developer events around Snapdragon Spaces. Now we have Claro as the eighth operator working with us, expanding on the collaboration with Adobe, Microsoft–Microsoft is on the MRDK. And then Niantic. We’re putting together the Niantic Lightship VPS, together with Spaces. Today we announced that we have 10 developers, early developers, early access developers using Spaces, plus the Lightship VPS.

VentureBeat: What does each party bring to the table there?

Swart: With Niantic, it’s their visual positioning system. In particular for public places and the outdoors. Who better than them, after building this infrastructure for their first-party applications? They’re now making it available to third parties. It’s really a combination of functionalities and APIs that we have from Qualcomm, from the Snapdragon Spaces, things like hand and head tracking, some of the environment understanding. But then, when it comes to the mapping, the VPS, you use the Lightship VPS. That’s how we put the two together.

I think the demos we’re going to show you are with a new feature we announced today. We call it Fusion. That’s the short version of it. But the full name of the feature is Dual Render Fusion. The best thing is really to experience it. But the idea is that you have an app made for a 2D screen–imagine a game. With the glasses you can now augment the application you have on the phone. Developers can choose to–maybe if I have a shooting game, I’ll just do explosions as AR with the glasses. One example we showed in the keynote this morning in our video–I have a furniture type of application, like Ikea. I can still use the 2D app for the phone as the main navigation on selecting. But once I select something, now I can see that couch in AR.

Qualcomm’s XR platform partners for Snapdragon Spaces.

Why do we think this is so meaningful, so important? It’s because from a developer perspective, you don’t need to start a whole new app development. You take the existing app, and you’re just adding some AR functionality on the glass. The barrier to entry is much lower for the developer.

Those are the main things we announced. Then we showed a couple of demos, including one with Red Bull, which was similar. They have the Red Bull app on the phone, and then when you have the glass, you can see–this was a mountain bike racing app. On a plane you have the mountain where the biker is going is rendered, and you have the path you follow and so forth. You still have the video in the background. And then the second one was Kitch, a cooking app. Again, it’s a mobile app, but when you put the glasses on, now the experience is augmented. It takes things from the phone and now I can see them in the real world.

VentureBeat: How quickly does that transition happen from looking at something on the phone, when it becomes aware that you have the glasses on?

Swart: That’s the good thing about–I don’t know if we have the video here with the Kitch demo, but in that case they’re using a cabled glass. Immediately, when you put the cable in, it knows to ask, “Hey, show the experience in your actual kitchen?” Press yes and then it comes in. Before you connect the glass, it’s just a mobile app. Connect the glass and it detects that it has this input and output available. Then it can ask the user if they want that. If it’s a wireless device, it’s a similar thing. The app detects that the glass is available and asks if it can show the augmented reality extensions. That will really help get more developers experimenting and adding functionality with augmented reality.

There are a lot of things happening in XR, of course. A new wave of devices is coming. We continue to be very bullish, very optimistic with both the VR/MR track and augmented reality.

VentureBeat: Given that [Apple’s event] next week is happening, is there a way you would describe how open or more open you are?

Swart: That was the title of the keynote. “The Future is Open.” Our intent is to make an open platform, in the sense that–from various angles. If you think about what we do with our processors, we make them available to startups like DigiLens. They can build hardware. Meta, they can build hardware. We’re open in the sense that we serve everyone. China, Korea, here. On the hardware side we make sure that everyone can build hardware on our platform.

When it comes to software and interfaces, we’re using OpenXR. Spaces essentially is adapting OpenXR. The goal is that you build it once and then everyone, all the OEMs, the hardware manufacturers that support Snapdragon Spaces, they’re going to support the applications. That’s why I say it’s from various angles, the ways we justify ourselves as open in driving XR forward. It’s just a healthier ecosystem. You can enable a lot more innovation in both hardware and software by building this framework of openness.

Qualcomm has partnered with Niantic for augmented reality.

VentureBeat: How do you make enough noise in that environment when Apple is coming into the market too?

Swart: The key thing for us is to show that we’ve been investing in XR for the longest time. AWE, the first AWE was I think in 2010. We were here. We had AR back then, smartphone AR. Stand-alone VR headsets, 2015 is when we started with the first reference designs. That’s what’s important to us. The industry, the world understands that we’ve been doing this for a long time. We’re trying to enable as many people as possible – the HTCs and the Picos and the small guys like DigiLens and Vuzix, you name it. Do you remember ODG? That was using Qualcomm. Those are important things for us.

It’s harder for us to reach the audience that some of these companies can get. But I think our story is–I feel it’s very solid, because of our investment and what we’re enabling today.

VentureBeat: How many of the existing headsets are based on Qualcomm?

Swart: The Magic Leap Two is not Qualcomm, but that’s probably it as far as notable headsets. The ones that are PC-connected, where you don’t have a lot of processing or intelligence on the headset, those are not going to be Qualcomm in general. The Varjo, a few things like that. But if it’s stand-alone, the great majority is Qualcomm. Last I counted, we launched more than 65 devices. I think that’s already outdated.

We’ve built the foundational technology, but also the ability to scale. Through our customer engineering, through our reference designs–our model is always to build that reference design. It’s kind of like a blueprint. Our customers can take as much advantage as they want from it. That facilitates their ability to quickly go to market.

VentureBeat: The $38 billion figure, I do wonder how much of that is hardware. Do you have a market share number relative to that?

Swart: No, we haven’t publicized anything. It’s no secret that the leading OEM today is Meta. Meta has a very strong partnership with Qualcomm. Last September Mark Zuckerberg and Cristiano Amon announced a multi-generation agreement. We’re working very strongly together on building the road map.

VentureBeat: For 2023, when you think about it–how many headsets do you think matter the most, as opposed to the full list of them?

Swart: It depends on how we define what matters the most. From a volume perspective, there aren’t many. But when it comes to driving innovation, there are quite a few that are doing different things. I was in China a month ago. I saw so much that some of these smaller companies are doing in the design. They look more like ski goggles than the early generations of VR headsets. In that sense, I think they matter. They’re showing what’s possible.

VentureBeat: One guy here was saying that he thinks Apple will be very focused on Lidar technology, while most others aren’t. Is that a dividing line, whether people think lidar is useful or not in this space?

Swart: It’s really a question of the ROI in using Lidar. Cost, power, size versus the range that you get for mapping a room and so forth. I do think that there is an opportunity for a spectrum of products, with and without. We have a dev sensor in our 845 reference design. These sensors are not new. They’ve been around for some time. There’s a cost in terms of power and in terms of size. It’s hard to put that kind of technology on a small device like AR glasses. It’s easier to put it in a VR headset. We see it typically in larger headsets.

A lot of what we’re doing in Spaces is using stereo cameras to generate depth maps, and then using 3D construction algorithms to build a spatial match, to reconstruct the real world in 3D so developers’ apps can interact with it. There are different approaches you can take. Our depth map is not generated from an active sensor today, but it could be. It’s built so that if the device has one, it can feed that in and Spaces can take advantage of it.

Qualcomm is pushing AR and VR technologies with its chip designs.

VentureBeat: There was a company I came across that was saying they would use XR sensors that were not cameras. When you have 10 cameras in the Meta Quest Pro–if you can reduce that number and still detect things like eye movement without the cameras, it’s potentially much cheaper. There was another one that would sense your wrist movements as well, and using that it would figure out what your hands were doing. Again, that would reduce the number of cameras. Is that an interesting direction to go in order to become more cost-effective?

Swart: The beauty of XR right now for engineers is that there’s just so much innovation. I don’t think that at this point there’s a right or a wrong. It’s all about tradeoffs. I do agree that if I have my smartwatch that has some sensors here, and it can track my hand, or help track my hand, you should use it. But can you really reduce the number of cameras? There will be a penalty for that.

The eye tracking solutions–I’ve heard people say they can do it with only one camera, or use a sensor that’s not a camera. The question is, when is it ready? Today I don’t have an option. For facial tracking, maybe we can use AI and limit the number of cameras. That’s all good, but it’s similar to the “Active depth or not?” discussion. There are options. One needs to choose the best balance between power, cost, and size.

What’s the problem with adding more cameras? Well, it’s costly. Each of these camera sensors, they add to the price. It’s power as well, and in two ways. There’s the power for the actual camera module, and then there’s the power to process the images. And then there’s size. If there are ways to eliminate those and still meet the performance bar I need for my application–some people will say no. The KPI that I’m measuring is getting too penalized, so I still want the camera. That’s why I don’t think there’s a right or a wrong. It’s just a choice. The right tradeoffs need to be analyzed.