Why Microsoft Is Investing In AR And Partnering With VR

Over the past week, we had a chance to see Microsoft's HoloLens on stage at E3, learn about a partnership with Oculus, and another with Valve. We sat down with Microsoft corporate vice president Kudo Tsunoda to discuss the company's investments in these new mediums.

HoloLens, which is being designed for release "in the Windows 10 timeframe," was given stage time during Microsoft's E3 press conference. With a recent acquisition of Minecraft developer Mojang, we got our first look at the wildly popular sandbox game through the HoloLens.

The demonstration included building, gesture manipulation of terrain, and moving from a flat, traditional display to a three dimensional digital machette that could be lifted, twisted, and turned. Tsunoda has some ideas for other gaming applications of the hardware.

"As you start to develop more games that are character- and story-driven, just having the characters in your own living room and having the narrative play out in your own home drives a much higher level of emotional attachment and engagement to characters and stories when it's taking place in your real world," he tells us. "Those are the big areas of differentiation we've explored so far with gaming with holograms. I don't think that any title that can incorporate your real-world environment or is character- or story-driven I think works really well with holograms. We're just scratching the surface about what you can do with holograms in gaming."

As anyone who has used gesture controls knows, the sensors don't always work properly. We watched someone manipulate a Minecraft world with his hands at the Microsoft press conference, and wanted to know how the existing challenges will be overcome. Tsunoda explained that HoloLens uses three different types of sensor input in cooperation to create the experience.

"You can look at a hologram, and HoloLens will know what you're looking at," he explains. "We have a really good way of knowing what you want to interact with, because you're looking at it. Then we've got our gestures. That's all understood, because we have a miniature depth sensor in the HoloLens and read your gestures and understand what you're doing. Then we've got voice commands. There are microphones on the device, so you can interact with the content with your voice."

Microsoft is able to achieve improved gesture reading when compared with Kinect, in part because of distance between the sensor and your hand. "It has a wide field of view. It's super precise," Tsudnoda tells us. "It is also just an advantage when you think about what you need to do with Kinect. Your console is sitting over there. You're on the sofa. That's a pretty far distance to have to read motion. You have a depth sensor on HoloLens. You're reading a hand gesture that's a lot closer, so you have a much higher level of precision."

While Microsoft is persuing its mixed-reality project, the company is also building bridges to the virtual reality community. At Oculus' pre-E3 press conference, the companies announced that the first Oculus Rift consumer kit will come bundled with an Xbox One controller. Microsoft also announced that it will be working with Valve's VR initiative.

"I think they are complementary technologies. I'm not saying that one is better than the other," Tsunoda tells us. "Virtual reality and the immersive experience is great. I think it has awesome applications for gaming and entertainment. It's stuff that I enjoy doing as well. The stuff we're doing with mixed reality is blending digital assets with your real world environment, because we think there are inherent advantages of having your digital applications where you can use them the most effectively."

HoloLens has applications far beyond gaming, and Tsunoda likens the technology to a home computer's multifunctional intent. He explains how one might use HoloLens to learn how to do new tasks with the help of someone on the other end of a HoloSkype call.

"We have a holographic Skype application. Someone can sit on a tablet and draw holograms into the world of the person wearing Microsoft HoloLens," Tsunoda tells us. "Let's say I want to go and rewire a light switch. I don't know how to rewire a light switch, but somebody can sit on a tablet and draw you holographic instructions right on the light switch, step-by-step as to what you are supposed to do, to learn something you might not have known how to do before."

Microsoft isn't being specific about timeframe for the HoloLens retail release. However, Tsunoda suggests we won't have to wait for years.

"It's definitely not long term. It's much more in near term than it is long term," Tsunoda says. "What we want to do is have a really polished experience for people in [version one]. Obviously we're not going to ship anything that in any way compromises that, so we'll get to a place where you have a great experience seeing holograms. You'll have a great experience interacting with them. We'll provide you with high value use cases, so the holograms aren't a novelty, but something you can integrate into what you do. Once we have a good set of those things (and I feel really confident about that even where we are right now) that you're going to see that coming to market."