Sony’s Heads Of PlayStation VR Discuss Virtual Reality’s Future

PlayStation VR is poised to take the console market by storm as the first virtual-reality system to be released from any of the three major home-console manufacturers. Rather than the typical competition of Microsoft and Nintendo, however, Sony Computer Entertainment must focus on challengers from the PC side of the industry. PlayStation must now compete with the likes of Oculus and Vive as while simultaneously blazing the trail for any other console manufacturers looking at making the leap to VR.

This interview was originally published on December 14, 2015 as a part of our month of virtual reality coverage. We're sharing it again today in conjunction with the announcement of the PlayStation VR price.

I spoke with Sony’s president of worldwide studios for Sony Computer Entertainment, Shuhei Yoshida, and director of PlayStation Magic Lab, Richard Marks, on the beginning of PlayStation’s interest in VR, the steps the company took to get to where it is today, and more. Because my two conversations hit on many of the same topics, I’ve combined them into one discussion, which you can read below.

Where did PlayStation VR start, and how did it get to where it is now?

Shuhei Yoshida: The VR project that we used to call Project Morpheus was started as a grassroots project amongst different teams geographically spread across Japan, the U.S., and Europe almost simultaneously. When we released PS Move, many people found that just adding PS Move to head-mounted displays of the time like Sony’s Movie Viewer or some other company’s viewer creates like a handmade, semi-VR headset, and they started experimenting with it with PS3. All of these people who are using their spare time after work creating the handmade kit and some games to play with as a hobby almost realized that many of us are doing the same thing, and the company was like, “With the PS4, we could make a real virtual-reality system.” So this really came from the passion of these people who had waited for the time that virtual reality could be made real.

Richard Marks:  The official project started around 2010 and early 2011, but as I talk about this – I started in 1999 and we were working on 3D interaction. Not VR, but it's trying to give new capabilities to interaction, and then those capabilities ended up culminating in the PlayStation Move at some point. And the PlayStation Move really was a 3D spatial-input device that – it wasn't easy to use because the display is a TV flat image, and so trying to do things in 3D while you're looking at them in 2D was really tricky. We've always wanted to do more immersion, but the actual, “Let's make a product now,” was around 2011, the start of official product cycle.

How did the headset evolve from when you really started working on it and to what we're seeing now?

RM: The very first prototypes that we made used the PlayStation Move bolted to them as the tracking solution, and then the screen was an off-the-shelf LCD screen with some special optics. So that was a pretty quick, hacked-together system, and then those same elements were refined into real versions of each thing, so the Move got removed, taken off, and we put in real markers for tracking that would do better. The optics were refined quite a bit more and the screen in particular went from an LCD 60 frames-per-second screen to the current 120 frames-per-second OLED screen, which is really a very big step in the level of presence you get, when you change the screen that much. So I'd say probably the screen technology, integrating that, and furthering that, is the biggest thing that happened, but the tracking already felt like we could do this with this kind of level of tracking. We won't put a big Move on your head. That was never the intention, but we knew we could revamp that into something that would work similarly.

The past decade has seen a few premium peripherals get introduced into the market. Motion controls, 3D TVs, etc., seemed like they could be game-changers, but they ultimately failed to make real lasting impacts. What do you think separates PlayStation VR and virtual reality in general from the past peripherals that may not have performed as well as everyone had hoped?

RM: There are a few different things I would say on that. First of all, this one is less like a peripheral and more like a platform of its own. You pretty much have to make the content to match it, so it is a really different experience because the content is made for it. So, it's like a new platform. The other thing is that, in a lot of the cases of peripherals, either they're tied to one game or they're tied to one company and they aren't really compatible with anything else, so you have one company trying to get a bunch of content made for this one device and that's asking quite a bit of a game developer for that to happen. We've tried, and in some cases have been successful, in some cases not. Kinect or the Wii Fit board, you could argue whether those were successful or not. They have some level of success. But in this case, there are multiple platforms for VR that are occurring simultaneously, so a content developer can make things and potentially see it go across multiple platforms, and that helps them out a lot. It helps them be able to make it economically viable to make a game for it.

SY: The clear difference [between VR and 3D TV] is the quality of experience. When you remember 3D TV and 3D stereoscopic gaming, every single game, or even movie, pretty much the experience was the same as playing on your regular TV except for there’s a little bit of depth inside the TV. The relationship between the player and the TV was exactly the same and unfortunately – this was actually a technical limitation that I was personally disappointed in – because the tech was such that in many cases, there were compromises, or a choice where when you get the depth of 3D stereoscopic tech for games, you could lose either resolution or framerate, and that’s a huge trade-off. If you remember, the 60 frames-per-second games would become 30 frames-per-second, or the resolution could become half depending on the tech. In addition to pretty much the same games, only instead we were playing using 3D stereoscopic tech, we had to put up with some lost framerate or responses, or the resolution in exchange for added depth inside the TV.

Virtual-reality tech allows, for the first time in the history of games, [developers] to put the player inside the game, not in front of the TV outside of the game’s walls. They create totally different experiences, so that’s the clear difference between virtual-reality tech, and 3D stereoscopic tech.

Why are you so confident that VR will be different than 3D tech?

SY: If you remember, 3D stereoscopic gaming was actually the consumer-electronics industry initiative. It was not Sony Computer Entertainment’s initiative. It was becoming a large thing from the TV manufacturers industry and of course, Sony has the Bravia team. We were like, “Yeah, we could do it for PS3, so why not?” They have moved on to other things like 4K or now HDR or some Smart TV or the “in thing” of the day. So that’s why, in addition to the compromise we had to make with the gaming experiences, we are like, “Oh, they have moved on. Why do we have to keep trying?” But VR is definitely our initiative. For the hardware and system software and the content is done, so that’s a huge difference.

What do you see as the major differentiation between PlayStation VR and the other major headsets?

SY: The difference between the console-based VR systems that we are working on versus PC-based or mobile-based systems that they are working on – the clear difference is that the philosophy we are approaching PS VR is the same as we’ve always approached PlayStation as a console device, meaning that you don’t need any technical knowledge to use it. You’ve just bought PlayStation in the past and connected it to your TV and enjoyed the game. We want people to just buy PS VR and connect it to a PS4 and enjoy VR experiences. The ease of use, the quality of the experiences – because every single PS4 is the same hardware whether it’s a consumer unit or a developer unit the developers are using to test their games in development. So before a developer or publisher releases these titles, they can make sure the experience is great for consumers. They can playtest and we have our certification test the same way as we approach consoles. Developers can fully test their games with the same environment that consumers have. That’s a huge benefit that we believe the consumer will notice – not just the ease of use of the hardware and the system, but the expected quality of the games or applications that will become available on PS VR. Because we make the same hardware for a long time, we tend to be able to provide a high-tech device for an affordable price.

Virtual reality is the kind of thing where it’s immersive, but you really need to experience it to understand why it works. How do you overcome that when looking at how to market PlayStation VR?

RM: Yeah, you're right. This is one of the biggest challenges. You can't broadcast VR, so you can't see a commercial for it and really get the idea of it. We are trying to do as many outreach things as we can, but there are only so many different people we can reach that way. So really, the answer to this is once the product goes on sale, those initial purchasers will be showing their friends and they'll be able to reach far more people than we can reach with direct demos ourselves. So we need to make sure those people have the best VR experiences to share with their friends, in order to show what the potential of VR is. 

We just expect, or we hope, that it will snowball – first 10,000 people will show 100,000 demos, then 100,000 people will show a million demos. I think that's the only way that people will be able to experience it. It's already happened at my home. I brought it home and I showed my kids, and then they called their friends and bring them over. It just seems like that's likely to happen, and it tends to happen with all new technologies. If they're new enough, you can't really understand what they are. You have to see them, and usually it's through someone who already started by taking the leap.

On the next page, we get into whether or not VR is built for longer experiences and how PlayStation is trying to lure big developers to PlayStation VR.

To this point, we’ve only seen shorter gameplay experiences and tech demos. When do you think we’ll start seeing some longer experiences showing up on PlayStation VR and virtual reality in general?

SY: The reason that we’re showing short demos is because it’s so important for people to try VR. So we have to limit – we have an internal rule that for the demos, they have to be a maximum of five minutes, preferably like three minutes, so that we can get many people to try it. With Rigs, a match can be five minutes, but it’s an online game, so it can be played again and again and again like people do with online shooters. You have seen many devs announce working on adventure games – like a first-person VR adventure game. These games tend to have many hours of gameplay. Also, we just announced that Gran Turismo Sport is going to have PS VR support, and we showed during Paris Games Week DriveClub VR. It’s an amazing, natural experience. Driving games feel more natural to play in VR, actually. Once you try it, you see your body and you see the interior of the car, and perspective and the scale is exactly as if you’re driving a car. It feels so natural and so great, and with driving games people play for hours and hours at a time. 

Of course, you have to wear a headset, and there could be some motion sickness, but it depends on the person. The sensitivity to these things differs by person, but what we have been seeing – especially in the development people who use these headsets for many hours a day – people get acclimated to the use of VR devices. Like when we tried the very first 3D spaced games like Doom for the first time, many of us were not able to play these games because it was too intense, but after a few years, I had no problem playing first-person shooters. Human beings have amazing capabilities to adapt their senses, so that’s going to happen over time. 

The combination of limited resources and investments in the initial years for these smaller developers and the newness of the experience, the gameplay time of VR games for the launch titles could be short, but there will be, from day one, types of games, like online shooters or driving games, that lend themselves to extended play times naturally. As the market evolves in terms of the types of developers and publishers to be able to justify a larger investment to make VR titles, that will naturally extend the gameplay time and the people who are using these devices will get more acclimated for the longer usage.

RM: I think that the gaming experiences at the beginning will tend to be shorter. Part of it is that there are now headsets in the market, so the game developers are using this time to explore a lot of different things, or looking at what works and what doesn't. But we are already starting to see what I would call, “real games.” 

Something like Rigs is very equivalent to something like Rocket League, which you play in maybe five minute increments, but you could play many, many times over. So that's already got that same kind of structure of a real game to it. It's not finished, but it has that same structure of feeling like a real game. We just announced Until Dawn: Rush of Blood at Paris Games Week. It feels like a real game, again, where you feel like you finished some level; you want to do the next level, because you unlocked it now. And it has that kind of pacing of a game to it. Some of the things are more exploratory. Something like London Heist we're showing that as separate pieces, but those are related to each other. We don't show them in one long sequence ever, because that would take a long time to demonstrate, but those things can be crafted together into a game experience as well. It's just right now it's seems better to show them as individual things. 

I think as any new platform, when it happens, when it comes out, there are these first kind of gameplay exploration games, and then there's more depth added to those as it goes on in its cycle of maturity. So I expect that will also happen with VR.

How do you plan on having the PlayStation brand leverage its relationships with game developers to bolster the PlayStation VR lineup?

SY: Talking just about brands, when we announced that Project Morpheus is actually PlayStation VR, people in the industry said they liked the Project Morpheus name and wondered why we changed it. We answered that we want more people to understand what we’re doing in virtual reality. We don’t want to put some exotic name for a general audience. VR itself is new, but when we put the PlayStation brand with VR, people say, “Oh yeah, that’s the VR that I heard about on the news! It’s coming to PlayStation!” So that’s the purpose that we chose PlayStation VR. 

Of course, we have so many developers we have great relationships with – especially the last few years, where our focus has been to support the indie developers. We are such a big fan of small, new novelties and creative games coming from indie communities, and we are working closely with middleware companies like Unity, Unreal, and Crytek. It’s been natural to us to be able to say to these communities that now they have an option to make a VR game. And it’s okay for them to start by making a prototype on PC using Oculus because the Rift was already available to purchase, but as long as they are using middleware like Unreal or Unity, it’s going to be so easy to bring it over to PS VR.

That was before we were able to supply enough PS VR development kits. Now, we can supply in abundance, so some developers may choose to start their development on PS4 for PS VR instead of porting over from PC. Like we support indies who work on PC first for Steam, we have no problem and are very excited to work with developers who are working on any VR experience whether it’s on PC or mobile. We can say that they can move their game to PS VR as well and we’ll support the effort. That’s how we are approaching it.

Of course, we’d like to see our major triple-A publishers to start working on PS VR content as well, and you have seen some efforts and demos released, especially from Japanese publishers like Bandai Namco, Capcom, Sega, and Square Enix. That’s great, and I hope more will be done; like Ubisoft was showing Trackmania on PS VR during E3 already. The larger publishers, because of their enormous resources, they tend to wait for the market to mature before they can justify investing on a larger scale, but I’m very optimistic that this will come. 

This is so early for the VR industry. This is a great opportunity for smaller developers to create something amazing with limited resources. Because the experience is so new, you don’t have to have 100 people to create something amazing on VR for the initial couple of years. That game can come from anywhere. When you look back at when a new market appears for games like Facebook and social media or mobile, it was not the established, large publishers who really hit big; it was smaller, dedicated developers for that medium that became big, like Zynga, Supercell, or Rovio. The same thing, I think, will happen – even though we are working hard – but when this new market appears, it’s going to be developers who are really excited and focused on making VR experiences – games from the ground up – they will come up with something amazing. We are all waiting for that to happen.

On the next page, we talk about overcoming potential health risks associated with intense VR experiences.

With VR being so immersive, how do you take into account series like Until Dawn that are getting VR experiences? I don't think it would be too outlandish to imagine someone having a very strong physical reaction to horror games with jump scares. Do you think there's any concern for people having health problems from these types of experiences?

RM: I think it's similar to other things that already exist. Already if you don't like horror games, you probably aren't going to play Outlast because that would be quite scary. In the real world there's something like a haunted house. If you don't like that, you wouldn't go into it. We do need to communicate what it is. Making it clear that this is a pretty intense experience is important for that. 

And there are other things that are similar. We have an underwater experience. If you're afraid of being underwater, you would know that that wouldn't be for you. I think as long as that's communicated well, people will need to understand it is very powerful. VR tends to be more powerful than most people estimate when they first try it. We do want to make sure the first experiences people have aren't pushing the very limits of what is possible in that space. I think some of the experiences at launch need to be such that they could be pretty universal.

How do you compensate for the varying needs of the different end users? For example, how do you take into account someone who wears glasses or someone who gets motion sick?

RM: I think it's a challenge, and it's also one of our strengths, because we've made a lot of consumer electronic devices and end user products in the past. We have a good hardware team and a good mechanical design team, and I think they did a really good job so far in accommodating glasses, having the whole optic block telescope out so that it can support glasses of different sizes. 

And then also that telescoping out motion is also nice to be able to move the whole front play away from your face and then you can actually look at your phone, or drink something without having to readjust the whole headset. And it's basically one button press and it moves away once you get used to it. We don't usually let people who try it for the first time do it that way, but for us who develop, we can do that all the time. We just move it away from our face very quickly. That's really important to feel like that. I'm only that one kind of button press away from being back in the real world and then also the comfort of trying to keep a nice balance of the weight, that's a really important thing, not pressing the thing against your face was a big design decision, and keeping a good air flow so it did not feel off the face with something and allow the air to go in and avoid a lot of the overheating people complain about if they spent a long time in some VR systems. Those are all considerations the mechanical-design guys thought a lot about.

On the motion sickness front, there are so many factors that play with that and especially since it's a platform that is so conducive for it. How do you go about doing that balancing act?

RM: There are a couple things that, in the hardware, have already been addressed. That's having a low-persistence, high refresh rate with low latency, those are all important things. Then there's some design of the experience decisions that, you don't want to move the person around the space a lot without their permission kind of, so the virtual camera should be connected to the person's head because that's what they expect. Those are some rules that you really don't want to violate, and then when they do want to move through the space, controlling how they move through the space and moving in a way that is conducive to a good experience is what we try to encourage. 

And along with that there's some kind of things that people are still figuring out as we go, but there's a good set of things people have figured out already, which is having some frame of reference that's kind of connected to your body, it helps a lot, so, for example, in Rigs you have a cockpit drawn around you. That helps a lot. Another frame of reference if you don't have a cockpit or something like that is having hands or an equivalent of hands, whatever that might be for your game, and being able to look down and see your hands which are in the right frame of reference for you. Even though you're moving through the world your hands are there with you, it gives you some kind of grounding with the reference frame and that helps quite a bit. And so these things evolve and like I said a lot of the developers have been experimenting with what works and what doesn't and so we're trying to build a best practices document of those things and make sure that we give a good experience. There will be people who want to push, I mean people want to make a roller-coaster experience, those haven't existed exactly on our platform, but they've existed on other ones, and some people love those experiences, so that will exist but you just want to know what you're making.

Kind of along those same lines, are there any safeguards in place for things like glitches? The other day I was playing Assassin's Creed and my character just out of nowhere flew a million feet into the air, and I imagine that would be incredibly jarring in a virtual-reality experience. Are there any safeguards to prevent that in PlayStation VR?

RM: We have some things that are part of the system software for knowing where the player is in the tracking space and things like that, so they can give warnings if you're getting near the edge for example. Our system doesn't have quite some of the issues that you'd have from a system where you'd be walking around and not knowing if there's a wall there or something like that, we don't have to worry about that kind of thing. But the other thing we do have is the ability to exit the game. The same way with the PlayStation button, you can leave the experience, you can do that easily in our experiences so it's something where, I mean, we try to do as much QA as we can of course so that you don't have flying into the world like that. That sounds like a bug they would not have wanted to go through. But if that were to go, if you push the PS button, you can be out of there immediately.

On the final page, we get into how often consumers will be expected to buy a new headset and what the future could hold for VR.

Oculus is planning on iterating its technology rather frequently. How do you view the potential lifecycle of the PlayStation VR headset?

SY: What separates PlayStation VR and the other headsets is the ease-of-use and the very, very close integration between the headset and the mother system, which in our case is the PS4. Because we are making the hardware, the libraries, the OS, and the headset all in our company, we have a really tight integration between the PS4 and the PS VR headset, which means we can extract the best out of the hardware that the PS4 has. We can, for example, reduce the latencies under 18 milliseconds – and still the software guys are tweaking it to improve the performance. 

And in addition, the display we chose is 120Hz True HD OLED and each pixel of True HD has R, G, and B subpixels, so if you count the number of subpixels, our True HD has more resolution, more subpixels, than higher resolution than what you can see on a smartphone. We are adopting the really cutting edge display technology for PS VR and integrating it really, really tightly with PS4. That’s how we are ensuring the great experience coming from PS VR from day one and as time goes by, the PS4 hardware doesn’t change, but because the console allows game developers to really go deep to extract more performance out of. 

When you compare the launch games of PS3 to the late-year games on PS3 like The Last of Us, you can clearly see the difference, even though the hardware is the same. The same thing will happen on PS4 and PS VR. When you compare the launch titles with something in the third year, even though the hardware will be the same, you will see the technological advancement from the consumer standpoint.

Where do you see VR currently and where do you see it going in the next five or 10 years?

RM: Right now, it is a lot about exploring what experiences are good. To be honest, the London Heist for example, shooting in VR, people weren't expecting that to be as good as it was initially. Most are pleasantly surprised that this really works and we really like this and we can build around this. Some of the things about multiplayer are still being explored right now if you're in a VR space with somebody else. So all of that right now is being fleshed out and it's really, really active and creative time to be in VR. 

In five years what you'll see is some of these things will have been established and people will know this is a good thing to do in this space. There will probably some other kinds of things that are not game-related that will also have become popular by then and established. I'm not sure what there will be, our focus is to push forward on the gaming front and keep aware of what's going on the other spaces and make sure our platform can be compatible with them. 

Then 10 years… 10 years is I think when you'll see, we may be at the end of five years or maybe 10 years, is when you'll see another step change in technology that makes some things pretty much fundamentally different. But those are really hard to predict.

Is the team getting any closer to finalizing the contents of the box when you buy a PlayStation VR and what the price will be?

SY: As a company, we’re working on the price-point and what’s in the box and the launch lineup and, of course, launch day, and we have yet to be ready to announce them. We are still saying the first half of 2016 and the price of PS VR might be similar to the cost of a new console, so that’s as much as we are saying as of today.

Click on the banner below to enter our hub of exclusive content covering all things VR that throughout the month.