hile Johnny Chung Lee’s name might not ring a bell, his work with the Wii’s Remote has been viewed millions of times on YouTube. Lee’s innovative projects use the input device in some mind-bogglingly clever ways, from using it to track users’ fingers, to creating inexpensive multipoint interactive whiteboards (which is much, much cooler than it sounds), and creating a head-tracking system for VR displays. If you’re not familiar with these videos, you need to watch them immediately before reading on.

We spoke with Lee about his projects—which started off as a procrastination tool from his PhD studies—and his thoughts on how his ideas could affect gaming in general. With the attention the Wii has received for its intuitive and interesting game controls, it’s great to see someone push hard and inspire others to see that the system lives up to its potential.

Game Informer: Can you talk a little about your background and what led to your work with the Wii?

Johnny Lee: I’m a PhD graduate student at Carnegie Mellon University studying human-computer interaction. I’m actually just finishing up my PhD. I’ve been doing a lot of work and am in a community that focuses on new ways to interact with computers, either by using innovative software or hardware technology. The Wii Remote stuff actually started off as sort of a procrastination project, because it’s not my primary thesis topic. I was basically playing with it to do something on the side. The Wii Remote has a lot of really cool devices in it, and it’s an easy platform to experiment with. So these are just projects that I thought would be fun to explore personally, and it turns out that a lot of other people find them interesting as well.

GI: Have you found that the Wii stuff has dovetailed into your PhD thesis work at all?

Lee: A little bit. Toward the end of my thesis work I was looking for high-performing tracking technology. It turns out that the camera in the Wii Remote is pretty good and is relatively easy to work with. So, for some of the demos late in my thesis work, I talked about how it could be done with something like the Wii Remote. My work uses a lot of infrared-light-sensing technology, so it’s also relevant to the Wii Remote—not quite in the same way, but it’s relevant.

Using Lee's technique, the Wii's Remote can keep track of a user's fingers.

GI: Did you gravitate toward the Wii because of the system itself, or was it simply a good way to get all of the components in a relatively inexpensive package?

Lee: Probably more of the latter. I do play video games and I like video games, but I was much more excited about the capabilities of the controller than I was the Wii game console itself. It’s amazingly impressive that Nintendo just allowed the ability to connect it to a computer over Bluetooth, which I think has been great for both the hobby community as well as the research community. It’s a pretty amazing piece of engineering, containing a high-speed infrared camera, accelerometer, plus it has Bluetooth communication, a vibration motor, speaker, buttons—and it does this all running off two AA batteries for a long time. If you had talked about this a few months before it was released, I probably would have said it could not have been done, given my exposure to research. But they did it, and I was massively impressed.

GI: Have you worked with other game tech in the past?

Lee: A little bit. There’s sort of the consumer game technology and then there’s the experimental game technology that you sometimes see at conferences like SIGGRAPH [Special Interest Group on GRAPHics and Interactive Techniques] and UIST [User Interface Software and Technology]—these are sort of the big either industry or academic research communities, and sometimes they have prototypes of things you could do in the future but aren’t necessarily mass market yet. There’s things like eye tracking and 3D trackers and face-recognition systems. All of the high-end ones are pretty difficult to work with or expensive. Things that are at a consumer level are typically normal game controllers, with joysticks and buttons, so there wasn’t anything too interesting there to play with. So the Wii Remote was the first consumer product that I thought that was really interesting to play with.

GI: So you never messed around with the NES Power Glove or anything?

Lee: No, not really.