Can you ‘shake the pepper’ with Natal?

Wouldn’t you know that just as I was about to start translating/transcribing another VR interview about, I came across a video of Microsoft’s Natal project.

Launched yesterday, the project’s already generating buzz. I’ve read some critical speculation in the blogosphere about whether the scenarios shown in the video are feasible or not. What do you think? journalist Lev Grossman explains how Natal works:

What they came up with is a kind of self-contained module that you add onto your Xbox 360. It has a video camera in it that tracks where your body is and what you’re doing with it. It also has a monochrome camera (it works with infrared) that reads depth — how far away your body and its component parts are — and a highly specialized microphone that can pick up voice commands. Along with all this hardware, it’s got a ton of software that tells the Xbox how to find your body’s various joints (it tracks 48 of them), how to keep track of multiple players at the same time, how to tell your Hawaiian shirt apart from the colorful wallpaper behind you, and so on. Microsoft even did an acoustic study of living rooms, so Project Natal can tell when you’re talking, when your buddies are talking and when somebody in the game is talking, so it knows whom to take voice commands from.

I can’t help but think about my chat with Christophe Chartier published in Equipping Our VR Future. He said:

If we want the development of VR equipment to be really mainstream the technology must be transparent. So forget set-ups including specialized gloves and headsets, which are today’s emblems of virtual reality equipment. In my opinion they won’t permit the mass deployment of VR technology. The set-up should integrate with the home environment, and for this we need equipment that’s more and more intelligent.

I think Natal will bring us a step closer. Yet it’s not quite in line with David Nahon’s description that VR is about,

Sensing and acting, perceiving and acting in the virtual world with your body.

After all, you’re sensing and perceiving in your living room, no matter how large the screen.

I’d certainly like to try it out though. ‘Beats my Intellivision days! Although I’m not sure I could play BurgerTime on Natal. Shake that pepper!



  • Marc D.

    Kate, given my knowledge regarding motion capturing and tracking I don’t think we are quite at a point where things like controling a football (soccer simulation) or the “choose your gear” freedom is something current hardware can really provide (but I might be very wrong of course).

    Also I’m not sure if there’s a real market for such kind of “blow your mind away full motion” interface. I own a Wii and what I like with it is that with common simple gestures you can control a character – but the concept is not neccessarily that you 1:1 do the same gestures/motion that your virtual ego does. It’s more like “I do a small gesture and the virtual ego does a bomb-knockout in the boxing ring” (although Wii boxing can already be quite sweaty ;-). I mean: if real people want to do real sports … hey, why not go outside with your football and play soccer with friends? If you like martial arts and want to learn Kungfu … go to a fight club and learn it for real. If I need to do 95% of the moves I’d need to do in real life, I can do them real life and learn something for real.

    Also, what I really don’t like (and this maybe due to the fact that I love my Wii and that I am so exited to see Nintendo’s success with this piece of simple hardware with a completely new “killer” interface) is that Sony and Microsoft kept telling that they don’t want to go the Wii path with (how they call it) “casual games” but they want to find a different path of success. Now, Microsoft is doing what they often did: trying to copy a great concept and squeeze $$ out of it (my argumentation is not fair, I know. And if MS’ interface is really that good we will have a new console market and the buyers will be able to take a free descision which concept they like better – that’S all our market economy ;-).

    At a first glance, this looks super-cool … like a fantastic gimick, a must-have … but when I think twice, for me it just appears to be Microsoft vaporware they are announcing frequently in glossy videos – but never does come true.

  • Kate,
    Great article! These types of interaction will allow brand new experiences, like the Wii brought a couple of years ago thanks to the Wiimote.
    Note, Microsoft was the first one to “shoot”, but Sony is closely following with a similar 3D camera, which they announced at E3 yesterday afternoon for the PS3:

  • KDI

    Ohh yes. this technology seems completely amazing. However a technological showcase is no product at all and unlike Sony (with their Playstation Eye and their “wand”) and Nintendo (Wiimote + Wii Motion Plus), Microsoft didn’t annouce any release time frame for Natal. (Do you remember this wonderful interactive screen-table shown by microsoft some time ago?)

    It feels like Natal opens a new world of interactivity but so did the EyeToy, and MS camera : both attempts failed because of the lack of compelling software/line-up. I still have to see it for myself but going controller-less is risky as you completely remove the already limited tactile feedback the controllers provide today. IMO, this technology, if it ever gets a release date, could strive in applications where very short-time response are not needed, low real-time interactivity applications : it seems to be a wonderful proposal for the media entertainment center, for educational software and other life-style applications. Milo, demoed by Peter Molyneux (LionHead), could be great at baby sitting/entertaining your kids while you’re busy with your chores, telling stories, reading books to people adapting to their state (feeling asleep, or away for a break…), monitor and correct your movements while you’re exergaming, telling you that the onion slices in your tagine are not fine enough, guide you while trying to mount your last IKEA gift…

    Even without a touch feedback, manipulating a 3D object to examine it, review it could be much more accessible thanks to the Natal interface, given its ability to detect the body joints. I could go as far as to say that our (DS) “consumer” products/life-style software could greatly benefit from such a technology.

    However, I doubt that playing catch up on the motion sensing arena with an add-on will help them convince the casual crowd that Nintendo owns today. Price point, market penetration, consumer perception, killer-apps… Good luck to Microsoft on that area. Perhaps we could help them. What do you think?

  • Well, the video is a mockup, not even a prototype. At the r&d level, the technology for all of these capabilities already exists, but getting it both useable “in the wild” and at a reasonable pricepoint is going to be difficult.

    I am not particularly wowed with this mockup even as such; there is very little that is new here. Most of the gestural metaphors we’ve seen elsewhere.

  • Wow, some passionate perspectives ! I’ll try to respond to each of you.

    @Marc D: I also question the “blow your mind” market. I guess it depends on what developers do with this. For someone that lives in an urban setting where space is more limited that in Seattle, the mixed reality gaming applications would have to fit given my physical dimensions and square meters. But then again, if I’m going to play sports, it’s for real! I could see myself high-speed car racing on Wii or this though. That’s something I would never do for real! But for people who are into high-end home entertainment systems and toys, I’d imagine they’d be all over Natal. It will be interesting to follow how people really use it.

    @Virgile: Thanks for your gaming industry perspective; sounds like you think this is a big step in what’s possible down the line. Nice link about Sony’s 3D camera announcement. One of the people that commented said they’d rather buy an “Xbox 720 or PS4 rather than a couple of overpriced peripherals that will quickly become obsolete.” Excellent point. I guess the take-away is that the technology is advancing to provide great new possibilities, and like with anything equipment related, the more you wait, the more integrated, cheaper it will be.

    @KDI: I agree that the body-joints detection could be very interesting for the world of product design and manufacturing! I wonder what type of spill-over we’ll see into the industrial world? And I agree with you that Natal will only be as good as the quality of the games people develop for it. But then that’s so subjective. For example, I’m not at all compelled to use Milo for babysitting my kids! How could Milo prevent one of them from getting injured? I see the applications being more interesting in retirement homes where people cannot get out and really go bowling. Maybe ironically, Natal applications will best serve the ageing.

    @Michael: Price is always an issue with new technology. I’d think it’d mellow down with time—once this hits the market. When will that be do you think?


  • It’s possible to get a limited version of what they show into a system today for reasonable cost. The hardware (camera and fast cpu) are nearly capable already. By “limited” I mean single-person with gross mocap and voice recognition, *maybe* handling 2 people who don’t stand too close to each other and use a very limited set of commands.

    But for this mockup to be (exactly) a product, the segmentation problems have to be solved both for voice and video: properly tracking, separating, identifying, and modeling multiple people in one space. This is a much-studied research problem, but I’d wager it’ll be 10 years before you can get the system “as shown”. At least.

  • KDI

    @Kate : Probably not babysitting per se, but entertaining AND watching them in the living room while you’re busy elsewhere in the house… A bit like a surveillance camera that also knows how to talk and show you things…

    @Michael : I read a blog post from the “wiimote head tracking guy”.
    He claims he works on the project and that the work done by MS on the subject is far beyond any academic research he has ever seen. I don’t know how much credit we can give him, but, still, his claims are quite strong.

  • @KDI: That’s an interesting article. Even more interesting are the 102 comments people left. I stopped at around 30 though. The comment that intrigued me the most was asking if Microsoft really has the gaming community in mind for Natal. What industrial uses would profit from Natal?

  • I see that on the NYT blog they posted the impressions of someone using the product:

    This performs “as expected”, and should end any debate about the similarities between the original promo video and the product’s actual performance.

    It’s very easy to make a great video showing advanced capability, but there is a danger to implying a lot more than you can deliver.

  • Thanks for the great article, Michael. It’s always good to hear from an outsider who’s actually tested Natal.

    I’m surprised nobody’s mentioned Softkinetic?

    I got to try it at our developers’ conference last year, and it’s cool. When I say the Natal video I thought abotu Softkinetic actually.