IAmA guy who has helped develop VR/AR concepts for the last 20 years, including early Google Earth, Second Life and Disney. I'm now helping Bing build an open real-time read/write world combining maps and user-media in 3D. AMA.

Yes. I was very fortunate to also get to spend some time with Dr. Vinge at the last Augmented Reality Event in Santa Clara (he was a guest of honor) talking about this kind of thing. Amazingly cool guy in person too.

Okay, 20 years out, AR will not even exist as a term, except in history books (and by books, I mean the thing that replaces them using AR). The tech will be so ingrained in our daily lives, we won't think of it as unusual to have magical consensual illusions around us and superpowers to control our environment, real and imagined.

Biggest software challenge? In order to overlay every surface with meaningful augmentations, we need to first know what those surfaces are. Otherwise, we're just painting 3D ghosts on markers like the current AR demos.

To do this, we need to understand the color, shape, and meaning of everyplace we look with AR. That's a huge data ingest problem, and computer vision, machine learning, and AI are not quite up to it today.

This is one of the main reasons we're trying to build this data layer using maps and media we can collect today. It's a much more tractable problem to solve. That's Read/Write World.

Here's one. I've been wanting to build and use the "AR contact lens" for about 20 years now. I mostly want it so I can build cool experiences with it -- superhuman vision, true AR, etc..

University of Washington has made some decent progress, but it's still 10 years out IMO. A company called Innovega has announced some near-term version that uses a combination of simpler contact lenses and simpler glasses, which is pretty cool. I'm mostly a software guy, so at best I can only inspire others to solve this particular output problem for me.

However, I'm optimistic this will be solved in much less than 10 years.

Hmm. I'd suggest asking the internet using one of those new-fangled "decision engines."

Yes, but in the angst-filled artist sense of loving and toiling at what one does and not being able to imagine doing anything else.

A writing teacher of mine once described writing a novel as like coughing up a mattress. Having tried it twice (not entirely successfully -- the writing, not the mattress), I agree. Writers write because they must.

It's exactly the same with me and tech. It's all about the breakthrough, and working with some really smart people and playing with really amazing toys... But I often stress about the process until it's done, mattresses and all.

I'm guessing by 'fully' you mean 'indistinguishable from reality.' Because for many cases, I'd say we already have that -- 'immersive' in the sense of a good book, games have been so for a long time. It's more about how your higher brain responds than your lower level senses, right? Haven't you ever played a game and forgot you were sitting in front of a computer? Flinched or started visualizing the map as if it was real?

But in the strict definition (say: realistic tactile/physical-force feedback, photorealistic imagery and analogous sound, unconstrained motion like in the Holodeck, and 3D actors who are indistinguishable from real people), I'd say 10-15 years still.

Tactile is the really hard one, when you consider that it's impractical to make anyone wear a special suit or full-body-force contraption just for playing games. Short of that, you need to stimulate nerves correctly, which is not a very mature science, nor something we should be so casual about. And that still doesn't let you walk freely, which is a dead give-away that it's not real when you hit a wall...

If you can squint a little and design your immersive VR games around the limits of motion and haptics and virtual humans, it's very much sooner.

Perfect human actors are still really hard too, but if you can go stylized the way say TF2 does or otherwise digitize real people verbatim, it will work just fine. The graphics and audio parts are in a race down to commodity hardware and very cheap devices within 3-5 years. The optics have been coming down as well recently, with some impressive solutions within reach.

Those issues come up in any creative profession, afaik. Chefs have trouble not critiquing other chef's work when eating out. OTOH, Chefs probably cook because they love food, and that never goes away.

I guess the best way to say it is that I have really high standards, which drives me to always try to make things better and that includes being [sometimes overly] critical, of myself first and foremost. But I can always appreciate when someone does something well, or perhaps better than anyone did before. Most times I work in areas where people around me are also very excited by seeing something new, which helps, and it's less about competition than a sense of "holy shit that was hard," or only obvious in retrospect.

What got me into this was that I always wanted to make movies, but didn't have the ability to get millions of dollars and hundreds of people carrying out the ideas in my head. Writing novels seemed like a good alternative early on. But I found I was a better technologist, esp. on the visual side of things. So I started creating the tools I wanted to use and have been doing so ever since. Nowadays it's more about trying to improve the world as best I can, but that's part of why I got into storytelling too.

Stupidly, at the time Neal hung out at Worldesign (1992-94), I hadn't yet read Snow Crash so I didn't know how cool he was. When I wrote my first novel, he was very generous and hooked me up with a few agents. (that novel endeavor is worth a whole story someday). I haven't met Gibson yet, unfortunately.

I think there were two different issues on the 'flying penises.'

Oprah was talking about allegedly organized child pornography and all-out rape, which is obviously extremely disturbing (if true). The only problem is the '9000' thing has been mostly debunked. I hope the real number is zero, but alas probably not.

The 'flying' part seems to refer to Second Life where flying 3D phalluses were used to disrupt/protest events. As long as the people present were not kids, it's not the Oprah issue.

But it does prove that with any open system that allows user editing, it's likely that someone will eventually create a penis. In games, it's known as TTC. This seems to have been a trend since cave walls up through bathroom stalls and now on-line. Some cultures embraced this trend in the form of fertility figures. Go figure.

I'm confident we'll be able to avoid the issues you bring up. That's as much of an answer as I can give right now. I will say that we intend to make our client open source, so you'll see soon enough. :)

"Augmented Reality" is really a misnomer. We're not augmenting the world (the world is unaware of our endeavors). We're really augmenting Human Perception -- making visible information that would otherwise be invisible. And with perception (as you probably know), that's as much about what we elide as what we add.

The canonical AR example is a busy city scene with everything labeled, including people and their birthdays. Really? How about showing that scene without ads we don't want to see? How about highlighting the car that might hit me if I cross the street while damping down the visual and audio noise that is keeping my brain busy with the wrong things.

In terms of what to study, it's determining what are the right and wrong things my brain needs to know in any given situation, and how best to present those.

Outside of military (label friend/foe), I don't think this has been studied academically nearly enough. I know Steve Feiner's work at Columbia touches on how to best present tasks in AR for comprehension. Any others?

Yes. The next big shift for GIS IMO is "real-time."

Think about what you can do when you have information streaming in real-time from lots of sources, when your analyses must take milliseconds instead of minutes to days, and when the name of the game is accumulating lots of small data from mostly non-authoritative (but meaningful in aggregate) sources to build up big-data we can trust.

For GIS+Urban Planning, look at an app like Waze and think about what could be done to both visualize and act on this kind of information. (btw, Diann Eisnor has given talks on related subjects; see Where 2.0 and ARE2011).

Check out http://www.hitl.washington.edu/artoolkit/ which should be free for educational uses, afaik. The guy who runs this has been doing AR for something like 20 years as well. Great guy.

Commercial apps like Layar, Metaio and the like are getting much better at using the equivalent of web content skills for content development. Eventually this just turns into the web, but in 3D.

I don't have any strong opinions on modeling tools. I have the most experience with 3DS Max myself.

We recently started doing something with AR and education. We use BuildAR and Google sketchup to make models and scenes. Check us out: http://www.jsnet.eku.edu/ARBlog/ and email us if you have any questions. You can download sample files and projects as well.

I can refer you to people in Bing who would seriously love to hear your feedback. :)

Thanks! I can offer you a free Photosynth app in exchange.

Seriously, I have lots of friends at Google :) But, alas, I've already managed to turn my FB account into an anti-social network by sheer neglect. Besides, as soon as I join Google+ it will no longer be cool.

VR file formats: there are quite a few. You have to decide if you want richness or efficiency for say streaming. Collada is doing very well for the former. VRML never quite took off, but the real answer will be more like HTML+CSS for 3D, plus some 3D models, plus spatial info. The guy who pushed Collada successfully is trying again with 3D models for WebGL. And Read/Write World is working on the spatial aspect full steam.

Map Data: as you point out, crowd sourcing this is the future and a big current focus for this Read/Write World project at Bing, at least in terms of user created content.

However, the key ingredient is that you own your data, including GPS traces and other signals, and are licensing it out under some terms. If you want to contribute (anonymously if desired) to the greater "Data Commons," then that's a good choice IMO. But it has to be opt-in to succeed or people will shut it off.

A slightly less clear-cut option some services take is when it's not explicitly opt-in data sharing, but implicit, like a loyalty card at the supermarket. At least there is a trade of value for information in that case. Perhaps you get to use the real-time map that comes from this in exchange for sharing in some small part of the data collection. Not a bad trade, but the question has to be asked about why can't this be an explicit user choice?

batman map: would be very hard to use the pic-up microphones and lossy compression in phones for a good 3D acoustic map (if that's how it was supposed to work). However, using all of the cameras at once (voluntarily, of course) is only maybe only 5 years out, assuming our phone networks get it together regarding bandwidth.

I've been very lucky to be at the right time and place to work on some very cool things. I'm kind of like the Forrest Gump of AR. :)

Never go up against an author in his own universe. Good authors really make their characters suffer.

I had the honor of working with Randy at Disney when I was there. He did more work with Disney after I left as well.

The Disney team I was on didn't need any persuading on VR, but other parts of the company did and his research was very helpful. It was one of the first large N (100k+) studies of a real-time VR installation anywhere. Honest to goodness science happening in the middle of an entertainment company.

I've also had the honor of working with other luminaries like Jaron Lanier(MS) and Danny Hillis(also Disney).

I loved that movie. I also liked the "Standalone Complex" series, but not in the same way.

If we wind up with viable brain/machine interfaces, there is always the possibility of them being hacked.

But the movie is really about the concept of a soul, apart from the machine and apart from the body. Making such an analogy of bodies = machines is the way they teased out that there must be something more to consciousness.