Most likely the coolest machine vision video you've seen in a long time. 3D object reconstruction using only a webcam!
Posted Nov 12, 2009, 3:51 PM UTC
1.2K362
Unfortunately all I can think of at the moment is speedy prop creation for Source engine games, shame on me.
75
Or a wearable version that creates a 3-d model of your surroundings as you walk around the city or inside a building.
62
And then play in that environment in a source engine game :)
77
Or you could upload yourself into games.
21
gaming future suddenly got much brighter :)
6
Half-Life: U
5
[deleted]
-1
soo am right to guess you like source games?
-4
Better, download the google street views and model the planet.
22
Google has already done rudimentary analysis of its images to figure out where fences, houses, etc are. You can see this when you move the cursor around. I guess due to processing required they haven't made a 3d model of anywhere yet; they also don't have near enough images.
2
God dammit the future is going to be awesome.
16
Yeah, because the present sucks.
/Presses tab and enter. Some crystals resonate at high speeds. Electrons are exited and shoot through a thin copper thread. A grey .png file is replaced by an orange one.
30
Meanwhile another redditor on an iPhone a continent away reads the above, fashions a reply on an illumated touch-screen then turns to the pilot and asks how soon they'll be touching down in Bristol.
15
The file isn't actually replaced, the HTML points to the other one.
THE MORE YOU KNOW...
6
I never tab-enter comments. Using the mouse takes time and serves the same function as a pop-up box saying "Are you sure you want to say that? Think real hard buddy."
3
Yes, sadly most people will not appreciate the awesomeness: http://www.youtube.com/watch?v=-LkusicUL2s&feature=player_embedded
0
This is what will replace Google Street View someday. Just imagine Google vans driving all over the world making detailed 3D models of everything. And then they'd have people just walking around, mapping the insides of every building.
6
Couldn't they use the street view data they've already collected?
And I'm pretty sure I saw a you tube video where they already did this. If I remember it was a different company but the same google van style setup.
5
[deleted]
3
They don't have enough references for each location (say.. a given building) from different angles to be able to produce an accurate model. More 'frames' as they drive down each street and cameras at different heights would in the right direction though..
1
Google is the Central Intelligence Corporation
2
Uh, what? I can't figure out how your link bears out your remark, but I also can't figure out why your accusation makes so much sense.
1
The british ordnance survery are already there: http://www.telegraph.co.uk/news/uknews/6308577/Ordnance-Survey-3D-map-of-Bournemouth-is-most-detailed-ever.html
1
Well google is extremely interested in cell phones right now. Remember that scene from Dark Knight? Replace sonar with cell phone cameras.
1
MMORPG: Planet Earth anyone?
1
You mean kinda like real life?
2
They already are using laser measuring to get shape information... if you take a look at the European recent street views, you can see when you drag your cursor over sides of buildings etc. Hopefully soon they will make Grant Theft Auto - Real Streets
1
.. and then uses that 3d model to project an altered reality scene onto your video glasses.
2
Dammit Vuzix, give us HDMI and cameras already!
1
Oh right. Just wait until the anti-video game hears about that.
"IT'S A MURDER SIMULATOR!"
1
Still good. all I can think is how to use it for porn.
8
Any form of new awesome technology must always first be applied to porn before getting used in other areas.
5
You could capture women once and then have them do anything you wish by the click of a button. cackle
4
Whoo hoo, look at dem ragdoll physics go! Whee!
5
That's actually a rather sweet idea.
3
Isn't that awesome?
3
I wager I could create a model of that thing at higher quality in less time than it took the thing to scan it. And mine would use the proper 15 or so polygons that the thing actually is and not the 200+ the machine calculated.
2
That sounds like a testable claim. How about a youtube video of real-time editing in a 3d modeling program?
22
We would first require a real-time version of this demo. (I'm sure the video was taken at full-speed, but the rendering I doubt.)
1
Well the sidebar specifies that it takes "on-line (live) generation of textured 3D models". Which I would think means that it all took place in real-time.
1
For now, yes. But I bet 40 years ago people used to say "I'll do this on a slide rule just as quick as it takes you on that new fangled calculator of yours".
21
You would become a modern day John Henry! Only to then die, and be replaced by the machine you'd defeated.
7
HA! i am not that geeky... i thought of scanning (naked) girls while they sleep! Maybe i could fap to it afterwards when they leave my house...
0
Interesting that they would choose to use as a model an innocuous church, instead of the obviously intended naked woman.
38
It's image processing, not machine vision, that has the naked women. Although... the pictures have to be cropped.
(Second paragraph in 'Circulation')
Edit: The link is to the 'playboy' article on wiki... so mostly SFW.
6
In 1970, Playboy became the first gentleman's magazine to be printed in braille.
WTF?!??! who would want to actually read playboy?
i mean in the 70s playboy might have been considered to be real porn...
You know whats funny?: nowadays playboy is quite the only porn that we men can openly admit to watch to in the presence of females.
Eventhough I am sure nobody actually faps to it anymore because its lame.
For real hot stuff we still have to be all like "yeah... i really dont know how some men can actually fnd that enjoyable!"
"what? gang midget bondage? nooooo i have never watched such filthy stuff"
i bet in 30 years all our actual hot stuff will be ok to admit in front of female humans and i cant imagine what extremes we will have come to...
-2
How come it didn't model his hand!? Are vampire assistants required?
23
It looked like the assistant kept moving his hand, and every time he moved it, where the new angle no longer had his hand, it subtracted it from the model. In other words it only kept elements from the object that were consistent from all angle keyframes.
37
One of the creators posted in the (surprisingly informative for once) YouTube comments:
The hand, like the background, is discarded from the reconstruction as it does not conform to the rigid body motion of the object. The only assumptions made are that the object is a rigid body and at least part of it is within the white circle you see at the start.
26
In other words, don't try to do anything likely to flex or bend while scanning it.
8
As soon as a demo comes out, I'm gonna make art with things flexing and bending in front of the webcam.
3
It would be interesting to see how far you could do that and have the system keep track.
2
So don't jiggle your boobs.
1
did you not see parts of his hand modeled in and then edited out?
3
I haven't seen the video because my internet connection sucks, but human skin tends to cluster in a pretty specific area of YCbCr. Maybe they took it into account this way? Another option would be some learned saliency function made to ignore limbs but that seems a bit overkill.
Edit: loldownmods.
-4
Edit: loldownmods.
Yes, because if you did watch the video you would see that's clearly not the right explanation.
2
I did watch the video, and there's nothing preventing them from having cheated. In fact, until I see more demos, I find it very difficult to believe they didn't cheat a bit. Not to marginalize their accomplishment (this sort of stuff is really difficult even if you do cheat), but notnaughtknot has a good point, and I doubt there are many people who have a clue what they're talking about downvoting him.
1
as a 3d modeler, allow me to be the first to say. DAY DOOK OUR JOB!!!!!!
23
[deleted]
6
[deleted]
3
[deleted]
2
[deleted]
1
Tulibu Dibu Douchoo
1
yeah, cuz man that 20 polygon church model looks hard to model otherwise.
Interestingly, if this kind of software takes your job away, you could continue to do it by making real-life models for the software to scan.
2
[deleted]
1
which brings us to an interesting point. Using cheap labor in third world countries and using technological advances, are they same indeed?
2
It doesn't seem to create a mesh that you would want to modify much. The triangles are all over the place.
1
true but why would you modify a building. unless you would have parts of it blow out. usually hard surfaces that don't animate can get away with having nasty geometry.
1
Here is a short article about it on Physorg.
21
Link to the developers webpage from the end of the video: http://mi.eng.cam.ac.uk/~qp202/
You can also ask to be updated when demo's come out!
"12/Nov/2009 - In a few months time when code porting is complete, a Linux-based demo of ProFORMA should be released. This will be followed by a windows based demo. To keep updated, please sign up here:"
More detailed video with comments:
18
They have webcam drivers for Linux now?!
4
No, you have to run wine.... then load the drivers.
/Sarcasm
5
nice, thanks for that. although i am not looking forward to installing linux or windows just to play with this!
edit: wait, what platform is it on at the moment?
1
[deleted]
0
Pretty freaking cool.
0
Another thought: could this same technology be used on existing footage? e.g. Rendering a location using footage from a video camera.
12
I would doubt it. It's based on motion detection and seems to rely on the background being stable.
11
It could work as long as the scenery (say a room) is completely static and the camera moves - its the same principle, just flipped.
10
The current system would not have to take account ANY background movement, in your system the background would still move relative to the camera. Saying that it looks like it only senses changes in movement you would have to process much much more data.
Introducing stereo vision would really help I think.
4
It is all based on motion tracking software. I use motion tracking software all the time and indeed you do have to option to track stereo footage, however this process is based on motion - a stereo option works in your head, but the computer does not see it that way.
It looks at the movement of something it considers a static, non changing object and makes a point field based on the tracking points - a separate point field will not really help it and the benefit you might expect from a second view point is replicated by rotating the object.
3
just flipped
Must. Resist. Urge. To. Meme.
2
I am not resisting any urges to meme. This is the same concept as your post, just flipped.
1
with point tracking it's really easy to remove a moving background; you just ignore tracked points beyond a certain depth.
1
It doesn't work like that. It is simply using the same principles as motion tracking software - the exact same principles - it is working out the object based on motion. Only the single rigid object (no moving parts, one solid piece) can be registered by the software - anything else is going to be considered 'noise'. Now, if you were to move the CAMERA instead of the object, then it would consider everything to be the object and MIGHT start creating the room as an object, however I am unsure if they have set up the software to be capable of this.
2
From the physorg article:
The system uses points detected on the object to estimate object structure from the motion of the camera or the object
2
The system is probably designed to assume a fixed background for simplicity. Should be possible to track other parts of the image and work out which are background and which are foreground.
1
perhapss not this exact piece of software, but companies like 2d3 can do it with existing footage.
I believe something similar aided in turning the 2d movie nightmare before xmas into 3d (although I could be wrong)
1
That would be so awesome! Imagine just filming buildings and getting models from it... but I'm guessing that we're not there yet.
1
I'm guessing it doesn't need stereo vision to figure out depth because its looking at the changes in points as you rotate the object? Either way that is phenomenal.
11
Yes, you just need different angles. As the camera does not move, it makes it a lot easier. I have made many 3D models of existing buildings for site models using PhotoModeler and ImageModeler (before it was swallowed up by Autodesk). This sort of modelling is called photogrammetry (or now videogrammetry?).
7
Ten years ago I was playing with Canoma using Kite aerial photos. This little scene rendered in low res for 56k modems was based on 2 photographs taken from about 150 feet up from a kite.
11
Yes, Canoma was the first such program I used. Pretty interface, but it was quite basic. It was for web models, and not that good for architecture.
That Tudor house looks like a great subject for this sort of software. The first major model I did with Photomodeler was also an old country mansion house.
3
This + RepRap = almost teleportation / emailing physical items
10
Yeah, not even close. This doesn't tell you what the object is made of. With wider spectrum imaging you might be able to figure out the surface composition but that still tells you nothing about what's below the surface.
1
What if i use it to send the things I've printed on my reprap?
1
Hmm, but proforma digitizes only surfaces. Today, you could just email the original RR instructions.
1
Or any other rapid prototyper. Why do you say the Reprap specifically? It's cheap but the quality reflects its price.
0
If you combine this with a 3D printer, it will make camming with guys a whole new experience. You don't have to just accept their "seven inches" claim: you can build a model and then print it out and test it on yourself!
Edit: you'd have to have something to reference for size, I guess, and guys could provide a bogus size reference. I'm working on it.
7
Accepting the... odd... premise for a moment, can't you just tell them to hold a ruler next to it?
12
I'm not accepting that premise.
9
Then you are much better off than any of us :(
3
They could print an out-of-proportion ruler.
5
Next, you'll be criticizing our use of Visual Basic GUIs to look at IP addresses.
3
Oh, no, no... I mean, how else are you supposed to do it?
2
This still has the advantage of being able to build a facsimile. Why leave to fantasy what you can hold in your hands?!
0
Dude...no.
edit: I've read your comment history and, well, you really need to keep the flamboyancy to a minimum. Every 10 posts is about a penis or how cute a dude is. We get it, you're gay. Enough already.
7
It's like an automated version of this.
7
Absolutely stunning. I'm curious as to how it knows which "object" to reconstruct, because if you think about it the webcam will feel that the surface of the table and the hand are both "moving" and therefore possibly objects as well?
6
[deleted]
6
Actually, if you look closely, there were flaws in both the mesh and texture where the hand had held it.
17
Additionally, subsequent 'frames', it appeared to realize that the smudged 'hand' object it expected to find was missing, and consequently it removed that from the 3D model.
19
Later in the video those were refined away as the algorithm was learning more and more of the model.
6
It attempted to, yes. But in the corners near the chimney, some stray triangles persisted where it should have been empty.
5
i imagine more "training" of rotating the object would get the computer to deduce more and make increasingly better attempts until it builds up the whole model with less errors.
0
It's based on movement - in this video only the object and a hand are moving. The hand is not stable, it keeps changing and so the software ignores it, but the church is a stable object and it can keep track. If you hand a free-hand camera AND a moving object you would just confuse it and you would not get a good result.
6
A bit like Jurassic park: "They can't see us if we stay absolutely still."
1
Actually, yes.
1
that's not true per se. by tracking points over time you get a 3d location. With a moving camera it will become very clear what is truly in "the background" (meaning distance), anything way far away can be automatically culled. This is how it works with non-realtime 2d->3d systems.
1
Perhaps it will be able to move onto that -indeed I don't doubt it will, but currently it is unable to (hence the static camera) and even in current software, getting a clean point field is not always easy.
1
As I mentioned below, they could also be using colour information to find (naked) limbs and discard them.
-1
This is extremely unlikely, and you can actually see the hand being "modeled" then eventually discarded due to expected models failing to appear as it revolves. This is most clearly shown when the chunk between the upper part of the model and the lower roof is gradually deleted.
8
an interesting example. what about concave surfaces though? The model has pretty sharp edges and simple facets.
6
Pretty sure this is basic research at this stage. But they may get it refined to a point where it does scan surfaces much more complex. I don't see any reason why this wouldn't work with more elaborate surfaces. The algorithm is essentially the same. They may have used something clean and simple to show it working in the video. Acquisition and processing still appears to take more than a second per frame. I think it may be a function of requiring more processing power.
10
there's ambiguity that is resolved only in small details that get hidden or not.
0
That is what I was thinking when watching the video.
1
I bet some 3d modeller will think that the polygon formation isn't clean. A geek out there is thinking the same thing and also thinking of the solution, if not already implementing it.
6
Who cares. If any intern can model that object in 5 minutes, while it would take a skilled 3d modeller an hour to do, I'll find a way to dealwith inefficient polygon formation.
9
I know this isn't really the point, but a skilled 3d modeller could model that object in 5 minutes.
I suppose in terms of content creation the limitation of this process if that you already need a physical object to scan, whereas the majority of 3d content creation tasks involve creating 3d models of objects that don't actually exist.
7
But won't it be more efficient even for pro content creators to make play-dough creatures and then scan them once this technology gets sophisticated enough?
1
Perhaps at some point. there's a lot of stuff 3d lets you do that clay does not. Like symmetry, or creating many variants based on the base mesh.
3
clay takes much longer. There are no macros or keyboard shortcuts for physical objects.
simple objects are easy to dish out. Complex objects just get more complex when modelling it physically
I can see the tech for people who have a webcam and don't have the skills to do it themselves, the money to make some schmuck to do it, or a friend who can do the 3d for free
1
or exist only in google images
1
I would say that is wrong. A lot of modeling is based off clay sculptures that someone created. I believe video game devs build clay models and use a 3d scanner to get that. This technology would allow basically a cheap way to do a 3d scan.
0
No, overall video game devs rarely build clay models; when they do it's almost always for characters, not buildings. Scanned data is really difficult to work with and needs a lot of clean up for video games.
Movies sometimes do for characters; I think Pixar still does. Still, it's not mandatory.
2
Automatic polygon refinement and decimation algorithms have been around for a long time in the 3d scanning world. They do a pretty good job, originally they were developed to take meshes of several million polys and bring them down to a few tens or hundreds of thousands, but they still work well bringing things down to low poly models.
They work based on surface curvature and will maintain a specified tolerance to the original mesh.
1
COOL! I can now make my very own papercraft rpg!
6
As someone trying to get into 3D modeling... I hope this doesn't make me obsolete.
Thankfully the geometry is still pretty awful... but still it's really awesome.
5
Are you good at making 3d objects that can be scanned? Maybe you can work with clay. But seriously, you won't be obsolete. You can't scan creativity into a computer
8
You can't scan creativity into a computer
yet.
6
|
My favorite part of the your comment
2
Fixed it. Stupid Markup. Thanks.
1
no, but you can sure fake it with algorithms
1
What does Al Gore have to do with creativity?
1
You might be able to still "get into" 3D modeling, just as a subject instead of a an artist.
Also, there may still be work for 3D Caricature Artists in malls in the future. Step 1: Capture 3D model of person. Step 2: Stretch the crap out of it to make them look funny. Step 3: Your Dreamworks movie is complete.
3
Looks like making N64 games just got a whole lot easier.
Seriously, though, that is pretty freaking impressive.
5
can't wait until it is refined
2
Can't wait to do my penis.
5
While there is a tendency to want to model large objects such as buildings, I can see there would also be a demand for modeling the very small as well.
2
It doesn't use silhouettes alone either!
Lots of programs I've seen like this one in the past, can only do edge detection against a plain non-textured background.
It means they can't do concave objects (like the wholes in a bowling ball) because the hollowed out bit doesn't create a silhouette.
This one appears to use parallax texture detection... part of the scene with the green arrow rotating, shows the object being sliced up on the face pointing towards us... wow!
4
uses feature detection to make up a series of points which are then tracked....
2
I'm definitely amazed. Very impressive. Does anyone happen to know what kind of machine they're using to do the processing?
3
It's not like that info is in section six of the paper or anything.
Odd that they chose 15fps when the 9000's are quite capable of 30...
6
Thanks. (Admittedly I was too lazy to look). But, I'm even more impressed now.
2
Amazing. I'm working in the field of computer vision myself and this work blows my mind.
The good question is: what processing power does it require?
Making it work on a desktop computer is one thing, and this work is certainly sound. However, making it work efficiently on a battery-powered robot is another.
Outsourcing your computing to a server farm via the internet?
If you want your vision to come with a 2000 ping time then sure.
At the Vrije Universiteit Amsterdam they have a computer vision lab that uses an Aibo (Sony's robot dog) with a 'brain' that covers about a dozen of computing grids from Amsterdam in The Netherlands to Melbourne in Australia.
Apparently even the Australian grid could do useful work for their soft-realtime vision.
Unrealistic. If you are connected via wireless, it usually drains more battery to transmitt than processing.
Source? Last time I checked desktop computers used more than 100 watts at least...
The uploader posted in the comments on youtube that this video was generated on a 2.6 gig dual core with 2 gigs of ram. Makes his feat even more impressive.
I don't think that robots are the main goal of this.
I've been looking into this kind of thing recently in an attempt to come up with a solution to mobile robot navigation using a webcam, and incorporating 3d reconstruction. Just hobby stuff, but I find it fascinating. This guy has some really impressive work along similar lines:
http://www.cs.washington.edu/homes/furukawa/
Wish I could play with his code for "Reconstructing Building Interiors from Images", but he hasn't posted it.
My question is a little more hopeful. When will this tech be advanced enough to turn older films into fully 3d films?
It's not that amazing, if you consider that it is like photogrammetry software that already exists, but instead of the camera taking pictures at different angles, the model moves instead, but the camera being the fixed point. See my other comment. The fact that it takes frames from video, and makes the reference points automatically is the innovation.
This stuff is (nearly) real time. It's an entirely new ball game as far as performance goes. It also does constant refinement of the model not just one scan. It's very dynamic. Yes, it is not fundamentally new as an idea but it's a huge step forward compared to the pre-existing technology in this space.
I think the biggest impact of this would be how it dramatically make a lot of things easier, and essentially cheaper. This will enable a lot of people to build 3D scenes with little cost, and much quicker.
This is a potential downfall though at least until they find a way to properly optimize it. Notice how it takes longer and longer every time it scans, as it is going through and verifying every node its done so far. The more nodes it adds the longer it takes every time going forward.
Buy 10 then or a 100.
Not really, when you consider laser scanning. I work in this field.
Damn, this is like watching two Samurai argue about swords.
That's exactly what the posted physorg article mentioned. Laser scanning is (obviously) much more expensive than using a webcam. Yet the result these guys are getting are just as good.
I look forward to seeing you present your work for public review so you can receive feedback exactly as insightful as what you have provided here.
Oh yeah? Well I work in fields, with lasers, scanning other fields.
[deleted]
you're getting voted down a lot but I really have to agree, this is real time but 3d scanners already exist. Other than this is amazing I would like to hear why this is any better than those in its "amazing" factor, other than a pure cost thing.
I hereby coin kwh's law of false novelty, anytime someone comments on reddit on how amazing a new technology is, someone else will reply that it is not amazing at all and discussion will ensue about how it is actually outdated/boring/stupid/old hat.
That's a dumb law. that's probably not even a new law either. I'm sure other redditors have realized the same thing and have come up with the same idea long before. Let's all discuss how outdated/boring/stupid/old-hat this is.
Nazi!
What in Godwin's name are you talking about?
You are just jealous you didn't think of it first
I though of it 20 years ago. So did a shit ton of other folks. We just had other stuff that we were doing...
It's cool im just joshin ya, I believe you.
I agree. Plus, I do not understand why the feature of 'robust tracking' is even mentioned (it is truly robust).