Meant to be Seen

Hi everyone!

I'll be doing a presentation this year @ HPG 2012 about computer vision and computer graphics. Its going to have some cool augmented reality stuff in there set in the Firefall universe (http://firefallthegame.com), so come by if your in the area and check it out!

- Jon

The video from the conference was released =D

[youtube-hd]http://www.youtube.com/watch?v=yeXWmKzbFwI&hd=1[/youtube-hd]

That is rad. What are you using for head tracking? Is it just the camera on the headset addressing to the play field markers? I didn't really notice any glitching either; really cool!

Incredibele! So cool. Especially the bomb crater. Can you tell anything about the hardware set up? Much respect!

Very cool. Tracking looks spot on too, I did not notice any jumps at all.

The tracking is amazing! What hardware are you using? Can you make a demo video really pushing the tracking? Like, just a cube on the floor with the camera moving rapidly. I'm really impressed; the game looks pretty decent too.

I ran into this thread:

http://www.firefallthegame.com/communit ... ots.43470/

Where you wrote:

All code was written from scratch in just under one month's spare time of my brother Dave and I It uses a markerless AR system derived from PTAM (ours is better IMO). Most of the art is from the PC game (with permission), but a few are from my brother Matt. The game was developed on the PC with attention to the limitations of an iPhone or Android in mind when making it (for an easy port). In other news though Mark Kern may be setting up a deal with a certain HMD vendor - nothing official yet though =D.

Can I conclude that you made this for handheld / Phone, with no markers? Does your version of PTAM work on android and / or iphone / ipad? Looking forward dowloading it on my ipad

http://ewokrampage.wordpress.com/

zoost wrote:I ran into this thread:

http://www.firefallthegame.com/communit ... ots.43470/

Where you wrote:

All code was written from scratch in just under one month's spare time of my brother Dave and I It uses a markerless AR system derived from PTAM (ours is better IMO). Most of the art is from the PC game (with permission), but a few are from my brother Matt. The game was developed on the PC with attention to the limitations of an iPhone or Android in mind when making it (for an easy port). In other news though Mark Kern may be setting up a deal with a certain HMD vendor - nothing official yet though =D.

Can I conclude that you made this for handheld / Phone, with no markers? Does your version of PTAM work on android and / or iphone / ipad? Looking forward dowloading it on my ipad

http://ewokrampage.wordpress.com/

yup All true. The tracking uses only the camera itself and with no markers what-so-ever. It most definitely can work on an iPad, but the final platform it will land on is unannounced at this time. The HMD vendor is silicon micro display for future hardware models

cybereality wrote:Very cool. Tracking looks spot on too, I did not notice any jumps at all.

The camera you see in that video is actually a virtual steady-cam running at about 120fps. It looks and plays absolutely awesome in person MTBS. For the HMD though I'm currently thinking a motion measurement device similar to the ones used in standard VR to improve inter-camera-frame tracking would be better in terms of immersion.

Any papers on what you guys did? Would it be very hard to get as good a result using just PTAM or do you need to have a steady-cam for best immersion?

Very cool demo. Is it actually fun to play this way though? I could certainly see players wearing VR headsets playing in the virtual world and then having spectators on the outside watching the game from the outside on a table or something using AR as you showed

druidsbane wrote:Any papers on what you guys did? Would it be very hard to get as good a result using just PTAM or do you need to have a steady-cam for best immersion?

Very cool demo. Is it actually fun to play this way though? I could certainly see players wearing VR headsets playing in the virtual world and then having spectators on the outside watching the game from the outside on a table or something using AR as you showed

There is, but it doesn't look like its online yet? I don't cover the improvements over PTAM in the presentation too much as the audience wasn't a computer vision audience and probably wouldn't be too interested in the details there. Sufficed to say, PTAM is from 2007 IIRC and it has some very clear short-cuts in what they did that can be improved. First, their choice of corner detector and feature descriptor are not the best - not even close. There has been advancements in that area which significantly improve quality. I'd say while there were other improvements, those are probably the most important ones. In the coming weeks/months I'll be re-writing everything I did from scratch to incorporate some other improvements which should require less key-frames in the algorithm as well as a faster bundle adjustment (hoping for 60fps).

Very impressive. You're right, the paper isn't up yet though other slides are. In any case, was interesting reading up on what's out there. My own attempts at feature detection and the like were back in college, very static, quite mediocre and certainly a few years behind 2007 tech Will definitely be following your work after this! Certainly more fun to see than sparse voxel tree stuff I read from a few years ago, no matter how useful that was

Don't know how I missed this, but that is really cool. How much environmental texture do you need to implement it well. That carpet has a lot of edges to grab onto. Would it work on something rough but homogenous like grass?

brantlew wrote:Don't know how I missed this, but that is really cool. How much environmental texture do you need to implement it well. That carpet has a lot of edges to grab onto. Would it work on something rough but homogenous like grass?

yes, but the tracking tends to be better when you have actual things it can track well. We found that while it works well on a white carpet under normal conditions, the tracking fails at extreme angles on mostly random texture like carpet (haven't tried grass yet, but its similar to carpet). Of course there are always degenerate cases like a flat, textureless walls and in that case there isn't much any method based on a standard camera can do. Since that demo though we've improved things quite a bit in tracking and that is our primary target right now. We want the tracking to be flawless and playable under any reasonable condition.

Could you post a video that demonstrates the current limitations of the tracking? It literally looks perfect in that video I was like

I imagine it would have trouble with something occluding the camera, really violent camera movements and smooth/plain backdrops (as you've said), anything else?

Also, what kind of set up does it need? Just the "move camera sideways" set up? If so have you considered using stereo cameras - could this let you do the set up stage by comparing feeds from the two cameras?

It really looks so close to a perfect solution, hope your development continues as well as it has been going so far.

I'll add to Chriky's questions... Can it handle a 360 degree turn or does it need to be tightly calibrated on a small region?

My question is the same as brantlew's I think, but anyway: can it reacquire its absolute position from point cloud data or other means if the FOV totally changes and then returns?