Pixy (CMUcam5) to include 50hz tracking

[cadcoke5]

On Kickstarter is a greatly updated CMUcam system, the Pixy (i.e. CMUcam5), by Charmed Labs and Carnegie Mellon.
http://www.kickstarter.com/projects/254 ... ion-sensor

It is designed to store 7 different colors, and can track multiple copies of them at 50hz. They do not comment on lag time. IIt can also see a two color tag then report its location and angle. Kick start supporters can pre-purchase for $60.

One of the questions was about head tracking. They replied;

Facial recognition is a more difficult problem than face detection, so let's just focus on face detection (for now). Pixy will not ship with face detection/tracking functionality, but since we've had such a huge response for this functionality, we're committed to bringing it to you as soon as possible. We're looking at a few possible algorithms. We have experience with Viola-Jones and have seen good results (http://www.cmucam.org/projects/cmucam3/wiki/Viola-jones) But-- yes, expect this to come out before the end of the year!

This product obviously has a potential application in VR for position tracking. I can imagine a system with color tags on top of user's VR helmets who are walking around in an arena. Alternatively, I can imagine a user's computer determining a user's view direction and their location by the camera mounted on the user's VR helmet that looks for colored tags around the room.

-Joe Dunfee

[zalo]

For most home VR setups, the PSEye will be ideal. It's $20 (used at the local store), and capable of 75Hz at 640x480 at 75 degrees field of view. The new $60 PS4 Eye will just be ridiculous for the purposes of tracking. 2 x 1280x800 @ 60FPS @ 85 degrees FOV. You might never have to buy another camera again...

The Pixi starts getting interesting when you take it out of the traditional VR space, and move it to something else entirely...

For instance, using face tracking to turn the monitor into a one-wall CAVE Johnny Chung Lee style. Combined with a 120hz monitor and a leap motion, you can probably get some pretty compelling holographic interactions.

[cadcoke5]

For most For instance, using face tracking to turn the monitor into a one-wall CAVE

With large stereoscopic TV's, that CAVE starts to get very available for home users. I just attempted some research on the PS Eye, but I cannot determine if it does its tracking with the internal hardware, or requires the host PC to do that. The Pixy does it all internally. This means it can be made to give position information to a VR Head Tracker.

If there turns out to be significant lag in the track time, for "fish tank VR" that would not be a problem. If something were to appear to wiggle around a bit while it is "underwater", it is not a problem. Though, for a true cave, or even an immersive game on a TV, having your entire world wiggle about would make the Pixy a deal killer. I will inquire about lag on the Kickstarter page.

I imagine the Pixy can be made to put out its data in a variety of formats, since it is open source. The hardest part will be getting the CAD or Game software to accept the data stream, and do something with it.

-Joe Dunfee

[zalo]

I doubt the Pixy will acheive 50hz using markerless face tracking. The algorithms required for face tracking are an order of magnitude more complex than for simple blob tracking.

The PSEye does not do onboard tracking, but chances are you're connecting it to a PC anyway, so that's not an issue. Having the application that does the tracking interface with the VR program is fairly simple (or you could have them both talk to FreePIE). See the Leap using websocket messages to communicate data to programs. You'll get far better performance out of a PSEye+Computer than you will out of a Pixy+Arduino communicating with the computer over serial.

Also, CAVEs are the only thing these newfangled 120hz 3D TVs are good for. They'll enhance the experience quite a bit.

Check out the Code Laboratories driver for the PSEye.

Of course, why bother with simple head tracking when The Eye Tribe now has a monitor mounted Eye Tracker that you can get for $99? http://theeyetribe.com/

[cadcoke5]

I doubt the Pixy will acheive 50hz using markerless face tracking.

Since for 3D, you are always wearing something in front of your eyes, adding a marker is certainly not a barrier.

The PSEye does not do onboard tracking, but chances are you're connecting it to a PC anyway, so that's not an issue. Having the application that does the tracking interface with the VR program is fairly simple ... You'll get far better performance out of a PSEye+Computer than you will out of a Pixy+Arduino communicating with the computer over serial.

If the PSEye has to send a lot of data over the USB, that will itself introduce a lag. I also remember trying out some mouse control with a camera doing some movement detection (I am not sure it was true face detection). There was definitely a lag issue when I used it with the Inventor CAD [Computer Aided Drafting and Design] program. But, this was 10 years ago. So much changes with time that it is hard to keep track of all the players. The bottom line is that the reliability and lag time numbers will indicate who is the current winner.

Of course, why bother with simple head tracking when The Eye Tribe now has a monitor mounted Eye Tracker that you can get for $99? http://theeyetribe.com/

We may be on the edge of an explosion of tracking technologies. Though, on the Eye Tribe web site that they base their technology on seeing the reflection of IR LEDs in the user's eyes. So, wearing a headset or shutter glasses will likely kill their particular technology for VR usage. Still, over the next year, I imagine there will be a zillion new potential devices to check out for VR applications.

My personal usage would not be for games. Rather, I would want it for designing things. CAD programs are less likely to have the necessary software hooks to use any of the open source interfaces. The CAD vendors themselves would have to want the technology to take hold, and give up on the idea of charging a zillion dollars for such an upgrade.

-Joe Dunfee

[zalo]

Oh, I was writing all that under the One-wall cave idea, which is probably better for cad and eye tribe, but worse for the Pixy.

Something I should probably mention: from my more modern experience with optical tracking w/ the psye, latency over USB is negligible (and unavoidable). Whether you have a camera offloading processing onto the computer via USB or the arduino uploading emulated serial information over USB, the latency from USB won't matter nearly as much as the speed at which you can process the data (and get it sent to the CAD program if its not already the one DOING the tracking).

The eye tribe might work with the rift if you put two spots of IR retroreflective tape on the front (you can buy some near the bottom of this page: http://www.naturalpoint.com/optitrack/p ... s-markers/ ).

[Fredz]

If the PSEye has to send a lot of data over the USB, that will itself introduce a lag.

The PS Move API has a 68ms total end-to-end latency using the PS Eye to track a PS Move glowing orb on Linux. I guess processing should be a bit more complicated for AR markers but I don't think it would necessarily induce a significant lag.

See : http://thp.io/2012/thesis/ (details in the Thesis PDF)

[FingerFlinger]

This is a post from the creator of the CL driver on latency:

Internal LED Capture Latency Test

I implemented a simple latency test program (it will be included in the SDK) that uses PS3Eye’s internal red LED to trigger the latency timing. The program is implemented using two threads, one that captures and checks the video frame for LED light and other one that after a random interval turns on the camera’s internal LED.
The camera was then put in the dark shoebox with a mirror in front of it (so that can clearly see its red LED). The blue LED was masked out.

Here are the latency results (each one was run for a few minutes):

Grayscale Mode

- 640x480 @ 75fps Latency Min: 11.6151ms Max: 25.565ms Avg: 18.2541ms
- 640x480 @ 60fps Latency Min: 13.9225ms Max: 30.373ms Avg: 22.0655ms
- 640x480 @ 30fps Latency Min: 23.1228ms Max: 56.288ms Avg: 39.7403ms
- 640x480 @ 15fps Latency Min: 38.0911ms Max: 104.65ms Avg: 70.1092ms

Color Mode

- 640x480 @ 75fps Latency Min: 15.3318ms Max: 28.234ms Avg: 21.5017ms
- 640x480 @ 60fps Latency Min: 17.2339ms Max: 33.576ms Avg: 25.2352ms
- 640x480 @ 30fps Latency Min: 23.5517ms Max: 56.527ms Avg: 38.9832ms
- 640x480 @ 15fps Latency Min: 36.9829ms Max: 102.48ms Avg: 68.4732ms

From the results we can see that the both grayscale and color modes give us consistent results.
We can conclude that the average latency given frame rate is ~1.5 frame time.

AlexP

Thread here.

A naive examination of his results suggest that the transfer latency is between 5-7ms.

[cadcoke5]

I have an answer in regards to lag on the Pixy. I asked that question on the Kickstarter page, and here is the reply;

Charmed Labs and Carnegie Mellon says:

Hi Joseph,
The lag time (latency) is between 20 and 40 ms depending on the number of
objects being tracked. Less than 10 and the latency is very close to 20
ms. The latency is the amount of time between the event happening (ball
moves) and the updated information appears on the communication port.

thanks for your interest!
--rich

-Joe Dunfee