Meant to be Seen

I got my Hillcrest FSRK-USB-2 tracker chip (the tracker used in the Oculus Rift) 2 days ago and started implementing it as a FreePIE module. It's been quite difficult so far because the Hillcrest documentation is sparse and unfriendly, the sample code has bugs in it, and C# has been fighting me every step of the way. But I think I'm finally past all the hard parts now and expect to have support completed by this weekend.

C# has proven to be a frustrating language choice for FreePIE where module development is concerned because all device API's are written and maintained in C and proxying function calls between .NET and C is awkward. There are sometimes managed wrapper libraries available, but they always seem to be one or two versions behind the most recent API spec - even managed Microsoft libraries. This has been a difficult issue for several modules, but for the Freespace SDK it has been the most difficult. libfreespace (the C version) is on version 6 now but libfreespace.NET is 3 years old and is useless. After several failed attempts to interface the tracker from C#, I just decided to implement the tracker interface in C (via libfreespace 6) and then write a proxy DLL to shuttle the values to C#. A messy solution but one that saves me a lot of hassle.

So hopefully I will have a working module soon.

Cool, how good is the trakcer?

Code completion is soon complete, i'll release a new binary soon, it would be really cool if we could get axis support for the WiiMote since thats the mainstream device.

Yeah its a bit of hassle to invoke C code from managed code, but if we wrote the program in C we would have needed to use the win32 api for the GUI and thats an even bigger PAIN

Easiest way is either to build a managed C++ project and wrap the SDK, or write a unmanaged static C++ library and wrap the SDK.

Calling static C++ methods (As long as they return simple types) is really easy in .NET

The tracker is fine, but nothing special..equivalent to other trackers I have used. The best thing about it is the ease of use. Since the board has an integrated USB and the interface library is mature and already loaded, you literally just plug it in and it's ready to go.

CyberVillain wrote:Calling static C++ methods (As long as they return simple types) is really easy in .NET

Yeah that's basically what I'm doing now. The libfreespace function parameters are complex. The functions return
union {
struct;
struct;
...
}

objects so difficult to map easily to C#. So I'm just wrapping all that with easy functions that return a few floats instead.

Cool. I ordered one of these trackers the other night. I'm hoping its at least as good as the one on the Vuzix, I was happy with that.

@cyber: Here, I'll save you a couple hours of debugging. The example code on the website has bugs in it for the FSRK-USB-2 and none of the compiled examples will work either (except MotionStudio). The bug is all over the code and looks like this...
should be changed to...

Thanks for the heads up.

Can't you just define FREESPACE_SUCCESS as true and save altering the code in all the different places?
Or is it used elsewhere for an actual purpose besides as a 'true'...

brantlew wrote: Yeah that's basically what I'm doing now. The libfreespace function parameters are complex. The functions return
union {
struct;
struct;
...
}

objects so difficult to map easily to C#. So I'm just wrapping all that with easy functions that return a few floats instead.

As long as its structs (Its a value type) it shouldn't be a problem, its classes that's a pain, and a union is just another struct if i recall correctly

Implemented support for the device and uploaded the code. I created a small C dll called piefreespace.dll that proxies data between the FreePI FreeSpace plugin and libfreespace. The C code is checked in as well, but I didn't upload a project or make-file with it, but it has very few dependencies. Developers interested in providing direct C++ support for the device may find that code useful. One thing that I would like to implement later is a way to programmatically recalibrate the sensor direction. No binary build available yet.

Here is the implementation in action.

[youtube]http://www.youtube.com/watch?v=sQOCLonMErQ[/youtube]


And here is the script that I used. (@CyberVillain... I LOVE the new syntax now with the member accessors so thanks for that)

This is amazing news! With Emerson's Biclops, the RIFT will be almost fully compatible with most games right off the bat!
This will help to thrust it out into the common video-gaming space.

I wonder if Palmer will be getting more than he bargained for?

@zalo: Well if what Palmer says comes to fruition, we may be having a whole lot of native game support coming down the pipe. Still, there's a lot of room for hacking to get all the legacy games to work. Not sure if FreePIE meshes exactly with what Emerson & cyber are doing (I think they are going for a direct interface) but it doesn't hurt to have more options.

brantlew wrote: And here is the script that I used. (@CyberVillain... I LOVE the new syntax now with the member accessors so thanks for that)

Lua devs will be confused, but who cares?

I received my FSRK-USB-2 and started hacking with it. It seems the linear Position fields are all set to 0 because the acceleration sensors are not good enough:

Have you tried to implement your own position tracking with a filtering method ?

You can get linear acceleration values from the sensor. Position values cannot be determined accurately. Theoretically you can estimate the integral of the acceleration (sum the area under the acceleration curve) to get velocity and then integrate the velocity to get position. But it just doesn't work because the sensor errors are so large and the double-integration just amplifies that error. At best your position values will be off by several centimeters in just a couple of seconds - and often it will drift meters per second!!!! This is true of all affordable MEMS accelerometers - not just the Hillcrest.

Snap ! I read somewhere John Carmack mentioning the FSRK-USB-2 allowed to build a full matrix update and I assumed it would be a 4x4 with rotation and translation but it seems we can only build a 3x3 rotation accurately....I wonder how this is going to impact the occulus rift: "Yaw" and "Pitch" should be fine but human "roll" movement are also accompanied with translation since the movement starts at the base of the neck....I have never tried a VR Headset so maybe the brain won't notice....

A solution would be to hard code the head translation for the Yaw...

Carmack put in some "fake" translations so that all three rotations model the shape of the head and neck. So for example, looking down is not just a rotation around the camera midpoint, but is a rotations around a modeled head originating at a neck. Same with yaw and roll. These translation animations are "canned" however and not derived from actual translations pulled from the sensor.

brantlew wrote:Carmack put in some "fake" translations so that all three rotations model the shape of the head and neck. So for example, looking down is not just a rotation around the camera midpoint, but is a rotations around a modeled head originating at a neck. Same with yaw and roll. These translation animations are "canned" however and not derived from actual translations pulled from the sensor.

Source ?

I'm pretty sure he mentioned it on one or two of the E3 videos that circulated.

I think he mentions it in this interview (but it could be another, there were a bunch of them):
http://www.giantbomb.com/e3-2012-john-c ... w/17-6164/

brantlew wrote:You can get linear acceleration values from the sensor. Position values cannot be determined accurately. Theoretically you can estimate the integral of the acceleration (sum the area under the acceleration curve) to get velocity and then integrate the velocity to get position. But it just doesn't work because the sensor errors are so large and the double-integration just amplifies that error. At best your position values will be off by several centimeters in just a couple of seconds - and often it will drift meters per second!!!! This is true of all affordable MEMS accelerometers - not just the Hillcrest.

What's the reliability on the acceleration values? As in, can it reliably tell between if the user is stationery and moving in one direction or another? (I'd imagine that the drift would create a small degree of acceleration even when stable - but as long as it reports a reliable difference between stationary and moving, then you'd be able to discard the smaller drifting values.

If the reliability is good enough, wouldn't it be possible to use that data to initiate the translation movement in the view - while integrating slower camera based (i.e. kinect) absolute positioning information, to allow for some degree of view translation that should be adequate (i.e. better than nothing) for now?

The way I figure it - the accelerometers should help start the vector of translation at less than 20ms - the next 50ms, the view point accelerates at any arbitrary rate (determined by the average movement speed of a person), at the 70ms mark (a number I pulled from Carmack's mentioning about attempting to use the kinect (may have heard him wrongly)), the data from the kinect kicks in, overriding the assumed vector intiated by the accelerometers.

Of course there'd have to be some degree of prediction to smooth out the transition from accelerometer data to camera positional data - otherwise the view point would jump back 50ms to the kinect starting. But then doing that would probably add a few dozen more milliseconds as the system attempts to match the assumed trajectory up with the predicted, extrapolated data from the kinect.

But when head and torso movements regularly range into the second+ range, the jankiness of a couple hundred milliseconds may be an adequate compromise to having translation tracking at all. Additionally, there shouldn't be much in the way of overshoot - as you'd use the accelerometer data for stopping to halt the view point immediately, rather than continuing to use the lagging kinect data.

What do you think guys? Am I on the right track here?

I think it's instructive to view what raw accelerometer readings actually look like. Here is a short sample I took at 60Hz, holding an accelerometer as stable as I possibly can and then slowly moving it forward for about 1 meter. You can see the motion along the green axis starting from about 2 seconds to 5 seconds.
This is what we mean when we are talking about IMU sensor noise. And this is a very constrained example. Things get much more chaotic as you start turning and moving at normal speeds. Can you see the motion signal - yes. Is it easy for the software to pick it out - sometimes. But it's rarely as easy as a simple cutoff and you generally need to smooth the signal to increase accuracy which always introduces delay. There is always a fine line that you traverse between accuracy and timeliness. You can have high accuracy and low responsiveness or you can have low accuracy and high responsiveness. Getting both is difficult. Complex mathematics and multi-sensor fusions definitely help but no one solution has really been the magic bullet to solve this problem in the general case with high accuracy.

brantlew wrote:I think it's instructive to view what raw accelerometer readings actually look like. Here is a short sample I took at 60Hz, holding an accelerometer as stable as I possibly can and then slowly moving it forward for about 1 meter. You can see the motion along the green axis starting from about 2 seconds to 5 seconds.

RawAccel.jpg

This is what we mean when we are talking about IMU sensor noise. And this is a very constrained example. Things get much more chaotic as you start turning and moving at normal speeds. Can you see the motion signal - yes. Is it easy for the software to pick it out - sometimes. But it's rarely as easy as a simple cutoff and you generally need to smooth the signal to increase accuracy which always introduces delay. There is always a fine line that you traverse between accuracy and timeliness. You can have high accuracy and low responsiveness or you can have low accuracy and high responsiveness. Getting both is difficult. Complex mathematics and multi-sensor fusions definitely help but no one solution has really been the magic bullet to solve this problem in the general case with high accuracy.

Thanks for the response. I understand quite clearly now the issue - by the time the acceleration signal overcomes the natural noisiness of the data, you've already been in motion for a few hundred millisecond - rendering this on-off initiation unusable.

I wonder if the Freespace lib will be library of choice for Rift games, if so we should work on emulating that lib from FreePIE so any generic tracker can be used, for example from Cyber's Rift driver etc

The Oculus SDK will be the library of choice and we should offer a plugin as soon as the SDK comes available.

brantlew wrote:The Oculus SDK will be the library of choice and we should offer a plugin as soon as the SDK comes available.

I've totally missed that there will be a Official SDK, who will be doing it? As I understand it Palmer isn't a developer?

Oculus has 10 or 15 employees now. I think the ScaleForm guys that joined Oculus have a big hand in the SDK development.

brantlew wrote:Oculus has 10 or 15 employees now. I think the ScaleForm guys that joined Oculus have a big hand in the SDK development.

I have to read up in the RIFT forum I hear

Does anyone have any advice for getting the FSRK USB ver 2 to work with FreePIE? I'm not having any luck. I used GlovePie before to map the mouse provided by the USB module to TrackIR so games will work with 2DOF, but I can't get the script you used to work. It'd be nice to get pitch/yaw/roll in games that support TrackIR.

Maybe I need to set my device to a different mode, don't know...

Thanks

Edit: I did get the Y/P/R values in the latest freepie and mapped to mouse (!) so I think this is just a trackir emulation problem.

FreePIE recently changed their script format from LUA to Python scripting so many of the scripts will not work any longer. It should be pretty simple to change the syntax. If you look at CyberVillain's scripts in this thread you can get a flavor for the new syntax.

http://www.mtbs3d.com/phpBB/viewtopic.p ... 052#p74672

Thanks - I figured that out eventually I added diagnostics.watch() and was able to see the freespace.* values coming from my FSRK-USB-2/FSM-6.

It would be nice to get support built for the 250hz firmware (or just doing XYZ with custom fusion). Maybe just a toggle switch to use the Body Frame or User Frame for XYZ calculation. It still works right now with FreePie but since no fusion is being done it's not smooth.

Something like http://www.mtbs3d.com/phpBB/viewtopic.p ... 035#p90637

I am trying myself with a couple fusion algorithms (like the one shipped with FreePIE) but can't seem to get values in range.

I do think an open source fusion DLL for the Hillcrest is a good idea if tmek wants to donate the code, but I believe something like that should be handled at the "driver" level before it reaches FreePIE. Not necessarily a kernel level driver, but at least in dedicated user space code - like a DLL built on top of libfreespace that just handles fusion and exposes the values. I think there are too many unpredictable timing pathways in FreePIE to guarantee a solid fusion algorithm within the framework since a single thread is used to run the Python script and interact with all the devices. So you can't predict the duty cycle because you don't know what sort of Python loops or blocking operations that thread might encounter on its way back around to your plugin.

brantlew wrote:I do think an open source fusion DLL for the Hillcrest is a good idea if tmek wants to donate the code, but I believe something like that should be handled at the "driver" level before it reaches FreePIE. Not necessarily a kernel level driver, but at least in dedicated user space code - like a DLL built on top of libfreespace that just handles fusion and exposes the values. I think there are too many unpredictable timing pathways in FreePIE to guarantee a solid fusion algorithm within the framework since a single thread is used to run the Python script and interact with all the devices. So you can't predict the duty cycle because you don't know what sort of Python loops or blocking operations that thread might encounter on its way back around to your plugin.

You could handle it in the plugin thread (Have the plugin return a action from the Start method)

CyberVillain wrote:You could handle it in the plugin thread (Have the plugin return a action from the Start method)

Sure, but there's not much benefit from running it in C#. Code samples for fusion and hardware interface libraries are always written in C, so porting it to C# and marshalling all of the data structures is just an exercise in tedium with no clear benefit.

brantlew wrote:

CyberVillain wrote:You could handle it in the plugin thread (Have the plugin return a action from the Start method)

Sure, but there's not much benefit from running it in C#. Code samples for fusion and hardware interface libraries are always written in C, so porting it to C# and marshalling all of the data structures is just an exercise in tedium with no clear benefit.

Just giving all the options, but if you can do it in the driver thats fine

brantlew wrote:I think there are too many unpredictable timing pathways in FreePIE to guarantee a solid fusion algorithm within the framework since a single thread is used to run the Python script and interact with all the devices. So you can't predict the duty cycle because you don't know what sort of Python loops or blocking operations that thread might encounter on its way back around to your plugin.

The PS Move API uses the fusion code from Sebastian Madgwick and it can work at varying FPS with the time period being feeded to the algorithm at each refresh. Maybe that would be a good addition to FreePIE.

brantlew wrote:Code samples for fusion and hardware interface libraries are always written in C, so porting it to C# and marshalling all of the data structures is just an exercise in tedium with no clear benefit.

The fusion code from Sebastian Madgwick which looks like state of the art at this time (ie. better than Kalman filters) is also available with an open source C# implementation :
http://www.x-io.co.uk/open-source-imu-a ... lgorithms/

Also this algorithm is able to do good fusion even at slow frequencies, in his paper the author said that the quality was equivalent at 50Hz and 500Hz, so the 250Hz firmware for the Hillcrest may not be needed.

The paper : http://www.x-io.co.uk/res/doc/madgwick_ ... report.pdf