It is currently Sat Aug 19, 2017 4:30 pm



 [ 12 posts ] 
 Leap Motion Plugin Development 
Author Message
One Eyed Hopeful

Joined: Fri Mar 28, 2014 2:09 pm
Posts: 2
I started creating a Leap Motion plugin for FreePIE using the latest Skeletal Tracking Beta SDK. The current version, which can be downloaded below tracks the pitch, roll, yaw, x, y, z positions of the right most and left most hands. I also added a single detectable gesture (typing with index finger) just to have some input to get past the calibration stages of Razer hydra emulated applications. It currently doesn't differentiate gestures by hand and needs a lot of work, but my focus right now is on the position and orientation.

Current Version Download Link: https://dl.dropboxusercontent.com/u/413 ... in-001.zip

It currently needs quite a bit of work, but the basic hand tracking is in. I got some poor, but promising results emulating the Hydra in the Tuscany Hydra demo. A big part of my problem is that I don't have a Hydra, so I don't know the actual ranges used/expected. For example, normalizing the x,y,z position in the leap motion results in a 0 to 1 range relative to the left and top of the field of view around the Leap. They refer to it as the interaction box: https://developer.leapmotion.com/docume ... onBox.html

The script below is what I am using by trial and error that gave me the best results emulating the Hydra so far. I included the diagnostic.watch for each of the existing values captured by the plugin for your reference. The orientations are on -/+ pi on each axis. The positions are normalized from 0 to 1 and I converted them to -1000 to 1000 range, in the script below, because some of my initial searches turned up Hydra positional values in this range, but that's only a guess at this point.

If you want to use this very early version of the plugin,
1) Acquire a leap motion or take it out of that drawer
2) Install the Latest Skeletal Tracking Beta https://developer.leapmotion.com/
3) Install .net 4.0 or higher framework if you don't already have it installed
4) Unzip the archive into your Freepie folder so the LeapMotionFreepiePlugin.dll file is in the plugins folder (e.g., C:\Program Files\FreePie\plugins) and the other dlls (Leap.dll, LeapCSharp.dll, LeapCSharp.NET4.0.dll, msvcp120.dll, mscvcr120.dll) are in the root of the Freepie folder (e.g., C:\Program Files\FreePie)
5) If you want to emulate the hydra copy the sixense_fake.dll (located in the root folder of your FreePIE installation) over the sixense.dll in the folder of the application you want use this plugin with.
6) Run a script like the one below, but hopefully better and more compatible with the Razer or any device you want to emulate with the Leap supported by FreePIE.
7) You will need to bind a key to the triggers of the hydra 0 and 1 for the calibration stages (e.g., mouse click). I used my gesture, which is a bit wonky right now, but it's serviceable.

Some caveats, right versus left hand detection only works when two hands are detected. It detects the right most and left most hand. So one hand is detected as both right and left, since it is technically the right most and left most hand when its alone. At a later stage you could restrict left and right hands to either side of the Leap to avoid glitchy behavior. But that introduces a few other issues, like restricted movement (left hand couldn't cross over to the right side of leap physically).

I hope someone with experience with the Leap can help test and sort out the translation to other devices like the Hydra, either by suggesting improvements to the script or my Leap API/plugin development, which I just started learning. The big issues right now are the positional/orientation translation to Hydra, and figuring out how best to emulate calibration with the Leap (i.e., where should your hands be when it says touch your shoulders or point at the base).

Update 6/27 - I had better results using the Sixense test app from the SDK. It also has very nice logging capabilities to see the hydra reported position and orientation for comparison against the leap values. The position mapping worked surprising well, as well as the pitch, yaw, and roll (once reversed). I experimented some with different ranges, and increase the range of the x position increases the currently restricted range of motion left to right (e.g., 1000 to 2000). I don't think the calibration will be much of a problem; the sdk documentation says its mainly for determining the hemisphere of the unit relative to the base. Make index finger typing motions with your right hand to emulate hydra button presses to move past the calibration, if you are using my example script.

Link to Sixense app (it's in the samples/win32 folder) : http://sixense.com/windowssdkdownload
Don't forget to replace the sixense.dll with sixense_fake.dll (discussed above).

Code:
diagnostics.watch(leap.rightpitch)
diagnostics.watch(leap.rightyaw)
diagnostics.watch(leap.rightroll)
diagnostics.watch(leap.rightxpos)
diagnostics.watch(leap.rightypos)
diagnostics.watch(leap.rightzpos)

diagnostics.watch(leap.leftpitch)
diagnostics.watch(leap.leftyaw)
diagnostics.watch(leap.leftroll)
diagnostics.watch(leap.leftxpos)
diagnostics.watch(leap.leftypos)
diagnostics.watch(leap.leftzpos)

diagnostics.watch(leap.typekeytapgesture)


hydra[1].yaw = leap.rightyaw
hydra[1].pitch = leap.rightpitch
hydra[1].roll = -leap.rightroll
hydra[1].x = filters.mapRange(leap.rightxpos, 0, 1, -1000,1000)
hydra[1].y = filters.mapRange(leap.rightypos, 0, 1, -1000,1000)
hydra[1].z = filters.mapRange(leap.rightzpos, 0, 1, -1000,1000)



hydra[0].yaw = leap.leftyaw
hydra[0].pitch = leap.leftpitch
hydra[0].roll = -leap.leftroll
hydra[0].x = filters.mapRange(leap.leftxpos, 0, 1, -1000,1000)
hydra[0].y = filters.mapRange(leap.leftypos, 0, 1, -1000,1000)
hydra[0].z = filters.mapRange(leap.leftzpos, 0, 1, -1000,1000)

hydra[0].trigger = leap.typekeytapgesture
hydra[1].trigger = leap.typekeytapgesture


Updates:
6/27 - Updated example script to use mapRange function to increase legibility.


Last edited by nonverbal on Fri Jun 27, 2014 9:53 am, edited 4 times in total.



Thu Jun 26, 2014 9:50 pm
3D Angel Eyes (Moderator)
User avatar

Joined: Sat Apr 12, 2008 8:18 pm
Posts: 11355
Nice work.

_________________
check my blog - cybereality.com


Thu Jun 26, 2014 11:20 pm
Petrif-Eyed

Joined: Mon Jun 22, 2009 8:36 am
Posts: 2156
Location: Stockholm, Sweden
Nice work! I hope you get some help from Leap owners! This could be a built in plugin once its tested and complete

_________________
FreePIE
My blog


Fri Jun 27, 2014 2:22 am
One Eyed Hopeful

Joined: Fri Mar 28, 2014 2:09 pm
Posts: 2
UPDATE 7/2 - I added much improved gesture support.

Version 002 linked below, now distinguishes between left and right hand gestures. I also added three gestures that are detected 1) Hand Swipe (regardless of direction at the moment), 2) Finger Key Type (regardless of finger, downward motion like typing an invisible keyboard), and 3) Screen Tap (forward finger motion like pushing a button on screen).

I also exposed the properties of this gestures to better fine tune them from the FreePIE interface for different applications. These control the threshold for what is considered a gesture. The current properties and their default values are:

SwipeMinLength - 150 mm
SwipeMinVelocity - 750 mm/s
KeytapVelocity - 50 mm/s (minimum downward velocity)
KeytapMinDistance - 3 mm
KeyTapHistory - .1 sec (the time window for evaluating movement)
ScreentapVelocity - 50 mm/s (minimum forward velocity)
ScreentapMinDistance - 5 mm
ScreenTapHistory - .1 sec (the time window for evaluating movement)


Link to version 002: https://dl.dropboxusercontent.com/u/413 ... in-002.zip

Right now the plugin relies on the DoBeforeNextExecute method where it grabs the current frame from the Leap and reports all the gesture and hand data. Is this the preferred way to write plugins, or should it be more event/listener driven from the leap itself when it registers a valid gesture? At the moment, I am just checking the frame during each poll and if there is a detected gesture, determine the hand and report its status via a property in the global plugin object. What is the update rate of FreePie? I think under optimal lighting/computer conditions the Leap motion is capable of up to 200 fps.

If anyone has had any opportunity to test/find bugs problems or wants features I don't currently have, let me know. I have been working on the tracking individual fingers and a tool (e.g., pencil) to further improve gesture recognition and accuracy (i.e., limiting recognition to specific fingers)--right now it is watching all fingers for a valid gesture occurrence.

Here is an updated testing script to emulate the Razer Hydra that includes diagnostic.watch for all the current values provided with this plugin.
Code:
diagnostics.watch(leap.rightpitch)
diagnostics.watch(leap.rightyaw)
diagnostics.watch(leap.rightroll)
diagnostics.watch(leap.rightxpos)
diagnostics.watch(leap.rightypos)
diagnostics.watch(leap.rightzpos)

diagnostics.watch(leap.leftpitch)
diagnostics.watch(leap.leftyaw)
diagnostics.watch(leap.leftroll)
diagnostics.watch(leap.leftxpos)
diagnostics.watch(leap.leftypos)
diagnostics.watch(leap.leftzpos)

diagnostics.watch(leap.righttypekeytapGesture)
diagnostics.watch(leap.lefttypekeytapGesture)
diagnostics.watch(leap.righttypescreentapGesture)
diagnostics.watch(leap.lefttypescreentapGesture)
diagnostics.watch(leap.rightswipeGesture)
diagnostics.watch(leap.leftswipeGesture)


hydra[1].yaw = leap.rightyaw
hydra[1].pitch = leap.rightpitch
hydra[1].roll = -leap.rightroll
hydra[1].x = filters.mapRange(leap.rightxpos, 0, 1, -1000,1000)
hydra[1].y = filters.mapRange(leap.rightypos, 0, 1, -1000,1000)
hydra[1].z = filters.mapRange(leap.rightzpos, 0, 1, -1000,1000)

hydra[0].yaw = leap.leftyaw
hydra[0].pitch = leap.leftpitch
hydra[0].roll = -leap.leftroll
hydra[0].x = filters.mapRange(leap.leftxpos, 0, 1, -1000,1000)
hydra[0].y = filters.mapRange(leap.leftypos, 0, 1, -1000,1000)
hydra[0].z = filters.mapRange(leap.leftzpos, 0, 1, -1000,1000)

hydra[0].trigger = leap.lefttypekeytapGesture
hydra[1].trigger = leap.righttypekeytapGesture




Wed Jul 02, 2014 8:32 am
Petrif-Eyed

Joined: Mon Jun 22, 2009 8:36 am
Posts: 2156
Location: Stockholm, Sweden
The default frequency of freepie is 60hz (OS default system clock)
It can be overridden for example

Code:
if starting:
   system.setThreadTiming(TimingTypes.HighresSystemTimer)
   system.threadExecutionInterval = 1


Would set FreePIE to run at 1000hz

I like the event driven model, that the plugin has a update event so that your python script only does something if new data has arrived. Note that this doesn't neccery mean that your plugin needs to use the Leap API's event model.

Also python isnt thread safe so if you add a update event to the plugin make sure you trigger it from the DoBeforeNextExecute method, this will ensure thread safty

_________________
FreePIE
My blog


Wed Jul 02, 2014 3:49 pm
One Eyed Hopeful

Joined: Wed Jun 04, 2014 12:51 pm
Posts: 3
Hi nonverbal,

I know the last update was in July, but hopefully you're still around and working on this.

Really keen to give this a try, but on my Win8.1 machine (.NET 4.5.1 + x86 and x64 versions of MSVCPP 2013 [12.0.21005]) FreePIE crashes after copying over the dlls.

The error details I'm getting are:

Code:
Problem signature:
  Problem Event Name:   CLR20r3
  Problem Signature 01:   FreePIE.exe
  Problem Signature 02:   1.6.512.0
  Problem Signature 03:   541b1cbd
  Problem Signature 04:   mscorlib
  Problem Signature 05:   4.0.30319.34014
  Problem Signature 06:   52e0b679
  Problem Signature 07:   2726
  Problem Signature 08:   12
  Problem Signature 09:   System.NotSupportedException
  OS Version:   6.3.9600.2.0.0.256.48
  Locale ID:   2057
  Additional Information 1:   5861
  Additional Information 2:   5861822e1919d7c014bbb064c64908b2
  Additional Information 3:   84a0
  Additional Information 4:   84a09ea102a12ee665c500221db8c9d6


I have the Leap SDK, but by the looks of it you hook directly into the provided DLL anyway, so don't think that could be it.

I've been trying the Sixense output with some success when getting data from AndroidIMU, so I don't think it's on the Sixense emulation side.

Any help at all in getting this working would be greatly appreciated. I am struggling to find a reasonably priced razer hydra in the UK and this seems like a workaround until things like STEM are properly available.

EDIT: Fixed... when the dll's are extracted on Win8.1 it appears they are immediately blocked and marked as "untrusted". Had to right-click all of the supplied dll's and "unblock" them. FreePIE now starts. Will continue further down this rabbit hole.

EDIT2: ARGGggg... So close... can see the diagnostics data coming through in FreePIE, but seem to have some sort of Windows 8 specific bug whereby if FreePIE loses focus it stops sending out updates.

Cheers
Cpldave.


Thu Nov 06, 2014 4:36 pm
Petrif-Eyed

Joined: Mon Jun 22, 2009 8:36 am
Posts: 2156
Location: Stockholm, Sweden
Quote:
EDIT2: ARGGggg... So close... can see the diagnostics data coming through in FreePIE, but seem to have some sort of Windows 8 specific bug whereby if FreePIE loses focus it stops sending out updates.

Hmm my dev.machine is on 8.1 and it works there, try sixsense emulation without the leap plugin, does that work?

_________________
FreePIE
My blog


Fri Nov 07, 2014 1:41 pm
One Eyed Hopeful

Joined: Wed Jun 04, 2014 12:51 pm
Posts: 3
Yep, I can output to Hydra using the android IMU from my phone and that worked fine. I checked the Leap setting and I definitely have "Allow Background Applications" setting enabled.. Very strange. Any ideas?


Fri Nov 07, 2014 1:54 pm
Petrif-Eyed

Joined: Mon Jun 22, 2009 8:36 am
Posts: 2156
Location: Stockholm, Sweden
Sorry no. and no code to look at. Maybe have to wait until nonverbal is heard of.

edit: Have you looked at the leap forum? Maybe he is active there?

_________________
FreePIE
My blog


Sat Nov 08, 2014 5:17 am
One Eyed Hopeful

Joined: Sun Apr 27, 2014 5:20 pm
Posts: 9
Digging up an old thread here, I know.

Does this plugin still work?
I put the files in the main dir and plugins dir as directed and now FreePIE won't start?
Would it be a version thing?

I'm super keen to get FreePIE working with Leap Motion but I'm just stuck at the moment.
The end goal is to mix Leap data and button data from an other controller (doesn't matter which) since the Leap's "trigger" button (your finger) moves the aim point too much.

Thanks


Wed Aug 03, 2016 2:41 pm
One Eyed Hopeful

Joined: Fri Sep 16, 2016 4:49 pm
Posts: 1
I would also be interested in this for the purposes of VR. I would like to use LeapMotion in Combination with 2 Wii Remotes, to output to the Hydra and then Vive Wand. I was originally going to build in C++ on top of the already existing Leap/SteamVR Driver, but seeing this made me think that a FreePie script may be possible, and easier. I understand this specific project is probably dead, but I would like to look at LEAP support for FreePie, if I was pointed in the right direction, Just Getting the Hand Tracking working while on a HMD (as seen in the visualiser), would be enough, no need for any gestures, as I would be using it with Wii Remotes.


Thu Sep 29, 2016 6:23 pm
One Eyed Hopeful

Joined: Thu Dec 01, 2016 7:36 pm
Posts: 14
I've managed to get everything to work for VR applications, but the Leap data is normalized and clipped inside the "InteractionBox" and I'm told that undesirable. Is there a way to get the raw data from the Leap into FreePIE?


Mon Dec 05, 2016 4:12 pm
Display posts from previous:  Sort by  
   [ 12 posts ] 

Who is online

Users browsing this forum: Google [Bot] and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Powered by phpBB® Forum Software © phpBB Group
Designed by STSoftware.