Leap Motion Plugin Development

Official forum for open source FreePIE discussion and development.
Post Reply
nonverbal
One Eyed Hopeful
Posts: 2
Joined: Fri Mar 28, 2014 2:09 pm

Leap Motion Plugin Development

Post by nonverbal »

I started creating a Leap Motion plugin for FreePIE using the latest Skeletal Tracking Beta SDK. The current version, which can be downloaded below tracks the pitch, roll, yaw, x, y, z positions of the right most and left most hands. I also added a single detectable gesture (typing with index finger) just to have some input to get past the calibration stages of Razer hydra emulated applications. It currently doesn't differentiate gestures by hand and needs a lot of work, but my focus right now is on the position and orientation.

Current Version Download Link: https://dl.dropboxusercontent.com/u/413 ... in-001.zip

It currently needs quite a bit of work, but the basic hand tracking is in. I got some poor, but promising results emulating the Hydra in the Tuscany Hydra demo. A big part of my problem is that I don't have a Hydra, so I don't know the actual ranges used/expected. For example, normalizing the x,y,z position in the leap motion results in a 0 to 1 range relative to the left and top of the field of view around the Leap. They refer to it as the interaction box: https://developer.leapmotion.com/docume ... onBox.html

The script below is what I am using by trial and error that gave me the best results emulating the Hydra so far. I included the diagnostic.watch for each of the existing values captured by the plugin for your reference. The orientations are on -/+ pi on each axis. The positions are normalized from 0 to 1 and I converted them to -1000 to 1000 range, in the script below, because some of my initial searches turned up Hydra positional values in this range, but that's only a guess at this point.

If you want to use this very early version of the plugin,
1) Acquire a leap motion or take it out of that drawer
2) Install the Latest Skeletal Tracking Beta https://developer.leapmotion.com/
3) Install .net 4.0 or higher framework if you don't already have it installed
4) Unzip the archive into your Freepie folder so the LeapMotionFreepiePlugin.dll file is in the plugins folder (e.g., C:\Program Files\FreePie\plugins) and the other dlls (Leap.dll, LeapCSharp.dll, LeapCSharp.NET4.0.dll, msvcp120.dll, mscvcr120.dll) are in the root of the Freepie folder (e.g., C:\Program Files\FreePie)
5) If you want to emulate the hydra copy the sixense_fake.dll (located in the root folder of your FreePIE installation) over the sixense.dll in the folder of the application you want use this plugin with.
6) Run a script like the one below, but hopefully better and more compatible with the Razer or any device you want to emulate with the Leap supported by FreePIE.
7) You will need to bind a key to the triggers of the hydra 0 and 1 for the calibration stages (e.g., mouse click). I used my gesture, which is a bit wonky right now, but it's serviceable.

Some caveats, right versus left hand detection only works when two hands are detected. It detects the right most and left most hand. So one hand is detected as both right and left, since it is technically the right most and left most hand when its alone. At a later stage you could restrict left and right hands to either side of the Leap to avoid glitchy behavior. But that introduces a few other issues, like restricted movement (left hand couldn't cross over to the right side of leap physically).

I hope someone with experience with the Leap can help test and sort out the translation to other devices like the Hydra, either by suggesting improvements to the script or my Leap API/plugin development, which I just started learning. The big issues right now are the positional/orientation translation to Hydra, and figuring out how best to emulate calibration with the Leap (i.e., where should your hands be when it says touch your shoulders or point at the base).

Update 6/27 - I had better results using the Sixense test app from the SDK. It also has very nice logging capabilities to see the hydra reported position and orientation for comparison against the leap values. The position mapping worked surprising well, as well as the pitch, yaw, and roll (once reversed). I experimented some with different ranges, and increase the range of the x position increases the currently restricted range of motion left to right (e.g., 1000 to 2000). I don't think the calibration will be much of a problem; the sdk documentation says its mainly for determining the hemisphere of the unit relative to the base. Make index finger typing motions with your right hand to emulate hydra button presses to move past the calibration, if you are using my example script.

Link to Sixense app (it's in the samples/win32 folder) : http://sixense.com/windowssdkdownload
Don't forget to replace the sixense.dll with sixense_fake.dll (discussed above).

Code: Select all

diagnostics.watch(leap.rightpitch)
diagnostics.watch(leap.rightyaw)
diagnostics.watch(leap.rightroll)
diagnostics.watch(leap.rightxpos)
diagnostics.watch(leap.rightypos)
diagnostics.watch(leap.rightzpos)

diagnostics.watch(leap.leftpitch)
diagnostics.watch(leap.leftyaw)
diagnostics.watch(leap.leftroll)
diagnostics.watch(leap.leftxpos)
diagnostics.watch(leap.leftypos)
diagnostics.watch(leap.leftzpos)

diagnostics.watch(leap.typekeytapgesture)


hydra[1].yaw = leap.rightyaw
hydra[1].pitch = leap.rightpitch
hydra[1].roll = -leap.rightroll
hydra[1].x = filters.mapRange(leap.rightxpos, 0, 1, -1000,1000)
hydra[1].y = filters.mapRange(leap.rightypos, 0, 1, -1000,1000)
hydra[1].z = filters.mapRange(leap.rightzpos, 0, 1, -1000,1000)



hydra[0].yaw = leap.leftyaw
hydra[0].pitch = leap.leftpitch
hydra[0].roll = -leap.leftroll
hydra[0].x = filters.mapRange(leap.leftxpos, 0, 1, -1000,1000)
hydra[0].y = filters.mapRange(leap.leftypos, 0, 1, -1000,1000)
hydra[0].z = filters.mapRange(leap.leftzpos, 0, 1, -1000,1000)

hydra[0].trigger = leap.typekeytapgesture
hydra[1].trigger = leap.typekeytapgesture
Updates:
6/27 - Updated example script to use mapRange function to increase legibility.
Last edited by nonverbal on Fri Jun 27, 2014 9:53 am, edited 4 times in total.

User avatar
cybereality
3D Angel Eyes (Moderator)
Posts: 11394
Joined: Sat Apr 12, 2008 8:18 pm
Which stereoscopic 3D solution do you primarily use?: S-3D desktop monitor

Re: Leap Motion Plugin Development

Post by cybereality »

Nice work.

CyberVillain
Petrif-Eyed
Posts: 2166
Joined: Mon Jun 22, 2009 8:36 am
Location: Stockholm, Sweden

Re: Leap Motion Plugin Development

Post by CyberVillain »

Nice work! I hope you get some help from Leap owners! This could be a built in plugin once its tested and complete

nonverbal
One Eyed Hopeful
Posts: 2
Joined: Fri Mar 28, 2014 2:09 pm

Re: Leap Motion Plugin Development

Post by nonverbal »

UPDATE 7/2 - I added much improved gesture support.

Version 002 linked below, now distinguishes between left and right hand gestures. I also added three gestures that are detected 1) Hand Swipe (regardless of direction at the moment), 2) Finger Key Type (regardless of finger, downward motion like typing an invisible keyboard), and 3) Screen Tap (forward finger motion like pushing a button on screen).

I also exposed the properties of this gestures to better fine tune them from the FreePIE interface for different applications. These control the threshold for what is considered a gesture. The current properties and their default values are:

SwipeMinLength - 150 mm
SwipeMinVelocity - 750 mm/s
KeytapVelocity - 50 mm/s (minimum downward velocity)
KeytapMinDistance - 3 mm
KeyTapHistory - .1 sec (the time window for evaluating movement)
ScreentapVelocity - 50 mm/s (minimum forward velocity)
ScreentapMinDistance - 5 mm
ScreenTapHistory - .1 sec (the time window for evaluating movement)


Link to version 002: https://dl.dropboxusercontent.com/u/413 ... in-002.zip

Right now the plugin relies on the DoBeforeNextExecute method where it grabs the current frame from the Leap and reports all the gesture and hand data. Is this the preferred way to write plugins, or should it be more event/listener driven from the leap itself when it registers a valid gesture? At the moment, I am just checking the frame during each poll and if there is a detected gesture, determine the hand and report its status via a property in the global plugin object. What is the update rate of FreePie? I think under optimal lighting/computer conditions the Leap motion is capable of up to 200 fps.

If anyone has had any opportunity to test/find bugs problems or wants features I don't currently have, let me know. I have been working on the tracking individual fingers and a tool (e.g., pencil) to further improve gesture recognition and accuracy (i.e., limiting recognition to specific fingers)--right now it is watching all fingers for a valid gesture occurrence.

Here is an updated testing script to emulate the Razer Hydra that includes diagnostic.watch for all the current values provided with this plugin.

Code: Select all

diagnostics.watch(leap.rightpitch)
diagnostics.watch(leap.rightyaw)
diagnostics.watch(leap.rightroll)
diagnostics.watch(leap.rightxpos)
diagnostics.watch(leap.rightypos)
diagnostics.watch(leap.rightzpos)

diagnostics.watch(leap.leftpitch)
diagnostics.watch(leap.leftyaw)
diagnostics.watch(leap.leftroll)
diagnostics.watch(leap.leftxpos)
diagnostics.watch(leap.leftypos)
diagnostics.watch(leap.leftzpos)

diagnostics.watch(leap.righttypekeytapGesture)
diagnostics.watch(leap.lefttypekeytapGesture)
diagnostics.watch(leap.righttypescreentapGesture)
diagnostics.watch(leap.lefttypescreentapGesture)
diagnostics.watch(leap.rightswipeGesture)
diagnostics.watch(leap.leftswipeGesture)


hydra[1].yaw = leap.rightyaw
hydra[1].pitch = leap.rightpitch
hydra[1].roll = -leap.rightroll
hydra[1].x = filters.mapRange(leap.rightxpos, 0, 1, -1000,1000)
hydra[1].y = filters.mapRange(leap.rightypos, 0, 1, -1000,1000)
hydra[1].z = filters.mapRange(leap.rightzpos, 0, 1, -1000,1000)

hydra[0].yaw = leap.leftyaw
hydra[0].pitch = leap.leftpitch
hydra[0].roll = -leap.leftroll
hydra[0].x = filters.mapRange(leap.leftxpos, 0, 1, -1000,1000)
hydra[0].y = filters.mapRange(leap.leftypos, 0, 1, -1000,1000)
hydra[0].z = filters.mapRange(leap.leftzpos, 0, 1, -1000,1000)

hydra[0].trigger = leap.lefttypekeytapGesture
hydra[1].trigger = leap.righttypekeytapGesture



CyberVillain
Petrif-Eyed
Posts: 2166
Joined: Mon Jun 22, 2009 8:36 am
Location: Stockholm, Sweden

Re: Leap Motion Plugin Development

Post by CyberVillain »

The default frequency of freepie is 60hz (OS default system clock)
It can be overridden for example

Code: Select all

if starting:
	system.setThreadTiming(TimingTypes.HighresSystemTimer)
	system.threadExecutionInterval = 1
Would set FreePIE to run at 1000hz

I like the event driven model, that the plugin has a update event so that your python script only does something if new data has arrived. Note that this doesn't neccery mean that your plugin needs to use the Leap API's event model.

Also python isnt thread safe so if you add a update event to the plugin make sure you trigger it from the DoBeforeNextExecute method, this will ensure thread safty

cpldave
One Eyed Hopeful
Posts: 3
Joined: Wed Jun 04, 2014 12:51 pm

Re: Leap Motion Plugin Development

Post by cpldave »

Hi nonverbal,

I know the last update was in July, but hopefully you're still around and working on this.

Really keen to give this a try, but on my Win8.1 machine (.NET 4.5.1 + x86 and x64 versions of MSVCPP 2013 [12.0.21005]) FreePIE crashes after copying over the dlls.

The error details I'm getting are:

Code: Select all

Problem signature:
  Problem Event Name:	CLR20r3
  Problem Signature 01:	FreePIE.exe
  Problem Signature 02:	1.6.512.0
  Problem Signature 03:	541b1cbd
  Problem Signature 04:	mscorlib
  Problem Signature 05:	4.0.30319.34014
  Problem Signature 06:	52e0b679
  Problem Signature 07:	2726
  Problem Signature 08:	12
  Problem Signature 09:	System.NotSupportedException
  OS Version:	6.3.9600.2.0.0.256.48
  Locale ID:	2057
  Additional Information 1:	5861
  Additional Information 2:	5861822e1919d7c014bbb064c64908b2
  Additional Information 3:	84a0
  Additional Information 4:	84a09ea102a12ee665c500221db8c9d6
I have the Leap SDK, but by the looks of it you hook directly into the provided DLL anyway, so don't think that could be it.

I've been trying the Sixense output with some success when getting data from AndroidIMU, so I don't think it's on the Sixense emulation side.

Any help at all in getting this working would be greatly appreciated. I am struggling to find a reasonably priced razer hydra in the UK and this seems like a workaround until things like STEM are properly available.

EDIT: Fixed... when the dll's are extracted on Win8.1 it appears they are immediately blocked and marked as "untrusted". Had to right-click all of the supplied dll's and "unblock" them. FreePIE now starts. Will continue further down this rabbit hole.

EDIT2: ARGGggg... So close... can see the diagnostics data coming through in FreePIE, but seem to have some sort of Windows 8 specific bug whereby if FreePIE loses focus it stops sending out updates.

Cheers
Cpldave.

CyberVillain
Petrif-Eyed
Posts: 2166
Joined: Mon Jun 22, 2009 8:36 am
Location: Stockholm, Sweden

Re: Leap Motion Plugin Development

Post by CyberVillain »

EDIT2: ARGGggg... So close... can see the diagnostics data coming through in FreePIE, but seem to have some sort of Windows 8 specific bug whereby if FreePIE loses focus it stops sending out updates.
Hmm my dev.machine is on 8.1 and it works there, try sixsense emulation without the leap plugin, does that work?

cpldave
One Eyed Hopeful
Posts: 3
Joined: Wed Jun 04, 2014 12:51 pm

Re: Leap Motion Plugin Development

Post by cpldave »

Yep, I can output to Hydra using the android IMU from my phone and that worked fine. I checked the Leap setting and I definitely have "Allow Background Applications" setting enabled.. Very strange. Any ideas?

CyberVillain
Petrif-Eyed
Posts: 2166
Joined: Mon Jun 22, 2009 8:36 am
Location: Stockholm, Sweden

Re: Leap Motion Plugin Development

Post by CyberVillain »

Sorry no. and no code to look at. Maybe have to wait until nonverbal is heard of.

edit: Have you looked at the leap forum? Maybe he is active there?

jimrooney
One Eyed Hopeful
Posts: 9
Joined: Sun Apr 27, 2014 5:20 pm

Re: Leap Motion Plugin Development

Post by jimrooney »

Digging up an old thread here, I know.

Does this plugin still work?
I put the files in the main dir and plugins dir as directed and now FreePIE won't start?
Would it be a version thing?

I'm super keen to get FreePIE working with Leap Motion but I'm just stuck at the moment.
The end goal is to mix Leap data and button data from an other controller (doesn't matter which) since the Leap's "trigger" button (your finger) moves the aim point too much.

Thanks

alec100_94
One Eyed Hopeful
Posts: 1
Joined: Fri Sep 16, 2016 4:49 pm

Re: Leap Motion Plugin Development

Post by alec100_94 »

I would also be interested in this for the purposes of VR. I would like to use LeapMotion in Combination with 2 Wii Remotes, to output to the Hydra and then Vive Wand. I was originally going to build in C++ on top of the already existing Leap/SteamVR Driver, but seeing this made me think that a FreePie script may be possible, and easier. I understand this specific project is probably dead, but I would like to look at LEAP support for FreePie, if I was pointed in the right direction, Just Getting the Hand Tracking working while on a HMD (as seen in the visualiser), would be enough, no need for any gestures, as I would be using it with Wii Remotes.

flo2
One Eyed Hopeful
Posts: 14
Joined: Thu Dec 01, 2016 7:36 pm

Re: Leap Motion Plugin Development

Post by flo2 »

I've managed to get everything to work for VR applications, but the Leap data is normalized and clipped inside the "InteractionBox" and I'm told that undesirable. Is there a way to get the raw data from the Leap into FreePIE?

uneasy
One Eyed Hopeful
Posts: 8
Joined: Thu Sep 07, 2017 2:58 am

Re: Leap Motion Plugin Development

Post by uneasy »

Plugin is not available for download anymore - Dropbox says file not found. Can someone reupload it?

ajayajayaj
One Eyed Hopeful
Posts: 3
Joined: Mon Jan 16, 2017 3:11 pm

Re: Leap Motion Plugin Development

Post by ajayajayaj »

I have reuploaded it in my Github repo. It's really sad dropbox broke so many links.

https://github.com/ajayyy/Wiimote-FreeP ... Plugin.zip

Post Reply

Return to “FreePIE”