DIY Vive Controller using Leap and Arduino

Official forum for open source FreePIE discussion and development.
Post Reply
flo2
One Eyed Hopeful
Posts: 14
Joined: Thu Dec 01, 2016 7:36 pm

DIY Vive Controller using Leap and Arduino

Post by flo2 »

UPDATE: Working source code https://github.com/fmaurer/FreePIE-VR-Controls
---
I've managed to pipe the data from an (arduino) orientation sensor, leap, and oculus into freepie and emulate a HTC Vive Controller using the Razer Hydra drivers:

http://imgur.com/a/obgP6

Here's the overview of how system works:

http://i.imgur.com/4uDer8r.png

Currently it's just raw data, but I'm trying to do the math transforms of the various position/orientation data and having a lot of trouble importing numpy into freepie.

I've placed it in the pylib folder of FreePIE install, but get the following error when I do "import numpy as np"

Code: Select all

cannot import multiarray from numpy.core
Any advice in getting the matrix math functions from numpy working in FreePIE?

The hope is I can position the emulated Vive controllers as a function of the HMD position, orientation, and Leap's hand data. From previous experiments, using orientation data in conjunction with the Leap positional data produced very good results: https://www.youtube.com/watch?v=UbbWZwtxqVA
Last edited by flo2 on Wed Dec 14, 2016 12:52 am, edited 1 time in total.

flo2
One Eyed Hopeful
Posts: 14
Joined: Thu Dec 01, 2016 7:36 pm

Re: DIY Vive Controller using Leap and Arduino

Post by flo2 »

I got around needing numpy by finding premade functions around the web using standard python. My linear algebra knowledge is almost nil and the code has a lot of issues, but as a proof of concept it functions.

The position of the right controller is a function of the HMD's position + orientation and the Leap's right hand position. I'm placing the code here if someone figures out the transformations faster than I can. I have a lot of linear algebra to learn.
https://www.youtube.com/watch?v=vbYr_GKlzSk

Code: Select all

*** REMOVED

EDIT: 12/7/2016 - code was fundamentally incorrect in too many ways to risk confusion, and it was an eyesore
Last edited by flo2 on Wed Dec 07, 2016 2:48 am, edited 2 times in total.

flo2
One Eyed Hopeful
Posts: 14
Joined: Thu Dec 01, 2016 7:36 pm

Re: DIY Vive Controller using Leap and Arduino

Post by flo2 »

Improved code, explanation, and request for help:

https://www.reddit.com/r/learnmath/comm ... formation/

flo2
One Eyed Hopeful
Posts: 14
Joined: Thu Dec 01, 2016 7:36 pm

Re: DIY Vive Controller using Leap and Arduino

Post by flo2 »

It works!
https://youtu.be/r-wGzPQHCVg
There were many pitfalls (e.g no documentation on the hydra driver, and I had no idea how it calibrated so everything was inverted for a while) but I hope someone has these numbers. What's the distance between the hydra base station and eye-center when re-centering the hydra base to 0,0,0?

If you look closely at the gif, you'll see the hand be stuck inside a "virtual bounded box." That's the Interaction Box function of the Leap sensor:
https://developer.leapmotion.com/docume ... action-box

Unfortunately, the author of the Leap plugin for FreePIE is no longer active as this was made over 2 years ago. All that would be needed is output the raw hand position data for the full range of motion the Leap captures. I hope this is a simple fix, but I know nothing about writing plugins. If anyone could help here, that would be awesome.

Next up is calibrate the Arduino-IMU orientation, and map the joystick+buttons. Should be in a playable state before the weekend :)

flo2
One Eyed Hopeful
Posts: 14
Joined: Thu Dec 01, 2016 7:36 pm

Re: DIY Vive Controller using Leap and Arduino

Post by flo2 »

Project files for DIY HTC Vive motion controllers, using the Leap Motion mounted to VR headset for positional tracking:
https://github.com/fmaurer/FreePIE-VR-Controls

Image
https://www.youtube.com/watch?v=S1-FHn2ySXA

Over the weekend I finished 3D printing a second DIY motion controller and playing a few Steam VR apps. There were a few glitches that were very specific to developer implementation, but overall very satisfying to finally have access to the Steam VR stuff with motion controllers. Without any industry standards for VR controllers, your mileage will vary in app support. Rec Room didn't recognize the pickup action, but paintball was fun.

While researching and reading through forums I found a lot of people with wii nunchucks trying to do this. With a few edits to the FreePIE script, it should be easily converted to any device/buttons/orientation data they want. The hard part was figuring out the 3D transforms, now it's just a matter of mapping the data.

More info and photos: http://flrnmrr.com/2016/11/30/arduino-v ... -emulation

The biggest limitation is the 2014 FreePIE plugin for the Leap Motion. It outputs the normalized (and limited) hand positions of the interaction box: https://developer.leapmotion.com/docume ... action-box. Developer thread: http://www.mtbs3d.com/phpBB/viewtopic.php?f=139&t=19753

The x,y,z positions of Left and Right hands are given in the InteractionBox's normalized space instead of the raw mm values of the full FOV from the Leap sensor. Until the driver is updated/recreated we're stuck using the limited InteractionBox volume.
If anyone with Windows DLL plugin development experience thinks it's straightforward, this driver could enable a flurry of VR applications for FreePIE community. Thank you again for everyone here helping out and the developer of FreePIE for such an awesome tool.

MeteorFalling2
One Eyed Hopeful
Posts: 41
Joined: Thu Dec 08, 2016 10:23 pm

Re: DIY Vive Controller using Leap and Arduino

Post by MeteorFalling2 »

Did you manage to figure out how to use numpy? I'm trying to smooth out the bumps in the y axis using the Wiimote and it seems like I need it as well.

Btw, i like your project. Motion controls really are special. I wish there was more documentation on FreePIE as it is hard to figure out how to do things.

flo2
One Eyed Hopeful
Posts: 14
Joined: Thu Dec 01, 2016 7:36 pm

Re: DIY Vive Controller using Leap and Arduino

Post by flo2 »

I couldn't get numpy to import. I rewrote the matrix transformations in pure python. It wasn't that bad since all I needed was a matrix multiplication function and the rest was performing the transformations in the correct order. Stepping through it helped me really understand and debugging was easy since only one function touched the data.

Documentation is a bit sparse but the source code is all there so it was straightforward to answer my own questions.

User avatar
cybereality
3D Angel Eyes (Moderator)
Posts: 11394
Joined: Sat Apr 12, 2008 8:18 pm
Which stereoscopic 3D solution do you primarily use?: S-3D desktop monitor

Re: DIY Vive Controller using Leap and Arduino

Post by cybereality »

That's awesome, flo2!

Post Reply

Return to “FreePIE”