Nexus 5 + Durovis Dive = Oculus Rift

[Grygiel]

Hi,

I am working on my project in which I want to use Nexus 5 and Durovis Dive as a VR goggle.
So far so good, but I need your help with head tracking from Android (UDP).

What I done...

1. I installed Splashtop Streamer so I can stream FullHD (30FPS) over WiFi.
Update - I switched WiFi to USB (Tethering) to lower latency (now working perfect)!

a) Connect your smartphone to your computer
b) Disconnect your network interface card
c) Enable USB Tethering to stream internet via USB
d) Done!

2. To split screen I use TriDef 3D (paid), but I heard you can use Geforce Experience Mod for free:
Link: http://forum.xda-developers.com/showthr ... ?t=2505510

I can't use Vireio Perception cos it can support only DX9, and I'm running DX10/11 games.

3. I installed FaceTrackNoIR and was able to move helmet in Project CARS by using FreeTrack 2.0 game protocol.
But the latency was too high ~800ms, for better motion control I decide to using accelerometer/gyroscope sensor from Nexus 5 and FreePIE software.
I installed both PC and android apk and was able to track my mouse on desktop using sample script.

"Android mouse emulation with deadband and continous yaw"

Code: Select all

def update():   
    #Apply deadband filter to avoid drift
    #And continousRotation filter to yaw axis to avoid jumps when passing tracker center
    x = filters.deadband(filters.delta(math.degrees(filters.continousRotation(android[0].yaw))), deadband)
    y = filters.deadband(filters.delta(math.degrees(android[0].pitch)), deadband)
      
    mouse.deltaX = x * multiply
    mouse.deltaY = y * multiply
   
if starting: 
    deadband = 0.01
    multiply = 5
    android[0].update += update

But "Android > Freetrack" doesn't work for me at all.

Code: Select all

def update():
    global yaw
    yaw = android[0].yaw
    global pitch
    pitch = -android[0].roll
    global roll
    roll = android[0].pitch

if starting:
    centerYaw = 0
    centerPitch = 0
    centerRoll = 0
   
    yaw = 0
    pitch = 0
    roll = 0
    android[0].update += update

freeTrack.yaw  = yaw - centerYaw
freeTrack.pitch = pitch - centerPitch
freeTrack.roll = roll - centerRoll

if keyboard.getKeyDown(Key.LeftControl) and keyboard.getPressed(Key.C):
    centerYaw = yaw
    centerPitch = pitch
    centerRoll = roll

YouTube: https://www.youtube.com/watch?v=Q1GW-nIctWI (before I stream over WiFi, now over USB and it is even better!)

HELP ME OUT! I NEED YOUR HELP TO FINISH MY PROJECT!

[CyberVillain]

What game are you trying to send Freetrack data to?

I've actually only tested it with one, and its Arma 2 and that works. I use the exact same pipeline like the real Freetrack program so should work with all Freetrack enabled games that works with the real program

edit: Nice DIY!

[Grygiel]

Thanks mate!
I already modified my Durovis Dive - I will post video on YouTube soon.

I am playing Project CARS, GRID Autosport and Oculus Rift Demos.
Project CARS support helmet tracking using FreeTrack 2.0 game protocol from FaceTrackNoIR (but this is slow webcam tracking).

There are two solutions...

1. Create special filter for UDP data for FaceTrackNoIR
2. Creare working script for FreePIE


UPDATE

Head Tracking is working for ARMA 3 but... I made some changes in code.
ARMA 3 see FreeTrack in controllers options.

Code: Select all

def update():
    global yaw
    yaw = -android[0].pitch
    global pitch
    pitch = -android[0].roll
    global roll
    roll = -android[0].yaw

if starting:
    centerYaw = 0
    centerPitch = 0
    centerRoll = 0
   
    yaw = 0
    pitch = 0
    roll = 0
    android[0].update += update

freeTrack.yaw  = yaw - centerYaw
freeTrack.pitch = pitch - centerPitch
freeTrack.roll = roll - centerRoll

if keyboard.getKeyDown(Key.LeftControl) and keyboard.getPressed(Key.C):
    centerYaw = yaw
    centerPitch = pitch
    centerRoll = roll

BTW AWESOME GAME!

It is working with my VR Goggles and FreePIE Head Tracking over USB (Tethering)!

1. Use USB Tethering
2. Your PC will get a new IP address
3. Run CMD and type IPCONFIG and copy IPv4 address to your Android FreePIE app.
4. Now you can stream without lag and use head tracking (same Android device)

OK I HAVE ONE PROBLEM - CyberVillain please help me!

How I can lower sensitivity using the above script?

[CyberVillain]

Both android and Freetrack uses absolute radian cordinates. So they should map 1:1. Meaning when you rotate the device 90 degrees the head in game should rote 90 degrees

Is that not the case? You can always add a multiplier like

Code: Select all

if starting:
   multiplier = 0.8

and

Code: Select all

def update():
    global yaw
    yaw = -android[0].pitch * multiplier

[Grygiel]

Thank you CyberVillain

The final script (ARMA 3) for Durovis Dive VR goggle:

Code: Select all

def update():
    global yaw
    yaw = -android[0].pitch * multiplier
    global pitch
    pitch = -android[0].roll * multiplier
    global roll
    roll = -android[0].yaw * multiplier

if starting:
    
    multiplier = 0.3
    
    centerYaw = 0
    centerPitch = 0
    centerRoll = 0
   
    yaw = 0
    pitch = 0
    roll = 0
    android[0].update += update

freeTrack.yaw  = yaw - centerYaw
freeTrack.pitch = pitch - centerPitch
freeTrack.roll = roll - centerRoll

if keyboard.getKeyDown(Key.LeftControl) and keyboard.getPressed(Key.C):
    centerYaw = yaw
    centerPitch = pitch
    centerRoll = roll

Next step... we have to find a solution for Project CARS!

[konstantin_lozev]

This looks awesome One question maybe: Do you plan to integrate a gun/wiimote in the VR script? I have experimented some with GlovePIE and the wiimote and I would have done VR+Gun in GlovePIE, but I don't think phone tracking will ever be implemented in GlovePIE. The idea is to control the looking around with the Dive/Android when the gun is on your hip and to control the aiming with the iron sights with the wiimote when the gun is up in iron sights mode. I have implemented a similar "switching" idea in my wiimote-only GlovePIE script and it is very natural. It can even be made "hybrid" in the sense of simply giving more weight (e.g. 90/10) to one of the two controllers depending on whether the gun is on your hip or you are aiming.
What do you think of that?

[Grygiel]

FullHD Streaming @ 30ms - DONE
FreeTrack from Android to PC Game - DONE
Split Screen - DONE

I'm waiting for FreeTrack support from other games (now it is working with ARMA3 & DCS World*)
*not tested

[izzo]

Hi Grygiel,

Are you using anything else then freepie + a game?

In freepie i can see the values change when i put a diagnostic.watch(freeTrack.yaw) in the arma3 script.
However when i launch the game (or any other freetrack game ie elite dangerous) nothing happens, no movement at all.

Regards,
Ruud

[NathankNel]

Hey everybody i have been trying to use my nexus 5 and googles cardboard vr kit to play games but i am having problems with freepie. I am using this code:


def update():
global yaw
global roll
global pitch
yaw = -android[0].googleYaw
roll = -android[0].googleRoll
pitch = android[0].googlePitch

if starting:
yaw = 0
roll = 0
pitch = 0
enabled = False
android[0].update += update

diagnostics.watch(yaw)
diagnostics.watch(roll)
diagnostics.watch(pitch)

deltaYaw = filters.delta(yaw)
deltaPitch = filters.delta(pitch)
deltaRoll = filters.delta(roll)

if math.fabs(deltaYaw) >= math.pi:
deltaYaw = 0

if (enabled):
mouse.deltaX = -deltaYaw*600
mouse.deltaY = -deltaRoll*1000

toggle = keyboard.getPressed(Key.M)

if toggle:
enabled = not enabled


but it only works sometimes and was wondering if you can help me?

[Grygiel]

I don't have any problem with my script.
Before I start the game I always do a test with android_mouse_emulation.py on the Windows desktop.
After that I run my script and the game.

Code: Select all

def update():   
    #Apply deadband filter to avoid drift
    #And continousRotation filter to yaw axis to avoid jumps when passing tracker center
    x = filters.deadband(filters.delta(math.degrees(filters.continousRotation(android[0].yaw))), deadband)
    y = filters.deadband(filters.delta(math.degrees(android[0].pitch)), deadband)
      
    mouse.deltaX = x * multiply
    mouse.deltaY = y * multiply
   
if starting: 
    deadband = 0.01
    multiply = 5
    android[0].update += update

http://www.youtube.com/watch?v=n66OluqGsDI

[NathankNel]

What program and app are you using for the head tracking?

[CyberVillain]

Grygiel: That looks pretty good! Hows the video latency? Do you get motion sickness?

[RoqueNE]

that looks awesome. sadly i can't get it to work with my Galaxy S3.
no movement in-game =/

the mouse emu script works.
Unfortunately, programming is like chinese language for me.
I have absolutely no idea what the code lines mean

but i try to understand. want to use it in Battlefield Vehicles.

[Grygiel]

that looks awesome. sadly i can't get it to work with my Galaxy S3.
no movement in-game =/

the mouse emu script works.
Unfortunately, programming is like chinese language for me.
I have absolutely no idea what the code lines mean

but i try to understand. want to use it in Battlefield Vehicles.

I'm testing a lot of games/software and today I had the same problem with ARMA 3... but I solved it.

1. Uninstall FreePIE, FaceTrackNoIR and all other soft/drivers for example vJoy driver.
2. Use CCleaner - find and fix all Registery errors.
3. Delete ARMA 3 profile: C:\Users\[USER NAME]\AppData\Local\Arma 3
4. Restart your PC
5. Install FreePIE and run ARMA 3

BTW...

I made 3 scripts for FreePIE for my Custom VR google...
- for FreeTrack portocol (for example ARMA 3)
- for Virtual Joystick/Controller (for example Project CARS)
- for Mouse (for example Porttal 2)

I will post everything on my website http://www.firstever.eu (including video and step-by-step guide) in 2 days.

I have something extra for you!
My Nexus 5 screen quality test (HD IPS+ 4.95 inches ~445 ppi pixel density)

[CyberVillain]

My guess is that it should be enough to remove

Current User/Software/Freetrack from the registry

That holds the path to the Freetrack client dll, FacetreckNoIR and FreePIE are not compatible becuase we use different dlls.
FreePIE and Freetrack is however compatible because we use the same method. So regardless if the Game uses the Freetrack dll or the FreePIE dll it will work.

Also, If Arma3 only reads the path once and store it somewhere locally (Why it would do that is beyond me, but sometimes developers do bad decisions) it wont work if it read the path to FacetreckNoIR instead of the updated path to FreePIE dll.

FreePIE will also restore the path to the original when you stop the test. This means that the FreePIE script must be started before you start the game otherwise it might not point to the correct dll

[Bounzze]

Thank you, Grygiel and CyberVillain!

I've been playing around with 3D gaming for a while now, and when I build my GoogleCardboard it was for the sole purpose of building my own hackulus rift. Here's my testfootage of the setup on July 17th:

https://www.youtube.com/watch?v=oAMN0LnsPh0

Because I worked with GlovePIE in the past, I figured it should be possible to get the accelerometer data from my phone to my PC and map it to joystick/gamepad input, but got stuck. I 'discovered' FreePIE a couple days ago, and thanks to this thread I already got a lot further.
I haven't had time to fully dive into freetrack and how to use it in-game, but I'm confident I'll get it working.

Once again: thank you!

[Grygiel]

Full version...

[youtube-hd]www.youtube.com/watch?v=-LqJAgWJ2TY[/youtube-hd]

[Bounzze]

Grygiel,

What streaming software do you use that uses the cable? Is that a feature of Splashtop?

I've been using LimeLight. I've altered the app slightly so it doesn't show the warning message when the latency gets high. The warning hampered the experience more then the actual lag in most cases.

[Grygiel]

Grygiel,

What streaming software do you use that uses the cable? Is that a feature of Splashtop?

I've been using LimeLight. I've altered the app slightly so it doesn't show the warning message when the latency gets high. The warning hampered the experience more then the actual lag in most cases.

I'm using Splashtop Streamer (better than Limelight) - connect it using USB Tethering!

[Bounzze]

I did like the wireless aspect of this setup, but you're right: responses are better when tethered. Although I got it wireless under 50ms when I used my PC as an access-point and had a direct connection between the Nexus and the PC.

Can you share the script for joystick/controller mapping? I did not get that working at all. I don't know if it's because I haven't setup vJoy correctly or if it's the mapping

[genius69]

Thank you CyberVillain

The final script (ARMA 3) for Durovis Dive VR goggle:

Code: Select all

def update():
    global yaw
    yaw = -android[0].pitch * multiplier
    global pitch
    pitch = -android[0].roll * multiplier
    global roll
    roll = -android[0].yaw * multiplier

if starting:
    
    multiplier = 0.3
    
    centerYaw = 0
    centerPitch = 0
    centerRoll = 0
   
    yaw = 0
    pitch = 0
    roll = 0
    android[0].update += update

freeTrack.yaw  = yaw - centerYaw
freeTrack.pitch = pitch - centerPitch
freeTrack.roll = roll - centerRoll

if keyboard.getKeyDown(Key.LeftControl) and keyboard.getPressed(Key.C):
    centerYaw = yaw
    centerPitch = pitch
    centerRoll = roll

Next step... we have to find a solution for Project CARS!

Hello Grygiel wanted to ask if you found a solution for PCars and if you would be so kind as to put the script to FreePie. Thanks in advance

[thoros]

Hey,

I have the problem that Splashtop cannot stream fullscreen output, but TriDef3d requires fullscreen for 3d to work.
I tried with Dishonored and couln´t get it to work.

How did you solve this?

Greetings
thoros

[Vali]

Hi guys, Grygiel,
Can you help me out please, I lost days and days doing this.
How I make the usb conection for Splashtop Streamer - connect it using USB Tethering.Cna you explain me step by step , like for a noob?.
Im using kainy and is laggy.

Thank you so much, and I hope you can reply me.
JAck

[RoqueNE]

Hey thanks for the reply

At time of my last posting i had a Refugio 3D for testing.
Now i have a Durovis Dive which is quite better for that.

and meanwhile i reinstalled my windows. hope everything work better now.
(after 3-4 years my system was a bit 'higgledy-piggledy')

will try your scripts when i'm back at home. my new idea is to use freepie
in games like Star Citizen and Elite Dangerous. freelook would be awesome.

lag of splashstream is also better now. connect my S3 direct per USB Tethering to my PC.
also tried WLAN Direct, but that don't work well until now.
wlan connection over router is unplayable. there are 1-2 seconds delay. i'm dead before i even know what hit me.

@Vali
For me it works when i go to Settings > More... > Tethering & Hotspot > enable USB Tethering.
then i have a new LAN Connection in Windows (SAMSUNG MObile USB Remote NDIS Network Device).
Here you check the assigned IP (right mouse button > status > details).

on android you must then insert the IP in Splashtop (i'm using Splashtop THD). just tap on that 'Plus'.

problem is, i lose internetconnection then from my PC. but other idea would be to use a WLAN Stick
and create a local Hotspot which you connect your Android to. that works with tools like "virtual router",
or similar. but here does not work yet. i'm working on it. the keyword is "Wi-Fi Direct" cause the oldschool 'Ad-Hoc' Wifi don't work with Android.

[Vali]

RoqueNE, thanks for the information.
Ill try that, now I am on stand by, I broke my nexus 7 display..I need to buy another one.
Btw, on your hmd do you see the edges, left-right? what kind of lenses do you have?

[RoqueNE]

RoqueNE, thanks for the information.
Ill try that, now I am on stand by, I broke my nexus 7 display..I need to buy another one.
Btw, on your hmd do you see the edges, left-right? what kind of lenses do you have?

Yes, i see the edges. but they are a little bit blurry.
but after a few minutes i stop notice it at all. It's like looking on a huge display. The immersion is really good.
tried that hang-glide simulator from the appstore and 10 minutes've looked around in amazement and open mouth^^
I use the provided standard lenses. i have bad eyes so i need to pull they nearly on the eye. more like contact lenses

[Vali]

I am using nexus 7 1280x800 and I see just a lil bit the edges from left and right.And I have the eyes like you , closer to the lenses for a better experience.
I have another ideea in mind, I want to use 2 nexus tablets, one next to each other in order to obtain a huge display but not sure if will work good.
What do you say, will work?
Or anybody from here, what do you think? splashtop on both tablets the signal with the same acuracy?

[NathankNel]

The best would probably be if you get a LG G3 or Galaxy Note 4 with 2k screen then also get the right lenses.

[RoqueNE]

Thank you CyberVillain

The final script (ARMA 3) for Durovis Dive VR goggle:

got your script running with Arma 3. but there is a small problem.
the directions are reversed. so left is right and up is down. when i look upward i see my feet.
any idea how i can fix this?


@vali
not sure about that. you would need to split the video signal. so the left eye signal is only showing on the left tablet, ect.
and both tablets must be absolute synchron. i don't even know how to start to get this working or if that is even possible
with splashtop (or other streaming tools).

[AndrewFX]

Hi guys! I'm a new member and I make my custom oculus rift with a color cross vr headset, my lg optimus g and kainy app but I don't have many knowledge of python. I'm trying to use freepie with Assetto Corsa and TrackIR emulation but I don't know how to configure axis. The movement in game doesn't rappresent my movement.

I use this script

Code: Select all

def update():
diagnostics.watch(trackIR.yaw)
global yaw
yaw = math.degrees(android[0].yaw)
global pitch
pitch = math.degrees(-android[0].roll)
global roll
roll = math.degrees(android[0].pitch)

if starting:
centerYaw = 0
centerPitch = 0
centerRoll = 0

yaw = 0
pitch = 0
roll = 0
android[0].update += update

trackIR.yaw = yaw - centerYaw
trackIR.pitch = pitch - centerPitch
trackIR.roll = roll - centerRoll

if keyboard.getKeyDown(Key.LeftControl) and keyboard.getPressed(Key.C):
centerYaw = yaw
centerPitch = pitch
centerRoll = roll

Can you help me?

[Cercata]

got your script running with Arma 3. but there is a small problem.
the directions are reversed. so left is right and up is down. when i look upward i see my feet.
any idea how i can fix this?

I had the same problem using a Galaxy S3.
I guess each phone can have the axis configured diferently, and the S3 has them inverted compared to Nexus 5.

I solved it just removing the "-"s in the update function. This is the script that is working for me in Arma 2:

Code: Select all

def update():
    global yaw
    yaw = android[0].pitch * multiplier
    global pitch
    pitch = android[0].roll * multiplier
    global roll
    roll = android[0].yaw * multiplier

if starting:
    
    multiplier = 0.3
    
    centerYaw = 0
    centerPitch = 0
    centerRoll = 0
   
    yaw = 0
    pitch = 0
    roll = 0
    android[0].update += update

freeTrack.yaw  = yaw - centerYaw
freeTrack.pitch = pitch - centerPitch
freeTrack.roll = roll - centerRoll

if keyboard.getKeyDown(Key.LeftControl) and keyboard.getPressed(Key.C):
    centerYaw = yaw
    centerPitch = pitch
    centerRoll = roll

[Cercata]

I'm starting to regret backing shareVR on kickstarter ... I don't mind having both, but this video is not honest, Trinus Gyre works much better that shown

https://www.youtube.com/watch?v=lrPjtpebB7A

At least we know Trinus Gyre exist, the videos of shareVR don't show UI of server neither client, they can be done With Splashtop + FreePIE

Anyway, I believe the kickstarted is not gonna succed.

[mystd]

I have a Galaxy S3 too. This code "works" but not usable for me.

Code: Select all

def update():   
    #Apply deadband filter to avoid drift
    #And continousRotation filter to yaw axis to avoid jumps when passing tracker center
    x = filters.deadband(filters.delta(math.degrees(filters.continousRotation(android[0].yaw))), deadband)
    y = filters.deadband(filters.delta(math.degrees(android[0].pitch)), deadband)
     
    mouse.deltaX = x * multiply
    mouse.deltaY = y * multiply
   
if starting:
    deadband = 0.01
    multiply = 5
    android[0].update += update

From all other scripts I get no response. I tried every script on http://andersmalmgren.github.io/FreePIE/samples.html and I could find here in the from. Nothing works is it because I'm using the latest freepie?

- I tried tinkering the scripts but I don't know what I'm doing guess I"ll have to play with a mouse until somebody writes a that works with my s3.

-ahh ok I get it now nobody has a working script so far, just freetrack enabled games work T_T Guess I'll wait for freetrack 2.3 then with freeimu support. Does someone know if the 2.3 pre release is available somewhere?