iPhone support

[brantlew]

Apple devices including the iPhone, iPad, and iPod Touch have inertial sensors that can accessed with FreePIE and used as orientation trackers. With magnetometer-assisted, near driftless readings and up to 100Hz data rate they are actually quite good trackers and make for a simple and inexpensive head tracking solution. To access the data you will need to purchase the Sensor Data app from the App Store.

http://itunes.apple.com/us/app/sensor-d ... 19802?mt=8

This app transmits sensor data from the phone to FreePIE using UDP transmissions so your phone will need to be connected by WiFi. Then simply go to this screen and configure Sensor Data for Unicast and set the IP address to the address of your PC.



Then launch FreePIE, execute an iPhone script, and hit Start Stream on the phone.



The script variables, "iPhone", "iPad", or "iPod" are all available and are interchangeable. Here is an example script to enable mouse emulation with the iPhone.

Code: Select all

if  starting:
   iPhone.continuousYawMode = True

yaw = math.degrees(iPhone.yaw)
pitch = math.degrees(iPhone.pitch)

mouse.deltaX = filters.delta(yaw * 10)
mouse.deltaY = filters.delta(pitch * 10)

NOTE: Unfortunately, there is a glitch in the iPhone that diminishes its effectiveness as a head tracker. A couple of times a minute it will enter into some sort of wireless hardware contention state and the data rate will drop to about 10Hz for a few seconds. No packets are lost but the latency is clearly noticeable when using the device as a headtracker. The iPod Touch does not seem to have this issue and I have not tried the iPad.

(Edit: Some users have reported that they don't experience any feed lag with the iPhone. Result may vary. It is currently unclear to me if the problem is related to specific hardware version, iOS version, or configuration.)

[youtube-hd]https://www.youtube.com/watch?v=gQS2Tdo9ZVM[/youtube-hd]

[space123321]

Thanks Brantlew!

As I mentioned ealier, I was able to get the simple mouse script running, however FreePie freezes up as soon as I attempt to run this Iphone script? Not sure what the issue is?

[brantlew]

Hmmm. I wonder if there could be a firewall issue? It's possible FreePIE is unable to listen successfully on that UDP port. Still...I'm not sure how that would manifest itself as a GUI freeze?


Edit: Is the app "frozen" or does it seem like it's just not getting data? If the latter, then some type of networking/firewall issue could be the problem and UDP packets are just not finding their way into FreePIE.

[space123321]

I can't click on anything once it is running... can't stop script, change from console to watch, etc... however the application still shows as running in task manager (does not show as not responding)?

[brantlew]

Hmmm. Out of curiosity, I pulled a fresh binary from GitHub and ran that script. It works for me and accepted iPhone input - however, it does pop up a message when the script first starts to force me to punch a hole in my firewall. Any chance something like that could be happening? Otherwise - I have no idea. The iPhone plugin is one of the simplest ones without external dependencies, so I don't know what could be happening. Maybe running FreePIE as administrator could help???

[space123321]

I do not recall a firewall message coming up, however I did a fresh install just to try again... on the fresh install I didn;t receive a firewall message? Looks like I will play again tomorrow...?

EDIT - a simple reset of my machine and no more freezing? Was able to get the script working first attempt and the iphone synched! On another postive note - I do not experience the same glitching that you experience on your video?

[CyberVillain]

Cool that you got it working and without lag!

Tonight if I get time i will add a plugin for Android, a plus is that the app is free for android

https://play.google.com/store/apps/deta ... LndpbXUiXQ

You'll need an highend Android with Magnometer for acceptable Yaw drift

[brantlew]

I do not recall a firewall message coming up, however I did a fresh install just to try again... on the fresh install I didn;t receive a firewall message? Looks like I will play again tomorrow...?

EDIT - a simple reset of my machine and no more freezing? Was able to get the script working first attempt and the iphone synched! On another postive note - I do not experience the same glitching that you experience on your video?

Good news. Might I ask which iPhone and iOS version you are using?

[space123321]

Iphone4 and 4.3.3

[mahler]

Cool that you got it working and without lag!

Tonight if I get time i will add a plugin for Android, a plus is that the app is free for android

Wireless IMU

You'll need an highend Android with Magnometer for acceptable Yaw drift

Played around a bit with my Galaxy Nexus....

The Wireless IMU app sends back the data in (x, y, z) values

Example:

Code: Select all

Accelerometer (m/s^2) [x, y, x]: 4,297, 7,882, -2,919 
Gyroscope (rad/s) [x, y, x]: 0,239, -0,114, 0,233 
Magnetometer (micro-Tesla uT) [x, y, x]: -16,266, -34,304, 23,004

Unfortunately I don't know how to best calculate Pitch-, Roll- and Yaw-samples from these values.... is 3d-pytharos enough?
(I'll be looking over the video of Sensor Fusion on Android Devices: A Revolution in Motion Processing again, after I get some sleep)
brantlew => Did I see correctly that you used the Angle Change method in your plugin?

I also found this app: SquidyClient BETA
Which does the same, but the output-String is not so clean and the data from different sensors comes in separate UDP packets, but it does seem to be more precise.

Example:

Code: Select all

ACCELEROMETER: 0.49876976, 0.49653563, 0.55741125
ORIENTATION3D [azimuth, pitch, roll]: 0.6157635, 0.50827295, 0.49071857
MAGNETIC_FIELD: 0.0013533693, -0.0011109433, -0.005039943

Note: There is no documentation on the units or values of the accelerometer and magnetic_field in this app

Anybody know how to calculate a Yaw-sample from these values....?
I Googled a bit and got some references to Tait–Bryan angles and I gave up

[brantlew]

It's on my todo list, but I haven't gotten around to looking at the Android sensor package at all. I had assumed that they would have a sensor fusion API just like iOS but maybe they don't. Maybe you can't put together a generic sensor fusion package that covers a wide range of hardware. Let me know what you find. (CyBerVillain is also working on an Android fusion plugin so he may have some helpful references)

Did I see correctly that you used the Angle Change method

Not sure what you mean, but the code you are referring to allows the yaw angle to be read continuously (360, 720, 1080, ...) instead of flipping from positive to negative on every revolution.

[mahler]

It's on my todo list, but I haven't gotten around to looking at the Android sensor package at all. I had assumed that they would have a sensor fusion API just like iOS but maybe they don't. Maybe you can't put together a generic sensor fusion package that covers a wide range of hardware. Let me know what you find. (CyBerVillain is also working on an Android fusion plugin so he may have some helpful references)

Well Android does have a way to retrieve the rotation matrix:
http://developer.android.com/reference/android/hardware/SensorManager.html#getRotationMatrix(float[], float[], float[], float[])
I don't yet have the experience or knowledge to fully know how to use this yet (and I get confused easily: Orientation vs Orientation)

I found this blog-post where somebody created an Android Sensor Fusion Tutorial with example code.
http://www.thousand-thoughts.com/2012/0 ... -tutorial/

But the two Android Apps mentioned above don't send anything more than I posted.

Did I see correctly that you used the Angle Change method

Not sure what you mean, but the code you are referring to allows the yaw angle to be read continuously (360, 720, 1080, ...) instead of flipping from positive to negative on every revolution.

What I meant was the first solution provided in the video I posted.
For mouse pointers, just using the "rotation-changes" and mapping it to pixel-movement would be the quick and easy method.
Then I saw something similar in your code where you calculated the delta of the sample. That's why I asked.

But I think I get it.
Why did you only do that for Yaw and not for the others?

So basically, the PitchSample, RollSample and YawSample you retrieve are the rotation angles? But compared to what? The previous measurement or to calibrated values?
And am I correct to understand that the Azimuth-value is the same as a Yaw angle, except that it is always calibrated North?

Edit: Ah found it:

Sensor.TYPE_ORIENTATION:

All values are angles in degrees.
values[0]: Azimuth, angle between the magnetic north direction and the y-axis, around the z-axis (0 to 359). 0=North, 90=East, 180=South, 270=West

values[1]: Pitch, rotation around x-axis (-180 to 180), with positive values when the z-axis moves toward the y-axis.

values[2]: Roll, rotation around y-axis (-90 to 90), with positive values when the x-axis moves toward the z-axis.

Note: This definition is different from yaw, pitch and roll used in aviation where the X axis is along the long side of the plane (tail to nose).

Note: This sensor type exists for legacy reasons, please use getRotationMatrix() in conjunction with remapCoordinateSystem() and getOrientation() to compute these values instead.

Important note: For historical reasons the roll angle is positive in the clockwise direction (mathematically speaking, it should be positive in the counter-clockwise direction).

[CyberVillain]

@mahler I havent had time to start with this are you sure x, y z isnt in fact Yaw, Pitch, roll?

If it only send 3 values it cant be the raw data, you need all 3 sensors to fuse correctly.

Anyway, here is a fusion lib written in C#

http://www.x-io.co.uk/node/10

[mahler]

@mahler I havent had time to start with this are you sure x, y z isnt in fact Yaw, Pitch, roll?
If it only send 3 values it cant be the raw data, you need all 3 sensors to fuse correctly.

The Wireless IMU documentation on the Google Play website is pretty specific. They just return the SensorEvent values.

But as you can see in the Android SensorManager-documentation, it directly supports orientation, so it should be easy to create an Android app to send this information.
It's just that Wireless IMU doesn't return this data. SquidlyClient BETA does, so I might end up using that. However then I'm still stuck on understanding how to get from Azimuth to Yaw (for the FreePIE plugin)

Anyway, here is a fusion lib written in C#
http://www.x-io.co.uk/node/10

Thanks. I'll have a look at it.
I'm now mostly questioning how the sensor-data in Android is fused and whether open-source solutions (or publicly available algorithms) might have better performances based on the raw data.

P.S. Maybe we should move these posts to an Android support thread.

[CyberVillain]

P.S. Maybe we should move these posts to an Android support thread.

Done!

[Yoder808]

I experience crashes every few minutes using the iPhone script, but I don't recall anyone else having any issues in other threads where I've seen people using the iPhone with FreePie. I'm using FreePIE v0.3 Build 121, with the script posted here. Is this a known issue, or am I just unlucky? Also, lowering the refresh rate to <80hz and/or turning off Bluetooth seemed to fix my 'jitter' problem.

[brantlew]

I experience crashes every few minutes using the iPhone script, but I don't recall anyone else having any issues in other threads where I've seen people using the iPhone with FreePie. I'm using FreePIE v0.3 Build 121, with the script posted here. Is this a known issue, or am I just unlucky? Also, lowering the refresh rate to <80hz and/or turning off Bluetooth seemed to fix my 'jitter' problem.

Do you know whether it's related to the phone script or just FreePIE in general? I would try running a different script against the same game you are using when you see crashes. I would also try running the phone script for a few minutes without a game running.

I've had a lot of strange behavior and crashes when using FreePIE during gaming. The FreePIE gui uses the WPF interface which is tied to DirectX and the graphics card and I suspect it contends with the gaming engine for resources. I've noticed the following strange behavior while gaming with FreePIE. These issues are not repeatable, but they only seem to happen while a game is running.

- FreePIE will stop polling when I minimize the gui and then up start again when I restore the window
- The FreePIE gui will "capture" the mouse so that as I move the mouse across the FreePIE window the mouse slows down and then speeds up again when it gets to the other side
- FreePIE crashes

All in all, I've stopped using FreePIE for the time being and have been using GlovePIE these days. I used to have a console version of FreePIE running but it doesn't compile anymore. I would like to recreate the console interface so that I can side-step any side effects of the GUI because currently it seems unstable to me. And once a script is completed there isn't any compelling reason to run it from the editor any longer.

[CyberVillain]

I've had other reports on crashes, the sad part is that I dont have these problems my self so its very hard to debug, also try to minimize FreePIE before starting game.

[Yoder808]

Thanks for the tips guys. I actually don't think I minimized FreePie, so I'll try that. I did try running it as administrator, compatibility mode, etc.

I'm not sure if the problem is in the script or the application. If I get crashed after trying minimizing it, I'll try other scripts with the same game.

[Yoder808]

I tried minimizing, etc, but no luck. When I debug in Visual Studio, I get:

A buffer overrun has occurred in FreePIE.exe which has corrupted the program's internal state. Press Break to debug the program or Continue to terminate the program.

For more details please see Help topic 'How to debug Buffer Overrun Issues'.

Here's the VS disassembly, but I don't know how much help that would be.

Is there a new binary available? If not, how difficult is it to compile from sauce?

[CyberVillain]

As I suspected, unmanaged code, and the call stacks hints that it's in msvcr and the only thing uses that lib is the Lua lib (iPhone plugin is 100% managed code). Can you please supply us with the script causes this, if we have any luck maybe you have written something wrong in the script.

For example if you listen to the Update event outside of the starting block

Code: Select all

if(starting) then
end

Compiling from source is really easy: If you have git installed do "git clone https://github.com/AndersMalmgren/FreePIE.git" to get a read only checkout of the code, if you dont have Git installed install it or download the source from the download page on Github. Open the SLN and compile, you use VSE it will give you some warnings but should compile according to brantlew

[brantlew]

I recreated a console version of FreePIE a couple of days ago. Running the console version eliminated the crashes that I was experiencing about once every 5 minutes. (although there are still some strange side-effects). I'm going to upload the console project this weekend and maybe that will help others with crashing.

[CyberVillain]

I had a report were

Code: Select all

diagnostics:watch

caused FreePIE to crash, I havent been able to reproduce this on my machine(s). Only use diagnostics:debug and watch for diagnostics and debugging, watch should never be used while gaming because it updates the FreePIE gui at a high frequency while it shouldnt crash it can slow down things

[FK87]

Hello
I have been trying to get this to work with my ipod touch, but i only get a error message, saying some thing like "the input string has an invalid format".

This is probably because i have not selected location data as the data capture in the sensor data app.
But it is not possible to select them, they are gray.
Is it possible to get it working with any of the other data captures?
Or am i doing something else wrong?

The ipod is a 4th gen touch.

[brantlew]

Try to deselect some of the data fields on the Config screen that you don't need like latitude, longitude, etc..

[FK87]

Okay, i looked at the plugin file at github, and figured out that "Euler angles" is the only one that needs to be selected in the data capture config, to get Timestamp,Roll,Pitch,Yaw.

Here is the data being sendt from the iPod,
Timestamp,Roll,Pitch,Yaw{0D}{0A}
1.001757,0.101939,-0.036664,0.001826{0D}{0A}

I'am using the script from first post.
But i still get the same error when i select "Start Stream" on the iPod.

Any suggestions?

[tania11]

As tablets support and run full programs, you can now transfer desirable files from your desktop to the tablet. Unlike earlier version tablets, in Windows tablets you can now play multiple HD videos which mean it has better multimedia capabilities.
Apple iPad Mini Skins

[Matoz]

Hi guys,

I am struggling to get FreePIE working with my iPhone. Have downloaded the App Sensor Data from the App Store and downloaded and installed last version of FreePIE. But I really have no clue what a script for an iPhone should look like. The Code from the first post does not work and gives me the following error message:

unexpected token 'then'

Having a look at the samples did not help either. What is the language/syntax for scripts? Python? As I have never used those.

Regards Matoz

[CyberRascal]

Hey

The script language has been changed since that sample was written due to instabilities in the lua engine that we used before.

Code: Select all

if  starting:
   iPhone.continuousYawMode = true

yaw = math.deg(iPhone.Yaw)
pitch = math.deg(iPhone.Pitch)

mouse.deltaX = filters.delta(yaw * 10)
mouse.deltaY = filters.delta(pitch * 10)

That should be the equivalent script in python. Let us know if it works or not

[Matoz]

This script gives me the following error:

name 'true' is not defined

Does FreePIE have any dependencies I have to install also? Sounds to me like something is missing in general.

[CyberVillain]

This script gives me the following error:

name 'true' is not defined

Does FreePIE have any dependencies I have to install also? Sounds to me like something is missing in general.

math.deg > math.degrees
true > True

ContinuousYawMode > ContinuousYawMode (This is a bug, all properties should have been changed to lower case)

Try this one

Code: Select all

if  starting:
   iPhone.ContinuousYawMode = True

yaw = math.degrees(iPhone.yaw)
pitch = math.degrees(iPhone.pitch)

mouse.deltaX = filters.delta(yaw * 10)
mouse.deltaY = filters.delta(pitch * 10)

[Matoz]

OK, now I can start the script. Now when I start the streaming in the app on my iPhone the mouse moves to the left top of the screen. Moving the iPhone does nothing to the mouse. Something is still wrong here. Can someone who has this running give some more tips what is going on. Which parameters should I stream in the iPhone app? What should I use as update rate?

Also can I add something like console output into the script, so I can get to know what actually is going on and what is received from the device?

Regards Matoz

[CyberVillain]

OK, now I can start the script. Now when I start the streaming in the app on my iPhone the mouse moves to the left top of the screen. Moving the iPhone does nothing to the mouse. Something is still wrong here. Can someone who has this running give some more tips what is going on. Which parameters should I stream in the iPhone app? What should I use as update rate?

Also can I add something like console output into the script, so I can get to know what actually is going on and what is received from the device?

Regards Matoz

Sorry dont own a iPhone, maybe Brantlew will check in here soon.
You can write to the console or watch window

for Watch
diagnostics.watch(iPhone.yaw)

for console
diagnostics.debug(iPhone.yaw)

[brantlew]

I just tested the Python script. There was a small syntax error, but otherwise it ran smoothly for me. It sounds like the data feed is returning a constant -1 for the orientation fields. Make sure that you are using an iPhone 4. Older iPhones do not have the hardware to support all of these fields and it's possible they just return non-values for those fields.


Corrected script:

Code: Select all

if  starting:
   iPhone.continuousYawMode = True

yaw = math.degrees(iPhone.yaw)
pitch = math.degrees(iPhone.pitch)

mouse.deltaX = filters.delta(yaw * 10)
mouse.deltaY = filters.delta(pitch * 10)

[CyberVillain]

Camelcase

continuousYawMode is only in the latest github source though, for the latest binary its pascal case

ContinuousYawMode

[Matoz]

Got it working now. Had to devide by 1.000.000. Seems either the app or iOS 6 has changed something in the data format.

Here is my working code:

Code: Select all

if  starting:
   iPhone.ContinuousYawMode = True
   
yawDegrees = math.degrees(iPhone.yaw) / 1000000
pitchDegrees = math.degrees(iPhone.pitch) / 1000000

yawDelta = filters.delta(yawDegrees * 10)
pitchDelta = filters.delta(pitchDegrees * 10)

mouse.deltaX = yawDelta
mouse.deltaY = pitchDelta

[pAnatiC]

Hey Guys!

Actually i found this thread while surfing in the project Cars & Oculus Forum.
Could someone send me a script for freepie that will just track, if my head goes left or right and then press a button?

I really would like to use this + iphone to get some kind of headtracking for Project Cars. Actually it isnt implemented yet, so you can just klick on a button to show left or right.
If it is possible to this with headtracking it would be great.

[CyberVillain]

I tried to make a scrpt for you but it looks like there is a bug in the Keyboard plugin. Only the modifiers can be pressed and kept down, all the others wont work like that. Do you have any input Brantlew?

[pAnatiC]

I tried to make a scrpt for you but it looks like there is a bug in the Keyboard plugin. Only the modifiers can be pressed and kept down, all the others wont work like that. Do you have any input Brantlew?

The modifiers are strg & alt, right? i guess that wouldnt be a big problem. should be possible to bind in game

[CyberVillain]

I tried to make a scrpt for you but it looks like there is a bug in the Keyboard plugin. Only the modifiers can be pressed and kept down, all the others wont work like that. Do you have any input Brantlew?

The modifiers are strg & alt, right? i guess that wouldnt be a big problem. should be possible to bind in game

Strange,if i use FreePIE to set left shift down and type something it will type everything in capital letters, so it works for typing. But if i assign a function in a game to left shift it does not work so it seems our keyboard emulation isnt 100%