Android

[mahler]

I can code, but didnt work with smartphone so far.
But looking into your code I saw you use the Sensor.TYPE_ORIENTATION

Playing around a bit, I realized that this sensor does this strange 180° jump when device is close to vertical, while Sensor.TYPE_ROTATION_VECTOR does not change that way.

Seems to be a different interpretation of the sensor data. Tried playing around with that?

This is a very interesting comment. I totally missed that.
Ever since the Sensor Fusion on Android video, I was really hoping to avoid issues like those of Euler angles (see 38:26)

In API 8 and above, there are "virtual" sensors which are generated by combining the inputs of all available sensors and appropriate filters. The "TYPE_ORIENTATION" sensor gives you the allover orientation of your device, but this interface is deprecated due to failure states at certain orientations. The new sensor is TYPE_ROTATION_VECTOR (API 9 and above) which gives your device orientation as a quaternion. This is really the best sensor to use, but the math behind it is a little heavy.

Failing that, what you do is call SensorManager.getRotationMatrix(), passing the latest gravity and magnetometer data. This will return a rotation matrix which could be used to convert a vector from device coordinates to world coordinates or vice-versa (just transpose the matrix to invert it).

The getOrientation() function can give you heading, pitch, and roll, but these have the same failure states as the TYPE_ORIENTATION sensor.

Source: http://stackoverflow.com/a/11068878

More documentation about the TYPE_ROTATION_VECTOR sensor:
http://developer.android.com/guide/topi ... ion-rotate
And here: http://developer.android.com/reference/ ... Event.html

I'm starting to think that we should have been using TYPE_ROTATION_VECTOR all along.
But the phone needs the hardware to support it (just accelerometer won't work) otherwise you might get a NULL-value according to this guy

Good catch Parity

[CyberVillain]

Good catch! I'm having the hangover from hell today so I wont be able to test this until later.

[Parity]

For what I am doing at the moment, the change in orientation like described at 37:00 through the Sensor Fusion Talk would work as well I think.
I am actually working with changes of the Euler Angles right now, not absolute values. But I run into problems in that "bad" vertical orientation, that is needed when using the device as HMD-Panel.

I presented my build here, you saw it working in the video too.
http://www.mtbs3d.com/phpBB/viewtopic.php?f=26&t=15607

[CyberVillain]

Ok tried the vector stuff, I cant see its much of an improvemt, the raw values + mahony fusion in FreePIE gives a better result. But it drifts.

Code: Select all

	public void onSensorChanged(SensorEvent sensorEvent) {
	    switch (sensorEvent.sensor.getType()) {  
	        case Sensor.TYPE_ACCELEROMETER:
	            acc = sensorEvent.values.clone();
	            break;
	        case Sensor.TYPE_MAGNETIC_FIELD:
	            mag = sensorEvent.values.clone();
	            break;
	            
	        case Sensor.TYPE_GYROSCOPE:
	            gyr = sensorEvent.values.clone();
	            break;
	        case Sensor.TYPE_ROTATION_VECTOR:
	        	orientation = sensorEvent.values.clone();
	        	break;
	    }	
	    
	    if(sendOrientation && orientation != null) {
            SensorManager.getRotationMatrixFromVector(rotationMatrix , orientation);
            SensorManager.getOrientation(rotationMatrix, orientation);
            imu = orientation; 
	    }
	    	    
	    if(debug && acc != null && gyr != null && mag != null)
	    	debugListener.debugRaw(acc, gyr, mag);
	    
	    if(debug && imu != null)
	    	debugListener.debugImu(imu);	    
	    
		if(sync.getNumberWaiting() > 0)
			sync.reset();			
	}

[mahler]

Ok tried the vector stuff, I cant see its much of an improvemt, the raw values + mahony fusion in FreePIE gives a better result. But it drifts.

What is 'better'? And which one drifts?

[Parity]

Hm.
Isnt the output of Sensor.TYPE_ROTATION_VECTOR -> getOrientation the same as the TYPE_ORIENTATION sensor?
With the same disadvantages as it just returns the Euler Angles resulting from that matrix?

Let me brainstorm that quick: For my purpose I would try to start like that:
Get the rotation vector of the device in a "default" device orientation (calibration "CAL").
Then aquire the actual rotation vector ("ROT") and calculate the angles that are needed to rotate "CAL" to "ROT".

Should work as long as you dont have your theta = 90°, but the difference is that I will not end up with that glitchy situation in my default orientation, what is the case at the moment.

Hopefully I am not totally wrong with that... I will keep thinking about it later.

Wait for it. I am just checking out the file you attached to your post. Looking awesome, will report back in a few minutes.


Okay, I checked it. Forget the theory out of the first part of this post. It is working FLAWLESSLY! No noise, to glitches. Huge improvement. Well done!

Essentially you only need:

Code: Select all

delta_yaw = filters:delta(android.GoogleYaw)
delta_roll = filters:delta(android.GoogleRoll)

mouse.DeltaX = delta_yaw*factorx
mouse.DeltaY = delta_roll*factory

[/color]

[mahler]

Okay, I checked it. Forget the theory out of the first part of this post. It is working FLAWLESSLY!
No noise, to glitches. Huge improvement. Well done!

Essentially you only need (Lua):

Code: Select all

delta_yaw = filters:delta(android.GoogleYaw)
delta_roll = filters:delta(android.GoogleRoll)

mouse.DeltaX = delta_yaw*factorx
mouse.DeltaY = delta_roll*factory

Yeah basically, I got the same result here
This code results in less jitter and less drift compared to Mahoney!
The only difference is that the yaw seems to slide a little bit to a halt...

[CyberVillain]

Ok tried the vector stuff, I cant see its much of an improvemt, the raw values + mahony fusion in FreePIE gives a better result. But it drifts.

What is 'better'? And which one drifts?

Mahony fusion works best for me, but it has some drift

[Parity]

May depend on the device and the quality or type of the sensors, but I couldnt ask for more than what the current version with rotation vector gives me.
Even with large factors of 1000, I can place the mouse pointer at a single pixel on my screen. (While beeing able to move all 1920 pixels within an instant if I wish.) Cannot see any drift either.

Awsome!

[CyberVillain]

Glad it worked for you, the sensors on the HTC One X should be top notch from what I understand. But i get really strane results. It's no drift, but behaves strange when coming to a halt. Mahony feels better and comes to a halt directly, but it has some drift so its not very good for absolut head postion like trackir and freetrack titles

[mahler]

Glad it worked for you, the sensors on the HTC One X should be top notch from what I understand. But i get really strange results. It's no drift, but behaves strange when coming to a halt. Mahony feels better and comes to a halt directly, but it has some drift so its not very good for absolut head postion like trackir and freetrack titles

I get the same strange results (only for yaw), but overall less jitter and drift over time.

[CyberVillain]

Can you guys test this version, beacuse I was playing around with the vector stuff and I do not know if I eneded up with a version that works as nice as the one you have tested

https://github.com/AndersMalmgren/FreeP ... id.imu.apk

[mahler]

Can you guys test this version, beacuse I was playing around with the vector stuff and I do not know if I ended up with a version that works as nice as the one you have tested
https://github.com/AndersMalmgren/FreeP ... id.imu.apk

Works about the same for me (I was actually not using your previous attachment, but used my own code in the app)

But I'm starting to also dislike the slow halt...

Just a question.. why did you add the code below?

Code: Select all

        rotationMatrix[ 0] = 1;
        rotationMatrix[ 4] = 1;
        rotationMatrix[ 8] = 1;
        rotationMatrix[12] = 1;

The way I read the documentation, these will be overwritten anyway...
(and inR-attribute is not used)

[CyberVillain]

Can you guys test this version, beacuse I was playing around with the vector stuff and I do not know if I ended up with a version that works as nice as the one you have tested
https://github.com/AndersMalmgren/FreeP ... id.imu.apk

Works about the same for me (I was actually not using your previous attachment, but used my own code in the app)

But I'm starting to also dislike the slow halt...

Just a question.. why did you add the code below?

Code: Select all

        rotationMatrix[ 0] = 1;
        rotationMatrix[ 4] = 1;
        rotationMatrix[ 8] = 1;
        rotationMatrix[12] = 1;

The way I read the documentation, these will be overwritten anyway...
(and inR-attribute is not used)

hehe, cut and paste from various examples, should remove that for the final version

[mahler]

hehe, cut and paste from various examples, should remove that for the final version

I noticed that the orientation-debugging displays the ROTATION_VECTOR data which doesn't seem very usefull. It would be better to display the calculated orientation values.

[CyberVillain]

hehe, cut and paste from various examples, should remove that for the final version

I noticed that the orientation-debugging displays the ROTATION_VECTOR data which doesn't seem very usefull. It would be better to display the calculated orientation values.

true, will fix that

[CyberVillain]

Hmm, its actually sending the correct data to debug

[mahler]

Hmm, its actually sending the correct data to debug

Yeah I saw that, yet the values in FreePIE are different.

By the way have you tested the Freetrack script with the Android app yet?

[CyberVillain]

Hmm, its actually sending the correct data to debug

Yeah I saw that, yet the values in FreePIE are different.

By the way have you tested the Freetrack script with the Android app yet?

With trackir yes, dont have a freetrack title installed right now.

[logicalChimp]

Hi

I have a couple of questions about the Android IMU app. Mainly, which version of Android was it intended for?

I've tried it on a Google Nexus One (running 2.3.3 iirc) and a Motorola Razr Maxx (running 4.0.x). The Razr gives me nothing at all (it runs, and it can talk to FreePIE, because the diagnostic code in the update function runs, but I just get '0' for all readings. I've tried various combinations of check boxes, but made no difference). The Nexus One only gives me the 'google' readings (and then only if I turn off 'send raw data')- and they are very very jittery/noisy. There is also significant latency (circa 800-1000ms) between moving the nexus, and seeing the numbers in FreePIE update - but that could be down to a poor wireless signal (will try to retest that aspect at work tomorrow, where I may be able to get on a more reliable wifi).

I also found a different 'Android IMU' app on Google Play - with the same default port number, strangely enough. However, whilst the numbers it displays on screen looked good (tracked well, not noisy/jittery, etc) the actual data as received by FreePIE was complete garbage - so I guess it actually sends it in a different format to what you're expecting.

So, after all that, I was wondering if you have any suggestions for more things I could try, or if there are any ' known issues' with certain hardware?

cheers

[CyberVillain]

Minimum SDK is 10 aka 3.3 of Android, I tried to change to SDK 8 (2.2) beacsue another user requested support for his Phone, but sadly we use SDK 10 dependencies. Very strange that you can even install it on a 2.3 device. The nexus probaly only has 2 of the 3 sensors thats why the result is noisy.

When you say Update do you breakpoint in the C# code? Its lots of overhead until the breakpoint will hit, a simple diagnostics.watch(android.googleYaw) script is more accurate to test performance. Also make sure to set To game or fastest sample rate.

Strange that it does not work with 4.0, I have a HTC One X which is a 4.0 Phone, and it works. Do you receive data in the Update method of FreePIE? Are the data all zero? Just because you see numbers in the debug doesnt mean FreePIE will receive the data, it uses UDP which fire and forgets the packages there is no guarantee that your receiving end will get the data. The debug window of the App is more of a tool to see if the Sensor part of the app works.

Let me know how it goes and I will try to help you as best I can

Hi

I have a couple of questions about the Android IMU app. Mainly, which version of Android was it intended for?

I've tried it on a Google Nexus One (running 2.3.3 iirc) and a Motorola Razr Maxx (running 4.0.x). The Razr gives me nothing at all (it runs, and it can talk to FreePIE, because the diagnostic code in the update function runs, but I just get '0' for all readings. I've tried various combinations of check boxes, but made no difference). The Nexus One only gives me the 'google' readings (and then only if I turn off 'send raw data')- and they are very very jittery/noisy. There is also significant latency (circa 800-1000ms) between moving the nexus, and seeing the numbers in FreePIE update - but that could be down to a poor wireless signal (will try to retest that aspect at work tomorrow, where I may be able to get on a more reliable wifi).

I also found a different 'Android IMU' app on Google Play - with the same default port number, strangely enough. However, whilst the numbers it displays on screen looked good (tracked well, not noisy/jittery, etc) the actual data as received by FreePIE was complete garbage - so I guess it actually sends it in a different format to what you're expecting.

So, after all that, I was wondering if you have any suggestions for more things I could try, or if there are any ' known issues' with certain hardware?

cheers

[mahler]

I also found a different 'Android IMU' app on Google Play - with the same default port number, strangely enough. However, whilst the numbers it displays on screen looked good (tracked well, not noisy/jittery, etc) the actual data as received by FreePIE was complete garbage - so I guess it actually sends it in a different format to what you're expecting.

Initially the FreePIE plugin was written to use the Wireless IMU app, but when it turned out there were other methods to retrieve orientation data (as another app SquidyClient BETA was doing) the decision was to make a custom app over which there was more control. This made it possible to customize the output into binary format and send select between fused or raw data. This reduced the size of the packets and resulted in better performance. The FreePIE Android app works fine here on my Google Galaxy Nexus (4.1).

[logicalChimp]

Ahh - that would make sense

Does this also mean that if I dig out one of the older Android plugins, it would be able to work with the Wireless IMU app? I might give that a go later today...

Thanks

[logicalChimp]

Minimum SDK is 10 aka 3.3 of Android, I tried to change to SDK 8 (2.2) beacsue another user requested support for his Phone, but sadly we use SDK 10 dependencies. Very strange that you can even install it on a 2.3 device. The nexus probaly only has 2 of the 3 sensors thats why the result is noisy.

Yeah - I wasn't particularly expecting it to work on the Nexus because other apps that use the IMU didn't (Parrot AR drone, for example). Oh well.

When you say Update do you breakpoint in the C# code? Its lots of overhead until the breakpoint will hit, a simple diagnostics.watch(android.googleYaw) script is more accurate to test performance. Also make sure to set To game or fastest sample rate.

Yup - just doing a diagnostics.watch on the individual values inside the 'update' function of FreePIE (i.e. define an 'update' function in Python with diagnostics calls, and then do 'android.update += update'

Strange that it does not work with 4.0, I have a HTC One X which is a 4.0 Phone, and it works. Do you receive data in the Update method of FreePIE? Are the data all zero? Just because you see numbers in the debug doesnt mean FreePIE will receive the data, it uses UDP which fire and forgets the packages there is no guarantee that your receiving end will get the data. The debug window of the App is more of a tool to see if the Sensor part of the app works.

With the Razr, I only get '0' in the update function (using the diagnostics.watch outputs). I also get no data displayed on the 'debug' screen (well, I get the names of the various sensors, but no data readings).

To be honest, the Android IMU is a temporary feature for me (as I am still waiting for my Sparkfun to arrive), but it would be nice to have a second option.

[CyberVillain]

Ah ok, if you dont see any values in the debug part of the Android app that means that the sensor manager isnt sending any data to the app, very strange, you have one or both of the two checkboxes checked right?

We use the core SDK functions for this so it should work

edit: Btw, if you have the 'Send raw' checkbox checked it will always be a initial delay of 1 second because the FreePIE plugin needs to sample the frequency for the Custom fusion to work correctly, but after that inital second it should sample very fast, i get around 250hz with my phone and the raw mode

[logicalChimp]

Yup - tried it (for at least 10+ seconds each time) with the 'Send raw' checkbox both ticked and unticked - no difference. I also get no error message, or any notification. The debug area is resolutely devoid of data.

Specific android version on the Razr is a stock 4.0.4 motorola image

Unfortunately, FreePIE doesn't work on my office laptop (won't start, probably something to do with the crazy security crap we have installed) so further testing / experimenting will have to wait until tonight.

[CyberVillain]

Yup - tried it (for at least 10+ seconds each time) with the 'Send raw' checkbox both ticked and unticked - no difference. I also get no error message, or any notification. The debug area is resolutely devoid of data.

Specific android version on the Razr is a stock 4.0.4 motorola image

Unfortunately, FreePIE doesn't work on my office laptop (won't start, probably something to do with the crazy security crap we have installed) so further testing / experimenting will have to wait until tonight.

Ok, cant help you much without more info, hope you find something more tonight. I will start to look at giving you the info you need for stop start event for your socket script

[shent1080]

Hi all

I'm new to this and have been getting some help from Cyber villain but it's not fair for him to keep answering my questions.

I've got the Freepie gui console, loaded the android apk to my tablet but i can't get a script to work so the 2 devices can communuicate, you guys all seem to be writing in various languages but the gui looks like it works on python as it asks for a .py script.

Does anyone have a .py script like mahony as it seems like this is the best for tracking?
Am i using the wrong version of FreePIE that prefers .py scripts?

Any help would really be appreciated.

I've attached the code that i've been given but i keep getting " " not expected's all over the place.

[brantlew]

Python is fickle about indentation. You need to indent after the "if" clause.

[shent1080]

No errors from the script now, cyber villain sent me his python script, thanks for the advice though. just need to get the 2 devices speaking to each other now, this is what i sent to cyber villain:

"Thanks cyber villain, the file works without any errors, but now the script runs, i start the android apk on my tablet and nothing happens, i feel like i've missed a step or something. can you have a look over this when you get chance.

1. installed freepie on pc
2. installed apk on tablet
3. opened freepie on tablet
4. opened freepie and script on pc
5. set settings on tablet to my ip and port 5555
6. check setting on android plugin and that is port 5555
7. run script
8. tun on tablet app
9. press z to start on pc
10. pressed right mouse on pc.

No errors come up just seems like they are not communicating, when i press z the cursor goes to the top left of the screen and stays there, when i right click the mouse the only thing that happens is a drop down menu appears like normal when you right click the mouse. any ideas?"

I'm guessing i've missed an obvious step, i'm using the android apk included in the freepie setup, if you get chance could you look over it, no probs if not though.

Thanks for the help.

[mahler]

The steps seem fine.

There are two things you could do.

1) Check if your tablet has the necessary sensors (which model? which Android version?) - To test this, enable debug-mode on the Android-app and see if the sensor-values appear.

2) Check the signal between the Wifi-connection and your PC. A firewall might block it somehow.
You can verify any values coming into the FreePIE application on your PC by adding the following to your update() code:

Code: Select all

    diagnostics.watch(android.yaw)
    diagnostics.watch(android.pitch)
    diagnostics.watch(android.roll)

[shent1080]

Hi mahler

The tablet i'm using is sensing all of the data magnometer, gyro, accelerometer etc.

It's a asus transformer tf101 with ICS

I'll turn the firewall off and try that.

I've been told that python is very sensitive, where exactly do i put the 3 lines of code you put in your last post, directly under update()? with an indent or not?

Thanks for the info.

[mahler]

I've been told that python is very sensitive, where exactly do i put the 3 lines of code you put in your last post, directly under update()? with an indent or not?

You can remove all other code and just use these lines - note I made a mistake, it should be lower case 'diagnostics'. Every new packet that is received by the Android plugin in FreePIE should update the values.

But I did find the probable cause... a bug

When I check both 'Send orientation' and 'Send raw data' in the Android app, the plugin can't seem to parse the raw data (android.yaw, android.pitch, android.roll) but the pre-calculated orientation values are picked up just fine (android.googleYaw, android.googlePitch, android.googleRoll)

You can work around this by unchecking 'Send orientation' in the Android app and tap the "On" button.

Note to CV: You may want to add the Android app to your download pages

[CyberVillain]

Cool, thanks will check that bug Mahler, havent noticed it myself. Strange that it stopped working.

edit: the location of the app is in the doc´s for the plugin, will attach a working version to this thread though once I have time to find the bug, wont happen today sadly

[mahler]

Cool, thanks will check that bug Mahler, havent noticed it myself. Strange that it stopped working.

Yeah I remember that it used to work just fine.

edit: the location of the app is in the doc´s for the plugin, will attach a working version to this thread though once I have time to find the bug, wont happen today sadly

I meant that it would be helpful to have a direct download link on this page: http://andersmalmgren.github.com/FreePIE/

[shent1080]

Thanks Mahler

All the diagnostic figures are being picked up from the tablet, i don't know what they refer to on the tablet though, Gyro, magno, or accelerometer?

this was checked without send orientation being ticked, if this is ticked it works for maybe half a second then all of the values go to NaN.

I've changed android.yaw etc in the script that cyber villain gave me to android.googleYaw and now i can move the cursor from left to right, don't know about getting the other axis to work though, do you know of any other good scripts and apk's, seen a lot of people talking about mahoney but will that work in the gui version of freepie? and what apk works with it?

Cheers

[CyberVillain]

If you only have data raw ticked you get values from android.yaw? Then it should work with the mouse sceipt?
mahony fusion is used to fuse the raw data into yaw, pitch roll.

Y does not work even with Google fusion? Pitch and roll can be swiched depending of you hold your phone. Try mapping mouse y to roll and vice verca

[mahler]

All the diagnostic figures are being picked up from the tablet, i don't know what they refer to on the tablet though, Gyro, magno, or accelerometer?

I admit that it's rather confusing at first.

yaw/pitch/roll are the rotation angles over the x,y,z axis.

The device sensors gyro (rotation), acc (positional) & magno (rotation) will output raw values.
These values need to be fused to get a reliable result.

The Android app has two options for fusion:

1) Let Android do the fusion itself (to googleYaw/-Pitch/-Roll) on the device using Android proprietary code and send those to FreePIE on the PC

2) Only send the raw values from the sensors and perform custom (Mahoney) sensor fusion in the Android plugin in FreePIE on the PC

My personal experience is that using android.googleYaw/-Pitch/-Roll (on-device sensor-fusion) is not as responsive but has less drift compared to the android.yaw/pitch/roll (Android pluging sensor-fusion) - but you can experiment with that yourself.

[shent1080]

I've tested the script with android.googleYaw and pitch on skyrim, with both raw and orientation checked the screen goes crazy, with just orientation checked i can turn left and right plus look up and down, the cursor wont return back to the original position, also turning far left and right doesn't go 360 degrees but instead flips your view to the other side of the screen, finally looking far down goes into a u shape. Thanks for the reply's, i've got a lot to check out.

Mark

[mahler]

with just orientation checked i can turn left and right plus look up and down, the cursor wont return back to the original position, also turning far left and right doesn't go 360 degrees but instead flips your view to the other side of the screen, finally looking far down goes into a u shape.

This gives me the best results for using the raw-data + custom fusion

Code: Select all

def update():
	yaw = math.degrees(android.yaw)
	roll = math.degrees(android.roll)
	pitch = math.degrees(android.pitch)

	#diagnostics.watch(yaw)
	#diagnostics.watch(roll)
	#diagnostics.watch(pitch)

	deltaYaw = filters.delta(yaw)
	deltaPitch = filters.delta(-roll)
	
	if (enabled and hotkey):
		mouse.deltaX = -deltaYaw*multiply
		mouse.deltaY = deltaPitch*multiply

if starting:
	enabled = False
	multiply = 20
	android.update += update

hotkey = mouse.rightButton
toggle = keyboard.getPressed(Key.Z)

if toggle:
	enabled = not enabled

Play around with the multiplier-value to get the right sensitivity