Head Tracking/Gun controller help and suggestions required!!

Post by **pierreye** » Tue Nov 08, 2011 12:56 am

Very nice demo. So the tracking are based on both HMD and the gun device in the same ratio? You can try out in a soccer field for more immersive experience.

Post by **fireslayer26** » Tue Nov 08, 2011 11:10 am

Great job Cyber!! You are a pioneer my friend! Oh, and congrats on making the Front page of MTBS3D!

@pierreye- Really looking forward to your video with your setup when you get the Sony HMD!!

Post by **brantlew** » Tue Nov 08, 2011 12:18 pm

WiredEarp wrote:How hard would it be to rig up your system so you can actually 'walk' around inside the VR if you had a sufficiently large room? That would add immersion as well (seems a waste of a freedom VR setup to be stuck in one spot?).

Stay tuned. I've got a freedom VR system in development (10%) right now.

Post by **pierreye** » Tue Nov 08, 2011 5:32 pm

Too bad I preorder from avsforum and all dealers do not get the first batch stock, only those preorder from sony website. I don't know when I can get a unit. Have all the accessories ready but without a HMD

Post by **cybereality** » Tue Nov 08, 2011 7:26 pm

Thanks guys!

WiredEarp wrote: How hard would it be to rig up your system so you can actually 'walk' around inside the VR if you had a sufficiently large room? That would add immersion as well (seems a waste of a freedom VR setup to be stuck in one spot?).

The Vuzix 1200VR has some rudimentary support for translation tracking. Its not very accurate, but with a good filter it should be able to track basic motion. There is also the accelerometer on the Wiimote, that could be used for this purpose as well. So I don't think it would take long to add support like that. The problem is that I live in a big city, and there is not a lot of free space around here. I mean, I could go to a park or something, but I'd be afraid of either getting mugged (with a few thousand dollars worth of equipment on my back) or getting shot by the cops! So I am a lot more interested in things that can be accomplished in small spaces: omni-directional treadmills, cyber-shoes and things like that.

pierreye wrote:Very nice demo. So the tracking are based on both HMD and the gun device in the same ratio?

The headtracking is mapped about 1:1 with the game. You see in the video, I make a full 360 degree turn and end up facing the same way in the game. The gun is also mapped 1:1 (about) with the gun in the game. However, the gun is disabled if the head is moving (past a very low threshold). This actually works really well. It allows you to walk around and navigate with full 1:1 rotational accurate, but if you stop you can then fine-tune the aim with the gun. I like this a lot more than using a hot-key to toggle the gun on or off. Its also a lot better than having the head and gun active all the time, which gets confusing and causes headaches.

---------------------------------------------

My next step will probably be integrating the Kinect and using the skeletal tracking to map to movements. Mainly: walking, jumping and ducking. The idea would be that you could walk in place and that would trigger movement. I am not sure how well this would even work (for example, walking backwards may be hard to track, as well as circle strafing) but its worth exploring. I think this will add some immersion, but certainly not as much as you would with full tracking in a big open space. Then maybe I can look into getting a manual treadmill (since this is affordable and off-the-shelf) and seeing how that works. So this certainly isn't the end of this project.

Post by **Fredz** » Wed Nov 09, 2011 12:33 am

cybereality wrote:My next step will probably be integrating the Kinect and using the skeletal tracking to map to movements. Mainly: walking, jumping and ducking. The idea would be that you could walk in place and that would trigger movement.

You can also have a look at this paper which explains how to navigate in a virtual world by walking in place using only head-tracking information :
- Taking Steps: The Influence of a Walking Technique on Presence in Virtual Reality

They used a neural network to analyze the stream of coordinates from the head-tracker and determine if the user was walking. They said the implementation was quite straightforward, they got training data from someone doing several tasks, then they trained the neural net with that data. They didn't handle running or jumping but they did handle climbing or descending steps, so I guess there is room for extension.

Post by **cybereality** » Wed Nov 09, 2011 10:55 pm

@Fredz: Sounds intersting. And here I was looking for a reason to play with neural nets. This looks like fun.

Post by **brantlew** » Wed Nov 09, 2011 11:20 pm

Hate to play devil's advocate but neural nets are infuriating. They work well up to about 80% of your solution, but then you are stuck at that plateau and since the solution is "black-box" by nature, you have no way of diagnosing and iterating on the solution. Direct engineering is tedious and difficult but it gives you a thorough understanding of the problem and an easier pathway to problem solving.

If you must use neural nets, I would still use the Kinect skeletal model as the inputs because the signals are orders of magnitude clearer and separable than head bobbing alone.

Post by **cybereality** » Thu Nov 10, 2011 9:49 pm

@brantlew: Thanks. I definitely want to explore what can be done with Kinect (neural net or not).

Post by **fireslayer26** » Wed Nov 23, 2011 9:14 pm

Any new videos or creations??

Post by **mAchiNE** » Thu Nov 24, 2011 3:53 am

I'm not sure about how a neural net will be useful for VR applications but OCZ brought out a Neural Impulse Actuator back in 2008
http://techreport.com/articles.x/14957
They don't seem to sell it anymore but if you think this might be useful you could look for one on ebay, I always thought it would be kinda cool to use it in fantasy games like Skyrim to cast spells etc, other than that it kinda seems pointless, I mean its not very realistic to fire a gun with your mind (or facial movements) etc

Still, it was a cool device though...

EDIT: maybe they still do make it http://www.ocztechnology.com/nia-game-controller.html you can get it on Amazon as well

Post by **cybereality** » Thu Nov 24, 2011 12:03 pm

Hmm.. The NIA is only $99 bucks. Why do I feel the urge to buy it? I know it probably works like crap and will just sit on my desk with the rest of this junk.

Post by **WiredEarp** » Thu Nov 24, 2011 10:22 pm

Hahah yeah, theres also another brain headset as well (eMotive?). I've always been tempted, but they are basically just biometric readers and apparently require significant setup and training. Be fun for meditation training etc but think it would just be too much hassle to use for gaming regularly.

Post by **cybereality** » Thu Nov 24, 2011 10:25 pm

Totally agree. I think they are basically useless for gaming. Was thinking more about doing dream research and other stuff like that with it. I've got a lot of stuff on my hands as it is though.

Post by **bobv5** » Fri Nov 25, 2011 1:34 pm

"Theres no reason that this 1:1 motion could not be done with relative movement. Assuming the tracker had no drift, this would be basically 100% identical to absolute movement. The only problem areas would be looking straight up or straight down (since the game will clip your movement). But if you avoid those areas, it should be the same. Now, in reality, trackers do have some drift, so you will need a way to "reset" or "recenter" the tracking. But this seems like only a minor annoyance."
(From a differnt page)

Not sure if this was mentioned, but a way to recenter the vertical view is for the software to detect when the motion tracker is tilted up or down as far as you neck can go, and move the mouse very far up or down. This way when the view gets out of line you just need to look at your feet for a second. Anyway, not my idea, and the other guy explains it better than me.

"One problem I've noticed with the improved mouselook and general mouselook scripts Carl gave out is that general gaming lacks one thing: recentering. If your pitch goes out of line, the GlovePIE can't recognize it since it doesn't get feedback from the game. This is a major problem. Whenever you respawn, look around too fast, something moves, something moves you, or something moves where you're looking, your yaw goes out of line, a MAJOR problem that makes it much less realistic. The TRACKIR emulation part of the script allowed for recentering at user level, but anything else requires the mouse to change anything, with varying results.

However, I've found one excellent feedback mechanism: No game lets you look down more than, well, DOWN. Same for looking up.This script moves the mouse as far as it can down and up when the VR920's actual pitch is down or up.

Use is simple: Calibrate VR920 accelerometer (for crouch detection, no jump), run script, and look all the way up or all the way down. This will line up your pitch, do this every time you respawn, pitch gets misaligned, etc etc"

http://forums.vr920.com/Topic4679-8-1.aspx" onclick="window.open(this.href);return false;

Post by **cybereality** » Fri Nov 25, 2011 3:37 pm

@bobv5: Thats a good idea.

Post by **pierreye** » Wed Jan 04, 2012 4:22 pm

Need some input from you guys on below ideas. I'm trying to work out the maths required to get an absolute pitch for any FPS.

As we know, the gyro from motion plus will drift from absolute center in FPS but I found out it is possible to get the absolute value from wiimote pitch (in glovePIE as wiimote.pitch in degrees). The problem that prevent me from using wiimote.pitch is it is not smooth and a little bit jumpy. Also it is possible to read the absolute value of MouseDirectInput so we can know which value is absolute center in the game. By combining all three info from gyro, pitch and MouseDirectInput, we can have a perfect pitch control using just a wiimote with motion plus.

I'm thinking to use gyro data (relative) that is calibrated against pitch reading (absolute). Also at 0 degree pitch, I can read the DirectMouseInputY and calibrate that as center position. If we can nailed the perfect pitch in FPS, I think the whole setup would be perfect as we do not need to constantly recalibrate in game. COD series normally is easier to center the pitch as there is a function to center view but not Battlefield series.

Post by **CyberVillain** » Wed Jan 04, 2012 5:23 pm

Mouse.DirectInputX is just a relative value, there is no way for the Mouse to now where in the viewport you are looking.. sorry

edit: Mouse.DirectInputX even have some drift without headtracking.. run this script, debug = Mouse.DirectInputX click run and be sure to hold the mouse icon in center of Play icon, the vaue will be zero, move the mouse left or right, do not go all the way out to the edges of screen because the cursor will stop but not the counter (Thats the difference with Mouse.X and DirectInputX) Return the mouse to center of Play icon, notice that the counter will be off...

Post by **cybereality** » Wed Jan 04, 2012 10:26 pm

I am trying to hook into DirectX and, if I can pull it off, it will be possible to obtain the absolute orientation of the camera. I won't make any promises, but I believe it can be done.

Post by **pierreye** » Thu Jan 05, 2012 1:48 am

CyberVillain wrote:Mouse.DirectInputX is just a relative value, there is no way for the Mouse to now where in the viewport you are looking.. sorry

edit: Mouse.DirectInputX even have some drift without headtracking.. run this script, debug = Mouse.DirectInputX click run and be sure to hold the mouse icon in center of Play icon, the vaue will be zero, move the mouse left or right, do not go all the way out to the edges of screen because the cursor will stop but not the counter (Thats the difference with Mouse.X and DirectInputX) Return the mouse to center of Play icon, notice that the counter will be off...

I actually try out DIrectInputY tie directly to the pitch value and seems like the mouse cursor always return to center when the pitch = 0 in Windows.

Post by **CyberVillain** » Sun Jan 08, 2012 1:43 pm

cybereality wrote:I am trying to hook into DirectX and, if I can pull it off, it will be possible to obtain the absolute orientation of the camera. I won't make any promises, but I believe it can be done.

That works for FPS shooters where you want to stear the gun sight, but it will not help for freelook games :/

edit: But with the data you can ofcourse probablt calibrate the relative freelook to be more absolute, so let me know when its done

Post by **pierreye** » Thu Jan 12, 2012 5:08 pm

I had improved the script by taking roll into consideration. Previously, I just tie the raw data from gyro directly to x,y. But if you turn the wiimote by certain angle, the x,y axis will be slightly tilted thus the drift is greater when everytime the angle of the wiimote is different. Example, if you turn your wiimote 90 degrees, yaw would become pitch and vice versa.

Will try out the new script later.

Post by **bobv5** » Thu Jan 12, 2012 5:41 pm

Was playing about with your script earlier, seems good, although I was just playing sat on the chair with wiimote in hand. I noticed that it had some code in it for the sensor bar. Do you know what this does? Would be really great if the script used it as a fixed refrence point, you could have it sat somewhere that the gun would end up pointing ocasionally while playing, and compensate for gyro drift.

Have you looked at using the accelerometers for jump and crouch? Not sure jumping would feel right, but crouching while also leaning and fireing round a corner would be good.

Post by **pierreye** » Thu Jan 12, 2012 9:00 pm

The IR sensor bar code is use to recenter the cursor but I think it is useless in our case as pitch is relative plus we are using 360 degree movement.

Post by **CyberVillain** » Fri Jan 13, 2012 6:26 am

pierreye wrote:The IR sensor bar code is use to recenter the cursor but I think it is useless in our case as pitch is relative plus we are using 360 degree movement.

Hmm, when it detects the center bar (And that its in center) maybe you can send a pitch up (To the top) very fast and down to center (You have to know the mouse speed to be able to get center as its relative).. Also have a timer or require that the center bar is out of screen and then visible again for the code to trigger so it does not do this too often...

Post by **pierreye** » Fri Jan 13, 2012 10:03 am

I don't think that is necessary as you can get the absolute pitch from wiimote.pitch which show you the degrees but the problem with wiimote.pitch is a bit jumpy to use for smooth aiming. A quick scroll up and back to center might not work in sniping mode which is quite annoying. I do experiment with auto center which detect if the pitch is within 1 degrees from absolute horizontal level every 10 secs, it will auto center in COD series. I found out it kind of screw up my aiming that it auto-center when I'm about to shoot the enemy especially in sniping mode.

Another update on the roll compensation that I try out earlier in COD:MW3 and it work perfectly. The drift had been minimized and only to recenter occasionally. You can now hold your rifle 90 degree and still the X,Y movement is working correctly. This setup would be great for Dead Space 2 where you can turn 90 degree to fire in alternative mode while having the correct movement.

Post by **CyberVillain** » Fri Jan 13, 2012 11:59 am

IN arma2 you can use TrackIR and freetrack at the same time, so if you bind Freetrack to headtracking and TracKIr to movement you could have absolute movement both for looking around (headtracking) and when moving body (Moving your weapon)

Youll need two wiimotes offcourse and you need to calibrate the headtracking one to the other so that where the weapon is pointed is center for the headtracking yaw.. Would be reaaaaaly cool

edit: sorry, I just assumed that you could have absolute movement since the game has absolute movement for the head. But i tried to assign the movement to my freetrack, and its relative :/ Really stupid of them

Post by **pierreye** » Fri Jan 13, 2012 6:30 pm

Here's the updated script with roll compensation.

Post by **Syntax** » Tue Jan 17, 2012 9:08 am

Seems like we are reinventing the wheel in here

My setup i have posted before used a hmd, with a gyration air mouse mounted on top and a wii zapper gun consisting of a wii remote, motion+ and nunchuck mounted in a cheap plastic gun. Basically it is dual mouse emulation with bound keys for movement and other options. You can hold a button on teh wiimote to use teh tracking or not so you can easily switch between head or gun tracking or both, it works quite well. Cant recommend gyration air mouse any more highly, these things are great.

(I think) this the wii script i used, the mouse tracking is jittery but fun enough. Note i always bind 'e' to jump in my fps and space to duck, makes sense right?

Quote:
Key.W = Wiimote1.Nunchuk.up
Key.a = Wiimote1.Nunchuk.left
Key.s = Wiimote1.Nunchuk.down
Key.d = Wiimote1.Nunchuk.right
Key.E = Wiimote1.Nunchuk.ZButton
Mouse.LeftButton = Wiimote1.Nunchuk.CButton

if starting then
// define deadzone, calibration and sensitivity
var.calibrationGyro = [-1.5, -0.5] // [Yaw, Pitch]
var.deadzoneGyro = [7, 7] // [Yaw, Pitch]
var.sensitivityGyro = [0.55, 0.65] // [Yaw, Pitch]; lower values result in slower movements, negative values will invert the corresponding mouse axis
endif

// hold B to move mouse with WiiMote
if (WiiMote.B) then
var.Gyro = [0, 0]
if (Abs(WiiMote.MotionPlus.YawSpeed + var.calibrationGyro[1]) > var.deadzoneGyro[1]) then var.Gyro[1] = RemoveUnits(WiiMote.MotionPlus.YawSpeed + var.calibrationGyro[1]) - (Sign(WiiMote.MotionPlus.YawSpeed) * var.deadzoneGyro[1])
if (Abs(WiiMote.MotionPlus.PitchSpeed + var.calibrationGyro[2]) > var.deadzoneGyro[2]) then var.Gyro[2] = RemoveUnits(WiiMote.MotionPlus.PitchSpeed + var.calibrationGyro[2]) - (Sign(WiiMote.MotionPlus.PitchSpeed) * var.deadzoneGyro[2])
Mouse.DirectInputX = Mouse.DirectInputX + (var.Gyro[1] * var.sensitivityGyro[1])
Mouse.DirectInputY = Mouse.DirectInputY + (-1 * var.Gyro[2] * var.sensitivityGyro[2])
endif
if wiimote.b = false then var.1 = 2
if wiimote.b = true then var.1 = 10

mouse.LeftButton = wiimote.A
mouse.RightButton = wiimote.Plus

if wiimote.down = true then mouse.directinputy = mouse.directinputy + var.1
if wiimote.up = true then mouse.directinputy = mouse.directinputy - var.1
if wiimote.left = true then mouse.directinputx = mouse.directinputx - var.1
if wiimote.right = true then mouse.directinputx = mouse.directinputx + var.1

The first thing i realised once i got this setup is that i needs real movement tracking

Great work Okta, but I ve a problem with the nunchuck.
When I start the script with no nunchuck plugged in the tracking(gyros) is working fine.
But when I plug the nunchcuk in and start the script I can only use the nunchuck means the tracking is mysteriously disabled. Also have tried to add a second wiimote but there was the same probelm. So How can I track and use the nunchuck at the same time and is it possible to only track with the first wiimote and use the second as controller?

Plz help

P.s: I finally got my hands on my very own HMZ-T1 and its so amazing once you make it comfortble and set the picture settings right.

Post by **pierreye** » Tue Jan 17, 2012 9:42 am

My previous script using Nunchuk with MotionPlus had the same issue.

What you need to do is connect the Wimote + Motion Plus without Nunchuk first. Click RUN in GlovePIE and connect the Wiimote by pressing 1 + 2. Once connected and calibrated, stop the script and plug in the Nunchuk. Then run the script again and it should be working fine. Again, I suggest getting the PS3 Navigator as it come with more button and wireless. You can press down the joystick as "Shift" for either sprint or steady sniping mode.

Post by **CyberVillain** » Tue Jan 17, 2012 9:46 am

pierreye wrote:My previous script using Nunchuk with MotionPlus had the same issue.

What you need to do is connect the Wimote + Motion Plus without Nunchuk first. Click RUN in GlovePIE and connect the Wiimote by pressing 1 + 2. Once connected and calibrated, stop the script and plug in the Nunchuk. Then run the script again and it should be working fine. Again, I suggest getting the PS3 Navigator as it come with more button and wireless. You can press down the joystick as "Shift" for either sprint or steady sniping mode.

Cant you detect when you hold your breath and steady the scope?

Post by **Syntax** » Tue Jan 17, 2012 10:42 am

What you need to do is connect the Wimote + Motion Plus without Nunchuk first. Click RUN in GlovePIE and connect the Wiimote by pressing 1 + 2

my wiimotes are already connected via my toshiba bluetooth driver software, so I dont know what you mean by saying "press 1+2 to connect"
They are allready connected.
But when the nunchuck is pluged in the gyros are disabled... WHY???

Post by **pierreye** » Tue Jan 17, 2012 7:40 pm

CyberVillain wrote:
pierreye wrote:My previous script using Nunchuk with MotionPlus had the same issue.

What you need to do is connect the Wimote + Motion Plus without Nunchuk first. Click RUN in GlovePIE and connect the Wiimote by pressing 1 + 2. Once connected and calibrated, stop the script and plug in the Nunchuk. Then run the script again and it should be working fine. Again, I suggest getting the PS3 Navigator as it come with more button and wireless. You can press down the joystick as "Shift" for either sprint or steady sniping mode.
Cant you detect when you hold your breath and steady the scope?

That's pretty hardcore for VR

Post by **pierreye** » Tue Jan 17, 2012 7:42 pm

Syntax wrote:
What you need to do is connect the Wimote + Motion Plus without Nunchuk first. Click RUN in GlovePIE and connect the Wiimote by pressing 1 + 2
my wiimotes are already connected via my toshiba bluetooth driver software, so I dont know what you mean by saying "press 1+2 to connect"
They are allready connected.
But when the nunchuck is pluged in the gyros are disabled... WHY???

I'm using standard MS Bluetooth stack and always have to press 1 + 2 on Wiimote for first connection to GlovePIE. In this case, just connect first without Nunchuk, then stop GlovePIE and plug in Nunchuk. Run GlovePIE the second time and it should work.

Post by **mayaman** » Wed Jan 18, 2012 3:55 am

Your video on YouTube is unreal. I'm hoping to emulate it. Please upload more videos and hopefully some tutorial videos for the more "green" among us.

Post by **Syntax** » Wed Jan 18, 2012 6:35 am

thx guys for all your suggestions.
Finally I got it to work. When your Wiimotes are already connected, I had to start the script first without motion plus but when the script starts to calibrate the wiimote I plugged in the motion plus + nunchuck whlie the script is running and there you go

Post by **Max3d** » Wed Jan 18, 2012 11:54 am

Hi guys,
Completely new here. I was looking for the guy who posted the amazing HMZ-T1 wireless VR FPS gaming video on youtube. Uploaded by rjyap1 on Dec 27, 2011
I would like to hear how he build this setup or something similar for a blog I maintain. It´s focused on the European Tech scene so maybe this is completely the wrong spot to ask.

(and yes I have been in stereo since ´90´s, wrote a 3D renderer in 1990, have an auto-stereoscopic monitor etc. so I´m not just some dumbo

Post by **bobv5** » Wed Jan 18, 2012 12:23 pm

The one near the top of this page?

http://www.mtbs3d.com/phpBB/viewtopic.p ... &start=780" onclick="window.open(this.href);return false;

You could be in the right place.....

Post by **pierreye** » Wed Jan 18, 2012 8:30 pm

Max3d wrote:Hi guys,
Completely new here. I was looking for the guy who posted the amazing HMZ-T1 wireless VR FPS gaming video on youtube. Uploaded by rjyap1 on Dec 27, 2011
I would like to hear how he build this setup or something similar for a blog I maintain. It´s focused on the European Tech scene so maybe this is completely the wrong spot to ask.

(and yes I have been in stereo since ´90´s, wrote a 3D renderer in 1990, have an auto-stereoscopic monitor etc. so I´m not just some dumbo

That's me! If you need a step by step video on the setup, I'll do that next 2 weeks when I have free time.

Post by **mayaman** » Fri Jan 20, 2012 1:41 pm

pierreye wrote:
Max3d wrote:Hi guys,
Completely new here. I was looking for the guy who posted the amazing HMZ-T1 wireless VR FPS gaming video on youtube. Uploaded by rjyap1 on Dec 27, 2011
I would like to hear how he build this setup or something similar for a blog I maintain. It´s focused on the European Tech scene so maybe this is completely the wrong spot to ask.

(and yes I have been in stereo since ´90´s, wrote a 3D renderer in 1990, have an auto-stereoscopic monitor etc. so I´m not just some dumbo
That's me! If you need a step by step video on the setup, I'll do that next 2 weeks when I have free time.

OMG yes please!!! This would be fantastic. I have the ability to get all the equipment but not the procedure of putting it all together. Please be as detailed as possible.

Thanks!