VR/AR Windows "Desktop" development

[NickK]

@LeeN:
Alright. Sounds like you have the problem taken care of already.

@druidsbane:
I am trying to understand how the frame buffer and pixel buffer work in 3D. And whether you code can be generalized to map 2D windows into 3D outside of the projected desktop.

If my understanding is correct, in the present implementation the frame and pixel buffers contain culled 3D scene. Thus, parts of windows hidden by other windows are not rendered into the framebuffer. This will prevent mapping them into 3D. Is it correct?

However, there appears to be a workaround. I'm reading that the OpenGL framebuffer does NOT have to have the same dimensions as the monitor. Thus, conceptually I can use the framebuffer the same way LeeN uses Xephyr to get window textures in memory. Suppose you tweaked the window manager to always position new windows on a humongous virtual monitor in an non-overlapping fashion. Next, you can create a humongous framebuffer and render different windows into different parts that framebuffer. This way the windows will not overlap in GPU memory (framebuffer). Then, you can allocate a part of that buffer to actually be displayed by setting the viewpoint on it. Rendering windows into 3D will be equivalent to copying data from other parts of the framebuffer into the part that's visible. Since that copying is GPU memory -> GPU memory it should be fast. Is it feasible to do it? Have I missed anything that makes it impossible?

[druidsbane]

Two things:

1) http://blog.mecheye.net/2012/06/the-lin ... ics-stack/ - an interesting coverage of the Linux graphics stack, the part at the end about Wayland makes me want to give it another shot once what I have fully works. Maybe I'll reach out to their mailinglist for help, however I just prefer to Google things and I guess it hasn't been stable long enough to get that info out there

2) @NickK: The code I have runs in two modes, at the very top there is a constant I set that says use texture or something like that. That one renders the desktop orthographically to a texture and that's that. Everything is proportional, etc... The other solution is if you turn it off the code renders it in 3D with perspective. In that case then you can place windows anywhere you want.

In terms of the framebuffer, you're right, they don't need to have the same dimensions as the screen at all. They are just surfaces you can use for different purposes, depth buffering, colour, etc... I have 3: 1 to render the desktop to a texture, and 2 to render the left and right views. The latter two as far as you care are fully 3D, they have a depth buffer and all they do is chop off the bits of the scene we don't care about for each eye. You can render your desktop there if you want in full 3D and ignore rendering to a texture which was my optimization so everything looks right. If you turn off the render desktop to texture constant in my code you'll see that windows that move off the surface of the desktop will hang off the edge That's b/c it is 3D and there is no clipping or anything. Give it a shot

The problem is with X11 and compositing that it is hard to get the mouse to move in 3D. There is a mapping function in the compositor, but I'm not 100% certain we'd want to use it that way. The best option is to create a virtual desktop per app, switch between virtual desktops, then we can transform the mouse so it appears on the right window surfaces if needed. This is getting into window manager & compositing territory.

Wayland would simplify all this as each Wayland surface is essentially its own desktop containing just one application. We own the mouse and can decide when it is on one surface or another, and when it is we can control where it appears. With X11 and virtual desktops I think we can do it using the WM_HINTS and assigning one per app, but still not sure how it would work 100%.

[LeeN]

druidsbane, another option... you can try to do what I originally was doing (and what I may go back to), and send mouse events directly to windows your self. It is quite a bit of work to get crossing events to work right though, and I never handled grabs so I'm not sure what that involves and I dont know how that will work in your case.

[druidsbane]

druidsbane, another option... you can try to do what I originally was doing (and what I may go back to), and send mouse events directly to windows your self. It is quite a bit of work to get crossing events to work right though, and I never handled grabs so I'm not sure what that involves and I dont know how that will work in your case.

I agree with that. I think there are two main options: grab the mouse pointer and forward events to any window or virtual workspace you want, or to go the Wayland route which might be the proper solution (though I found it confusing with what I played with at first). I do grab the mouse pointer in navigation mode so that's not too difficult so I guess events can be forwarded that way but as I'm not that familiar with the details of X I'm sure I'd miss forwarding some crucial event so best to not even try With Wayland it might be different as it seems less hackey or rather, less global everything, global windows, global devices, global surfaces, no clear heirarchy, etc... that seems to be the X way. Though it may still not be the most ideal for a 3D UI I think it has the best chances once it is properly released with a stable API, well documented and supported. Definitely keeping my eye open

Looking forward to seeing your future updates as well, especially the new input methods that you want to test. I don't have anything novel to try out, nor do I have any devices yet so I'm stuck with mouse everything.

[StreetRat]

One thing i am doing is creating a new type of mouse pointer.
For now its just a cube that can move in full 3d and have it change color when it interacts with something.
I hope to make it a circle at some point. As you move it behind objects it disappears behind them. Move it though something and it changes color. Anything to help give a visual representation of depth can only be usefull

I have it so mouse movement up down is still move up down, left right is still left right, but shift up down becomes move in out along the z axis. Could probably change that to mousewheel at some point

That way you just run though a check of the position of all items in regards to the mouse via basic collision detection, and then the element once you have the application.
If you recreate your own border/titlebar across the top then all movement / rotation is done through that, and any other interaction is sent on to the app normally.

If your creating a 3d desktop then the standard 2d mouse pointer no longer seems appropriate.

As much as we all want it, theres no way to make a 3d desktop compatible with everything unless its built as a desktop manager from the ground up. Trying to program to work with everything will be a nightmare

[druidsbane]

Just a quick update. I now have it so the mouse is captured and rendered correctly (though I haven't taken into account the hot zones where I should offset the mouse slightly to I think). All the menu and window issues have been fixed and the original startup crash. I still get another one on my NVidia desktop, might be related to using OpenGL before the context or window are ready... we'll see. Next up will be to try and get the view warped correctly and optimize the FBO's. It is completely useable now and around 250fps Screenshots on my site's project page though need to get a video up at some point.

[NickK]

@druidsbane:
There are still a few memory leaks that need to be fixed.

Line 1113: leaking memory because it is never freed:
eventmask.mask = (unsigned char *)calloc(eventmask.mask_len, sizeof(char));

Line 1239-1240: cursor_image gets allocated by XFixesGetCursorImage() but never freed:
XFixesCursorImage *cursor_image;
cursor_image = XFixesGetCursorImage (dpy);

Line 626: I don't understand what's leaked here but valgrind is complaining that something's wrong with XQueryTree() call:
==3845== 250,216 bytes in 385 blocks are definitely lost in loss record 1,177 of 1,188
==3845== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3845== by 0x589D63F: XQueryTree (in /usr/lib/x86_64-linux-gnu/libX11.so.6.3.0)
==3845== by 0x40CFED: renderDesktopToTexture() (ibex.cpp:626)
==3845== by 0x40D9D5: renderGL() (ibex.cpp:800)
==3845== by 0x40F92E: main (ibex.cpp:1437)

[LeeN]

XQueryTree needs to have XFree called on the returned children pointer if it is not already doing that.

[druidsbane]

@NickK and @LeeN: you guys are the best! I hadn't profiled it at all and didn't know about the free's required also, my only suspicion was about the cursor, hehe. Thanks!

[NickK]

@druidsbane:
I'm afraid there is another problem in the code. There is a performance bottleneck on XGetWindowAttributes() functions. Please, see the profiler output below.

Functions sorted by runtime:
http://img560.imageshack.us/img560/1538/functions1.png


This is the hotspot in the body of renderDesktopToTexture() function:
http://img411.imageshack.us/img411/3572/detailed1.png


The above screenshots indicate that you are spending 80% of the runtime by calling XGetWindowAttributes() function in the loop inside renderDesktopToTexture() function.

I'm wondering if we really need to constantly query the attributes of child windows? We'll probably need to test if this query is actually slowing down the rendering loop.

[NickK]

@LeeN:
I finally received my Hydra. Also, I was recommended to try its usability on Portal 2 game. Not sure if you heard about it. In any case, if you enable the Hydra input in your code I will be able to do some independent testing (and of course complain like users always do).

One thing i am doing is creating a new type of mouse pointer.
For now its just a cube that can move in full 3d and have it change color when it interacts with something.
I hope to make it a circle at some point. As you move it behind objects it disappears behind them. Move it though something and it changes color. Anything to help give a visual representation of depth can only be usefull

I have it so mouse movement up down is still move up down, left right is still left right, but shift up down becomes move in out along the z axis. Could probably change that to mousewheel at some point

That way you just run though a check of the position of all items in regards to the mouse via basic collision detection, and then the element once you have the application.
If you recreate your own border/titlebar across the top then all movement / rotation is done through that, and any other interaction is sent on to the app normally.

If your creating a 3d desktop then the standard 2d mouse pointer no longer seems appropriate.

As much as we all want it, theres no way to make a 3d desktop compatible with everything unless its built as a desktop manager from the ground up. Trying to program to work with everything will be a nightmare

I am not an expert on Windows platform, so take it for what its worth. If I was to pursue the 3D desktop on Windows I'd be looking at Windows Remote Desktop protocol:
http://msdn.microsoft.com/en-us/library ... s.85).aspx

An example of its use is the well known VNC that delegates your mouse events and keyboard events to the remote Windows host. Now, in your case the "remote" host just happens to be the same as the local host which is perfectly fine from the IP protocol perspective.

There are also open source projects built on top of the RDP protocol that claim to work with all Windows systems, please see this:
http://www.tightvnc.com/
http://www.rdesktop.org/

I am not sure if this is the best route for you to pursue, but at the very least you know that it is feasible to construct your own client desktop on top of this protocol.

[druidsbane]

@druidsbane:
I'm afraid there is another problem in the code. There is a performance bottleneck on XGetWindowAttributes() functions. Please, see the profiler output below.
...
I'm wondering if we really need to constantly query the attributes of child windows? We'll probably need to test if this query is actually slowing down the rendering loop.

Good point. I think that I can optimize it to only perform the query when a window has an event associated with its ID, eg: mapping/unmapping; it shouldn't be too hard to implement. I'm still trying to fix the memory leaks but on the VM which is all I have access to this weekend since I'm travelling again has issues where it is reporting tons of problems inside the gl helper code I have and so drowns out the rest of the errors that you pointed out. I should also start cleaning up my code into separate files.

But its good that you found the renderDesktopToTexture function as you can see easily how I'm doing it now. If you ever do decide to intercept and redirect the mouse through another device then you can transform the windows here and forward the events using the main event loop. Also, if you're using something other than a mouse then you can even capture the mouse just for the application and use the other device like the hydra as a fake mouse

[TilliK]

Hi!

I don't know if this has been mentioned here before, but i found this on youtube, and it seems pretty good imho, http://www.youtube.com/watch?v=4JdjWhXrq68

[druidsbane]

I don't know if this has been mentioned here before, but i found this on youtube, and it seems pretty good imho, http://www.youtube.com/watch?v=4JdjWhXrq68

I definitely saw this on one of the earlier posts somewhere on this forum. Very impressive I'm glad we'll have a number of alternatives by the time the Rift is out!

[StreetRat]

@NickK: Thanks for that, didnt even know that existed. ill have a look at it when i get back to programming

[LeeN]

I've pushed the Hydra code (the left controller is used for head tracking, the right is just a floating box), and I've also added support for headtracking with OpenNI (Kinect) which I demonstrate in this video:
[youtube-hd]http://www.youtube.com/watch?v=qP9CqWGFRik[/youtube-hd]

You will need the sixense sdk:
http://sixense.com/linuxsdkdownload
I've had trouble with installing it or it only halfway installs, and so I made my code work with just having the sdk uncompressed in an adjacent directory to the vrx source.
By default the hydra code is disabled. I recommend using ccmake to configure the build:
# ccmake ../vr-x-wrapper
or just use cmakes interactive mode:
# cmake -i ../vr-x-wrapper
From there you can enable the hydra code.
and then build and install:
# make install
and start Xephyr
# cd release/bin
# ./server0.sh &
and run vrx
# ./run.sh
The run.sh script is really only necessary if you think the sixense sdk failed to install all the way.



The panel code is not done yet. I'm planning on integrating some of my old code with Hydra as a mouse, since I am practically rewriting it from scratch with the panel code changes I've been trying to make.


I also updated one of my computers to the latest Ubuntu and found that fullscreen (alt-enter) doesn't work! This used to work for me in Ubuntu 10.04 but apparently not in 12.4. From what I can tell, the window manager is either ignoring or not receiving events from my call to XChangeWindowAttributes with override_redirect flag. So I think I am going to have to create and destroy windows when I want to toggle going fullscreen.

[cybereality]

Looking good man!

[brantlew]

That is super cool. That is your head moving forward that causes the zoom - correct? Very nice.

[druidsbane]

Great work

[NickK]

@LeeN:
No more stereo cat in the interface? That's completely unacceptable! Just kidding.
Do you have any objections if I try to port your Hydra code to druidsbane's version?

@druidsbane:
I've cleaned up your code a little bit: reformatted indentation to get rid of tabs, replaced explicit calloc/free calls with std::vector, moved global rotation and position data along with walk() method into a desktop3D class, added a few comments, etc. I'm wondering if you'd be interested in merging the cleaner code into your repo?

I plan to break it into clean stand alone pieces so that I could add other hardware. Right now, everything depends on everything through global variables which is something I'm gradually trying to eliminate. If the code is more modular and if LeeN doesn't mind, we can enable Hydra and/or a virtual keyboard. Personally, I think it's better to work on different things and then share the code. If the architecture is clean enough, sharing of code between LeeN's and your repo will become easier.

By the way, in today's repo version window move operation doesn't work (before any changes I made).

[LeeN]

Thanks guys!

@brantlew, it is Z axis motion (I'm not sure that is considered zoom since traditionally zoom means narrowing the FOV) and yeah I was intentionally moving my head forward. You can actually see my beard if you look carefully at the dots on screen.

@NickK, yeah, no dubstep cat this time (LOL, I didn't notice that it looks like he is wearing stereo glasses). Friends of mine I met at Pandemic studios have started their own kickstarter game, Football Heroes, that's why I put that instead.
Anyway, I'm fine with you integrating the hydra code, I tried to keep it self contained so the code wasn't scattered everywhere. At some point I plan to integrate druidsbanes code or maybe move to his code base in general for doing these experiments, it really depends on where things go. My bigger goal is to develope an API/protocol for full 3D applications, and to integrate x windows applications into that.

[druidsbane]

@druidsbane:
I've cleaned up your code a little bit: reformatted indentation to get rid of tabs, replaced explicit calloc/free calls with std::vector, moved global rotation and position data along with walk() method into a desktop3D class, added a few comments, etc. I'm wondering if you'd be interested in merging the cleaner code into your repo?

I plan to break it into clean stand alone pieces so that I could add other hardware. Right now, everything depends on everything through global variables which is something I'm gradually trying to eliminate. If the code is more modular and if LeeN doesn't mind, we can enable Hydra and/or a virtual keyboard. Personally, I think it's better to work on different things and then share the code. If the architecture is clean enough, sharing of code between LeeN's and your repo will become easier.

By the way, in today's repo version window move operation doesn't work (before any changes I made).

I have no objections. I have made a quick change to use my iPhone as an orientation sensor and stream data since I can't get my Wii controller hooked up to my virtual machine. I'll ping you later today?

Any code cleanup much appreciated. I still can't get valgrind to work on my machines honestly, did you use the suppression file to ignore errors from other GL libraries? I didn't get that set up so maybe that's why. I'll also check the window move thing. I think I know why. I tried to cache the XGetWindowAttributes whic you mentioned is a hotspot. If I do that then I need to read in a move from XEvents, hehe. Good catch, I hadn't tried moving windows in my code since I was testing other things Really glad you're helping out! I still haven't looked LeeN's code but if we can get it to a modular enough point where we can share things that would be awesome, but for now I'd like to keep it relatively easy to just hack stuff in and see if we like it and delete quickly if we don't

[druidsbane]

At some point I plan to integrate druidsbanes code or maybe move to his code base in general for doing these experiments, it really depends on where things go. My bigger goal is to develope an API/protocol for full 3D applications, and to integrate x windows applications into that.

I agree with your last thought. I want something similar but my goal is to have a 3D world associated with things. For a first pass I want to try to keep something light, but if we have a modular system it should be easy to plug in: 3D apps or non-3D apps and solid desktop, as well as 3D world or plain simple one... etc... Another goal is to abstract some of the code to work on Windows and possibly OSX as well at some point... the simple fact that there are 3rd party VNC/Remote Desktop applications at least gives me hope we can get a reasonable desktop even if not fully hardware accelerated on the virtual desktop side of things.

[druidsbane]

Quick update. Thanks to your info NickK I fixed the window moving bug. I've also added a new project to the repository, it's a folder called IPhoneHeadTracker and basically it is a simple gyroscope for the iPhone. All it does is let you enter a local IP address where your virtual desktop is and hit go and it sends down the phone's orientation as a 3x3 matrix. One inversion later in the code and it is looking around The idea was inspired by brantlew's video on another thread and it does have the same periodic lag that he experienced, but I just wanted to see for myself what kind of latency there was. It works great and if anyone has access to the SDK they can compile and run on their own phones. If anyone's interested I can post a binary on my site for those with jailbroken phones that don't want to compile (ps: jailbreaking is completely legal according to US law at the moment). Below is a really ugly screenshot of the iPhone version (the iPad version does the same thing but ridiculously huge ).

[druidsbane]

@NickK: I've integrated your changes. I do have lots of cleanup to do but hopefully this gets it to a cleaner state where you're more comfortable working on your own components! Hopefully we can make it more modular in the near future so code sharing to or from other projects is easier, though to be honest I'd say it is easy even now since the code is tiny and mostly just a collection of functions that perform very specific tasks and easy to hack in stuff even if it is ugly

[brantlew]

@brantlew I'd recognise that jitter anywhere! It's the broadcom bluetooth chip. I had the same problem with my old Galaxy S.

If I was using a bluetooth mouse, and I did something online over wifi, the mouse movements would become jittery.

The chip in the galaxy s and most iphones (bcm4329) does bluetooth and wifi at the same time by rapidly switching between the two. I think your phone is probably scanning for bluetooth devices every couple of seconds, causing the stream to become jittery.

To fix it, just switch off the bluetooth (if that's possible on an iphone)

If that's not the issue, I'll look like a right moron

I haven't gotten around to testing this out, but you might try to turn off BluTooth and see if it improves.

[druidsbane]

@brantlew I'd recognise that jitter anywhere! It's the broadcom bluetooth chip. I had the same problem with my old Galaxy S.

If I was using a bluetooth mouse, and I did something online over wifi, the mouse movements would become jittery.

The chip in the galaxy s and most iphones (bcm4329) does bluetooth and wifi at the same time by rapidly switching between the two. I think your phone is probably scanning for bluetooth devices every couple of seconds, causing the stream to become jittery.

To fix it, just switch off the bluetooth (if that's possible on an iphone)

If that's not the issue, I'll look like a right moron

I haven't gotten around to testing this out, but you might try to turn off BluTooth and see if it improves.

Unfortunately that didn't work for me. I still get stutter. It might be related to reading UDP packets out of order, I haven't tried to ensure order at all, but more likely it is just a stutter on the phone.

[brantlew]

I tried both UDP and TCP (Nagel disabled) and got the same behavior. Very frustrating because otherwise, it's a fantastic tracker.

[druidsbane]

@brantlew: thanks for your help with this anyways, at least now I don't have to bother trying TCP

Everyone, I've uploaded a couple of YouTube videos to demonstrate this project embeded below The first video is from a distance and shows the iPhone for interaction and how it looks like on a laptop. The second video is a direct screencast so you can see all the colours and details up close.

[youtube-hd]http://www.youtube.com/watch?v=Ec0JgisJqGA[/youtube-hd]
[youtube-hd]http://www.youtube.com/watch?v=nglk96pxaLQ[/youtube-hd]

[brantlew]

Excellent work!

[cybereality]

Awesome work, druidsbane!

[android78]

That is totally cool and something extremely useful!
Just a shame that we can't do that with MS windows. I see that you currently have the depth rendered for crosseyed view, and assume it will be possible to flip for the rift? Also, how do you control the position of the window and your view in the windows world?

[LeeN]

That's very good Hasham!

Your stereoscopic 3d looks a lot better in the videos than it did in the screenshots for some reason. If you want to, you can tag youtube videos with "yt3d:enable=true" and those with a stereo monitor/tv or an LG Optimus 3D/Thrill can see the SBS videos in stereoscopic 3d, they can also flip the left and right or view it in anaglyph.

One idea I've had for the vr-x-wrapper was to allow users to tag windows or part of windows as being stereoscopic. This way people can watch 3d movies in vlc/mpc or 3d youtube videos.

I hope that the Rift will build into it the ability to inform applications when the user puts on the rift. This way they can be on their laptop and it looks like a normal desktop, and when they put on the Rift it changes to stereoscopic 3d.

[NickK]

I have no objections. I have made a quick change to use my iPhone as an orientation sensor and stream data since I can't get my Wii controller hooked up to my virtual machine. I'll ping you later today?

Any code cleanup much appreciated. I still can't get valgrind to work on my machines honestly, did you use the suppression file to ignore errors from other GL libraries? I didn't get that set up so maybe that's why. I'll also check the window move thing. I think I know why. I tried to cache the XGetWindowAttributes whic you mentioned is a hotspot. If I do that then I need to read in a move from XEvents, hehe. Good catch, I hadn't tried moving windows in my code since I was testing other things Really glad you're helping out! I still haven't looked LeeN's code but if we can get it to a modular enough point where we can share things that would be awesome, but for now I'd like to keep it relatively easy to just hack stuff in and see if we like it and delete quickly if we don't

I had the same problem with valgrind ??? records that are due to stripped down function names in nVidia's drivers. Judging by abundance of info on the web, they are either false alarms or nVidia's binary drivers have bugs (or both). I did not use the suppression file. Usually, I just supply a filename with “--log-file=<file>” valgrind command line option, then scrolled through that file and ignore all the ??? records. Look for the clean records with your own function names.

Also, the latest version works alright visually from the user's perspective but I'm not sure it is stable: there are some X errors showing up on the screen:

X Error of failed request: GLXBadPixmap
X Error of failed request: BadValue (integer parameter out of range for operation)
X Error of failed request: BadMatch (invalid parameter attributes)
X Error of failed request: BadPixmap (invalid Pixmap parameter)
X Error of failed request: BadDrawable (invalid Pixmap or Window parameter)
X Error of failed request: GLXBadPixmap

To reproduce: launch ibex and press “Application Menu” of the main desktop panel. Right after that the errors are printed in the console. Is it expected? The error messages first appeared in Sep 5 code. In Sep 4 code they were either not present or just not printed in the console.
In addition, when I try to run the profiler, I can no longer exit by pressing Ctrl-C. It results in a temporary freeze with data corruption on exit.

By the way, excellent demo videos. On my desktop I'm now consistently in the 150-170 fps range. It runs very smooth on a high power GPU.

[druidsbane]

Thanks everyone! I just wanted a quick way for the layman to understand what this is all about

@LeeN: I've tagged the video as recommended, seems under advanced settings you can just set it, no need for confusing tags (also, that tag is no longer in use, they require you specify LR or RL for the views, I just used a drop down ) I agree that it would be nice to label windows or parts stereoscopic, then you can just render the proper views for each and it would look 3D in your 3D desktop! Also that last suggestion about the Rift informing apps that it is on as an event or something would be great, the thing is, as an extra monitor, you'd really just need to query for which one the Rift is and display 3D on that, you can leave your regular desktop 2D on your old monitor by default (or let you render SBS for debugging purposes).

@android78: for the YouTube video I flipped it since it wasn't intended to be cross-eyed, can you let me know if that is the right way to do it? I always thought the right view was on the right and the left view was on the left Please let me know so I can change that if its wrong? This was a very quick proof-of-concept for 3D and i didn't actually test it out personally. I guess my NVidia 3D Vision kit that I could only use for 5 minutes at a time b/c it is so annoying will finally come in handy

@NickK: now that's dedication When I saw the thousands of errors I didn't bother with a suppression file nor to scroll that far down! Next time I will. In terms of those errors, I haven't checked to see which windows I'm mapping that cause those errors but from what I've read online it is almost expected that some windows will fail to map sometimes on NVidia and we need to catch those errors. That may have been older information and not apply but these errors seem harmless (also I wasn't printing them!). I'll look into the Valgrind issues at some point as well but I think you helped find the biggest bugs so far and we can work on cleaning up on exit later once the code has destructors in the right places. Right now I don't intercept the quit signal and cleanup or anything.

[druidsbane]

Just wanted to share my first quick and dirty attempt at barrel distortion in ibex for the Rift.

[druidsbane]

Quick update. I've gotten a simple symmetric barrel distortion done. Its harder than I thought to get this done in a pixel shader as I can't really find good documentation on performing the distort, especially with variable FOV for vertical and horizontal. Nonetheless, here is the first attempt that works. Toggle navigation mode then toggle this on and off using the B key (for barrel distort )

[LeeN]

druidsbane, after looking at your video in 3D, I've noticed that there is also something wrong with the stereoscopic 3d in ibex, I thought it was just the photos before. The problem is noticable when turning left or right.

I think the problem is that you need to move the i2 glTranslate to be before the glRotates, in fact it should be the very first GL_MODELVIEW transform or the last GL_PROJECTION transform.

The logic is that, with the current order of transforms it is translating the desktop left and right on the x axis in world space, but you should instead be moving the world left and right for the eyes in camera space.

I am going to over explain this for others not as familiar with OpenGL.

If you are transforming an object from one space to another, you write those transforms in opposite order in OpenGL.

So if you are transforming an object from : object space > world space > camera space
In OpenGL you would write the transforms in opposite order:
first world-to-camera transform
then object-to-world transform

So the problem with the code I think, is that the translation for the eye offset is mixed in the object-to-world transform instead of the world-to-camera transform, so the desktop is moving left or right relative to the world but not relative to the eyes. And I think really the eye translation should be the very first GL_MODELVIEW transform or the last GL_PROJECTION transform.

The stereoscopic code I have been using in vr x is similar to the link in the comments of the ibex code, it uses glFrustum to change the left and right edges to do "anamorphic" projection for the offset of each eye. I don't know what the rift display will utlimately be like but I have doubts that the eye will be in the exact center of each side of the display, so an "anamorphic" projection will probably be necessary as well to compensate for this offset.

[NickK]

Guys, I have some good news and some bad news.

Good news:
I've been able to get IBEX view orientation controlled by one of Hydra controllers. Basically, it works like iPhone in the demo video. I have not yet tried to enable full Hydra functionality but I intend to. My code is still in an early prototype stage and is used as a proof of concept to test feasibility. At this time it looks doable without major complications. I had to change most of LeeN's code but it was an excellent starting point nonetheless: it saved me a lot of time. Thanks, LeeN!

Bad news:
Wayland developers are considering to introduce XY global coordinates into the Wayland protocol. Remember all the 3D goodies of Wayland I was talking about? How it allows fully 3D clients (= 3D objects) because the protocol itself doesn't define a surface content. How Wayland doesn't allow clients to choose their global coordinates, so that the window manager had the luxury of rendering them as needed. How Wayland keeps clients isolated from each other so that their popups are grouped their own surfaces. All of these top advantages of Wayland for our application will be wiped out, just like that, if they change the protocol to interpret any client content as some global XY coordinates.

I've communicated with some of their developers. My goal is to make sure that they understand the impact of their decision on *any* 3D desktop environment. I don't think there is much more that we can do since 3D desktops are a niche application and we don't even have RIFT hardware out on the market. Let's just hope that W won't degenerate into another X.

[druidsbane]

@LeeN: Thanks for that very much. Since reading your message I moved that portion up as you suggested, but after looking at the video in 3D and playing around I feel that my camera code is still wrong in general. When I rotate it feels like objects stretch towards me and get larger rather than stay in place and get smaller... weird. I've also been playing around with different orders for rotation and translation of the world into camera space but I end up pivoting around the origin or other points rather than myself. You have given me some great points to ponder and I'll try to make sure the 3D is really up to snuff!

@NickK: That's great. I was about to ask if we were using the FreePIE library that I keep reading about on the forums before I looked it up and discovered it was C# only. Probably dependent on Windows API's as well. Still, excellent work