VR/AR Windows "Desktop" development

[LeeN]

At this point I don't have a plan to start a company or to make money. I'm mostly doing this because I think it should be done and as far as I know no one is doing it the way that I am. If some one wants to start a business that creates the first VR OS sometime soon that would be awesome, there would be no more reason for me to continue. There is no urgency for me.

when I say hand in the air, I literally mean my hand and not my elbow, mostly it's because if you are surrounded by windows you have to turn your wrist and it's much easier with your hand off the table. Maybe if I reintegrate it later you'll have a better understanding and maybe even have ideas how it can work better, but for now I'm interested in a virtual tablet.

Even if you don't like tablets, they are dominating the market and many people believe they are easier, more intuitive to use than traditional computers. Even in John Carmacks talk at Quakecon, he was using an iPad for notes, in fact that image of him using it gave me the idea.

If you have ideas you'd like to try, feel free to use my source code to try things out.

[druidsbane]

While researching various solutions to the bugs I still have I came across this: http://www.opencobalt.org/ Pretty cool concept.

I like the ideas you guys are bouncing around of using tablets as well as those suggesting virtual laser pointers. I am personally setting myself up so I can try using the Leap Motion to good effect. We'll see how each of these pan out.

I had made much progress in my spare time while on vacation and now that I'm back I'm hitting rather large roadblocks with the native nvidia hardware. So while I have SBS rendering and the virtual desktop mostly working on a VM, on a native Linux install it is pretty bad (and even slower than my VM!) I'm liking more and more your solution of not acting as a window manager at all as that gives more flexibility to work on the more interesting parts of the project than window management which has been done and redone a million times!

Please keep up the good work LeeN and others and I'm sure we'll each get there in our own way to represent the virtual desktop/workspace we all want

[LeeN]

Also, one thing I forgot about last night is that at least for X windows, windows are detached and you have no easy way of detecting that a popup window is intended to be on top of another. This make free space windows difficult to do normally. My idea to overcome this is to bundle windows created by the same process together into a panel, in the same position and order in Xephyr. The problem with doing this is, now you have the 3d movment of panels and the 2d movement of windows in the panel to deal with, having a virtual tablet solves this issue as you can now attach panels to the tablet to move them around in 3d and then use the touch screen or mouse and keyboard to manpulate them.

[druidsbane]

Finally got it to work for the most part reasonably fast on both a virtual machine and a real desktop. I took a screenshot from the VM and you can see it is going around 40-60fps. On the desktop it is 90-100fps. It would be completely useable except for the fact that I'm currently unable to grab the position of the xpointer. I'll do that next and render it on the view so it is easy to interact with the windows. Other than that it works alright using XFCE as you're doing LeeN (it misbehaves badly on any compositing window manager), and even OpenGL windows render fine through it on my nvidia-based desktop which was surprising. Next up is load up some 3D models and figure out a way to move around that world and keep the virtual workspace as it is

[LeeN]

druidsbane, that's great the performance has improved, and I really like the fact that OpenGL works through it. Is it full OpenGL or is it indirect or software rendered? Xephyr can only be used indirect which really doesn't matter since it is software rendered , you probably saw how bad blender performs in my video.

Nice Job!

[druidsbane]

druidsbane, that's great the performance has improved, and I really like the fact that OpenGL works through it. Is it full OpenGL or is it indirect or software rendered? Xephyr can only be used indirect which really doesn't matter since it is software rendered , you probably saw how bad blender performs in my video.

Nice Job!

I figured the performance issues in the video were due to the recording process. But I guess not since I think you did a handheld recording right? The OpenGL is directly rendered by nvidia. I tested Blender just now and it is quite fast I just used the game engine thingy and it was locked at the 60fps I saw before I started the manager. Also, the fps at that point dropped to around 80fps. Still alright for our purposes. That is probably the only good thing about compositing is the performance, though honesetly I expected much more from the desktop with what was once a great card (GTX 480) than my virtual machine on a tiny macbook air. I don't know how much further I'll be able to take this but I want to get code out soon My dream honestly is to have the compositing manager run inside of a nice game engine like the Unreal engine or something so we can do some even more interesting things

Edit: to clarify, the 60fps is for Blender, the 80fps is for the compositing manager.

[NickK]

@LeeN:
I didn't mean to tell you what you should or should not do. If you believe that tablets are the best input device, please, work on it first. The only message I tried to convey is that I'm not optimistic about its applicability to VR. But I may be wrong.

Carmack's example doesn't apply here because he wasn't actively using his tablet to control a complex environment. In fact, he wasn't actively using it at all, just scrolling and reading. And it's not really that I dislike tablets - I use my Thinkpad Tablet to keep notes, check email, do basic sketches that I need to exchange over email. Yet, despite having a keyboard attachment for it, I don't view it as a good device for productive use. I also have a convertible tablet with Win7: a larger, more powerful tablet that I use as a travel computer. If I disliked tablets I wouldn't be spending my own money to buy all this hardware.

If you can somehow solve the usability issues of tablets in VR environment and find a way to make them as productive as a mouse+keyboard then you deserve to have a patent on it because it is a *highly* non-trivial problem. And everyone would benefit if you can solve it for them.

@druidsbane:
I've seen Open Cobalt before. Somehow I feel that they started out ahead of times. But what they do is one of likely ways we'll communicate in the future, where you simply enter the 3D world of another person to talk to or work with him/her. Voice and video can be integrated into the VR GUI in a seemless way that you don't even need to launch skype or any other application. That said, they have a rather questionable choice of a programming language - Squeak. For a wide adoption, I suspect a widely used programming language would have been a better choice.

[druidsbane]

@druidsbane:
I've seen Open Cobalt before. Somehow I feel that they started out ahead of times. But what they do is one of likely ways we'll communicate in the future, where you simply enter the 3D world of another person to talk to or work with him/her. Voice and video can be integrated into the VR GUI in a seemless way that you don't even need to launch skype or any other application. That said, they have a rather questionable choice of a programming language - Squeak. For a wide adoption, I suspect a widely used programming language would have been a better choice.

I agree about the language, however Python had just been released when this first started and one of the authors of Open Cobalt is Alan Kay, one of the authors of Squeak It is an old project but the ideas of dynamically updated and editable worlds and so on are definitely the future and while I don't think this project will be it, we have inspiration from so many other places (especially sci fi) to help guide us as well!

Also, another small update, I figured out how to hide the mouse cursor and render my own ugly one (small red triangle you can see on the console) so at least now I can fully interact. Next up figure out a toggle to interact with the world and interact with the desktop. I'm still unable to intercept events fully just yet, only acting as a pass through.

[druidsbane]

Added library to load OBJ files exported from Blender, loaded in a test monitor and superimposed the desktop on that. Next up figuring out how to rotate the point of view using a toggle instead of manually coding it so I can try to simulate using the headset.

[LeeN]

I didn't mean to tell you what you should or should not do. If you believe that tablets are the best input device, please, work on it first. The only message I tried to convey is that I'm not optimistic about its applicability to VR. But I may be wrong.

Carmack's example doesn't apply here because he wasn't actively using his tablet to control a complex environment. In fact, he wasn't actively using it at all, just scrolling and reading. And it's not really that I dislike tablets - I use my Thinkpad Tablet to keep notes, check email, do basic sketches that I need to exchange over email. Yet, despite having a keyboard attachment for it, I don't view it as a good device for productive use. I also have a convertible tablet with Win7: a larger, more powerful tablet that I use as a travel computer. If I disliked tablets I wouldn't be spending my own money to buy all this hardware.

If you can somehow solve the usability issues of tablets in VR environment and find a way to make them as productive as a mouse+keyboard then you deserve to have a patent on it because it is a *highly* non-trivial problem. And everyone would benefit if you can solve it for them.

Part of the reason I am doing this is to explore different input methods. I'm not saying I am not going to try the 'red beam' wiimote kind of control of windows either.

Really the biggest challenge I think, is figuring out how to solve that windows are detached from each other. I found a few ways of determining a window was intended to appear on top of / relative to another, such as looking at it's indirect_override flag and there is an X atomic/property that I found using xtrace which is also sometimes used to indicate a window is to appear over another, but I found these are not 100% in fact they are probably not even 80%.

That is why bundling windows of the same process into a 2D panel, will allow them to be positioned relative to each other (in X server 2D coordinates), and then being able to move the panel around in 3D. The problem with this is that now you have windows that can move in 2D on the panel, and panels that move in 3D, and how to differentiate between moving windows and moving panels. (note, the panels are not the same thing as the virtual monitors in my current source code)

The Carmack reference wasn't so much how or why he was using it, I was just thinking about this issue when I saw him using the tablet when I thought that these panels are like tablets, and then I thought a physical tablet like input device could be a good fit for solving this. And I think it would be a device that would be appealing to the most people.

And this goes both ways, this is something that could enhance a tablet experience. Tablets have small displays to make the light and small, in VR you can then positions windows around you.

Another way of solving this, is with Kinect or Leap motion, and you wave your hand through a panel to select it (faster than alt-tab) and then you can use the mouse and keyboard in that panel. And if you want to move that panel, you clench your fingers and it attaches to your hand, and then move it to where you want it and then release your fingers to set it. This would also be something that can be done with Hydra, but again Hydra feels awkward because it isn't designed to be used along side a keyboard and mouse.


Also, I found the problems with Hydra again. Hydra is good if you always point forward and don't turn to large degrees, but once you starting turning beyond about 90 degrees, things start behaving strange. I found this when I was trying to mount the Hydra to the HMZ-T1, it felt nature to mount it side ways across my head but then rotations became distorted. There could be something else going on though with the rotation matrix that I get from the SDK. I'll probably release the source code before I can figure out if there is something else going on with the rotation matrix.

[NickK]

@druidsbane:
Your screenshot includes "wayland demos". Are you working on Wayland? Anyone from their dev list?

Another question: you mentioned that your VR desktop allows other applications to render OpenGL. This puzzles me a bit. Suppose that I have my own OpenGL code that uses my own pixel shader. I compiled the shader in my code and send it to the GPU. How will your desktop be able to intercept my shader?

@LeeN:
Unfortunately, I can't seem to come up with any reasonable suggestion how to deal with the detached windows. The only one I can think of is to develop a new X or a W server, which obviously is a challenging task.

Do you happen to have any *good* links that describe how to construct a simple barebone X server that would be compatible with Nvidia's or ATI's driver blobs? My experience with the open source driver stack has not been very good. Nothing against their developers - I understand that they work for free.

@All:
Rumors have it that Nvidia allocated resources to evaluate if it's worth providing binary support for Wayland. Just please don't ask Nvidia questions on the W mailing list. W developers seem to have Torvalds' attitude towards driver blobs.

[druidsbane]

@druidsbane:
Your screenshot includes "wayland demos". Are you working on Wayland? Anyone from their dev list?

Another question: you mentioned that your VR desktop allows other applications to render OpenGL. This puzzles me a bit. Suppose that I have my own OpenGL code that uses my own pixel shader. I compiled the shader in my code and send it to the GPU. How will your desktop be able to intercept my shader?

...

@All:
Rumors have it that Nvidia allocated resources to evaluate if it's worth providing binary support for Wayland. Just please don't ask Nvidia questions on the W mailing list. W developers seem to have Torvalds' attitude towards driver blobs.

Hey! I tried working on wayland first and failed to make much progress. There seems to be even less documentation than X11's extensions and given the poor performance and lack of driver support I gave up for now. I figured just write the main code in OpenGL and make it modular and someone else more familiar than I can port it to wayland if it is still useful at that point!

In terms of OpenGL I don't need to do anything really, the NVidia driver just natively supports redirecting OpenGL rendering to an offscreen buffer. I will ensure that I don't try to intercept fullscreen apps so they can run at full speed though.

Lastly, in terms of wayland support for NVidia binary drivers, that is something being considered according to Phoronix but there is nothing to report. That's why I figured if I wanted to hit the 60fps needed on today's hardware then X is the most practical choice for now.

Also, I'm working on getting my code to a point I can release in whatever little free time I have

[LeeN]

@LeeN:
Unfortunately, I can't seem to come up with any reasonable suggestion how to deal with the detached windows. The only one I can think of is to develop a new X or a W server, which obviously is a challenging task.

Do you happen to have any *good* links that describe how to construct a simple barebone X server that would be compatible with Nvidia's or ATI's driver blobs? My experience with the open source driver stack has not been very good. Nothing against their developers - I understand that they work for free.

Even writing a new x server will probably not help. There is no standard or requirement that an application let the x server know that a window is intended to be shown over another window, except that the window is created with 2D coordinates that overlap the other window. Detecting this intention is not 100%.

Panels are pretty much the only way I can think of, and that is likely what I will focus on next.

[NickK]

I figured just write the main code in OpenGL and make it modular and someone else more familiar than I can port it to wayland if it is still useful at that point! Also, I'm working on getting my code to a point I can release in whatever little free time I have.

I was thinking to do the same with LeeN's code: to split OpenGL away from the desktop management so that I could plug in another OpenGL engine. Then, another issue came up related to all the input hardware that needs to be supported: head tracker, tablets, Hydra, etc. Now, I'm thinking that a hardware abstraction layer may also be useful. Otherwise, debugging will become unmanageable as the code grows.

Fortunately, LeeN was able to collect all the code into a small prototype: it's easier to start with it. I am wondering if your first prototype can also be made relatively compact? Can your code be combined with LeeN's?

That's why I figured if I wanted to hit the 60fps needed on today's hardware then X is the most practical choice for now.

This is one of the things that I want to be careful about. Carmack said that 60Hz would not suffice for a visor with head tracking. The problem is that people move their head very quickly. If you render at 60 fps only, any sharp head movement will result in visual artifacts. Over time these artifacts will lead to headaches and users will not want to use your interface any longer. Ideally, we need a solution that in the future can be scaled to 120 Hz. I believe the commercial RIFT is targeting 120 Hz as well.

Even writing a new x server will probably not help. There is no standard or requirement that an application let the x server know that a window is intended to be shown over another window, except that the window is created with 2D coordinates that overlap the other window. Detecting this intention is not 100%.

I believe that XCreateWindow() tells me the parent window. If I understand it correctly, an X server knows the whole tree of parent-child dependencies. Would it not be sufficient to solve the problem if I can develop my own X server?

Window XCreateWindow(display, parent, x, y, width, height, border_width, depth, class, visual, valuemask, attributes)
Display *display;
Window parent;
int x, y;
unsigned int width, height;
unsigned int border_width;
int depth;
unsigned int class;
Visual *visual
unsigned long valuemask;
XSetWindowAttributes *attributes;

[druidsbane]

NickK: the code should hopefully be short but right now it is definitely not clean as I'm trying out do many different things. When I release something it shouldn't be too hard to follow.

The 60fps is the best the hardware will support in the initial implementation. In terms of smoothness, if I can't hit120fps for the compositing but the hardware supports it, I'll just update the desktop components every other frame but render the scene at 120fps and do the proper head tracking

Lastly, have you looked at XQueryTree? That coupled wit XGetWindowAttributes should be enough to figure out the parent of a window. The attributes contain a root window for each window, I expect that to be the parent. If not, the other choice is to get the child windows of the root window, and perform XQueryTree on each of the children returned to build up a tree that way, but hopefully you don't have to do that. Also be sure to grab the X server so nothing else pops up while you are processing the tree.

[LeeN]

I believe that XCreateWindow() tells me the parent window. If I understand it correctly, an X server knows the whole tree of parent-child dependencies. Would it not be sufficient to solve the problem if I can develop my own X server?

Window XCreateWindow(display, parent, x, y, width, height, border_width, depth, class, visual, valuemask, attributes)
Display *display;
Window parent;
int x, y;
unsigned int width, height;
unsigned int border_width;
int depth;
unsigned int class;
Visual *visual
unsigned long valuemask;
XSetWindowAttributes *attributes;

This is not the same thing. When a window is a child of another window, it is cropped by the parent window, it never is outside of the parent window. This is mostly used for creating input regions, and drawing regions with in a window.

For pop up windows, like tooltips, menus (drop down stuff), context menu, etc, they can be bigger and exist outside of the window they are intended to show on top of so they use 'root' as parent.

[NickK]

Guys, what about this crazy idea? For every application started from VRX we create an individual window manager (WM), while all these WMs are managed by the global window manager (GWM). Every popup window spawned by an application will still reside within the same WM as the main window. Therefore, if we have our own GWM/WM designed specifically for VRX, we'll know which windows need to be grouped. Individual WMs can be developed small enough to avoid hogging resources.

[druidsbane]

It's interesting but I'm actually trying to reproduce individual desktops as virtual monitors in a virtual world so everything just works as usual The window manager just does its work as usual.

[NickK]

It's interesting but I'm actually trying to reproduce individual desktops as virtual monitors in a virtual world so everything just works as usual The window manager just does its work as usual.

What if I don't want to have virtual monitors or panels and just want to have windows around me in 3D space. And I want to be able to move them around freely in 3D without having artificial constraints on their location. Would that be possible with your implementation?

[LeeN]

I think if I understand what your are saying druidsbane, it is basically virtual monitors similar to what vrx is doing except with one desktop per virtual monitor instead stretching one desktop across the virtual monitors. I tried to do that with Xephyr but I couldn't seem to get the windows to move from one desktop to another.

NickK, that is not that crazy, what you are saying sounds similar to what I've referred to as panels, your idea is probably different but it effectively does the exact same thing. Basically all windows from the same application will be in the same panel, using 2D coordinates from the x server and so will appear on top of each other correctly. That is also the primary reason I like the idea of using a 3d tracked tablet input device to attach to a panel and you can use the touch screen to interact with the 2D windows in a panel, then you can place a panel where you want it in 3D space and detach it from the tablet.

[LeeN]

Also after playing with PTAM, it turned out to be kind of a dud, at least for head tracking, I talked more about it here:

http://www.mtbs3d.com/phpBB/viewtopic.php?f=138&t=15312

[druidsbane]

I think if I understand what your are saying druidsbane, it is basically virtual monitors similar to what vrx is doing except with one desktop per virtual monitor instead stretching one desktop across the virtual monitors. I tried to do that with Xephyr but I couldn't seem to get the windows to move from one desktop to another.

That's correct, virtual monitors like vrx. There is no reason to not make them 3D except I'd have the exact same problem as you have right now finding parents of windows. Since the windows are rendered separately I did play around with giving each its own depth or moving them around (eg: spriral, or stack) but that wasn't very useful for the same reasons you mention. Also, my goal from the outset has been to simply virtualize the desktop as it exists with whatever the user is comfortable with.

In any case, I have a version of the code ready to at least play around with if anyone is interested. Very raw, but I've attached it below. I've also attached a screenshot taken with my phone as this was running on my desktop not the virtual machine and I was too lazy to write the code to save out screenshots I've added a simple skybox to demo a simple world, there is still no ground and I've disabled rendering the monitor b/c it kills the framerate for some reason. The screenshot shows that I have around 85fps with blender running on my setup. There are many inefficiencies and the code needs much work so I've attached it here rather than github since I didn't get time to upload so anyone can play around with to start.

To compile you can either do it in Eclipse or from cmake. I have a readme that shows the libraries I had to install on Ubuntu to get it to work. I recommend making a build directory below the code directory and running:

mkdir build
cd build
cmake ../ibex-20120903
make install
./ibex

To run, just launch it. Do it from a commandline for two reason - 1) it crashes on startup sometimes so you might need to relaunch till it starts, 2) when you are switching to navigation mode I haven't disabled the keyboard from the other applications so it types both in the applications (hopefully terminal is up so it gets it isntead of your important docs) and in the manager. On launch it will take your first desktop and composite it twice. It you can use the mouse and keyboard to interact as usual and the FPS will be printed to the console. Press: ctrl+shift+y to toggle navigation mode or desktop mode. In navigation mode you can use W to move forward and S to move backwards and the mouse to look around and change the walking direction on the ground plane.

(Edited for typos)

[NickK]

@druidsbane:
Doesn't seem to work for me. I'm getting white space instead of background, see below.
Other symptoms:
+ Right clicking works to bring up a popup menu.
+ I can guess that my gnome panel is at the top and I can blindly open its menus which will show up on white background.
+ After I click W and S many times, a grassy background shows up. After that any mouse movement results in a crash.
+ After the crash my mouse pointer works in X but it doesn't generate any events, so I can't click on anything.

My system: Ubuntu LL, GTX 9800+, latest nvidia prop drivers.

[druidsbane]

@druidsbane:
Doesn't seem to work for me. I'm getting white space instead of background, see below.
...
Other symptoms:
+ After I click W and S many times, a grassy background shows up. After that any mouse movement results in a crash.
+ After the crash my mouse pointer works in X but it doesn't generate any events, so I can't click on anything.

One thing, I'm also running XFCE with no effects turned on. NVidia seems rather flakey that way and I haven't figured out how to detect and turn that off. On the VM it works fine with the Unity 2D if i remember correctly but not with Unity obviously, on NVidia neither of those options work. Thanks for trying it out though

[druidsbane]

I finally got a chance to update the FOV code for it and set up a public repository. It can be found at: https://bitbucket.org/druidsbane/ibex (or in my signature). Also, performance has improved to around 60fps in the virtual machine, and is still around 90-100 on my desktop. I have some ideas that can bring that up to around 300fps when I'm done.

edit: fixed screenshot!

[NickK]

@druidsbane:
I've managed to install it on another Ubuntu box with Ubuntu PP and XFCE as you requested. Looks fantanstic! I'm getting about 150 fps on my oldie GTX 9800+. With youtube video playing and crazy rotation around it drops to 125 fps. Looks already sufficient for RIFT.
The only gripe I have now is that the desktop is too small, see a screenshot below. Is there any way I can zoom in and out?



If the screenshot doesn't show up, please click on this link to see the image:
http://s15.postimage.org/etv263b8a/Scre ... _59_00.jpg

[druidsbane]

@druidsbane:
I've managed to install it on another Ubuntu box with Ubuntu PP and XFCE as you requested. Looks fantanstic! I'm getting about 150 fps on my oldie GTX 9800+. With youtube video playing and crazy rotation around it drops to 125 fps. Looks already sufficient for RIFT.
The only gripe I have now is that the desktop is too small, see a screenshot below. Is there any way I can zoom in and out?

I'm glad to hear that the FPS is so good.

In terms of zooming in and out you just need to press: CTRL+SHIFT+Y then use the A and W to walk around and aim with the mouse.

If you mean even while close up it isn't filling screen, that's because of the FOV so it will look weird on a desktop monitor. Next up I need to smooth out the motion with the framerate instead of using fixed increments per frame. Also, I'm rendering only the primary desktop, I need to see if I can get more than one on there, we'll see. I want to also render the desktop only once instead of once per eye. I also think I'm not using framebuffers efficiently. Lastly, the goal is to integrate the OGRE engine or something else to actually load in models and a level of some sort. If you don't mind testing I'll keep posting

Ideas welcome, but I hope the basic idea is clear that I'm trying to push for, virtual space with a nice environment of some sort but same desktop as usual.

[NickK]

I'm glad to hear that the FPS is so good.

In terms of zooming in and out you just need to press: CTRL+SHIFT+Y then use the A and W to walk around and aim with the mouse.

If you mean even while close up it isn't filling screen, that's because of the FOV so it will look weird on a desktop monitor. Next up I need to smooth out the motion with the framerate instead of using fixed increments per frame. Also, I'm rendering only the primary desktop, I need to see if I can get more than one on there, we'll see. I want to also render the desktop only once instead of once per eye. I also think I'm not using framebuffers efficiently. Lastly, the goal is to integrate the OGRE engine or something else to actually load in models and a level of some sort. If you don't mind testing I'll keep posting

Ideas welcome, but I hope the basic idea is clear that I'm trying to push for, virtual space with a nice environment of some sort but same desktop as usual.

I wasn't using it correctly (without CTRL+SHIFT+Y), now W and S work alright for walking around.
I do have other issues though:
+ Left mouse click is very flaky, it registers a click on "Applications Menu" panel but the drop down menu doesn't appear.
+ Sometimes I need to click left mouse buttom several times before a single click gets through.
+ I can't move windows around with my mouse.
+ Left click doesn't seem to be aimed correctly. Sometimes I click on a title bar and it open a menu item that is located underneath the title bar.
+ Some of popup menus (right mouse button) come out OK while others come out blank with no menu items.

Have you encountered these issues as well or is it my system misbehaving?

[druidsbane]

I've definitely noticed the issues you mentioned. Moving windows works for me when not in navigation mode, but popups and some new windows like context menus sometime have issues. Also, with the XFCE terminal I've notched it takes a while to start up and creating new tabs is problematic.

I intend to split my time between new feature on the fun tasks and the boring fixes of mapping windows and inputs. As it is open source patches are always welcome also, I've updated it slightly so motion is much smoother and you can strafe with the A and D keys as well!

[Fredz]

If the screenshot doesn't show up, please click on this link to see the image:
http://s15.postimage.org/etv263b8a/Scre ... _59_00.jpg

[druidsbane]

Thanks for trying it out guys. I just checked in another update to make it not crash randomly on startup and render the desktop properly to a texture. Also the framerate on my machine has jumped from 150 fps to 250 fps with this latest change.

I figured out that I'm not picking up the mapping/unmapping events of windows which is why menus aren't rendering right. I am struggling to figure that one out but I'm pretty sure that once that's fixed many of the problems you mentioned will be gone. I know it's related to that b/c if I open a menu and mouse over it, the first time it reacts and highlights, the second time it is static like I'm using a stale pixmap from last time. Obviously it's been remapped but I haven't picked it up. Once I fix this I'll post another update then work more on the world aspect.

[NickK]

Thanks for trying it out guys. I just checked in another update to make it not crash randomly on startup and render the desktop properly to a texture. Also the framerate on my machine has jumped from 150 fps to 250 fps with this latest change.

Interesting: my frame rate barely improved. I've downloaded and compiled a fresh version and got 150 fps. I guess different graphics cards have different symptoms to the same problem.

I've updated it slightly so motion is much smoother and you can strafe with the A and D keys as well!

Whatever you did with this update fixed all the issues I had:
+ Mouse clicks work fast, with no errors or redundant clicks.
+ “Application menu” opens up correctly.
+ Popup menus are no longer blank.
It looks like a major improvement to me. The segfault is also gone. Excellent work!

I figured out that I'm not picking up the mapping/unmapping events of windows which is why menus aren't rendering right. I am struggling to figure that one out but I'm pretty sure that once that's fixed many of the problems you mentioned will be gone. I know it's related to that b/c if I open a menu and mouse over it, the first time it reacts and highlights, the second time it is static like I'm using a stale pixmap from last time. Obviously it's been remapped but I haven't picked it up. Once I fix this I'll post another update then work more on the world aspect.

I got the same problem with highlighting. So far it seems the only artifact that I can notice. If I see anything else I'll let you know.

As it is open source patches are always welcome

I will need to get familiar with your code first. Right now it looks like an assembly of disjoint codes: GLM from GLSDK, framebuffer code, and X code. Lots of commented out code and indentation all over the place. It's very hard to follow what's going on. A little cleanup may be needed to avoid hidden bugs: they'll be very hard to catch once you start growing the code by adding new features.

LeeN's code is pretty good. I've already figured out how to add new key functions to move virtual monitors closer and farther away. I believe it'll save your time in the future to clean up the code as well while it's still small.

@druidsbane and @LeeN:
Can the framebuffer implementation be used to separate windows in 3D? Or it maps the entire desktop into 1 main window only?

Another thought: Do we really need to map the existing 2D desktop into separate 3D windows? Instead, the 2D desktop may be displayed as a single virtual monitor. In addition, we can provide an alternative 3D panel+terminal that allow users to launch applications into 3D and control hardware to use: head tracker, Hydra, tablets, etc. The 3D panel will control if and how that application maps to 3D (e.g. using LeeN's 3D window container he calls a panel). Regular 2D applications that have issues with popups will be constrained (as a group of windows) to reside within the main projection of the desktop window. There is a small inconvenience to the user but it is not significant. What do you think about it?

[LeeN]

Nick, very few programs don't have popup windows of some sort, the most common popup probably being menus. So in that way, just about all of them will not work right in 3d. You can see all the problems I had in my early videos.

A virtual 2D monitor will work and while it does have some value my interest is in 3D and figuring out how things will work in it.

I haven't had much time to work on vrx but I have come to the conclusion that future versions of vrx will not be compatible with window managers, so I've searched for and found a really basic WM to learn how to replicate their functionality.

[druidsbane]

@LeeN: do you think the window manager protocol stuff might help? I've been looking through lots of code and many of the compositing managers also set "WM_HINTS" or something like that so they can get some extra hints. I didn't delve too deep, but you're right from before that not enough info exists from the raw X11 queries. Maybe these hints help. Interested in seeing where this goes. Maybe Wayland fixes some of these issues, maybe not. I don't know if it can for the X11 compatibility layer. If you can somehow get the PID of the XID, then maybe you can group them that way, at least you'll know which app group it goes with and just pick the biggest window to center it on or something

@NickK: I'm guessing the framerate hasn't changed due to the fact that my hardware is different from yours and so the texturing stage makes no difference. As I mentioned, on the VM it is actually slower, though as you pointed out the results are more correct. In terms of the code, you're right, it is a HUGE mess, lol. I'm embarrassed, however I just wanted to get it out there and it is still a prototype. Once I get all the mapping issues fixed I can get rid of all the dead code and pull things into the right files, I promise

Lastly, small extra update today to handle more mapping/unmapping of windows which fixes new tabs in the XFCE terminal for me. I do know that if I can capture all of them I'd be fine b/c I hacked my code to always map every time and while it worked and fixed the menu issues it was extremely slow. I just want to have a conversation with one of those compositing manager authors at some point and pick their brains on this, probably fix everything in a few minutes

[StreetRat]

You guys have made a lot of progress on the unix version, but ive got next to no where on the windows version. Partly because i get easily sidetracked for days on end.
Anyway, it seems Windows wants to block you at almost every step.

I am using VB.net with XNA (directx) so it does have some limitations, my python, OpenGL isnt up to scratch so im not sure how to convert some parts.

Progress
I have got a screen showing within directx of an app, no movement or manipulation yet.
Im using PrintWindow, works most of the time but its not guaranteed to work with all windows.
Then comes input.
Using windows post/send messages isnt guaranteed to work either, and when it does you need to access the right textbox or button for the input to work, cant just send it to the main window and hope it works. I have away around it, but not sure if itll work in 3d.
Then theres the menus to deal with.
Most apps have a proper menu system, but some (like ms paint) have their own style. No idea how to interact with them.

On the menus though, not sure how you guys are dealing with them, but if need be, i thought about recreating the whole menu structure within the desktop app, wont look as good but might work. That way the original menus are never touched, just simulated.

At most, under windows it looks like youll only get about 80% of the apps to work.
Something just to navigate explorer is doable, i have parts of this in 3 different apps, just need to combine them

If anyone knows anything about windows and wants to chime in with ideas, please do.

[LeeN]

Someone posted this on youtube:
[youtube]http://www.youtube.com/watch?v=4JdjWhXrq68[/youtube]

The camera pivots weird (and is kind of annoying after a while).

But this goes back to what I said before, the challenge is in full 3D windows, virtual displays are not hard to do.

[NickK]

@LeeN:
I have been communicating with some Wayland developers to see if I could get any advice. Some of them are very helpful. I can share contacts if you'd like.

The message I got was this. They are also working on X compatibility component, called XWayland, to allow older X clients to run on W. And they've run into the same problem with popups that we are dealing with. There is a code full of hacks in "src/xwayland" directory of Weston code that attempts to cover most of these cases. It doesn't work 100% though because X clients can poke around other clients' windows and can also use absolute coordinates bypassing a window manager. XWayland component needs this window grouping because it converts an X application with all its popups into a single Wayland "surface" data structure.

In Wayland protocol clients are completely isolated and don't get access to absolute coordinates. That's how they enable a demo with rotating windows: W compositor (Weston) has 100% control over how that window is rendered, so it can choose to render it rotating. Any spawned dialogs or popup windows are all part of the same client's "surface" data structure that the client must populate. Think of the "surface" as a data storage for pixels or a collection of textures owned by the client. I also believe it is similar to what you called a "panel": a collection of graphical data of all windows of the same application. The client can only specify where the popup pixel data starts within the "surface" but doesn't choose how to render it.

In summary, it sounds like what you are trying to accomplish with panels and window groups is very similar to what W developers put into the Wayland protocol as a "surface". So, if they've already covered most of the well-behaving X clients, it may be worth taking a look at src/xwayland to see if we can reuse that code. Hopefully, this info can be helpful.

@StreetRat:
We are having a hard time solving our problems with an open source OS and a stable API like OpenGL, while you are taking on an even bigger task of dealing with a closed source system with an unstable API like Direct3D. Perhaps, it's time to switch away from the dark side?

[StreetRat]

@LeeN: Sweet, looks like progress is being made in one way or another

@NickK: Once i know what im doing, converting to OpenGL is part of the plan. As for converting to a unix platform, ive tried too many times, im mainly a windows guy
Although the advantage of DirectX is Nvidias 3d vision works with it for those that dont have the Rift, Give it TrackIR support and it could be interesting.
As were all finding out, baby steps first.

[druidsbane]

Once i know what im doing, converting to OpenGL is part of the plan.

I would advise against that on Windows simply because you are most likely to get the best performance from DirectX for something like this. The Windows compositing manager (DWM, http://en.wikipedia.org/wiki/Desktop_Window_Manager) is accelerated using Direct3D in Vista and higher, so your best bet is to get the window surfaces that way. I was looking into that as well as a future direction for Ibex but that depends on many other factors you mentioned including passthrough on inputs and so on. I'm nearly done with all that on Linux though I fixed the remaining rendering issues I had and am now working on getting the proper mouse cursor bitmap and associated events so it is a complete reproduction of a desktop.

Good luck and I'll definitely share what I can if I get more info.

[LeeN]

Nick, I already have grouping of windows into a panel. Where I am currently at right now is dealing with mouse input, which is why I'm finding I need to do my own window management because current ones like to control focus and stacking order, which are things I will need to control.

What I'm calling panels are in many ways like 'desktops' in X, except each application gets it's own. I also want to add support for merging them and breaking them apart.