VR/AR Windows "Desktop" development

Post by **cybereality** » Wed Oct 10, 2012 8:46 pm

I was just commenting on the Ogre head, since that is equivalent to "Hello World" in regards to the 3D engine. Never really made anything substantial with Ogre, used it mainly to test tracking devices and things like that. But its a nice engine, with a clean interface.

Post by **druidsbane** » Sun Oct 14, 2012 9:42 pm

Got it

Well, the Ogre head is gone now :p Instead I've added a simple landscape exported from Blender, water, skydome, fixed the proportions of desktop and made it a regular screen not that crazy rotating cube, integrated camera motion so you can fly through the landscape as well as look around using an orientation sensor (ie: Rift!!! currently only the iPhone sensor is supported for looking around obviously). Lots of cleanup and code removals. Screenshots below. libtinyxml and Ogre 1.8.0 are required, you may need to tweak config files to point to them... maybe. I still need to add the SBS rendering and barrel distortion, collision detection and then actually find pretty models to integrate with the scene in Blender!

This is the desktop showing the water... ugly but will improve once I write or find a better shader:

This is a closeup of the desktop showing the Gimp where I took that last screenshot... desktop is useable in 3D, I just hope that the Rift will be good enough as it stands or in next iteration to be useable:

Post by **druidsbane** » Mon Oct 15, 2012 7:52 pm

Improved the water shader and added stenciling on the terrain in Blender so that I can create paths and so on with textures on the terrain. Ogre is more powerful than I thought and the combination of Blender and Ogre is excellent for creating a world design. UV texturing is still a pain to figure out and if anyone knows how to "Bake" in the automatically generated texture coordinates please share

Everything is checked-in in case anyone wants to try with the "-o" extension. Make sure to have all the dependencies. Next up will try to get the render-to-texture stuff done and the distortion. Any thoughts or suggestions would be quite welcome. Oh, and I probably want to add some sound to the world

Post by **druidsbane** » Tue Oct 16, 2012 9:24 pm

Another boring update, but just got barrel distort working on Ibex for the Ogre 3D plugin. The world is beginning to shape up. Performance dropped from 600fps on my desktop to 350fps, so there is definitely a toll. I have some ideas on how to speed things up, but not just yet. I think I might need to tweak the texture settings, and lower to the Rift's resolution, FOV is all wrong on this as well due to how Ogre works, hopefully that can be fixed. Also, I don't have a way of toggling the barrel distort off yet. Looking forward to adding the neck rotation model, improving text sharpness, zooming in on texture so no wasted space on edges as can be seen.

Post by **cybereality** » Tue Oct 16, 2012 9:34 pm

Cool.

Post by **druidsbane** » Tue Oct 16, 2012 10:03 pm

Is that NY stereoscopic meet up still going on? I see only two meetings took place but might be cool to meet other VR folks!

Post by **cybereality** » Tue Oct 16, 2012 10:05 pm

Yeah, its still going on, just been busy. Planning on holding a meeting maybe in December to demo the Rift.

Post by **druidsbane** » Tue Oct 16, 2012 10:14 pm

Cool! I'll bring mine too if I have it by then. Will be interesting to try out your driver as gaming is still my main goal with the Rift even I also want to use it for a more productive desktop

Post by **druidsbane** » Fri Jan 04, 2013 10:57 pm

I finally got the first beta release ready for the Mac. It requires OS X 10.8. I hope this works for most users. It currently works on only a single monitor. You can move around using the keyboard commands above. To switch between desktop control and virtual world control press Fn+Shift+F1. You can use the open source iPhone app to simulate a motion sensor if you have an iPhone developer account using the source on bitbucket above. I will try to get a binary out on the App Store once Ibex is more stable and officially released. I hope you enjoy and please feel free to send me feedback. The download link is below. There isn't a source-release yet as it isn't actually using any open-source libraries or linking against them. I will try to figure out how to make this work with the already out-there Linux version of Ibex. Happy New Year!

http://hwahba.com/resources/IbexMac-b1.zip

Post by **cybereality** » Fri Jan 04, 2013 11:32 pm

Cool man!

Post by **brantlew** » Sat Jan 05, 2013 5:12 pm

Cool. The Rift warping is a nice touch.

Post by **DeeKej** » Sat Jan 05, 2013 6:27 pm

This is really cool! I want to try spending the summer trying to create an open source virtual workspace to use with Rift and Leap, if something like this would be ready by that time...

Awesomeness to say the least

Post by **druidsbane** » Sat Jan 12, 2013 5:42 pm

Thanks for your comments guys

I'm still on vacation so I've been working on this sporadically here and there.

For anyone that's still following this, a small update for the Mac version today to enhance mouse and CPU performance. Tiny memory leak aside I think it is looking much better. All that's left now is an actual Rift, releasing the iPhone client and adding dual-display support so we can run on the Rift with a higher resolution virtual screen instead of a desktop with the Rift's resolution.

http://hwahba.com/resources/IbexMac-b1.zip

Post by **coresnake** » Sun Jan 13, 2013 10:15 am

Awesome, would be cool if someone made a Matrix-esque 'white room' SKU for it too, some of us don't like the outdoors

Post by **druidsbane** » Fri Feb 08, 2013 6:28 pm

Just a heads-up. While the source isn't available yet like the Linux and Mac ports of Ibex, the Windows one is finally making headway. Performance is actually quite good even using the brute-force method that I am right now. I don't know yet how functional it will be compared to the others until I get a Rift or test with a second monitor, but it is moving along nicely. Took a few weeks to finally get something to work with all the false starts and so on. For those that care, I'm actually going for brute-force Win32 screenshots, not DirectX or a virtual display driver or anything like what remote desktop systems would use for example. Currently it displays the desktop, you can move around the world and look around and it is quite fast. I haven't tested on my laptop yet so I can't speak to that performance though. Hopefully we'll have a solution ready by the time the Rift is out

Post by **TheHolyChicken** » Wed Feb 27, 2013 9:53 am

Thought I'd just chime in to say keep up the good work! This is a project I've been keeping an eye on, as the prospect of working in a virtual environment is very appealing to me. I want to do programming in the middle of a forest or on the beach, not in a dreary office IRL!

Post by **geekmaster** » Wed Feb 27, 2013 11:41 am

TheHolyChicken wrote:Thought I'd just chime in to say keep up the good work! This is a project I've been keeping an eye on, as the prospect of working in a virtual environment is very appealing to me. I want to do programming in the middle of a forest or on the beach, not in a dreary office IRL!

ME TOO!!! (Shouting from the rooftops)...

Post by **druidsbane** » Wed Feb 27, 2013 11:53 am

Glad you guys like what little is out there

Question, are you guys Linux/Windows/Mac users? Until the Rift SDK is out I can't tell for sure which platform will be ready to support out of the box other than Windows though. Also, anyone care about building worlds in Blender? The core version of ibex only does the skybox with a floating desktop, but if there is interest I'll port the Ogre3D stuff to OSX and Windows and maybe we can get support from others to build some nice environments with sound and animations and all that!

Also, another option is to try to get it integrated somehow in UDK or Unity, no reason it can't be done other than time and money, but that might be easier to target depending on who is out there, but my preference is Ogre3D since completely open-source and we can use it on Linux easily. Ideally I'd like to target other platforms but I'd rather not mention anything till I actually get started on those if ever.

Post by **geekmaster** » Wed Feb 27, 2013 12:47 pm

I use linux and Windows. I also like Ogre. But I also like drawing raw pixels directly into the framebuffer.

But for big complex modern games, the other comprehensive solutions commonly discussed around here are probably a better fit. It all depends on your application and target audience.

Post by **hal10000** » Fri Mar 08, 2013 9:01 am

I'm on Windows and I'm really excited for this. It will be it so much easier to start games and apps while still wearing the rift.

Post by **druidsbane** » Fri Apr 12, 2013 9:30 am

Also, just a heads-up, the win beta for ibex is at (http://hwahba.com/ibex). The source is on bitbucket. If you have a Rift it will detect it and run fullscreen on it, otherwise it will just run in a 1280x800 window on your main display. I'm sure there is much to do but it should be fun if you can give it a shot. I might need to statically link against the MS C++ libraries or you might need to install a runtime, please let me know how it goes if you give it a try

Post by **TheHolyChicken** » Sat Apr 13, 2013 1:22 am

druidsbane wrote:Updated the binary to release mode, had to rebuild the Rift lib using VS2012. You may still need the VC++2012 Redistributable (http://www.microsoft.com/en-us/download ... x?id=30679) but I hope not. If you do I'll try to figure a way to get it to copy along with the binary somehow. Hope it works now with less extra compilers installed

I don't have the VC2012 redistributable installed. The application ran, but there were problems.

I tried the keyboard commands suggested by the website (ctrl+shift+y to enable movement etc) but they did not appear to do anything. Ibex takes my mouse control, even after alt-tabbing to another application - fortunately I know how to use task manager using keyboard only, otherwise I would have been unable to kill it! (alt+f4 did not close ibex).

I did not see a mouse within ibex, and keyboard commands did not seem to do anything - how to you use the virtual desktop environment? I have a suspicion that my framerate is essentially zero, but I have no way of telling (how can I display FPS?). Despite this, I did manage to get a screenshot

Why am I looking at the floor despite just claiming the controls don't work? Somewhere in all my button pressing and alt-tabbing the view (instantly) jumped from its initial vector to this one!

Should I install the 2012 redistributable and try again....?

Post by **druidsbane** » Sat Apr 13, 2013 9:53 am

Ah, my instructions are really bad and the main website is out of sync with the bitbucket instructions from the source. I'll work on getting a help screen up in there somehow. So, turns out for Windows it is Ctrl+Shift+G

I ran into issues with the key bindings from each OS meaning something else in different programs, so ctrl+shift+F meant something in Visual Studio on Windows so I decided to change it. I think that I can accept the same keys in all OS's but have a primary one in case you have an application that uses that key binding so you don't get stuck.

Also, in terms of framerate, even without switching modes, the mouse should give you mouselook and the arrows should work or the asdw keys to walk around. Lastly, when in Rift mode, you'd need to switch to desktop mode then move the mouse over to the main screen. for it to be visible.

Hopefully your framerate isn't as bad as 0fps

I'll try to add some stats now to the main UI!

Post by **druidsbane** » Sat Apr 13, 2013 11:34 am

I've just updated it to accept Ctrl+Shift+F2 as well to switch like on the Mac and made it resolution independent from the current Rift so it will scale up with future hardware with hopefully no changes.

The most interesting part was lowering the resolution to 1280x800 just for playing around and finding that I could get work done and use it quite comfortably even on the current Rift hardware! This is pretty cool because now one can browse or work or whatever using current hardware and it gives me a baseline for the virtual display resolutions to start off. I might make it an option to choose your desktop resolution on startup so you can use it easily. It has also exposed the issues with navigation switching back and forth between the desktop and ibex control mode. I'll integrate my Hydra code soon as I had it working on the Mac but considering its size I'd still like to use something like the Leap Motion if they every actually send me the one I preordered a long time ago, beginning to give up on those guys.

If anyone has a Rift it will be tons of fun to use now

Post by **TheHolyChicken** » Sat Apr 13, 2013 12:09 pm

Just installed the x64 Visual 2012 Redistributable, and having the same problems. The rendering/warping works and looks really cool.... for one frame. Then the ibex application is essentially frozen with 0 FPS. Moving the mouse or using WASD doesn't seem to have any effect (they might be, but 0 FPS!). Using ctrl+shift+g I can now get mouse control and close the application easily now, thankfully, so no need to do awkward task manager navigation and killing via keyboard shortcuts anymore!

EDIT: I should say that I do have a strong computer (eg 500+ FPS on the C++ Tuscany demo) so it's not just due to my system being shite

EDIT2: wasd and mouse control definitely work. When I start the application there is a ~1 second period before ibex fully loads up; if I use WASD or move the mouse during this period, ibex has definitely responded! But it's 0 FPS after that. Hope that's of some help. I'm on 64bit win7, latest patches etc etc etc installed for everything.

Post by **druidsbane** » Sat Apr 13, 2013 12:29 pm

One question though about it locking up. When you say ctrl+shift+g allows the mouse to respond, do you mean outside of the app? I'm assuming from the screenshot earlier that you weren't wearing a Rift so that means that you weren't moving the mouse on the virtual desktop but rather the regular one. It is probably crashed or something, I might need to add logging at some point to debug this or run it on another non-dev machine at home and see

Thanks again! Your dream of the beach and resting your feet in the surf while using the desktop is near at hand

Post by **TheHolyChicken** » Sat Apr 13, 2013 1:33 pm

druidsbane wrote:One question though about it locking up. When you say ctrl+shift+g allows the mouse to respond, do you mean outside of the app? I'm assuming from the screenshot earlier that you weren't wearing a Rift so that means that you weren't moving the mouse on the virtual desktop but rather the regular one. It is probably crashed or something, I might need to add logging at some point to debug this or run it on another non-dev machine at home and see

Correct, I don't have my Rift yet; I'm expecting (hoping?) it roughly at the end of next month.
By default, the app 'swallows' my mouse. It's hidden when ibex has focus. If I alt-tab to another application I can SEE my mouse, but when I attempt to move it it is stuck in position (it vibrates around). After pressing ctrl+shift+g, I can now see AND move my mouse freely, both in ibex and my desktop in general.

Thanks again! Your dream of the beach and resting your feet in the surf while using the desktop is near at hand

Haha, you remember

I'm counting on you!

Post by **Hermit** » Sun Apr 14, 2013 8:19 am

the infinite recursion is a bit of overkill

Also guys, this will only work with an aero enabled windows desktop theme.

Post by **druidsbane** » Sun Apr 14, 2013 8:25 am

Hermit wrote:the infinite recursion is a bit of overkill Also guys, this will only work with an aero enabled windows desktop theme.

The infinite recursion thing is mainly because the screenshot is on the desktop instead of from the second display that is the Rift. I'd prefer that ibex be transparent to screenshots on the Windows version like it is on the Mac and Linux ones but it seems that Microsoft took away that API call in Win7 and later. I'll check again though. I'm currently just using the regular WINAPI which means that it should work on anything from windows xp to windows 7. Is that not your experience? Aero shouldn't have any impact, in fact it actually hurts a bit because when alt-tabbing it decides to render the desktop only and all the windows transparent which can be very annoying, so Aero off is even better

Post by **Hermit** » Sun Apr 14, 2013 11:11 am

It definitely works, but it does require aero on my machine.

Putting aero off will make the cpu do more work, this is slower than the gpu so unless your application is gpu bound you will want it on.

I have been working on a project like yourse btw. Mostly by hooking the directx desktopmanager to steal the texture handles from every application, so i can render windows individually in 3d space. there is someone else that already does this: http://www.youtube.com/watch?v=Fpvh16VjrRE

Post by **TheHolyChicken** » Sun Apr 14, 2013 4:29 pm

Had a few spare seconds and tried with aero and it actually worked, and worked with good FPS. Without aero enabled I get ~0 FPS.

Weird. Haven't had a chance to experiment further.

Post by **druidsbane** » Sun Apr 14, 2013 4:32 pm

Cool find! Now I have something to debug

Post by **druidsbane** » Sat Apr 20, 2013 11:07 am

TheHolyChicken: I've fixed the bug with Aero being disabled

My desktop has stopped starting up once again so I didn't test to make sure the binary auto-detects the Rift as before, but all I changed was that little bit related to the Aero being disabled and getting the window to work, so you can now use your desktop as before and still test this

Also, next up I'll integrate the lens distortion caching so it runs even faster than before. The Mac version already does that and I'll integrate it to the Windows code-base soon. Thanks for your test, never would have thought to try that on my own

Post by **TheHolyChicken** » Sat Apr 20, 2013 11:35 am

Yep, works fine now without Aero.

Post by **druidsbane** » Sat May 18, 2013 9:52 pm

Hi all! The Ibex betas for Windows and Mac at http://hwahba.com/ibex have been updated. They add better support for the basic video playing support that was mentioned earlier (press the / key to bring up the UI, W/S keys to navigate, Enter to select a file, Escape to exit. You can only play a video once before restarting ibex or it will crash). They also enable basic Razer Hydra use: press the LT and then RT buttons on your controllers, thereafter you should be able to walk around using the thumbstick.

Video and audio are very choppy on Windows but I'm working on it. I want to change the main monitor to a curved display in the next update and get more Hydra actions hooked up as well as walking where you look support.

Post by **AngelJ** » Thu May 30, 2013 9:44 am

I've been following this thread since the beginning so "shame on me" for not trying Ibex yet. Anyway, it looks really awesome. Keep up the good work!

Post by **ThePhilosopher** » Wed Jul 24, 2013 3:13 pm

How about the problem of resolution for a useful desktop?
I mean, I've read that with OR DevKit 1, reading little texts in games is very difficult.
Or do we have to wait for a long time (longer than the wait for a consumer OR) to see a practical VR desktop?

Post by **druidsbane** » Wed Jul 24, 2013 3:20 pm

ThePhilosopher wrote:How about the problem of resolution for a useful desktop?
I mean, I've read that with OR DevKit 1, reading little texts in games is very difficult.
Or do we have to wait for a long time (longer than the wait for a consumer OR) to see a practical VR desktop?

Well, I've tested it and it works quite fine at a desktop size of 1280x800 or 1440x900 (the resolution of my MacBook Air). This means that one desktop is readable at current resolutions and to get more you'd either have to go HD or create more virtual monitors (which isn't so bad

). The HD version will help a lot though

I'm working on the Linux one once again so that will be the first one that supports multiple displays I hope. Development is slow but getting there. Been distracted by other things in life, but always find a little time to work on it every day or few days. I'm unifying the build system for all the OS versions so that I can finally not waste time porting my code to each one and modifying flags whenever I add a new library.

Post by **Icbribob** » Fri Aug 23, 2013 9:33 pm

I am currently trying to use ibex in a AR project.

My hardware setup is a bt-100 moviero I have modified to take a HDMI input. Running ibex on a Mac PowerBook.. Mouse/keyboard for control. Building a 9dof headtracker.

I would like to have just a big transparent "room" as my background. It's nice to have a "world", but with the moverio I could use my real location as my "world"

Any guidance on how to do this? I really just want a blank empty world, no color.....empty space that I can move around my windows in.

Would it be a simple as building a large room in blender and making the texture transparent, then exporting to ibex?

By the way, as is it is still a trip.

Bob

Post by **ThePhilosopher** » Wed Apr 16, 2014 6:56 am

I'm still following this amazing (and unique!) project, made by only one man!

I hope there will be more people interested in it when the screen resolution of our HDM will be more usable for this desktop project.
Maybe with the RiftUp! more people will join this ambitious project