VR Desktop - Method of Loci

Talk about Head Mounted Displays (HMDs), augmented reality, wearable computing, controller hardware, haptic feedback, motion tracking, and related topics here!
Post Reply
User avatar
unsilentwill
Cross Eyed!
Posts: 171
Joined: Wed Sep 12, 2012 6:12 pm
Contact:

VR Desktop - Method of Loci

Post by unsilentwill »

Hello everyone! I've been lurking this forum since E3 after John Carmack's demo, just amazed by the intelligence and creativity going on here. I'm a just young artist and indie game designer, finally getting into Unity and excited to make a game with/on the Rift!

I'm completely new to VR but a huge fan of 3D and motion control (in theory anyway) so forgive me if this idea has been discussed before here, or in a book or movie. Essentially when thinking about the Rift and VR actually happening my head spins and goes off in crazy directions. I saw the amazing work of the in-game 2D desktop screen displayed in-game, I imagined an operating system and/or desktop built from the ground up in a 3D space instead of 2D desktop.

Image
Imagine each object as a program or file, organizing them physically or with special commands

Image
They could replace folders and even shortcuts to other rooms/areas, think Futurama's internet + Paraboxes

Basically each file or program or shortcut would be displayed as a customizable object with its own physics. Now that sounds stupidly complicated to the tried-and-true desktop, and don't worry it is, but I consider it an improvement for a few reasons. One, who needs Windows when you have the great outdoors? Two, I'm personally very visual so traversing through a series of folders is often more confusing to me than searching through even chaotic piles on my desk. And three, it taps back into our memory unlike traditional desktops.

The "method of loci" aka "thought cathedral" or "memory palace" is a memory technique where physically or mentally walking around a space and assigning arbitrary names/dates allows you to revisit the space mentally and remember massive amounts of information compared to just off the top of your head. Given the amount of useless game information and maps I currently remember without even trying, I figure its very successful and could increase productivity back in the real world immensely, depending what files you work with. Especially with a search/categorizing function.

Also I'm a bit biased as an artist, because I'd love it if all of my digital artwork could be hung up around my office at different sizes, or 2D games projected, 3D models fully visible in my workspace (not to mention a VR sculpting program... be still my heart). And I suppose there's a bit of an escapist in me who'd rather work in one of these worlds: http://imgur.com/a/sA4kL or a house that I designed in 3D, or even working in the game and designing the world around me instantly seeing changes, than my small room. It's basically AR, but inside VR I guess? Better than a laptop, you could take your programs with you as you wander through a virtual forest or city or spaceship or whatever (amazing images in the link running on CryEngine, and from polycount, I can still dream of 3D SBS 60 fps Crysis ...someday).

This would probably require writing a whole new OS, or seriously modding Linux, and I don't even know who would attempt such a thing, but what do you all think? Functional concept or waste of time?
User avatar
cybereality
3D Angel Eyes (Moderator)
Posts: 11406
Joined: Sat Apr 12, 2008 8:18 pm

Re: VR Desktop - Method of Loci

Post by cybereality »

Yeah, I think its a great idea. Probably modding linux would be the best bet, no need to invent a whole OS. Also, this could even work on Windows just as an app (its been done before). Would be very cool.
User avatar
rhinosix
Cross Eyed!
Posts: 136
Joined: Thu Aug 30, 2012 10:19 am

Re: VR Desktop - Method of Loci

Post by rhinosix »

Love that gallery you posted!

This is one of my main wants for VR - to be able to escape into a beautiful virtual world to work creatively without being distracted. I'd love to be able to have notes, whiteboards for mindmapping, sketches etc. plastered on my virtual walls for reference as I'm working on something at the desk in front of me.

I think after the HMD, the most important hardware is haptic gloves. It's going to be hard to interact with an abstract 3D operating system if you can't interact with the world in 3D. But even word processing, drawing using a tablet or LEAP motion, or regular 2D computing would be cool in a virtual environment.

An idea I had when I first heard about the Oculus Rift was a word processor which using natural language processing to construct a surreal 3D environment around you. Something similar to Scribblenauts. So if you start writing, and use words such as 'sand', 'dunes', 'wind', 'hot sun', your basic environment would become a desert. If you mention an animal or an object there's a chance one would be generated in the world. As you write more the landscape would be constantly changing until you're in a completely different environment and things that haven't been mentioned in a long time would fade away.

I think it would be inspiring to work in a fantasy, or beautiful natural environment even if all the functionality isn't possible yet.
EdZ
Sharp Eyed Eagle!
Posts: 425
Joined: Sat Dec 22, 2007 3:38 am

Re: VR Desktop - Method of Loci

Post by EdZ »

One thing to remember: without a haptic interface (i.e. one that allows you to touch and feel virtual objects), a lot of the proprioceptive 'cheats' we take for granted when interacting with objects, for example riffling through papers on a desk, are unavailable in a VR environment.
STRZ
Certif-Eyed!
Posts: 559
Joined: Mon Dec 05, 2011 3:02 am
Location: Geekenhausen

Re: VR Desktop - Method of Loci

Post by STRZ »

Wayland, the new Replacement for X in Linux could be a very nice playground for such new things.

https://www.youtube.com/watch?v=_FjuPn7MXMs

Imagine yourself with a HMD walking around in a virtual office and throwing a unwanted file into the trash bin :shock:
User avatar
Chriky
Binocular Vision CONFIRMED!
Posts: 228
Joined: Fri Jan 27, 2012 11:24 am

Re: VR Desktop - Method of Loci

Post by Chriky »

@ STRZ

That's amazing! It shouldn't be too hard to get that rendering in Rift-mode. That'll be one of the first things I work on when I get my Rift, if not before.
STRZ
Certif-Eyed!
Posts: 559
Joined: Mon Dec 05, 2011 3:02 am
Location: Geekenhausen

Re: VR Desktop - Method of Loci

Post by STRZ »

There is really crazy potential with that new technology in Linux. For example take the video, you could have multiple players in the virtual environment looking at the same screen, or lets say a virtual cinema watching the same movie side by side. Or programming each with their own virtual keyboard on the same program and then watching what the others are doing, you just walk to their screen :woot

It's very useful for servers and virtual machines, cloud computing and that stuff. I doesn't matter where your computer is, you can call Program Windows from wherever you want/have acess, to your device, or like you see in the video into a virtual environment.

It could be crucial for work too, you have not to change your location anymore if you do office stuff, you have your boss just walking by in the virtual world staring at your screen :twisted:

Wayland 1.0 has been released some weeks ago, there is currently afaik one Distro showing it off for testing purposes http://sourceforge.net/projects/rebeccablackos/

Another Linux Distro wich will support wayland is this one http://www.maui-project.org/

Others like Arch, Debian or Gentoo should have it in their repositories.

The most amazing thing is that it's free software and anybody can tinker with it and use it. Looking forward to see the development of this technology and how different tools like quicktime 5 or maybe compiz and other new compositors will use it.
User avatar
unsilentwill
Cross Eyed!
Posts: 171
Joined: Wed Sep 12, 2012 6:12 pm
Contact:

Re: VR Desktop - Method of Loci

Post by unsilentwill »

cybereality wrote:Also, this could even work on Windows just as an app (its been done before).
Oh yeah? I'd love to see it, I've just seen some digital galleries which are pretty cool, but nothing full blown.
STRZ wrote: https://www.youtube.com/watch?v=_FjuPn7MXMs

Imagine yourself with a HMD walking around in a virtual office and throwing a unwanted file into the trash bin :shock:
Indeed! This is what inspired me think start thinking bigger and crazier. The biggest difference would be related to core OS functions, since rendering a full 3D space would be more intensive than just 2D, you wouldn't want to be running both desktops. So how difficult would it be to display files as 3D models by default?

Also programs would be a trick, mapping their functions to basic game functions (like Amnesia, Half Life 2) picking things up, turning knobs. I'm imagining a firelit room with an easel that I can render Photoshop on, a still life made of my computer files, and reaching over to the turntable where I can play a record (rendered from a folder of MP3s and played through iTunes or whatever). Is it just me, or is deleting an unread email completely different from crumpling a seal letter and tossing it into a fireplace?

I guess there's something about physicality I miss especially with Windows 8 looking the way it does being so abstract. Maybe the visual themes of Windows 98 spoiled me with custom cursors, icons, colors and desktop. Not to mention being able to visit other people's desktops over the internet, could even be a new form of website-slash-office?

So many questions. Too excited for January you guys.
STRZ
Certif-Eyed!
Posts: 559
Joined: Mon Dec 05, 2011 3:02 am
Location: Geekenhausen

Re: VR Desktop - Method of Loci

Post by STRZ »

unsilentwill wrote: Indeed! This is what inspired me think start thinking bigger and crazier. The biggest difference would be related to core OS functions, since rendering a full 3D space would be more intensive than just 2D, you wouldn't want to be running both desktops. So how difficult would it be to display files as 3D models by default?
Depends on the OS you use, Windows would be very difficult because you can't modify the system deep enough, to replace the desktop environment completely for example. So everything you do would exist on top of the existing one. Displaying files as 3d models by default means that you create a different desktop environment and need to get rid of the default one completely.
Also programs would be a trick, mapping their functions to basic game functions (like Amnesia, Half Life 2) picking things up, turning knobs. I'm imagining a firelit room with an easel that I can render Photoshop on, a still life made of my computer files, and reaching over to the turntable where I can play a record (rendered from a folder of MP3s and played through iTunes or whatever).
You'd need to modify the source code of those programs and incorporate/translate/create the GUI (graphical user interface) into your new desktop environment. You can't do that with closed source proprietary software like Photoshop or iTunes because you don't have acess to the source code. It's like Windows itself, it's not meant to be modified by the end User.
Is it just me, or is deleting an unread email completely different from crumpling a seal letter and tossing it into a fireplace?
Imagine sitting in your virtual office, and everytime you get an E-Mail a secretary knocks your virtual door to bring you the E-Mail. I think deleting unread E-Mail is the same than throwing a letter into a fire, just not so emotional.
I guess there's something about physicality I miss especially with Windows 8 looking the way it does being so abstract. Maybe the visual themes of Windows 98 spoiled me with custom cursors, icons, colors and desktop. Not to mention being able to visit other people's desktops over the internet, could even be a new form of website-slash-office?
I don't use Windows anymore for other stuff than gaming, because your ability to modify the system to your liking is very limited. You can't dig deep enough into the system and because of that you will never understand how computers really work. It's just a waste of time to mess with Windows if you are a technically interested person because it only lets you touch the surface of a operating system. I see the concept of Windows 8, trying to make it easier for people who are not interested in computers and are used to smartphones, so that the product appeals to a greater mass of people and sells more copies.

The less people translate/reabstract stuff wich is going on on their computers into the real world the better for those who want control over the people and benefit from it. That's a reason why many people do stupid things related to privacy in the internet, they just don't think of what it would translate to in the real world.

It's sort of a similar concept like lawspeak, abstracted to the point that it's difficult for normal people without a lawyer to get around it. It would be cool to eliminate this abstraction layer, to make it clear for everybody what they're dealing with, wouldn't it? 8-)
Post Reply

Return to “General VR/AR Discussion”