VRPaint (and VRBrowser) (crosspost from OculusRift.com)

[Kajos]

Hi all,

I am hereby introducing my app, VRPaint, for the Oculus Rift. I spent about a month of my spare time developing it. It is still somewhat rough around the edges (sloppy code at some places, ssao is not optimal yet, texture atlas is not 100%, stereo could use some work), but it is a functional prototype and the idea behind the construction of the engine is very smart, if I may say so myself.

It is a panoramic 3D paint application coded in Java and works on Linux, MacOSX and Windows (using LWJGL). (Note: MacOSX users need to change the start file)

The project is opensource and licensed under GPLv3.

Download: http://code.google.com/p/vrpaint/downloads/list (be sure to read the readme for controls)
Screenshot: http://imgur.com/Pdtxhfs ( nothing fancy just threw some things together, I'm no artist )

A brief explanation of the engine:
What it basically does is rendering the geometric scene to a skycube with the depth stored in the alpha channel. After a new object is added, the geometric scene (the object) is rendered and merged with the previous scene (which is stored as 6 textures, a skycube) in a fragment shader.

The engine works at an excellent 60 fps also on slower graphics cards. To do this I split the rendering of the scene to the skycube, which is a fairly hard task, to happen over multiple frames. The only real thing that is somewhat heavy is the ssao and godrays; these can be turned off.

A stereoscopic view is generated with the stored depth using reprojection. I used a very simple form of reprojection; it gives some artifacts. I used a trick I learned which is to use the reprojected frame for the non-dominant eye, which most of the time is right, but can be switched with a shortcut. I use the original frame for the dominant eye. Artifacts are a lot less visual then! Also, I was working on a better reprojection shader, but got tired of it, it took quite some time to perfect.

I integrated animations by storing the texture coordinates and using texture atlassing. I change the texture coordinates in a shader, to simulate movement of the texture (there are 4 types of movement atm). I did this because I didn't want the scene to be dull, plus a simple pixel picking can show what texture was used (lava, water, stone), which I did because I wanted to create an Alchemy type of gameplay.

As a final note to clarify; the engine does not (actually hardly) use polygons, but instead it uses pixels to store the environment. Therefor, the scene can have a lot of detail. Try it! Place as many objects as you can, the engine won't slow down!

VRBrowser:
I incorporated links as well at some point, this works pretty good. In theory this engine could very well be suited for a low cost VRBrowser. It is easy for webdevelopers to create a scene, there only needs to be taken account for the looks (since the engine always runs at 60fps on most gpu's). Only disadvantages I can think of, is ofcourse the restricted walk space, and the loading times for a scene (a scene is 10mb, a lot more than your average website. This can become a lot smaller, but then rendering time will appear). To note; the linking feature is removed, but if there is demand for it, I will begin working on an actual VRBrowser with this engine.

Greetings,
Kajos

[PasticheDonkey]

runs well is just really unintuative control wise and i only seem to be able to select red or grey or animated textures. also the interlaced mode seems to have a strange warp to the display. do you think you could write to two z depth buffers simultaneously for the indepentant views. little more memory requirement but that doesn't seem a problem.

[Kajos]

If you want you can remove the animations alltogether by removing the following lines in singlepass.frag in the shaders folder.

Code: Select all

if (color.r > 0.8) {
		savedC.x = mod(savedC.x + sin(time / 1000.0 * (color.r * 3.0)), 1.0);
		savedC.y = mod(savedC.y + cos(time / 1000.0 * (color.r * 3.0)), 1.0);
	} else if (color.r > 0.6) {
		savedC.x = mod(savedC.x + time * (color.r) / 1000.0, 1.0);
		savedC.y = mod(savedC.y + time * (color.r) / 1000.0, 1.0);
	} else if (color.r > 0.4) {
		savedC.x = mod(savedC.x, 1.0);
		savedC.y = mod(savedC.y + cos(savedC.y + time / 1000.0 * (color.r * 6.0)) * waveLength + waveLength, 1.0);
	} else if (color.r > 0.2) {
		savedC.x = (cos(savedC.x * 5.0 * PI + time / 1000.0 * (color.r * 6.0)) * waveLength + waveLength) / 2.0 + savedC.x * 5.0;
		savedC.x /= 5.0 + waveLength;
		savedC.y = (sin(savedC.y * 5.0 * PI + time / 1000.0 * (color.r * 6.0)) * waveLength + waveLength) / 2.0 + savedC.y * 5.0;
		savedC.y /= 5.0 + waveLength;
	}

Controls are indeed not too intuitive. I considered implementing Wiimote support that would help a lot, especially with rotation. I use a texturepack which you can replace with e.g. Minecraft texture packs, although it's best to keep it to 16x16 maps. (Note: I think it needs to be png 24 bits. If you have another type, open in paint and save as png!)

Edit: I don't see the warp you are talking about. Can you maybe show it with paint or something?

I see btw; that one of the shadow modes is broken.

[Diorama]

VRBrowser:
I incorporated links as well at some point, this works pretty good. In theory this engine could very well be suited for a low cost VRBrowser. It is easy for webdevelopers to create a scene, there only needs to be taken account for the looks (since the engine always runs at 60fps on most gpu's). Only disadvantages I can think of, is ofcourse the restricted walk space, and the loading times for a scene (a scene is 10mb, a lot more than your average website. This can become a lot smaller, but then rendering time will appear). To note; the linking feature is removed, but if there is demand for it, I will begin working on an actual VRBrowser with this engine.

This is a possibility that I have been really interested in since I first started thinking of the real implications of the Rift.
I remember in like 1997 my friends dad said to him "one day, I'm going to take you on a car-ride through the Internet". That doesn't seem so silly and futuristic now. 2017?

[PasticheDonkey]

a mouse keyboard control system that could work would be.

mouse freelook
mouse wheel object z depth

then when right click is pressed as a toggle
mouse x and y rotation
mouse wheel z rotation

a key as a toggle gives menu for selection of colour shape size etc

or the shape is controlled similar to the above using a key as a toggle instead of a mouse click
so while key pressed
mouse is x and y dimensions scaling
scroll wheel is z dimension scaling

simliary colour and shape can be controlled by a mouse while toggled. shape with the mouse wheel. colour you're basically looking around a dome that contains every possible colour to pick from.

anyway i think that gets you quick easy access to most common features using just two keys and the mouse.

obviously undo redo save load etc should still be keys.

[Kajos]

a mouse keyboard control system that could work would be.

mouse freelook
mouse wheel object z depth

then when right click is pressed as a toggle
mouse x and y rotation
mouse wheel z rotation

a key as a toggle gives menu for selection of colour shape size etc

I like the first part, that might work well indeed. I don't know how much information a menu can take on a Rift display however. On a computer screen it could work for sure.

[2EyeGuy]

It reminds me of this:
http://graphics.cs.brown.edu/research/s ... nting.html

[Kajos]

It reminds me of this:
http://graphics.cs.brown.edu/research/s ... nting.html

Ha, that's awesome!

Also, for any tourists coming to the Netherlands or locals; http://panorama-mesdag.com/ is a really incredible panorama painting. I can imagine seeing it is very comparable to wearing a Rift.

[STRZ]

Nice that you develop it the FOSS way

i thought about starting to get into researching virtual spraypainting, because this is what i'm good at in real life, and would like to do it in a virtual environment. Like taking a Razer hydra and mod it into a spraycan and painting trains and walls in virtual worlds. it would be well suited, because in reality you have no friction with the surface you paint on using a spraycan, and could translate all the skills from the real world into the virtual world if the tracker in the spraycan/Hydra works well enough, and you get the physics of the virtual spraypaint on different surfaces and the spraying in general right.

[PasticheDonkey]

you'd still need a nice way to make masks as well.

[Kajos]

Nice that you develop it the FOSS way

i thought about starting to get into researching virtual spraypainting, because this is what i'm good at in real life, and would like to do it in a virtual environment. Like taking a Razer hydra and mod it into a spraycan and painting trains and walls in virtual worlds. it would be well suited, because in reality you have no friction with the surface you paint on using a spraycan, and could translate all the skills from the real world into the virtual world if the tracker in the spraycan/Hydra works well enough, and you get the physics of the virtual spraypaint on different surfaces and the spraying in general right.

I didn't see a way of making money out of it, so I thought it would make a good opensource project

Someone mentioned Hydra support before, I unfortunately don't have the money to buy one though. It would be sweet with it for sure

[Kajos]

you'd still need a nice way to make masks as well.

It's all very possible, but I think I leave it at this release at the moment. It will take some time before the Rift becomes mainstream, I'll probably add much more features then. As of now it's a proof of concept; the engine shows the only proper way to do real painting (although the brush is not implemented yet :p I think it would be hard to draw in 3D hence I left it out ) in VR without limitations of polygoncount. I haven't seen an engine like it elsewhere (if someone knows though.. let me know)

[PasticheDonkey]

i've actually thought of similar before but i'd tread multiple objects as z projections from separate cube maps so you could actually walk around. then anything animatable has to be series of connected objects to animate etc. basically since i heard of displacement maps i thought what's the point of the polygons any more. the extra memory needed for the geometry is saved with simpler textures which become colour maps instead.

i believe zbrush must work somewhat similary.

[Kajos]

i've actually thought of similar before but i'd tread multiple objects as z projections from separate cube maps so you could actually walk around. then anything animatable has to be series of connected objects to animate etc. basically since i heard of displacement maps i thought what's the point of the polygons any more. the extra memory needed for the geometry is saved with simpler textures which become colour maps instead.

True, but I think you can only do this to an extent, then it becomes probably very noticeable with artifacts. But it is worth experimenting with, if you see what Minecraft has achieved with its engine (or Doom, etc.), you might hit the jackpot

I was thinking of creating an engine where objects in the far distance (which are fixed) are rendered to a skycube. Then update this skycube every now and then as the player moves. You get great renderdistance and more room for details etc. Could even do this with more layers probably.

[PasticheDonkey]

i've generally been too ill to have the mental stamina to actually do anything i think of. hoping i can get well enough and start actually doing stuff.

with enough data throughput rate you could essentially have a scrolling sky box. just becomes loading detail through scrolling or zooming really on the sides of the cube map.

[Kajos]

Yeah, it consumes a LOT of time and effort to produce anything new/innovative, and often you don't see any return. I just wish engines like Unity and UDK were more low level friendly, just the model and texture loading alone would be a relieve.

Do get well btw!

[Kajos]

also the interlaced mode seems to have a strange warp to the display. do you think you could write to two z depth buffers simultaneously for the indepentant views.

I think your dominant eye might be left instead of right, so try switching the dominant eye setting and see if that helps.

I've thought about storing a skycube for each eye, but I have to get back to that; at the moment I don't see it being possible (easily at least) hence I used reprojection. But this might change, I would be happy with that a lot as well

I got mentioned on the Oculus facebook btw, which is awesome! ----->>> https://www.facebook.com/Oculus3D

Lastly, I would very much like to know what everybody here would like most in the ultimate VR painting app (you can cite multiple or cite your own). Let your imagination go!

• Wiimote support
  Hydra support
  Simply proper mouse support (how many people work with a graphic tablet?)
  Instagram like features
  Brush/pen support next to the current 3D models
  I (don't) want moving textures (I think this needs to be removed, but I would like your input very much!)
  MD2/3DS (animated) model support
  Text support
  Sharing through a portal
  Export to regular HTML/Javascript website
  Focus on VRBrowser features
  Other...

Thanks!

[PasticheDonkey]

in interlaced mode you actually get image offset so i just have it so i am seeing the left most image with my left eye and right most with the right, and that's better than switching it with the button. but my eye dominance can actually vary back and forth. anyway true 3Ds always better.

great that you got the acknowledgement

i'd actually like move support, there is an open source library out there. cos i've actually got one of those.

ideally it would end up like that cavepainting shown earlier. i want to be able to paint mould chisel all the visual artistic mediums synthesized into one you can treat like any. but also have things from computer art production as well.

about the warp it looks pincushion rather than barrel or neutral.

[Kajos]

in interlaced mode you actually get image offset so i just have it so i am seeing the left most image with my left eye and right most with the right, and that's better than switching it with the button. but my eye dominance can actually vary back and forth. anyway true 3Ds always better.

great that you got the acknowledgement

i'd actually like move support, there is an open source library out there. cos i've actually got one of those.

ideally it would end up like that cavepainting shown earlier. i want to be able to paint mould chisel all the visual artistic mediums synthesized into one you can treat like any. but also have things from computer art production as well.

about the warp it looks pincushion rather than barrel or neutral.

There are some technological limitations, i.e. once an object is placed you cannot scale/move/rotate it. It is in that way like regular paint. I could incorporate layers eventually, but this will add complexity and increase the save size significantly (which is not beneficial for people viewing it online / downloading; it is a digital art form in the end).

Brush, pen support is definately coming soon. Fill (bucket) support is a little trickier since it needs to be implemented through a shader. I am going to aim this at true designers, so it will become more than a gimmick, so I thought I should orient on quality graphic tablet/mouse support.

Model loading I already implemented but I left out in the distributed files. So this will come soon as well (support for MD2 and 3DS only atm).

I figured out a way to do true 3D, and it will come with its disadvantages, but as you said it might be the way to go. I am not sure still, with reprojection one can change the focaldepth afterwards, which might be very handy. Also it will increase save size. I might have to try to find a better compression than png..

I'm still thinking about the animated textures. I thought scenes might become too dull without them, but maybe true stereoscopic support will make up for it in combination with animated 3DS and MD2?

[Mystify]

Why can't you have a different working save format from your published format? If layers will increase the save save. why not have them when editing, and published files have it all baked down to one?

[Kajos]

Why can't you have a different working save format from your published format? If layers will increase the save save. why not have them when editing, and published files have it all baked down to one?

You're right on that. Mmm.. Need to add that to the list as well.

[hal10000]

I would love a combo of the Leap motion 3d drawing support combined with the rift. You could then draw in 3d space and walk around your drawing and add things to it in 3D space. You could create 3D paintings and art that could be a lot of fun.

[STRZ]

A VR game engine where you walk around in space and create objects while beeing in the game would be crazy.

Something similar to this:

[youtube-hd]http://www.youtube.com/watch?v=VYfBrNOi ... r_embedded[/youtube-hd]

[Kajos]

Now we're throwing voxel engines around, why not voxelstein?

[youtube]http://www.youtube.com/watch?v=6ZUdsRdokO4[/youtube]

Engine made by the famous Ken Silverman of Doom etc.

[Diorama]

A VR game engine where you walk around in space and create objects while beeing in the game would be crazy.

Something similar to this:

http://www.youtube.com/watch?v=VYfBrNOi ... r_embedded

His voxel concrete looks fantastic:


With 10x the resolution it would be mind-blowing.

[hal10000]

http://youtu.be/mQkKyOOyLSs?t=2m30s This is kind of what i'm talking about, you can see him draw and then rotate the drawing in 3d space. If you could just view and walk around that same 3d space then it should be "fairly" easy to pull off by one of the geniuses that seem to be in this forum.

[omeDev]

choo choo~!

[Kajos]

True stereoscopy: achieved!

[PasticheDonkey]

i'm finding that hard to view.

[Kajos]

i'm finding that hard to view.

I need to tweak it some more maybe I think the effect shows really nice on my 3D monitor, it probably needs less seperation for the Rift. But the technique is there and it's much nicer than reprojection, it takes up twice the size to save however.

EDIT: I think the eyes are switched maybe..

[PasticheDonkey]

i might have been too close for the interocular it's higher than you normally get in 3d content.

[drifter]

A VR game engine where you walk around in space and create objects while beeing in the game would be crazy.

Something similar to this:

http://www.youtube.com/watch?v=VYfBrNOi ... r_embedded

Crazy amount of details, though no dynamic lighting (is it even possible in voxel ?)
It reminds me of 3D-Coat, a voxel modeler.