Riftsculpt - an Oculus Rift 3D Sculpting app-Community ideas

[TheRealistWord]

I've been tossing around an idea for a 3D sculpting app built from the ground up with 100% Oculus Rift integration, tentatively titled "RiftSculpt". After seeing the PS4 keynote a few weeks back and the 3D sculpting demo they showed, I decided that now would be a great time to pursue this idea! It'd be great if, as a community, we could plan something out here that could really display the Rift's potential in a sculpting program!

I took a moment to write a brief description of what RiftSculpt could be like:

"Riftsculpt is a powerful and simple digital sculpting application that takes advantage of the Rift’s immersive FOV and head tracking capabilities. Users experience a more immersive and connected experience over standard digital sculpting applications. Riftsculpt will combine the familiarity and hands-on approach of traditional sculpting with the power and speed of digital sculpting by allowing the user to dynamically create sculptures right inside of their own virtual environment. Utilizing a series of simple hand gestures, the user can quickly switch between an assortment of tools and interface modes (doing away with the need for a keyboard/mouse).

Sculptures created in Riftsculpt will be saved directly in the Riftsculpt format, .RSP, along with any special modifications for the current sculpting session. Additionally, sculptures can be exported to common 3D model formats (such as .obj) and imported into other digital sculpting applications for further refinement. Sculptures created in Riftsculpt will also be created to scale, meaning the object you create in your virtual sculpting environment will appear in true-to-life dimensions when imported into other digital sculpting applications.

Users will have the ability to open Riftsculpts created by other users and view them at the same scale as they had been created in their own virtual environment. "

For hand gestures, I imagine either rigging a Leap, or perhaps just using a PSMove controller or a Razer Hydra. Buttons on the controllers could be used for quickly changing tools, or alternatively, different hand gestures could also be assigned to different tools while using the Leap or any other hand-sensing devices. Basically, what I'd imagine for this program is:

Put on the Rift, then easily sculpt by moving your hands in thin air. Grabbing clay, pushing clay, watching your creation unfold directly before your eyes in a 1:1 true scale environment. Easily creating virtual content and sharing it with your friends. Once we rig the Rift with a stereo pair of cameras and set up augmented reality (like USC's mod), we can bring the virtual sculpture into the real world.

I'm extremely excited about this! As someone who's not too keen of building large armatures and applying huge masses of clay, using the Rift to make a digital sculpture would be really cool. (and who knows, maybe someday 3D printing the sculpture!)

Anyway, before I ramble on too much... What would all of you think would work well in a 3D sculpting app for the Rift?

[gjpetch]

If positional tracking worked then it could be really amazing, walking and leaning around a model to get different views could make the process so much more intuitive, that's something I've dreamed of for years. Could be a great way to show models to clients for feedback and signoff too. Here's hoping that the community can work out a good positional tracking solution!

[TheRealistWord]

If positional tracking worked then it could be really amazing

Positional tracking would be incredible I would think! Since you'd pretty much remain within a certain perimeter (walking around your sculpture instead of your regular PC game where you'd be traveling a mile across a huge grassy terrain), I'm wondering if positional tracking could be done with something simple like a Kinect? Also, another idea I had:

If there's no augmented reality, then you wouldn't be able to see the real world environment around your sculpture, meaning you could accidentally bump into chairs and furniture. A possible remedy for this would be clearing out a small space in your livingroom or wherever (maybe 5'x5'), and in the sculpting program, you specify the size of the free space and it'll draw dotted lines on the virtual ground to indicate the boundaries. So long as you don't walk past those boundaries in the virtual world, you'll be certain you won't be bumping into anything in the real world.

Another idea with that approach (crude mouse drawing I just did, heh, check the attached file):

1. When you're setting up your virtual sculpting environment, you hit a key and it asks you to walk around the border of your real world sculpting environment.
2. Remove the Oculus Rift for a moment, and walk around the room getting close to the furniture/edges of the wall, but making sure not to bump into anything. The software will map your position as you move, drawing a red line and defining the boundaries of your virtual sculpting space.
3. Put the Rift back on, and based on where you walked while defining the boundaries, the software will create your virtual environment, with any area located outside of the sculpting space rendered as a drop off, the dark abyss, so it appears that you're standing ontop of a cliff suspended in time. So, naturally, you won't walk off the cliff, meaning you'll stay within the boundaries you yourself defined, and you won't go crashing into any furniture!

I'm wondering if that would be an effective way to sort of build a virtual environment so when you put the Rift back on, you'll know where you can and can't step. Couple that with accurate position tracking, and you might have a nice little virtual environment that's sort of mapped to your real environment, as far as the walkable perimeter goes!

[spire8989]

This sounds cool, but damn, we really need some good VR gloves for this kind of thing. With finger tracking, positional tracking, etc. I hope Oculus is working on VR gloves. =\

Uh, anyway. The idea is pretty awesome, although I don't know about ledges for the perimeter, that might mess with my head while I'm walking around. Maybe an island or something?

[Scog]

I think for this to work well, we will need absolute position tracking on the rift.
Once we get that though, this is one of the things I am most excited about. Not just sculpting either, other 3d content creation tools as well. (poly modeling, scene creation etc)

The standard tools are powerful, but slow and take a long time to learn. I think that these tools built with the right approach could really make digital content creation so much faster and intuitive than it is now.

Oh and +1 on the leap, for me it's the obvious choice.

Now that I think of it, with good visual feedback it could still work reasonably well even without the position tracking.

[Aabel]

It's a cool idea but no haptic feedback would keep it from ever being used in a serious manner. Simply making gestures in the air won't provide enough feedback to sculpt in an intuitive manner. I imagine it would be at least as frustrating as trying to paint with a mouse, maybe even worse. I would not consider using the leap as a main input for sculpting, for some sort of navigational input sure, but for actual sculpting with a rift as a display the only device that will really work is a Sensable

Do you sculpt either using traditional or digital media?

[geekmaster]

This sounds cool, but damn, we really need some good VR gloves for this kind of thing. With finger tracking, positional tracking, etc. I hope Oculus is working on VR gloves. =\

Uh, anyway. The idea is pretty awesome, although I don't know about ledges for the perimeter, that might mess with my head while I'm walking around. Maybe an island or something?

As mention in a previous post awhile back, I was planning to put obstacles in the environment to match obstacles in my living room. That way you just bump into both of them at the same time, instead of dropping to your knees from fear of heights, then clobbering your head on a coffee table that looks like a cliff in your VR world. Less fear and more safety, for free walking in the confines of a typical cluttered room. The biggest problem I see is with kids or pets that are missing from your VR world, leading to potentially costly mishaps. Until we get it right though (perhaps with a camera detecting out-of-place RL objects and modifying the VR world to compensate), sitting in a chair would be safer. But even then, Nate said that people actually fell out of their chair during a demo back when they were still using headphones during the demos. Perhaps we need a fully-padded "Rift Room" for our free walking VR...


And yes, I want virtual Play-Doh too.

[fredrik]

This software would probably work almost out-of-box with the Rift: http://doc-ok.org/?p=493 . It is a voxel-based sculpting tool that uses two Razer Hydra for input. There is already some HMD support in the toolkit the application is built around, and I guess Rift support will be added soon after the author gets his dev-kit.

Regarding haptics, Sensable currently holds a patent on the use of haptic feedback in this type of software: http://www.google.com/patents/US6831640 . So I guess we won't see other haptic sculpting tools besides FreeForm (Sensable's own software) for another ten years... Too bad, since this could easily be implemented with a low-cost device like the Novint Falcon.

[geekmaster]

This software would probably work almost out-of-box with the Rift: http://doc-ok.org/?p=493 . It is a voxel-based sculpting tool that uses two Razer Hydra for input. There is already some HMD support in the toolkit the application is built around, and I guess Rift support will be added soon after the author gets his dev-kit.

Regarding haptics, Sensable currently holds a patent on the use of haptic feedback in this type of software: http://www.google.com/patents/US6831640 . So I guess we won't see other haptic sculpting tools besides FreeForm (Sensable's own software) for another ten years... Too bad, since this could easily be implemented with a low-cost device like the Novint Falcon.

Haptic feedback is generally obvious, and should apply generally to VR. A patent that prevents using it in a specific case, such as sculpting, is completely absurd.

[MSat]

This software would probably work almost out-of-box with the Rift: http://doc-ok.org/?p=493 . It is a voxel-based sculpting tool that uses two Razer Hydra for input. There is already some HMD support in the toolkit the application is built around, and I guess Rift support will be added soon after the author gets his dev-kit.

Regarding haptics, Sensable currently holds a patent on the use of haptic feedback in this type of software: http://www.google.com/patents/US6831640 . So I guess we won't see other haptic sculpting tools besides FreeForm (Sensable's own software) for another ten years... Too bad, since this could easily be implemented with a low-cost device like the Novint Falcon.

Haptic feedback is generally obvious, and should apply generally to VR. A patent that prevents using it in a specific case, such as sculpting, is completely absurd.

Sadly, a lot of granted patents are quite absurd.

[zalo]

I know I sound like a broken record with this leap head tracking thing, but I really do think it has potential:

The leap (in this case) would track an actively lit fiducial on top of the head and at the end of a PSMove wand. The stereo cameras on the leap will triangulate the 3D position of both! The leap has ample FoV to cover a large area, and subpixel blob tracking will make up for resolution at a distance. You can use the orientation data from the rift and PSMove for the fusion.

"But zalo, how do you determine the difference between the head and the PSMove?"
Simple! Both the rift and PSMove have accelerometers. If the leap detects one point moving, and one device moving, it will easily be able to match the two together! The rest of the time we'll have to rely on temporally coherent tracking.

Good luck with your idea! I foresought(?) this as being an endgame VR scenario, but if we can make this happen sooner, I'll be that much happier!

[Pingles]

They've done some amazing htings with "virtual" sculpting with just mouse and/or touch input. You certainly don't NEED anything more to begin working on it.

That stuff can always be added later.

Getting the core interface done first would definitely be my first priority.

Sculptris for iOS is pretty remarkable. Check out the videos at the bottom of the page.

http://pixologic.com/sculptris/

If you combine an interface like that with the 3D, positional view of the Rift I think people could be doing amazing things even with just a mouse.

[ChrisJD]

The leap (in this case) would track an actively lit fiducial on top of the head and at the end of a PSMove wand. The stereo cameras on the leap will triangulate the 3D position of both! The leap has ample FoV to cover a large area, and subpixel blob tracking will make up for resolution at a distance. You can use the orientation data from the rift and PSMove for the fusion.

The Leap tracks pointy things not lights. Currently the software only works reliably when the Leap is placed on a desk facing upwards. At the moment, if you want a solution like you describe, you'd be better off with a couple of ps3 eyes.

If someone puts something together to read raw data from the Leap or new features are added to the official SDK there may be more ways you can use it in the future.

[zalo]

David said a future version will implement a "Blob API".

[Doc Ok]

This software would probably work almost out-of-box with the Rift: http://doc-ok.org/?p=493 . It is a voxel-based sculpting tool that uses two Razer Hydra for input. There is already some HMD support in the toolkit the application is built around, and I guess Rift support will be added soon after the author gets his dev-kit.

That's correct. Vrui fully supports HMDs; the "only" things to add are a driver for the rift's internal orientation tracker, and image undistortion to account for its fisheye lenses. The Razer Hydra is just one option for input devices. Vrui supports a lot of other things, from professional tracking systems to homebrew stuff like the Wiimote, and I'm currently working on a driver for the Playstation Move.

That said, the rift is not really ideal for this kind of application. Head-mounted displays, and specifically this one, are good to place a user into a large environment and let them look around, they're for "viewing from the inside out," so to speak. This sculpting application, and other potential 3D modeling apps using similar user interfaces, on the other hand, work best when "viewed from the outside in," meaning that you have a relatively small model that you can hold in your hands and work with. HMDs can do that as well, in principle, but it requires good internal calibration (something the rift developers are not really talking about), and, more importantly, positional head tracking (which the rift doesn't have at all). These applications are more the domain of head-tracked 3D TVs or high-end CAVEs, or turn-key holographic displays like the zspace.

[Mystify]

This software would probably work almost out-of-box with the Rift: http://doc-ok.org/?p=493 . It is a voxel-based sculpting tool that uses two Razer Hydra for input. There is already some HMD support in the toolkit the application is built around, and I guess Rift support will be added soon after the author gets his dev-kit.

That's correct. Vrui fully supports HMDs; the "only" things to add are a driver for the rift's internal orientation tracker, and image undistortion to account for its fisheye lenses. The Razer Hydra is just one option for input devices. Vrui supports a lot of other things, from professional tracking systems to homebrew stuff like the Wiimote, and I'm currently working on a driver for the Playstation Move.

That said, the rift is not really ideal for this kind of application. Head-mounted displays, and specifically this one, are good to place a user into a large environment and let them look around, they're for "viewing from the inside out," so to speak. This sculpting application, and other potential 3D modeling apps using similar user interfaces, on the other hand, work best when "viewed from the outside in," meaning that you have a relatively small model that you can hold in your hands and work with. HMDs can do that as well, in principle, but it requires good internal calibration (something the rift developers are not really talking about), and, more importantly, positional head tracking (which the rift doesn't have at all). These applications are more the domain of head-tracked 3D TVs or high-end CAVEs, or turn-key holographic displays like the zspace.

I don't think I buy that at all. It allows you to move what you are working on from something inside a screen to something within arm's reach. Combine that with a positional tracking system for oyur hands, and it should be HUGE improvement is usability, even without haptics. Head tracking will probably be very useful for it, I agree, but even without that you can have extremely intuitive gesture controls for rotating and scaling your view of the object. Why model a 3D object on a 2D screen with 2D interfaces when you can work with it in 3D with 3D interfaces?

[Aabel]

Why model a 3D object on a 2D screen with 2D interfaces when you can work with it in 3D with 3D interfaces?

Because current top of the line 2d interfaces offer a basic haptic input that the proposed 3d solution does not. When it comes to sculpting workflow the haptic input is much more valuable than a low resolution head tracked 3d display. Besides in a professional work environment it will not be difficult to send the current 3d model you are working on in Zbrush or whatever to a viewer application with the push of a button.

Use Zbrush on a 21 inch cintiq display, then you will see what the fuss about pressure sensitivity is.

[mrklaw]

Tumble on the PS3 works really well in 3D with two move controllers. You get a great sense of moving your 'hands' in 3D space.

Combined with a HMD, move controllers could be a fantastic input. I think for sculpting though, positional tracking would be almost essential, due to you looking in on an object rather than out at the world around you.

[Mystify]

Why model a 3D object on a 2D screen with 2D interfaces when you can work with it in 3D with 3D interfaces?

Because current top of the line 2d interfaces offer a basic haptic input that the proposed 3d solution does not. When it comes to sculpting workflow the haptic input is much more valuable than a low resolution head tracked 3d display. Besides in a professional work environment it will not be difficult to send the current 3d model you are working on in Zbrush or whatever to a viewer application with the push of a button.

Use Zbrush on a 21 inch cintiq display, then you will see what the fuss about pressure sensitivity is.

Haptic input, as in pressure sensitivity? That seems trivial to accomplish. Look at an airbrush, that is effectively what they have. In fact, an airbrush seems like an ideal thing to model with a 3d interface. You don't actually touch the object, so you don't need the haptic feedback.
I 100% believe that actually having your object n front of you so you can interact with it directly is a better approach. Sure, current 2D methods have decades of refinement at the moment, but they are dealing with extremely fundamental limitations, and 3D will be able to go past that. We just need to work on adding that same level of refinement to it.

[Aabel]

Look at an airbrush, that is effectively what they have. In fact, an airbrush seems like an ideal thing to model with a 3d interface.

I haven't used a wacom airbrush with a hmd, but I have used it with a cintiq, it's not that great, especially for sculpting as you want to be able to press into the model, not scroll a wheel, that is unintuitive for sculpting. It would be cool for doing some, but not all 3d polypaint tasks. In fact the more I think about it and look at how Zbrush works, the next easiest step would be to just get Pixologic to support the rift, but at such a low resolution I don't think any great sculpture will be produced with the devkit. Zbrush already pushes polycounts to such a point that the density is greater than one polygon per pixel, the Rift devkit will hit that barrier even quicker. However the ability to see your hands in relationship to what you are working on is extremely important, and I am not sure I want to risk losing that 1:1 mapping of what my hands and tools are doing.

I 100% believe that actually having your object n front of you so you can interact with it directly is a better approach.

It would be very nice indeed! however if it comes at a cost to the intuitive use of the main toolset, I will take a pass as will many other artists as it will result in a loss of control over the toolset and be a step backwards. I already plan on using the rift as a viewer to check my work but I do not believe content creation technology is ready to make the leap to VR. I guess that's why we are getting devkits though! I am glad people are working on the problem, I just hope they understand what it is artists are looking for and what is desirable in a work flow.

[Mystify]

I am not sure I want to risk losing that 1:1 mapping of what my hands and tools are doing.

But that is what this can actually offer- a 1:1 mapping in actual space.

I will grant that it will not be superior on day 1. There has been too much development on doing things the current way. But it is a better road to go down, which will eventually lead to a superior place.

[Doc Ok]

I don't think I buy that at all. It allows you to move what you are working on from something inside a screen to something within arm's reach.

That's what I'm saying. I'm not comparing oculus rift to 2D displays, I'm comparing it to 3D screen-based fully immersive displays. The idea with those is exactly that you have a virtual 3D object float in front of your hands, and can touch it and interact with it. The point is that those displays work better for this kind of application than head-mounted displays. I'm not saying HMDs won't work for this; I'm saying there are better approaches. You want to use 3D interfaces on a 3D display, just maybe not a head-mounted display.

See these videos, for example. That's what I'm talking about:

http://www.youtube.com/watch?v=HR3lwWhFUD4
http://www.youtube.com/watch?v=EWZPan4mFCQ
http://www.youtube.com/watch?v=8fZJoKRjJBg

[Mystify]

I don't think I buy that at all. It allows you to move what you are working on from something inside a screen to something within arm's reach.

That's what I'm saying. I'm not comparing oculus rift to 2D displays, I'm comparing it to 3D screen-based fully immersive displays. The idea with those is exactly that you have a virtual 3D object float in front of your hands, and can touch it and interact with it. The point is that those displays work better for this kind of application than head-mounted displays. I'm not saying HMDs won't work for this; I'm saying there are better approaches. You want to use 3D interfaces on a 3D display, just maybe not a head-mounted display.

See these videos, for example. That's what I'm talking about:

http://www.youtube.com/watch?v=HR3lwWhFUD4
http://www.youtube.com/watch?v=EWZPan4mFCQ
http://www.youtube.com/watch?v=8fZJoKRjJBg

eh, I have 0 faith in those being immersive enough, but maybe that is my lazy eye talking. Ok, lets assume that the 3d effect is compelling, and that the issue on that front is with me.
Without head tracking, you can't look around the object. To me, at least, that has a big impact on how easily you can work with something.
You can't reach behind an object on a 3d tv. Wit ha HMD and proper hand tracking, you can. I'm also not sure that 3d tvs work well with things in front of them in general, as the occulsion seems like it would get weird.
You have to be looking at the screen. I mean this from a possible viewing angle front. If your screen is on the wall, you can't look down on your object. Conversely, if your screen is on the ground, you can't look sideways at it.
A large screen like that is not very accessible. It takes up a lot of space, and it probably fairly pricey. I can see a serious modeler working with that, but not a hobbyist or amateur. Plus, the HMD should have a wider set of alternative uses, and assuming VR takes off, people are more likely to already possess one. You can also utilize improved interface technologies designed for VR more directly. What works great for a HMD may or may not translate to a screen like that.

I will grant that it has a resolution advantage, but that seems like a transient thing, as HMD resolutions will improve.