"PTZ Tweening" for low-power low-latency head-tracking

This is for discussion and development of non-commercial open source VR/AR projects (e.g. Kickstarter applicable, etc). Contact MTBS admins at customerservice@mtbs3d.com if you are unsure if your efforts qualify.
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

"PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

SYNOPSIS: "PTZ Tweening" as discussed in this thread is a variation of "digital image stabilization" that is used here to stabilize the most recent unwarped framebuffer image relative to the tracked head position in the virtual world, in the absence of newly rendered video frames. It is essential to maintain high-speed low-latency head tracking to anchor the head position in the virtual world, even in underpowered or harsh computing environments. The method proposed here meets these requirements to a sufficient degree, even with a low-speed high-latency rendered video source, by uncoupling fast head tracking from potentially slow frame rendering. The goal here is to provide low-latency head tracking in a low-power environment, or to serve as a low-overhead backup method for head-tracked approximate lost frame synthesis in a desktop or "backtop" VR environment.

"PTZ" means "pan/tilt/zoom". Think of a camera on a tripod, for which you can pan left and right, tilt up and down, and zoom in and out.

"Tweening" means creating artificial "in-between" video frames to interpolate between different adjacent frames of "real" content. This can be used to increase the perceived framerate of captured or rendered video.

Now imagine yourself in the Oculus Rift HMD, looking at an unwarped portion of a spherical projection (perhaps from a camera with a fisheye lens). When you move your head, a nearby portion of the spherical projection is unwarped so that your view moves slightly to track your head rotation. This would be immersive. Experimental evidence shows that you can substitute a small amount of rotation for sideways movement, and you can substitute a small amount of zoom for forward or backward movement. You can even tilt the image to match head tilt to toward the shoulders. And moving toward or away from the horizon can be simulated by adjusting the zoom.

The process described above would allow creating tweened (linearly interpolated) frames that are perceived as BELONGING in between sequentially rendered frames, when the content difference between the frames is low. In the case of head tracking, we can use simple displacement mapping to simulate small head motions with very low latency, while waiting for the next "real" frame from the rendering pipeline. This allows us to render the world at a slow pace, such as when using low-power graphics hardware such as the Raspberry Pi.

We can even do this in a client/server arrangement where the PTZ Tweening is performed by the display device (i.e. Raspberry Pi) while the heavy rendering is done by a remote PC over a wireless connection.

As long as the immersive (but STATIC) image around us tracks our typical quick but small regular head movements, your brain should adapt reasonably well to any minor irregularities like a lack of parallax effects. Instead of perceiving that as motion sickness, it will just look "interesting", like when moving your head sideways while viewing a 3D movie. When the next rendered frame arrives, it may jerk a little, but not nearly as much as if you were just viewing a frozen image of the last rendered frame that you received.

Head tracking a static image will certainly be better than viewing a frozen image that does NOT track your head movements, and should keep the PERCEIVED head tracking latency low even with a low rendered framerate or slow video communications channel.

For really slow rendering updates, to help eliminate the jarring effect of a sudden new rendered frame, you can fade or morph into the new image, providing you have sufficient local processing power. But as long as your FRAME OF REFERENCE remains head tracked, it is less important that the world around you evolves and changes in larger steps, as long as it is fast enough to be perceived as motion (perhaps as low as 10 FPS for low-contrast scenery, but 16 FPS is usually consider the minimum, and motion pictures typically use 24 FPS).

The IMPORTANT point that I am trying to make is that we should UNCOUPLE head tracking from animation frame rate, and we can use PTZ Tweening to accomplish this with fairly low overhead, especially in a client/server environment (i.e. remote rendering PC).

For even simpler processing, we can do a simple "skybox" effect instead of using spherical projections. For really limited motion, we can substitute translation for small rotations. And translation can be performed in hardware, or with simple software, just by changing the beginning row of column of a flat image to be displayed. Think of sliding a video on your screen around on the screen with your head tracking acting like a mouse, but only for very small "autonomic" motions where there is little difference between rotation and translation (lateral motion). And it is easy to control zoom (resizing) with a simple differential blitter operation.

The reason that this is important to me is twofold:
1) I want to use my Nexus 7 tablet with a simple DIY "Fov2Go" style HMD accessory. PTZ Tweening can help prevent motion sickness from a low framerate.
2) I want to use my Oculus Rift connected to a Raspberry Pi computer, for low power portable applications (perhaps with the assistance of my Nexus 7, or a smart phone, or a wireless remote desktop PC). PTZ Tweening can help for the same reason, even if it is ONLY used as a supplemental backup method to provide continuous head tracking even when frames are lost due to temporary low framerates or communications bandwidth limitations.

More to come later...

Suggestions? Ideas?
Last edited by geekmaster on Thu Feb 28, 2013 1:10 am, edited 27 times in total.
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

Here are my quotes on this subject copied from other threads, gathered here for historical context:
geekmaster wrote:I have some ideas about software rendering that may be a novel approach. The latency problem with head movement until pixels hitting the eye can be bypassed with some perceptual tricks using the power of my Nexus 7 as the display. If the host PC sends an image with MORE horizontal and vertical FoV than you plan to display, quick head movements can do local pan/tilt/zoom on the current image, which will be replaced (or morphed) with a newly rendered image as soon as it becomes available. I believe that substituting an appropriate PTZ portion of the most recent image will substitute nicely for a missing NEW image, to avoid motion sickness from latency issues. Of course, the rendered image must be larger than the displayed image, to provide sufficient peripheral margin to pan into while waiting for the next rendered image. Depending on the application, this might even work okay for high-latency network connections.

The idea for this came from videos I have seen where 4 FPS video had intermediate frames inserted that used morphing instead of motion interpolation. It was a interesting "dream-like" effect, which I think would be better than motion lag.

I will find out more how effective this will be when I get around to testing it. Even though I just "invented" this idea recently, for all I know somebody else may have invented it before me. It has happened before and it will happen again. I just hope some patent troll will not prevent me from using my idea...

In summary, I think that simple head-tracked pan/tilt/zoom can compensate for high-latency rendering to prevent motion sickness, and I want to test this with my "Fresnel Lens Stack Fov2Go Clone for 7-inch Tablets"...
geekmaster wrote:As I mentioned elsewhere, I think that rendering a larger FoV than gets displayed could be useful when the rendering FPS slows down, so that head-tracked PTZ (pan/tilt/zoom) could be used on the previous image to "simulate" a higher FPS to help prevent motion sickness. Allowing fast low-latency head-tracked PTZ using the last-known image could work well for other slowdowns such as while loading assets, or during network congestion. Although it may exist, I have seen no prior art on this idea. It came to me after reading John Carmack's blog about latency issues.

It seems especially useful for my "fresnel lens stack Fov2Go clone for 7-inch tablets" that I am developing, where the simple "last received image" PTZ function could be offloaded to the tablet. As mentioned above, it would help to have some extra unused image content in the peripheral margins, in which to pan during head movement. But even without it, just panning the central area should help a lot.

So while downsampling/supersampling/anti-aliasing, we could render a little extra FoV for PTZ "backup" latency compensation. Just an idea. But worth trying. ;)
geekmaster wrote:
AdaAugmented wrote:I would anticipate problems doing this with 6DOF head tracking since parallex won't change as the image pans and rotates. It could be rather jarring if there are any foreground objects in the FOV and you end up with perspective latency that doesn't match the head tracking latency. It would be interesting to play with though, maybe it wouldn't be such a big deal.
I am concerned primarily with all those little head shakes and twitches that contribute so much to VR immersion when the VR display content moves accordingly with low latency.

It is your peripheral vision that is most sensitive to motion (and probably the source of motion sickness) and that is much less sensitive to parallax (and does not even have stereoscopic overlap for the most part).

It may be noticed by the user if PTZ (pan/tilt/zoom) is substituted for rendered frames for too long of a dropout period, but I suspect that will just be perceived as an environmental curiosity rather than potential poisoning from bad 'shrooms (or whatever)...

The whole point of even doing this PTZ substitution is to provide a backup method for when rendered frames are not available, such as dropouts, or to interpolate missing "tween" frames when using a low-power video card that can only provide less than optimal framerates (such as a tablet PC or a Raspberry Pi). I am making an educated guess that for "tourism" type VR apps, we may be able to go as low as 10 FPS rendering with PTZ interpolation. But for FPS games, prepare to get fragged.
;)

Surely, head tracked PTZ will be drastically superior to just being stuck with a frozen image frame, while waiting for the image next frame from the rendering pipeline or communications channel. Right?
:)

BTW, if anybody incorporates head-tracked PTZ into their VR software, please give me a little recognition in your credits (and maybe a free copy of your game). Thanks!
geekmaster wrote:
PasticheDonkey wrote:well i can be a passenger in a car or on a rollercoster etc. the copied view to another vr hmd is different cos you have no perspective control. on a snow board a shaking bouncing camera would probably make you sick tho. but that's why there are steady cams.
Just try reading a book or watching a hand-held TV while riding in a moving vehicle. Not so good. Perhaps reading an ebook or watching a movie (or playing a video game) while wearing an HMD (with rotation and translation head-tracking) in a moving vehicle will be a great experience though. Steadycams have heavy mass to keep their angle and translation path moving in a graceful arc. It is the quick unexpected small jarring motions that are the greatest problem, which a steadycam attempts to correct.

With an HMD, we could provide a SOFTWARE steadycam technique (as I have proposed elsewhere) using head-tracked PTZ (pan/tilt/zoom) to compensate for missing frames, and in this case to debounce the scenery so it remains coupled to the head instead of to the bouncing vehicle.
geekmaster wrote:I described elsewhere how I think we can get away with lower framerate VR rendering for tablet computers (or Raspberry Pi?). To keep the head-tracking latency low for small shaking or jittering motions (or even small head turns), we should be able to use PTZ (pan/tilt/zoom) to move around in the static image while waiting for the next "animated" image to arrive. I strongly believe that viewing low-latency head-tracked stereoscopic 3D (perhaps even a full spherical projection) will be effective at providing useful immersion even when the scenery update speed is limited. If necessary, we can use a faster IMU than what the tablet PC uses, perhaps with bluetooth or near-field communications.

Any thoughts on these ideas, guys?
geekmaster wrote:EDIT: This quote is from a thread where the new Oculus Latency Tester is being discussed, and it applicability to other CPUs such as the Raspberry Pi:
Dycus wrote:The dependency is in the game - you'll add a bit of code that flashes the screen under the sensor when you press the button. Easy stuff.

So yeah, you could use it on a Pi. I've used one before and they're awfully underpowered, though, unless you offload just about everything to the GPU. I wish you luck in getting anything more than a small room working well! Little techdemos would probably be best. I'm sure you can get something running, though. Do you have a Pi yet?
I have "too many" dev kits of all flavors, but not a Pi yet. I will order one soon. A Pi is sort of a "big boy's arduino" with the HDMI output and all that.
:D
One thing I was thinking was to use a client/server app on my Nexus 7, using the RasPi as an HDMI display device. That is where my idea of using head-tracked PTZ (pan/tilt/zoom) comes in, where the Pi lets the image track with the head during little head movements to maintain immersion, while the Nexus 7 computes the next frame.

I think that decoupling head tracking from frame rendering is VERY IMPORTANT, so that the visual image maintains very low latency with respect to head tracking, and the world can update around you as slowly as it needs to for the available hardware and power consumption budget.

Even a smartphone may be good enough to render useful interesting content at at acceptable update rate (perhaps 10 FPS or better), while the RasPi keeps the immersion going with 60 FPS PTZ.

I have been pushing this decoupling idea here for awhile, where PTZ can be a useful substitution for limited motion while waiting for slower frame updates from the render pipeline and communications channels.

Regarding underpowered, a Pi has WAY MORE power than the old 'puters like the Atari ST or Amiga, and there were some interesting and immersive games on those. The Pi should be able to draw LOTS of extra pixels. Worst case, even an Arduino should be able to supply simple but immersive wire-frame mazes (with hidden-line removal) and perhaps even some intersting procedural textures as well. You just need to remember the clever tricks used in the olden days (or dig out those "antique computer books").

One of my bigger interests is playing with some of my extensive robotic parts collection, to build some sort of exoskeleton haptic feedback suit. That would be a "killer" app -- oops, maybe a poor choice of words on that one.
:shock:
Last edited by geekmaster on Wed Feb 27, 2013 9:57 pm, edited 8 times in total.
Mystify
Certif-Eyed!
Posts: 645
Joined: Fri Jan 11, 2013 5:10 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by Mystify »

I'd be eager to hear how this works out for you in practice. I don't see how this could really be worse than than the alternative, i.e. doing nothing, unless you run into pixel switching time issues. It will also be interesting to see how low the "real" framerate can be with the PTZ tweening providing the responsiveness.
One question that needs to be address is what to do with the border where you have panned the the image away and don't have anything the replace it with. Do you smear the edge and hope its far enough in your peripheral and the hap is small enough that it doesn't matter?
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

Mystify wrote:I'd be eager to hear how this works out for you in practice. I don't see how this could really be worse than than the alternative, i.e. doing nothing, unless you run into pixel switching time issues. It will also be interesting to see how low the "real" framerate can be with the PTZ tweening providing the responsiveness.
One question that needs to be address is what to do with the border where you have panned the the image away and don't have anything the replace it with. Do you smear the edge and hope its far enough in your peripheral and the hap is small enough that it doesn't matter?
Regarding framerate, we are accustomed to perceiving continous animation for 24 FPS motion pictures. My experiments (published elsewhere) show that for eink displays that do NOT emit light AND have a slow pixel update rate, even an average framerate as low as 7.6 FPS looks MUCH better than expected.

Because the Rifts will use light-emitting LCD panels, I would recommend a faster framerate such as 24 FPS, but all a slow WORLD update rate would do is to cause the shape of the world around us to change. Our positional frame of reference in the world (important for minimizing motion sickness) would be maintained at low latency for typical small head motions over a short (between rendered frames) period of time. Even larger head motions would just cause a novel or curios distortion of parallax effects, but should pretty well compensate for things in our peripheral view, which I think is more important for our world frame of reference.

Regarding offscreen pixels, as I mentioned previously in different threads (to be quoted in the first post later), it would help to either render a frame larger than the Rift FoV to give us something extra to view when we move our heads, or we could just put "background noise" there (such as extending border pixels outward for an "ambilight" effect). That should be perceived much like walking past a huge flat video screen, which should not cause motion sickness when it is tracked to head movements with low latency.

At least that is how I understand it. And like you said, head-tracked synthetic motion interpolation is certainly MUCH better than having HUGE head tracking latency caused by missing video frames or slow framerate. PTZ Tweening provides EXACTLY the high FPS head-tracked synthetic video frames that we need.

Like I said, we need to UNCOUPLE head tracking from VR content rendering. That way we can keep our heads stuck into a stable frame of reference to minimize motion sickness, and we can update our virtual world at whatever rate our processing power and energy budget will allow. It may look strange for things around us to move in a jerky fashion, but as long as they are close to where we expect them to be when we move our heads, that will be only a minor curiosity that we will get used to as we suspend our disbelief (just like watching cartoon animation). Eventually, a good story will overcome mediocre special effects and you may even stop noticing them.

All my life, every time I really believed in something, it was proven true (or I made it come true). I believe in this idea, and I hope to see it flourish here. I may need to provide plenty of interesting and fun working examples that run on lowly hardware, but eventually others will pick up my ideas and run with them.

Enjoy! :D
Last edited by geekmaster on Thu Mar 14, 2013 8:13 pm, edited 1 time in total.
Mystify
Certif-Eyed!
Posts: 645
Joined: Fri Jan 11, 2013 5:10 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by Mystify »

I thought the reason film can get by with a low frame with without issue is motion blur. The motion for the timespan is essentially baked into each frame, and so we perceive it smoothly, whereas a video game has a sequence of crisp images, and hence needs a much higher framerate to get the same smoothness of motion.
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

Mystify wrote:I thought the reason film can get by with a low frame with without issue is motion blur. The motion for the timespan is essentially baked into each frame, and so we perceive it smoothly, whereas a video game has a sequence of crisp images, and hence needs a much higher framerate to get the same smoothness of motion.
Actually only animated films use motion blur. Real film cameras have rather short exposure times, and there is no afterimage from previous frames to mix in the new exposure on a whole new frame of film.

Even CRT monitors which DO have some persistence as the phosphor light emission decays, are still fast enough to support LCS (liquid crystal shutter) 3D glasses.

The persistence of vision (POV) occurs inside the eye, and it is independent of whether the content is from a movie or from a video game.

The reason people like high FPS (frames-per-second) in FPS (first-person-shooter) video games is to prevent jerky rotation (also a problem in movies when panning the camera across high-contrast scenes), but really, mostly to help with getting a high frag count due to the game update cycle framerate being synchronized with the video framerate in many games. The human reaction time is measured in HUNDREDS of milliseconds, so a lot of that perceived motion (other than for smooth camera panning) is subjective in regards to affecting game frag counts. Experience and repetitive familiarity can allow a much faster response time in some cases though so hard-core gaming enthusiast may vehemently disagree. High framerate helps reduce flicker in 3D shutter glasses, so 120Hz monitors are required for that. 120Hz would be much better in an HMD too, because a moving display attached to your head will increase the perception of flicker when pixels are not were you expect them to be.

In reality, we perceive flicker from the light flash frequency. We perceive continuous motion when we see more than 16 updates per second (more or less, depending on light intensity and contrast). Motion pictures were typically 16 to 18 FPS in the silent film era (more or less, depending on how much film they could afford for their hand-cranked cameras). When "talkies" were introduce, the camera framerate was standardized at 24 FPS so that the sounds encoded on the film would be correct. Sadly, even though older film projectors contained a switch to select silent 18 FPS or sound 24 FPS, most older silent films were INCORRECTLY digitized at 24 FPS, making people move around MUCH TOO FAST, which you can see on just about any old silent film. They really should be played back at their intended speed of 18 FPS (or have motion-interpolated frames inserted to upscale their framerate).

Now, where we are headed with this, is that to prevent perceived flicker, professional movie projectors contain a shutter that closes MULTIPLE times per film frame advance. That way even though the scenery only changes at 24 FPS, the flicker is at 48 Hz (or ocassionally 96 Hz). We no longer see annoying flicker at that higher frequency. And we can still see the slower animation as relatively smooth at the lower framerate.

What is really interesting is that while movie theaters uncouple flicker (48 Hz) from framerate (24 FPS), head tracking can (and should) also uncouple frame updates (60 Hz or faster) from content updates (as low as 16-24 Hz). Interestingly, the minimum head tracking latency of about 20 msec corresponds nicely to the movie flicker rate of 48 Hz. In both cases, the change in CONTENT can be slower and still be perceived as continuous motion of the OBJECTS, not of your head relative to the virtual world frame of reference. Frame of reference needs the same higher frequency needed to prevent perception of flicker, which can also induce nausea (or even epileptic seizures). It is my belief that this higher head tracking frequency can be done with PTZ Tweening, while the world can be rendered at a much slower update rate.
Last edited by geekmaster on Wed Feb 27, 2013 5:47 pm, edited 9 times in total.
Mystify
Certif-Eyed!
Posts: 645
Joined: Fri Jan 11, 2013 5:10 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by Mystify »

That was informative, thank you for taking the time to type it all out. I appreciate that.
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

Even a "super simple" method of head rotation tracking might be able to prevent motion sickness. I want to experiment with it. The simplest thing I can think of is to simply shift the onscreen image up/down/left/right to the nearest pixels (or fraction of a pixel) to track head motions.

For framebuffer hardware, you can normally set the display viewing window position in a much larger framebuffer. We can fill the framebuffer with a much taller image, and just change the window starting position to match the vertical head position. Most framebuffer support also allows setting the horizontal column, which has the effect of moving the image with wraparound at the screen edges. When that happens, you can either ignore small amounts of wrap, or overwrite them the "wrong" content with new offscreen content. For update speed, many framebuffer devices support high speed copies (blits) from offscreen video memory (often used for storage of sprites). We can store extra side content offscreen far up the framebuffer (or copy it from RAM).

With enough mapped framebuffer RAM, it is also possible to support qpel (quarter pixel) movements, by storing 16 images of the screen, with quarter pixel shifts in vertical and horizontal directions. Then you select the image that matches your H and V fractional offsets, and use the viewport start address to set then full pixel offset positions.

This could be tested with an ordinary desktop display panel, by using mouse emulation with a head tracker, and watching the mouse pointer move as if it was attached to your eyeglasses (provided it is calibrated accordingly). Attaching the background to the mouse pointer should allow you to view the world as if your monitor was a window, just with simple window dragging (and perhaps window resizing tracked with forward and backward head positions. You can use both rotation and translation (6DoF) to move the onscreen window, making it look like you are viewing a distant billboard through a window in your monitor.

This same effect should work well in the rift to lock your head to a virtual frame of reference (the other side of the virtual window), even without 3D rendering, to prevent motion sickness.

Another documented perceptual effect is that there can be significant changes in scenery that occur during an eye blink, and the majority of people WILL NOT NOTICE any changes. We could perceptually smooth parallax errors from significant head motion by delaying the next rendered frame update until during an eye blink. Perhaps not practical in an FPS game, but it would make an interesting demo running on a little RasPi, and would work perceptually better when transitioning from interpolated to rendered frames. EDIT: Eye blink detection would probably require a camera too, but a "Fov2Go-style" HMD make with a 7-inch tablet PC would have a camera and might be able to detect eye blinks. Alternatively, eye blinks may be detectable with EMG electrodes near the eybrows (perhaps built into the HMD foam pads).

I dream about this stuff... ;)

EDIT: The above description was based on stuff I did with framebuffers used in displays that did not use pre-warping. For the Rift display, it would require a pre-warp step AFTER doing any pixel shifting of the display contents. This removes the ability to shift the screen just by setting the window start position in hardware. However, it could still be used to set the window start position for the pre-warp/displacement algorithm (in other words, combine warp displacement with head-tracked offset displacement). This means we can pipeline head-tracked shifting with pre-warp on a line-by-line basis right inside the software blitter.
Last edited by geekmaster on Wed Feb 27, 2013 10:32 pm, edited 6 times in total.
User avatar
PasticheDonkey
Sharp Eyed Eagle!
Posts: 450
Joined: Sun Jan 06, 2013 4:54 am

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by PasticheDonkey »

i thought movies were 72hz. triple gated.
tbowren
One Eyed Hopeful
Posts: 22
Joined: Fri Jan 25, 2013 12:29 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by tbowren »

Interestingly I have been having the same conversation with a graphics programmer friend. The idea would be to have the engine render to a framebuffer queue that is not yet warped. At a certain point in time the engine reads the current head rotation matrix, calculates the difference from when the frame was started, offsets the stored image in the buffer by the appropriate amount, and sends it to the display.

This was talked about by Carmack's recent latency blog post the other day. He brought up the point that you must only offset the 3d world elements and then composite the UI before warping because you don't want that to shift. I had not thought of that.

When running through this idea the problem came up regarding what to do with that frame that was delayed. Do you use it for the next frame (thus increasing your input latency from the game itself) or ditch it and start a new frame at some point. You could potentially end up with the currently displayed frame being very delayed in time from when the game collected the current status.

I think this is an important idea to experiment with right now because people will be throwing games at their Rift ( and eventually higher rez Rift) and they may not have the proper hardware to support it. If the first generation of VR experiences is causing headaches and producing bad reviews, it can give the device a bad "reputation".

I agree that doing some cheats to maintain a low latency head tracking with a guaranteed high frame rate is essential until the next generation or 2 of video cards come out that are expecting the kind of demands that high rez VR displays will demand of it.

Ogre3D appears to have all the render pipeline trigger events that could make this idea possible to test. Last night I got my Razer hydra working inside the Ogre engine so I am going to try and test the idea out.
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

PasticheDonkey wrote:i thought movies were 72hz. triple gated.
Older theaters used 48Hz. New ones use digital projection. I do not know any that used 72Hz but it makes sense to do so, being the next multiple of 24Hz beyond 48Hz.
http://en.wikipedia.org/wiki/Movie_projector
Modern shutters are designed with a flicker-rate of two times (48 Hz) or even sometimes three times (72 Hz) the frame rate of the film, so as to reduce the perception of screen flickering. (See Frame rate and Flicker fusion threshold.) Higher rate shutters are less light efficient, requiring more powerful light sources for the same light on screen.
Because faster shutters are less efficient, 72Hz may be less popular than 48Hz. I had a friend who was a projectionist at an outdoor drive-in theater (when such things still existed). Their projectors used carbon arc lamps and 48Hz shutters. 72Hz not only require brighter (and hotter) lights, but the faster shutters may require more maintenance, making them less popular for economic reasons. Brighter lights may also more quickly damage or bleach the dye in color film emulsion.
Last edited by geekmaster on Wed Feb 27, 2013 8:51 pm, edited 2 times in total.
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

tbowren wrote:Interestingly I have been having the same conversation with a graphics programmer friend. The idea would be to have the engine render to a framebuffer queue that is not yet warped. At a certain point in time the engine reads the current head rotation matrix, calculates the difference from when the frame was started, offsets the stored image in the buffer by the appropriate amount, and sends it to the display.
...
I agree that doing some cheats to maintain a low latency head tracking with a guaranteed high frame rate is essential until the next generation or 2 of video cards come out that are expecting the kind of demands that high rez VR displays will demand of it.

Ogre3D appears to have all the render pipeline trigger events that could make this idea possible to test. Last night I got my Razer hydra working inside the Ogre engine so I am going to try and test the idea out.
Great! Keep us posted on your testing results. :D

Yes, the warping needs to be done after any shifting, which means that my "framebuffer offset" method mentioned previously willl not work as-is. I was thinking it could be done in an unwarped buffer, then blitted with warping into the hardware buffer. The framebuffer offset would be done by the blitter instead of setting a hardware viewport window (like I did in the olden days).

I really need to change my thinking to include pre-warp into the mix where it belongs (at the last possible moment).

My idea will work, but it will need a CPU that can do the warping (probably not an Arduino)... ;)

Obviously, if the head moves too much, the tweening errors will accumulate, making a potentially jarring transition to the next rendered frame (parallax errors), but the all-important frame of reference that prevents motion sickness should be close enough as-is. I think...

Remember that the effect of this "PTZ Tweening" cheat will be very subjective. Some people may love it and others may hate it. But the alternative (no frame updates) is even worse, so head-tracked linear interpolation will be good enough when that is all the hardware can support. Even with good hardware, it would be a nice backup plan for when a rendered frame is just not ready yet, for whatever reason.
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

The second post in this thread was a placeholder, but now it contains quotes of my various posts gathered from other threads to this new dedicated thread. You may want to read the second post if you wish to follow the historical context of this idea, as it was being developed here.

Enjoy!
:mrgreen:
User avatar
PasticheDonkey
Sharp Eyed Eagle!
Posts: 450
Joined: Sun Jan 06, 2013 4:54 am

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by PasticheDonkey »

it's basically the same principle as shooting so you can crop it later (there's a name for that i can't remember). which allows software stabilising. and in the case of a arc projection pans tilts and zooms.
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

PasticheDonkey wrote:it's basically the same principle as shooting so you can crop it later (there's a name for that i can't remember). which allows software stabilising. and in the case of a arc projection pans tilts and zooms.
Yes, this is related to camera digital zoom (cropping an oversized image from the image sensor and expanding the smaller image to fill the viewfinder), and also to digital image stabilization (which shifts that cropped image around by mulitple pixels using accelerometer feedback to compensate for motion). Most digital cameras have extra pixels around the border of the image sensor, so that you can do digital image stabilization even on a full-sized (uncropped) image.

In our case, what I am calling "PTZ Tweening" also supports digital zoom via image resizing, to compensate for small forward or backward camera motions as well. For a little extra overhead, we can also rotate the image to compensate for sideways head rotations (toward the shoulders). These are all flat 2D effects, but they should be fine for small head movements, to keep head-tracking perception at low latency even with high latency or slow scene rendering.

As you suggested, you can do software video stabilization in post-processing, using only computed motion vector analysis. I have used the VirtualDub DeShaker plug-in with excellent results for this. Some newer cameras can actually record the acceleration sensors into the video stream, allowing for low-overhead accurate image stabilization in post processing.

In our case, we are using the accelerometer (and other IMU) data to do real-time image stabilization, to anchor the head in the virtual world even in the absence of recent rendered video, by "stabilizing" the most recent image frame (side to side, up and down, forward and back). For small head motions, we can substitute translation for rotation, or vice-versa. This is all just an approximation "trick" to help maintain immersion and to help avoid motion sickness.

But what do you mean by "arc projection pans tilts and zooms"?
zalo
Certif-Eyed!
Posts: 661
Joined: Sun Mar 25, 2012 12:33 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by zalo »

When moving into unrendered territory, a blurry vignette could be implemented under the pretense of motion blur.

But you'd have to integrate it with the game's existing motion-blue to make it look seamless.
User avatar
PasticheDonkey
Sharp Eyed Eagle!
Posts: 450
Joined: Sun Jan 06, 2013 4:54 am

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by PasticheDonkey »

geekmaster wrote: But what do you mean by "arc projection pans tilts and zooms"?
i dunno i think i was just talking about normal photography, how wider fields of view give you a bigger arc view. things like cinerama have a larger arc view. it's another term for FoV really but i think more appropriate for cinema discussion. arc projection would be the projection of the arc view.
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

zalo wrote:When moving into unrendered territory, a blurry vignette could be implemented under the pretense of motion blur.

But you'd have to integrate it with the game's existing motion-blue to make it look seamless.
As mentioned previously, digital cameras with digital image stabilization often have a border of unallocated pixels surrounding the "declared" maximum resolution, and these extra border pixels are then used "when moving into unrendered territory" as you said. We can do the same in this method, by either rendering more than we need OR by displaying less than we rendered. The Rifts may already have some extra non-visible pixels at the top and bottom (depending on lens set, whether eyeglasses are worn with then, and if they were upgraded with aspheric anamorphic lenses), and these extra pixels can be used for vertical image stabilization. For the horizontal direction, we may need to draw pixels from some other offscreen region or buffer into that "unrendered territory" as needed, or we could just fudge it "amiblight" style (i.e. "blurry vignette") by duplicating edge pixels as needed. Because of lens and user variability, we may need to treat vertical border pixels the same way as horizontal border pixels.

On a side note, I added a new synopsis to the beginning of the first post, that I think summarizes this entire thread nicely:
SYNOPSIS: "PTZ Tweening" as discussed in this thread is a variation of "digital image stabilization" that is used here to stabilize the most recent unwarped framebuffer image relative to the tracked head position in the virtual world, in the absence of newly rendered video frames. It is essential to maintain high-speed low-latency head tracking to anchor the head position in the virtual world, even in underpowered or harsh computing environments. The method proposed here meets these requirements to a sufficient degree, even with a low-speed high-latency rendered video source, by uncoupling fast head tracking from potentially slow frame rendering. The goal here is to provide low-latency head tracking in a low-power environment, or to serve as a low-overhead backup method for head-tracked approximate lost frame synthesis in a desktop or "backtop" VR environment.
Last edited by geekmaster on Thu Feb 28, 2013 1:10 am, edited 6 times in total.
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

PasticheDonkey wrote:
geekmaster wrote:But what do you mean by "arc projection pans tilts and zooms"?
i dunno i think i was just talking about normal photography, how wider fields of view give you a bigger arc view. things like cinerama have a larger arc view. it's another term for FoV really but i think more appropriate for cinema discussion. arc projection would be the projection of the arc view.
Ahh... It confused me a little because the last time I was in a projection booth, we were using "carbon arc" projectors, which actually had a super bright electrical current arcing across carbon electrodes (much like the light from arc welding). I thought you were referring to just such a projector with your "arc projection" terminology...

And what do you guys think of my new "synopsis" (in my first post, and quoted above)?
User avatar
Diorama
Binocular Vision CONFIRMED!
Posts: 273
Joined: Mon Jan 28, 2013 10:37 am
Location: Brighton, UK (Sometimes London)

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by Diorama »

This is semi-relevant at best, but after growing up with grainy, shaky 3rd generation footage transferred to cheap VHS stock, this video was a breath of fresh air:
[youtube-hd]http://www.youtube.com/watch?v=5cKpzp358F4[/youtube-hd]
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

Diorama wrote:This is semi-relevant at best, but after growing up with grainy, shaky 3rd generation footage transferred to cheap VHS stock, this video was a breath of fresh air:
That video output is the same effect you get with the VirtualDub DeShaker plug-in, except I liked the option to fill in the borders with content extracted from past and future frames. The effect works quite well.

Regarding the lunar landing videos, the last I heard was that all that custom "High-Definition" video content was still "lost". Did they find the missing reels of magnetic recording tape?
http://en.wikipedia.org/wiki/Apollo_11_missing_tapes
News that these analog data tapes were missing broke on August 5, 2006 when the printed and online versions of The Sydney Morning Herald published a story with the title One giant blunder for mankind: how NASA lost moon pictures. The missing tapes were among over 700 boxes of magnetic data tapes recorded throughout the Apollo program which have not been found. On August 16, 2006 NASA announced its official search saying, "The original tapes may be at the Goddard Space Flight Center … or at another location within the NASA archiving system", "NASA engineers are hopeful that when the tapes are found they can use today's digital technology to provide a version of the moonwalk that is much better quality than what we have today." NASA also had ongoing research reasons for finding these higher resolution tapes, since Project Orion was planned to carry out tasks similar to those of the original Apollo program, to "Get a team of astronauts to the moon and back safely".
Photo of lost "High-Definition" original video content:
Image

Only known content, video recording from TV camera pointed at HD monitor:
Image
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

Additional ideas for "uncoupled head tracking":

We could use separate client/server processing, where the server does all the high-powered VR rendering, and projects that onto a "skybox" (or spherical projection) for each rendered frame. The VR content rendering does not have to be fast, and can even have significant dropouts. Then the client displays the most recently received skybox (or spherical projection) using high-speed low-latency head tracking, while applying any (optional) "de-fisheye" or other filters, and also applying the Rift pre-warp filter.

The point is that the complex VR world is rendered and projected onto a much simpler format that APPEARS the same to the end user, at any point in time. If the end user moves around in that skybox world, it will look like he is a large box with mural painted on its walls. But this mural will change continuously when new rendered frames are received.

As long as the user moves slower than the rendered frame update rate he will see no difference from viewing rendered content. But if the rendered content is too slow (or freezes for awhile), the user will see that he is in a skybox (cubical projection booth, such as a CAVE).

In any case, there should be no motion sickness, because the skybox will be head tracked fast enough to prevent illness. Uncoupling head tracking from VR rendering should help immensely with head tracking latency issues. Other latencies such as body tracking or world updates can tolerate much longer latencies.

For people who already know and use these ideas, please provide some links or references. I want to learn more about them and other related information. I prefer to get this information "out there" so we can all use it, rather than locking it away in a private toolbox to keep as a "competitive advantage". I want ALL of our new VR content to be the best it can be, so that it can grow and thrive, unlike past VR iterations that were never good enough for general daily consumption. It would be GREAT if this new VR can replace (or supplement) traditional TV and video games.
:D
Pyry
Two Eyed Hopeful
Posts: 85
Joined: Mon Aug 13, 2012 5:55 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by Pyry »

I think this might actually cause problems with head translations. Consider what happens if you're looking at a distant object (like say a mountain): as you translate your head, the visual position of the mountain should remain constant (assuming it is very far away). However, if you project this view onto a sphere and use that to interpolate, in the interpolated frames the mountain will move, and then it will snap back with every real frame, meaning that it will constantly shake or shimmer as you translate your head.

For example, if you're moving your head to the right while looking at a distant mountain (R = real frame, I = sphere projection interpolated frame):

R ----M----
I ---M-----
I --M------
I -M-------
R ----M----
I ---M-----
I --M------
I -M-------
R ----M----
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

Pyry wrote:I think this might actually cause problems with head translations. Consider what happens if you're looking at a distant object (like say a mountain): as you translate your head, the visual position of the mountain should remain constant (assuming it is very far away). However, if you project this view onto a sphere and use that to interpolate, in the interpolated frames the mountain will move, and then it will snap back with every real frame, meaning that it will constantly shake or shimmer as you translate your head.

For example, if you're moving your head to the right while looking at a distant mountain (R = real frame, I = sphere projection interpolated frame):

R ----M----
I ---M-----
I --M------
I -M-------
R ----M----
I ---M-----
I --M------
I -M-------
R ----M----
It will cause problems for large head movements when there are too many consecutive missing frames of rendered content. The goal here is to approximately simulate just enough change based on the last received rendered frame so that small head motions are tracked, allowing your frame of reference to follow your head regardless of missing rendered content. In the case of large motions (from significant missing rendered content), the VR world will appear to be painted onto a skybox (the walls/ceiling/floor of a very large room). Nearby things will not move as much as you expect them too, causing an effect similar to moving your head sideways significantly while watching a 3D movie. I do not believe that will cause motion sickness. It will be like you are anchored in a room, looking at a picture on a distant screen. But hopefully this will be rare, and new rendered scenes will replace the skybox fast enough to appear as continuous motion (at least 16 FPS), which will also make 3D objects move correctly (with correct parallax) for both eyes.

I have though about the shake or shimmer when snapping from an approximate tweened frame, but hopefully that will be minor and rare, from small barely perceptiple head motions. And if it is big, as I mentioned in other posts, we can either blend the images together (fade in), or morph them together with motion vector interpolation (MPEG-style), or in some cases perhaps delay the transition until an eye blink (which most people will not detect). The amount of processing to use depends on the power available to the display client.

With enough power in the client (HMD display server), a more rich and realistic 3D subset could be handed from the remote render PC (or render farm) to the client. I was starting from the simplest case here (simple skybox) that should be handled okay by even a little Raspberry Pi connected to a Rift HMD. Remember that the client will still have to do any pre-warp (if required). From other posts, it appears that pre-warp should not be needed for an HMD with meniscus lenses (such as high-power magnifying reading glasses).

Regarding your "mountain" example, the jumps would just look like watching a 3D movie with a low framerate. The movie screen tracks your head just fine, but the display content updates slowly or has content that freezes now and then (like a weak digital satellite TV signal). You do not get motion sickess at the movie theater (well, some people do for virtual roller coaster rides and such, but those people are probably not VR enthusiasts anyway)...

Remember, these are all just my own independent ideas that are being developed as I type them here, and as I experiment with my lenses as stuff. As far as I know, I am inventing this stuff. Eventually, when I refactor and rename parts of it, I might begin to find that others have done this long ago. Stuff that cannot be found with an early set of keywords are often found later when you refactor and rename stuff as the idea matures. It has happened to me many times before, where I thought I had invented something really cool. Usually, with enough research, you can find prior art when using the RIGHT choice of keywords...
User avatar
KBK
Terrif-eying the Ladies!
Posts: 910
Joined: Tue Jan 29, 2013 2:05 am

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by KBK »

How about correct rendering in center and leaving the PTZ to be done only with the peripheral? A potential best of both worlds. however, possibly more complex to execute. It would probably depend on the speeds involved in the given motion.

Although I am wary of anything that warps the image, even slightly, regarding being a stack on top of other manipulations. What I mean is anything that disturbs geometry as interpreted by the eye. Digital audio has taught me at least one thing:that digital decisions on fine detail, that are stacked on top of one another, do indeed end up being heard. The same tends to apply when you stack manipulations in digital video, you see them.

All of this (thread subject), I feel, will only be a problem with the first generations of the hardware/software. In the context of portables, this proposed solution would potentially be critical. But you kinda said that already....
Intelligence... is not inherent - it is a point in understanding. Q: When does a fire become self sustaining?
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

KBK wrote:How about correct rendering in center and leaving the PTZ to be done only with the peripheral? A potential best of both worlds. however, possibly more complex to execute. It would probably depend on the speeds involved in the given motion.
That is essentially what I have been saying. Render the VR world in the "center" (desktop PC or render farm), and do the head-tracked PTZ (and pre-warp if your lenses require it) in the peripheral (client/display server). We are on the same wavelength there, I think...

EDIT: Oh, wait, I think we used alternate definitions here on TWO different terms, according to two different contexts. Terminology is everything! We must be precise in defining our context when using words that might mean something else in a related but different context, such as here. I though of "center" as "computer center" (i.e. the place rendering is done). I thought "peripheral" was "peripheral device" (display client). Now, looking at it from an optical context instead of a computing context, I think you meant "center" as "middle/foveal FoV", and I think you meant "peripheral" as "outer/peripheral FoV". In that different context, I still agree, if the client processor has enough power to render to the foveal window of the HDM display. The problem is, a computer like a Raspberry Pi (which I want to support) will not have enough storage to hold the game assets for a significantly rich VR world, so it alone cannot render that foveal view.
KBK wrote:Although I am wary of anything that warps the image, even slightly, regarding being a stack on top of other manipulations. What I mean is anything that disturbs geometry as interpreted by the eye. Digital audio has taught me at least one thing:that digital decisions on fine detail, that are stacked on top of one another, do indeed end up being heard. The same tends to apply when you stack manipulations in digital video, you see them.
Pre-warping is already required by the Rift because of its use of a distorting (when held near the eye) aspheric lens. Using meniscus lenses (such as in high powered reading glasses) does not cause such distortion or require pre-warping (according to my experiments, which are not yet complete). The 7-inch display is rather shy on pixels, which means that a little blurriness can actually reduce the annoying "screendoor effect" to some degree. Some distortion is not always a bad thing. Take a musician's fuzz pedal for example. And I LOVE Moiré patterns created by optical interference distortion. But this distortion is only a side-effect of the current limits of affordable technology. And such geometrical disturbances to the eye will be perceived as novel behavior of objects in the environment. I do not think they will affect your head-tracked anchor point in the VR world frame of reference, which is the whole reason for this thread.
KBK wrote:All of this (thread subject), I feel, will only be a problem with the first generations of the hardware/software. In the context of portables, this proposed solution would potentially be critical. But you kinda said that already....
Agreed. These things will no longer be problems that concern us when we get sufficient resolution and fast enough low-power portable processors to do what we need at an affordable price.
User avatar
KBK
Terrif-eying the Ladies!
Posts: 910
Joined: Tue Jan 29, 2013 2:05 am

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by KBK »

You and I have our disagreements, or whatever.. but everybody knows that things grow best in crap. No matter how much it stinks, or how many flies are on it, it's still the best fodder. :)
Last edited by KBK on Fri Mar 01, 2013 1:25 am, edited 2 times in total.
Intelligence... is not inherent - it is a point in understanding. Q: When does a fire become self sustaining?
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

KBK wrote:You and I have our disagreements, or whatever.. but everybody knows that things grow best in crap. No matter how much it stinks, or how many flies are on it, it's still the best fodder. :P
I think we are both accustomed to getting things our own way, and having a lifetime of special priveleges (or whatever). I am trying to mellow with age, but some people become grumpy old men. It depends mostly on your outlook on life, I think...

Anyway, other than our confusion over context (which I think I figured out, in my EDIT above), it looks like we are mostly in agreement on the subject of this thread.

My point was that hackish "crude aproximate interpolation" cheats may well suffice for low-latency head tracking, keeping your head anchored in the VR frame of reference, even when the VR rendering pipeline cannot provide low latency for whatever reason. Disturbances in the VR content (such as parallax errors) should just be perceived as novelty in the objects around you, and not as a vestibular disturbance inside you. At least that is how I think about this, based on my own personal experience.

It is okay to strive for perfection in yourself and your deeds, but you must also accept mediocrity in others (and sometimes in yourself) and certainly in the lack of maturity in our technology when we choose to be on the cutting edge, when that is sufficient and all that is available at this time. It will make you a happier person in this world, at this time and place.
User avatar
KBK
Terrif-eying the Ladies!
Posts: 910
Joined: Tue Jan 29, 2013 2:05 am

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by KBK »

A second concern arose. One that came to mind, as soon as I looked into the thread. It's about how hardware manufacturers might leave this sort of 'manipulation' in place, even when the portable computing horsepower grows to a point that proper rendering can be done with no manipulations. They'd probably leave off removing such established systems and techniques (the systems proposed), and devote their new found horsepower elsewhere.

Thus manipulations in imagery could become established, even when they can be corrected out. I have no disagreement with the reasons and sentiment of the thread, nor disagree with the usefulness of the techniques, though! :)

Another side thought. The less expensive it is, the more likely it will reach children.
Intelligence... is not inherent - it is a point in understanding. Q: When does a fire become self sustaining?
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

KBK wrote:A second concern arose. One that came to mind, as soon as I looked into the thread. It's about how hardware manufacturers might leave this sort of 'manipulation' in place, even when the portable computing horsepower grows to a point that proper rendering can be done with no manipulations. They'd probably leave off removing such established systems and techniques (the systems proposed), and devote their new found horsepower elsewhere.

Thus manipulations in imagery could become established, even when they can be corrected out. I have no disagreement with the reasons and sentiment of the thread, nor disagree with the usefulness of the techniques, though! :)

Another side thought. The less expensive it is, the more likely it will reach children.
That is certainly a concern, especially when you look at things like a QWERTY keyboard which was intentionally designed as the WORST POSSIBLE LAYOUT to solve a problem with hardware limitations that were resolved long ago. It was necessary to prevent fast typing, so that type hammers in a typewriter had time to retract and would not jam together making it unusable until manually unjammed (perhaps by a typewriter repairman). Now that hardware can go faster we are still stuck with QWERTY keyboards. I had a friend who switched to Dvorak keyboards at home and at work (which are optimally designed for FAST typing of English). But he later switched back because of the accuracy problems from having to deal with QWERTY at client sites and other places, when typing is a REFLEX action that is hard to change. There are other examples, such as digital telephone services inserting white noise (hiss) on the otherwise silent line, so people do not think the line was disconnected. But hopefully the modern incarnation of VR will still be young when 1080p (or better, 8k displays) can be used in some future Rift HMD, so we can switch to meniscus lenses and stop with the nasty pre-warp stuff.

Sadly, business decisions often choose the crappier technology if it will save a few pennies, such as VHS over Beta (I used mostly Hi-8 myself, with a pair of edit decks that had Faroudja Time Base Correction and were immune to Macrovision copy protection). Hopefully, Oculus will remain a privately held company with the best product for the least cost as the driving force, so we can see periodic improvements that use the best affordable appropriate technology available.

Anyway, we need to get along, and to do that, it would help if you continue to use sensible langage that we can all understand, and try to limit things that MIGHT sound boastful or demeaning to us "commoners". It does not hurt to go back and edit your posts to make them more agreeable when you can. Thanks.

EDIT: Think of my proposal to use a low-power processor for head-tracked video as analogous to the "lizard brain", where we can still function enough to survive even with significant damage to or loss of the more evolved portions of our brains. In our VR context, I suggest that we have a low (but sufficient) power processor (like a RasPi) that drives the Rift and handles only head-tracking and pre-warp, to keep the "VR frame of reference" anchored to the head with very low latency. It supplies approximate interpolated frames when they are missing from the render pipeline (supplied from a more powerful computer or computing cluster). That interpolation can be as simple as possible to supply a sufficient perception of stability in the VR world, perhaps just a simple pan/tilt/zoom on a projected skybox (that acts like a flat 3D movie screen). The render pipeline projects the VR world onto the skybox and sends it to the display processor (RasPi), and missing frames just cause objects in the VR world to stutter (or morph) while the head contentedly looks around that VR world, safely anchored with low-latency to the VR world frame of reference. By uncoupling head tracking from rendering, it makes the whole render pipeline latency issue much less critical for the success of VR. IMHO, of course.
User avatar
TheHolyChicken
Diamond Eyed Freakazoid!
Posts: 733
Joined: Thu Oct 18, 2012 3:34 am
Location: Brighton, UK
Contact:

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by TheHolyChicken »

I think it sounds like a good idea, and certainly worth exploration. There's nothing stopping you doing the software right now if you have the skills - are you working towards this now? This is definitely something that will need real testing rather than relying on the theory :)
Sometimes I sits and thinks, and sometimes I just sits.
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

TheHolyChicken wrote:I think it sounds like a good idea, and certainly worth exploration. There's nothing stopping you doing the software right now if you have the skills - are you working towards this now? This is definitely something that will need real testing rather than relying on the theory :)
The most difficult part will not be the software development, but because this is designed to affect subjective perception, the effectiveness of this idea and process will need to be tested on a large population of people of varying cultures and backgrounds. Because Americans and East Asians are known to perceive the world differently[1], and because people have different reactions to motion sickness and simulator sickness, and because we can learn to overcome such illness with practice and repeated increasing exposure, the efficacy of this software for its intended purpose will need to be analyzed statistically for a large number of test subjects from multiple cultures, social and experiential backgrounds, and ages.

But even before such studies are performed for a large enough number of test subjects, we can provide an option that allows the end user to turn enable or bypass it, so that he can decide for himself if it helps or hinders his overall immersive VR experience.

References:
[1] http://www.apa.org/monitor/feb06/connection.aspx
Regarding this reference and claim, I remember reading years ago about a gaze-tracking study, that showed how westerners (especially Americans), when shown a photo of a panoramic jungle environment, quickly brought their attention to a tiger peering at them mostly obscured by bushes and leaves, whereas the East Asians viewed the same photo by first noticing the overall esthetics (Feng Shui?), followed by flowers and natural beauty, and eventually the dangerous beast clandestinely watching them. This may affect how we are affected by head-tracking anamolies, because which portions of our FoV we study first.

Conclusion:
Regardless of whether such studies are true, many of us DO have first hand knowledge of simulator sickness, from playing games such as Descent on a large FoV monitor or projection screen. Providing an (optional) fallback mechanism to assure that head-tracking is sufficiently anchored in the VR environment, when low-latency rendering alone is not adequate for that task, would be a good thing for many people, at the cost of a little extra processing layer (Raspberry Pi or similar, or a software layer on the rendering PC). This proposed "uncoupling" layer would go between (possibly slow or variable latency) complex rendering to a simple skybox, and head-tracking/interpolating/warping that skybox at a much faster framerate and lower latency.

EDIT: I did not order a Raspberry Pi yet, but I have invested in research time (thought experiments, documented in this thread). I need to order a RasPi soon. I have also been playing a bit with my "DIY Fov2Go clone for 7-inch tablets" minimalistic design, and I purchased some more "dollar store" supplies for its construction. If I stick with pseudo-anamorphic lenses (offset fresnel lens stacks), I will need to write my own pre-warp filter for it. I will probably build another one using high-magnification meniscus eyeglass lenses too, when they arrive. I hope that the Rift SDK/API has support for DIY pre-warp plug-ins to go along with its replaceable lens cups. Regarding my software skills: yes, but sufficient time: no. The gray side of my "leet haxor skillz" has been dormant since the DMCA and the Patriot Act went into effect, but I still prefer coding to the bare metal in C and assembler language (because I am a latent bit basher and pixel pusher).

Bit bashing secrets (with SWAR):
http://aggregate.org/MAGIC/

Pixel pushing secrets (from our beloved Michael Abrash):
http://nondot.org/~sabre/Mirrored/Graph ... BlackBook/

The Depths of Doom (John Carmack is our software god):
http://kotaku.com/5975610/the-exception ... ource-code
:lol:
Krenzo
Binocular Vision CONFIRMED!
Posts: 265
Joined: Tue Sep 07, 2010 10:46 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by Krenzo »

I don't think this is possible with the Raspberry Pi. There's no high speed digital input to the Pi ie HDMI input, only composite input. Its GPIO ports are not designed for high speed as the traces are not length matched for high speed differential signaling. USB 2.0 is the fastest input connection available.
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

Krenzo wrote:I don't think this is possible with the Raspberry Pi. There's no high speed digital input to the Pi ie HDMI input, only composite input. Its GPIO ports are not designed for high speed as the traces are not length matched for high speed differential signaling. USB 2.0 is the fastest input connection available.
I am NOT proposing processing HDMI input data, which as you said, is "impossible".

As mentioned, I want the RasPi to receive "skyboxes" from a remote rendering PC (or render farm), at a refresh rate adequate for objects in the environment to appear to be changing (or moving within the VR environment) fast enough to be perceived as relatively smooth animation (to avoid unwanted "robot dance" jerkiness from our NPCs). This skybox input stream would probably be coming from a wireless network connection between the RasPi and a nearby game rendering PC.

For strictly standalone portable use, some nice simple low-asset virtual environments that can run directly on the RasPi, such as "3D algorithmic art animations" or a head-tracked virtual theater, so you can watch movies with your HMD in a moving vehicle without getting premature motion sickness.

The Rift can use the RasPi HDMI OUTPUT, however.
:D
Krenzo
Binocular Vision CONFIRMED!
Posts: 265
Joined: Tue Sep 07, 2010 10:46 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by Krenzo »

geekmaster wrote:As mentioned, I want the RasPi to receive "skyboxes" from a remote rendering PC (or render farm), at a refresh rate adequate for objects in the environment to appear to be changing (or moving within the VR environment) fast enough to be perceived as relatively smooth animation (to avoid unwanted "robot dance" jerkiness from our NPCs)
With an uncompressed 24-bit 720p image source, each frame is (1280*720*24) 22.1 Mbit. Raspberry Pi's USB can handle 248 Mbit/s (31 MB/s). That will give you 11 fps. After rereading your first post, I see that that frame rate is perfectly acceptable to you. Personally, I don't see the benefit in working with such a low frame rate. I think the best use of this technique would be to remove latency from trackers and wireless video for gaming where a higher frame rate is necessary.
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

Krenzo wrote:With an uncompressed 24-bit 720p image source, each frame is (1280*720*24) 22.1 Mbit. Raspberry Pi's USB can handle 248 Mbit/s (31 MB/s). That will give you 11 fps. After rereading your first post, I see that that frame rate is perfectly acceptable to you. Personally, I don't see the benefit in working with such a low frame rate. I think the best use of this technique would be to remove latency from trackers and wireless video for gaming where a higher frame rate is necessary.
I have not yet played with a Raspberry Pi and I do not know its limitations. Are you saying it cannot output 60FPS video? If so, then we clearly need a more powerful device that DOES meet our needs as described above, but substitute the name of this device wherever you see "RasPi" (or equivalent) in my recent posts on this topic.

If you are ONLY concerned about USB, who said anything about USB? And why would anybody try to send UNCOMPRESSED video over that and expect to reach full HDMI speeds? I sure did not suggest any such thing. Because the skyboxes only need to be updated at slower "Persistence of Vision" speeds (such as 16Hz or more), and because the skybox is so far away, little will change between them and they can be updated incrementally (with little priority to stuff not near the current FoV, and almost nothing to stuff behind you).

The concept of using a skybox is so that a PORTION of a model of it can be maintained inside the "RasPi" (or whatever). This portion only needs to be what the user can currently see, plus a little more around the borders to allow for small head motions while waiting for the next "skybox frame". I only used the term "skybox" so you could envision how it would be projected, and how it could be mapped to a pre-warp SBS-Half pair for the Rift.

As I intended here, the Raspberry Pi is just a suggested processing TOOL. I would be perfectly happy to use whatever device has sufficient power to meet the minimum sufficent requirements for immersive head tracking, which may work well just by shifting (with pan/tilt/zoom) the framebuffer (extended to at least one wall of a large skybox), using as little processing power as required. If a RasPi can do that, then great. If not, I used the term "RasPi" with the intention that it also includes more-capable devices, as suggested in previous posts. My posts are already wordy enough, without needing to include weasel-word disclaimers every time I mention the device that does head-tracked skybox manipulation.

If you are correct and a RasPi cannot even do simple skybox manipulation at 60Hz, it would still be worth using it for OTHER cool little demos when used with the Rift. But then we need a more powerful portable device to do the things I mentioned in previous posts here.

Nothing has changed, except perhaps the NAME of the device referred to in my posts above. I will know more when I get a RasPi, and I try doing what I suggested using various methods, with or without its onboard GPU. I will be disappointed if a RasPi cannot do 60 FPS video as you suggested above... But I will find something suitable to replace it then.

To be more clear would require the use of even more words, and these posts are wordy enough...
;)
These posts are not intended to be detailed plans. They are just ideas that will be experimented with and expanded as time permits. They give you food for thought, which is enough for people with sufficient background and experience to duplicate my experiments and to use my ideas. There are certainly rough edges to be dealt with, but I feed confident that they are solvable. It does not matter if a RasPi is powerful enough, because we can substitute something else that IS powerful enough. Again, although these thoughts and ideas are new to me, I am probably not the first person to think them...
:lol:
Krenzo
Binocular Vision CONFIRMED!
Posts: 265
Joined: Tue Sep 07, 2010 10:46 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by Krenzo »

I was pointing out there are limitations if you wanted to use a Raspberry Pi, namely that you're stuck with using USB 2.0 as your input bus. I don't know of any other dev boards that would allow anything better than what the Pi offers unless you go with FPGA.
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

Krenzo wrote:I was pointing out there are limitations if you wanted to use a Raspberry Pi, namely that you're stuck with using USB 2.0 as your input bus. I don't know of any other dev boards that would allow anything better than what the Pi offers unless you go with FPGA.
Well, even the WiFi input (USB dongle) I suggested is USB limited, but no matter WHAT input source it does not make sense to push raw uncompressed video over it. USB is not the limitation. Choice of data representation and choice of algorithm is EVERYTHING. And as I mentioned, the RasPi was only a example. I would like to see just how much of this we can do with a RasPi, but there is no reason to limit ourselves to that specific hardware.

If you haven't, you really SHOULD read Jon Bentley's "Programming Pearls" books. After that, it should be obvious that communicated updates to the (partial) skybox model should be differential, limited to areas near the viewing region, and perhaps even compressed. There are some EXCELLENT fast compression methods that work well on ARM processors. One that I have ported is based on LZ4 compression.

You only need enough compression to keep the data representation models synchronized between the render PC and the RasPi (or whatever gets used to head-track and drive the Rift). We can worry about such details later. It takes enough words just to describe preliminary basic concepts. Getting buried down in details too soon can impose artificial limts on a project. We can always upgrade our hardware (including the RasPi, or whatever). Details such as RasPi or USB only affect our choice of data representation or algorithms, but not our basic concepts (and we are still working at the conceptual level here). Specific details are premature at this time.

What I am describing CAN be done, and I plan to do it. I am just describing my thoughts and ideas openly and freely here and now. Specific constraints such as hardware choices can be decided later, and even then they are subject to change if something better comes along.
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

Here is a nice example of "PTZ Tweening", as I describe it here:
http://www.mediavr.com/artgallerystereoana.htm
Use your mouse wheel to zoom in and out, and you left mouse button to pan and tilt the view.

This could be handled by a low-latency head-tracked process, independent from the rendering app that delivers new content (allowing it to be much slower that head tracking and PTZ).
geekmaster
Petrif-Eyed
Posts: 2708
Joined: Sat Sep 01, 2012 10:47 pm

Re: "PTZ Tweening" for low-power low-latency head-tracking

Post by geekmaster »

Here are some demos of what I was talking about:
http://priteeboy.deviantart.com/art/Vue ... -209276862
http://priteeboy.deviantart.com/art/Vue ... -186873878
http://priteeboy.deviantart.com/art/Vue ... -190720438

Use the arrow keys to control pan and tilt, and the Ctrl and Shift keys (or mouse wheel) to control zoom.

interestingly, while moving the image around, the outer corners look a little stretched (exactly what the Rift needs).

I resized my browser window so that the image just filled my FoV when viewing through a 2-inch 5x aspheric lens (recommended for DIY Rift clones), with my face very close to my monitor. It was virtually perfect. The lens warped the corner distortion back where it belonged. This example would look great in a Rift, even if both eyes saw the same view. Even better with a stereoscopic pair controlled like this.

My plans are to do exactly the same thing (with head tracking), while some background rendering process replaces the viewed image periodically (same computer, or different computer(s) on network, running the "game", preferably fast enough to maintain the illusion of continuous motion). That way, even if the image does not update quickly, or has drop-outs, you can still look around in the current image.

But just for tweening (to compensate for delayed or missing rendered frames), we do not need the full 360-degree view. Perhaps just an extra 15-degrees beyond the screen edges may be enough, if replacement images centered on the current viewpoint arrive quickly enough.

While the rendering device sends new frames to the head-tracking and viewing device, we could still send a full 360-degrees, but we may need to prioritize the data so that pixels nearest the viewpoint are sent soonest/fastest, while stuff behind us is low proirity and can get filled in when bandwidth becomes available. We could do progressive resolution enhancement on the stuff behind us, so it starts low-resolution (for peripheral vision if we turn quickly), and gets progressively finer detail (ready for when we turn around).
Post Reply

Return to “VR/AR Research & Development”