EBU-ITU-SMPTE 3DTV Workshop

[DmitryKo]

Some interesting quotes from the presentations posted at the EBUs 2009 3D TV Workshop,
http://tech.ebu.ch/events/3dtv09" onclick="window.open(this.href);return false;


The 3D TV toolbox - the results of the 3D TV NoE

Code: Select all

•Stereoscopy is the simplest and oldest technique
•It does not create physical duplicates of 3D light
•Quality of resultant 3D effect is inferior
	•Lack of parallax
	•Focus and convergence mismatch
	•Mis-alignment
•“Motion sickness” type of a feeling (Eye fatigue)
•Main reason for commercial failure of 3D techniques


•Mutli-view video provides some horizontal parallax
•Still limited to a small angle (~20-45 degrees)
•Jumping effect
•Viewing discomfort similar to stereoscopy
•Requires high-resolution display device
•Leakage of neighboring images


3D Video R&D
Long-term (10+ years):
– Realistic/Ultra realistic displays
– “Natural” interaction with 3D displays
– Holographic 3D displays, including “integral imaging” variants
– Natural immersive environments
– Total decoupling of “capture” and “display”
– Novel capture, representation and display techniques

3D TV situation for consumer electronics manufacturers

Code: Select all

Conclusions of CEA’s 3D Task Force (to date)
1. The general scope of CEA's work should focus on the interfaces between consumer sources, sinks, repeaters, converters, and glasses. CEA should not address the selection or recommendation of a specific 3D encoding format for use in the content creation, production or distribution chain.
2. Extend CEA-861 to support the carriage of 3D content and bidirectional 3D format signaling. In order for sources and syncs to be interoperable, CEA will study the possibility of whether there can be at least one mandatory 3D content signal format required of sources and sinks for “3D-READY” products.
3. Investigate the development of standards for 3D glasses including interface, signaling, setup, control and polarization.
4. Investigate the need to document terminology that conveys interoperability to the consumer.

Coming Up
3D Techzone at International CES, January 2010

MPEG developments in multi-view video coding and 3D video

Code: Select all

Stereo and Multi-view Video Coding
•"Classic" Stereo requires only two views which are taken "as is" – i.e. the capture must already take display properties into account
	•Compression of stereo video is straightforward
	•Simulcast
	•Combination of two views into one
•Exploitation of inter-view redundancy
•This does not support
	•N-view displays (autostereoscopic, holographic)
	•Additional functionality: Baseline adaptation
•For these purposes, either coding of multiple views (if available) or depth-based synthesis is needed

Multi-view Video Coding (MVC)
•Multi-view and 3D video representations require multiple synchronized video signals that show the same scenery from different viewpoints
•Huge amount of data with need to be compressed efficiently
•Multiview typically has a larger amount of inter-view statistical dependencies than stereo

Stereo Video
•MPEG-4 part 10 AVC Stereo SEI message and Frame Packing Arrangement SEI message (the latter in 14496-10/5e Amd.1, to be finalized by July 2009) allow various methods of L/R packing
	•Temporal, spatial row/column, spatial side-by-side/up-and-bottom, checkerboard (quincunx)
•MPEG-4 AVC Stereo High Profile (new in Study 14496-10/5e Amd.1, to be finalized by July 2009)
	•Subset of MVC, restricted to 2 views, allows progressive and interlaced stereo

MVC Standard – Profiles and Levels
•Multiview High – finalized, part of original AVC amendment Supports same subset of coding tools for inter-view prediction as the existing High Profile of AVC (but no interlaced)
•Stereo High – draft spec in 14496-10/5e Amd.1, expect to finalize by July/October 2009 
Includes support for interlaced and limits number of views to stereo only

3D 4 U

Code: Select all

Stereoscopic 3D Formats

side-by-side interleaved

• Standard format for 3D Cinema (+)
• Standard format for glasses-based consumer displays (+)
• No support for non-glasses-based multiview displays (-)
• Allows adjustment of zero-parallax (+)
• No scaling of depth (-)
	– No adjustment to display size
	– No personal preferences, kids mode
• No occlusion information
	– No motion parallax


Video + Depth

• Successful demonstrated by ATTEST project (2002-2004), MPEG-C Part 3
• Not the standard format for 3D Cinema (-)
• Depth-Image-Based-Rendering
	– Support for stereoscopic glasses-based consumer displays
	– Support for non-glasses-based multiview displays (+)
	– Allows scaling of depth (+)
		• Adjustment to display size
		• Personal preferences, kids mode
	– Views must be extrapolated (-)
• Allows adjustment for zero-parallax (+)
• No occlusion information (-)
	– Reduced quality of depth-image-based-rendering


Layered Depth Video - LDV
Video with per-pixel depth map and occlusion layer with depth map
• Not the standard format for 3D Cinema (-)
• Depth-Image-Based-Rendering
– Support for stereoscopic glasses-based consumer displays
– Support for non-glasses-based multiview displays (+)
– Allows scaling of depth (+)
– Views must be extrapolated (-)
• Allows adjustment for zero-parallax (+)
• Provides occlusion information (+)
– Better quality of depth-image-based-rendering


Depth Enhanced Stereo (DES)
2 videos with per-pixel depth map and occlusion layer with depth map
Left View Right View
• Not the standard format for 3D Cinema (-)
• Easily usable for stereoscopic glasses-based consumer displays (+)
• Depth-Image-Based-Rendering
– Support for non-glasses-based multiview displays (+)
– Allows scaling of depth (+)
– Views are interpolated or extrapolated
• Allows adjustment for zero-parallax (+)
• Provides excellent occlusion information (++)


Multiple Video-Plus-Depth (MVD)
Interpolation of Intermediate Virtual Views from
Multiple Video-Plus-Depth (MVD)
• Not the standard format for 3D Cinema (-)
• Easily usable for stereoscopic glasses-based consumer displays
• Depth-Image-Based-Rendering
– Support for non-glasses-based multiview displays (+)
– Allows scaling of depth (+)
– Views are interpolated (+)
• Allows adjustment for zero-parallax (+)
• Provides good occlusion handling due to redundant information

Conclusions
• Stereoscopic 3D is the de-facto standard from 3D-Cinema
– Directly usable for glasses-based displays, but no scaling of depth
– Not usable for non-glasses-based displays
• A 3D distribution format must be generic for all display types
– Support for stereoscopic displays
– Support for multi-view displays

• Computer vision solutions can be used for formats conversion
– Estimation of depth maps
– Generation of virtual views for multi-view and for adjustment of baseline

Issues in broadcast delivery of 3D

Code: Select all

Potential Delivery Methods
• Frame Compatible
	• Side-by-side
	• Over/Under
	• Line Interleaved
	• Checkerboard
• 2D Compatible
	• Simulcast
	• MVC/Interrelated Views
	• 2D+DOT
	• 2D+Difference (TDVision)

Common Characteristics of Frame Compatible
• All can be deployed right now
• All can be reformatted upon entry into the display
• Full resolution can be easily added using layering techniques
• similar to JPEG2000 and AVC/SVC/MVC
• All frame compatible systems can be delivered from the network to the home including existing mezzanine level compression systems
• Standards impact is trivial


Common Characteristics of 2D compatible
• All methods require new consumer devices for 3D
• 2D is available
• All require increased relative bandwidth
• Ranges from 60% to 100% extra channel capacity
• Offers full resolution once deployed
• Recoding at the local station or MSO is problematic
• New broadcast infrastructure is required
• Some need new standards or are limited to specific standards


2D + Depth
• 2D Compatible
• Characteristics
• Stereo (multi-view) images need to be rendered from a depth map
• Creating depth map is a difficult task !
• Occlusions and reveals are a weakness
• Specified by MPEG as MPEG-C
• Codec independent


AVC Inter-related (MVC) Streams
• 2D compatible
• Characteristics
• AVC core compression; MPEG-2 supports but not used
• MVC (Multi-View Codec) describes this method with many views
	• Efficiency gain of 20% has been shown (over simulcast)
		• If first stream needs 10 Mb/s, second stream needs 6 Mb/s
	• Two 10 Mbps streams nets at 16 Mbps
	• Ten 10 Mbps streams nets at 64 Mbps
• New infrastructure
• Requires new consumer devices


Technical Summary
• Frame compatible systems can be deployed now
	• Deployment costs are small
	• Fits in the current channels and infrastructure
	• Only the consumer needs to buy new equipment
	• Full resolution can be added later bundled with other features
• 2D compatible systems require buying new equipment
	• Content production weaknesses for some systems
	• Distribution bandwidth is almost twice the 2D bandwidth
	• Both the consumer and provider need to buy new equipment
	• Full resolution available when 3D is fully deployed

Report of the SMPTE Task Force on 3D to the home
An updated version of presentation discussed in SMPTE Home 3D Task Force: useless

[cybereality]

Cool, sounds like they've got all the bases covered.

I am really interested to here more about this:

Layered Depth Video - LDV
Video with per-pixel depth map and occlusion layer with depth map

It sounds like just what I was thinking about for converting 2D video to 3D. No need to reinvent the wheel if this already exists.

[funkee]

Cool, sounds like they've got all the bases covered.

I am really interested to here more about this:

Layered Depth Video - LDV
Video with per-pixel depth map and occlusion layer with depth map

It sounds like just what I was thinking about for converting 2D video to 3D. No need to reinvent the wheel if this already exists.

Several companies, but most notably Philips 3D Solutions have researched this topic and have built demonstrators to prove their point. At this moment 3D SOlutions does not exist anymore, but 3D4YOU is still working on the topic.

[martinlandau]

Yah that depth map stuff looks really cool.

"Stereoscopic 3D is the de-facto standard from 3D-Cinema"

It must be nice when you have a pretty much predefined display type. Let me think of all the ways I viewed digital information today. LED billboards, 37 inch tv, 72 inch tv, 15 inch lcd, 8 inch lcd, 3 inch lcd, 1 inch hmd lcd, 10 ft projection CRT. I know manufactures need to start expanding thier horizons. I use polarized driving glasses to reduce sun glare, but some GPS screens (you know - something you use when driving a lot) polarized the wrong way and I can't wear my glasses and use the GPS screen at the same time - talk about a screwup and shortsighted thinking by the engineers! This reminds me of a snafu that happened at IBM once. They had designed this laptop keyboard called the butterfly, it was really cool - it expanded when you opened the laptop up and let you get a large keyboard on a small laptop. Only the idiot engineers forgot to test it with certain PCMCIA cards plugged into the laptop. Many IBM PCMCIA cards were too big for the butterfly keyboard to expand when plugged in! GOOD LORD! Talk about angry customers who couldn't understand the engineering geniuses that let this get out the door! LOL!

[BlackShark]

Cool, sounds like they've got all the bases covered.

I am really interested to here more about this:

Layered Depth Video - LDV
Video with per-pixel depth map and occlusion layer with depth map

It sounds like just what I was thinking about for converting 2D video to 3D. No need to reinvent the wheel if this already exists.

Several companies, but most notably Philips 3D Solutions have researched this topic and have built demonstrators to prove their point. At this moment 3D SOlutions does not exist anymore, but 3D4YOU is still working on the topic.

How are you supposed to create an occlusion layer when this data is not seen from the camera ?

[DmitryKo]

How are you supposed to create an occlusion layer when this data is not seen from the camera ?

They use image processing algorythms which take multiple angle views of the same scene and try to reconstruct depth information, as in Multiview Video Coding compression. This particular setup uses at least 5 different cameras (see page 13 of the presentation), as stereo-only (2-view) capture wouldn't provide enough information for high quality processing.