Yo-Sung Ho: Biographical Sketch 3DTV: Technical Challenges for Realistic Experiences November 04 th, 2010 Prof. Yo-Sung Ho Gwangju Institute of Science and Technology 1977~1983 Seoul National University [BS, MS] 1984~1989 University of California, SB [Ph.D.] 1983~1995 Electronics & Telecomm. Research Institute (ETRI) 1990~1993 North America Philips Labs, NY, USA (NAPL) 1995~Now Gwangju Institute of Science & Technology (GIST) 1981~Now Senior Member of IEEE, SPIE, IEEK, KICS, KSBE 1991~Now Delegate to MPEG/JVT Meetings 1999~Now Tutorial Lecturer at Various International Conferences 2003~Now Director of 2004~Now Associate Editor of IEEE Trans. on CSVT < 2 > Gwangju Institute of Science and Tech. Research-oriented Graduate School Funded by the Korean Government Founded in 1993 (16 years old) M.S. and Ph.D. Degree Programs Five Departments in Applied Science and Engineering Information and Communications, Material Science and Engineering Mechatronics, Environmental Science and Engineering, Life Science 800 Graduate Students (10% Foreigners), 105 Faculty Members All lectures are delivered in English Top ranked in terms of SCI paper publication and research fund Financial Support Tuition waved, Dormitory and living expenses are provided For more information, please visit www.gist.ac.kr Outline Trend of Broadcasting Technologies History of 3D Technologies 3DTV System: Cameras and Displays Current MPEG Activities for 3DTV Multi-view Video Coding 3D Video Processing Techniques Conclusions < 3 > < 4 > Trend of Broadcasting Technologies Evolution of TV Technologies Quality 2012 ~ ~ 2001 2002 ~ 2006 2007 ~ 2011 UHDTV Stereoscopic Realistic Broadcasting 실감방송 3DTV 3DTV Digital TV/HDTV Intelligent TV B&W TV Color TV Digital TV HDTV Color TV DCATV IPTV B/W TV Internet DMB UHDTV Radio DAB Interactivity < 5 > < 6 >
Hit of 3D Movies History of 3D Technologies 1838 1891 1908 1915 1989 2006 ANALOG Technology Era DIGITAL Technology Era 1844 1903 1950s 2000s < 7 > < 8 > 3DTV System Stereoscopic Cameras < 9 > < 10 > Multi-view Cameras 3D Display Monitors < 11 > < 12 >
Anaglyph Polarization < 13 > < 14 > Polarized Glasses Shutter Glasses Screen MUX Synchronization Right Image Shutter Glasses Left Image < 15 > < 16 > 3D Display without Glasses Parallax Barrier Parallax barrier Lenticular sheet < 17 > < 18 >
Lenticular Sheet Auto-stereoscopic 3D Display < 19 > < 20 > History of 3DAV History of MVC 2001 /12 2002 /05 2002 /12 2003 /10 2004 /10 time 2005 /01 2005 /04 2005 /07 2006 /01 2006 /04 2006 /07 time First Proposal on 3D Video EEs on 3DAV 3DAV Seminar CfC on 3DAV CfE on MVC First Draft CfP Second Draft CfP CfP on MVC Evaluation of Proposals Core Experiments MVC Work in JVT 3D Video 3DAV activities MVC works in MPEG MVC works in JVT Applications and requirements Test Data Representation format and camera parameters Fix Test Conditions EEs on 3DAV MVC Work < 21 > < 22 > Multi-view Video Coding Multi-view Camera System z Decoded Multiple Views Scope of MPEG Standardization Multiple Camera s x Multi-view Multi-view Video Video MVC Bitstream Encoder Decoder Displays 2D TV/HDTV From Y Axis Stereo Display Multi-view Display < 23 > < 24 >
Multi-view Video Acquisition Display of Multi-view Video Camera Calibration Shot of 3D Broadcasting Content Extraction of Camera Parameters Using Multi-view camera system Multi-view Auto-Stereoscopic Monitor N Stereoscopic Monitors < 25 > < 26 > Technical Problems Technical Problems Data acquisition N cameras (N=2-128 or more) How to calibrate multiple cameras? How to translate and rotate each camera? Elaborate camera control system is required (Hardware) All the camera parameters should be stored Color/luminance inconsistency among multiple views Data size A huge amount of data Raw data rate with no compression (example) VGA color video, 8 views, 30 fps, 10 sec. 1024 x 768 x 3 bytes (R, G, and B) x 8 views x 30 fps x 10 sec. = 5,662 Gbytes For 1 multi-view video, 8 CDs are required If the image resolution is HD (1920 x 1080) Over 14 Gbytes for a single 10 sec. multi-view video More than 3 DVDs for storage only < 27 > Synchronization among multiple cameras (capture, display) Transmission of a huge amount of data Is it possible to transmit raw multi-view videos in the current network? Real-time rendering Is it possible to render multi-views/free-views in real-time? Partial decoding and view /temporal/spatial random access are important Multi-view 3-D display devices No glasses, simultaneous multiple view displays < 28 > History of 3DV/FTV 2007 /04 2008 /01 2008 /04 2008 /07 time Request for FTV Work Call for 3D Test Data EEs on 3DV Vision on 3DV Preparing for CfP Vision on 3D Video Vision on 3D Video To develop a new 3D video format To support stereo or auto-stereoscopic displays To make a standard for a new 3D vodeo codec within next two years. FTV/3DV Applications and requirements on 3DV Viewing test for evaluation Limited Camera Inputs Data Format Stereoscopic displays Variable stereo baseline Adjust depth perception Data Format Constrained Rate (based on distribution) Left Right Bit Rate 3DV should be compatible with: existing standards mono and stereo devices existing or planned infrastructure Simulcast MVC 3DV Updating DERS and VSRS Auto-stereoscopic N-view displays Wide viewing angle Large number of output views 2D 2D+Depth 3D Rendering Capability < 29 > < 30 >
Free-viewpoint TV Ray-Space Representation FTV Free viewpoint functionality View generation for auto-stereoscopic displays FTV System < 31 > < 32 > N Video + Depth Challenging Issues Depth/Disparity Estimation Sub-pixel accuracy Temporal enhancement to reduce flickering effects Depth map refinement for distorted depth map Coding of Multi-view Video + Depth Map Coding structure Depth map coding scheme Bit allocation for depth map coding Intermediate View Synthesis View synthesis method for depth map distortion Filtering along object boundaries < 33 > < 34 > Camera Calibration Finding out camera parameters Relationship between 3-D object point and its 2-D image projection Form a 3x4 projection matrix P Homogeneous coordinate representation of points Camera parameters Intrinsic parameters: matrix A Extrinsic parameters: matrix R and vector t m ~ ~ ~ PM = A[R t]m Multi-view Image Rectification Why Multi-view Image Rectification? To compensate for non-ideal conditions Non-ideal conditions are due to Manual adjustment of multiple cameras Hard to use mechanical instruments for camera alignment Non-ideal conditions cause High complexity in finding pixel correspondence or matching Unclear viewpoints and viewpoint change Before After < 35 > < 36 >
Before Multi-view Image Rectification Overlapped image by nine original images After Multi-view Image Rectification Overlapped image by nine rectified images < 37 > < 38 > Color Consistency Examples Single Camera System Multi-view Camera System Race Uli < 39 > < 40 > Global and Local Properties Histogram Matching Global Property - Average brightness, histogram Local Property - Color Chart Histogram Color View 1 View 1 View 2 - Correspondence Color View 2 Differenc e View 1 View 2 < 41 > < 42 >
View Synthesis by 3-D Warping 3-D Scene Rendering Key idea of intermediate view synthesis using depth map Project the pixels of the reference view into world coordinates Re-project the scene in the world coordinates into the desired view Backward Projection Object Forward Projection + Depth Map C n-1 Desired View C n Reference View < 43 > < 44 > Generated Intermediate Intermediate View Generation Views Conclusions 3 rd viewpoint Color and Depth Intermediate images at virtual viewpoints 4 th viewpoint Color and Depth History of 3D Technologies 3DTV System: Cameras and Displays MPEG Activities on 3D Video Coding 3D Audio Visual (3DAV) Multi-view Video Coding (MVC) 3D Video Coding (3DV/FTV) Challenging Issues for 3D Video Capturing of Multi-view Images Depth/Disparity Estimation Coding of Multi-view Video + Depth Map Intermediate Virtual View Synthesis < 45 > < 46 > Acknowledgments Contact Information MPEG/JVT Contributions Prof. Yo-Sung Ho +82-62-970-2211, +82-10-3162-3669 hoyo@gist.ac.kr http://vclab.gist.ac.kr/ GIST RBRC Members GIST VCL Members (RBRC) +82-62-970-3131 http://rbrc.gist.ac.kr/ Gwangju Institute of Science and Technology (GIST) http://www.gist.ac.kr/ < 47 > < 48 >