Research & Development on 3DTV at NHK Keiichi Kubota Science & Technology Research Laboratories Japan Broadcasting Corporation (NHK) June 2, 2010 1
NHK Science and Technology Research Laboratories Founded in 1930. This year is NHK Laboratory s 80th anniversary. The purpose of the foundation is to start research on television for Tokyo Olympic Games that was planned in 1940 but was cancelled because of the war. NHK Laboratory have promoted research and development that is related to the entire field of broadcasting technology. Major research outcomes are satellite broadcasting, HDTV, digital broadcasting, and flat panel display. 1930-1961 1961-2002 2002-2
Super Hi-Vision System Super Hi-Vision has extremely high spatial resolution, 33 million pixels per frame, and it can provide viewers with stunning images. Various parameters other than spatial resolution is under investigation now. The wide visual angle of 100 degrees provides viewers with an immersive feeling. The purpose of Super Hi-Vision to home is to provide a totally new viewing experience to enjoy wide and extremely high resolution images from any viewing distance. 7680 100-200 inch Viewing distance: 1-2m 4320 65 inch Viewing distance: 65cm Viewing distance : 0.75 x Picture height Viewing angle : 100 degrees A3 wide Viewing distance: 25cm 3
Targets and Roadmap of Super Hi-Vision R&D To start an experimental broadcasting in 2020 using a broadcasting satellite in 21 GHz band. To develop an international standard for the Super Hi-Vision signal format that reflects R&D on new specifications on frame rate, opto-electronic transfer characteristics, number of bits, and colorimetry. To develop a terrestrial broadcasting system for the future Super Hi-Vision service beyond 2020. 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 Target London Olympic Rio Olympic Experimental Broadcasting in 21 GHz band Camera 2.5 60Hz Full Pixel Prototype, 2.5 120Hz Full Pixel 1.25 60Hz Full Pixel Display Compression Recording Prototype 100-150 100 Mbps, Dual-Green Improved Prototype 100 New coding scheme for 120Hz 200 Mbps, Full Pixel Commercial 65 Prototype 100 Sheet-type HDD for studio use (non-compressed, 2 hours) Optical disc for home use (compressed, 1hour) 4
Camera and Display of Super Hi-Vision 1 st Generation Camera (2002) CCD with 8-million pixels x 4 2.5 inch optics, 80 kg 2 nd Generation Camera (2004) CMOS with 8-million pixels x 4 1.25 inch Optics, 40 kg 3rd Generation Camera (2010) CMOS with 8-million pixels x 4 1.25 inch Optics, 20 kg Super Hi-Vision Flat Panel Display Spring of 2011 Dual-Green Projector (2002) 3840x2160 LCD x 4 Full-Resolution Projector (2009) 8192x4390 LCOS x 3 Full-Resolution Camera (2010) CMOS with 33-million pixels x 3 2.5 inch Optics, 80 kg 5
Research on 3DTV at NHK 1960 1980 1990 2000 2010 Basic Research Exploring various 3D systems Visual comfort/discomfort Visual fatigue Phase 1 Phase 2 Phase 3 - Parallax barrier system - 3DTV systems - Stereo vision - Human factors etc. First demo of 3D HDTV (1989) Demo of 3D HDTV at NAB(1991) -Requirements for comfortable 3D viewing (system and programs) - Parallax distribution analysis Full parallax, No glasses Integral photography Holography Phase 4 Electronic Holography Integral 3D TV 6
3D HDTV System 3D HDTV demonstration at NAB 1991 using dual CRT projectors Compact 3D HDTV camera (1998) 7
Major Visual Depth Cues Convergence: Angle of visual lines Binocular Disparity: Image difference caused by viewpoint difference Accommodation: Focusing of the lens reflects distance to object Motion Parallax: Image difference according to the change of viewpoint 8
3DTV Systems and Visual Depth Cues System Conergence Binocular Disparity Motion Parallax Accommodation Psychology Stereoscopy Two cameras and two displays Multi-viewpoint Multi-camera and multi-display Voxel display Arrange displays along depthaxis Spatial image reproduction reproduce field of light rays from objects Optical and electronic technology 9
Human Factors in Stereoscopic 3DTV NHK conducted research on major human factors such as (un)naturalness, visual comfort, and visual fatigue, and clarified relations between these factors and shooting and display parameters. (1995-2003) Subjects Naturalness Visual comfort Visual Fatigue Human factors Depth perception Fusion of L&R images Inconsistent convergence & accommodation Physical factors Depth distortion Parameters difference between L&R cameras, display Excessive disparity Excessive disparity change 10
Visual Fatigue Caused by Stereoscopic 3DTV -1- Stereoscopic viewing is different from natural viewing in the sense that convergence and accommodation points do NOT necessarily coincide. Convergence = Accommodation = A A A screen B Convergence = A Accommodation = B Natural view Stereoscopic view 11
before after one hour viewing after rest after two rests before after one hour viewing after rest after two rests before after one hour viewing after rest after two rests before after one hour viewing after rest after two rests Relative fusion width Visual Fatigue Caused by Stereoscopic 3DTV -2- Subjective evaluation of visual fatigue caused by discrepancy in vergence and accommodation. 融合幅比 1.2 2D 平面画像 still image 3D 両眼視差が still image with large 一定で大 disparity p=0.020 3D 両眼視差の image with small 3D 両眼視差の image with large time 時間変動が小 varying disparity time 時間変動が大 varying disparity p=0.003 1 0.8 0.6 視観聴前視前 視観聴直視後直後 wide rfw 融像幅広 narrow rfw 融像幅狭 11 回休憩憩後後 22 回休憩憩後後 観視聴視前前 観視聴視直直後後 11 回休休憩憩後後 22 回休憩憩後後 観視聴視前前 p=0.048 観視聴視直直後後 11 回休休憩憩後後 22 回休休憩憩後後 More fatigue for 3DTV than 2DTV More 時刻 fatigue for time varying 時刻 disparity than 時刻 3D still image More fatigue for larger time varying disparity Recovery after some period of rest in the short range experiment 観視聴視前前 観視聴視直直後後 時刻 11 回休休憩憩後後 22 回休休憩憩後後 12
Considerations on Stereoscopic 3DTV Carefully produced 3D motion pictures have a power to afford special experience to the viewers. In case of broadcasting, especially free over-the air broadcasting, the following points should be considered: Visual fatigue Uncontrolled viewing conditions including viewing distance, display parameter differences, viewing time, and audience profile. Compatibility You need to consider those who do not want to watch 3D programs or those who only have 2D display. Image quality and acceptance Will viewers accept special glasses and image degradations due to stereoscopic 3DTV viewing conditions? 13
3DTV Systems and Visual Depth Cues System Vergence Binocular Disparity Motion Parallax Accommodation Psychology Stereoscopy Two cameras and two displays Multi-viewpoint Multi-camera and multi-display Voxel display Arrange displays along depthaxis Spatial image reproduction reproduce field of light rays from objects Optical and electronic technology 14
Spatial Image Reproduction System Spatial imaging systems do not require special glasses to see 3D images, because an object is reconstructed as a spatial image. Two kinds of methods, Holography and Integral method. System structure is simple, but the requirement for the imaging devices and display devices is very high, and devices that are available now do not meet the requirement. Image s appearance changes 上下左右 as どthe の viewer s 方向に移 position moves しても画像が変化する Reconstruction 光線群を再現 of light Display 表示装置 Device Viewer 寝ころんだお兄さん 被写体が立体空間像と Object reconstructed as a spatial image して再現される 15
Integral 3DTV System display Lens array A group of rays forms a real image Imaging method Shoot and display through lens array System requires a camera and a display of very high resolution to reproduce an adequate 3D image. 16
Integral 3DTV System Prototype Integral 3D TV system based on a SHV with 33 million pixel Depth control lens Converging lens Diffusion screen Reconstructed image I P B G I P Object Virtual Images GRIN lens array 250(V) x 400(H) R UHDTV camera (4000 scanning lines) UHDTV Projector (4000 scanning lines) Full-pixel structure 33M pixels (7680(H)x4320(V)) This prototype has been developed with JVC Kenwood Holdings Inc. of Japan A part of this research is supported by National Institute of Communications Technologies. 17
Reconstructed Images by Integral Method -1- Upper Viewpoint Left Viewpoint Right Viewpoint Lower Viewpoint 18
Reconstructed Images by Integral Method -2- Reconstructed spatial 3D image Projected image on a diffuser set 10 mm away from lens array Projected image on a diffuser set 40 mm away from lens array 19
Prototypes of Integral 3DTV System 100,000 Full resolution SHV based (2009) 33M pixel full resolution, 400x250 lenses 10,000 50,000 4Kx2K system based 4M(Dual-green)/8M, 2002 Full 8M. 2004 160x117 lenses Camera 1 HDTV based Display Camera 2 HDTV camera/ SXGA LCD 54x59 lenses (1997) Dual-green SHV based (2007) 16M(Dual-green), 182x140 lenses pixel count of the base video system 2M 8M 16M 33M 20
Demonstration at NAB 2009 21
Future of Integral 3DTV 16Kx8K system based (201X) 300,000 (NTSC) 100,000 SHV(8Kx4K) based (2009) 10,000 8M 16M 33M pixel count of the base video system 132M 22
History of R&D on Television in Japan 1925 1926 1930 1940 1953 1960 1964 1989 2000 2003 2011 Radio NHK Labs B&W TV Broadcasting Tokyo Takayanagi イ (Cancelled) Color TV Tokyo BS Analog BS Digital Terrestrial Digital Analog Termination HDTV 1964 Research Start 1984 LA 1985 Tsukuba Expo 1988 Seoul 1994 1998 2000 MUSE BS Digital Nagano Olympic games have always been the target for the broadcasters to develop new technologies. HDTV debuted at LA Olympic Games and Tsukuba Expo 20 years after the start of R&D. Then 15 years after LA Olympic, HDTV became a full service in Japan. Super Hi-Vision debuted at Aichi Expo 10 years after the start of R&D. When will it be a full service? 20 years 15 years Super Hi-Vision 3D-TV (Spatial Reproducing) 1995 Research Start 10 years 1996 Research Start 2005 Aichi Expo 2012 London 2016 Rio de Janeiro 23
Thank you for your attention. 24