Research Results 3 Studies for Future Broadcasting Services and Basic Technologies OUTLINE 3.1 Super-Surround Audio-Visual Systems With the aim of realizing an ultra high-definition display system with 4,000 scanning lines, we developed a 2,000 scanning line 8-million-pixel CCD camera. Research was also conducted on 3-D TV systems, on the conditions needed to produce natural, strain-free binocular type 3-D HDTV, and on a stereoscopic TV system that will reproduce natural 3-D images without the need for special glasses. Regarding advanced audio and video coding, we made further research advances on a 3-D video and audio coding scheme that presents video and sound using 3-D data derived from the subject and sound source. With the goal of a future super-surround audio-visual system, we also pursued sound field control technology to achieve room acoustical simulations. 3.2 Frequency Resources Development In preparation for revision of the international Channel Plan for satellite broadcasting services in the 12 GHz band, we carried out technical investigations, including interference calculations. As a result, at the World Radiocommunication Conference (WRC-2000) in May 2000, Japan was able to gain four additional channel assignments along with the eight channels already allocated. Further research was done on an advanced satellite broadcasting system that aims to overcome broadcasting service interruption by concentrating the beam pattern in areas of severe rain attenuation. In order to examine future broadcasting systems using the wide-band characteristics of millimeter-waves, we continued studies, which included field trials, of the propagation characteristics of millimeter-waves. 3.3 Recording With a view to application to compact large capacity recording systems for portable terminals, we conducted a wide range of studies involving disk media and heads for perpendicular recording. Magneto resistive (MR) heads and ultra-thin magnetic layer tape research was also pursued in order to realize large-capacity tape storage systems. 3.4 Imaging Devices Research on solid-state HARP imaging devices was promoted, moving toward the development of a compact, lightweight, low power, high sensitivity camera. This field also included studies concerning field emitter array image sensor with the HARP target to be used for future ultrahigh sensitivity cameras, resulting in the trial manufacture of a 128 96 pixel device, which performed successfully in imaging experiments. New photoconductive films, such as nanometer-sized silicon dot film and films using organic materials, were examined for their application potential in next generation highperformance, high-function imaging devices. 3.5 Display and Optical Devices With the goal of large and ultra high-definition display development, we began studies on FED (field emission display) element technology. Continuing research on the PDP (plasma display panel) was carried out in an attempt to increase both the luminance and the efficiency. In order to realize lightweight, easy-to-carry flexible displays, advanced studies were made of organic and inorganic electroluminescent and liquid crystal devices. NHK STRL ANNUAL REPORT 2000 Studies for Future Broadcasting Services and Basic Technologies 33
3.1.1 Ultra High-definition Video Technology Ultra High-definition Image Acquisition Technology Studies are underway on an ultra high-definition image acquisition device that is regarded as a component of the future super-surround audio-visual system. Following the development in the previous year of an experimental monochrome ultra high-definition imaging device using the 8-million-pixel CCD imager, we manufactured a prototype color imaging system using three such 8-million-pixel CCDs (Figures 1 and 2). One problem observed with the experimental imaging element was that this element performs partition parallel processing, resulting in the characteristic multiformity of individual divided signals. Various compensation methods to resolve this problem were promoted. We conducted gain and offset compensation using a scheme that utilizes a silence period as a black reference and reference beam input at a blanking period as a white reference. This realized a level correction error for adjacent pixels at the area boundary at less than the level resolution (correction sector: 8 bit) of the present hardware. Research continued on a high-speed drive CCD image sensor. A double speed drive on a 2/3-inch 2-million-pixel Hi- Vision (HDTV) CCD was realized. The CCD has a horizontal driving frequency of 148.5MHz and a vertical driving frequency of 1.16MHz. We constructed a prototype RGB camera using this image sensor and obtained a color video image in the HDTV progressive scanning (1035/60P) format. Subject-adaptive Pick-up Technology We continued research on a wide-dynamic-range camera that can shoot a high luminance object and a low luminance object simultaneously. Demonstration experiments were performed on a scheme that combined a single CCD system with a 3-CCD system, which was invented in the last fiscal year. We constructed a prototype imaging device using 2/3- inch HDTV CCDs, and verified its effectiveness. Ultra High-definition/Wide-screen Image System For the realization of a future television system with highly realistic images, research is underway on an ultra high-definition, wide-screen display system. During this fiscal year, we started research to achieve the goal of creating 4,000 scanning line images. Regarding display parameters (display size, aspect ratio, preferable viewing distance, etc.) for ultra high-definition and wide-screen display systems, we obtained general Figure 1: 8-million-pixel color imaging system camera head tendencies based on experiments conducted to derive such parameters. These included ways to obtain a desirable aspect ratio on a large display, as well as the psychological effects of adjusting variable display sizes and viewing distances independently. The results did not conflict with the experiment results conducted for HDTV development, and the respective parameters cover a wide range. With regard to a 4,000 scanning line display system, we started further studies of the effects of using such a system on both large and small displays, in addition to investigating display performance through simulations using printing. Visual Wide-field Display As for PDP stereoscopic display systems, we manufactured a prototype AC type PDP. We drastically reduced the crosstalk between right and left images in comparison with the conventional DC type PDP by using phosphor with short persistence, and also improved gradation characteristics. The construction of a stereoscopic signal processor enabled long experiments on 3D motion pictures using 3D HDTV images. Regarding projection type display systems, we developed a position detection technique having a simple structure, yet with the capability of producing seamless video presentations by using an experimental multi-screen highresolution display system consisting of four LC projectors, manufactured in fiscal 1999. CCD Mechanism shutter Camera head Lens f-50mm 2.5-inch prism 16ch R 16ch G B 16ch CCD driving circuit Analog signal processing AD conversion CCU * Digital signal processing 8 HD-SDIs Transmission, Recording, Display devices CCU * (Camera Control Unit) Figure 2: Prototype system structure 34 NHK STRL ANNUAL REPORT 2000 Studies for Future Broadcasting Services and Basic Technologies Super-Surround Audio-Visual Systems
3.1.2 3-D Hi-Vision (HDTV) System Stereoscopic HDTV, or 3-D Hi-Vision, is a stereoscopic television system that has the potential for relatively easy realization through the application of fundamental HDTV technologies. The system will provide extremely realistic and dynamic 3-D pictures by taking full advantage of the high resolution and wide-screen features of HDTV. We conducted a psychological effect analysis using a stereoscopic image, and promoted research to examine the relationship between this analysis and the spatial and temporal distribution patterns of binocular parallax. Secondary factor(easier to view ) 2D 4 6 2 9 8 7 7 9 5 3 1 6 Numbers in the figure indicate video material number 3 1 5 2 8 4 3D Subjective evaluation tests were performed using a standard 3-D test chart and past 3-D video programs and two factors were extracted, the first related to the "sense of presence" and the second related to "ease of viewing." Based on these factors, we determined trends using evaluation images (inserted figure). With regard to a comparative analysis between 2-D and 3-D images, it was revealed that 3-D images have a more significant effect on the sense of presence; regarding the secondary factor, ease of viewing, however, scores varied depending on the material. Experimental analysis of the relationship between the parallax distribution pattern and subjective evaluation test results indicated that for ease of viewing 3-D images, the upper part of the screen should be located further away from the viewer with less parallactic irregularity, and the entire image should be positioned at the back. Primary factor(smaller Sense of presence Greater) 3-D and 2-D image characteristics NHK STRL ANNUAL REPORT 2000 Studies for Future Broadcasting Services and Basic Technologies Super-Surround Audio-Visual Systems 35
3.1.3 Eye Strain Factors for Stereoscopic Image Viewing The examination of human visual characteristics for stereoscopic image and strain-free stereoscopic display conditions continued with the aim of achieving the reproduction of natural, strain-free-viewing stereoscopic images. Concerning human visual perception related to stereoscopic images, we measured the range of disparity for which a reconstructed stereoscopic image can be observed by the elderly. A stereoscopic display device, using the lenticular sheet shown in the inserted picture, was used in the measurement. The results of the evaluation test conducted on the 71 elderly subjects revealed that the viewing ability for stereoscopic images showed no difference from the characteristics already observed among younger subjects. In order to acquire appropriate display condition data to produce strain-free stereoscopic images, experiments to study the relationship between stereoscopic images and visual fatigue were also performed. Both subjective and objective evaluation tests measuring accommodation response, were conducted in the experiment. Regarding stereoscopic image presentation, we attempted to reduce influences from geometrical gaps, crosstalk, and picture frames to a minimum. This experiment clarified that visual fatigue from a still stereoscopic image, at a fixed location close to the screen, was not significantly different from the strain caused by viewing a two-dimensional image. Studies also continued on the "motion-sickness" Binocular fusion characteristic evaluation test using a 20-inch 3D image display device. discomfort that occurs when viewing stereoscopic images containing a great deal of motion at close range on a large display. We found that the physiological indexes closely indicating the degree of discomfort were the subject's heart and respiration rates. We also carried out comprehensive research on factors such as crosstalk and the video characteristic differences between right and left images, induced by an imperfect system structure. Past research results on crosstalk acceptability levels and allowable limits were compiled. Subjective tests revealed that the degree of interference caused by crosstalk tends to be greater than that of the ghost disturbances on a 2-D image, and that the desirable crosstalk visibility threshold for a natural image should be less than 1-2%. 36 NHK STRL ANNUAL REPORT 2000 Studies for Future Broadcasting Services and Basic Technologies Super-Surround Audio-Visual Systems
3.1.4 Future 3-D Television System Basic research on integral photography (IP) and holography are underway to realize an advanced 3-D TV system to produce 3-D pictures that can be viewed from any viewpoint without special glasses. Experimental Holography System Use of a beam generated from aliasing as an object beam can enlarge the viewing zone. The inserted picture shows that at the viewpoint, where only the "D" was observed with the conventional system, the entire word "3D" can be viewed with this system. Broken lines show the viewing zone of the conventional system. In general, when shooting a 2-D image, focusing on a specific object will make other objects located in the depth direction out of focus, resulting in deteriorated resolution in the area. On the other hand, integral photography (IP), which picks up and displays 3-D images, has to attain high-resolution subject images distributed in a wide area and depth direction. Research was promoted on a pickup technique that considers these factors. Such analysis revealed that constant response of the overall MTF in a wide range of depths could be obtained through focusing on an observer or at infinity. With regard to a holographic scheme, we continued with basic research to realize a moving holographic display using a liquid crystal display (LCD). The pixel pitch of conventional LCD is not small enough for holographic displays, limiting the area in which a three-dimensional reconstructed image can be observed (viewing zone), and making it difficult to perceive the images with both eyes as autostereoscopic images. Aliasing occurs in a sampled hologram displayed on a LCD, and the beams generated from such aliasing disturbs the reconstructed image observation. We had invented a method utilizing the beams from aliasing as object beams by fiscal 1999, and during fiscal 2000, we constructed a prototype system with a viewing zone four times greater than that of the previous systems. NHK STRL ANNUAL REPORT 2000 Studies for Future Broadcasting Services and Basic Technologies Super-Surround Audio-Visual Systems 37
3.1.5 Coding for 3D Audio and Video Broadcasting Systems Regarding the 3D video system, research is advancing toward a stereoscopic system that reproduces 3D video images from any viewpoint by extracting object depth information using multiple cameras. As for object depth information measurement, we attempted to improve on the measuring accuracy and reduce computing time by constructing a hierarchical depth estimation* search method (a technique for searching from low-resolution images to high-resolution images progressively). We also proceeded with examinations of a smoothing method for depth information measurement errors in order to improve the picture quality of a composed video from multiple viewpoints, and confirmed that application of a median filter 3D sound image display system using array-speakers for preprocessing can effectively reduce errors. A prototype arbitrary viewpoint stereoscopic display system was constructed to conduct composed video image evaluation tests using this smoothing technique, and it was shown that high-quality 3D video images could be obtained. For the 3D sound image reproduction system, research is being promoted to realize a super-realistic broadcasting system with the transmission of such information as the sound source location and recording site reverberation characteristics, in addition to signals from individual sound sources. During fiscal 2000, we investigated the human perception of sound image depth, and verified through subjective assessment tests that perceived distance control for sound image can be performed by adjusting the direct and indirect (reflected sound, etc.) sound level ratio. Furthermore, other subjective evaluation tests showed that smooth direction control on an array-speaker system could be achieved by adjusting the horizontal direction level difference. This is similar to the ordinary stereo sound control process. Based on these findings, we fabricated a new algorithm that can localize sound images to an area that is distant from the array-speaker. This has made feasible the localization of a sound image in front of a listener at any position in a horizontal plane. (* Retrieval of information that relates which pixel in each camera image captures light originating from a specific point in the object, in the case where the same object is shot by two cameras.) 38 NHK STRL ANNUAL REPORT 2000 Studies for Future Broadcasting Services and Basic Technologies Super-Surround Audio-Visual Systems
3.1.6 Sound Field Control We continued studies on sound field controls for use with a future super realistic audio system. During fiscal 2000, research was promoted to achieve highly accurate room acoustical computer simulations and to acquire system parameters for the arrangement of sound sources. To develop highly accurate acoustical computer simulations, we developed a division technique based on an icosahedral. This allowed the polarization of the sound ray radiation density to be equalized, which was not possible with the conventional simulation technique. The variety of sound absorption coefficients caused by the difference in the incident angle was taken into consideration, by measuring the oblique incident sound absorption coefficient at various wall surfaces. As for the arrangement of sound sources in a super realistic audio system, we examined the problem from the viewpoint of the reproductivity of sound spatial impressions, such as those caused by late reflection. The results revealed that a sense of sound spatial impression in a diffuse sound field could be realized by arranging a total of four sound sources, in the case of frequencies lower than 1.5KHz. When the front of the listener is given a value of 0 degrees, one pair of sound sources should be located symmetrically, at either an angle between 0 and 45 degree or at an angle between 135 and 180 degrees, and the other pair should be at an angle between 45 and 135 degrees. In a virtual reality audio system linked to CG images, which CG image on the screen of VR audio system we have been studying since 1999 as an application for the super realistic audio system, we incorporated a function to move the sound source in virtual space as the program commands. We have also increased the maximum number of sound sources in virtual space up to four. In addition, we fabricated a compact prototype system with four loudspeakers, which is a reduction from the previous system using eight. With the aim of reducing reverberation processing, which is the most significant load in the system, research proceeded on an important factor for the audibility field and reflection structure. NHK STRL ANNUAL REPORT 2000 Studies for Future Broadcasting Services and Basic Technologies Super-Surround Audio-Visual Systems 39