Table of Contents. Greetings 1. Accomplishments in NHK Science & Technology Research Laboratories Outline 50

Transcription

1

2 Table of Contents Greetings 1 Accomplishments in Next-generation broadcasting media Super Hi-Vision Super Hi-Vision format Cameras Displays Coding Satellite transmission technology Terrestrial transmission technology Wired transmission technology Audio systems providing a strong sense of presence Three-dimensional television Integral 3D television Generating 3D content from multi-viewpoint images 13 2 Advanced digital broadcasting services Technologies for fusing broadcasting and communications Hybridcast TM SNS, CGM Security platform Transmission technology for handheld and mobile devices Technology for improving reception Enhanced ISDB-T 21 4 Advanced contents production technology Networked program production systems Video production technology High-quality speech synthesis Acoustic devices Contribution transmission technology Millimeter-wave mobile camera Millimeter-wave TV camera 33 5 Devices and materials for broadcasting Advanced imaging systems Super-high-sensitivity image sensors High-frame-rate imaging devices Organic image sensors Advanced storage systems Magnetic recording technology Optical storage technology Advanced display systems Flexible displays Next-generation display materials 40 6 Research-related works Joint activities with other organizations Publication of research results Application of research results 48 3 Universal broadcasting services User-friendly information presentation Closed-captioning through speech recognition Speech and audio signal processing for the elderly Language processing for language barrier-free services Content search and recommendation technology TV user interface technology Technology for estimating viewer psychological state 27 NHK Science & Technology Research Laboratories Outline 50

3 Greetings Keiichi Kubota Head of NHK Science & Technology Research Laboratories The NHK Science & Technology Research Laboratories STRL celebrated its 80 th anniversary in June Throughout these years, successive generations of Japanese have benefited from the fruits of STRL s research on black-and-white television, color television, Hi-Vision and satellite broadcasting, and digital broadcasting. The STRL Open House in May 2010 commemorated the anniversary with a historical review of advances that we have made over the years, together with exhibits of our latest research results. Another anniversary event was held in October. NHK STRL is the sole research facility in Japan specializing in broadcasting technology, and as part of the public broadcaster, NHK, it has the role, from a research and development perspective, of leading Japan in broadcasting technology and contributing to building a rich broadcasting culture. To fulfill its role, NHK STRL conducts research on two fronts: short -term research to resolve current operational issues, and long-term research to create new services for the next generation. To bring useful and diverse broadcast services to viewers in the age in which broadcast and communications technologies are merging, we are conducting wide-ranging R&D on physiology, devices and systems, so that the next generation of Japanese will have an appropriate infrastructure for their lives. Fiscal 2010 was the second year of our latest three-year administrative plan, whose management policy calls for us to promote research and development on technology that will lead to the age of integrated broadcasting and communications through a wide variety of research. Research for the near future includes finding ways to resolve issues facing the impending switchover to all-digital terrestrial television broadcasting, as well as R&D on new services and terminals that have fully integrated broadcasting and communications functions. For the longer term, NHK STRL is concentrating on developing new media, including Super Hi-Vision and 3D television technologies and the infrastructure to support them. We also want to strengthen cooperation with broadcasting and research facilities in other countries by collaborating internationally on research. It is my hope that this annual report will be helpful for your understanding of NHK STRL research and development activities, for building collaborative relationships toward new research and development, and for making use of the results of our research and development. I would like to express my gratitude and request your continued support and cooperation in the future. NHK STRL ANNUAL REPORT

4 Accomplishments in 2010 Super Hi-Vision We have developed a 33-megapixel camera having the full-resolution of Super Hi-Vision, a high-performance compact encoder, and a practical compact projector. In cooperation with the BBC, NTT, and other organizations, we performed international transmission tests between Tokyo and London using IP networks. We are continuing with our development of a Super Hi-Vision image sensor with a 120 Hz frame rate and have completed the design and prototyping of the device. We are also developing ways to increase the transmission capacity to support SHV and other high-capacity services through digital terrestrial broadcasting. In particular, we have developed a dual-polarized multiple-input multiple-output MIMO ultra -multi-level OFDM Orthogonal Frequency Division Multiplexing method. We have developed a reproduction system for the home environment to convey the presence of 22.2 multichannel audio on a few speakers. We built a high-resolution 58-inch PDP prototype with a 0.33-mm pixel pitch, the level required for a 150-inch SHV-PDP, in cooperation with a commercial electronics manufacturer. This direct-view SHV display is a major milestone towards the introduction of SHV into homes. We released our R&D roadmap for SHV at the STRL Open House The ultimate objective is test broadcasting with 21-GHz-band satellites in At IBC2010 in September in Amsterdam, we gave theater presentations using the full-resolution projector and 22.2 multichannel audio, and we exhibited the full-resolution camera, encoding equipment, 22.2 multichannel audio, and production equipment. We also conducted a live transmission of uncompressed SHV over optical fiber from within Amsterdam. See p.4 for details. Three-dimensional television We are researching integral photography IP as means of making a three-dimensional television. IP uses spatial imaging rather than the common stereo format. It can be viewed without special glasses, and it produces less visual fatigue than the 3D systems in use today. In FY2010, we improved the overall image quality by increasing the lens-array resolution and processing to compensate for distortion. In particular, we used the full-resolution SHV pixel-offset technology to improve the resolution limit and 3D image quality in the depth direction. We also looked into ways of converting multi-viewpoint images captured with multiple cameras into IP 3D images See p.12 for details. Advanced digital broadcasting services We are developing a new technology that fuses broadcasting and communications, called Hybridcast TM. Hybridcast is the product of a lot of research we have done on employing communications networks in broadcasting. A basic Hybridcast system built with PCs was exhibited at the STRL Open House 2010 in May. We have also advanced research to quickly establish and implement the Hybridcast specifications. As part of this effort, we are studying technologies for synchronizing broadcasting and communications channels and playing back content from each of them, security technologies such as ID federation, encryption, CAS, and digital watermarking, and P2P technology for implementing high-capacity live video distribution. We also conducted feasibility tests of the teleda social television service for NHK programs. We are planning a new multimedia broadcasting service for mobile terminals using the VHF- Low-band and have established an experimental test station and conducted field tests. We have also developed a small co-channel interference canceller, which expands the range of environments for receiving digital terrestrial broadcasts. The canceller has been implemented in a set-top box See p.14 for details. Full-resolution SHV camera Compact SHV projector IP 3D display equipment Hybridcast TM Universal broadcasting services We are committed to developing technologies to enable people of all abilities to benefit from broadcast services. For instance, we aim to enrich broadcasts for the hearing impaired. We have expanded the range of sign-language broadcasts by developing a way to convert Japanese text into signlanguage computer graphics, and we have developed a compact and work-saving closed-caption production system using speech recognition. For people with visual impairments, we are working on a haptic-display system in which users touch three-dimensional objects that represent images and Computer-graphic sign language EN-Vision text. We have also developed a technology that turns Japanese in broadcasts into English or easier-to-understand Japanese. We exhibited these user-friendly broadcasting technologies at IBC2010 in September, and they made a great impression on broadcasters from around the world. We are also researching technology to analyze images and language in video content and have developed a system to give recommendations for programs based on the results of the analysis. This system is being used by the NHK Creative Library EN- Vision We are continuing our work on using brain-activity data and behavioral data so that we can analyze and estimate viewers psychological states while they are watching television programs. See p.22 for details. 2 NHK STRL ANNUAL REPORT 2010

5 Advanced contents production technology We added a new millimeter-wave-band send-back system to our low-delay, high-imagequality Hi-Vision millimeter-wave mobile camera. This camera was used during the Kohaku Utagassen program in December, to shoot in circumstances that would difficult for cabled cameras. We also developed a MIMO receiver system for outside broadcasting of sports. The system provides more robust video transmissions in comparison with previous gigahertz-band field pick-up units FPUs This receiver was used for the first time to cover the Lake Biwa Mainichi Marathon in March. We are developing technology for transmitting video materials without interruption using the Internet Protocol and a distributed-server broadcast system using cloud technology. We are planning to post the program source code on the STRL website in the near future. We are trying to use high-quality speech synthesis to expand the range of speech databases efficiently. We have developed equipment for reading out weather reports and have incorporated a speech-rate conversion function to make the read-out speech easier to listen to. We have increased the sound-generation efficiency and improved sound quality in our light-weight electroactive elastomer speakers and are continuing to make advances on low-operating-voltage silicon condenser microphones. See p.28 for details. Devices and materials for broadcasting We are developing organic image sensors. This year, we made a full-color sensor and increased the number of pixels by using a layered organic-film structure. We also made advances in cold-cathode HARP image sensors for compact, ultrasensitive cameras. Regarding our efforts to realize high-speed, magnetic recording devices with no moving parts, we confirmed that magnetic nano-domains in nano-wires can be used for recording information. We also advanced near-field optical recording technology to increase the capacity of thin optical disks. Regarding our work on holographic recording technology, we increased the Full color image from organic image sensor Millimeter-wave mobile camera Flexible display speed of wavefront compensation techniques to stabilize the light wavefront and increased the recording density through angular multiplexing. We are continuing with our development of display panel production technology, display elements, and thin-film transistors TFT for flexible displays, and we have prototyped a 5-inch-diagonal color panel. We also verified the performance of PDP electrodes for SHV. These electrodes reduce the required driving voltage significantly compared to earlier electrodes. Finally, we are developing an ultra-high-resolution, high-speed spatial light modulator for use in optical spin devices for a spatial imaging display system with a wide viewing area See p.34 for details. International collaboration and standardization activities We internationally exhibited ISDB-T, Japan s digital terrestrial broadcasting format. To promote ISDB-T, we conducted demonstrations of multiband transmitters and receivers in southern Africa and other regions that use the 7 or 8 MHz bandwidth. We also invited visiting researchers from Brazil, a country which has already begun using ISDB-T. We are working with international standardization organizations such as the ITU-R 1 and SMPTE 2 to standardize technologies such as Super Hi-Vision. We submitted a proposal to MPEG 3 for a new High Efficiency Video Coding HEVC technique. For attendees of the Tokyo meeting of the ABU 4 in October, we held tours of 14 exhibits of the latest STRL technology and held lectures on three-dimensional television and Hybridcast See p.41 for details. ABU tour of NHK STRL Public relations activities 2010 was the 80th anniversary of STRL, and the STRL Open House, held each year in May, had the 80th anniversary as its theme. It featured 44 exhibits on our latest research results and was attended by 22,087 visitors. In October, Tokyo s Setagaya Ward office joined us in commemorating the 80th anniversary of STRL. The NHK STRL 80 th Anniversary Celebration Years with the People of Setagaya featured technical and interactive exhibits, entertainment at the entrance to the labs, and concert-hall shows and concerts. The event attracted 4,021 visitors over two days. STRL research results were also exhibited overseas at NAB Events commemorating STRL s 80th Anniversary IBC Exhibition 2010 and IBC2010, and in Japan at CEATEC2010 and events held by NHK broadcasting stations throughout the country See p.45 for details. 1 International Telecommunication Union - Radiocommunication Sector 4 Asia-Pacific Broadcasting Union 2 Society of Motion Picture and Television Engineers 3 ISO/IEC Moving Picture Experts Group Hybridcast TM is a trademark of NHK Engineering Services, Inc. NHK STRL ANNUAL REPORT

6 1 Next-generation broadcasting media STRL is advancing with its research on Super Hi- Vision SHV and three-dimensional television for next-generation broadcast media. We studied the system parameters of SHV and examined the frame frequency and colorimetry as well as the pixel count, aspect ratio, and bit depth We confirmed that SHV will provide a sense of realness even on a midsize screen of approximately 70-inches for households after conducting thorough assessment tests. Our R&D on SHV cameras has led to a threechip, full-resolution SHV camera system using 33- megapixel imaging devices. Our work on SHV displays has yielded a practical, compact projector and real-time signalprocessing for a high-dynamic-range projector. We also developed an 85-inch full-resolution LCD display in collaboration with a commercial electronics manufacturer and produced a prototype 58-inch high-resolution PDP as a milestone to our goal making of a 100-inch SHV display. We conducted successful test transmissions between London and Tokyo using an improved AVC/ H. 264 SHV codec. We also submitted proposals for standardization of a new video coding scheme, HEVC, to MPEG and researched a new video coding technology called reconstructive video coding. Regarding the audio system for SHV, we confirmed the effectiveness of the loudspeaker arrangement for 22.2 multichannel sound and developed a sophisticated and practical production system. We also studied a home audio system that uses fewer loudspeakers than channels and a transmission coding system. We are studying transmission technologies with improved bandwidth efficiency for the next generation of wideband satellite broadcasting and digital terrestrial broadcasting to provide highcapacity transmissions such as SHV. We also produced a prototype of cable transmission equipment that uses multiple 6 MHz channels conforming to existing standards. In other research, we studied methods of transmitting uncompressed SHV signals over optical fiber as a means of transmitting program materials and conducted transmission tests at IBC We submitted proposals for standardizing SHV to ITU-R, SMPTE, and other standards organizations. We also issued our R&D roadmap of SHV for the future. We will continue to research integral 3D television IP 3DTV with the ultimate goal being a form of video that shows natural 3 D images without special glasses. We increased the resolution of IP 3DTV using full-resolution SHV devices with pixel-offset technology. We also studied ways of converting multiviewpoint images captured with multiple cameras into IP 3D imagery, so that we can capture subjects that would otherwise be difficult to capture with ordinary IP techniques. We submitted research results on 3 D video studies on the dependency of visual comfort and naturalness of 3D images and display conditions to the ITU-R for standardization. 1.1 Super Hi-Vision Super Hi-Vision format We are conducting studies on the video parameters of Super Hi-Vision SHV Evaluating the sense of realness SHV is a video system that is able to provide a very strong sense of presence due to its extremely high resolution and wide field of view. SHV also provides a sense of realness as though the real object were actually there In FY2010, we evaluated the relationship between image resolution and the sense of realness in experiments where subjects compared actual objects and video imagery. Our goal is to achieve a level of realism whereby images cannot be distinguished from the actual objects 1 Figure 1 shows the relationship between angular resolution cycles-per-degree: cpd and the sense of realness when viewing images. The higher the angular resolution, the greater the sense of realness, and the sense gently saturates above about 60 cpd; above 155 cpd, images are indistinguishable from the Sense of realness ln p PLASTER BUST MODEL SHIP 6 BUTTERFLIES Real object Resolution cpd Figure 1. Relationship between image resolution and sense of realness 4 NHK STRL ANNUAL REPORT 2010

7 1 Next-generation broadcasting media 1.1 Super Hi Vision real object. The results of converting the angular resolution to field-of-view FOV angle or viewing distance for SHV pixel horizontal resolution are shown in Figure 2. Combining this study s results with the past study s on the relationship between the sense of being there and FOV shows that the SHV system can provide both a sense of being there and a sense of realness for FOV angles ranging from 30 to 100 degrees viewing distances of 0.75 to 3 times the picture height SHV is expected to be viewed on large screens in theatres, medium screens in homes, and on small, portable screens. Frame frequency In the past, we studied the relationship between temporal sampling parameters and degradation in motion portrayal, including motion blur, stroboscopic effects, and flicker, as factors for determining the frame frequency. Taking the results of these studies into consideration, we have evaluated the relationship between frame frequency and motion video quality on a 100- inch screen. The improvement in image quality as a result of increasing the frame frequency was found to be 0.5 on a fivegrade interval scale in going from 60 Hz to 120 Hz and a further 0.2 in going from 120 Hz to 240 Hz. contributing the system colorimetry research results described above, we have proposed a UHDTV video parameter set based on research to date see Table 1 1 K. Masaoka, Y. Nishida, M. Sugawara and E. Nakasu: Comparing Visual Realness between High Resolution Images and Real Objects, ITE Technical Report, Vol. 35, No. 16, HI , 3DIT , pp in Japanese 2 K. Masaoka, K. Omura, Y. Nishida, M. Sugawara, Y. Nojiri, E. Nakasu, S.Kagawa,A.Nagase,T.KunoandH.Sugiura: Demonstrationofa wide-gamut system colorimetry for UHDTV, ITE Annual Convention, in Japanese Sense of realness High Sense of realness Sense of being there High Sense of being there System colorimetry We conducted experiments on capturing and displaying widecolor-gamut natural images based on a wide-gamut system colorimetry that uses RGB primaries equivalent to monochromatic light sources on a spectral locus 2 Figure 3 shows color distributions of the objects. These objects include colors that cannot be reproduced by the HDTV color gamut. We confirmed that they could be covered by the proposed wide-gamut system colorimetry for SHV. A comparison of images reproduced by a laser display color gamut and those limited by the HDTV color gamut confirmed that the wider color gamut reproduces colors with higher saturation, enabling better color reproduction that is closer to reality. This research was conducted in cooperation with Mitsubishi Electric Corporation. Standardization We continued with our efforts to standardize the SHV format UHDTV in ITU-R terminology at ARIB and ITU-R. In addition to Low Low Field-of-view deg. Viewing distance H Figure 2. Sense of being there and sense of realness for different field-of -view angles Table 1. Basic video parameters of SHV Spatial resolution H V Framefrequency Bit depth System colorimetry CIE 1931 Red Green Blue Reference white D pixels 60 Hz Under study forover60hz 12bit x = y = x = y = x = y = x = y = y 0.8 y x x Figure 3. Color distribution of wide-color objects and RGB primaries of SHV outer and HDTV inner NHK STRL ANNUAL REPORT

8 1 Next-generation broadcasting media 1.1 Super Hi Vision Cameras We are making advances on a full-resolution Super Hi-Vision SHV camera with a 7680-pixel by 4320-line resolution for red, green and blue 1 Figure 1 In FY2010, we reduced the size of equipment used to transmit the video signal between the camera head and camera control unit prototyped in FY2009. This is so it can be incorporated into the camera head and be more portable. This development also helped to reduce the total weight of the camera head, including lens, to 65 kg. We are developing a lateral chromatic aberration correction system using signal processing in real time. The accuracy of correction is increased by adding a function to support not only the zoom parameter but iris and focus parameters. We also devised a method for generating and presenting a focus-assist signal, to help camera operators focus using a lowresolution viewfinder, and we implemented this function in the camera. The assist signal is generated by applying a maximum filter to a pre-selected high-frequency component of the signal, and this is overlaid on the viewfinder image generated by a lowpass filtering. This method enables focusing to an accuracy equal to that of using an ultrahigh-resolution monitor, and it has low computational cost. It is thus very practical. We were able to shoot outdoor scenes using this prototype camera and to exhibit SHV video that was captured and displayed at the full-resolution at the STRL Open House We also proceeded with our work on single-chip color imaging devices using 33-megapixel image sensors for fullresolution SHV, for compact SHV cameras. We also made advances on demosaicing 2 to be applied to ultrahigh-resolution Figure 1. Full-resolution SHV prototype camera single-chip color imaging. 1 T. Yamashita, R. Funatsu, T. Yanagi, K. Mitani, Y. Nojiri and T. Yoshida: A Camera System Using Three 33-M-pixel CMOS Image Sensors for UHDTV2, SMPTE Annual Technical Conference & Expo R. Funatsu, T. Soeno, K. Omura, T. Yamashita and M. Suagawara: High-resolution Demosaicing Using an Even-tap Symmetric FIR Filter, ITE Winter Annual Convention, in Japanese Displays Projectors We have been developing a high-dynamic-range projector with dual modulation since FY The dual modulation method requires processing of the outputs of the primary and secondary modulators to obtain the optical output, and initially, the video signals had to be created computationally. In FY2010, we implemented real-time video signal processing equipment to directly display Super Hi-Vision SHV video signals. We also developed a compact projector capable of displaying SHV by using 4K display devices for each of R, G and B signals and shifting them diagonally by a half-pixel every second frame 1 Figure 1 This projector was jointly developed with NHK Engineering Services and JVC Kenwood Holdings. Direct-view displays We are developing a self-emissive direct-view SHV displays for home use. In collaboration with Panasonic, we successfully prototyped a 58-inch ultra-high-resolution plasma display panel PDP with a pixel pitch of 0.33 mm Figure 2 It has four times the number of pixels of Hi-Vision displays, the same as a Figure 1. Compact projector Figure 2. Ultra-high-definition, 58-inch, 0.33mm pixel-pitch PDP display exhibited at the STRL Open House 2010 and IBC 2010 in Europe 6 NHK STRL ANNUAL REPORT 2010

9 1 Next-generation broadcasting media 1.1 Super Hi Vision 103-inch PDP display which we developed, and it matches the pixel-pitch required for implementing a 100-inch full-resolution SHV display. In cooperation with Sharp Corporation, we developed an 85- inch, full-resolution, SHV Liquid Crystal Display LCD with a pixel pitch of 0.25 mm. Ultra-realistic communication system This is the final year of our research undertaken for the National Institute of Information and Communications Technology NICT This project began in FY2009 and is entitled, Ultrarealistic communication system. We verified the effectiveness of a binaural audio system designed for a dome theatre, conducted a subjective assessment of the relationship between brightness, contrast and screen size of the display 2 and outlined video and audio specifications for dome theatres. 1 F. Okano, M. Kanazawa, Y. Kusakabe, Y. Haino, M. Furuya and M. Sato: Proposal for a scanning method based on complementary field offset sampling and its application to Super Hi-Vision projector, ITE Technical Report, Vol. 35, No. 4, IDY2011-9, pp in Japanese 2 M. Kanazawa, J. Nishigaki, K. Takeuchi, R. Harada and T. Imamura: Image Presentation for Dome Theater, ITE Technical Report, Vol. 35, No. 16, HI , 3DIT , pp in Japanese Coding We are studying video compression coding methods for Super Hi-Vision SHV broadcasting. SHV video encoder We improved the AVC/H.264 SHV video encoder developed in FY2009. In FY2010, we improved the video format converter that divides up the SHV video signal. The conventional system does not transmit the periphery of the image due to overlap processing at the partition boundaries. As a remedy, we added an interpolation function that allows pixels at the edge of the image to be transmitted. This research was conducted in cooperation with Fujitsu Laboratories, Ltd. On September 29 and 30, 2010, we conducted live international SHV transmission experiments in cooperation with NTT, the BBC, and academic networks. In these experiments, video captured at a BBC studio in London was compressed using an encoder developed by NHK and transmitted to STRL in Tokyo via an advanced internet Figures 1, 2 We confirmed that high-quality SHV can be transmitted using multiple networks managed by different operators and without any guaranteed bandwidth. In parallel with these efforts, we are moving forward with our research on video coding methods for program production. We developed functions for extending the JPEG2000-standard video encoders developed in FY2009 and earlier, including a reversible RGB-YUV conversion system and an extended pixel dynamic range system. Figure 1. Encoder London Figure 2. Decoder Tokyo Next-generation video coding Standardization of High Efficiency Video Coding HEVC for ultra-high definition video is in progress at ISO/ IEC and the ITU-T. We are studying next-generation video coding methods to be used for this standard 1 We developed a technology that adaptively use a discrete cosine transform DCT or discrete sine transform DST HEVC s advanced intra-prediction and motion compensated prediction techniques de-correlate the residual signals after the predictions, which means that the DCT is not necessarily suitable for HEVC. By using a DCT and DST together, we were able to improve performance for images that include complex textures. We also developed a new intra-prediction technology that improves video coding efficiency, while maintaining continuity between boundary regions and predicted regions. This research was conducted in cooperation with Mitsubishi Electric Corporation. Reconstructive video coding For transmitting high-data-volume video such as SHV, we are looking into a new concept called reconstructive video coding, in which video is coded after being reduced in size according to parameters such as transmission bandwidth and video content and is reconstructed using super-resolution techniques. This method uses existing conventional video coding schemes such as AVC/H.264 as its core, but in a pre-processing stage, it optimizes the super-resolution parameters with adaptive processing according to texture and local reconstruction. These parameters are sent to the post-processing stage as auxiliary data, and they make possible high-quality image reconstructions Figure 3 In FY2010, we added parameter optimization to the preprocessor for it to control non-uniform spatial and temporal subsampling and aliasing due to pixel decimation. The parameter optimization uses super-resolution in-loop simulation. We developed two reconstruction methods for the post processor including a sequential Monte Carlo method 2 and a method based on wavelet analysis 3. We also developed techniques for iterative synthesis of high-frequency components based on self-similarity. The sequential Monte Carlo-based reconstructive method generates a super-resolution image using weightings and by choosing from multiple hypothetical images. The resolution of the images is increased by iteratively processing them based on the similarity between reduced hypothetical images and the decoded image. We confirmed the effectiveness of this method to reconstruct images Figure 4 NHK STRL ANNUAL REPORT

10 1 Next-generation broadcasting media 1.1 Super Hi Vision Input image Local reconstruction Optimization/Image analysis Pixel reconstruction Video reconstruction Parameters Side-information coding Geometric transforms Pixel decimation Reduced image Pre-processor Coding Decoding Side information Side-information decoding Figure 3. Reconstructive video coding Parameters Video coding Bit-stream Video decoding Decoded reduced image Inverse geometric transforms Super-resolution Reconstructed image The wavelet reconstruction method computes superresolution parameters that minimize the difference between the source input image and the locally reconstructed image on the Core coding Post-processor Figure 4. Test results Left: Conventional coding method, Right: Reconstructive coding method encoding side. It uses these parameters to obtain a highly accurate super-resolution image. The technique for generating high-frequency components focuses on the similarity between image blocks and estimates high-frequency components upon magnification from other blocks to generate a super-resolution image. 1 Y. Shishikui and K. Iguchi: Iterative Adjustment Intra Prediction for High Performance Video Coding, IWAIT S. Sekiguchi, A. Minezawa, K. Sugimoto, A. Ichigaya, K. Iguchi and Y. Shishikui: A novel video coding scheme for Super Hi-Vision, Proc. PCS2010, O3-4, pp T. Misu, Y. Matsuo, S. Sakaida, Y. Shishikui and E. Nakasu: Novel Video Coding Paradigm with Reduction/Restoration Processes, Proc. PCS2010, S2-2, pp Satellite transmission technology We have continued with our studies on Super Hi-Vision SHV broadcasting via satellite using 12- and 21-GHz-band. Transmission systems for next-generation satellite broadcasting We evaluated the performance of a transmitter incorporating a pre-distortion compensator built in FY2009. The compensator can reduce the transmission power needed for SHV broadcast signals by improving the carrier-to-noise power ratio C/N required for multi-level modulation on a satellite channel. It estimates distortion vectors in the satellite channel at the transmitter and adds the opposite vectors to the signal before transmission. We conducted transmission tests on a 12-GHz-band satellite transponder simulator and found that incorporation of the compensator yielded an improvement of 0.4 db for 16 APSK code rate: 3/4 and 1.3 db for 32 APSK code rate: 4/5 in the required C/N + Output Back Off OBO: the ratio of maximum transmission power for the unmodulated signal to the actual modulated-signal power in the satellite transponder 1 To improve the required C/N on the receiving side, we began prototyping a receiver conforming to ARIB STD-B44 Transmission System for Advanced Wide Band Digital Satellite Broadcasting The receiver incorporates an adaptive equalizer utilizing a pilot signal Figure 1 To evaluate its performance, we will conduct tests on it over the satellite transponder simulator. To evaluate the effect of the satellite channel characteristics, we prototyped a 21-GHz-band satellite transponder simulator composed of input and output filters, amplifiers and the other components. We will use this simulator to evaluate the wideband modulator/demodulators. 21-GHz-band broadcasting satellite system We continued with our research on an onboard phased-array antenna that can control the radiation pattern to compensate for rainfall attenuation in the 21-GHz-band GHz The antenna is an array-fed imaging reflector antenna composed of a main reflector with a 2.2 m diameter aperture, a subreflector with a 0.22 m diameter aperture, and 61 feed elements. We studied how to design radiation patterns that could be controlled with only the phase of the feeds in order to minimize the broadcast outages due to rainfall over the entire service area. We simulated outages by using radar and rain-gauge combined analysis precipitation data provided by the Japan Meteorological Agency. The simulation showed that outages could be reduced to approximately 40 % of the time over the entire service area as compared with a uniform-power radiation pattern using a conventional reflector antenna 2 For the 21-GHz-band receiver, we developed offset parabola antennas with 45, 60 and 120 cm diameter apertures and measured their radiation patterns. We confirmed that the measured co-polarization radiation pattern can be satisfied the reference pattern being considered by the ITU-R 3 We also studied the power feed for a dual 12- and 21-GHz-band receiver antenna, assuming that broadcast satellite s geostationary orbit position would be 110 degrees east longitude. In the future, we will build an antenna based on the results of this study and evaluate its performance. 8 NHK STRL ANNUAL REPORT 2010

11 1 Next-generation broadcasting media 1.1 Super Hi Vision I Front-end equalizer LDPC decoder Q Equalized signal Log likelihood ratio Pilot signal averaging Log likelihood ratio computation C/N estimation Update filter coefficients Detect pilot signal Filter coefficient update through training sequence Pilot signal replica LDPC : Low-Density Parity Check Figure 1. Adaptive equalizer using a pilot signal Engineering model of 21-GHz-band broadcasting satellite transponder We are developing a test model of a satellite transponder, called the engineering model, to evaluate the performance of 21-GHz-band onboard equipments in a space environment. It is composed of a shaped reflector antenna for transmission and reception, a branch filter, a receiver, an input filter, an amplifier, an output filter, and a multiplexer Figure 2 The shaped reflector antenna is a membrane structure made from tri-axial woven carbon fiber for lightness and surface accuracy. Assuming the band from 21.4 GHz to 22.0 GHz can be divided in two and used for wide-band signal transmissions, the bandwidths of the input and output filters should be approximately 300 MHz. The filters were designed to reduce the group delay deviation, which degrades a transmission signal. The output filter was designed to attenuate unwanted emissions by 80 db in the neighboring frequency band to 22.5 GHz for radio astronomy A travelling-wave tube TWT amplifier with an output power of 130 W was also developed. We will evaluate the electrical performances of each of these devices and verify their electrical and mechanical performance in a thermal vacuum test that simulates a space environment. 1 M. Kojima, A. Hashimoto, Y. Suzuki, T. Kimura and S. Tanaka: Performance Evaluation of Predistortion Transmitter Based on Signal Shaped reflector antenna Branch filter Receiver Commands Input filter Telemetry TWT Output filter Figure GHz-band broadcast satellite engineering model Multiplexer Point Error Estimation over Satellite Channel, IEICE Society Conference, B in Japanese 2 S. Nakazawa, M. Nagasaka, S. Tanaka and K. Shogen: Onboard Antennas and Simulation of Service Availability for 21-GHz Band Broadcasting Satellite, IEICE Technical Report, Vol. 110, No. 23, AP , pp M. Nagasaka, S. Nakazawa, S. Tanaka and K. Shogen: A Study on Receiving Antenna Patterns for the 21 GHz-band Broadcastingsatellite Service, IEICE Society Conference, B in Japanese Terrestrial transmission technology We are studying next-generation digital terrestrial broadcasting systems for delivering large-volume content services such as Super Hi-Vision. High-capacity transmissions for the next generation of terrestrial broadcasting In FY2010, we developed a dual-polarized MIMO multipleinput multiple-output transmission system to transmit two ultra-multi-level OFDM Orthogonal Frequency Division Multiplexing signals based on the ISBD-T signal format using the 4096QAM for carrier modulation scheme. We also conducted laboratory and field tests and evaluated the transmission characteristics of our prototype system. The transmission capacity was about 73 Mbps PES rate using 4096QAM with an inner code rate of 3/4 for FEC and dual polarized waves. Transmitter and receiver experimental stations were set up on the STRL premises. They were separated by approximately 100 m and had dual polarized Yagi antennas mounted on top of them Fig- NHK STRL ANNUAL REPORT

12 1 Next-generation broadcasting media 1.1 Super Hi Vision ure 1 The bit-error rates at various receiver input levels were measured by attenuating the receiver input level and by varying transmission parameters such as the carrier modulation scheme and inner FEC code rate 1 Figure 2 To reduce the degradation due to level differences between the each polarized waves, we conducted tests using skew polarized waves at ±45 degrees and right/left-handed circularly polarized waves. We found that the required C/N could be improved by approximately 2dB by using skew or circularly polarized waves instead of horizontally and vertically polarized waves when the difference in levels between the each polarized signals was 8 db. We also transmitted two ultra-multi-level OFDM signals each modulated by different data streams at 1 W from vertically and horizontally polarized transmitter antennas mounted at different locations on the roof of STRL. We measured the transmission characteristics at 27 reception points at distances of 5 km or less around STRL. We obtained error-free transmissions with an inner code rate of 3/4 at approximately 53 dbf or greater for 1024QAM, and 59 dbf or greater for 4096QAM. We are also making progress in our work to lower the required C/ N by changing the error correction scheme from a concatenation of convolutional and Reed-Solomon coding, as is used in ISDB-T, to a concatenation of low-density paritycheck coding and BCH coding. Transmitter station Dual polarized Yagi antenna Receiver station Bit Error Rate BER Figure 2. Field test results Reception power dbm 1024QAM 2048QAM 4096QAM Study of the UHF-band for next-generation terrestrial broadcasting To increase the likelihood that the UHF-band will be secured for next-generation terrestrial broadcasting, we conducted computational simulations based on the supposed state of frequency usage after the switchover to all-digital terrestrial broadcasting takes place. We studied frequencies available for new stations and began a study of possible interference from transmitters that use white spaces in the UHF-band. The interference model included two types of interference from stations using white space: on the broadcast area and on reception at relay stations. We also built a UHF-band experimental test station conforming to the ISDB-T transmission format at the Nabeta radio broadcasting station in Aichi Prefecture. This station was used to test limited-area One-Seg use in disaster stricken areas and is a model for Special Whitespace Areas selected by the Study team for a new radio-frequency-use vision of the Ministry of Internal Affairs and Communications Figure 1. A view of field test within STRL premises 1 S. Asakura, K. Murayama, M. Taguchi, T. Shitomi and K. Shibuya: Technology for the next generation of digital terrestrial broadcasting -Transmission characteristics of 4096 QAM-OFDM-, ITE Technical Report, Vol. 35, No. 10, BCT , pp in Japanese Wired transmission technology We are proceeding with our research on wired transmission technologies using fiber-optic and co-axial television cables for transmitting uncompressed Super Hi-Vision SHV signals and distributing SHV to homes. Transmission of uncompressed SHV signals Uncompressed video signals, unlike compressed video, have minimal delay and no image degradation. They are thus used for the contribution links between broadcast stations or from on-site locations to stations. To allow the use of Wide Area Networks WAN as a contribution link, the SHV signal must be converted into a WAN signal. We have developed a technology to do this conversion and have built optical transmission equipment for handling full-resolution SHV signals. At the IBC 2010 broadcasting equipment trade show held in The Netherlands in September 2010, we transmitted live, uncompressed, 24 Gbps dual-green SHV signals over optical fiber within the city of Amsterdam over a distance of approx km We are also conducting research on ultra-high-speed networks for transmitting SHV within broadcast stations. As part of a study for a system to accommodate two 72 Gbps, fullresolution SHV signals on a 160 Gbps optical LAN signal, we built equipment that can carry a single full-resolution SHV sig- 10 NHK STRL ANNUAL REPORT 2010

13 1 Next-generation broadcasting media 1.1 Super Hi Vision nal on two 40 Gbps signals and carried out laboratory tests on it 1 We collaborated with the National Institute of Advanced Industrial Science and Technology and the National Institute of Information and Communications Technology in a demonstration of an optical packet and optical path integrated network. This demonstration successfully transmitted a dual-green SHV signal over an optical path network. Part of this research was conducted under contract with the New Energy and Industrial Technology Development Organization for the project titled Development on Next-generation High-efficiency Network Devices Technology. Cable TV transmission of SHV We are also researching technology for transmitting compressed SHV signals over cable television networks to households. We developed a multiplexing method that divides up a compressed SHV signal and efficiently multiplexes the divided signals on multiple carriers. The system supports Transport Stream TS signals of various bit rates, including those of highcapacity MPEG-2 TS. It also reduces the power consumption of receivers. We built test equipment 2 and exhibited it at the STRL Open House 2010 Figure 1 Our technology using multiple Figure 1. Prototype test equipment carriers has the benefits of being easy to introduce to existing facilities and facilitating early introduction of SHV broadcasting. 1 T. Nakatogawa, M. Nakamura and K. Oyamada: Fundamental experiment of converting a full resolution Super Hi-Vision signal into two OTU3 signals, ITE Winter Annual Convention, in Japanese 2 T. Kusakabe, T. Kurakake and K. Oyamada: A division and transmission method of various bitrate transport streams over coax cable television, ITE Annual Convention, in Japanese Audio systems providing a strong sense of presence We are making progress with our R&D and standardization efforts on 22.2 multichannel audio 1 Figure 1 to be used by Super Hi-Vision SHV 22.2 multichannel audio production systems We are conducting R& D on easier and more sophisticated production systems for 22.2 multichannel audio. In FY 2010, based on the results of research that we contracted to McGill University in Canada, we developed a system to add 3D reverberation to audio and used the system during the production of SHV content. We also developed a man-machine interface to make 3D audio panning operations easier and exhibited a 22.2 multichannel audio production system at IBC2010 for the first time. To control the virtual distance of the audio image in the 3D audio space, we developed an algorithm that simultaneously changes the frequency response, amplitude, amount of reverb added, and distance of sound sources. Subjective evaluation of 22.2 multichannel audio To confirm the effectiveness of 22.2 multichannel loudspeaker arrangements, we arranged multiple loudspeakers on the surface of a sphere and conducted a subjective evaluation on the sense of spatial uniformity of sound especially as regards the location of the speakers in the vertical direction. We found that if the interval between adjacent loudspeakers is less than 45 degrees, there is a good sense of uniformity to the distribution of sound levels in the vertical direction Figure multichannel sound for households Households in Japan have little space available for 22.2 multichannel audio when one speaker is used for each channel, so we are researching signal processing that will enable the high sense of presence of 22.2 multichannel audio to be conveyed by fewer speakers. In FY2010, we developed a method for downmixing 22.2 channels to 8 channels while maintaining the pressure and direction of sound at the listening point 2 We also developed a method of reproducing 22.2 multichannel audio with only three forward facing loudspeakers through use of a headrelated transfer function that incorporates the propagation characteristics of sounds reaching both ears from various directions. These advances were exhibited at the STRL Open House We conducted subjective evaluations of the size of the listening area for the 8-channel reproduction method and found that up to three viewers can experience the full sense of presence at the same time. For the three-loudspeaker reproduction method, we studied methods to suppress high-frequency noise produced by signal processing. We also prototyped a system to play back the forward channels of 22.2 multichannel audio using a loudspeaker array with several small loudspeaker units arranged on the periphery of a TpBL BL TpSiL SiL TpFL FL BtFL BC TpC TpBC FLc LFE1 TpFC TV screen BtFC BR TpSiR LFE2 TpFR TpBR FC FRc FR SiR Upper level: 9 channels Mid-level: 10 channels BtFR Lower level: 3 channels LFE: 2 channels Figure 1. Labels and positions of 22.2 multichannel audio channels based on SMPTE ST NHK STRL ANNUAL REPORT

14 1 Next-generation broadcasting media 1.1 Super Hi Vision 1.2 Three-dimensional television flat panel display. We have begun research on a format for converting 22.2 multichannel audio into a down-mixed 8-channel signal and a signal for expanding listening area, encoding and transmitting those signals, and decoding the signals and reproducing 22.2 multichannel audio at the receiver side. In FY2010, we reduced the energy of the listening-area-expansion signal by 10 db or more, which means that the information can be compressed by reducing the bits allocated to the listening-area-expansion signal. Standardization We continued to promote 22.2 multichannel audio at various standards bodies. This year, the IEC standard for general channel label and channel assignment, including 22.2 multichannel audio, was approved. This should facilitate the development of digital audio interfaces and other devices including I/O devices. The specification is compatible with those of 22.2 multichannel audio in the SMPTE ST standard. At SMPTE, we are interested in standardizing metadata for describing multichannel audio in the Material exchange Format MXF At ITU-R, to support the 3D multichannel audio studio specifications that we proposed in FY2009, we added proposals for 3D multichannel audio format requirements and contributed to the rapporteur documents regarding the latest trends in 22.2 multichannel audio. Acoustical cognitive science In research contracted by the National Institute of Information and Communications Technology, titled R&D on Ultra-Realistic Communication Technology with Innovative 3D Video Technology: Recognition and transmission of sensitivity information, we studied acoustical factors affecting the sense of presence and the relationship between the sense of presence and depth of emotion Kandoh We found that there is a strong correlation between the sense of presence and the impressions of substance and motion and that this correlation is strengthened when sound sources are moved closer. We also found that Spatial uniformity of sound (degradation metric) Speaker spacing deg. Azimuthal angle of vertical circle on which loudspeakers were arranged 0deg. 45deg. 90deg. Figure 2. Vertical spacing of speakers on a spherical surface and sense of sound connectedness a large change in the sound image width results in a change in the Kandoh evaluation value. 1 K. Hamasaki, T. Nishiguchi, R. Okumura, Y. Nakayama and A. Ando: A 22.2 Multichannel Sound System for Ultrahigh-Definition TV UHDTV SMPTE Motion Imaging Journal, Vol. 117, No. 3, pp A. Ando: Conversion of reproduced sound field based on the coincidence of sound pressure and direction of particle velocity, Proc. ICA Three-dimensional television Integral 3D television We are researching a form of spatial imaging technology called integral 3D television. This form of television promises a heightened sense of presence without eye strain. Integral photography IP is a system able to reproduce three-dimensional images by using an array of small lenses for capture and display. Integral photography does not require the viewer to wear special glasses, and it can display natural 3 D images that change with the viewing position. The drawback of IP is that it requires a huge amount of data to produce good-quality images. We developed integral television equipment using fullresolution Super Hi-Vision SHV images in FY and in FY 2010, we improved the quality of the 3D images by using pixeloffset methods with the full-resolution SHV green element Figure 1 To capture images, we offset the positions of two green imaging elements G1,G2 by half a pixel width in the vertical and horizontal directions. For display, we use a single green display element together with an optical element able to shift the optical path by half a pixel width in the horizontal and vertical directions and compose the images by controlling the shift in time. This increases the resolution of the elemental images images captured using the lens array which in turns improves the resolution of the reproduced 3D images in the depth direction. In the future, we will investigate the resolution characteristics of this equipment. We have also developed methods for detecting and adjusting the amount of pixel offset by using aliasing in the image produced by the lens array 2 3 With these methods, pixel offsets can be adjusted with high precision, either visually or using simple measuring equipment, with the lens array in place. Part of this research was conducted under contract with the National Institute of Information and Communications Technology NICT for the project titled R&D on Multi-Parallel & Spatial Imaging 3-Dimensional Television System. We also participated in R&D on proving test systems for expansion of digital museums with Tokyo University under contract by the Ministry of Education, Culture, Sports, Science and 12 NHK STRL ANNUAL REPORT 2010

15 1 Next-generation broadcasting media 1.2 Three-dimensional television Technology. We investigated the potential for using this method for exhibiting museum artifacts using 3D displays. Finally, we continued in our collaborative research with NICT in the area of electronic holography. We were able to reproduce full-color holographic images of real objects by converting images captured in SHV and integral photography into holograms. 1 J. Arai, F. Okano, M. Kawakita, M. Okui, Y. Haino, M. Yoshimura, M. Furuya and M. Sato: Integral Three-Dimensional Television Using a 33-Megapixel Imaging System, IEEE Journal of Display Technology, Vol. 6, No. 10, pp M. Kawakita, S. Sasaki, J. Arai, M. Okui, F. Okano, Y. Haino, M. Yoshimura and M. Sato: Projection-type integral 3-D display with distortion compensation, J. SID, 18/9, pp H. Sasaki, M. Kawakita, K. Masaoka, J. Arai, M. Okui, F. Okano, Y. Haino, M. Yoshimura and M. Sato: Pixel-offset position detection using lens array for integral three-dimensional display, Proc. of SPIE, 7863, Image-capture lens array gradient-index lens Display timing control signal Display lens array Object B R G 2 G 1 G 1 : HD-SDI 16ch G 2 : HD-SDI 16ch B : HD-SDI 16ch R : HD-SDI 16ch G 1/G 2 Frame switcher HD-SDI DVI interface converter DVI Time-division pixel-offset SHV projector Reproduced 3D image SHV camera using the pixel offset method Diffusion screen Figure 1. Integral 3D television incorporating the Super Hi-Vision pixel-offset method Generating 3D content from multi-viewpoint images We are developing technology to capture subjects with multiple cameras and generate integral 3D images from the resulting multi-viewpoint images. This method of generating 3D images works on subjects that are difficult to capture with the normal integral 3D method. In FY2010, we studied methods of setting up multi-viewpoint robotic cameras, generating shape models from multi-viewpoint images, and converting these shape models into integral 3D images. We built multi-viewpoint robotic cameras to take 3D images of distant objects with zoom lenses 1 Figure 1 shows a group of cameras mounted on a motion control platform. With this system, the orientation of the cameras can be controlled in coordination with the movement of a single master camera operated by a person. We studied the phase-only correlation method and the belief propagation method as ways of generating shape models from multi-viewpoint images. The precision of a shape model built with the phase-only correlation method was improved by using region partitioning. For the belief propagation method, we devised a hierarchical method for reducing the effect of video level differences among the cameras. To convert shape models into integral 3D images, we developed a procedure using oblique projections parallel light rays are projected obliquely onto the projection plane 2 Oblique projections do not cause the geometric distortions often accompanying projections, and hence, they enable conversions to be done more efficiently. This processing can be sped up by using a Graphics Processing Unit GPU Figure 1. Multi-viewpoint robotic camera Part of this research was conducted under contract with the National Institute of Information and Communications Technology NICT for the project titled R&D on Ultra-Realistic Communication Technology with Innovative 3D Video Technology. 1 K. Ikeya,K. Hisatomi,M. Katayama and Y. Iwadate: Control Method for Multiple Robot Cameras, ITE Winter Annual Convention, in Japanese 2 Y. Iwadate and M. Katayama: A generation method of integral 3D Image by oblique projection, ITE Technical Report, Vol. 34, No. 43, 3DIT , pp in Japanese NHK STRL ANNUAL REPORT

16 2 Advanced digital broadcasting services Ten years have passed since digital broadcasting began, and the transition to all-digital broadcasting will be completed in most areas of Japan in July, STRL is continuing its research with the goal of making digital broadcasting more convenient and user-friendly. NHK On Demand, which began in 2008, is an example of how, with the recent spread of broadband networking, content similar to broadcasts can be viewed at any time through communications channels. With the spread of digital broadcasting and broadband, we have entered a time when linking and fusion of communications and broadcasting will be commonplace. For such an era, NHK has added management policies to its three-year management plan started in 2009 to promote R&D on technology that will lead to the fusion of broadcasting and communications, and STRL is engaged in such R&D. At the STRL Open House 2010, we unveiled a new system linking broadcasting and communications, called Hybridcast TM. Hybridcast incorporates various technologies developed by STRL, and it will go beyond simply providing interactive programs and communications content by connecting the Internet to a television receiver. It will include technology for implementing sophisticated services, for example, services that simultaneously use broadcast and communications content, applications that can be enjoyed in various ways, and services that make use of media such as social networking and mobile terminals. After the open house, we began a line of research that would quickly lead to us establishing and implementing specifications for Hybridcast, starting with a study on how to synchronize content for presentation accurately. At the same time, we are engaging in research on related security technologies, including ID federation technology that will make it easier to link mobile terminals and televisions, authentication technology that will enable easy and efficient key updates, and electronic watermarking technology for identifying video content. We are also advancing with our research on a downloadable version of the Conditional Access System CAS Anticipating a time in the future when broadcasting and communications are completely IP, we are researching extensions to the multiplexing format currently used for digital broadcasting, MPEG-2 Systems. In particular, we are studying an advanced multiplexing that combines content from multiple transmission paths and have submitted related proposals to MPEG. We have also developed a peer-to-peer P2P technology for distributing live video to many terminals over the Internet at the same time and have tested a prototype to confirm its practicality. We are developing a web-based computer graphics CG video production system to provide services linked to a variety of information available on the web. We also conducted mediumscale demonstrations of the teleda social TV broadcast service. We are conducting research on transmission methods and receiver antennas for mobile multimedia broadcasting in the VHF-band of channels 1, 2, and 3 that will become available after analog television broadcasting ends. We are also developing a co-channel interference canceller and equalizer for suppressing multipath interference outside the guard interval of digital terrestrial broadcasting; in 2010, we reduced the size of this equipment. We also made an ISDB-T compatible system for the 7 and 8 MHz bandwidths for international markets that use these bands. 2.1 Technologies for fusing broadcasting and communications Hybridcast TM Hybridcast TM works through a beneficial combination of broadcasting and communications, and it will be used to provide new broadcast services that are difficult to implement usingbroadcastingalone 1 Hybridcast platform technology The Hybridcast platform is versatile and extensible, and it enables us to build various services in accordance to the needs of viewers Figure 1 At the STRL Open House 2010, we exhibited Hybridcast TM is a trademark of NHK Engineering Services, Inc. 14 NHK STRL ANNUAL REPORT 2010 a customization service, a Social TV service, a program recommendation service, and a device linkage service on PC-based receivers 2 The customization service enables content to be presented in a way that better meets individual viewer s needs and preferences by sending information related to the current broadcast program over a broadband network. Examples of what it provides include multi-lingual subtitles for foreigner and speechrate-adjusted audio for the elderly. One key technology for it is the stream synchronization for content delivered over multiple delivery channels. The Social TV service integrates broadcasting with social networking services SNS that have become popular on the Internet. At the Open House, we exhibited a

17 2 Advanced digital broadcasting services 2.1 Technologies for fusing broadcasting and communications Broadcast Multi-destination, high quality, reliable Program Viewers have access to a wealth of programs Broadcaster Communications according to individual viewer needs Additional program-related information Digital receiver Network cloud Program DB Various services Various functions User DB CGM, SNS Content management/ distribution technology Synchronized presentation technology Mobile-device linkage technology Content-rights protection Viewer privacy protection Private home CGM Consumer Generated Media SNS Social Networking Service Figure 1. Hybridcast system overview service in which viewers can post comments and impressions about a program they are watching to an SNS and have them appear on the television screen. The service also lets viewers give their impressions by choosing a group of comments with four colored keys on their remote controller. The program recommendation service offers, from a broad library of Video-on- Demand VOD content, programs that are related to the current program or that appear to the user s interests or needs. It also allows programs to be bookmarked. The device linkage service links a mobile device to a television set with the help of a 2D barcode, and users can operate their mobile device to view recommended programs, schedule recordings, etc.. We established a set of requirements for the Hybridcast system and studied how to ensure that it would be flexible enough to deliver various services and be compatible with existing broadcasting systems. We focused on the vital parts of the system, that is, the overall system architecture, receiver APIs, and signaling. We verified the system s feasibility with the cooperation with device manufacturers. We also improved the stream synchronization technology by developing a set-top-box prototype receiver with video-frame-rate accuracy that can support multi-angle and stereoscopic TV services. The findings of the above-mentioned studies and tests were incorporated into the draft specifications on Hybridcast s system architecture, service model, and technical requirements. We also demonstrated Hybridcast prototypes at various exhibitions and events, including the STRL Open House 2010, the ABU General Assembly in Tokyo, IBC2010, and the ITE Winter Symposium. In the future, we will conduct detailed studies related to issues such as security, create specifications for the platform and overall system, and a develop verification system. New transport schemes We are conducting research on transport schemes for combinations of video, audio, and data to be transmitted over broadcast and communication networks. DTS/CTS Code DTS/CTS Code Video Audio Timed data Files FEC data Broadcast DTS/CTS UTC Code Component1/broadcast DTS/CTS UTC Code Component2/communications ATS packet UDP IP clock1 UTC clock2 Component 1 Control metadata Program A Communications Component 2 DTS Decoding Timestamp CTS Composition Timestamp Figure 2. Synchronization method of ATS components Figure 3. ATS protocol stack Control metadata The current digital broadcast systems require all components in a program to be multiplexed into a single transport stream TS using MPEG-2 Systems. A limitation of MPEG-2 Systems is that components transmitted in separate TSs cannot be combined and used together on the client side. We designed a new type of control metadata to link components transmitted on multiple delivery channels and a method for synchronizing the presentation of these components. Specifically, we developed two transport schemes: a TS scheme and an IP scheme. In the TS scheme, TS components are deliv- NHK STRL ANNUAL REPORT

18 2 Advanced digital broadcasting services 2.1 Technologies for fusing broadcasting and communications ered in the form of IP on communication networks. In the IP scheme, all components are delivered in the form of IP since all data in communication networks are delivered in the form of IP packets In the TS transport scheme, the time stamps are attached to program components to be sent over communication networks. Time stamps are used because it is difficult to deliver an accurate clock over the Internet. They enable components sent over the broadcast and communication networks to be synchronized with each other when they are played through the receiver. We designed a new kind of control metadata, called Advanced PMT APMT which can link components in another TS to those in the TS in which the APMT is multiplexed. The APMT can be sent on broadcast or communication networks because its section structure is a generic TS control metadata format. The IP transport scheme is applicable if both networks can transmit IP packets. We developed the Advanced Transport Scheme ATS by which synchronized presentation of components is possible by adjusting the presentation times of each component using the attached Universal Time Coordinates UTC Here, the time stamps of each component are not based on the identical clocks Figure 2 File data, forward error correction FEC data, and control data are delivered in ATS packets, and content delivered on various channels can be easily integrated Figure 3 We submitted proposals on these transport schemes to MPEG, which is in the process of standardizing new transport schemes called MPEG Media Transport MMT IP distribution technology Viewing screen ( on web browser Figure 4. P2P live distribution system P2P network monitor Viewing-state monitor We are researching IP content distribution technology using overlay networks to enable low-cost and stable delivery of large-scale content over communication networks. In FY2010, we developed a live streaming technique using peer-to-peer P2P technology and built a prototype to distribute live video to multiple terminals simultaneously over the Internet. The technique features a mesh distribution network with multiple delivery paths between terminals to increase fault tolerance and connections that are controlled in consideration of the terminal communication environment. We developed the connection functionality so that the larger the uplink bandwidth of a terminal is, the more lower-ranked terminals can connect to it, and terminals with large uplink bandwidths are ranked as high as possible as to their connectivity in the distribution network. We also developed a connection function to allow terminals to relay content, even if they are positioned behind a network address translator NAT and cannot normally receive connection requests from the Internet by having the terminal connect to a terminal outside the NAT. We prototyped a distribution system with the above functions and used it at the STRL Open House 2010 and the NHK School Chorus Contest. The distribution network was stable in performing these tasks and it reduced server load Figure 4 We will increase the network s stability by developing control methods that autonomously change the connection state according to conditions at each terminal, and we will conduct simulations to confirm their effectiveness. Future directions We will cooperate with other NHK departments to finalize the system specifications of Hybridcast and develop verification systems as quickly as possible. We will continue to promote Hybridcast at events such as the STRL Open House, and we will post draft specifications and open discussions with operators and vendors. We will work toward standardization and introduce actual Hybridcast services. 1 H. Katoh: An Overview of Research at NHK STRL on Technologies for Fusing Broadcasting and Communications, NHK STRL R&D, No. 124, pp in Japanese 2 K. Matsumura and Y. Kanatsugu: Hybridcast TM System and Technology Overview, NHK STRL R&D, No. 124, pp in Japanese SNS, CGM We are researching social networking services SNS in which broadcasting is a starting point to promote interactions among viewers and consumer generated media CGM where viewers publish their own video content. Social TV We are conducting research on a Social TV system, with the goal of creating a public forum by using TV program viewing as the starting point. This public forum will make use of various devices familiar to viewers, such as televisions, PCs and mobile devices, and provide a communications space on the Internet for them to post and share information and exchange opinions. We are studying various elemental technologies that generate activity on social networks, such as video distribution, video processing, program recommendation, and user-operation analysis. We are also studying platforms integrating these elemental technologies. In FY2010, we proposed a method for recommending programs using SNSs. Comments related to programs are entered as in a famous SNS, and method selects the recommendations of popular, i.e., charismatic, users. Comments of users linked to the current user and charismatic users are used to make inferences about programs they have seen, and these inferences in turn form recommendations about programs to be broadcast. We developed a technology to analyze comments in real time and in a way that suits the display device being used. We prototyped a presentation methods of personalized services to multiple devices Figure 1 16 NHK STRL ANNUAL REPORT 2010

19 2 Advanced digital broadcasting services 2.1 Technologies for fusing broadcasting and communications Viewer content creation and distribution systems Television PC Mobile terminal Figure 1. Various device presentation of real-time comment-analysis result We are advancing with our research on Omni-Resource Media, an Internet TV system that allows viewers to create, publish, and view programs in a simple, blog-like style. Our previous system for producing computer graphic CG video content from program descriptions written in TV program Making Language TVML ran locally on a PC. In FY2010, we developed a cloud-based system by putting all of the functionalities onto a server 1 and exhibited it at the STRL Open House By making the system cloud-based, various other services can be provided. Programs can be made to incorporate news materials automatically and have links with other information available on the web. Also, to create more sophisticated direction styles, we developed a method that allows CG characters to change their behavior automatically based on rules defined by the user and in accordance with their current state 2 We also extended TVML to give characters more detailed facial expressions. TVML-related technology was used in programs and events held at the Broadcast Center and at regional stations and during a program celebrating the 80th anniversary of the Fukuoka Station. For example, high school students in the Digista Teens program and participants in the NHK Bunkasai festival used TVML tools to create video content. 1 N. Hamaguchi, H. Kaneko, M. Doke, S. Inoue and I. Kumazawa: A study on live server-side rendering of CG-based video content, FIT 2010, No. 3, K-039, pp in Japanese 2 M. Doke, N. Hamaguchi, H. Kaneko and S. Inoue: Prototype of Advanced APE Combined Template Type and Adaptive Type Processes for Production Direction, FIT2010, No. 3, K-040, pp in Japanese Figure 2. teleda trial site We are continuing our work begun in FY2009 to build the teleda social TV service system. We developed APIs for extending teleda s functionality and analyzed social-network data, and incorporated our findings in the recommendation technology described above, a system for generating snapshot video that automatically summarizes a program in a 10- second clip, and in an analysis of users viewing tendencies. We conducted a medium-scale service trial with about 1000 people by linking teleda with the NHK Net Club authentication function. The trial staged exclusive events and provided services related to programs, and we collected data on viewing habits and comments. We learned that a social TV service can be the impetus for people to view many and varied programs and can stimulate communication among viewers. Related services External web services Materials search TVML scripts CG video TVML scripts CG video Web interface Materials search function CG video content creation function Live distribution function Cloud-based CG video content production system on the Internet Figure 3. Cloud-based CG video content production system NHK STRL ANNUAL REPORT

20 2 Advanced digital broadcasting services 2.1 Technologies for fusing broadcasting and communications Security platform We are conducting research on the security of Hybridcast and on protecting broadcasting content rights and personal information in broadcasting and communications environments. Security technology in Hybridcast receivers We are studying security system in the Hybridcast environment which includes broadcasting and communications networks. In FY2010, we designed a basic management model for applications to be provided through communications networks. We developed a method to control an application appropriately when emergency warning broadcasts are being presented. We also designed the basic syntax of security policies for controlling the presentation of applications and developed a method for transmitting them. The World Broadcasting Unions WBU and the Asia-pacific Broadcasting Union ABU have held meeting to discuss how to preserve the viewing experience on Hybrid and Internet Television. We submitted a proposal to ensure that emergency warnings have a highest priority to be presented to the ABU task group. We developed a digital signature scheme to verify that an application originated from a certified developer or distributor and has not been modified when it is distributed to users. This scheme verifies signatures by using only one public key stored in the receiver. The signing keys of application developers or distributors can be securely updated through communications networks. Moreover, the scheme enables the signing key to be revoked quickly 1 Identity-web services framework for broadcast and personal communications services, and device linking technology To enable broadcasting services to be fused with personalized communications, we developed an ID federation framework that includes web service federation and methods for the receiver to integrate and display personal communications data and broadcast content, and we built a prototype system 2 Figure 1 The framework ensures privacy when communications providers offer services through data broadcasting. This research was conducted in cooperation with the Tokyo Institute of Technology. Multiple users sometimes access their personal communications services on the same receiver device. We studied ways of securely managing personal information and controlling access to the receiver, and we developed a prototype broadcast receiver with an account manager. We also studied methods for linking mobile devices to receivers and studied the management requirements for such links. Advanced conditional access system We are researching a downloadable conditional access system CAS that can securely update and add processing functions such as encryption key management or encryption algorithms by sending them through broadcasting and communications channels. In FY2010, we designed a downloadable CAS system that works with the current form of digital broadcasting and developed a downloadable CAS receiver prototype based on the experimental CAS system that we built from field programmable gate arrays FPGA in FY2009. We also clarified the issues of implementing the downloadable CAS module in the system LSIs of Figure 1. Example of displaying broadcast services and personal communications services simultaneously using an ID federation framework pension data the receivers. We also developed an entitlement management message EMM technology for content protection system in conformity with the ARIB standard and submitted the specifications of this technology to ARIB for standardization. Encryption and authentication We developed a public key encryption scheme for a secure content delivery system that has multiple content providers. The scheme enables each receiver to have a distinct decryption key and decrypt multicast content encrypted with a single public key. The CAS of the current form of digital broadcasting has a message encryption function for individual viewers, and we made it so that this scheme can be used to encrypt personal messages. Only the viewer specified by the provider can decrypt the encrypted message. We also developed a broadcast encryption scheme that can identify the originating receiver when a private key leaks from a receiver. The scheme is secure and the sizes of public key, private key, and ciphertext are smaller than those of the existing schemes. This research was conducted in cooperation with the National Institute of Advanced Industrial Science and Technology. Digital watermarking We developed digital watermarking technology for the purpose of preventing unauthorized uploads of broadcast programs and identifying content within the broadcast station. We made improvements to parallel processing algorithms for embedding digital watermarks in content and made it possible to embed more than 80 bits of data in real time without degrading image quality. We also optimized the detection algorithms so that they could detect watermarks quicker. We made a generaluse software library for the embedding and detection software, so that watermarking functions could be easily added to filebased systems within broadcast stations. The watermarks are very robust to various video encoding methods and can be detected in video that has been reduced in size or converted into analog. We used this technology to watermark and send live video over the Internet and found that watermarks in the encoded video could quickly be retrieved 3 Also, in cooperation with the NHK Engineering Administration department, we developed a real-time system for embedding and detecting identifiers in source video from weather cameras and other systems. This research was conducted in cooperation with Mitsubishi Electric Corporation. 18 NHK STRL ANNUAL REPORT 2010

21 2 Advanced digital broadcasting services 2.2 Transmission technology for handheld and mobile devices 1 G. Ohtake and K. Ogawa: Application authentication for hybrid services of broadcasting and communications network, SCIS2011, 3E in Japanese 2 C. Yamamura and A. Fujii: Federated Identity Management and Web- Services Framework for Broadcasting-Communications Hybrid Services, Broadcast Technology, No. 43, pp H. Ohmata, K. Majima, T. Inoue, T. Sugai, H. Yamada, and H. Nishikawa: Applying Digital Watermarking Technology to Video Distribution System over the Internet, ITE Annual Convention, in Japanese 2.2 Transmission technology for handheld and mobile devices VHF-Low-band multimedia broadcasting We are advancing research toward implementing multimedia broadcasting for handheld terminals using VHF-band channels 1 to 3 VHF-Low-band after analog television broadcasting ends. In FY2010, we established a VHF-Low-band experimental test station on the roof of NHK STRLandtestedthereception area for real-time broadcasts. We also evaluated reception characteristics for storable broadcast services by using data carousel transmission methods. For the real-time broadcast reception area, we conducted field tests using the VHF receiver equipment built in FY2009. We determined the relationship between received electric field strength and received TS-packet error rate in urban areas and verified the link budget for mobile reception 1 At the same time, we measured man-made noise in urban areas and studied the influence of this noise. We elucidated the reception characteristics of data-carousel storable broadcasts by conducting laboratory experiments. We studied the relationship between the number of content retransmissions and storage success rates in a mobile reception environment by using the size of download data blocks DDB and content size as parameters 2 Storable broadcasts should be able to be classified by genre e.g., political or economic news and usual news items like weather reports should be updated by overwriting with the latest information. We devised a transmission method that attaches storage instructions appropriate to the type of content news, weather report, drama series, etc before transmission so the receiver can automatically classify and store the content. This method was included in the draft operational regulations for VHF-Low-band multimedia broadcasting. We also updated the compact receiver equipment prototyped in FY2009 by adding new functionalities, including update, series scheduling, and content-type identification, and we verified the operations of these functions Figure 1 Handheld and portable antennas We are researching a compact antenna for handheld receivers to be used in VHF-Low multimedia broadcasts.we have applied a folded-type rectangular loop antenna structure and using the grounded plane and case to resonate in VHF-Lowband 3 It measures mm, so it can fit in an Figure 2. Prototype compact antenna for handheld receivers e-book or car navigation devices Figure 2 We also conducted field tests in which we connected the antenna to a receiver in a simulated handheld-device environment. Next-generation digital terrestrial broadcasting transmission system for handheld and mobile reception ISDB-T is the current digital terrestrial television broadcasting system in Japan. It simultaneously provides Hi-Vision broadcasting for fixed reception and One-Seg service for handheld and mobile reception. We are conducting research on a largecapacity transmission scheme that can transmit Hi-Vision programs to handheld and mobile terminals even in severe reception conditions. The transmission method involves multipleinput multiple-output orthogonal frequency division multiplexing MIMO-OFDM In FY2010, we conducted laboratory experiments with the modulator and demodulator developed in FY2009 and measured the channel response perfomance of orthogonal coded scattered pilots. To improve reception performance, we used 2 2 MIMO and added tuner and automatic gain control AGC components to the demodulator. We also made a prototype dual-polarized antenna and conducted field experiments with it where we evaluated the reception field strength of each horizontal and vertical signal. Figure 1. Compact prototype receiver 1 H. Sanei, M. Okano and S. Nakahara: A study of Contents Schedule Table for mobile multimedia broadcasting in the VHF-Low band, ITE Technical Report, Vol.34, No.33, BCT , pp in Japanese 2 M. Nakamura, H. Sanei, M. Okano and S. Nakahara: Field Experiment of mobile multimedia broadcasting signal in the VHF-Low band, ITE Winter Annual Convention, in Japanese 3 N. Kogo, M. Nagasaka, S. Tanaka, K. Shogen and K. Ito: Folded Type Rectangular Loop Antenna in VHF-Low Band, IEICE Society Conference, B in Japanese NHK STRL ANNUAL REPORT

22 2 Advanced digital broadcasting services 2.3 Technology for improving reception 2.3 Technology for improving reception Interference cancellation technology As a countermeasure against co-channel interference in digital terrestrial broadcasting, we developed a pre-fft Fast Fourier Transform adaptive array technology that uses a remodulated time-domain signal with channel distortion as a reference signal. In FY2010, we developed a booster-type, compact interference canceller prototype incorporating an integrated signalprocessing chip and exhibited it at the STRL Open House We also conducted field tests on the prototype with an FPGA signal processing component built in FY2009. For the tests in Tanogawauchi in Hiroshima Prefecture, broadcasting waves from the Akichiyoda station were affected by interference from waves from the Geihoku Hachiman station, and waves from the Miyoshi station were affected by interference from the Hamada station. We confirmed that all channels could be received when the interference canceller was operating. For the tests at Makubetsu in Hokkaido, it was initially thought that reception characteristics would be degraded by interference from digital terrestrial broadcasting waves from different programs coming from other regions, but upon closer study, the cause of the degradation was found to be multi-path interference exceeding the guard interval GI due to reflections from mountains. The interference cancellers improved the characteristics. We also confirmed that reducing the separation between receiver antennas provided an improvement in interference cancellation performance under the condition of a wider arrival angle spread. In tests conducted in Nishi-izu in Shizuoka Prefecture, we received broadcasting waves from the Shizuoka station and confirmed that interference-cancelling equipment improved reception affected by multi-path interference exceeding the GI from stations in Hamamatsu and Fujieda. In other research, we developed a prototype post-fft adaptive array for an on-air relay compensator. We improved the signal processing algorithm by adding reciprocal filter processing to determine suitable frequency domain weighting coefficients for low-d/u multi-path environments Figure 1 multi-path equalization in the frequency domain. We are working on a practical implementation of this technology. In FY2010, we field-tested a prototype with enhanced robustness against low-d/u multi-path interference inside the GI at Nishi-izu and other locations, and we also participated with the Reinan Cable Network in testing the equipment. In other research, we developed a technique called adaptive leak that detects the delay profile of multi-path echoes when the multi-path delay spread is large. We also reduced the size of the equalization circuit and built a set-top box incorporating it. Improving mobile reception performance of Hi-Vision digital terrestrial television broadcasting We are progressing with our research to enable stable reception of Hi-Vision programs in automobiles and buses and to enable reception of digital terrestrial television broadcasting over a wider reception area. In FY2009, we prototyped an eight-branch space diversity receiver with iterative decoding. The receiver has eight receiving antennas, and it features eight-branch space diversity 8Br-SD with maximal ratio combining and error correction using iterative Viterbi and Reed-Solomon decoding. In FY2010, we conducted laboratory experiments to confirm that the prototype receiver improved reception performance and field experiments to see if it could expand the reception area Figure 2 In the field experiments, the conventional fourbranch space diversity 4Br-SD receiver obtained reception rates of 90% or higher in areas with a fixed reception field strength of 75 dbμv/m or greater calculated for a receiver antenna height of 10 m whereas the 8Br-SD receiver with iterative decoding obtained 90% or higher reception rates in areas with 70 dbμv/m or greater 1 1 Y. Narikiyo and M. Takada: Field experiments of ISDB-T HDTV mobile reception using 8-branch space diversity with iterative decoding, ITE Technical Report, Vol. 35, No. 5, BCT , pp in Japanese Equalizer for multi-path interference outside the guard interval As a countermeasure against multi-path interference outside the guard interval GI we developed a technique that performs an FFT on several OFDM symbol durations and performs dBμV/m Without reciprocal filter processing With reciprocal filter processing Bit Error Rate BER Interference : D/U 6dB, Arrival angle: 50deg. Multi-path delay : 120μs, Arrival angle 20deg. CNR : 30dB Required BER multi-path D/U db Figure 1. Interference cancellation characteristics in an SFN environment multi-path D/U characteristics Both 4Br-SD and 8Br-SD with iterative decoding reception possible Only 8Br-SD with iterative decoding reception possible 4Br-SD or 8Br-SD with iterative decoding reception impossible Figure 2. Mobile reception field experiment results 20 NHK STRL ANNUAL REPORT 2010

23 2 Advanced digital broadcasting services 2.4 Enhanced ISDB-T 2.4 Enhanced ISDB-T Multiband ISDB-T system The channel bandwidth for terrestrial television broadcasting varies from 6, 7 or 8 MHz depending on the country and region of the world in which it is deployed. ISDB-T is an internationally recommended digital terrestrial television system developed in Japan, and its transmission parameters have been standardized for all the bandwidths, while its modulators, receivers and other equipment are available only for the 6 MHz bandwidth. ISDB-T is being used or being deployed in 11 other countries, mostly in Latin America, and all of them use a 6 MHz bandwidth. To enable ISDB-T in Asian and African countries that use 7 or 8 MHz bandwidth, we developed a prototype compact multiband ISDB-T receiver set-top box able to receive any of these bandwidths Figure 1 In FY2010, we conducted laboratory tests on the prototype to assess its minimum input power, Gaussian noise characteristics, multi-path characteristics, and interference characteristics. We confirmed that its characteristics are as good as those of the ISDB-T receivers used in Japan Figure 2 We exhibited this receiver in South Africa and in Brazil MHz (GI duration 126μs) 7MHz (GI duration 108μs) 8MHz (GI duration 94.5μs) Required D/U db Multi-path delay time μs Figure 1. Appearance of prototype receiver Figure 2. Performance against multi-path interference NHK STRL ANNUAL REPORT

24 3 Universal broadcasting services Japanese lifestyles are diversifying as the information infrastructure continues to expand, and digitalization is diversifying the variety of content and functions that viewers desire of broadcasting services. Moreover, Japanese society is aging and becoming more internationalized ; this means people s physical condition, including their vision and hearing abilities, and cultural background, such as their native language, are also diversifying. The most fundamental function of broadcasting is to provide the basic information to every person in society. As such, we must conduct research to respond to these changing conditions through the use of information technology. Our goal is to make presentations of information more user-friendly for everyone. To this end, we have developed a receiver that translates data broadcasts into braille and speech for visually impaired users and have built a haptic system for presenting information and evaluated its cognitive characteristics. We have also developed elemental technologies to expand the range of videodescription and sign-language broadcasting services. Regarding our closed-caption technology using speech recognition, we improved the word accuracy for dialog and other speech in news programs, and we helped to develop a news-archive search system for program creators and editors. We are also studying methods to suppress background sound and to enhance speech, by using speech and acoustic processing to modify program audio at the receiver and make content easiertohearfortheelderly. In natural language processing, we are developing language transformation and analysis methods to provide barrier-free services in various languages. We made progress on statistical Japanese-to-English news translation and on support for re-writing Japanese in news articles into easier forms for non-japanese living in Japan. We also continued to study how to increase the precision of categories for analyzing reports on programs gathered from viewers. We conducted research to promote the use of our video archives. In particular, we studied technologies to extract semantic structures from video and to process texts about programs so they can be used to make searches and recommendations. We also worked on a metadata production framework and developed a system to search for and recommend programs. These technologies are being used by NHK On Demand, NHK Public Video Library, and NHK Creative Library. Also, in user-interface technology, we began research on recognizing viewers situation, and offering services that are appropriate for the situation, including a basic user study and building a prototype system. In other research, we conducted experiments on how programs are viewed and what psychological effects they elicit, by examining brain activity and eye-gaze directions of viewers. We also developed methods to measure the degree of discomfort viewers may feel when watching a video from its physical characteristics. 3.1 User-friendly information presentation STRL is engaged in research on user-friendly information environments that will allow people with vision or hearing impairmentstogetmorefrombroadcasts. Speech synthesis Barrier-free information digital broadcasting technology The barrier-free information reception and presentation system under development is for those with visual disabilities and can be adapted to the type and degree of the user s disability. It delivers data broadcasts and the Electronic Program Guide EPG and can also output braille subtitles for deaf and blind people. In FY2010, we developed a practical set-top-box system with excellent accessibility Figure 1 Braille, finger braille equipment Remote control set-top-box receiver Screen display for those with weak vision Figure 1. Barrier-free information reception/presentation system for the visually impaired persons 2D/3D haptic display technology We are advancing with our research toward a Touch TV that will convey 2D and 3D information through the tactile and kinesthetic senses. In FY2010, we used a 2D tactile information display to explore presentation methods that reduce the burden for visually impaired users when they search for information or try to remember the locations of specific buttons. By studying the effect of 22 NHK STRL ANNUAL REPORT 2010

25 3 Universal broadcasting services 3.2 Closed-captioning through speech recognition using sound and touch together on user s memory and learning, we found a tendency toward improved recall for an object when the user received a tactile clue at its location. To fashion guidelines for the design of equipment and Figure 2. Multi-finger haptic presentation presentation of 3 D information to fingers, we conducted shape-recognition evaluations comparing the characteristics of real and virtual objects. In FY2010, we evaluated how someone uses their fingers to feel the shape and size of objects, as well as how various factors affect the recognition of shape and size Figure 2 The results indicated that there is a significant touch illusion whereby objects are perceived to be larger than they actually are if the user touches it with only one finger, that the reaction force on the palm has an effect on the recognition of the correct size when multiple fingers are used, and that the presence of gravity affects shape recognition 1 Part of this research was conducted in cooperation with Tokyo Women s Christian University and KGS Corporation. Figure 3. Computer graphic face model Display region of direction from e-manuscript Display for standard-length descriptions Voice/non-voice/music determined automatically Current descriptive manuscript entry area Pursed lips Drawn straight Comment entry area Text exceeding standard length shown in red Display/selection of candidate text from stage directions Elemental technologies for a sign-language broadcast service To expand sign-language broadcasting services, we are studying the feasibility of machine translation from Japanese text into sign-language computer graphics. Non-manual signals such as facial expressions are important parts of sign language. In FY2010, we built a face-shape model Figure 3 and extended the TV program Making Language TVML to enable the face shape model to express non-manual signals 2 We also developed a simple method for automatically connecting CG signing words and prototyped a CG signing translation system that uses translation examples. This research was conducted in cooperation with Kogakuin University. Support system for production of audio description broadcast programs We are conducting research on production support tools to expand the range of audio descriptions for those with visual impairments. In FY2010, we built a prototype support system for producing audio descriptions more efficiently 3 Figure 4 The Figure 4. Support system for production of audio description broadcast programs prototype system can automatically detect non-voice and nonmusic sections where descriptions can be inserted and displays the length of description that can be entered. It also uses language processing technology to generate supporting text from the stage direction notes, if available in electronic form, and these can be referenced when creating the audio description broadcast text. 1 T. Handa, T. Sakai and H. Shinoda: An Evaluation of Size Recognition in Multi-Finger Haptic Environment, HCG Symposium, HCG2010-C6-2, pp in Japanese 2 H. Kaneko, N. Kato, T. Shimizu, S. Inoue and Y. Nagashima: A Study on Generating Sign Language Animation with Non-Manual Signals, HCG Symposium, HCG2010-A5-2, pp in Japanese 3 Y. Yamanouchi, N. Kato, T. Imai and N. Hiruma: Evaluation of Trial Manuscript Production Support System for Audio Description, IEICE General Conference, A in Japanese 3.2 Closed-captioning through speech recognition One of the goals of our speech recognition research is to expand the range of closed-caption broadcasts for the hearing impaired and elderly. Closed-captioning system for news programs Closed-captioning in sports programs such as professional sumo wrestling are created by recognizing utterances rephrased by a re-speaker. To produce captions more efficiently in news programs, we are developing a system Figure 1 to directly recognize speech of announcers, reporters and commentators and switch to a re-speaker for interviews and other scenes that are difficult to recognize 1 2 In FY2010, we built a compact closed-caption production system using a notebook PC for speech recognition and a touch-panel PC for error correction. We also increased word accuracy for free conversations and reduced the amount of operator work required to correct errors. We studied methods of automatically inferring the original news manuscript from the speech recognition result. We used a broadcast-news transcription system to develop a news archive search for program creators and editors. The transcription system recognizes speech in all of the news programs and stores the results together with the audio and video. NHK STRL ANNUAL REPORT

26 3 Universal broadcasting services 3.3 Speech and audio signal processing for the elderly Audio Speech recognition Error correction Text Figure 1. Closed-captioning system for news programs Spontaneous speech recognition Closed-caption broadcast...we now continue the news. Spontaneous speech in news programs is relatively freely spoken speech on a particular topic, and it is not read from a script. It includes free conversations between host and guests, and speech recognition performance falls because of errors in recognizing particular phrases and lack of clarity due to rapid speech. To deal with such recognition errors, we studied discriminative training methods using linguistic and acoustic features of words 3 and iterative adaptive training methods for features of individual speaker voices. We evaluated the system by using conversations from the program Today s close-up. The recognitionratewas88.7%.wewereabletoreducetheerror rate by half from that of FY2007 error rate of 11.3% vs S. Homma, A. Kobayashi, T. Oku, S. Sato and T. Imai: Live Closed- Captioning with Robust Speech Recognition for a Spontaneous- Spoken Style, APSIPA ASC 2010, PO3.2, pp T. Imai, S. Homma, A. Kobayashi, T. Oku and S. Sato: Speech Recognition with a Seamlessly Updated Language Model for Real-Time Closed-Captioning, INTERSPEECH 2010, P3.5, pp A. Kobayashi, T. Oku, S. Homma, S. Sato, T. Imai and T. Takagi: Discriminative Rescoring Based on Minimization of Word Errors for Transcribing Broadcast News, INTERSPEECH 2008, ThuSe2.P2-5, pp Speech and audio signal processing for the elderly We are developing technology for adjusting program sound to make it easier for the elderly to hear dialog and narration in the presence of background sounds. In FY2009, we developed a program sound production support device that calculates the perceived loudness measured in phon units of the narration and background sound and evaluates the difficulty that an elderly person would have in hearing the narration by taking the difference between the two loudnesses 1 In FY2010, we evaluated whether the volume balance which was adjusted according to the evaluated results was appropriate or not. We took sound judged by elderly listeners to have larger background sound measured at 72.5 phon or more and adjusted the volume balance down to an average narration level of 70 phon. Elderly subjects evaluated the adjusted sound, and the results confirmed that the adjustments were suitable; the number of subjects judging the background sound as too loud dropped from 25% to 15 2 We are also studying methods for suppressing background sound and enhancing speech to make content easier to hear by adjusting the program sound on the receiver side. To suppress background sound, we developed a method that uses a statistical model to find speech segments and then uses the correlation of signals between stereo channels on the speech segments to estimate narration sound and background sound and control the sound balance appropriately. In other research, we conducted subjective evaluation tests with elderly subjects and found that speech with no background sound could be made easier to hear by increasing the dynamic range of its pitch. 1 T. Komori, H. Dan, T. Takagi, K. Shoda, K. Kurozumi, S. Komiyama, H. Hoshi and K. Murakawa: An Investigation of the Audio Balance Control Based on the Loudness Level, IEICE Transactions, Vol. J92-A, No. 5, pp in Japanese 2 T. Komori, N. Seiyama and T. Takagi: A Verification on Objective Evaluation Method of Appropriate Broadcasting Sound Balance for Elderly Listeners, Proceedings of Autumn Meeting of Acoustical Society of Japan, 1-R in Japanese 3.4 Language processing for language barrier-free services Japanese translation/conversion assistance technology To provide useful services for non-japanese inside and outside of Japan, we are studying how to automatically translate news from Japanese to English and make Japanese news easier for non-japanese to understand. We have begun studying statistical machine translation from Japanese to English 1 There are few direct translations of NHK news available, so training a statistical model using existing methods is difficult. In FY 2010, we developed a method for training a phrase-based strings of several words translation model from news data. We also built a translation system that incorporates this method with existing decoders and other elements and prepared parallel texts from 1,000 news stories for an evaluation. We are interested in giving viewers an option of simplified Japanese in news. In FY2010, we developed a method to extract vocabulary in a news database, regardless of genre or timeframe, in order to determine the basic vocabulary to be used for writing news at NHK. Using this method, we extracted approximately 6,000 candidate words that cover 95% of all news in daily services. In addition to deciding on a provisional standard for easy Japanese, we developed an interface to assist editors in rewriting. We also prototyped a news-comprehension test system to evaluate the level of Japanese text and conducted preliminary evaluation tests using native speakers of Japanese 2 Opinion analysis technology We are developing opinion analysis technology so that we can quickly respond to the opinions of NHK viewers. In FY2010, 24 NHK STRL ANNUAL REPORT 2010

27 3 Universal broadcasting services 3.5 Content search and recommendation technology we tried to increase the accuracy of opinion classification. People express their opinions with large variations in sentence structure, expressions, and length and we developed a robust machine learning method that can overcome such variations and correctly predict the opinion category of the input sentences. 1 T. Kumano and H. Tanaka: Calculating translation probabilities for bilingual sentential alignment using monolingual and bilingual word co-occurrence statistics, Annual Meeting of the Association for Natural Language Processing, P in Japanese 2 H. Tanaka and H. Mino: Reading Comprehension Test of Easy Japanese News by Japanese Native Speakers, Annual Meeting of the Association for Natural Language Processing, B in Japanese 3.5 Content search and recommendation technology To make archives easier and more versatile to use, we are researching technology for identifying the semantic structures in video and processing text for program search and recommendation purposes, and are promoting our metadata production framework. We developed a program search and recommendation system incorporating these technologies. Video structure extraction for program search and recommendation We are advancing with our research on video structure extraction technology to divide a program into sections that have meanings and attach metadata describing the content to the each section. In FY2010, we developed methods for detecting objects and events in video and used them to identify the semantic structure and create video summaries of programs. We conducted experiments on dividing news and journal programs into scenes that span multiple shots but have one overall meaning. In particular, we developed a method for dividing news scenes using recurring features in the video. We also studied a method for dividing up journal programs that contain mixed scenes by focusing on closed-captions expressing changes in time or place. We also developed a method for detecting video segments with open captions and identifying where captions appear in the video frame, with the goal of extracting semantic information from the open captions. By detecting the region where the text appears by using combinations of image features, we were able to improve detection accuracy by approximately 20% over earlier methods 97% match ratio and by using a search region with a scanning window, we were also able to handle open captions that move or change in color or size over time 1 To generate summary videos automatically for browsing programs, we developed a learning algorithm that searches for objects and events occurring in video. The new method is faster than earlier methods, and it can search for 130 types of object and event. Text processing for program search and recommendation Along with video analysis technology, we are also advancing with our research using text processing techniques to search for and present information. In FY2010, we made improvements to the method developed in FY2009 that searches for relevant programs using text similarities based on word frequencies and named entity categories. We also worked on new search, recommendation, and presentation methods. Program summaries in archives contain limited amounts of text information to compute the relevance of programs to each other, so external text information must also be used. In FY 2010, we developed a method that uses Wikipedia as a source of external information to present programs prioritized for relevance to people and events attracting attention in the world 2 Expressions that are attracting attention are identified by checking Wikipedia s revision history. We also developed a method for automatically acquiring news-article structural elements and building a news ontology which describes relationships among concepts comprising the news by gathering and integrating information from Wikipedia s table of contents. We also prototyped a system that uses the news ontology to present search results for each of the structural elements in the news. In other work, we studied a method for structuring news by referring to the NHK Radio and Television Yearbook. The method structures news by making associations between sentences in the NHK Radio and Television Yearbook and sentences in the NHK News. We incorporated this method into a news search system that presents detailed information from the NHK News 3 Part of this research was conducted in cooperation with the National Institute of Information and Communications Technology NICT Metadata production framework Our Metadata Production Framework MPF uses media analysis to produce metadata efficiently, and in FY2010, we participated in the EBU EC-M SCAIE European Broadcasting Union, Expert Community on Metadata, Study of Content Analysisbased Automatic Information Extraction in Production in an effort to promote the MPF. The goal of the EBU body s study is to evaluate automatic metadata generation, and the body decided on a common metadata format that is an extension of the metadata model used in the MPF. We also contributed to international standardization of a common metadata format in the form of the new MPEG-7 profile at MPEG. Program search and recommendation systems Using these video structure extraction and text processing technologies, we developed a system that searches for and recommends relevant programs for viewers and a program scene search and browsing system for creators and producers. The relevant-program search and recommendation system for viewers uses text processing to calculate similarities between program descriptions and implements a general query interface accessible by the HTTP communications protocol. We also prototyped a system for the program recommendation service of the Hybridcast system that links broadcasting and communications. In cooperation with the NHK On Demand Bureau, we developed a system that stores relevant-program information by searching for relevant programs by looking for similar program overview texts and metadata naming actors and genres. Using this system, we began trials on presenting programs relevant to those selected by NHK On Demand users on the PC services NHK STRL ANNUAL REPORT

28 3 Universal broadcasting services 3.5 Content search and recommendation technology 3.6 TV user interface technology screen Figure 1 This technology was also incorporated in a relevant-program recommender system for the NHK Public Video Library, operated by the Copyright and Archives Center. We prototyped a search system for finding relevant video clips for the NHK Creative Library. The system calculates the similarity between textual descriptions of video and similarity between images 4 Figure 2 We devised a method to search for similar thumbnail images by comparing color and texture characteristics of block units. We increased the search processing speed by pre-classifying images using block-unit component data. We worked on a search method that takes query sketches drawings used as search queries and uses principal color lists and principal block images extracted from the search database. We made it usable on a web browser and began a public test of it on the NHK s website. The test will gather data from the general public for evaluating the search functions and usability of the system. In other research, we examined technology to adapt program recommendations to the preferences and viewing styles of individuals. To develop personalized ways of displaying search results, we conducted a survey about program recommendations and found that viewing behavior depends on the time frame, i. e., weekday or holiday, and viewers have particular genres that they often watch. With this information in hand, we built a system that increases the priority of particular genres in the search results. We also prototyped a video search and retrieval system for creators and producers. The system searches by analyzing videos for factors such as camera work, similar colors or patterns, and the presence or absence of faces or open captions, and by the similarity of textual descriptions of the content. The system is intended for use with large quantities of video, and it has functions to automatically extract and index the metadata required for the searches described above. Figure 1. NHK On Demand relevant-program display screen 1 Y. Kawai, H. Sumiyoshi and M. Fujii: A method for detecting display regions of open caption from single video frame, ITE Winter Annual Convention, in Japanese 2 J. Goto, M. Miyazaki, H. Tanaka and A. Aizawa: Relevant TV Program Retrieval Using Wikipedia Revision History, ITE Annual Convention, in Japanese 3 N.Kato,M.Miyazaki,J.GotoandH.Tanaka: Structuringnewsarticles by using NHK yearbooks, IEICE General Conference, D in Japanese 4 H. Sumiyoshi, T. Mochizuki, J.Goto and M.Fujii: Relevant Video Content Recommendation System, ITE Annual Convention, in Japanese Figure 2. Relevant-video clip search system for the NHK Creative Library 3.6 TV user interface technology We began research on television user interfaces in FY2010. We are using a combination of sensing and image recognition technologies to determine the status of viewers in order to provide appropriate services that take their status into account 1 As a basis for this research, we conducted a survey of television viewing behaviors in households and summarized the results into a set of viewing status patterns. We are studying human recognition technologies as ways of obtaining the viewing status. To identify multiple viewers in real time, we developed a rapid and robust face recognition method combining skin color selection, face detection, and facial feature point searches. Also, when we built and trained the database for face recognition, we automated the facial image registration by using a novel image alignment method. We developed a method to identify a variety of human actions during Recognition of various status elements Figure 1. Recognition results for the prototype user interface 26 NHK STRL ANNUAL REPORT 2010

29 3 Universal broadcasting services 3.7 Technology for estimating viewer psychological state TV viewing by bundling several motion-video feature points and classifying them according to the actions, and a gesture recognition method that uses temporal changes in object depth and image data. We also developed a way of classifying changes in facial features during TV viewing, on the supposition that there will be a correlation between changes in facial expression and amount of interest in the program. We built a prototype television user interface system incorporating the human recognition methods described above. The system uses various image recognition processes to determine a viewer s status Figure 1 and has a mechanism for taking this status into account when it recommends and presents relevant content. 1 M. Naemura, M. Takahashi, M. Okuda, H. Mitsumine, S. Clippingdale and M. Fujii: TV Interface Based on Viewing Context, ITE Journal, Vol. 64, No. 12, pp in Japanese 3.7 Technology for estimating viewer psychological state To be able to analyze objectively how a program is being watched and what psychological effect it is having on viewers, we are researching ways of measuring a viewers psychological states from their brain activity, viewing behavior eye-gaze movement, attention attitude, etc. and other factors. Analysis of brain activity In FY2010, we began developing technology to analyze viewers states of attention by using functional near-infrared spectroscopy fnirs Conventional research on brain activity has involved giving a subject watching stimuli various tasks to change his/her attention state and the corresponding brain activity would be measured. However, brain activity is affected not only by the attention state but also by various other factors, so it is difficult to extract only the effect of changes in attention from all the other activity. Our solution was to measure brain activity while subjects systematically changed the direction of Figure 1. Brain regions thought to be related to spatial attention different colors represent attention in different directions their attention within a field of view and to extract only the activity which was synchronized with these changes. As a result, we found brain regions that showed characteristic responses to the horizontal shifts of the direction of attention, mainly in the temporal areas 1 Figure 1 We will continue to analyze attention phenomena in more detail. Eye-gaze analysis We are analyzing eye-gaze to identify the areas of program video that attract a viewer s attention and interest. In FY2010, we conducted experiments to identify where attention is directed by measuring people s gazes and how image contrasts and hue changes attract the viewers attention 2 We also improved our system for measuring gazes by expanding the field of view over which measurements can be taken. We plan to use this system to measure the psychological effects of ultra-highpresence video systems such as Super Hi-Vision on viewers. Content can also have undesirable effects on viewers. In particular, with recent increases in the size of televisions in homes, the number of viewers experiencing discomfort due to motion or flashing in video may be increasing. In response to this issue, in FY 2010, we developed software to estimate the level of viewer discomfort from the physical features in the video. We conducted subjective evaluations that confirmed the effectiveness of this approach. 1 M. Harasawa and H. Ishikane: Continuously moving RSVP task revealed neuronal activities related to position of spatial attention: an fnirs study, Vision Sciences Society 11th Annual Meeting, S. Umeda, K. Komine and T. Morita: Fusion parameter extraction of saliency map from the saccadic latency in a visual search task, ITE Annual Convention, in Japanese NHK STRL ANNUAL REPORT

30 4 Advanced contents production technology We are researching program-production technology for efficiently producing high-quality content, realizing new forms of expression in video, and supporting emergency reports. We are researching the use of broadband Internet Protocol IP networks as a means of transmitting video materials and distributing video without interruption by controlling the video encoder on the basis of the state of the connection. We are also researching a distributed server broadcasting system using cloud technology for continuous storage of program files and effective use of content. We developed a camera that can simultaneously measure distance and surface reflectance information by using an optical system with polarized split aperture. We are also developing systems to obtain real-space information about photographic conditions and use this information to perform sophisticated composition and processing operations. We are studying efficient methods to enlarge speech synthesis databases and developing equipment that reads out weather reports and adjusts the speech to fit within the available broadcast time. In our research on acoustic devices, we devised a push-pull type of structure that increases efficiency and improves the sound quality of our lightweight electroactive-elastomer loudspeakers. We used our narrow-directivity microphones in program production and studied ways to increase their directivity. We also continued our work on silicon electret condenser microphones, which have excellent mobility and reliability. We increased the efficiency of charge accumulation in these microphones, so that elements can be manufactured in a short time. We are studying the use of a transmitter antenna for 800-MHz-band FPUs currently used for road-race coverage for more stable video transmission. We are researching microwave-band FPUs and reducing the number of receiver base stations required for road-race coverage. We are developing full-duplex FPUs and techniques for channel estimation and equalization in the frequency domain. We are also researching a millimeter-wave FPU capable of high-capacity transmission using 120-GHz-band signals. In our millimeter-wave mobile camera research, we developed a return-link transmission system using the same millimeter-wave-band as the main-link video. This gives the wireless camera s operator the same level of control as he or she would have with cabled HDTV cameras. We used this camera for coverage of golf tournaments and the Kohaku Utagassen year-end song festival program. The millimeter-wave camera, which emits millimeter-waves to form an image, is able to take images of objects that are obstructed by opaque substances, such as thick fog and smoke. This fiscal year, we increased the unit s frame rate and improved image quality by integrating transmit and receive antennas into one, thereby suppressing undesired reflections. 4.1 Networked program production systems We are researching the use of broadband Internet Protocol IP networks for transmitting video materials and a distributed server-based broadcasting system for continuous storage and effective use of program files. Enhanced video transmission over IP networks Best-effort IP networks, which are shared by multiple users, are affected by packet loss and/or shrinkage of available bandwidth. Without a means to alleviate these problems, such networks are unsuitable for transmitting video materials. We are developing technology to enable live video to be transmitted without interruption on best-effort networks. For one-to-one live video transmissions, such as live reports from the field, we have made progress in developing transmission equipment to estimate the available bandwidth and quality of the network and control the video compression rate so that the encoded data and the error protection data to recover from packet loss will fit within the bandwidth. We also improved the compatibility of the encoder 1 For one-to-many or many-to-many live video transmission, such as when multiple video contributions are monitored with multiple receivers, overload and/or congestion tend to happen at the streaming server that distributes encoded signals to the receiving terminals and the uplinks and downlinks before and after the servers. Congestion occurs and video quality deteriorates when the number of receiving terminals increases. To alleviate this problem, we developed a video distribution system that avoids interruptions. It estimates the bandwidth at the potential bottlenecks above by monitoring the current number of videos and receivers and knowing the encoding bit rate of each video. It calculates target encoding bit rates that satisfy the bandwidth for every potential bottleneck below its capacity and makes the video encoders stay within those bit rates Figure 1 Broadcasting station system Broadcasting station systems are using more and more program files. As the number of these files continues to grow, we decided to begin research on the next generation of post filebased distributed server-based broadcasting systems. This system is composed of a wide-area distributed file sys- 28 NHK STRL ANNUAL REPORT 2010

31 4 Advanced contents production technology 4.1 Networked program production systems 4.2 Video production technology Rate-controllable encoder Bottleneck point Video, audio Target bit-rates Counts number of viewing terminals, calculates target rates, and controls encoder Figure 1. Rate-controlled video distribution system tem with database functions for storing program files, a cloudbased content processing system for high-speed video processing, a browsing and viewing service able to switch between program files quickly, and a play-out and delivery system for TV program files. We are studying the basic and functional requirements of this system and are designing and developing it. The wide-area distributed file system has no centralized management mechanism and works as a single virtual storage system with scalability. In this system, broadcasting stations can store and share a continually increasing number of files. In FY2010, we conducted a study of a multi-record structured file-management method that attaches information such as indexing data for video, audio, and text to files. We also designed an interface for accessing multiple files distributed over thesystemasiftheywereasinglefile. In the cloud-based content processing system, servers on the network process files in parallel and more servers can be used to speed-up processing. In FY2010, we studied ways of allocating processes dynamically according to CPU and network resource usage. These methods were implemented in the file transmission protocol developed earlier. We are also studying a distributed server-based browsing and viewing service that can quickly switch between high-volume Streaming server Server-function extension 4.2 Video production technology Congestion point Web browser programs. In FY2010, we devised a method of distributed cache placement and developed simulation tools for evaluating it. In other research, we cooperated with the Engineering Administration department to study applications of the flexible program production system developed in FY2009 for the next version of the tapeless system. We also prepared to release the source code of this program production system on the STRL website to promote its development and use. 1 S. Oda, K. Kamimura, K. Aoki and M. Yamamoto : Performance Evaluation of HDTV IP Transmission System Adapting to Available Network Bandwidth, IEEE Broadcast Symposium, DTV Transmission I 2010 We have begun research on technology that obtains environmental metadata, such as depth information, camera poses, lighting, and so on about the real space in which a video is shot to simplify operations and computations involved in composing and processing video and increase accuracy of them. Our method to obtain environment metadata uses an optical system with a polarized split aperture Figure 1 A camera embodying it can capture images and at the same time obtain information about the lighting and the object, including its distance from the camera and surface reflectance information. We confirmed the feasibility of these concepts and prototyped a system with high-depth-resolution that mounts a laser range finder on a precision-controlled camera platform. We are also developing technology to estimate camera parameters in real time using 3D models of objects in real space, in order to obtain the camera parameters without using data from sensors on the camera platform. As part of this effort, we devised a method that detects small camera movements from the amount of blur in the image and used it to control jitter in the parameter estimation results 1 We are researching video synthesis and image processing techniques using environmental metadata to make it easier for Focal point Object location Polarized-split aperture Image plane a Object in front of focal point Parallax created by blurring b Object behind focal point Select each polarization component using a polarizing prism Used to separate mirrorreflected components of each polarized component Figure 1. Principle of depth estimation using an optical system with polarized split aperture Original image Depth image Object removed and filled-in Figure 2. Removal and filling-in of an object pink object at the bottom of the original by using depth information original image from the Middlebury Stereo Datasets NHK STRL ANNUAL REPORT

32 4 Advanced contents production technology 4.3 High-quality speech synthesis 4.4 Acoustic devices ordinary users to perform video synthesis and processing. One such technique removes an image of a specified object in a selected region of the video frames and fills it in with an appropriate background. We built algorithms for extracting and filling-in regions and confirmed that using depth information and color information together give better results than using only color information 2 Figure 2 We are incorporating the results of the above studies into a framework for image composition and processing that manages environmental metadata and can be used by various video processing modules. 1 H. Park, H. Mitsumine and M. Fujii: Jitter Suppression in Model-Based Camera Tracking, Proceeding of the 16th International Conference on Virtual Systems and Multimedia, pp H. Okubo and M. Fujii: Image Completion with Supplemental Depth Information, Proceeding of the 255th Conference of the Institute of Image Electronics Engineers of Japan, No. 255, , pp in Japanese 4.3 High-quality speech synthesis We are researching high-quality speech synthesis technology that can read out manuscripts for broadcast and automatically read out data broadcasts for visually impaired persons. In FY2009, we developed a high-quality text-to-speech synthesis method using a large amount of recorded announcer s speech in news programs 1 and built equipment to automatically read out Radio 2 s Kabushiki Shikyo State of the Markets broadcast 2 In FY2010, we developed equipment to read out weather reports, which contain a wider range of expressions than exist in stock market reports. To make this possible, we developed a high-quality speech synthesis method that is based on classifications of text patterns representing sentence structure and produces speech indistinguishable from a natural voice 3 We conducted subjective evaluation tests comparing the new method with previously developed text-to-speech synthesis methods ; the new method produced a more natural sounding speech in 66% of the cases. We combined this method with speech-rate conversion technology, prototyping automatic speech synthesis system for weather reports able to read out the weather report within the allotted broadcast period Figure 1 To enable replacement of segments for which no speech data is available with phonemes from another speaker in text-tospeech synthesis by concatenating waveforms, we conducted tests on the relationship between acoustical characteristics and naturalness of the sound and statistically classifying speakerdependent tendencies. To be able to synthesize a wide range of expressions, we began developing a way of complementing a small amount of speech data on one speaker with another speaker s data stored in a speech database. We consolidated our methods of speech synthesis based on statistical techniques and studied ways to complement a small new-speaker speech database. We devised a method that concatenates speech segments around missing segments by deriving mappings from data in an existing speech database for a different speaker. We also developed speech-rate conversion methods to control speech rates according to changes in pitch or sound pressure level of a voice, Figure 1. Screen-shot of software for automatically reading-out weather reports for reading or language study purposes involving high-speed playback of recorded content. 1 H. Segi, R. Tako, N. Seiyama and T. Takagi: Concatenative Speech Synthesis System Using Recordings of Japanese Broadcast News Programs as a Speech Database, IPSJ Journal, Vol. 50, No. 2, pp in Japanese 2 H. Segi, N. Seiyama, R. Tako, T. Takagi, S. Oode, A. Imai, M. Nishiwaki and R. Koyama: Developing Speech Synthesis System for Stock-price Bulletins and Trial Use in Digital Terrestrial Radio, ITE Journal, Vol. 62, No. 1, pp in Japanese 3 H. Segi, R. Tako, N. Seiyama, T. Takagi, H. Saito and S. Ozawa : Sentence-Generating System for Speech Synthesis Using Templates and Application for the Weather Report Radio Program, ITE Journal, Vol. 65, No. 1, pp in Japanese 4.4 Acoustic devices Excellent acoustic devices are essential for the production of high-quality audio content. We are conducting research on loudspeakers that utilize new principles and compact highperformance microphones for the next generation. Lightweight loudspeakers using polymer film We are conducting research on compact and versatile, lightweight loudspeakers that would be ideal for installing the Super Hi-Vision 22.2 multichannel sound system in private homes. In FY2010, we devised a push-pull structure using two electroactive elastomer rubber-like films that expand and contract in opposite directions, which produces higher efficiency and sound quality Figure 1 The loudspeakers are round, with a diameter of 16 cm, and weigh only 60 g approximately 1/20th the weight of conventional loudspeakers of the same size The push-pull structure yields a frequency response from 80 Hz to 15 khz Figure 2 We analyzed the structure, focusing on the excluded volume of air, and built prototype loudspeakers. We theoretically and experimentally showed that the best efficiency and biggest improvement in sound quality is obtained when the ratio of diameters of Figure 1. Push pull loudspeaker using electroactive elastomer 30 NHK STRL ANNUAL REPORT 2010

33 4 Advanced contents production technology 4.4 Acoustic devices 4.5 Contribution transmission technology the electrodes and diaphragm is 2:1. We also prototyped and measured rectangular, triangular and elliptic loudspeakers, and found that the acoustic characteristics are roughly proportional to the surface area and do not change significantly with different shapes. This means that the shape of the loudspeakers can be designed somewhat freely. This research was conducted in cooperation with Foster Electric Co. Ltd. Highly directional microphones We are conducting research on highly directional microphones with significantly reduced rear response, for recording clear sound in noisy environments. In FY 2010, we used the standard 30 cm short 15 cm and long 55 cm microphones prototyped in FY 2009 and earlier in many programs mainly sports and verified their performance and operability. In addition, we began studying ways to increase their directionality. Acoustic pressure level db k 2k 4k 6k 8k 10k 20k Frequency Hz Figure 2. Amplitude frequency response Silicon microphones Diaphragm single-crystal silicon Sound Backplate single-crystal silicon Dielectric stores charge Insulator Figure 3. Silicon electret condenser microphone structure Detector circuit Sound signal We are developing silicon electret condenser microphones that have the potential for being highly portable and reliable broadcast microphones. These devices store an electric charge internally, which eliminates the need for the 48 V bias voltage required to operate earlier microphones. We previously developed a new technology for storing charge in the dielectric formed inside the device. The technology works by exposing the microphone to an X-ray source. In FY2010, we prototyped silicon electret condenser microphones and improved the efficiency of the charge-storage technology. The prototype silicon electret condenser microphones use single-crystal silicon for the diaphragm and an inorganic dielectric for the facing backplate Figure 3 As a result of accumulating electric charge in the dielectric through corona discharge, the microphone could be operated without a bias voltage. We confirmed that the microphone has sufficient sensitivity -45 db and a nearly flat frequency response within the audio frequency range. On the other hand, the rate of charging by X-rays is not sufficient, and the time required to accumulate enough charge to operate the microphone is too long. To increase the charge accumulation efficiency, we developed a method involving shortening the wavelength of the X-rays used and injecting gas with a higher X-ray absorption rate into the microphone during charging 1 These measures dramatically increased the charging rate, and we concluded that this charge-storage technology can be put to practical use. This research was conducted in cooperation with RION Co., Ltd. and Kobayasi Institute of Physical Research. 1 K. Hagiwara, M. Goto, Y. Iguchi, T. Tajima, Y. Yasuno, H. Kodama, K. Kidokoro and Y. Suzuki: Effect of High-absorbance Gas Introduction on Soft-X-ray Charging Properties for Electret Generator, Proc. PowerMEMS2010, pp Contribution transmission technology Reliable microwave-band transmission technology To increase capacity and expand the transmission area for mobile coverage of events such as road-races, we have developed a novel field pick-up unit FPU receiver that can input up to 16 branches of signals received from base stations positioned around the course Figure 1 The receiver can correct delaytime differences among branches, select the best four branches, and combine and demodulate them, even if the lengths of optical fibers transmitting signals from the base stations differ by up to 160 km equivalent difference Reliable connections over a wide transmission area can be made by exploiting the macrodiversity effect. Also, for the multiple-input multiple-output MIMO transmission mode, we implemented a sophisticated branch-selection algorithm to increase transmission capacity and reliability, by taking into consideration not only the reception power, but also correlations among channels. This receiver was used for broadcast coverage of the Biwako Mainichi marathon 66th Lake Biwa Mainichi Marathon We also created a hardware prototype of the low density parity check LDPC - minimum mean square error MMSE - softinterference cancellation SIC channel coding system that we have been studying. We confirmed that the iterative decoding in the turbo-equalizer could process signals in real-time 10 ms or less 1 We conducted tests using a fading simulator to evaluate its transmission performance. The tests showed that iterative decoding yielded an improvement of 3.7 db or more in the required C/N in environments with correlated flat fading and that the reliability of MIMO channels can be greatly improved. In other research on microwave-band mobile transmission systems, we obtained reception data from the Kyoto Ekiden race course and analyzed the propagation paths. We found that the propagation path of signals from a transmitter mounted on a motorcycle can be categorized into one of three characteristic types. NHK STRL ANNUAL REPORT

34 4 Advanced contents production technology 4.5 Contribution transmission technology 4.6 Millimeter-wave mobile camera We began basic studies on a bidirectional FPU that will increase transmission speed by feeding back the reception state. This fiscal year, we developed basic test equipment for studying time division duplex TDD methods capable of bidirec- Figure 1. Prototype 16-input basestation MIMO receiver tional transmission without increasing the frequency bandwidth. Single-carrier frequency-domain equalization We are studying a single-carrier method to perform highspeed channel estimation and equalization in the frequency domain for a millimeter-wave-band FPU. In FY2010, we studied algorithms for block synchronization and automatic frequency control AFC and combined them with the algorithms developed in FY2009 for channel estimation, noise power measurement, and maximal ratio combining in the frequency domain. We created a computer simulation model of a hardware implementation and verified the algorithms. The results indicated good operations even in multipath environments, so we have begun work on building a hardware modulator/demodulator. 800-MHz-band transmitter antenna for road-races We conducted a study on a transmitter antenna for mobile transmission vehicles to increase the stability of road-race video transmissions using 800-MHz-band FPUs and reduce the number of receiver base stations 2 We made the transmitter array antenna to create a narrow-beam configuration suited to the course and receiver base station placement. We used the array antennas in the Zenkoku Koukou Ekiden 61 st All Japan High School Ekiden and Biwako Mainichi marathon. The video transmissions were stable and the number of receiver base stations could be reduced from what had been necessary to set up previously. Mesh reflector antenna for satellite news gathering We are developing a 14-GHz-band satellite news gathering SNG mesh reflector antenna, to transmit the initial contribution video in emergency reports. We attempted to reduce the radiation-pattern side lobes with modifications such as increasing the radial ribs of the mesh reflector from 24 to 32 as well as the ratio of the focal length to the diameter from 0.5 to 0.6. We plan to conduct video transmission tests on the antenna in the field. Reliable millimeter-wave-band wireless transmission technology We are developing an FPU using 120-GHz-band signals for transmitting uncompressed Super Hi-Vision signals 3 In FY 2010, we studied the requirements and the bit-rate and other parameters, also considering conditions for frequency sharing between other systems, in order for the 120-GHz-band, which is capable of high-capacity transmission, to be used by broadcasters in the future. We also conducted transmission experiments over 120-GHz-band wireless links using cross-polarization, and we participated in technical examination services conducted by the Ministry of Internal Affairs and Communications Study group on spectrum-sharing technologies and other issues for next generation broadcasting systems. The study group is examining various issues including interference in the 120-GHzband on radio astronomy and satellite earth exploration and the limits of measurement equipment for the 120-GHz-band radio station. We actively have made our research results available to the group and are making proposals for technical standards. In other research, we worked on high-speed modulation and demodulation using digital signal processing techniques. This research aims at a practical millimeter-wave-band FPU for transmission of uncompressed HDTV signals. In FY 2010, we developed a field programmable gate array FPGA program of a quadrature phase shift keying QPSK demodulator circuit. The unit demodulates a modulated signal with a center frequency of 750 MHz and bandwidth of 750 MHz. The digital demodulator circuit directly converts the modulated signal, which is input from an A/D converter, into baseband inphase and quadrature phase signals. The modulator and demodulator are able to transmit uncompressed HDTV signals with a bit-rate of 1.5 Gbps. We also studied error correction codes, in particular, the prospect of using LDPC on signals to be transmitted at 3 Gbps after encoding. We implemented the LDPC code in FPGAs on our prototype signal processing boards. 1 K. Mitsuyama, K. Kambara, T. Nakagawa, T. Ikeda and T. Ohtsuki : Prototyping and Performance Evaluation of LDPC-MMSE-SIC System with Time Interleaving, IEICE Technical Report, Vol. 110, No. 251, RCS , pp in Japanese 2 N. Kogo, S. Tanaka, F. Murakami and M. Suzuki: Radiation Patterns of Antennas on OB Van for Road Race in 800-MHz Band, ITE Technical Report, Vol. 35, No. 5, BCT , pp in Japanese 3 S. Okabe, T. Ikeda, F. Suginoshita and S. Shogen: Transmission Characteristics of 10 Gbps Class Forward Error Correction System Using Convolutional Code, ITE Annual Convention, in Japanese 4.6 Millimeter-wave mobile camera We are researching HDTV quality millimeter-wave mobile cameras with low delay. In FY2010, we augmented the returnlink radio transmission functions, such as the return image and HDTV camera control, to give the cameras the same level of operability as conventional cabled HDTV cameras. The return-link transmission system has to be as resistant to interruption as the main-link transmission, have a compact return-link demodulator in the camera, and support precise control of the camera through full-duplex communication. We developed return-link equipment implementing a space-time block coding STBC method with four transmitters that reduces required C/N while requiring few computations for demodulation which helps to make the demodulator board equipped by camera operator compact 1 We also developed multiplexer/ demultiplexer equipment for the control signals linking the main and return systems and for implementing bi-directional communication for controlling the camera and other signals. We designed the 42-GHz-band high-frequency transceiver for the main-link and return-link systems and prototyped bidirectional 42-GHz-band high-frequency transceiver 2 We used a shared local oscillator for transmission and reception in order to make the bi-directional high-frequency transceiver 32 NHK STRL ANNUAL REPORT 2010

35 4 Advanced contents production technology 4.6 Millimeter-wave mobile camera 4.7 Millimeter-wave TV camera compact. We also prototyped an optical interface, to allow IF signals to easily be transmitted between the high-frequency transceiver in the base station and the video control room on existing optical trunk lines. Till now, a dedicated receiver was needed for each transmitted channel, but we have simplified the receiver system with an architecture that uses a common front-end receiver module for multiple channels. We prototyped a 42-GHz-band, horizontally polarized, omnidirectional antenna required for dual-polarized MIMO transmissions in outdoor line-of-sight environments with strongly correlated radio paths. We confirmed that the antenna suppresses channel correlations and improves transmission characteristics. The millimeter-wave mobile camera was used during coverage of the Japan Golf Tour Championship, the Aomori Nebuta Festival, the Japan Open Golf Championship, the Japan Women s Open Golf Championship, Saturday Studio Park, and the 61st Kohaku Utagassen year-end song festival program Figure 1 In particular, we used two cameras equipped with a 42 GHz return-link transmission system and horizontally polarized omni-directional antennas in approximately 50 shoots during the program, showing the effectiveness of this STRL technology in actual programming. Figure 1. Millimeter-wave mobile camera in use 1 S. Suzuki, T. Nakagawa and T. Ikeda: Development of Return Transmission System using 42GHz-band for Millimeter-wave Mobile Camera, ITE Technical Report, Vol. 35, No. 10, BCT , pp in Japanese 2 T. Nakagawa, S. Suzuki and T. Ikeda: 42 GHz Band Transceiver and IF Optical Transmission Equipment for Millimeter-wave Mobile Camera, ITE Technical Report, Vol. 35, No. 10, BCT , pp in Japanese 4.7 Millimeter-wave TV camera We are continuing with our research on a millimeter-wave TV camera that obtains an image by transmitting millimeter waves toward the scene and receiving the reflected waves. The millimeter-wave camera can take images of objects that are obscured by optically opaque substances, such as thick fog and smoke, since millimeter-waves penetrate these substances. In FY2010, we evaluated a second-generation prototype that has an electronic scanning reflectarray antenna. The experiments showed the unit to have a frame frequency of 2.3 Hz 1 2 which is 23-times faster than the first-generation prototype. The camera should thus be able to follow slow-moving objects such as a person walking. However, noise appeared in the background of the captured images, blurring the outline of the subject. This was because a small horn antenna was used to transmit the millimeter-wave signals over the entire field of view, so not only does the main beam of the reflectarray antenna receive the reflected waves from the intended direction but also the associated side lobes receive the reflected waves from undesired directions. To alleviate this problem, we made it so that the transmitter and receiver share the reflectarray antenna. Sharing a large reflectarray antenna reduces the signals transmitted in the undesired directions and reduces the undesired reflected waves received by the side lobes. On the other hand, it also degrades the isolation between the transmitter and receiver; i.e., the transmitted signals can leak into the receiver. Hence, we employed a simple receiver structure using a single-ended mixer with a modified waveform of the transmitting signal so that the leakage signal can be exploited as a reference signal 3 As a result, when imaging a plastic mannequin at a distance of 2 m, background noise was reduced by approximately 10 db. This made the outline of the mannequin clearer and improved the image quality Figure 1 Left Image before improvements Right after improvements Figure 1. Image improvement as a result of transmitter and receiver sharing reflectarray antenna 1 H. Kamoda: Millimeter-wave TV Camera: Millimeter-wave Active Imaging System, NHK STRL R&D, No. 122, pp in Japanese 2 J. Tsumochi, H. Kamoda, T. Kuki and F. Suginoshita: Improvement of Frame Frequency for Millimeter-wave TV Camera, IEICE Society Conference, CS in Japanese 3 J. Tsumochi, H. Kamoda and F. Suginoshita: A Study on Monostatic Radar for Millimeter Wave Imaging System, IEICE Technical Report, Vol. 110, No. 447, MW , pp in Japanese NHK STRL ANNUAL REPORT

36 5 Devices and materials for broadcasting We are conducting research on image capture, recording, and display devices for next-generation broadcast services such as Super Hi-Vision SHV We are developing image sensors that have high sensitivity, high frame rates, and high picture quality. In particular, we continued with our development of high-sensitivity field emitter array image sensors with HARP films and found ways to make these sensors more compact and sensitive. We are also developing an SHV image sensor with a 120 Hz frame frequency and have completed the design and fabrication of a high-frame-rate test device for investigating its feasibility. We also completed the design of a back-side-illuminated ultra-high-speed CCD and launched a study of image sensors with a three-dimensional structure that reads out the signal in the depth direction. In our research on high-picture-quality single-chip color image sensors, we fabricated a full-color organic image sensor with a large number of pixels. Our research on magnetic and optical recording equipment is advancing. Related to magnetic storage, we fabricated data transfer rate smoothing equipment consisting of two hard-disk drives. We also developed technologies for magnetic recording devices consisting of magnetic nanowires to create nano-sized domains. Regarding our work on optical storage, we continued with our development of thin optical disk and holographic recording technologies for SHV. We fabricated a thin optical disk drive system and worked on near-field optical recording technology for increasing the data capacity of the disk. We also built experimental holographic recording equipment and evaluated its recording and reproduction characteristics. We increased the processing speed for wavefront compensation, which stabilizes the beam wavefront, and increased the angular multiplexing holographic recording density. We continued with our research on flexible displays, high-efficiency plasma displays PDP and spatial light modulators and conducted a survey of advanced technologies of future display devices. Regarding our work on flexible displays, we continued with our research and development of organic light emitting diodes OLED thin-film transistors TFTs and display panel manufacturing technology. We increased both the light emission efficiency of OLEDs and the field-effect mobility of TFTs and prototyped a 5-inch-diagonal color panel. Moreover, we developed a protective layer to reduce the driving voltage of highefficiency PDPs; this new layer can be formed in a high-temperature process. Research is also advancing on an ultra-high-resolution, high-speed spatial light modulator driven by spin transfer switching. This is a promising technology for future spatial image reproduction display systems with a wide viewing zone. 5.1 Advanced imaging systems Super-high-sensitivity image sensors We are developing field emitter array FEA High-gain Avalanche Rushing amorphous Photoconductor HARP image sensors in an effort to make a compact, high-sensitivity HDTV camera for reporting nocturnal incidents such as late-night emergencies. These image sensors are a new type of imaging device consisting of an array of many microscopic field emitters, which emit electrons by simply applying a voltage, facing a highly sensitive HARP photoconductive film, which makes use of an avalanche multiplication effect Figure 1 In FY2010, we developed elemental technology for a more compact FEA image sensor and worked on increasing the multiplication factor of the HARP film. To reduce the size of the FEA imaging sensor, we improved the performance of the electron beam focusing system, which does not require permanent magnet and instead applies a potential to electrodes above the array. The electrostatic focusing system prototyped in FY2009 did not yield a sufficient electron beam and the insulator between the gate and focusing electrodes was exposed to the electron beam, causing a drop in reliability. To resolve this problem, we designed an electrostatic focusing system with a new extraction electrode placed between the gate and focusing electrodes. Simulations of this focusing system showed the focused electron beam was double the intensity of the beam of the previous system and that the insulator is not exposed to the electron beam. The new design should therefore be good for improving the focusing system. In HARP films, the charge multiplication factor can be increased by increasing the voltage applied to the film, but at the same time, the dark current increases, practically limiting increases in the multiplication factor. We attempted to reduce dark current in order to allow the multiplication factor to be increased. Dark current in HARP films is due mostly to holes injected from outside the device. Conventionally, hole injection is suppressed with a layer of cerium oxide CeO2 to form an energy barrier, but this effect is not always adequate. Thus, we analyzed the structure of the CeO2 layer, determined the conditions necessary to improve hole blocking 1 and searched for new materials. We found that replacing CeO2 with zinc oxide or gallium oxide would further reduce dark current and should enable us to at least double the maximum multiplication factor. Development of the electrostatically focused field emitter array was conducted in cooperation with Futaba Corporation. 34 NHK STRL ANNUAL REPORT 2010

37 5 Devices and materials for broadcasting 5.1 Advanced imaging systems HARP film Field emitter array Pixel Focusing electrode Extraction electrode Gate electrode Emitter FY2010 FY2009 Figure 1. Schematic diagram of image sensor and electrostatic focusing system 1 K.Kikuchi,Y.Ohkawa,K.Miyakawa,T.Matsubara,K.Tanioka,M. Kubota and N. Egami: Hole-blocking mechanism in High-gain Avalanche Rushing amorphous Photoconductor HARP film, Proc. of ICOOPMA 2010, Vol. 64, No. 5, pp High-frame-rate imaging devices In efforts to increase frame rates of imaging devices, we have engaged in research and development on a 120 Hz Super Hi- Vision SHV imaging device, a highly-sensitive, backsideilluminated ultra-high-speed imaging device, and a 3 D- structured imaging device that reads out the signal in the depth direction. High-frame-rate SHV imaging devices We are developing an SHV imaging device with a frame rate of 120 Hz, in order to improve the image quality of SHV motion images. In FY2010, we designed and began prototyping devices based on the results of studies done in FY2009 to verify their operation. To implement the 120 Hz frame frequency, the analog signal output from each line must be converted into a digital signal in 1.9 μs or less, and the digital signal must be read out at a speed of 48 Gbps or greater. For the conversion, we used a compact cyclic column-parallel analog-to-digital converter ADC circuit with high speed and accuracy. Moreover, this twostage structure keeps power consumption low while further increasing speed. For the read-out operation, we increased the speed to over 51 Gbps by making a 32- fold parallelization of the signals read from each line. We designed a test device with approximately 30-megapixels and Figure 1. High-frame-rate SHV imaging device for operation testing completed a prototype Figure 1 This research was conducted in cooperation with Shizuoka University. Ultra-high-speed imaging devices We are developing ultra-high-speed CCDs able to reproduce instantaneous phenomena in ultra-slow-motion video, by capturing images at up to one million frames per second. In FY 2010, we began the layout design and prototyping of a backside-illuminated ultra-high-speed CCD 1 with six times the sensitivity of earlier devices. Earlier ultra-high-speed CCDs prototypes had poor device yield rates, so they used two devices joined together in order to increase the number of pixels. However, this technique is not compatible with the structure of back-illuminated ultra-high-speed CCDs so we used a new large-area-device manufacturing technique combining multiple lithographic exposures to fabricate a single, 300 kilopixels backside-illuminated ultra-high-speed CCD device. This research was conducted in cooperation with Kinki University and DALSA Corp. 3D-structured imaging devices We have begun development of an imaging device with a 3D structure that reads out the signal in the depth direction of the device, with the goal of achieving both high frame rates and high-resolution. In FY2010, we studied device manufacturing technologies and selected issues for study in preparation for starting the main research effort in T. Arai, H. Hayashida, K. Kitamura, J. Yonai, H. Maruyama, T. G. Etoh and Van Kuijk, H.: Simulation based design for backside illuminated ultrahigh-speed CCDs, Proceedings of SPIE, Vol , pp X X NHK STRL ANNUAL REPORT

38 5 Devices and materials for broadcasting 5.1 Advanced imaging systems 5.2 Advanced storage systems Organic image sensors Blue sensitive organic film Green sensitive organic film Red sensitive organic film Transparent TFT circuit Light B G R Figure 1. Conceptual diagram of the organic image sensor B output G output R output We are developing compact, highly sensitive, high resolution organic image sensors. The color image sensor illustrated in Figure 1 is composed of alternating layers of an organic photoconductive film sensitive to blue B green G or red R light, and transparent thinfilm transistor TFT circuit layers which read out the charges generated in the photoconductive films. Our previous prototype had 47 pixels horizontally and 30 pixels vertically with organic photoconductor films sensitive to green and red light. With it, we verified that color imaging with such a layered structure is possible. We have since made progress in increasing the efficiency of organic photoconductive films sensitive to blue light and have increased the integration size of transparent TFT circuits. In FY2010, we incorporated these advances into a fullcolor organic image sensor with approximately pixels 1 The transparent TFT circuits of the latest prototype were formed on three glass substrates. Organic photoconductor layers sensitive to blue, green and red light were formed on the circuits, layered in order of blue, green and red in the direction of incident light. The layers were made of fullerene-doped coumarin 30 blue a double semiconductor layer of a quinacridone derivative / perylene derivative Figure 2. Sample image from prototype layered sensor green and a double semiconductor layer of zinc phthalocyanine/titanyl phthalocyanine red The transparent TFT circuits were made from zinc-oxide, which is transparent to visible light. The circuit had 128 pixels horizontally and 96 pixels vertically, with a pixel pitch of 100 μm. In imaging tests, we obtained full-color motion images from the prototype device Figure 2 verifying for the first time that full-color imaging is possible using stacked layers of organic photoconductive films sensitive to the three primary colors. We also confirmed that the resolution corresponded to the number of pixels and that there was no image lag. Part of this research on forming organic photoconductive layers was conducted in cooperation with Saitama University, and the part on the ZnO TFT circuits was conducted in cooperation with Kochi University of Technology. 1 H. Seo, S. Aihara, T. Watabe, H. Ohtake, T. Sakai, M. Kubota, N. Egami, T. Hiramatsu, T. Matsuda, M. Furuta and T. Hirao: A 128 x 96 Pixel Stack-Type Color Image Sensor : Stack of Individual Blue-, Green-, and Red-Sensitive Organic Photoconductive Films Integrated with a ZnO Thin-Film Transistor Readout Circuit, Jpn. J. Appl. Phys., Vol. 50, No. 2, , pp Advanced storage systems Magnetic recording technology We are working to increase the recording / reproducing speeds of hard disk drives HDDs with the goal of developing compact equipment for recording uncompressed Super Hi- Vision video. We are also developing technologies that use nano-wires to make nano-domain magnetic devices. 2.5 inch hard disk drives Storage for uncompressed Super Hi-Vision video The key to making HDD recording equipment for uncompressed SHV video is increasing the data transfer-rate 1 However, a difficulty with this is that HDDs operate at a fixed rotational speed, so the data transfer-rate is lower for data stored on the inner part of the disk, approximately half that of data stored near the outer edge. We prototyped transfer-ratesmoothing equipment using two HDDs, where by one drive writes on the outer region of its disk while the other drive writes on the inner region Figure 1. This equipment achieved minimum transfer-rates that were 1.5 times higher than with the conventional configuration. This prototype consists of two 2.5 inch serial advanced technology attachment SATA HDDs oper- Figure 1. Transfer-rate-smoothing hard disk Transfer-rate smoothing controller 36 NHK STRL ANNUAL REPORT 2010

39 5 Devices and materials for broadcasting 5.2 Advanced storage systems ating at rpm, and it achieved sustained transfer-rates of 1.8 Gbps or higher. This approach allows a High-Definition Serial Digital Interface HD-SDI signal, which is the base unit of an SHV signal, to be stored and played back from a pair of 2.5 inch HDDs. This provides us with a way to develop compact equipment for storing uncompressed SHV video. Nano-domain magnetic devices We are proceeding with the development of recording devices that have nano-sized magnetic domains on magnetic nanowires, with the goal of realizing high-speed, high-density magnetic recording devices with no moving parts. In FY2010, we prototyped cobalt-palladium nano-wires with lengths from 10 to 20 μm. We confirmed that reducing their widths to 250 nm or less made them capable of holding multiple stable nanodomains, and thus they could be used for making a magnetic recording medium. We also verified that by flowing a direct current along the length of a nano-wire, the multiple magnetic nano-domains in it can be caused to move at the same time, and that the magnetic nano-domains can be stopped at the desired locations by forming local structures that are structurally and magnetically heterogeneous 2 3. These results almost complete our verification of the operating principles of these devices. 1 E. Miyashita, M. Kishida and N. Hayashi: Storage for Ultra-High Definition Video, Digest of the 9th Perpendicular Magnetic Recording Conference, 17aB M. Okuda, Y. Miyamoto, M. Kishida and N. Hayashi: Behavior of Magnetic Domains in [Co/Pd] Nanowires with Scratched Notches by Nanoindentation, Digests of the 34th Annual Conference of Magnetics in Japan, 6aE in Japanese 3 M. Okuda, Y. Miyamoto, M. Kishida and N. Hayashi: Magnetic Observation of Magnetic Domains in [Co/Pd] Nanowires, ITE Winter Annual Convention, in Japanese Optical storage technology We are advancing with our research on thin optical disks and holographic recording technology for future program archives and Super Hi-Vision SHV recording equipment Thin optical disks Thin optical disks with thickness of 0.1 mm are promising candidates for program archives and home SHV recording equipment because they can rotate at higher speeds, thereby giving them higher data transfer-rates. In FY2010, we developed a high-capacity drive system using thin optical disk drives. In parallel with this development, we proceeded with the development of near-field optical recording technology for increasing the recording density of thin optical disks. For high-capacity drive systems, we have developed a thin optical disk drive of approximately the same size as current commercial optical disk drives and equipment that automatically loads thin optical disks from a disk cartridge that can hold 100 thin disks 1 Regarding the drive, we improved the disk loading tray structure by using a mechanism that enables very stable rotation, and this helps to make the drive compact Figure 1 For the disk loading mechanism, we used air suction to transport disks from the cartridge to the drive without causing damage to the disks. In our research on near-field optical recording technology by using a solid immersion objective lens, we prototyped an optical head capable of projecting an optical spot on a disk of approximately half the diameter that current optical disk drive could make. To achieve highcapacity storage and high-speed readout with near-field light, the optical head must be positioned at 50 nm or Figure 1. Prototype compact thin optical disk drive less from the rapidly rotating disk, and this gap must be kept stable. An issue hampering stability is surface flutter synchronized with the disk s rotation. We developed our own head-position control technology that synchronizes the peaks of the opticalhead servo gain with not only the disk s fundamental frequency of rotation but also its harmonics. We confirmed that a headdisk distance of 50 nm or less can be maintained with this technology. Research on the high-capacity drive system was conducted in cooperation with Ricoh Co. Ltd. Research on the focus control technology for near-field optical recording was conducted in cooperation with Nagaoka University of Technology. High-speed holographic recording technology A recording system for SHV must have very high capacity and data transfer-rates. We are researching holographic recording technology using multiplexing as a means of high-density recording and reproducing a two-dimensional data array at once. We have been studying the basic technologies such as angular multiplexing and wavefront compensation and have built test equipment for holographic recording. In FY2010, we built highcapacity and high-speed optical systems for multiplexed recording and reproduction, and we studied wavefront compensation techniques to ensure a low bit-error rate ber and photopoly- Wavefront compensation mirror Reference beam during reproducing Galvanometer mirror Spatial light modulator Camera Reproduced beam phase conjugate Objective beam Objective lens Pseudo disk media Reference beam during recording Galvanometer mirror Figure 2. Holographic recording/reproducing test equipment top view NHK STRL ANNUAL REPORT

40 5 Devices and materials for broadcasting 5.2 Advanced storage systems 5.3 Advanced display systems mer recording media. We made an objective lens with numerical aperture of 0.85, up from 0.65 of the previous systems, and increased the recording density by a factor of four. We also developed a new opticalscanning angular-multiplexing optical system combining galvanometer mirrors and low-aberration relay lenses. This optical system eliminates the need for a mechanism that varies the media slant angle, as was used in the previous equipment, and it greatly reduces the time required to switch the reference beam angle, from 10 ms to 300 μs 2 Regarding the photopolymer recording media, the reproduced images had in the past been degraded by distorted interference fringes caused by volume shrinkage of the media. We are attempting to deal with this problem by using wavefront compensation technology. We have developed a new method that is able to compensate for interference fringe distortions in all pages when angle-multiplexing N pages, by using the compensated wavefront for the middle N/2-th page. Doing so has resulted in an improvement in ber of up to two orders of magnitude. This method reduces the number of times compensation must be done by a factor of 1/N, and hence it leads to faster data reproduction. We integrated these elemental technologies in our new prototype holographic recording and reproducing test equipment Figure 2 We also developed a new photopolymer recording materials with less fluctuation in its photosensitivity, and evaluated its characteristics using pseudo-disk media, made by mounting postage-stamp-sized media on a disk. We multiplexed 340 pages in a 1 mm thick media with a recording density of 180 Gbit/in 2, and achieved a practical ber on the order of 10-4.With batch processing, we successfully recorded and played back 15 seconds of H.264 compressed HDTV signal using 5-9 modulation, page interleaving, and low-density parity-check code. This research on photopolymer materials was conducted in cooperation with Nippon Steel Chemical Co., Ltd. 1 D. Koide, T. Kajiyama, Y. Takano, H. Tokumaru, N. Onagi and K. Tsukahara: Terabytes Archival Storage using Flexible Optical Disks, International Symposium on Optical Memory 2010 Technical Digest, Th-PO-03, pp N. Kinoshita, T. Muroi, N. Ishii, K. Kamijo and N. Shimidzu: Recording Density Improvement in Angle-Multiplexed Holographic Memory, Optics & Photonics Japan 2010, 9pAS2, pp in Japanese 5.3 Advanced display systems Flexible displays We are progressing with our work on light-weight, flexible displays that can be rolled up for transport, the ultimate goals of which are very large sheet displays for Super Hi-Vision and convenient television or information displays for mobile use. In particular, we are conducting research on display panel production technology for flexible plastic substrates, display devices, and thin-film transistors TFTs needed to electrically drive the display panels. Display panel production technology We are continuing to fabricate active-matrix panels dirven by organic TFTs on plastic substrates, with the goal of increasing their screen size, resolution, and image quality. In particular, we are developing panel production technology using solutionprocess and printing technologies. In FY 2010, we developed a new gate-insulator film using olefin-type polymer, which is easy to use in solution processes. This made it possible to fabricate an organic TFT array with a resolution of 80 pixels / inch Figure 1 on a plastic substrate polyethylene naphthalate at temperatures under 150 By using pentacene as organic semiconductor and finely patterning it with polymeric separator structures, we obtained a fieldeffect charge mobility of 0.05 cm 2 /Vs, an on/off current ratio of over 10 6, and good spatial uniformity. By depositing organic Figure 1. Flexible organic TFT array light emitting diodes Figure 2. Flexible OLED display OLEDs in three colors over the TFT array, we prototyped a 5-inch color panel 1 with QVGA resolution 320x240 RGB with four times the resolution of the array fabricated in Utilizing this panel, we built a color motion display with a 60 Hz frame frequency, good uniformity, and brightness of 30 cd/m 2 or higher Figure 2 We are also developing a transfer process for producing large-screen, high-resolution panels by forming organic-tft pixel circuits on a glass substrate and transferring them onto a plastic substrate. We used the new transfer method to make a 5.8-inch-diagonal organic TFT array with a 0.6 mm pixel pitch, and we found that it had essentially the same TFT characteristics before and after the transfer. Research on the development of the transfer process was conducted in corporation with Kyodo Printing Co. Ltd. 38 NHK STRL ANNUAL REPORT 2010

41 5 Devices and materials for broadcasting 5.3 Advanced display systems TFT technology TFTs, which drive the individual pixels, are essential to building flexible display panels. We are researching organic TFTs that have a flexible structure and can be fabricated at low temperature for use on plastic substrates. Till now, we have tried to reduce the operating voltage for organic TFTs with highdielectric-constant gate insulators, and we have increased field-effect mobility and current in organic TFTs by reducing the contact resistance with gate-insulator surface treatments that use a self-assembled mono-layer and by introducing a holeinjection layer. In FY2010, we increased the performance of organic TFTs by using olefin-type polymer as the gate insulator. Olefin does not contain hydroxyl groups or other structures that can trap charge carriers in its surface. This allows the previous issues to be controlled, such as the threshold voltage shift and hysteresis, and improves operational stability. The gate insulating layer can also be formed by using simple solution processes such as spin coating, thereby allowing us to improve TFT reliability and yield. We prototyped a complementary inverter circuit on a plastic substrate using a combination of this polymer insulator and p-type and n-type organic semiconductors and obtained a large inverter gain in ambient air 2 We are working to increase the current-driving performance of organic TFTs. We prototyped an organic TFT with a vertical structure with the source and drain electrodes positioned vertically rather than the conventional horizontally. By exploiting the molecular alignment of the organic semiconductor and having the current flow in the vertical direction, we achieved high current values of 2 A/cm 2 at a driving voltage of 5 V. We have also begun research on an oxide semiconductor In- GaZnO4: IGZO which has high electron mobility that is advantageous for driving high-resolution large screens. IGZO- TFTs can be formed at low temperature, so they are good candidates for driver devices in ultra-large-screen displays. In FY 2010, we optimized the IGZO film thickness and the oxygen partial pressure during sputtering and determined the formation conditions that would yield good transistor characteristics. We used the olefin insulator discussed earlier, which can be easily used in solution processes, to form IGZO-TFTs on plastic substrates that have low heat tolerance. Using a top-gate structure made it possible to limit the damage to the insulator that occurs during the IGZO sputter deposition, and we were able to create a fabricating process that can be conducted at a temperature under 130 For the IGZO-TFT formed on the plastic substrate, we confirmed that the switching behavior was good, with a field effect mobility of 4.4 cm 2 /Vs and on/off ratio of 10 7 or higher. We also began working on creating IGZO patterning methods. We fabricated a QVGA-resolution IGZO-TFT array on a 5-inch substrate, and achieved good spatial uniformity in operating characteristics. We then built a flexible panel by forming OLED layers for the three primary colors over the array. Display devices We are researching OLED and liquid crystal LC film devices for use as flexible displays. Regarding OLED devices, we are researching light-emitting materials and carrier transport materials to increase the emission efficiency of the displays. As for the emitting materials, there are fluorescent materials that have long lifetimes and phosphorescent materials that provide high efficiency, so the issue is to attain both a long life and efficiency at the same time. Most fluorescent materials lose 75% of their energy as nonradiative thermal energy. In FY2010, we designed a new fluorescent material with a molecular structure that is able to use more of its energy as light emissions. We tested the new material in prototype devices and obtained higher external quantum efficiencies that exceeded the earlier theoretical limits. For phosphorescent devices, it is relatively difficult to increase the efficiency of blue emissions. To deal with this problem and to extend the lifetime, we optimized the design of the charge transport material 3. On the other hand, phosphorescent polymer materials are advantageous for printing methods to make large screens. We quantitatively verified that the phosphorescent units can be a trap material by measuring the charge mobility. This provided us with useful information for designing new materials. By optimizing the solution process conditions of ink-jet printing with these phosphorescent polymers we established new technology for forming pixels with four times the resolution of the process developed in Part of this research on phosphorescent polymers was conducted in cooperation with Showa Denko K. K. We have also been developing LC film devices fabrication methods using printing processes and UV-curable polymers, and a flexible backlight using light-emitting-diodes and light guide plate. We built color video displays by driving the LC panels with polycrystalline silicon TFTs or organic TFTs. We believe that device structures should be developed so that large-screen display can easily be fabricated and have excellent tolerance to bending. In FY2010, we made detailed observations of the process of forming polymer walls that keep the substrate gap constant when fabricating devices using low-viscosity nematic LC, which has high tolerance to bending. This observation clarified that the polymer wall formation depends on the LC s molecular alignment. We also fabricated a flexible LC device structure by forming spacer walls and grooves on plastic substrates by using a roller nano-imprint technique that scales well to large areas and by bonding the interdigitated spacers using thermally curable resin. 1 Y. Nakajima, Y. Fujisaki, T. Takei, H. Sato, M. Nakata, M. Suzuki, H. Fukagawa, G. Motomura, T. Shimizu, K. Sugitani, Y. Isogai, T. Katoh, T. Yamamoto, H. Fujikake and S. Tokito: Characteristics of OTFTs using Olefin-polymer Gate Insulator and their Application to a 5-in. OTFT-driven Flexible AMOLED Display, Materials Research Society Symposium 2010, F Y. Fujisaki, Y. Nakajima, D. Kumaki, T. Yamamoto, S. Tokito, T. Kono, J. Nishida and Y. Yamashita: Air-stable n-type organic thin-film transistor array and high gain complementary inverter on flexible substrate, Applied Physics Letters, Vol. 97, No. 13, pp H. Fukagawa, N. Yokoyama, S. Irisa and S. Tokito: Pyridoindole Derivative as Electron Transporting Host Material for Efficient Deep-blue Phosphorescent Organic Light-emitting Diodes, Advanced Materials, Vol. 22, No. 42, pp NHK STRL ANNUAL REPORT

42 5 Devices and materials for broadcasting 5.3 Advanced display systems Next-generation display materials Reducing PDP power consumption We have continued with our research on new electrode materials to reduce the driving voltage and energy consumption of Super Hi-Vision plasma display panels PDP for home use. We developed electrodes that are highly stable chemically, and for which the driving voltage does not rise even under hightemperature atmospheric production conditions. We evaluated the chemical stability of electrodes made from calcium magnesium oxide CaxMg1-xO with respect to the proportion of Ca x and found that the electrodes become much more stable in a high-temperature atmospheric process at 450 when x is between 0.25 and 0.4. Panels with electrodes having Ca in this range can operate at voltages 20% to 30% lower than those of panels using conventional MgO electrodes 1 Figure 1 This will make possible significant voltage reductions relative to earlier panels and greater energy savings. Optical-spin devices Figure 1. Image displayed on a low-power panel using Ca0.36Mg0.64O electrodes The key to realization of holographic imaging with a wide viewing-zone angle is the development of a high-resolution, high-speed, spatial light modulator SLM as a display device. We are researching a spin transfer switching STS SLM with pixel pitches of 1 μm or less which able to produce large optical diffraction angles for viewing-zone angles up to 30 degrees. The direction of magnetization spin of the material comprising a pixel can be controlled by varying the direction of the injected spin current spin transfer switching and light can be modulated using the magneto-optic Kerr rotation, by which the polarization plane of incident light is rotated by an angle that depends on the direction of magnetization. Earlier, we developed spin-injection magnetic materials and built basic technologies such as single devices to test the operation of the magnetic switching effect 2 In FY2010, we demonstrated optical modulation by spin transfer switching, developed a process for fabricating device arrays, and began searching for magnetic materials to reduce the driving current and increase the MO Kerr rotation angle. For devices smaller than 1 μm, the wavelength of the modulated light is close to the size of the device. We built a highmagnification 3000 MO microscope to observe MO effect with nano-magnets. We observed, for the first time, monotone images produced by light modulation on a 500 nm 500 nm device, by using spin-injection current through transparent electrodes Figure 2 The device had a gadolinium-iron alloy free layer with a MO Kerr rotation angle of 0.12 To make the array, we used a damascene process to fabricate a flat, embedded, bottom-electrode array 620 nm width, 1 μm pitch of copper on a silicon substrate and formed microscopic pixel patterns with widths of 120 nm ±50 nm on the bottom electrodes by using electron-beam lithography. We used this process to make a 10-pixel 1 10 SLM prototype. For the magnetic material, we reduced the magnetic switching current to a quarter of previous levels by increasing the iron composition of the gadolinium-iron alloy in the free layer. Also, by creating a free layer with multiple layers of tantalum and cobalt-platinum, which has a large Kerr rotation angle, we achieved a MO Kerr rotation angle of 0.3 Part of this research was conducted under contract with the National Institute of Information and Communications Technology NICT. Transparent electrode or Free layer Spacer layer Pinned layer Bottom electrode Cu a Element structure cross section Free layer Pinned layer Off state before magnetic reversal On state after magnetic reversal Example observation of magneto-optical effect on the nano-scale device size: 500 nm 500 nm) b Light modulation by the device Figure 2. Observation of light modulation by prototype STS light modulator Increasing the performance of CRT viewfinders CRT viewfinders CRT-VF with excellent video display performance are used in handheld broadcast-quality HDTV cameras, but their resolution is at present inadequate. To increase their resolution, we are attempting to replace the conventional phosphor powders used in them with phosphor thin films. The particles in phosphor thin films are better for high-resolution displays because they are smaller than fine powders, but they are not as bright. Thus, in FY2010, we attempted to increase their brightness by using high-temperature growth to improve their crystallinity. Another issue is that the glass used for current CRT-VFs is suitable for maintaining a high vacuum, but it deforms at high temperature, and this in turn distorts the display image. Thus, we developed a method in which phosphor film is grown at high temperature on a different, heat-resistant glass, and it is adhered to another glass in a vacuum. As a result, we were able to increase the brightness of the green phosphor film by a factor of about two 170 cd/m 2 at 15μA 3 1 Y.Motoyama,D.KatoandM.Seki: AnACPDPwithanAir-Stable CaMgO Protective Layer, IDW 10 Technical Digest, PDP5-2L, pp K. Aoshima, N. Funabashi, K. Machida, Y. Miyamoto, K. Kuga, T. Ishibashi, N. Shimidzu and F. Sato: Submicron Magneto-Optical Light Modulation Device for Holographic Displays Driven by Spin-Polarized Electrons, IEEE Journal of Display Technology, Vol. 6, No. 9, pp K.Tanaka,T.Sakai,S.OkamotoandY.Shimizu: Highresolution compact CRT using phosphor thin-films for HDTV viewfinder, ITE Annual Convention, in Japanese 40 NHK STRL ANNUAL REPORT 2010

43 6 Research-related works This year was the 80th anniversary of the opening of NHK STRL. The STRL Open House held in May was on a larger scale than in previous years, with 22,087 visitors having the opportunity to view 44 exhibits and attend presentations by speakers from overseas. An event commemorating the anniversary was also held jointly with Setagaya Ward in October. This year, we exhibited or made presentations about our research results a total of 40 times within Japan and overseas. In particular, we exhibited at IBC2010 in Amsterdam our latest SHV projectors, full-resolution SHV cameras, and user-friendly broadcast technology. We also gave 93 tours a total of 1399 visitors of our laboratories. Of these, 40 were tours for visitors totaling 411 from outside Japan. We are also actively publishing STRL research results in domestic and international conference proceedings and journals. A total of 652 papers have been published. We are also managing rights to protect our research results. In FY2010, NHK as a whole submitted 323 patent applications and obtained 278 patents. As of the end of FY2010, NHK will hold 1571 patents. STRL is actively participating with standardization organizations including the ITU, ABU, Information and Communications Council, and the Association of Radio Industries and Businesses ARIB The 2010 general meeting of the ABU was held in Tokyo and hosted by NHK. In addition to contributing papers, we held tours of STRL. We are actively collaborating in the international arena to promote the Japanese digital broadcasting format ISDB-T. We have dispatched staff to overseas broadcasters and conducted various international activities. As of the end of March 2011, 11 countries besides Japan will have adopted the format. We are actively pursuing cooperation with outside organizations. We participated in 46 instances of cooperative research and 8 instances of commissioned research. We hosted 25 visiting researchers including 23 trainees and dispatched two of our researchers overseas. The fruits of our latest research have been used in production of NHK and other broadcasters programs. These include the millimeter-wave mobile camera, P 2 P live distribution technology, and video content-search technology. In FY2010, we collaborated in the production of 82 programs. In recognition of its research excellence, STRL received a total of 25 awards in FY2010, including the Hoso Bunka Foundation awards and the Meritorious Award on Radio. 6.1 Joint activities with other organizations Participation in standardization organizations NHK STRL is participating in standardization organizations within and outside of Japan by contributing its research results. The ITU-R is tasked with standardization related to radiocommunication broadcasting. As part of the WP6A and WP6B working parties of study group 6 SG6, we contributed to a transmission framework for mobile multimedia digital terrestrial broadcasts and encapsulation of IP packets within MPEG-2 TS packets. Furthermore, we contributed to WP6C activities on standards related to a wide-gamut system colorimetry for UHDTV. ITU-T is tasked with standardization of television and sound transmission and integrated broadband cable networks, and we contributed to SG9 of ITU-T on standards related to linking devices to other household networks. MPEG has begun working together with ITU-T on standardization of High Efficiency Video Coding HEVC the next generation of video coding. Related to this effort, we provided a test model and Super Hi-Vision SHV test video, evaluated encoding tools, and proposed an adaptive-selection technology for orthogonal transformations. We also made a proposal to MPEG Media Transport MMT, which is studying new transport formats, on a multiplexing scheme for implementing hybrid broadcast/communications transmission. Related to audio, we made contributions on home reproduction of 22.2 multichannel audio and descriptor formats for media-analysis results to be used for automatic metadata generation in MPEG-7. At SMPTE, the UHDTV interface standard that we have been working on was published in August as ST We also held a discussion on defining metadata for multichannel audio in Material Exchange Format MXF, which is used for exchanging media. We are especially pleased that Senior Research Engineer Kohji Mitani currently with the Engineering Administration Department was selected as a SMPTE Fellow and awarded the SMPTE David Sarnoff Medal Award for achievements related to development of HDTV and UHDTV systems. The FY 2010 annual general meeting of the Asia-Pacific Broadcasting Union ABU was held in Tokyo. In technical committee meetings held before the general meeting, NHK submitted 23 papers on topics such as production, transmission, and frequencies. We also held tours of STRL for committee members participating in the general meeting and promoted STRL technologies such as Hybridcast and SHV to the delegates of the meeting. In other ABU work, we promoted our emergency warning broadcasting system throughout the ABU region by participating in a six-country backpack road show and by conducting presentations and exhibits at the Digital Broadcasting Symposium 2010 held in Kuala Lumpur in March. Besides the above activities, we participated in standardization activities and promoted our technologies at the European NHK STRL ANNUAL REPORT

44 6 Research-related works 6.1 Joint activities with other organizations Broadcasting Union EBU, International Electrotechnical Commission IEC, Audio Engineering Society AES Japan Electronics and Information Technology Industries Association JEITA, and Telecommunication Technology Committee TTC Leadership for activities at major standardization organizations International Telecommunication Union ITU Committee name International Telecommunication Union, Radiocommunication Sector ITU-R Study Group 6 SG6-Broadcasting service Working Party 6B WP6B - Broadcast service assembly and access Asia-Pacific Broadcasting Union ABU Committee name Technical committee Transmission topic Audio Engineering Society AES Committee name Officer committee International region section Technical committee Transmission and broadcasting Audio recoding and mastering systems Information and Communications Council Committee name Information and communications technology subcommittee ITU-R section Work planning committee Spectrummanagement committee Radio-wave propagation committee Satellite services committee Broadcast services committee Scientific services committee ITU-T section Cable network, program transmission committee Next-generation networks committee IPTV special committee Low-power radio systems committee ITS radio systems committee Industry/academia/government collaboration promotion committee Information and communications policy section Study committee for promoting circulation of digital content Study committee for standardization policy for environments integrating and linking communications and broadcasting Leadership role Vice-chairman Chairman Leadership role Chairman Chairman Leadership role Vice president Vice chairman Chairman Chairman Leadership role Expert member Expert member Expert member Expert member Expert member Expert member Expert member Expert member Expert member Expert member Expert member Expert member Expert member Expert member Association of Radio Industries and Businesses ARIB Committee name Leadership role Technical committee Broadcasting international standardization working group Chairman Quality assessment methods investigative research group Committee chairman Monitoring evaluation methods working group Manager Flat-panel display image quality evaluation working group Manager Newbroadcast technology investigative research group 3D television working group Manager Digital broadcast systems development section Committee chairman SBTVD-T standardization task force Leader Identifier assignment ad-hoc group Leader Multiplexing working group Manager Download methods TG Leader Video coding methods working group Manager Data coding methods working group Manager Advanced data imaging H.264 TG Leader Broadcast service application metadata working group Manager Digital broadcasting systembased on homeserverworking group Manager Digital receivers working group Emergency broadcast transmission TG Leader AC security AHG Leader Digital satellite broadcasting advancement working group Manager Digital terrestrial broadcasting transmission path coding working group Manager Contribution transmission development section Committee chairman Terrestrial wireless contribution transmission working group Manager Broadcaster communication radio frequency TG Leader Millimeter-wave contribution transmission TG Leader Ultra-high-definition TV studio facilities development section Committee chairman Acoustic system study working group Manager Device interface study working group Manager Promotion strategy committee Digital broadcasting international promotion sub-committee Digital broadcasting experts group DiBEG Secretary 7, 8 MHz multi-country support task force Leader International expansion of the ISDB-T digital terrestrial broadcasting format Brazil chose ISDB-T as its digital terrestrial broadcasting format in June 2006, and it began broadcasting in December Encouraged by this success, we cooperated with the public and private sectors on a promotional campaign for other Latin American countries to chose ISDB-T as their format for digital broadcasting. As a result of these efforts, five countries had adopted ISDB-T by FY2009. Furthermore, FY2010 saw four more Latin American countries announced their adoption of ISDB-T: Costa Rica in May, Paraguay in June, Bolivia in July, and Uruguay in December. In June, the National Communications Committee in the Philippines announced it would adopt ISDB-T. So far eleven countries Costa Rica Venezuela Ecuador Peru Brazil Bolivia Paraguay Chile Uruguay Argentina ISDB-T Adopted ISDB-T Launched Figure 1. Adoption of ISDB-T in Latin American countries 42 NHK STRL ANNUAL REPORT 2010

45 6 Research-related works 6.1 Joint activities with other organizations besides Japan have adopted ISDB-T, and three Brazil, Peru, and Argentina have begun broadcasting Figure 1 With the decisions to adopt ISDB-T in 11 Latin American countries, we shifted the focus of ISDB-T promotional activity to South-East Asia and Southern Africa. NHK has also cooperated in these activities to expand internationally Table 1 STRL tours were held for Asia-Pacific Broadcasting Union ABU technical committee members from various countries during the ABU general meeting in Tokyo in October, including demonstrations of technology such as multiband ISDB-T. Table 1. Major collaborations in FY2010 Date 2010/4 2010/6 2010/7 2010/7 2010/7 Description Japan and Brazil give a joint presentation and demonstration of ISDB- T at digital terrestrial broadcasting symposiums held in South Africa and Lesotho. In South Africa, we gave an indoor exhibition of multiband ISDB-T modulators and receivers supporting the 6, 7 and 8 MHz bandwidths to South African cabinet ministers and broadcasters. We attended a workshop in Vietnam sponsored by ASEAN, on digitization ofbroadcasting and accommodation offrequencies freed-up after the end of analog broadcasting, and gave an address on Japan s initiatives toward digitization ofbroadcasting and the end of analog broadcasting. In Thailand, we participated in a seminar on digital terrestrial broadcasting sponsored by theministry of Internal Affairs and Communications and the Thai committee on electronic communications, bypre- senting an address and exhibit on ISDB-T. Parties related to broadcasting in countries affiliated with the Southern African Development Community SADC were invited to Brazil for a Japan-Brazil digital television workshop, where we demonstrated an 8 MHz bandwidth version of ISDB-T. Date 2010/ / / /12 Description In a tour to promote emergency warning broadcasting system, sponsored by the Asia-Pacific Broadcasting Union and including the six countries of Malaysia, Thailand, Cambodia, Vietnam, the Philippines and China, we gave talks introducing emergency warning broadcasting system and earthquake early warning using ISDB-T. Inaseminar and demonstration on digital terrestrial broadcasting using ISDB-T in South Africa, sponsored by theministry of Internal Affairs and Communications and the Japanese Embassy in South Africa, weexhibited ISDB-T using test signals transmitted at the location. At the general meeting of local broadcasters held in the Philippines, we gave a presentation introducing the features and services of ISDB- T. At the IECEP2010 international convention of electrical engineering and communications held in the Philippines, we gave a presentation introducing the features and services of ISDB-T Collaboration with overseas research facilities Collaborative research, cooperating institutes STRL collaborated with the BBC U.K. as part of its mutual research collaboration agreement made in 2007 with the laboratories of European public broadcasters, the BBC U.K., RAI Italy and IRT Germany We also conducted a collaborative research project with the TBRRI a Canadian research facility and the State University of New York in the U.S.A. In November, the Digital Group of Broadcasting Technology and Research DGBTR was formed by ABS China and KBS Korea in Beijing, with the goal of exchanging information on the latest broadcast technology and research. In FY2010, we conducted a total of 46 collaborative research projects on topics ranging from systems development to materials and basic research. We also continued our educational work with graduate schools at six universities Tohoku University, University of Electro-Communications, Tokyo University of Science, Tokyo Institute of Technology, Waseda University, and Chiba University We sent part-time lecturers to these universities and accepted trainees from them Visiting researchers, trainees, STRL staff overseas We invited two researchers from TV Globo in Brazil. Note that we accepted two applications from young visiting researchers from facilities affiliated with the Asia-Pacific Broadcast Union ABU but they were unable to attend. We commissioned four outside research projects involving three post-doctoral students and one researcher at a university overseas Table 1 We accepted a total of 23 trainees from ten universities Waseda University, Toho University, Tokai University, Tokyo Denki University, Tokyo University of Science, the University of Electro-Communications, Senshu University, Nihon University, Nagaoka University of Technology, Korea University in South Korea STRL also dispatched two researchers to research institutions in the U.S.A. and the U.K. Table 2 Table 1. Delegated research Delegate Post-doctorate Post-doctorate Post-doctorate McGill University, Canada Term 2009/10/1-2011/9/ /10/1-2010/7/ /11/6-2011/11/5 2009/3/1-2012/2/28 Research topic Research on increasing the degree of light modulation in spin-injection light modulator devices Discovering organic semiconductor thin films with ultra-high field-effect mobility Research on marker-less camera tracking Research on technology to add reverberation for 3D audio design tools, and control methods for them NHK STRL ANNUAL REPORT

46 6 Research-related works 6.1 Joint activities with other organizations Table 2. NHK STRL staff overseas Location NBC, U.S.A. Surrey University, U.K. Term 2010/8/ /2/ /8/ /2/25 Research topic Survey and research on the latest hybrid broadcasting and communications services in the U.S.A. as well as related technologies Research on generating 3Dinformation frommulti-viewpoint images Commissioned research We engaged in contract research for a number of national and public research organizations. The projects commissioned by NICT*, NEDO**, the Ministry of Education, Culture, Sports, Science and Technology, and JST*** are titled as follows: R&D on Multi-Parallel & Spatial Imaging 3-Dimensional Television System R&D on Ultra-Realistic Communication Technology with Innovative 3D Video Technology Issue 1: Device technology for innovative 3D video display Issue 2: Core elemental technologies for communications and broadcasting of 3D video Issue 4: Recognition and transmission of sensitivity information Issue 5: Ultra-realistic communication system Development of Next-generation High-efficiency Network Devices Technology R&D on proving test systems for expansion of digital museums mixed-reality digital museum Creation of science, technology and systems for enriching an aging society *NICT: National Institute of Information and Communications Technology **NEDO: New Energy and Industrial Technology Development Organization ***JST: Japan Science and Technology Agency Various committee meetings We held two meetings of the broadcast technology research committee and received input from committee members who are academic and experienced people. The research adviser meetings took input from research advisers and guest researchers and covered ten sessions. Broadcast technology research committee members Titles omitted March 2011 ** Committee chair * Committee vice-chair Name Affiliation *Kiyoharu Aizawa Professor, University of Tokyo Interfaculty Initiative in Information Studies ShigeyukiAkiba President/CEO, KDDI R&DLaboratories ** Makoto Ando Professor,Graduate School of Engineering, Tokyo Institute of Technology Shingo Ichimura Vice President, National Institute of Advanced Industrial Science and Technology AIST Shinichi Oishi Professor, Department of Applied Mathematics, School of Fundamental Science and Engineering,Waseda University Shunichiro Kudo Senior Director,National Association of Commercial Broadcasters in Japan Hiroshi Kumagai Vice President,National Institute of Information and Communications Technology Yasuhiro Koike Professor, Department of Applied Physics and Physico-Informatics,Faculty of Science & Technology,Keio University Hiromichi Shinohara Senior VP,NTT Group; Director, Research and Development Planning Department Atsushi Takahara Vice-Director, Institute for Materials Chemistry and Engineering,Kyushu University Hiroshi Tanaka Section Manager, Broadcast Technology Division., Information and Communications Bureau,Ministry of Internal Affairs and Communications Toshiei Toyonaka Director, Tokyo Broadcasting System Television Inc. Michihiko Minoh Director-General, Institute for Information Management and Communication, Kyoto University Hiroaki Muraoka Professor, Research Institute of Electrical Communication, Tohoku University Masayuki Murata Professor,Graduate School of Information Science and Technology,Osaka University Research advisers Titles omitted March 2011 Name Affiliation Tsuneya Ando Professor,Graduate School of Engineering, Tokyo Institute of Technology Koichi Ito Professor,Graduate School of Engineering, ChibaUniversity SusumuItoh Professor, Department of Electrical Engineering,Faculty of Science and Technology, Tokyo University of Science Tohru Ifukube Professor, Research Center for Advanced Science and Technology, Tokyo University HidekiImai Professor, Department of Electrical, Electronic, and Communication Engineering, Faculty of Science and Engineering, Chuo University Tatsuo Uchida President, Sendai National College of Technology Juro Ohga Emeritus Professor, Shibaura Institute of Technology Jiro Katto Professor, Department of Computer Science and Engineering, Faculty of Science and Engineering,Waseda University Yoichi Suzuki Professor, Research Institute of Electrical Communication, Tohoku University Fumio Takahata Professor, Department of Computer Science and Engineering, Faculty of Science and Engineering,Waseda University Kunihide Tachibana Professor,Graduate School of Science and Engineering, EhimeUniversity Takashi Taniguchi Professor,Faculty of Informatics,Osaka Gakuin University Jun-ichi Tsujii Professor,Graduate School of Information Science and Technology, University of Tokyo Yoshihisa Nakamura President, Iwate Prefectural University Mitsutoshi Hatori Emeritus Professor, University of Tokyo Hiroshi Harashima Emeritus Professor, University of Tokyo Takehiko Bando Emeritus Professor,Niigata University Sadaoki Furui Professor, Graduate School of Information Science and Engineering, Tokyo Institute of Technology Kuniyoshi Yokoo Emeritus Professor, Research Institute of Electrical Communication, Tohoku University Akihiko Yoshikawa Professor,Graduate School of Engineering, ChibaUniversity Guest researchers Titles omitted March 2011 Name Affiliation Tomoaki Ohtsuki Professor, Department of Information and Computer Science,Keio University Kenji Ozawa Professor, Interdisciplinary Graduate School of Medicine and Engineering, Yamanashi University Wataru Kameyama Professor, Graduate School of Global Information and Communication Studies, Waseda University Yoshimasa Kawata Professor, Department ofmechanical Engineering,Faculty of Engineering, Shizuoka University Yoichi Sato Professor, Institute of Industrial Science, University of Tokyo Satoshi Shioiri Professor, Research Institute of Electrical Communication, Tohoku University Hiroyuki Shinoda Associate Professor,Graduate School of Information Science and Technology, University of Tokyo Satoshi Sekine Research Associate Professor, Computer Science Department,NewYork University Takao Someya Professor, School of Engineering, University of Tokyo Yoshiaki Tsushima Post-doctoral Researcher,Harvard University 44 NHK STRL ANNUAL REPORT 2010

47 6 Research-related works 6.2 Publication of research results 6.2 Publication of research results STRL Open House The STRL Open House 2010 commemorated STRL s 80th anniversary. The theme was STRL 80th Anniversary Delivering the Future, and the event included 44 exhibitions and ten poster displays of the latest research results. Family events were held on Saturday and Sunday, with STRL researchers giving a series of talks, titled Let s learn how TV works, and introducing the latest broadcast technology. There were also interactive exhibitions and a stamp rally. The event attracted total of 22,087 visitors. May 25 Tuesday Opening ceremony May 26 Wednesday Open by invitation May Thursday to Sunday Open to the public The exhibition entrance A research presentation Let s learn how TV works Stamp rally Addresses Title Broadcast Technology Evolution - Aiming for new lifestyles - Hulu and Video Services on IP Networks 2010 and 10 Years Ahead - Anticipating technical developments for the real User Age - Broadcast Technology Supporting Seeing, Hearing and Speaking -Bornfrom solving the puzzles of sensation - Speaker Keiichi Kubota,Head of NHK STRL Glenn Reitmeier, Senior Vice President,NBC Universal Gota Iwanami, President/CEO, INFOCITY, Inc. Tohru Ifukube, Project Professor, Univ. of Tokyo Research Center for Advanced Science and Technology Research presentations Title Research on 3DTV at NHK STRL Toward the Realization ofhybridcast - Objectives of R&DatNHK STRL - Flexible Program Production Systems A Full Resolution Super Hi-Vision Camera System Speech Recognition for Live Closed-Captioning Millimeter-wave TV Camera Improving the Picture Quality of Ultra-high-sensitivity HARP Imaging Devices An Ultra-fine Spatial Light Modulation Device Speaker TakayukiIto, Deputy Head of NHK STRL Hisakazu Katoh,Head of Advanced Broadcasting Platforms Research Division, NHK STRL Shinya Takeuchi, Advanced Broadcasting Platforms Research Division, NHK STRL Takayuki Yamishita, Advanced Television Systems Research Division, NHK STRL Shinichi Homma,Human & Information Science Research Division, NHK STRL Hirokazu Kamoda, Broadcasting Networks Research Division, NHK STRL Yuji Ohkawa, Imaging & Storage Devices Research Division, NHK STRL Kenji Machida, Display &Functional Devices Research Division, NHK STRL Exhibits 1 Hybridcast:Fusing Broadcasting and Communications 16 Translating Japanese into Computer-animated Japanese Sign Language 31 Relevant Retrieval Systemfor NHK Creative Library 2 Context-aware TV User Interface 17 Live Closed-captioning using Speech Recognition 32 CG Content Production Services using Cloud Computing 3 Full-resolution Super Hi-Vision Camera System 18 Automatic Evaluation of Program Sound Balance for the Elderly 33 Flexible Program Production System 4 High-performance Super Hi-Vision Video Codec 19 Automatic Detection of Unpleasant Scenes 34 Lighting Device Technology using Organic Light Emitting Diodes 5 Integral Three-dimensional Television 20 Cable Television Systemfor Super Hi-Vision 35 Advanced FPU Mobile Relay System 6 Channel Equalizer ofmultipath Distortion for Community Reception Facilities 21 Optical LANfor Super Hi-Vision 36 Millimeter-wave TV Camera 7 Interference Canceller for Digital Terrestrial Broadcasting Receiver 22 Lightweight Loudspeaker using Polymer Films 37 The VFX World of SakanoUenoKumo 8 Mobile Reception Technology for Digital Terrestrial HDTV Broadcasting inch Diagonal Ultrahigh-Definition Plasma Display with 0.33-mm Pixel Pitch 38 Licensing and Applicational Development of NHK s Technology 9 Mobile Multimedia Broadcasting in the VHF-Low Band 24 Wide-gamut Colorimetry for Super Hi-Vision 39 Haptic Technology for Representing 3-D Objects 10 Large-capacity Transmission Technology for Next-generation Digital Terrestrial Broadcasting 25 Home Reproduction of Super Hi-Vision Audio 40 Organic Image Sensor 11 Robust Transmission Technology ofmobile Reception for Next-generation Digital Terrestrial Broadcasting Multichannel Sound Production System 41 HARP Imaging Device using Fiber Optic Plate 12 Consultation Desk for Answering Your Questions about Digital Broadcasting Reception 27 Identity-Web Services Framework between Broadcasting and Communication Services 42 Flexible Organic Light Emitting Diode Display 13 Super Hi-Vision Theater 28 Secure Content Distribution Services 43 Recording System using Thin Optical Disks Years of STRL Achievements 29 Digital Watermarking Systemfor Content Distribution over the Internet 44 High-density Holographic Recording Technology 15 Multi-modal Information Presentation Systemfor the Visually Impaired 30 Comment Analysis for Social TV Services NHK STRL ANNUAL REPORT

48 6 Research-related works 6.2 Publication of research results NHK STRL 80th anniversary events The NHK STRL 80th Anniversary Celebration - 80 Years with the People of Setagaya was held in cooperation with the Setagaya City Ward Office on October 16 and 17. This event included an experiential exhibit and stage, and people from Setagaya ward participated in the exhibits and the stage events. A total of 4,402 visitors attended. The Dream Jazz Band performed in the STRL Hall. This band is composed of students from middle schools in Setagaya Ward. There was also a joint performance of the NHK Friendship Choir and Orchestra and the STRL Jazz band under the direction of Mr. Akira Senju. The opening ceremony, the Setagaya Municipal Yamano Elementary School Choir, and the Setagaya Municipal Sakuragaoka Elementary School Band performed on the main stage. The celebration also featured a quiz contest and science lessons by STRL staff and a musical introduction to STRL s history. The experiential exhibits included photography in a virtual studio, 3D video, and games utilizing voice and face recognition technologies. There was also a parent-child handicraft corner, and displays of local goods from Setagaya, including Karuta traditional card games and the winning entries from the 2010 Setagaya Residents Art Exhibition. Dream Jazz Band performance Hands-on handicraft corner Musical introduction to STRL s history Game utilizing face recognition technology Exhibitions inside and outside Japan In September 2010, we exhibited our latest Super Hi-Vision SHV equipment at IBC2010, the largest exhibition of broadcasting equipment in Europe. The event, held at the RAI Convention Center in Amsterdam, was the first time to show our full-resolution projectors and cameras, along with the research results that made them possible. We also exhibited our latest dual-green devices, with improved compactness and performance, and content production equipment and technology. The event attracted 48,521 visitors. NHK broadcasting stations throughout Japan hosted exhibits of our latest broadcast technology. We developed many new exhibition materials and presented them at many of these events, including a TV no shikumi, which is a simple introduction to television principles such as video compression, a compact integral 3D TV display exhibit, and an exhibit called Moguraian no takara sagashi, which uses voice recognition to enable anyone to control a computer-graphic animated character. 2 overseas exhibitions Event name NAB2010 Las Vegas, U.S.A IBC2010 Amsterdam, Netherlands Dates 4/12-4/15 9/10-9/14 Exhibits Introduction to NHK STRL Super Hi-Vision 40 exhibitions within Japan Event name Major events Thanks inshibuya Shibuyadedomo Yokohama Broadcast Library NHK Yokohama 80th Anniversary of the NHK KanazawaStation, Future of Broadcasting NHK Kanazawa CEATEC2010 DIGITAL CONTENTS EXPO2010 Broadcast Experience Festival in Omoromachi NHK Okinawa Fureai Festa 2010 NHK Fukuoka NHK Open Studio 80th Anniversary Special NHK Okayama Dates 5/1-5/4 7/23-9/26 9/11-9/20 10/4-10/8 10/14-10/17 10/23-10/24 12/4-12/5 1/29-2/6 Exhibits Shabette asobo, Ultra-high-speed Camera TV no shikumi, PC-VRAS, Shabette asobo TV no shikumi, Virtual Puppet, Shabette asobo Voice recognition, CG Sign language,haptic Terminal, Integral 3D television EN-Vision New TV no shikumi, PC-VRAS, Integral 3D television, Virtual puppets, etc. Super Hi-Vision theater,3dhi-vision, etc. Moguraian no takara sagashi, Integral 3D television, etc. 46 NHK STRL ANNUAL REPORT 2010

49 6 Research-related works 6.2 Publication of research results Academic conference, etc. Besides presenting research results at many domestic and international conferences such as ACM Multimedia and SMPTE, we had many papers published in academic journals, including IEEE Transactions, Optics Letters, Advanced Materials, and Applied Physics Letters. Academic journals in Japan Overseas journals Academic and research conferences in Japan Overseas/international conferences, etc. Contributions to general periodicals Lectures at the other organization Total 63 papers 20 papers 269 papers 134 papers 73 articles 93 events 652 events Bulletins STRL publishes the bulletins describing its research activities and achievements. NHK STRL R& D has been completely redesigned to have special features on content use technology, technology linking broadcasting and communications, and special data-input device technology. Broadcast Technology is oriented to overseas readers; it introduces our latest research on topics such as 3 D video processing technology and content search technology. STRL Dayori Japanese, monthly No. 61 to No. 72 NHK STRL R&D Japanese, bimonthly No. 121 to No. 126 Annual Report Japanese, annually FY2009 Edition Broadcast Technology English, quarterly No. 40 to No. 43 Annual Report English, annually FY2009 Edition Press releases We issued eleven press releases on our research results and other issues. Date 2010/5/18 5/19 5/20 5/20 5/21 5/24 5/24 9/6 9/ /1/13 3/ Press release content Development of a full-resolution Super Hi-Vision camera - Achieving the real image quality of SHV!Making outdoor high-resolution capture possible - Development of high-performance, compact Super Hi-Vision coding equipment Development of a Super Hi-Vision audio reproduction method for homeuse - Reproducing 22.2 multichannel audio, even with some few speakers - Development of a 3D audio mixing system -Efficient production of 22.2 multichannel audio for Super Hi-Vision - Development of ultra-lightweight loudspeakers using polymer films -Able to reproduce wide-band audio at 1/20th the weight of conventional speakers - Development of technology enabling high-capacity terrestrial transmission - With the goal of terrestrial broadcast of Super Hi-Vision - Development of a large, millimeter-wave reflectarray antenna for a millimeter-wave TV camera -Enabling motion capture by controlling a millimeter-wave beam - 80th anniversary of NHK STRL, STRL Special Autumn Event Regarding the interruption of services for the digital NHK general channel in some regions Development of apractical, compact SHVprojector! -Toward SHV testbroadcasts in First use of anewly developed receiving system on a live broadcast ofmarathon -Expanding stable transmission area for mobile coverage - Visits, tours, and event news coverage STRL Dayori Web site NHK STRL R&D Broadcast Technology The STRL web site presents an outline of STRL and its research. It also provides information on conference presentations, patents, and events such as the STRL Open House, as well as up-to-date announcements and reports related to research. It also introduces STRL s journals and other publications. Participants in the Let s learn how TV works event, held Saturday and Sunday during the STRL Open House this year, were recruited by linking the NHK Net Club to the STRL web site. The web site was also used to promote events such as the STRL 80th anniversary event held in the autumn. To promote Japan s digital terrestrial broadcasting format overseas, we held study and inspection tours for officials from the Southern African Development Community SADC the Philippines, Argentina, and Chile. We welcomed many visitors from the European Broadcasting Union EBU CBS in the USA, KBS in South Korea, etc. STRL Open House web site STRL 80th anniversary event web site Inspections, tours News-media 93 instances 40 from overseas 1399 visitors 411 from overseas 51 instances NHK STRL ANNUAL REPORT

50 6 Research-related works 6.3 Application of research results 6.3 Application of research results Cooperation with program producers Our R & D results have been used in many programs and program-related activities. For example, we have developed a single-chip HDTV camera that makes use of technology resulting from our R&D on Super Hi-Vision. This camera s compactness and good imaging characteristics have been put to use in the production of dramas. Our insect microphone, ultra-highspeed camera, and Super HARP camera have been used in nature and science programs. Our rear-canceling microphone, which can focus on and record particular sounds clearly, and our durable silicon microphones have been used for covering marathons, golf tournaments, baseball games, swimming meets, and cultural festivals. Our millimeter-wave mobile camera, which does not restrict the movements of the camera operator, has been used to produce programs conveying a strong sense of presence. In FY2010, we collaborated in the production of a total of 82 programs. Use of the millimeter-wave mobile camera in Kohaku Utagassen and other programs The millimeter-wave camera is a wireless camera that frees the operator of encumbering wires that restrict his or her movements. In FY2010, we added a bidirectional communication link in the millimeter-wave-band to the camera. The link greatly improves the camera s operability by carrying a low-delay, stable return video signal and detailed remote configuration functions. The camera has been used in the Kohaku Utagassen year-end song festival program since 2008, and its mobility makes it easier for operators to capture striking video images. In 2010, it was used for over 50 cuts in Kohaku Utagassen and for coverage of golf and other outdoor events. Live P2P distribution of the NHK School Chorus Contest The NHK School Chorus Contest Shikoku block contest was covered live with STRL s peer-to-peer P2P technology, which simultaneously distributes live video to multiple viewers on the Internet. Over 2000 viewers who were unable to attend the event were able to enjoy the video. Use of EN-Vision with NHK On Demand We added a function to NHK On Demand NOD to display recommended programs when users select programs they wish to view. Technology from STRL s EN-Vision was used in this related-program search function. The EN-Vision system uses program metadata published on NOD to automatically select other programs that have descriptions similar to the selected program. It also recommends programs featuring the same actors or in the same genre. High-school TVML productions at the NHK Bunkasai Festival The TV program Making Language TVML developed at STRL was used by three high-school groups to create programs that were presented at the NHK Bunkasai festival and in a program titled Digista Teens that was broadcast to decide the winning production by popular vote. At the venue, TVML attracted much interest as a tool that allows easy production of content combining real imagery with computer graphics. Capturing with the millimeter-wave mobile camera Live streaming screen of NHK School Chorus Contest Related program recommendations with NHK On Demand Recording the Digista Teens program Patents NHK is participating in a Digital Broadcasting Patent Pool, which bundles licensing of patents required by digital broadcasting standards under reasonable conditions. The Pool promotes the use of patents held by NHK and contributes to the smooth spread of digital broadcast receivers as the final transition to digital broadcasting approaches in Japan. We are continuing in our efforts to protect rights for R&D results as part of NHK s intellectual property management effort, and we are promoting transfers of patented NHK technologies at events such as the Patent Solution Fair 2010, held by the Japan Patent Office, and Technical Show Yokohama 2011, hosted by the City of Yokohama. 48 NHK STRL ANNUAL REPORT 2010

51 6 Research-related works 6.3 Application of research results Patent and utility model applications submitted NHK Total Patents and utility models in use NHK Total Domestic Overseas Total Type Patents Utility models Designs New Total at end of FY Type Contracts Licenses breakdown Patents Expertise New Total at end of FY Patents and utility models rights granted NHK Total Technical cooperation NHK Total Domestic Overseas Total Type Patents Utility models Designs New Total at end of FY Type Technical cooperation projects, Commissioned testing Commissioned research projects Total Prizes, degrees STRL received 25 prizes in FY2010, including The Commendation by the Minister of Education, Culture, Sports, Science and Technology, Hoso Bunka Foundation prizes, and The Meritorious Award on Radio. Eight staff members obtained new academic degrees. At the end of FY2010, 92 STRL members held doctoral degrees. Satoshi Aihara Shuichi Aoki MasayukiTakada Kazuyoshi Shogen Nobuhiko Funabashi Madoka Nakamura Hiroyuki Kaneko Kazuyoshi Shogen Yoshihiro Fujita Kazuyoshi Shogen Mamoru Doke Award winner Akinori Hashimoto, Tomohiro Saito, Masaaki Kojima,Yoichi Suzuki and Shoji Tanaka representative Nobuyuki Yagi Digital terrestrial broadcasting auxiliary channel emergency earthquake warning transmission method development group Shizuo Tokito, Mitsunori Suzuki and Gen ichi Motomura Digital broadcast video quality monitoring equipment development team Institute of Electronics, Information and Communication Engineers IEICE Shinichi Sakaida Emergency earthquake warning AC transmission method development group Association of Radio Industries and Businesses ARIB representative Hiroyuki Hamazumi Telecommunication Technology Committee Masaru Takechi TTC 120-GHz-band program contribution transmission system development group, Fuji Television, NHK, Hoso Bunka Foundation NTT Production teamfor SakanouenoKumo Hoso Bunka Foundation Institute of Image Information and Television Engineers ITE Institute of Image Information and Television Engineers ITE Information Processing Society ofjapan IPSJ Institute of Electronics, Information and Communication Engineers IEICE Tokyo Metropolitan Government Japan Institute of Invention and Innovation Tomoyasu Komori Institute of Electronics, Information and Communication Engineers IEICE Yuichi Iwadate Takayanagi Foundation for Electronics Science and Technology Super Hi-Vision optical-transmission technology development group TEISHIN association representative Kimiyuki Oyamada Masafumi Nagasaka Institute of Electronics, Information and Communication Engineers IEICE ABU Awarded by Ministry of Education, Culture, Sports, Science and Technology The ITU Association ofjapan ITU-AJ The ITU Association ofjapan ITU-AJ Institute of Image Information and Television Engineers ITE Institute of Image Information and Television Engineers ITE Institute of Image Information and Television Engineers ITE Institute of Image Information and Television Engineers ITE Institute of Image Information and Television Engineers ITE Institute of Image Information and Television Engineers ITE ITU-AJ Award ITU-AJ Award Fujio Frontier Award Hi-Vision Technology Award The Information and Systems Society Distinguished Service Award The Meritorious Award on Radio ARIB Chairman s Award TTC Award Hoso Bunka Foundation Award Hoso Bunka Foundation Award For technical development and operation of production methods for the special drama, SakanouenoKumo For the publication of Magneto-optic observations of the structure of nano-magnetic regions at the 2009 ITE Annual Suzuki Memorial Incentive Award Symposium, and Evaluation of the MO characteris- tics of spin-injection optical modulators at the 2009 ITE Winter Symposium Suzuki Memorial Incentive Award Prototype of a device optical interface for full-resolution Super Hi-Vision FIT Young Researcher Award Decentralized cooperative program production using Network Ad-Lib System Fellow designation Research in satellite broadcasting and implementation of satellite-mounted beamforming antennas Tokyo Merit Award R&D on sophisticated, high-quality video capture and distribution technology ABU Technical Review Prize for 2008 Best Article Award Current status of UHDTV Super Hi-Vision delivery Local Commendation for Invention - Honorable Mention Automatic program production equipment Human Communication Award Research on voice/audio signal processing for the elderly, Award for best paper related to human communication Takayanagi Award Honorable Mention MaejimaAward Award name The Commendation for Science and Technology by the Minister of Education, Culture, Sports, Science and Technology Young Scientist Prize Niwa-Takayanagi Award Achievement Award Niwa-Takayanagi Award Best Paper Award Niwa-Takayanagi Award Best Book Award Technology Promotion Award Development Award Young Researcher saward In recognition of Research on photoconductor films using organic materials for imaging devices International Activity Award, Award for achievement in field International Activity Award, Award for international collaboration in field Contribution to increasing the capacity of digital satellite broadcasting ARIB demonstration tests for advanced digital satellite broadcast methods Book title: Video media technology Development of a method for encoding and transmitting emergency earthquake warning and earthquake center information, focusing on the one segment auxiliary channel carrier in digital terrestrial broadcasting signals Prototype of a flexible EL display forming flexible display elements with ink-jet methods using organic EL materials with phosphorescent polymers Development of digital broadcast video quality monitoring equipment able to estimate the resolution of an MPEG-2- encoded stream without the source images, using an original algorithm Contributions to activities of the Image Engineering Technical Group Digital terrestrial broadcasting AC emergency earthquake warning transmission method development group Achievements advancing standardization related to multimedia broadcasting Development of a high-capacity program contribution transmission system using the 120-GHz-band Development of a multi-viewpoint 3D video system Development of Super Hi-Vision optical-transmission technology Date 2010/4/ /5/ /5/ /5/ /5/ /5/ /5/ /5/ /5/ /6/1 2010/6/ /6/ /7/1 2010/7/1 2010/9/1 2010/9/1 2010/9/8 2010/9/ /10/1 2010/10/ /11/ /12/ /1/ /3/11 Study of receiver-antenna emission patterns for 21-GHzband satellite broadcasting 2011/3/15 Only the names of STRL staff-members are shown NHK STRL ANNUAL REPORT

52 NHK Science & Technology Research Laboratories Outline The NHK Science & Technology Research Laboratories NHK STRL is the sole research facility in Japan specializing in broadcasting technology, and as part of the public broadcaster NHK, its role is to lead Japan in developing new broadcasting technology and contributing to a rich broadcasting culture. History of broadcasting development and STRL 2020 Target : Super Hi- Vision test broadcasting 2011 : Switchover to all-digital television broadcasting 2006 : One-Seg broadcasting begins 2003 : Digital terrestrial broadcasting begins 2000 : BS Digital broadcasting begins B-SAT3a Provided by: B-SAT/LMCSS 1995 : Super Hi-Vision research begins 1989 : BS Analog broadcasting begins 1982 : Digital broadcasting research begins 1991 : Hi-Vision test broadcasting begins analog STRL Open House 1966 : Satellite broadcasting research begins 1964 : Hi-Vision research begins 1953 : Television broadcasting begins 1930 : NHK Technical Research Laboratories established 1925 : Radio broadcasting begins The STRL Open House is held every year in May, providing an easy introduction to STRL R&D results. STRL by the numbers Current research building Established in June 1930 June January 1965 January July 1984 July Present Technical Research Laboratories Technical Research Laboratories, Broadcast Science Research Laboratories Science & Technology Research Laboratories Employees 247 including 222 researchers Degree-holding personnel 92 Qualified patent agents 2 Patents held Domestic 1,329 NHK Total International 234 NHK STRL organization at end of FY2010 Head of STRL Keiichi Kubota Executive Research Engineers Kazuyoshi Shogen, Yuji Nojiri, Takayuki Ito Completed March 2002 High rise building : 14 floors above ground, 2 below ground Mid height building : 6 floors above ground, 2 below ground Total floor space : Approx. 46,000m 2 Includes research area : Approx. 16,000m 2 Total land area : Approx. 33,000m 2 Deputy Head of STRL Shuichi Fujisawa Planning and Coordination Division Patents Division Advanced Broadcasting Platforms Research Division Broadcasting Networks Research Division Advanced Television Systems Research Division Human & Information Science Research Division Imaging & Storage Devices Research Division Display & Functional Devices Research Division General Affairs Division Planning and management of research, public relations, domestic/international liaison, etc. Patent rights management, technology transfers, etc. Broadcast and communications hybrid systems, IP network, security, digital rights management, social network services, etc. Next-generation digital broadcasting, satellite broadcasting, cable broadcasting, contribution system, optical link, etc. Super Hi-Vision, three-dimensional television, high-presence audio systems, video and audio coding, audiovisual perception, etc. Image recognition and representation, speech recognition and synthesis, natural language processing, information retrieval, cognitive science, information-barrier-free technologies, etc. High-sensitivity imaging, high-speed imaging, organic imaging, high-density magnetic recording, high-speed optical recording, etc. Flexible displays, plasma displays, organic devices, holographic recording, spin electronics, etc. Staff administration, accounting, facilities management, etc. Director Nobuyuki Yagi Tadashi Nakayama Hisakazu Kato Toru Kuroda Eisuke Nakasu Masahiro Shibata Norifumi Egami Naoki Shimizu Norimitsu Yamada at end of FY NHK STRL ANNUAL REPORT 2010