1 Television with a strong sensation of reality

Transcription

1 1 Television with a strong sensation of reality NHK STRL is researching Super Hi-Vision SHV and three-dimensional 3D television, to create television conveying a strong sensation of reality. Standardization of the SHV format is advancing at ITU-R, and Recommendation BT. 2020, Parameter values for ultra-high definition television systems for production and international programme exchange, was established in August, This specification includes the fullspecification SHV frame frequency of 120 Hz and wide-gamut system colorimetry. We developed equipment to convert from the SHV wide-gamut system colorimetry to the HDTV color gamut. We devised and prototyped an interface specification for transmitting the signals described in BT We developed a compact single-chip color SHV camera head weighing only 5 kg using a 33 megapixel image sensor and prototyped equipment capable of operation within a theatre. We also prototyped a three-chip full-spec SHV image-capture equipment using the SHV image sensors prototyped in FY 2011 and able to support a 120 Hz frame rate. In addition to increasing the sensitivity of image sensors, we worked on the design of image sensors for a single-chip full-spec camera. In our work on SHV displays, we improved on the quality of the 145-inch PDP developed in FY 2011 and used it for public viewings of the Olympics in London. We also developed an SHV projector capable of displaying 120 Hz video. In our work on coding, we continued with our standardization activities of the HEVC video compression, and were instrumental in drawing up an international standard including support for the SHV image resolution and frame rate. We also prototyped an SHV hardware encoder based on the HEVC standard. We also prototyped 4K hardware based on a new reconstructive video coding using super-resolution technology and tested this method. In our work on audio, we developed a reverberator using measured impulse responses and improved the directional characteristics of our spherical microphone. We used these technologies in production of 22.2 multi-channel audio for the London Olympics. We demonstrated the effectiveness of 22.2 multi-channel format through subjective evaluations in which we changed the positions of the speakers. We analyzed the characteristics of our display-integrated loudspeaker array for home playback of 22.2 multi-channel audio and improved processing that enables the front speakers to reproduce audio from the sides and behind. We also developed a 3D audio coding able to play back two-channel and 5.1-channel signals from parts of a 22.2-channel bit stream. We also contributed to standardization related to 22.2 multi-channel audio with ITU-R, SMPTE, IEC, AES, and MPEG. In our work on 3 D television, we made advances in our research on integral 3D television, which provides natural, viewable images and does not require special glasses. We developed a method for configuring the overall recording and playback system. We also prototyped a fine lens array approximately the same size as the imaging sensor and developed a compact capture device that does not require an imaging lens to focus the images from the lens array. We began studies to improve the quality of 3D images reproduced by integral 3D systems. We also improved the way we obtain integral 3 D images from multiviewpoint images, create 3 D models of objects, and use information from depth cameras. We developed a multi-viewpoint robotic camera system that eliminates the need for camera calibration using special patterns. Regarding our work on SHV transmission, we studied wide-band transmission methods for high-capacity transmission using 12 GHz and 21 GHz band satellites, and efficient, high-capacity transmission technologies for the next generation of digital terrestrial broadcasting systems. We also distributed SHV programs using multiplecarrier transmission equipment in an actual cable television facility. We also studied optical wavelength division multiplexing 144 Gbps and Ethernet 72 Gbps technologies for transmitting SHV signal contributions for production. 1.1 Super Hi-Vision Super Hi-Vision format We have advanced with our R&D and standardization activities related to the Super Hi-Vision 8K/SHV video system. Video format standardization We continued our work on standardization of the SHV video format called UHDTV in standardization terminology with ITU- R. In FY2011, we reached an agreement on a preliminary draft 4 NHK STRL ANNUAL REPORT 2012

2 Table 1. Main parameters of UHDTV and HDTV image formats UHDTV HDTV ARIB Standard STD-B56 S-001C Resolution Horiz. Vert 7,680 4,320, 3,840 2,160 1,920 1,080 Frame frequency Hz 120, 60, 60/ , 60/ , 30/1.001 Field frequency Hz 60, 60/1.001 Scan Progressive scan Progressive scan Interlaced scan Chromaticity coordinates of the primary colors and reference white x, y Luminance Red 0.708, , Green 0.170, , Blue 0.131, , White D , , Chrominance Pixel structure for luminance and chrominance 4:4:4, 4:2:2, 4:2:0 4:2:2 Bit depth bit 10, 12 8, 10 new Recommendation, including a wide-gamut colorimetry and a frame frequency of 120 Hz, but the formats for the luminance and color-difference signals were outstanding issues. We did a comparative study of constant luminance and nonconstant luminance the conventional format and obtained results showing that there is less degradation when converting to a constant luminance from an RGB source image but that there is no subjective difference in quality between the two. Moreover, we found that there are differences when processing the luminance and color-difference signals with constant luminance compared with the results when processing RGB signals. In FY2012, we worked at ITU-R to standardize UHDTV based on these results. This resulted in agreement that both constant luminance and non-constant luminance have advantages and disadvantages and the draft new Recommendation would include both methods. The approval process resulted in the establishment of Recommendation ITU-R BT. 2020, Parameter values for ultra-high definition television systems for production and international programme exchange, 1 in August The SMPTE standard was revised and ARIB standard STD-B56 2 Table 1 was established in light of the ITU-R Recommendation. Among the formats specified in the ITU-R Recommendation, ARIB STD-B56 specifies the formats that will be used for UHDTV broadcasting in Japan. We also contributed to creating standard test materials at ARIB for UHDTV still-images and HDTV-resolution still-images conforming to the UHDTV wide-gamut system colorimetry. Video system We developed equipment to convert the colors of the SHV wide-gamut system colorimetry to colors in the HDTV color space, while maintaining image quality. We increased the performance of computations by using a 3D Look-Up Table LUT conversion provides better image quality than those based on RGB values by maintaining the luminance and hue of colors falling outside of the HDTV color gamut and only decreasing the chroma when mapping into the HDTV color gamut. To clarify the resolution requirements for SHV cameras, we studied the Modulation Transfer Functions MTF of various HDTV broadcast and 4K digital cinema cameras and drew up guidelines for the required resolution characteristics. Interfaces Earlier UHDTV interfaces were based on HD-SDI standards and assume a field frequency of 60 Hz frame frequency of 30 Hz, with a 4:2:2 format and 10 bits. In contrast, the SHV signal format includes a frame frequency of 120 Hz, formats from 4:2:0 to 4:4:4, and 12 bits. We developed a new 10-Gbps multi-link interface capable of handling various UHDTV signals in a unified way, including full-spec SHV signals pixels, 120- Hz frame frequency, 4:4:4 format, 12 bit and others such as pixels, 60 Hz frame frequency, and formats including 4:2:2 and 4:2:0. We also prototyped a transmitter and receiver to verify the effectiveness and practicality of this interface. 1 Recommendation ITU-R BT. 2020, Parameter values for ultra-high definition television systems for production and international programme exchange ARIB Standard STD-B56, Ultra-high definition television format studio standards 2013 Japanese NHK STRL ANNUAL REPORT

3 1.1.2 Cameras We are researching Super Hi-Vision 8K/SHV cameras toward realization of a full-specification SHV camera, including single-chip compact imaging devices, a 120-Hz framefrequency imaging method and the wide color gamut. We have also advanced development of full-specification SHV image sensors. SHV imaging research Toward the development of practical, compact cameras, we prototyped a compact single-chip camera head using a 33 mega-pixel image sensor Figure 1 1. It is comparable to current Hi-Vision broadcasting cameras, at a weight of 5 kg. We conducted a test shooting using this camera head at the Kohaku Uta Gassen year-end song festival. We also began prototyping ultra-high-pixel-count image sensors toward achieving full SHV resolution for each color with only one image sensor. We conducted field test recordings using experimental equipment able to capture at 120-Hz frame frequency using four-chip imaging with 33 mega-pixel image sensors, toward realizing a full-spec camera. We exhibited the captured images at the NHK STRL Open House, displaying them using a 120-Hz framefrequency projector. We also designed a new wider-colorgamut prism for SHV cameras based on the results of studies done in FY 2011, and prototyped a SHV camera using this prism. We studied ways to increase sensitivity, which has been an issue with current SHV cameras, using a combination of large image sensors and signal processing, and we prototyped equipment capable of operation within a theatre. Increasing sensitivity of full-specification SHV imaging devices We are developing 33-mega-pixel image sensors with 120 Hz frame frequency 120 fps, based on the full specifications of SHV international standard ITU-R Recommendation BT In FY 2012, we prototyped a 120-fps three-chip color imagecapture device using three of the image sensors prototyped in FY Figure 2 3. In imaging tests, it achieved performance of the device sensitivity of F 4.8 with a SN ratio of 51 db at a light intensity of 2000 lux. We also developed a signal processor and 4K- and 2K- resolution 120-Hz displays for this device. Motion video capture experiments showed that the video captured at a frame frequency of 120 Hz displayed less motion blur than at 60 Hz for fast-moving objects. We conducted technical presentations and demonstrations of this equipment at the NHK STRL Open House and at IBC2012 in Amsterdam, Netherlands. As the 120-Hz frame frequency is 4 times higher than that of conventinal Hi-Vision system, the light-accumulation period of the image sensor is only 1/4, therefore the sensitivity of the 120-Hz SHV image sensor is important. In FY 2011, we improved the sensitivity 2.4 times over earlier devices by reexamining the pixel structure and incorporating an on-chip micro-lens. Then in FY 2012, we newly prototyped an image sensor with the semiconductor process of 0.11μm design rule which design improved the sensitivity further by a factor of 2.5 through simulation. Full-specification single-chip-color SHV imaging devices Figure 1. Super Hi-Vision camera head < Compact prototype left and earlier device right Figure Hz Super Hi-Vision three chip color imaging device We are developing a single-chip color image sensor that captures color images on a single imaging device by having microscopic color filters on each pixel on-chip Color Filter Array, CFA of the image sensor, with the goal of realizing a compact, full-specification SHV camera. For full-specification SHV, 33 mega-pixels are needed for each of red, blue and green signal, so the overall image sensor requires over 100 mega-pixels. Operating these many pixels at a frame frequency of 120 Hz is important. In FY2012, we checked and confirmed a feasibility of high-speed operation and low power consumption of main circuits including power, AD conversion, and output interface circuits through simulation. Research on the full-specification SHV imaging devices and the single-chip color SHV imaging device was conducted in cooperation with Shizuoka University. 1 R. Funatsu, T. Yamashita, T. Soeno, T. Yanagi, Y. Takahashi, T. Yoshida : Development of Compact Camera Head for Super Hi- Vision, ITE Annual Convention 2012, in Japanese 2 K. Kitamura, T. Watabe, T. Sawamoto, T. Kosugi, T. Akahori, T. Iida, K. Isobe, T. Watanabe, H. Shimamoto, H. Ohtake, S. Aoyama, S. Kawahito, N. Egami: A 33-Megapixel 120-Frames-Per-Second 2.5-Watt CMOS Image Sensor With Column-Parallel Two-Stage Cyclic Analogto-Digital Converters, IEEE Transactions on Electron Devices, vol.59, no.12, 2012, pp K.Kitamura,T.Watabe,H.Shimamoto,H.Ohtake,S.Kawahito,T.Kosugi, T. Watanabe, T. Yanagi, H. Kikuchi, T. Yoshida, N. Egami: DE- VELOPMENT OF 33-MEGAPIXEL 120-HZ CMOS IMAGE SENSOR AND EXPERIMENTAL COLOUR CAMERA SYSTEM, International Broadcasting Convention 2012 IBC 2012 Conference NHK STRL ANNUAL REPORT 2012

4 1.1.3 Displays We are continuing our research on Super Hi-Vision 8K/SHV direct-view displays and projectors. Plasma display panels We continued to develop self-emissive, direct-view, ultrahigh-definition Plasma Display Panels PDPs for large-screen SHV displays. In FY2012, we improved the image quality and reduced the size of peripheral devices for the 145-inch diagonal full-resolution SHV PDP 1 that we developed in FY2011. A slight ununiformity in brightness of the panel occur as the size and resolution increase. We developed a front-end device that performs video signal pre-processing to compensate for this issue. High-performance PDPs utilizing this technology were used in the SHV public viewings at the London Olympics 2,promotional exhibitions and other such events 3. We also studied ways to reduce the size of this front-end device to make it more practical. We showed that it can be made 30% smaller and that a PDP display can integrated with the video signal preprocessing function. We also cooperated with Foster Electric Company, Ltd. to develop a loudspeaker array that reproduces 22.2multichannel audio for a full-resolution SHV flat-panel monitor integrating video and audio playback Figure 1. Figure inch plasma SHV display with built in speaker array High-frame-rate display technology We continued to develop displays for full-spec SHV video. In FY2012, we cooperated with JVC Kenwood Corp. to build a projector that can show 120-Hz SHV video Figure 2. Thisprojector is based on our previous 60-Hz compact SHV projector equipped with three eight-megapixel reflective LCDs and an e- shift device. The unit drives the display elements and e-shift device 120 Hz. We also began developing technologies to increase the frame rate of direct-view displays for the home and panel driver technology for hold-type displays. 1 Press release: Japanese. Figure Hz SHV projector 2 Ishii, M. Sugawara, Y. Shishikui, T. Itou: SUPER Hi-VISION Public Viewing at London 2012 Olympic Games, International Display Workshop in conjunction with Asia Display 2012, pp Murai, H. Abe, T. Oue, K. Ishii, Y. Murakami, N. Shimizu: Development of 145-in. Diagonal SUPER Hi-VISION PDPs, International Display Workshop in conjunction with Asia Display 2012, pp Coding We are researching video compression coding methods for Super Hi-Vision 8K/SHV broadcasting. Next-generation HEVC/H.265 video coding scheme We continued with our standardization work began in FY 2010 on the next-generation video coding scheme called High Efficiency Video Coding HEVC /H.265. We defined new level in the standard supporting SHV video resolutions and frame rates and a 10-bit per pixel profile main10 for high-quality broadcasts. An international standard regulating HEVC will be published in We compared the performance of HEVC with the conventional coding scheme, AVC/H.264, using various SHV video images by the common software encoder in the standardization work. We confirmed that HEVC has approximately double the performance of AVC/H.264 for more than half of the SHV videos used in the tests 1. We developed a prototype HEVC encoder supporting SHV video Figure 1. This device compresses 60-Hz SHV video using the draft HEVC main10 profile specification and outputs locally decoded video. The AVC/H.264 SHV coding equipment developed earlier partitions video spatially and encodes the segments independently, so it is not possible to perform motion or intra prediction between the partitioned video segments. This results in image degradation near segment boundaries. The current prototype HEVC encoder also partitions images, but it can perform prediction crossing segment boundaries and derives to avoid such image degradation. We accepted government-commissioned research project from the Ministry of Internal Affairs and Communications, Japan. For this project, titled The project for Resilient Information and Communication Networks The third supplementary budget of the general account, FY2011, we developed an HEVC encoder for HDTV and verified in a subjective evaluation that it has approximately double the performance of AVC/H264. This research was conducted in cooperation with Mitsubishi Electric Co. Ltd. NHK STRL ANNUAL REPORT

5 Figure 1. HEVC prototype encoder supporting SHV Reconstructive video coding Figure 2. Reconstructive video coding hardware Common platform We are researching reconstructive video coding system that improve image quality at very high compression rates by combining image reconstruction techniques and video coding methods. We prototyped hardware processors for image reduction and super-resolution reconstruction Figure 2 2 implemented upon FPGA-based common hardware platforms that support up to 4K video signals. The image reduction processor is equipped with adaptive convolution filtering cores that are controllable on a per-pixel basis for altering aliasing and sampling positions. The super-resolution processor is composed of adaptive convolution filters in parallel-series connection that are capable of controlling basis functions and edge modulators for wavelet superresolution. We assembled the image reduction processor, super-resolution processor and an HDTV AVC/H.264 codec and conducted tests of super-resolution reconstructive coding. We also investigated reconstructive video coding with framerate, quantization, and colorimetric conversions with the aid of our key technology of side-data transmission for controlling the reconstruction processor. First, we developed a method for encoding 120-Hz video with a 60-Hz codec system by alternatively dropping video frames. The receiver incorporates a multi-frame super-resolution technique for interpolating unsent frames. The encoder also tries multi-frame interpolation to indentify the optimal parameters to control the reception-side interpolation. We conducted experiments, confirming that the image quality of the interpolated frames is better than with conventional framerate conversion techniques 3. Next, we made improvements to the coding methods with bit-depth conversion developed in FY We reduced the bit depth by quantifiers that suppress noise and preserve smooth gradations in the image as much as possible. Intermediate gradations are reconstructed at the receiver by referring to the gradation values of neighboring pixels. The results of experiments show that these improvements effectively suppress false contouring artifacts. Finally, we developed methods using adaptive color conversion along color-space axes on the input video. Six signal components from two consecutive color frames are transformed using principal component analysis into new signal components suitable for coding. A video encoder compresses the principal components after the color and inter-frame de-correlations by the transformation. We confirmed that the technique is very effective on video whose luminance changes over time. 1 Y. Sugito, Y. Matsuo, A. Ichigaya, and S. Sakaida: Coding Performance Comparison between HEVC and AVC/H.264, Proc. ITE annual convention 2012, in Japanese 2 T. Misu, Y. Matsuo, S. Iwamura, S. Sakaida, and Y. Shishikui: Reconstructive Video Coding System - Broadcasting UHDTV to Homes with the Aid of Super-resolution Techniques, Proc. IEEE GCCE 2012, pp T. Misu, Y. Matsuo, S. Iwamura, K. Iguchi, and S. Sakaida, A Multi- Frame based Frame Interpolation and the Application to Video Coding, Proc. FIT 2012, No.3, I-019, pp in Japanese Sound systems providing a strong sense of presence We are conducting R&D and promoting standardization of a 22.2 multichannel sound system for use with Super Hi-Vision 8K/SHV. The multichannel sound system uses nine top layer channels, ten middle layer channels, three bottom layer channels and two Low Frequency Effects LFE channels. SHV sound system For production systems, we developed a practical reverberator, used it to implement 22-direction impulse responses measured at stadiums in Japan, and used it for SHV sound production at the London Olympics. We also developed a new mixing console able to change the sense of reverberation based on the direction of the sound source. Moreover, we added a 3D headtracking function to the 22.2 multichannel sound binaural headphone processor that we are developing in cooperation with the NHK Engineering Administration Department, to improve sound image localization. We improved the shape of the acoustic baffles on our 45 cm diameter spherical microphone, to improve its directivity, and we used it at the London Olympics. We also devised signal processing techniques to compensate for the lack of directivity resulting from the more compact microphone. In our research on spatial audio perception, we performed experiments to verify the effectiveness of the 22.2 multichannel audio system. The results showed that if the loudspeaker separation is 60 degrees or less, the arrival direction of the sound can be reproduced with the same accuracy as with a real sound source Figure 1 1. We also found that if the loudspeakers in the top layer deflect at approximately 30 degrees to the front, or the loudspeakers in the middle layer deflects at approximately 10 degrees to the rear, there is no significant deterioration of the spatial impression. 8 NHK STRL ANNUAL REPORT 2012

6 Coincidence of sound arrival directions Middle layer ear height Top layer Elevation : 45 deg. Bottom layer Elevation: 30 deg. Coincidence of sound arrival directions Medium plane 0 180deg. Diagonal direction deg. Frontal plane deg angular interval angular interval Loudspeaker spacing Loudspeaker spacing Figure 1. Relationship between loudspeaker spacing and accuracy of sound arrival directions Left: in the horizontal plane, Right: in the vertical plane Standardization At the April meeting of ITU-R, we proposed a workplan toward developing a Recommendation on advanced multichannel sound systems beyond 5.1 channel sound system. At the October meeting, we submitted a document on draft new Recommendation on advanced multichannel sound systems including 22.2 multichannel and 10.2 multichannel sound systems signed by Japan and South Korea. At the IEC, we worked on revisions to IEC , for transmitting 22.2 multichannel sound channel assignment information, as standardized in IEC62574, over digital audio interfaces. The draft was approved in TA4. At AES, we proposed a new standard for transmitting audio signals with up to 32 channels on the same physical layer as AES3. At SMPTE, we initiated a study group on 3D sound and submitted information on 22.2 multichannel sound. At MPEG, we contributed documents related to 3D audio coding, including methods for adjusting audio levels of the 22.2 multichannel sound system and requirements of 22.2 loudspeaker arrangements for subjective evaluation, and assisted to issue a call for proposals for MPEG-H 3D Audio. SHV sound transmission and reproduction We are developing a Loudspeaker Array Frame LAF integrated into display for home reproduction of 22.2 multichannel sound, and in FY2012, we conducted measurements of generated sound field in front of the LAF to confirm a wavefront generated from virtual sound sources on the screen by using wave front synthesis technology 2. The sound filed of virtual sound sources can be almost the same as from real sound sources. We alsostudiedonanerrorfunctiontoshowthatthedistortionin this wave front is frequency-dependent. Side and rear sound images can be reproduced by using signal processing technique with frontal loudspeaker array. We studied the signal processing methods in the time and frequency domains and developed a reproduction method using the multiple loudspeakers in the LAF to broaden listening area. With the LAF and two linear loudspeaker arrays upper and bottom of the screen, we studied wave front synthesis and acoustic beam-forming techniques as methods to reproduce side and rear sound images, in cooperation with the Electronics and Telecommunications Research Institute ETRI in South Korea. We also developed a method to minimize the deterioration of the frequency response and the shape of the wave surface when the sound image moves away from the screen. With this method, the sound depth control that will be necessary in the future 3D television can be realized. For reproduction of 22.2 multichannel sound with headphones, we optimized the Head-Related Transfer Function HRTF which is used to convert the 22-direction signal into a binaural signal for individual listeners. We devised a method to categorize HRTFs according to the frequency response and other factors and developed a selection method to adapt the appropriate HRTF for individual listener. We implemented this method on a portable device to reproduce 22.2 multichannel sound. To maintain compatibility with the existing broadcast systems, we developed coding methods for 22.2 multichannel sound which has a capability for decoding conventional twochannel stereo and 5.1-channel signals without full decoding of the 22.2-channel bit stream 3. Ultra-reality meter We examined methods to evaluate objectively the subjective factors such as sense of presence and emotional effect. In FY 2012, we studied changes in acoustic impressions over time. We found that when impressions correlate well with objective evaluations of the overall composition, the time series of impressions also correlated well with those of the objective evaluations. Moreover, the impression over time are synchronous with the objective evaluation even when the correlation with the evaluation of the overall composition is low. Part of this research was done under contract by the National Institute of Information and Communications Technology NICT for its project, R&D for enhanced sense-of-presence technology through innovative 3D imaging. NHK STRL ANNUAL REPORT

7 1 H. Ookubo, K. Watanabe, S. Oode: Report of studies on advanced multichannel sound system beyond 5.1 ch sound sensation of listener s envelopment and localization of phantom sound images on loudspeaker configurations with height channels, ITU, 6 C / 85-E H. Okubo, T. Sugimoto, S. Oishi, A. Ando: A Method for Reproducing Frontal Sound Field of 22.2 Multichannel Sound utilizing a Loudspeaker Array Frame, AES 133rd Convention, Convention paper A. Ando: Coding and transmission of three-dimensional sound using its spatial features, Acoust. Sci. & Tech. Vol. 33, No. 5, pp Satellite transmission technology Next-generation satellite transmission scheme We prototyped a 300 MHz-class wideband modulator and demodulator modem with a feed forward blind adaptive equalizer and performed transmission tests using a simulated satellite transponder. The tests confirmed an improvement of 4.7 db in the required C/N + Output Back Off OBO using modulation scheme - coding rate of 8PSK-3/4, a symbol rate of 247 Mbaud, and a roll-off factor of 0.1 Figure 1 1. For advanced wide-band digital satellite broadcasting, we conducted simulated satellite transmission tests with distortion compensation on the transmitter and adaptive equalization on the receiver. These measures improved the required C/N+OBO by approximately 0.2 db, 0.5 db, and 1.1 db for 8PSK-3/4, 16 APSK-3/4, and 32APSK-4/5, respectively. We also prototyped a modem for a satellite transponder with a 69-MHz transmission bandwidth, maximum symbol rate of 69 Mbaud, and roll-off factor configurable in the range from 0.0 to 1.0. We designed a Low-Density Parity-Check LDPC matrix taking bit-level error correction capability into consideration for 8 PSK set partitioning, in order to improve transmission performance. We performed a computer simulation of LDPC coding with code rate as a parameter on the first and second bit and determined the coding rate that provided the best transmission performance. We performed satellite transmission tests using the Wideband InterNetworking engineering test and Demonstration Satellite WINDS for high-capacity contribution links for Super Hi- Vision 8K, etc. We conducted single-carrier transmissions and bulk transmissions with two-adjacent-carrier Frequency Division Multiplexing FDM using 8PSK, 16APSK, and 32APSK and found that all modulations performed almost like that of interfrequency loop-back transmission. Required C/N + OBO db No equalization dB Equalization update filter coefficients set according to C/N OBO db Figure 1. OBO vs. required C/N + OBO characteristic 21-GHz-band satellite broadcasting system By giving shaped reflector to the array-fed antenna, we confirmed that it could generate the radiation pattern shaped to cover Japan needed for a 21-GHz-band satellite, even if it has only a small number of elements. We also evaluated the electrical performance of devices used to measure differences in phase and amplitude between the monitored elements. These devices can be used to monitor the radiation pattern produced. To minimize power losses in the feed wave guide, we arranged the high power amplifier next to the array feed and considered different thermal transport methods for this arrangement. We prototyped a single-device thermal test model consisting of a TWT amplifier, filter, wave guide, and other components, in order to evaluate the array feed thermal characteristics by computer simulation based on the thermal vacuum test of the prototype device. This research was conducted in cooperation with the Japan Aerospace Exploration Agency. We verified the rain attenuation occurrence rate for tenminute precipitation intervals using measure precipitation and rain attenuation values for 12- GHz-band satellite broadcasts at Tokyo from 2005 to 2010 and the Kagoshima station during 2001, 2002, Even when the 10-minute precipitation values were the same, the Kagoshima station, which has more precipitation than Tokyo over the whole year, experienced more rain attenuation. We designed the feed elements and feed circuits for use with a dual 12- and 21-GHz-band satellite broadcast receiver antenna. The four-element micro-strip array antenna had a stacked structure Figure 2, and we conducted simulations showing that it could cover both bands 2. Wideband and high-output power transponder for 21-GHz-band broadcasting satellite We are researching a wideband and high-output power transponder for satellite broadcasting of Super Hi-Vision 8K in the 21-GHz band. We designed an output filter with a 300 MHz range from 21.7 to 22.0 GHz, for an engineering model of a 21-GHz-band satellite output filter that will pass the unwanted wideband signal 12 GHz band element 21 GHz band element Slot Feed line Cross-section : stacked structure Dielectric substrate Ground plane 12 GHz band 21 GHz band Top view : 2 2 array Figure 2. Four element microstrip antenna for 12 GHz and 21 GHz bands 10 NHK STRL ANNUAL REPORT 2012

8 with low distortion, while attenuating undesired emissions in the to 22.5 GHz range, which is used for radio astronomy. For a 21 GHz-band broadcasting satellite with an array-fed imaging reflector antenna, we designed a beam-forming network,feedarray,andreflector.wealsostudiedtheon-board layout of a 32-element array-fed imaging reflector antenna This research was sponsored by the Ministry of Internal Affairs and Communications, Japan, under its project, Research & development of efficient use of frequency resource for nextgeneration satellite broadcasting system. 1 Y. Suzuki, A. Hashimoto, Y. Matsusaki, S. Tanaka, T. Kimura: Performance Evaluation of the Wide-band Modem for 21-GHz band Satellite Transmission with the Adaptive Equalizer, IEICE Society Conference, B-3-6, 2012, p.273 in Japanese 2 M. Nagasaka, S. Nakazawa, and S. Tanaka: 12/21 GHz Dual-Band Feed for Circularly Polarized Satellite Broadcasting Receiving Antenna, 2012 IEEE International Symposium on Antennas and Propagation and USNC/URSI National Radio Science Meeting, IEEE, M. Kamei, S. Tanaka, Y. Itoh: Research and Development of Next Generation Satellite Broadcasting System using 21-GHz band, 56th Conference on Space Science and Technology, 2S in Japanese Terrestrial transmission technology Large-capacity transmission for the next generation of terrestrial broadcasting We are researching large-capacity transmission technologies for terrestrial broadcasting of Super Hi-Vision 8K/SHV. InFY 2012, we adopted a concatenated code consisting of Low- Density Parity-Check LDPC code and BCH code for the error correction code, and we implemented 32k point FFT, optimizedbit-interleave and multi-dimensional interleaving, in which transmission data interleave the time and frequency directions and inter-polarizations, to the dual-polarized Multiple-Input Multiple-Output MIMO ultra-multi-level Orthogonal Frequency Required C/N db Vs. ISDB-T signal, delay time difference 61.5 μsec Vs. ISDB-T signal, delay time difference μsec Between dual-polarized MIMO signals, delay time difference 61.5 μsec Between dual-polarized MIMO signals, delay time difference μsec No interference FFT size 8k GI ratio 1/8 Carrier Modulation scheme 1024QAM FEC: LDPC r=3/4 Number of decoding iterations Delay time adjustment Delay time adjustment Interference D/U db Figure 1. Required C/N for dual polarized MIMO ultra multilevel OFDM transmission in co channel interference environment Division Multiplexing OFDM transmission test equipment. We combined two sets of test equipment with bulk transmission technology. We used them to conduct terrestrial transmission field tests of compressed SHV video over two channels in UHF band UHF 31ch, 34ch 1. We showed the actual SHV video transmission at the NHK STRL Open House. We transmitted test signals consisting of two 1 W signals of both polarization for each channel totalling four signals from a dual-polarized transmitting antenna installed on the roof of NHK STRL and received it with a dual polarized Yagi antenna installed on the roof of a building in an urban area approximately 4.2 km away. The four received RF signals were converted directly to optical signals, transmitted to NHK STRL by Radio-On-Fiber ROF transmission using wavelength division multiplexing technology, demodulated, and decoded. The transmission parameters were: number of signal points: 4096, FFT size: 32k, Guard Interval GI ratio: 1/32, and LDPC code rate: 3/4. Transmission capacity was Mbps/channel 188 byte packetized Transport Stream TS rate. We evaluated the co-channel interference characteristics between a current ISDB-T signal and a dual-polarized MIMO ultra-multilevel OFDM signal, as well as between two dual-polarized MIMO ultra-multilevel OFDM signals, by computer simulation. Results showed that in each case, when the symbol lengths of the interfering and interfered signals were the same, the interfering signal caused more degradation than in the case of interfered signal is Additive White Gaussian Noise AWGN because of the strong correlation between the Scattered Pilots SP, butthatit was possible to reduce the degradation by adjusting the relative delay Figure 1. When the interfering signal was ISDB-T, only one polarization was used, and we obtained some improvement due to multi-dimensional interleaving 2. We studied how to apply our idea of decoding the LDPC code with iteratively estimating the Log Likelihood Ratio LLR. The circuit scale for the basic Belief Propagation BP method would be very large, so we tried various methods to reduce the amount of operation and performed a computer simulation to evaluate their characteristics when decoding with iterative LLR estimation. Results showed that for an ultra-multilevel signal, when using the Offset UMP BP method in an AWGN environment, deterioration in the required C/N ratio from the basic BP method is limited to about 0.2 db. We studied a Space-Time Coding, Space-Division Multiplexing STC-SDM transmission scheme, combining dual-polarized MIMO, which is a type of SDM, with STC, as a next-generation technology for Single Frequency Networks SFN in which signals broadcast from two different transmitters are received simultaneously with no degradation. Computer simulations confirmed that the system has excellent characteristics 3 Figure 2. NHK STRL ANNUAL REPORT

9 35 With STC Without STC FFT size 32k GI ratio 1/32 Carrier modulation scheme 1024QAM FEC LDPC r=3/4 Number of decoding iterations 50 Delay time difference 10 μsec With out channel reliability information With channel reliability information Carrier modulation scheme 64QAM FEC LDPC r=3/4 Required C/N db BER D/U 0dB Delay time difference 1 μsec SFN signal D/U db Figure 2. Required C/N for dual polarized MIMO ultra multilevel OFDM transmission in SFN environment We also made improvements to experimental equipment for conducting SFN field experiments with the STC-SDM scheme. To improve the required C/N in multipath environments, we revised the LLR computations in decoding the LDPC code to reflect the noise distributions for each carrier symbol. We also continued our research on a multi-carrier modulation scheme using a lapped basis TMUX as an alternative to the OFDM transmission scheme. In FY2012, we showed the feasibility of the scheme by conducting hardware implementation and evaluated it. We evaluated the transmission characteristics in real broadcast wave propagation environments through computer simulations and confirmed that it is superior to OFDM schemes. We also proposed an estimation method of channel reliability that can be used in decoding process of error correction codes. We showed its effectiveness in a computer simulation Figure 3, and proceeded with a hardware implementation C/N db Figure 3. TMUX transmission bit error rate performance in a multipath environment 1 Murayama,M.Taguchi,T.Shitomi,S.Asakura,K.Shibuya: Technology for the next generation of digital terrestrial broadcasting - Field experiment of Super Hi-Vision transmission using two channels in the UHF band - ITE Technical Report, Vol.36, No.30, BCT , pp in Japanese 2 Asakura, K. Murayama, M. Taguchi, T. Shitomi, K. Shibuya: Transmission Technology for the next generation of digital terrestrial broadcasting - A study of co-channel interference suppression - ITE Technical Report, Vol. 36, No. 51, BCT , pp in Japanese 3 Shitomi, K. Murayama, M. Taguchi, S. Asakura, K. Shibuya: Technology for the next generation of digital terrestrial broadcasting A Study on SFN Based on STC-SDM Transmission Scheme - ITE Technical Report, Vol.36, No.51, BCT , pp in Japanese Wired transmission technology Optical transmission of uncompressed Super Hi-Vision signals We are researching a system to transmit uncompressed Super Hi-Vision 8K/SHV signals over optical fiber, with the advantage of not incurring delays or image degradation resulting from image compression. To make SHV equipment that will be used in broadcasting stations more compact, the interfaces between the equipment need to be made smaller. An effective way to achieve this is to decrease the multiplicity of wavelength multiplexing by increasing the speed of each optical signal. Accordingly, we designed the optical transmission section of an equipment interface capable of high-speed transmission at 25 Gbps per wavelength. We also used an 8B/10B transmission coding to make it easy to recover the reception signal clock and checked that 25 Gbps signals can be transmitted error-free using prototype equipment. We also researched a transmission system using Ethernet to transmit program contributions between broadcast stations. In FY2012, we designed a system that converts a full-resolution SHV signal 72 Gbps into a 100 Gbit Ethernet signal. We prototyped experimental equipment and confirmed that it operated correctly. Cable TV transmission of Super Hi-Vision signals We are researching transmission technology for distributing SHV signals to homes through cable television networks. SHV requires 16 times the amount of data as Hi-Vision HDTV, soa single digital cable television channel is not sufficient to trans- Monitor Receiver 256QAM 64QAM Spectrum Partitioning / multiplexing equipment Head end 64/256QAM modulator Private home Cable TV transmission path Cable TV operator Figure 1. Transmission of SHV over the cable distribution system using channel bonding technology 12 NHK STRL ANNUAL REPORT 2012

10 mit it. To transmit SHV using existing cable television systems, we developed a system that partitions a compressed SHV signal MPEG-2 TS and uses multiple carriers to transmit it. With current cable distribution systems, digital broadcast signals MPEG-2 TS are transmitted using 64QAM carriers. In the future, we plan to introduce 256QAM, which has higher transmission capacity than 64QAM. On the other hand, 256QAM has higher symbol density in amplitude and phase, so it is more susceptible to the effects of noise and distortion than 64QAM, and bit errors can occur more easily. Thus, to use the channel efficiently, we prototyped equipment 1 that partitions the signal and transmits it using combinations of channels with different modulation schemes 64QAM and 256QAM that fit the noise and distortion characteristics of the cable transmission channels. At a receiver, the received signals are synchronized to regenerate the MPEG-2 TS signal. We used this equipment to conduct transmission experiments on SHV cable distribution within the facilities of a cable operator in Kofu City in Yamanashi Prefecture Nihon Network Service co., ltd. 1 Y. Hakamada, N. Nakamura, K. Oyamada, T. Kurakake, and T. Kusakabe: An UHDTV Cable Television Distribution in Combinations of Multiple 64 and 256 QAM Channels IEEE ICCE2013 Vol.2, pp We held public viewings of the 2012 London Olympics in Super Hi-Vision. The viewings were conducted in cooperation with the BBC in the U.K. and Olympic Broadcasting Services OBS. Preparations and production were carried out according to the production system agreed upon by the broadcasters in FY2011. Production tasks were done at the Olympic venues in London, and editing and transmission tasks were done at BBC studios. Public viewings were held at nine locations in the U.K., U.S.A., and Japan. The NHK Sports Center, News Department and Broadcast Engineering Department handled most of the production and editing tasks, and NHK STRL handled most of the transmission tasks from BBC studios to the viewing venues as well as the screenings at the venues. A broadcaster in the U.S.A. cooperated with us in transmitting the signals to the venue in that country. In June, 2011, NHK STRL assigned a leader to handle each of the public viewing venues and a team of specialists to support them in making preparations. Because of power supply problems in the Kansai area in May, plans for the Osaka public viewing were abandoned, and a new public viewing was scheduled for the Fukushima station instead. Preparations for the transmissions began in We conducted transmission tests from London to NHK STRL in September, This was followed by loop-back transmissions between NTT Musashino and London in February, 2011, between London and NHK STRL in July, 2011, between London and Washington and from NHK STRL to three venues in Japan in September 2011, from London to Bradford and Glasgow in March, 2012, and from London to three locations in the U.K., NHK STRL, and Fureai Hall in April, The public viewings in Japan were mainly handled by NHK STRL, while related events were handled by the NHK Audience Relations & Cultural Promotions Department. The technical managers, business managers, and theater directors were appointed at the beginning of Promotional activities including on-line media news, programs, sports, notices, etc., off-line media magazines, newspaper announcements, posters, leaflets, and digital madia Web pages, NHK Net Club, Twitter, etc., were carried out by the NHK Public Relations Department and the departments involved. Preparations in the U. K. included investigating the U. K. venue candidates in April, 2011, training of the BBC staff in September, 2011, and re-testing of venues in November, NHK staff for the U.K. venues arrived at the venues approximately ten days before the start of the public viewings and operated the facilities in cooperation with the BBC staff. Regarding the International Broadcast Center IBC, weheld meetings with the OBS in order to share a temporary theatre that would also present 3D screenings. The IBC needed to set up the equipment beforehand, so in June, staff visited the sites to check the status of the equipment and communication lines for the public viewings. Preparations in the U. S. were carried out in cooperation with a U. S. broadcaster. We conducted on-site inspections during the international transmission tests in August, 2011 and in September, 2011 and decided to use a meeting room in the NBCU building for the demonstrations. After that, we decided the details of the demo and used the occasion of the international transmission tests in April, 2012 to further our preparations. We held public viewings at four venues within Japan from July 28 till August 12 Figure 1. NHK STRL, which was a distribution point for transmissions, handled the time differences between the U.K., the U.S.A., and Japan. Events were held at the NHK Studio Park, starting on July 21 and continuing throughout the Games. Many events related to the programs and for families and young people were also held at the venues in Shibuya, Akihabara, and Fukushima. We also promoted events as widely as possible through various media and business networks to link them with other programs in various genres and with NHK s Olympic broadcasts. We also Figure 1. Screening at the NHK Fureai Hall NHK STRL ANNUAL REPORT

11 1 Television with a strong sensation of reality 1.2 Three-dimensional television issued newsletters and posted information about the events on NHK Net-Club in order to attract visitors. These promotional activities helped attract over 200,000 visitors to the venues in Japan. We also held a press conference on July 28 and a private viewing by invitation on July 31. Many dignitaries visited the venues, and the events were covered by the mass media. The BBC oversaw the public viewings held in London, Bradford, and Glasgow, and OBS supervised the viewings at the IBC, with NHK providing support. Pre-event content was screened, starting on July 23. Visitors to the BBC-managed venues had to reserve admissions, whereas the IBC venues operated on a free entry and exit basis and received many guests, including broadcasters from various countries and VIPs. Promotion of events within the U.K. was mainly handled by the NHK Public Relations Department and included presentations and press events held jointly by NHK, OBS, and the BBC. The venue held in the U.S.A. was mainly operated by the U.S. broadcaster, with NHK providing support. These events were by invitation only and were attended by many VIPs from government, the media, and electronics industry. We received many comments of praise and anticipation from visitors to all of the public viewings. Visitors to the events in Japan gave a very high satisfaction rating of over 82%. One visitor commented, It felt so realistic, it was as though I was at the Olympic site! Another asked, When will you begin actual broadcasts? 1 M. Sugawara, S. Sawada, H. Fujinuma, Y. Shishikui, et al.: Super Hi-Vision at the 2012 London Olympics, SMPTE Motion Imaging Journal, vol.122, pp Three-dimensional television Integral 3D television We are researching a form of spatial imaging technology called the integral method, for realizing more natural and viewable 3D television that does not require special glasses. Integral photography IP is a technique that produces three-dimensional images by using an array of very small lenses for capture and display. In FY2012, we studied ways to improve the quality of 3 D images and 3D image reproduction. Increasing spatial reproduction quality of 3D images Generating high-quality 3D images using the integral method requires many more pixels than for ordinary 2D imaging equipment, both for recording and for display. To overcome the pixel count limitations of single imaging devices, we studied ways to build imaging equipment with more pixels using arrangements of multiple imaging devices. For capture, we prototyped recording equipment with a seven-camera array Figure 1. We found that by integrating the images from each of the cameras, we were able to expand the viewing zone the range in which the 3D image can be viewed by a factor of three in the vertical and horizontal directions. For display, we prototyped equipment consisting of two projectors and found that we could expand the viewing zone by a factor of two in the horizontal direction 1. As far as the imaging equipment goes, we had been using a lens array that was larger than the image sensor. With this configuration, a camera lens was needed to adjust the size of the optical image, in order to project the tiny optical images of the object generated by the lens array the elemental images onto the image sensor. This was a problem because it made the capture equipment larger. It also degraded the quality of the 3D images because of decreased resolution and distortion due to the extra camera lens. To address this problem, we prototyped a very fine lens array of the same size as the image sensor and integrated the two to develop a compact, high-quality imaging device Figure 2 2. Part of this research was conducted under contract with the Ministry of Internal Affairs and Communications, Japan as part of a project titled, R&D on spatial information acquisition system using multiple image sensors. We also participated in the Camera array Image sensor Lens array Lens array Figure 1. Integral recording equipment using multiple cameras Figure 2. Integral imaging device incorporating an image sensor and lens array in one unit 14 NHK STRL ANNUAL REPORT 2012

12 1 Television with a strong sensation of reality 1.2 Three-dimensional television project, R&D on proving test systems for expansion of digital museums, under contract with the Ministry of Education, Culture, Sports, Science and Technology, Japan, in which we built a prototype display to exhibit museum artifacts using integral 3D methods. Research on 3D image spatial reproduction quality In FY2012, we began studying the quality of reproduced 3D images using the integral method and conducted experiments on the accommodation response with respect to the 3D image and on the resolution of 3D images. With stereoscopic 3D methods, the vergence of the two eyes coincides at the 3D image, but the accommodation is fixed at the display surface and does not correspond to the 3D image. This is thought to be one of the causes of visual fatigue. The integral method reproduces the light-ray from the object, so in principle, vergence and accommodation should be at the same depth position, as if one is observing the actual object. To verify this theoretical characteristic, we conducted experiments in FY 2012 to evaluate the depth perception and measure the accommodation response in viewing reproduced 3D images using the integral method. The data suggests that the subjective depth perception of depth and accommodation conform to the depth position of the 3D image 3. The resolution of 3D images reproduced using the integral method depends on factors including the pitch and focal length of lenses in the lens array and the pixel pitch in the flat-panel display. Since the resolution is related to many parameters in this way, it was thought that computer simulation would be effective in evaluating resolution. Accordingly, we developed a simulation of the integral method by generating integral 3D imagesfrom3ddatasuchascomputergraphicsandthenconverting them into stereoscopic 3D images. We conducted subjective evaluations of the resolution of 3D images generated for various parameters and verified that this method is effective. 1 H. Sasaki, M. Miura and J. Arai: Integral three-dimensional display with enhanced viewing zone using MEMS laser projectors, ITE Annual Convention 2012, p in Japanese 2 J.Arai,T.Yamashita,M.Miura,H.Hiura,N.OkaichiandF.Okano: Integral imaging system with adjacent arrangement of lens-array and image sensors, IEICE General Conference D-11-5, p in Japanese 3 H.Hiura,S.Yano,M.Emoto,T.Mishina,J.Arai,K.HisatomiandY. Iwadate: A Study on Accommodation Responses in Viewing Integral Imaging, ITE Technical Report Vol.37, No.11, 3DIT2013-3, HI , pp in Japanese Generating 3D content from multi-viewpoint images We are studying methods for capturing integral 3D images of objects that are difficult to capture using optical equipments usually used well, such as very distant or very large objects. In FY2012, we studied methods to generate integral 3D images from multi-viewpoint images and using depth cameras, and we also studied multi-viewpoint robotic cameras. In FY2011, we had developed a method for generating 3D models of an object from multi-viewpoint images. In FY2012, we devised a method to integrate these 3D models over the elemental images the tiny images corresponding to the lens-array lenses. With this method, parts of the object obstructed by the foreground can be compensated for with information from the 3 D models generated from multiple viewpoints 1. Figure1 shows reconstructed integral 3D images on a display taking from different viewpoints. Starting in FY2012, we began research on methods to generate integral 3D images using depth maps from a depth camera. As part of this, we prototyped a system that converts a depth map into an integral 3D image. We also conducted experiments related to methods for calibrating depth cameras and cameras arranged in a line. Regarding the multi-viewpoint robotic camera, we added a camera self-calibration method to the system we built in FY This eliminated the need to photograph special patterns when calibrating the cameras and increases the convenience for on-site shooting. Parameters such as camera position and orientation, obtained from the self-calibration, can be used to control the orientation of multi-viewpoint robotic cameras and in the computation to generate 3D models from the multiviewpoint images 2. They are also useful in video production, enabling the multi-viewpoint robotic camera to follow objects, and enabling a time-slice video a video captured as though a Left Up Down Right Figure 1. Image reproduced on integral 3D display, generated from multiviewpoint images from 11 cameras single camera travelled a circumference around the object to be generated instantly from the video. 1 K. Hisatomi, K. Ikeya, M. Katayama and Y. Iwadate: A generation method of an integral 3D image from multiple depth images, ITE Technical Report, Vol.36, No.24, pp in Japanese 2 K. Ikeya, K. Hisatomi, M. Katayama and Y. Iwadate: Direction Control Method for Weakly-Calibrated Multiple Robot Cameras, ITE Winter Annual Conference 2012, in Japanese NHK STRL ANNUAL REPORT