8K 240-HZ FULL-RESOLUTION HIGH-SPEED CAMERA AND SLOW-MOTION REPLAY SERVER SYSTEMS

Size: px

Start display at page:

Download "8K 240-HZ FULL-RESOLUTION HIGH-SPEED CAMERA AND SLOW-MOTION REPLAY SERVER SYSTEMS"

Augustine Farmer
5 years ago
Views:

1 8K 240-HZ FULL-RESOLUTION HIGH-SPEED CAMERA AND SLOW-MOTION REPLAY SERVER SYSTEMS R. Funatsu, T. Kajiyama, T. Yasue, K. Kikuchi, K. Tomioka, T. Nakamura, H. Okamoto, E. Miyashita and H. Shimamoto Japan Broadcasting Corporation (NHK), Japan ABSTRACT We developed an 8K 240-Hz full-resolution high-speed camera and slowmotion replay server system for the creation of sport content. The highspeed camera acquires 8K 240-Hz video through three 1.25-inch 33- megapixel 240-Hz CMOS image sensors. Signals are transmitted between a camera head and camera control unit (CCU) through a four-core optical camera cable, and the CCU then processes the sensor signal and outputs an 8K 240-Hz RGB 4:4:4 video signal. The 8K 240-Hz slow-motion replay server system can record up to 240 min of 8K 240-Hz video while simultaneously playing back an 8K 60-Hz slow-motion video. The input 8K 240-Hz video is compressed by a factor of 12 and stored on solid-state drives. The replay server offers loop-recording, live-editing, and slowmotion replay functions with a dedicated remote controller. We evaluated the 8K slow-motion video by combining the camera and slow-motion replay server system. INTRODUCTION An 8K ultra-high-definition television (UHDTV) test broadcast was launched via satellite in 2016, and this will move to a regular satellite broadcasting service in Japan on December 1st, 2018 (1). Various 8K content is required for practical broadcasting, and there is strong demand for an 8K slow-motion system, mainly for sports content creation, to capture and replay decisive moments effectively. In general, high-speed cameras used for slow-motion replay should capture high-speed video faster than the conventional broadcasting frame frequency (60-Hz), to ensure smooth motion during playback. In addition, slow-motion replay servers need to continue recording video through these high-speed cameras during video playback. Although conventional commercial high-speed cameras (2)(3) and slow-motion replay servers (4) achieve a frame frequency of 480-Hz or higher, they only support up to 4K resolution, and there are no products that currently support 8K resolution. To realize high-speed shooting and slow-motion playback at 8K resolution, we have used 8K 120-Hz cameras (5)(6) and a compression recorder (7) supporting the highest format of UHDTV (8). However, in some sports, higher shooting speeds are desirable for 8K sport production. Moreover, the 8K 120-Hz compression recorder we developed cannot record high-speed video during playback. We also tried to shoot 240-Hz slow-motion video using a full-featured 8K camera and uncompressed recorder (9). This method allowed us to obtain 240-Hz high-speed video, but the resolution in the vertical direction was reduced to half that of 8K resolution.

2 In this paper, we present an 8K 240-Hz high-speed camera and slow-motion replay server system that can capture and record 8K 240-Hz slow-motion video while simultaneously playing back 8K 60-Hz slow-motion video. By combining the camera and slow-motion replay server systems, we evaluated the image quality of the 8K slow-motion video. CONFIGURATION OF SLOW-MOTION SYSTEM The configuration of the proposed 8K slow-motion system is shown in Figure 1. The 8K 240-Hz high-speed camera consists of a camera head and camera control unit (CCU). The camera head is equipped with three newly developed 1.25-inch 33-megapixel 240-Hz CMOS image sensors (11) to capture 8K 240-Hz full-resolution video. Signals are transmitted between the camera head and the CCU by a four-core optical camera cable. The CCU outputs 8K 240-Hz RGB 4:4:4 video using two ultra-high-definition (UHD) signal/data interfaces (U-SDIs) (10). The 8K 240-Hz high-speed 8K slow-motion replay server can record 8K 240-Hz video and playback 8K 60-Hz video simultaneously. A dedicated slow-motion controller allows the server to achieve loop-recording, live-editing, and slow-motion replay functions. The server outputs 8K 60-Hz YCbCr 4:2:2 video using four 12G-SDIs. 8K 240-Hz Camera Head 4-core camera cable Slow-motion controller 8K 240-Hz CCU 8K 240-Hz video 8K 240-Hz Slow-motion Replay server U-SDI x 2 12G-SDI x 4 8K 60-Hz Replay video Figure 1 Configuration of 8K slow-motion system. 8K 240-HZ FULL-RESOLUTION HIGH-SPEED CAMERA SYSTEM 1.25-inch 240-Hz 33-megapixel CMOS image sensor To capture 8K 240-Hz video, we have developed a 1.25-inch 33-megapixel CMOS image sensor that supports 240-Hz image shooting. Figure 2 shows a block diagram of the sensor. The pixel array has a 2.1-μm pitch and a vertically shared 2.5-transistor structure. Each pixel column has duplicated source followers, a correlated double sampling (CDS) circuit, and programmable gain amplifier (PGA) circuits. The PGA circuits are connected to a multi-functional three-stage pipelined analog-to-digital converter (ADC) that has a 120-Hz low-noise mode and 240-Hz high-speed mode.

3 Timing Timing Row drivers Timing Row drivers Timing The first-stage ADC is used in the 120-Hz low-noise mode according to the folding-integration technique (12), which is skipped in the 240-Hz highspeed mode. The second-stage ADC is a Cyclic ADC that converts the higher six bits in the 120-Hz highspeed mode. The third-stage ADC is a successive approximation register (SAR) ADC that converts the lower six bits with low power consumption. In 240-Hz mode, the ADC offers 12-bit resolution. Digital readout 3-stage ADC CDS/PGA Pixel Array 2.1 μm x 2.1 μm (Effective 7680 x 4320) The ADC outputs 16-bit data at 240- CDS/PGA Hz operation (three bits from the first ADC in 120-Hz mode, 13 bits from the 3-stage ADC second and third ADCs, including a sign bit). Output signals from the 46 Digital readout columns of the ADCs are collected and converted into a serial signal of Figure 2 Block diagram of the image sensor 864 Mbps. The image sensor has 184 data channels of 864 Mbps and 12 clock channels. The aggregate data rate output from the sensor is approximately 159 Gbps. The sensor is fabricated using a 110-nm 1P4M CIS process. The conversion gain is 85 μv/e -, random noise is 4.3 e -, and power consumption is 9.8 W. Camera system configuration A block diagram of the camera system and the appearance of the camera head and CCU are shown in Figures 3 and 4, respectively. In the camera head, three monochrome image sensors capture 8K full-resolution 240-Hz images through a colour separation prism that is compliant with the wide colour gamut standardized in ITU-R Rec. BT.2020 (8). The sensor drive circuit (SNS DRIVE) supplies clocks and pulses to the sensor and receives 184-channel sensor output signals. The received signals are then converted into 16-channel 9.8 Gbps high-speed serial signals. The optical interface circuit (OPT/IF) receives 48-channel 9.8 Gbps data from three sensors and converts them into four 112 Gbps optical signals using a wavelength division multiplexing (WDM) technique; these are then transmitted to the CCU through the fourcore fibre optic camera cable. The OPT/IF circuit also receives four channels of 10 Gbps return signal from the CCU through the same camera cable using bi-directional optical signal transmission, which is also based on WDM. The CCU receives four 112 Gbps sensor signals and divides the image area into four vertical strips in the OPT/IF circuit. These areas are processed in parallel to generate an 8K 240-Hz video signal and accumulated in the buffer circuits. The accumulated frames are interleaved into odd and even frames, and are output as two 8K 120-Hz RGB 4:4:4 video signals through two U-SDIs.

Camera Head SNS DRIVE 1.25-inch CMOS (G) SNS DRIVE 864Mbps x 184 CLK/Pulse 9.8Gbps x 16 SNS DRIVE 9.8Gbps x 16 9.

4 Camera Head SNS DRIVE 1.25-inch CMOS (G) SNS DRIVE 864Mbps x 184 CLK/Pulse 9.8Gbps x 16 SNS DRIVE 9.8Gbps x Gbps x 16 3G-SDI x 4 HD-SDI x 1 OPT I/F (HEAD) RET out Sensor signals 112Gbps x 4 4-core camera cable 10Gbps x 4 VF/RET video OPT I/F (CCU) RET in Proc A Proc B Proc C Proc D 3G-SDI x 8 HD-SDI x 4 CCU BUF L BUF R U-SDI Even frames U-SDI Odd frames U-SDI even U-SDI odd Figure 3 Block diagram of 8K 240-Hz full-resolution camera system Figure 4 Images of the camera head and CCU Item Optical size Image sensor Active pixel count Frame rate Colour gamut HDR System Transmission between Head and CCU Value 1.25-inch 33-megapixel CMOS 7,680 (H) 4,320 (V) 240-Hz/120-Hz/60-Hz Rec. BT.2020 Hybrid Log-Gamma Head to CCU: 448 Gbps (1310 nm) CCU to Head: 40 Gbps (1550 nm) Signal Output U-SDI 2ch (RGB 4:4:4) Size Head:162 (W) 206 (H) 353 (D) mm CCU: 8U (19-inch rack size) Weight Head: 18 kg/ccu: 37 kg Power consumption Head: 300 W/CCU: 750 W Table 1 Specifications of the camera system

The specifications of the camera system are summarized in Table 1. The frame frequency of the 8K video signal can be switched to 120-Hz and 60-Hz by frame averaging.

8K 240-HZ SLOW-MOTION REPLAY SERVER System configuration Figure 5 shows the block diagram of the 8K 240-Hz slow-motion replay server, and Figure 6 shows the appearance of the server and controller.

Each 8K 120-Hz video is converted to YCbCr 4:2:2 format and compressed in parallel to 1/12 its original size by an intra-frame compression module.

The playback is processed by reversing the flow of the recording process at 60 Hz through a decompression module, and output through four 12G-SDIs in 8K YCbCr 4:2:2 video format.

5 The specifications of the camera system are summarized in Table 1. The frame frequency of the 8K video signal can be switched to 120-Hz and 60-Hz by frame averaging. The Hybrid Log-Gamma (13) scheme is adopted as the high dynamic range (HDR) video system. The camera head weighs 18 kg and the CCU has a 19-inch rack size of 8U. 8K 240-HZ SLOW-MOTION REPLAY SERVER System configuration Figure 5 shows the block diagram of the 8K 240-Hz slow-motion replay server, and Figure 6 shows the appearance of the server and controller. The server receives an 8K 240-Hz RGB 4:4:4 video signal through two U-SDIs as a pair of 8K 120-Hz video signals from the high-speed 8K camera. Each 8K 120-Hz video is converted to YCbCr 4:2:2 format and compressed in parallel to 1/12 its original size by an intra-frame compression module. The compressed signals are then recorded onto solidstate drives (SSDs) via an SSD array controller. The data rate of an 8K 240-Hz compressed video is 16 Gbps. The playback is processed by reversing the flow of the recording process at 60 Hz through a decompression module, and output through four 12G-SDIs in 8K YCbCr 4:2:2 video format. The SSD array can access recording and playback data simultaneously. Thus, 8K 240-Hz high-speed video is continuously recorded during playback operation. In addition, the scheme for processing video signals using 120- Hz units has the advantage that it can be expanded to record video signals at frame rates above 240-Hz. 8K 240 Hz RGB 4:4:4 U-SDI I/F (120 Hz) U-SDI I/F (120 Hz) Compression (120 Hz) Compression (120 Hz) SSD array controller SSD array controller SSDs SSDs Main board 8K 60 Hz YCC 4:2:2 12G-SDI I/F x4 (60Hz) Decompression (60 Hz) Figure 5 Configuration of 8K slow-motion system. Figure 6 Appearance of 8K slow-motion replay server and controller.

6 Specifications of slow-motion replay server The specifications of the 8K 240-Hz slow-motion replay server are listed in Table 2. The total capacity of the SSD array is 32 TB and the maximum recording time is 240 min with 1/12 video compression. The server has a 19-inch rack size of 5U. Using a dedicated slow-motion controller, the server offers instant replay, loop recording, variable replay speed control, and live editing functions. In live replay and editing, slow-motion controllers require quick-response replays. Therefore, the read latency should be as low as possible. To guarantee low-latency read operations, we benchmarked the worst possible latency under mixed read, write, and erase operations. Based on these results, the SSD parallelization number was set to 64 and the latency was less than 10 frames. Item Value Video format Input: 8K 240-Hz RGB 4:4:4 Output: 8K 60-Hz YCbCr 4:2:2 Signal input U-SDI 2 Signal output 12G-SDI 4 Compression ratio 1/12 (Intra-frame compression) Recording media/capacity SSD/32 TB Recording time 240 min SSD parallelization number 64 Replay latency Less than 10 frames Size 5U (19-inch rack size) Functions Instant replay, Loop recording, Variable replay speed, Live editing Power consumption 800 W Table 2 Specifications of the slow-motion replay server system IMAGE ACQUISITION EXPERIMENT To examine the performance of the proposed system, we implemented an image acquisition experiment. The imaging characteristics of the captured 8K 240-Hz high-speed video are listed in Table 3. The sensitivity of the camera is F2.8 at 2000 lux with a dynamic range of 450% and the signal to noise ratio (S/N) is 50 db. The modulation transfer function (MTF) for the luminance signal at 3200 TV lines and 4300 TV lines was measured to be 12% and 5% at F4.0, respectively, and the camera response is at the Nyquist frequency. Figure 7 compares the motion blur of 8K videos at frame frequencies of 240 Hz and 60 Hz. The magnified images are cropped from part of the 8K image, and the 60-Hz images are generated by frame averaging. From the magnified images, it can be seen that the 240-Hz video has much less motion blur than the 60-Hz 8K video.

Item Value Sensitivity F2.8/2000 lux S/N 50 db Dynamic range 450% MTF (F4.

240-Hz and 60-Hz videos. We have developed an 8K 240-Hz full-resolution high-speed camera and slow-motion replay server system.

The slow-motion replay server can record 8K 240-Hz video for 240 min and simultaneously play back 8K 60-Hz slow-motion video by optimizing the SSD parallelization number.

7 Item Value Sensitivity F2.8/2000 lux S/N 50 db Dynamic range 450% MTF (F4.0) 3200 TVL: 12% 4320 TVL: 5% Table 3 Imaging characteristics of 8K 240-Hz high-speed video Full-size images Magnified images 240-Hz 60-Hz CONCLUSIONS Figure 7 Comparison of motion blur between 240-Hz and 60-Hz videos. We have developed an 8K 240-Hz full-resolution high-speed camera and slow-motion replay server system. To capture an 8K 240-Hz full-resolution video, the camera is equipped with three newly developed high-speed CMOS image sensors, high-speed optical transceivers, and high-speed signal processors. The slow-motion replay server can record 8K 240-Hz video for 240 min and simultaneously play back 8K 60-Hz slow-motion video by optimizing the SSD parallelization number. The motion blur apparent in the 8K 240-Hz video is highly improved compared with that of conventional 8K 120-Hz and 60-Hz video, and a resolution of over 4000 TV lines can be achieved. In the future, we will utilize the slow-motion system developed in this study for various program productions and attempt to enhance the frame frequency. REFERENCES 1. Y. Nishida, A. Ichigaya, T. Yamashita, S. Aoki, K. Masuhara and T. Ikeda, Launching 8K/4K high dynamic range television (HDR-TV) broadcasting NAB BEC Proceedings. pp. 259 to

8 K. Kitamura, T. Yasue, T. Soeno and H. Shimamoto, Full-specification 8K Camera System NAB BEC Proceedings. pp. 266 to H. Shimamoto, T. Yasue, K. Kitamura, T. Watanabe, N. Egami, S. Kawahito, T. Kosugi, T. Watanabe and T. Tsukamoto, A Compact 120 Frames/sec UHDTV2 Camera with 35mm PL Mount Lens. SMPTE Motion Imaging Journal. Vol. 123, 4, May pp. 21 to K. Kikuchi, T. Kajiyama and E. Miyashita, K Super Hi-Vision Compression Recorder Using Compact Bit-rate Control Method. Proceedings of International Workshop on Advanced Image Technology 2016 (IWAT 2016). IC ITU-R Recommendation BT Parameter Values for Ultra-High-Definition Television Systems for Production and International Program Exchange. 9. T. Yamazaki, K. Kitamura, T. Yasue, R. Funatsu, T. Kajiyama, K. Ogura, K. Kikuchi, E. Miyashita and H. Shimamoto, fps slow motion shooting experiment using fullfeatured 8K camera and recorder. Proceedings of 31st International Congress on Highspeed Imaging and Photonics. 1A-P02, pp.744 to ITU-R Recommendation BT Real-Time Serial Digital Interfaces for UHDTV Signals. 11. T. Yasue, K. Tomioka, R. Funatsu, T. Nakamura, T. Yamasaki, H. Shimamoto, T. Kosugi, S.-W. Jun, T. Watanabe, M. Nagase, T. Kitajima, S. Aoyama and S. Kawahito, A 2.1μm 33Mpixel CMOS Imager with Multi-Functional 3-Stage Pipeline ADC for 480fps High-Speed Mode and 120fps Low-Noise Mode IEEE International Solid- State Circuits Conference Digest of Technical papers. DS pp.90 to M.-W. Seo, S.-H. Suh, T. Iida, T. Takasawa, K. Isobe, T. Watanabe, S. Itoh, K. Yasutomi and S. Kawahito, A Low-Noise High Intrascene Dynamic Range CMOS Image Sensor with a 13 to 19b Variable-Resolution Column-Parallel Folding- Integration/Cyclic ADC. IEEE Journal of Solid-State Circuits. Vol.47, No , pp.272 to ITU-R Recommendation BT Image parameter values for high dynamic range television for use in production and international programme exchange.

Development of Program Production System for Full-Featured 8K Super Hi-Vision

Development of Program Production System for Full-Featured 8K Super Hi-Vision Daiichi Koide, Jun Yonai, Yoshitaka Ikeda, Tetsuya Hayashida, Yoshiro Takiguchi, and Yukihiro Nishida Test satellite broadcasting