High Precision and High Speed TV Picture Quality Enhancement Method based on Compactly Supported Sampling Function Heeburm RYU, Koji NAKAMURA and Kazuo TORAICHI TARA Center, University of Tsukuba, 1-1-1 Tennodai, Tsukuba, Ibaraki 305-8577, Japan Paul W.H.KWAN CREST Program, Japan Science & Technology Agency YasuoMOROOKAandKoichiWADA Graduate School of Systems and Information Engineering, University of Tsukuba ABSTRACT In this paper, a method for high precision and high speed TV picture quality enhancement based on an application of the Fluency Information Theory is proposed. We have introduced the fluency DAC using C-type fluency sampling function (or compactly supported) that has received. For high speed and high precision processing of TV picture, several essential requirements were identified. The new inpulse response satisfying these requirements is introduced for only NTSC signal based on C-type fluency sampling function. The NTSC image signal is interporated in real time to obtain high precision image. Finally, an image signal that has doubled the horizontal resolution is sent as output to cathoderay tube. Keywords: High speed, High precision, TV picture quality enhancement, Fluency sampling functions 1 Introduction In our earlier research, we have developed a method for improving the quality of video hardcopy image[1]. The objective was to obtain from the video data still images that have sufficiently high quality to be included as photographs in an archived photo album. In order to obtain still images that have the same level of quality as the moving picture, it was necessary to increase the resolution of the moving picture by 1.5 times. This decision was based on an experimental study in vision physiology. The end result was that an interpolation technique for improving the quality of still image to the same level as the original moving picture was proposed. Specifically, to achieve our objective, an impulse response tailored for moving picture was first developed which was then convolved with the input image to obtain the output image. As verification, the output images used in our experiments were checked by subjective evaluation. However, the impulse response used in this earlier research exhibited an infinite attenuation in the time domain. As a result, the problem that error was introduced due to truncation was unavoidable, making it difficult to adapt the impulse response for high speed processing. In recent years, we proposed and developed an impulse response that is suitable for DVD-Audio that has a maximum sampling rate of 192KHz[2]. We named this impulse response C-type (or compactly supported) Fluency sampling function. Because of its compactly supported property, C-type sampling function is capable of high accuracy and high speed processing. In the frequency domain, it has characteristics that are comparable to the impulse response proposed in the earlier research. Practically, DVD-Audio players equipped with the D/A converters developed based on C-type Fluency sampling function have been commercialized. By these DVD-Audio players, a number of awards have been received including The Audion Excellence Award 2001. Furthermore, we have performed research on high speed and high precision TV picture quality enhancement algorithm based on the interpolation approach. Unlike the DVD-Audio signal, TV signal processing involves 2-dimensional signal which incurs an extraordinary amount of data. For high speed and high precision processing of TV picture, the following requirements were checked: (1) Specification in the time domain, (2) Specification in the frequency domain. In this paper, a new impulse response satisfying the requirements and suitable for interpolation processing is derived. The new impulse response is derived for reducing the cost of computation based on C-type Fluency sampling function. The proposal technique focused on the relationship between a C-type fluency sampling function and its derivatives.
After deriving the new impulse response, the algorithm of convolution processing between the TV picture signal and the impulse response is proposed. In the algorithm, horizontal sampling is performed for every pixel in the input image. The input image is decomposed into a RGB signal, and the digitally sampled input image is then processed by the convolution process. By convolving the input image with the newly proposed impulse response based on the C-type Fluency sampling function, the horizontal resolution of TV picture image is doubled. Finally, D/A conversion is carried out. The image signal which has double the horizontal resolution horizontal is send as output to cathode-ray tube. In the chapter 2, the preliminaries of fluency information theory is explained. Also new impluse response only for TV picture is derived. The chapter 3 described the enhancement algorithm of TV picture quality. The subjective evaluation of proposal technique is performed at the chapter 4. 2 Preliminaries In this chapter, the preliminaries of fluency information theory are briefly explained. Thereafter, the problem encountered when the conventional sampling function is being used in the TV picture quality enhancement with 2-dimensional signal will be clearly described. 2.1 Overview of the C-type Fluency Sampling Function In this section, signal spaces that the fluency information theory deals with and the sampling functions that characterize these signal spaces are explained. The space of all signals that signal processing dealt with in general exists in the open infinite interval (, ) and is considered a sub-space of the typical Hilbert space, L 2 (R), ½ Z ¾ u u(t) 2 dt <+, (1) with the inner product expressed as (u, v) L2, Z u(t)v(t)dt. (2) Here, R denotes the set of all real numbers. Given a natural number m andanintegerτ, the signal space dealt with in this paper is definedinterms of independent function systems of B-spline functon of degree (m 1) with only (m 2) times continuous Figure 1: C-type fluency sampling function differentiability, Z µ m m sin πfτ [b] ψ(t), e j2πft df, (3) πfτ m =1, 2, 3,..., obtained by shifting them on the time axis[3]. When m =1, m [b] ψ(t) is a rectangular function that is noncontinuous but non-differentiable at t = ±τ/2. When m =2, m [b] ψ(t) is triangular function that is continuous but non-differentiable at t = ±τ. We proposed C-type fluency sampling function which can carry out high speed and high accuracy calculation[2]. It is proposed for DVD-Audio with maximum sampling rate of 192 KHz. The sampling function with compact support is represented as linear combination of function systems composed of 2 degree. C-type function which are m = 3 class is denoted as follows: 3 [c] ψ(t) = 1 3 [b] 2 ψ(t +1/2) + 23 [b] ψ(t) 1 3 [b] ψ(t 1/2) (4) 2 Fig.1 shows the C-type fluency sampling function with compact supported. The 3 [c] ψ(t) has next features. The function has compact support which converges to 0 at the left and right 2 nd sample ψ(t) =0,for t 2 3 [c] The function is symmetrical on the center of t =0. 3 [c] ψ(t) =3 [c] ψ( t) Only one time continuously differentiable In this section, the signal space dealt by the Fluency information theory is overviewed. Also C-type fluency sampling functions is introduced and its characteristics are explained. 2.2 The Adaption of C-type Fluency Sampling Function to TV Picture As described previous section, we have proposed the sampling function, named C-type fluency sampling
function. Reproduction of high quality DVD-Audio signal which has 192KHz frequency range can be performed using the C-type function. C-type has the feature which can perform high speed processing and quality processing. Based on the C-type fluency sampling function, we haveperformedresearchonhighspeedandhighprecision TV picture quality enhancement algorithm based on the interpolation approach. Because C-type is compactly supported, the high speed operation is possible. TV picture signal is 2-dimensional signal. This means that needed the extraordinary amount of information unlike 1-dimensional signal like DVD-Audio or CD signal. In proportion to the amount of information, the amount of operations also increases extraordinarily. Therefore, C-type may not be of use at processing speed. At the case of C-type, some problems may arise on the side of calculation speed. In consideration of processing speed, it is necessary to add an improvement from C-type. On the other hand, human subjective impression of image quality is highly influenced by edge information. The edge information is a portion which changes abruptly in an image. When the edge information is not expressed clearly, people do not feel the impression of high precision from an image. In other words, people acquire a good impression in the image when the edge information is expressed finely. 2.2.1 Specification of time domain and frequency domain In the previous section, we described the problem which occurs when C-type is applied to TV picture. This section shows the specification which solve the problem. The specification of time domain and frequency domain is examined at same time. Specification in the time domain The TV picture signal from an NTSC system is processing in 30 frams/sec. This means that onescanlinemustbeprocessedinnomorethan 63.5ms. Therefore, in order to improve the picture image quality in real time, it is required that very high speed processing should be possible. Specification in the frequency domain Human subjective impression of image quality is highly influenced by edge information. Since edges in the image exhibit an abrupt change in spatial frequencies, in order to obtain high precision image by interpolation, the interpolation function used must be able to reconstruct the high frequency componets accurately. TV picture signal has about 4.2MHz frequency band. The twice Figure 2: Impulse response as much of band is necessary to apply high precision processing to the TV picture signal. This subsection is described the examination result about specification of time domain and frequency domain when C-type is applied to TV picture signal. 2.2.2 The derive of impulse response especially for TV The new impulse response satisfying the above requirements and suitable for interpolation processing is derived. As stated above, TV picture signal has 4.2 MHz frequency domain. This means 30 frames must be processed in a second. When C-type is applied to interpolation process in TV picture signal, the new impulse response is added an improvement based on C-type fluency sampling function. To reduce calculation time, the proposal technique focused on the relationship between a C-type fluency sampling function and its derivatives. The 1 st derivative function is calculated when C-type is differentiated one time. This function become m = 2 class function which we define. C-type becomes the function of a stairs form which we say m = 1 class function, by performing the second degree differentiation. From now, the reverse operation, i.e. integration, is considered. The 2 nd dervative function is the class of Walsh function. If Walsh class function integrates with this function once, it will become the function of a polygon class function. Furthermore, when one more integration is carried out, there is a relation from which C-type is obtained. The proposal technique observed the correspondence relation. The integration performed in analog space corresponds to addition in digital space. The impulse response only for TV picture signals is introduced by digital addition. The procedure of introduction begins from the function of m = 1 class function. The coefficients of m =1 class function are 1, 3, 5, 7, 7, 5, 3, 1. This coefficient sequence is shifted to the right. The digital
addition is performed as below: µ 1, 3, 5, 7, 7, 5, 3, 1 1 2, 7, 0, 7, 2, 1, 0, Consequently, the number sequence of -1,2,7,0,-7,- 2,1,0 is obtained. This sequence approximates the triangular function which performed the 1 st derivative in an analog. Furthermore, digital addition is performed to approximate the C-type fluency sampling function. µ 1, 2, 7, 0, 7, 2, 1, 0 1 1, 8, 8, 1, 1, 0, 0, The result of 2 times digital addition becomes 1, 1, 8, 8, 1, 1, 0, 0 Fig.2 shows the number sequency derived by digital integration. The drawn impulse response is a square waveform. If this response is used for convolution processing, the interpolation processing will become possible only integrally and high speed calculation can be performed. Moreover as the number shows, 8, 1, 1 is the number sequence which do not need multiplication. Since the number sequence is consisted in the power of 2, in a computer, it can calculate by shift operation and improvement in the speed of operation can be realized further. Z transform is performed and the frequency feature is examined. The number sequency has frequency feature as Fig.3. The width of main band till 3dB is 0.249f s to sampling frequency. The proposal technique set to one of the targets to express a high frequency components correctly in order to obtain a clear picture. If the conventional digital filter exceeds 3dB as an basis, energy of frequency will be cut rapidly. However, the proposal technique has energy also in the range exceeding 3dB. That is, it turns out that it is suitable for expressing a high frequency components as the purpose. In this chapter, it outlined about the fundamental feature which fluency sampling function and its function have. The problem generated when using C-type fluency sampling function for highly precision processing of TV picture signal was described. According to Figure 3: Frequency feature the problem, the examination of demand specification was performed. C-type was improved in order to fulfill demand specification. Finally, the impulse response only for TV picture signal was derived. 3 High Precision Process of TV Picture This chapter describes the algorithm which obtains a high precision TV picture signal by the derived impulse response. 3.1 The procedure of the system This section explains the procedure from an overall viewpoint. The NTSC signal becomes the one frame, when the scan of the even number field and the odd number field is completed. That is, the frame consists of the two fields. The NTSC signal from common TV, Video or DVD output is made into the incoming signal. After deriving the new impulse response, the algorithm of convolution processing between the TV picture signal and the impulse response is proposed. In the algorithm, horizontal sampling is performed for every pixel in the input image. The input image is decomposed into a R G B signal, and the digitally sampled input image is then processed by the convolution process. By convolving the input image with the newly proposed impulse response based on the C-type Fluency sampling function, the horizontal resolution of TV picture image is doubled. Finally, D/A conversion is carried out. The digital image is changed into an analog signal based on NTSC standard. The image signal which has double the horizontal resolution is send as output to cathoderay tube. The next section describes the detailed algorithm of the high precision processing of TV picture signal by interpolation processing. 3.2 High precision process by proposal impulse response By convolution process, the horizontal resolution of TV picture is raised twice. High precision process is realized by the process. The High precison process is consisted by Step 1 Step 3 (Step 1 ) 8 pixels which adjoins in the horizontal direction, the perpendicular direction, and the direction of slant of a target pixel are used.
Figure 4: High precision process (Step 2 ) Four pixels are newly interpolated around a target pixel. (Step 3 ) Standardization based on NTSC signal. As shown by Fig.4, Step 1 use eight pixels which adjoin a target pixel for every field. P 0 in Fig.4 shows the target pixel, and P 1 P 8 expresses adjoining 8 pixels. In order to process for every frame, the scanning line of n, n +2, and n + 4 is used at same process. In Step 2, input siginal is interpolated using by derived new impulse response. In 1 In 4 of Fig.4 corresponds to newly interpolated pixels by convolution processing. Four new pixels are generated to a target pixel by process of (Step 3 ), and resolution increases by 4 times. The result of interpolation, each frame has twice numbers of scanning lines. However, one frame is constituted from the 525 scanning line in the NTSC system.anaverageistakeninthedirectionofavertical axis, and standard processing is performed. Finally the signal from which the output doubled only at the horizontal axis is outputted by D/A. In this chapter, the algorithm of interpolation using a target pixel and neighboring input data is proposed. The new impulse resuponse is interduced to the improvement of a TV picture signal by convolution. 4 The Experiment Result To improvement of TV picture signal, the interpolation by convolution using new impulse response is performed, and the quality of image is evaluated. 4.1 The simulation result to digital still image ISO standard still images were introduced for simulation. The original image is enlarged by conventional interpolation technique and the proposed algorithm. As for an input signal, interpolation processing is performed horizontal direction. Therefore, unlike TV picture of the NTSC system which is an analog signal, output signal has double length horizontally as simulation result. Fig.5(a) is the partial image of the simulation result by the proposal technique. Fig.5(b) is the parital image by conventional technique(bi-cubic). In the case of Bi-Cubic, expression of a portion with a sharp change such as a orange and a carpet, cannot be reconstructed clearly. Since the energy of high frequency components will be cut, the part with a abrupt change is not clear. The image reconstructed by the proposal technique looks clearer than the one by the conventional method. One scanning line of a middle from each simulation image was taken and analyzed. Fig.5(c) shows the spectrum of the proposal technique and the conventional technique. The both spectrums shows similar distribution in low frequency domain. However, when frequency becomes high, it turns out that the proposal technique cotained much energy compared with the conventional technique. The experimental results confirm that the proposed technique is able to reproduce higher frequency componentswhencomparedwiththeconventionalinterpolation technique. Visually, the proposal technique produces clearer picture when compared with that of the conventional method. 4.2 The evaluation of analog TV picture Evaluation of actual TV picture performed by subjective evaluation was also performed. The subjective evaluation was performed based on variation II recommended by ITU-R BT-500. The evaluation is experimented for 23 persons who do not know the proposal technique. The conditions of evaluation are as follows. Four scenes(1.live, 2.movie, 3.soccer, 4.scenery) were prepared. Each scean was shown 4 times repeated during each 10 seconds. Con(=conventional) Pro(Proposal) + Con Pro The shown order of Con and Pro was randomly selected. Fig.6(a) was the answer to Is it precision compared with previous picture?. Fig.6(b) was Which one do you like?. The vertical axis shows the number of people. It turns out to four scenes that almost persons have answered that the proposal technique is high
(a)proposal technique (b)bi-cubic (c)frequency domain Figure 5: Result of Simulation (a) (b) Figure 6: The evaluation result precision and good. As the result of subjective evaluation, we could confirm the validity of the proposal technique. 5 Conclusion In this paper, high precision and high speed TV picture quality enhancement process based on the Fluency Information Theory was proposed. Specifications in the time domain and frequency domain were examined when the interpolation approach is applied to enhance the TV picture quality. In the proposal technique, to fulfill the specifications, improvement was added to the C-type fluency sampling function. The proposal technique focused on the relationship between a C-type fluency sampling function and its derivatives. The new impulse response only for TV picture signal is derived based on the C- type. The new impulse response is consisted of the integer sequence. Since the impulse response is constructed by power of 2, in a computer, shift operation is carried out instead of multiplication. Furthermore, high speed operation is attained. The proposed technique was able to reproduce higher frequency components when compared with the conventional interpolation technique. Visually, the proposal technique produced clearer picture when compared with that of the conventional method. Finally, by the subject evaluation, the validity of proposal technique was confirmed. Acknowledgemtnt This research was partially supported by the grant of the Core Research for Evolutional Science and Technology (CREST) Program under the Japan Science and Technology Agency (JST), and the competitive research fund of R& D support scheme for funding selected IT proposals from the Ministry of Public Management, Home affairs, Posts and Telecommunications. The authors would like to acknowledge here these organizations. References [1] K.Toraichi, M. Kamada, S.Ishiuchi, S.Yang and R.Mori, Improvement of Video Hardcopy Image Quality by Using Spline Interpolation IEICE,Vol.J71-D,No.7,pp.1276-1285(1988) [2] K. Toraichi and K. Nakamura, Sampling Function of Degree 2 for DVD-Audio, Trans.IEE,Vol.123-C,No.5,pp.928-937(2003). [3] M.Kamada K.Toraichi and R.Mori, Periodic spline orthonormal bases J.Approx.Theory Vol.55 pp.27-38 (1988)