Estimating PSNR in High Definition H.264/AVC Video Sequences Using Artificial Neural Networks

Transcription

1 RADIOEGIEERIG, VOL. 7, O. 3, SEPTEMBER Estiating PSR in High Definition H.64/AVC Video Sequences Using Artificial eural etworks Martin SLAIA, Václav ŘÍČÝ Dept. of Radio Electronics, Brno University of Technology, Purkyňova 8, 6 00 Brno, Czech Republic slanina@ieee.org, ricny@feec.vutbr.cz Abstract. The paper presents a video quality etric designed for the H.64/AVC codec. The etric operates directly on the encoded H.64/AVC bit strea, parses the encoding paraeters and processes the using an artificial neural network. The network is designed to estiate peak signal-to-noise ratios of the video sequence fraes, thus enabling coputation of full reference obective quality etric values without having the undistorted video aterial prior to encoding for coparison. We present the etric fraework and test its perforance for LDTV (low definition television) as well as HDTV (high definition television) video aterial. Keywords H.64/AVC, video quality, obective quality etric, HDTV, artificial neural network.. Introduction Digital video processing techniques and ethods, however advantageous and efficient, can be characterized with a coon issue concerning perforance testing and quality assurance. Let us focus only on the video copression algoriths at this tie, and study the ipact of the source coding process on the visual quality of the processed video aterial. Errors and faults appearing during the copressed video transission and/or storage will not be taken into account. The aboveentioned quality assessent issue has its roots in the fact that for digital video processing systes, it is not only the syste settings affecting the resulting video quality, but to a significant etent the actual video content plays a decisive role. Highly detailed video fraes (high spatial activity) with rapid changes over tie (high teporal activity) are typically the worst-case scenario. The word quality has been entioned in the previous paragraphs with no real eplanation of its eaning in the area of video processing. Ideally, the video quality is described and quantified using subective quality tests, typically eploying a group of huan observers to rate the quality, identify visibility of distortions, etc. Such tests are well understood, however their usability is liited due to the cubersoeness of organizing test sessions. The coon ai in video quality research is thus in replacing such subective procedures with obective easureents, i.e. ethods capable of evaluating the quality autoatically. The benchark of obective tests is coonly the correlation with subective test results. Full reference obective etrics are based on a coparative approach, as the original as well as processed (and distorted) video aterial is available. They reach fro the siplest piel-based etrics (such as the peak signal-tonoise ratio PSR) to the ore sophisticated, based on either the psychophysical approach [], [] or engineering approach [], [3]. The proble of full reference etrics is quite well understood and evaluation studies show good results. On the other hand, no reference etrics, especially for the eerging video copression standards, are still in their infancy. A typical solution for the older, especially DCT-based (discrete cosine transfor) algoriths, is in detecting typical artifacts such as blocking, blur, etc. [4]. The H.64/AVC, however, does not have such typical artifacts, as a deblocking filter is eployed at its output to adaptively sooth the block areas. Our approach is thus in eaining the encoded H.64/AVC data and ake a stateent on video quality using the encoding paraeters present in the bit strea. In this paper, a ethod capable of estiating PSR values of the encoded video fraes is presented, only relying on the inforation present in the encoded bit strea, and reoving the necessity of having the undistorted original available for coparison. A siilar proble was recently considered in [5], using a different approach and different bit strea paraeters. Section briefly describes the full reference etric whose outputs are desired by our syste the PSR, sections 3 and 4 present the fraework and design details of the no reference syste. The test settings are described in section 4 and the perforance over different video sets is discussed in section 5.

2 4 M. SLAIA, V. ŘÍČÝ, ESTIMATIG PSR I HIGH DEFIITIO H.64/AVC VIDEO SEQUECES USIG. The Full Reference Approach Aong the full reference obective quality etrics, the peak signal-to-noise ratio (PSR) holds its strong position and is quite often used although it is known to correlate poorly with the subective scores []. However, it is well understood and, above all, fairly easy to ipleent. The PSR is given by PSR = log () where is the aiu value a piel can take and is the ean squared error, defined as M T ~ [ f ( i,, k) f ( i,, k) ] =. () MT i= = k = The sybols M,, T represent the video frae width, frae height and the nuber of fraes the PSR is being coputed for, respectively, f(i,,k) and ~ f (i,,k) are the lua piel values of the original and the distorted video, respectively. A nuber of different full reference quality algoriths has been presented in the last decades. Soe of these algoriths are capable of reaching quite high correlations with the subective scores. 3. Proposed PSR Estiation Schee As noted above, the PSR estiation algorith is supposed to work only with the encoded bit strea of H.64/AVC. Let us observe what paraeters available in the encoded bit strea are likely to carry inforation on the quality of the decoded video: 3. Prediction Modes There are two groups of prediction odes in which data can be predicted in the H.64/AVC intra prediction odes and inter prediction odes. For intra odes, the prediction is done fro neighboring saples within the sae iage (slice), in the siplest case, ust copied in a selected direction. The predicted block size for intra prediction is not fied, but can be altered depending on the encoder s choice fro 6 6 piels, over 8 8 piels down to 4 4 piels as listed the first colun in Tab.. The IPCM ode enables the encoder to code the data directly with no prediction. The block size and the prediction direction is then signaled in the bit strea. Large predicted blocks are likely to be chosen in sooth areas with low spatial activity. Choosing large blocks requires fewer bits to signal prediction process to the encoder, and thus in soe situations the encoder ight choose large blocks to spare bits even though the prediction accuracy decreases. In inter prediction odes, previously encoded and decoded pictures are used as reference. Blocks are then predicted using otion copensation in either one or two directions. Again, the block size is not fied here. The available block sizes are listed in the second colun in Tab.. Even though there is a kind of hierarchy described in the standard, the block size used is a sufficient paraeter for our application. In the direct ode, the piel values are siply copied fro the reference picture. Intra prediction Intra 66 Intra 88 Intra 44 IPCM Prediction odes Inter prediction Direct 66 Inter 66 Inter 68 Inter 86 Inter 88 Inter 84 Inter 48 Inter 44 Tab.. Prediction odes and prediction block sizes available in the H.64/AVC. 3. Quantization The predicted data are copared to the original iage blocks and the differences reain to be encoded. There are several available transfor algoriths in the H.64/AVC [6]. What we are interested in is the coarseness of transfor coefficient. The standard defines a paraeter, ranging fro 0 to 5. The higher the paraeter, the coarser is the and the lower quality of the decoded video can be epected. 3.3 Syste Fraework The operation of the PSR estiating syste can be described by a flowchart displayed in Fig.. For each frae (or field if processing interlaced video), the encoding paraeters are first read fro the network abstraction layer AL [6], which takes care of proper encoded data handling for a selected application. The net step is then decoding variable length codes, which is not eplicitly entioned in the flowchart but can be included in the read block. Having the frae encoding paraeters available, the syste continues based on the frae type in case of intra frae, the frae only includes intra coded blocks as listed in the left colun in Tab.. On contrary, inter fraes ay include any block type fro Tab.. For intra fraes, the average paraeter is calculated as it can be altered throughout the frae. However, the encoder does not necessarily have to have this feature ipleented, which is also the case in our setup (see Sec. 5). The net input paraeter for the PSR evaluation is fored by the ratios of the respective prediction odes throughout the frae. The inputs are then fed to an artificial neural network (A) which outputs an esti-

3 RADIOEGIEERIG, VOL. 7, O. 3, SEPTEMBER ated PSR. At this point, PSR estiation is done for one frae. In case an inter predicted frae is being processed, the whole syste operates on the sae basis as for intra predicted fraes, with one significant difference. For inter prediction, reference fraes are used for otion copensated prediction, so the quality of the reference picture (reference PSR) naturally ipacts the quality of the predicted block or frae (estiated PSR). One additional paraeter is thus introduced for each inter prediction ode, defining the average reference PSR (see Sec. 4). START frae 4. Parsing the Bit Strea As entioned above, we need a tool to read the required H.64/AVC coded data fro the bit strea and to decode the variable length codes used by the standard. Even though a straightforward approach would be to ipleent these functions directly according to the standard, there is a software available that takes care of all this the H.64/AVC encoder/decoder reference software [7]. Its code is written in C and is freely available for use and odification. We odified the C code in order to etract the desired paraeters. As there is no need to decode the actual piel values within the fraes, a significant part of the decoder can be disabled. Intra Read frae properties fro AL Frae type? Inter 4. Reference PSR Let us now up over to the last block prior to A operation for inter predicted fraes. As noted above, we need to have an estiate of the PSR in the reference pictures, the otion copensated prediction is done fro. For each of the inter prediction odes (right colun in Tab. ) we want one nuber representing the average reference PSR coputed fro the reference pictures. Calculate average paraeter Calculate average paraeter Assue we have an area consisting of blocks having the peak signal-to-noise ratio PSR, an area of blocks with PSR, etc. The corresponding of each area can be epressed fro () as Calculate prediction ode ratios Mapping - A Store estiated PSR et frae or ED Calculate prediction ode ratios Calculate reference PSR for each prediction ode Mapping - A Fig.. Basic syste flowchart for intra and inter predicted fraes. 4. Syste Design Details This section is intended to ore thoroughly analyze soe of the syste key blocks introduced in Sec. 3. =. (3) PSR 0 Calculating overall ean squared error of the whole area using a weighted average yields + + L+ = + + L+ + + L+ PSR PSR PSR = + + L+ i PSRi i= =, = where is the total nuber of different PSR values in the eained area. Substituting the result of (4) back into () gives the overall reference PSR as PSR = log = log = i= = PSR log i PSRi i=. (4) (5)

4 6 M. SLAIA, V. ŘÍČÝ, ESTIMATIG PSR I HIGH DEFIITIO H.64/AVC VIDEO SEQUECES USIG The epression in (5) is evaluated for each prediction ode and the PSRs of the associated reference pictures. If the video is encoded in Main, Etended or High profile, the inter prediction ay be bi-directional. In such case, (5) is used first to copute one value of PSR for each predicted block as there is one reference in each direction, the is equal to. 5. Video Sequences The artificial neural networks have been trained using a set of short video sequences in CIF forat (35 88 piels). The first fraes of the training sequences are displayed in Fig., the sequences are available at [8]. The sequences were encoded using the H.64/AVC reference encoder in Main profile at Level 3.0 with seven different configurations each [7]. The altered paraeters are listed in Tab.. Setting nuber Target bitrate [kbps] Initial paraeter Fig.. Video sequences in CIF resolution used for network training. Tab.. Encoder configuration for training video sequences. To verify the PSR estiation algorith for a different set of video sequences with a different resolution, two sets were used, including video sequences in 70p HDTV resolution (80 70 piels) and in full HD resolution 80p (90 80 piels). The sequences are again freely available [9]. The variable encoder paraeters for the 70p video sequences are the sae as those for the training sequences in CIF resolution (Tab. ). As the encoding of 80p sequences is quite deanding, fewer configurations were used for this forat as listed in Tab. 3. The HDTV sequences were encoded at Main profile, but the Level had to be changed to 3. and 5.0 for the 70p and 80p sequences, respectively. Setting nuber Target bitrate [kbps] Initial paraeter Tab. 3. Encoder configuration for 80p video sequences. Fig. 3. HDTV video sequences used for evaluation and verification of network generalization ability. 6. Results Having the set of low resolution training video sequences created and desired paraeters etracted, different network configurations were trained on training set of paraeters using the least ean squares algorith with Bayesian regularization []. The intra and inter predicted fraes were treated separately, which eans different networks were designed and trained for each frae type. eural networks were

5 RADIOEGIEERIG, VOL. 7, O. 3, SEPTEMBER trained on the CIF resolution training sequences and their perforance was verified for HDTV set of video sequences. The optiization criterion for the training was the ean squared error of the real and estiated PSRs over the training set. 6. Intra Fraes For the intra fraes, there are only five input paraeters to the artificial neural network. The siplest network configuration is the linear unit, which is capable of representing any configuration of linear neurons []. Surprisingly, even such a siple configuration gives quite nice results. For the 70p video sequence test set, the linear unit reached a correlation coefficient between the real and the estiated PSR values and a ean squared error () of Anyway, as the perforance of the syste for a whole video sequence vastly depends on the estiation accuracy of the first frae in the sequence, we will use a ore coplicated network configuration in our consideration. Out of the several tested configurations, a three layer network with five units in the first layer, two units in the hidden layer and one linear unit in the output layer was selected. Such network reached a correlation of and equal to.088. with the first layer having as any units as there are inputs to the network. The nuber of units in the second (hidden) layer is arbitrary and should be selected to give best results for the desired application. The nuber of units in the third (output) layer is deterined by the desired nuber of network outputs in our case it will only have one unit as the only output we require is the estiated PSR. Tab. 4 lists the results for the hidden layer having one to si neuron units. In the coputation of of the training set, only inter fraes are considered. For and correlation coefficient of the test set, even the intra fraes are taken into account and the results represent the whole considered video sequences. It is obvious the results differ very slightly for the varying network configurations. The changes are ore likely to be caused by different initial network weight setting rather than liiting network capabilities. As we are using a regularized training algorith, the risk of overfitting is iniized and the networks are usable even when the nuber of hidden units is larger than necessarily needed. Fig. 5 shows a scatter plot diagra of the real and the estiated PSR values for the whole set of 70p testing sequences. A siilar diagra for 80p sequences is displayed in Fig. 6. In both diagras, an MLP with three units in the hidden layer was used. Hidden units CIF 70p 80p Training Corr. Corr Tab. 4. PSR Estiation results for high definition video sequences. Fig. 4. Scatter plot diagra: Real versus estiated PSR values for intra coded fraes. 70p test set, three-layer network. The scatter plot diagra of the real and estiated PSR values for the three-layer network is shown in Fig. 4. Please note that even though there are 35 copressed video sequences available in the 70p test set, there are fewer values in the diagra as for soe configurations the first fraes were encoded in the sae way. 6. Inter Fraes The PSR estiation proble sees to be ore difficult as it turned out that a siple linear neuron is not capable of reaching satisfactory results. The configuration of a ulti-layer perceptron (MLP) was thus used instead. The MLP is supposed to be ade up fro three layers, Fig. 5 Scatter plot diagra: Real versus estiated PSR values for inter coded fraes. 70p test set, three-layer network, four units in the hidden layer.

6 8 M. SLAIA, V. ŘÍČÝ, ESTIMATIG PSR I HIGH DEFIITIO H.64/AVC VIDEO SEQUECES USIG Fig. 6. Scatter plot diagra: Real versus estiated PSR values for inter coded fraes. 80p test set, three-layer network, four units in the hidden layer. otice the declination of the results fro the ideal values in the left part of Fig. 5. Obviously, there are sequences in the test set for which the algorith gives wrong outputs and the peak difference can reach up to decibels which is an unacceptable value. This issue ight be solved by a different selection of the training sequence set. The network outputs are uch ore accurate for the video sequences with high signal-to-noise ratios. References [] WIKLER, S. Digital Video Quality: Vision Models and Metrics. Chichester: Wiley, 005. [] WU, H. R., RAO, K. R. Digital Video Iage Quality and Perceptual Coding. Boca Raton: Taylor & Francis, 006. [3] WAG, Z., LU, L., BOVIK, A. C. Video quality assessent based on structural distortion easureent. Signal Processing: Iage Counication, February 004, vol. 9, no., p. -3. [4] WAG, Z., BOVIK, A. C., EVAS, B. L. Blind easureent of blocking artifacts in iages. In Proceedings of 000 International Conference on Iage Processing, vol. 3, p , 000. [5] EDE, A. o-reference estiation of the coding PSR for H.64- coded sequences. IEEE Transactions on Consuer Electronics, May 008, vol. 53, no., p [6] ITU-T Recoendation H.64, Advanced Video Coding for Generic Audiovisual Services. Geneva: The International Telecounication Union, 006. [7] SUEHRIG, K. The H.64/MPEG-4 AVC reference software JM. [online] Available: [8] Arizona State University, Video Traces Research Group. CIF Sequences. [online] Available: [9] HAGLUD, L. The SVT High Definition Multi Forat Set. Sveriges Television AB, 005 [online] Available: [] DEMUTH, H., BEALE, M. eural etwork Toolbo for Use With MATLAB. User s Guide, version 4. atick: Mathworks, Inc., Conclusion We have presented a syste capable of estiating PSR for real H.64/AVC copressed video sequences. The results show the approach is quite universal in ters of video resolution as it was designed using low resolution video sequences and with increasing resolution it still perfors reasonably well. As the syste is designed quite universally, it should be possible to train the artificial neural network to estiate different target values. Eperients with subective score estiation will follow. Acknowledgeents The research described in the paper was financially supported by the Czech Grant Agency under grant o. / 08/H07, and by the Czech Ministry of Education by the research progra MSM About Authors... Martin SLAIA was born in 98. He received his aster s degree fro the Brno University of Technology in 005. Since then, he is pursuing his Ph.D. degree at the Departent of Radio Engineering at the sae university. His research is focused on television technology, iage and video processing and video quality assessent in particular. Václav ŘÍČÝ was born in 937. He is a professor at the Departent of Radio Electronics, Brno University of Technology. His research interest includes, in particular, video and television technology, analogue and digital processing and digital easureent.