Image-Based Synchronization in Mobile TV for a Multi-Broadcast-Receiver

Transcription

1 Image-Based Synchronization in Mobile TV for a Multi-Broadcast-Receiver Tobias Tröger 1, Henning Heiber 2, Andreas Schmitt 2 and André Kaup 1 1 Multimedia Communications and Signal Processing University of Erlangen-Nuremberg Cauerstr. 7, Erlangen, Germany {troeger, kaup}@lnt.de Abstract In this paper we propose a novel technique for imagebased synchronization of two mobile received TV sequences having different spatial resolutions, deviant image qualities, and possibly lost image blocks. The sequences are aligned spatially and temporally based on an affine transform. The optimal transformation parameters are determined numerically by the Levenberg- Marquardt-Algorithm due to complexity reasons. The presented algorithm leads to accurate synchronization of mobile TV and is robust even in case of image cropping and arbitrarily shaped error patterns. Keywords DVB-H, DVB-T, Mobile TV, Multi-Broadcast Reception, Synchronization, T-DMB INTRODUCTION Currently, more and more TV programs are broadcasted simultaneously based on terrestrial transmission standards like DVB-T, DVB-H, or T-DMB. Digital Video Broadcasting - Terrestrial (DVB-T) provides terrestrial broadcasting of SDTV or HDTV video, audio streams, and data. The DVB-T technique was developed for stationary and portable reception. Although mobile reception was not aimed at by the standard originally, mobile DVB-T receivers are available in the meantime. Based on the internet protocol, low-resolution digital multimedia can be transmitted by Digital Video Broadcasting - Handheld (DVB-H). DVB-H is an adapted version of the DVB-T technique which is especially optimized for mobile reception conditions. Terrestrial Digital Multimedia Broadcasting (T-DMB) is an extension of the well-known Digital Audio Broadcasting - Standard (DAB) providing terrestrial transmission of digital lowresolution TV signals. Perfect temporal and spatial synchronization of video sequences is required for various multimedia applications like multi-view video coding [1] or inter-sequence error concealment of TV signals [2]. In this paper, our 2 Development Infotainment Audi AG Ingolstadt, Germany {henning.heiber, andreas.schmitt}@audi.de focus is on synchronization of mobile TV sequences received in a multi-broadcast scenario. Fig. 1 shows such a scenario for simultaneous reception of DVB-T and T- DMB signals with a car-mounted device. Assuming multi-broadcast reception of DVB-T, DVB-H and T-DMB signals, redundant TV programs having similar image content are available on different broadcast channels. This redundancy offers new possibilities to enhance robustness of mobile TV reception. In particular, errors which occur in TV broadcasts being transmitted over terrestrial channels can be concealed by pixel-wise combination of corresponding TV sequences. Assuming a prospective multi-broadcast-receiver, however, these TV sequences are usually shifted in time against each other as they are transmitted according to different broadcasting standards. In detail, the processing times of source coding, error protection and packetization differ for instance. Further aspects are buffer sizes and burst-wise transmission. Summing up, the delay is hard to predict in advance for a given reception scenario. As a-priori information about the delay between corresponding TV programs is not available at a future multibroadcast-receiver (MBR), it has to be determined periodically for successful synchronization. Due to the deviant broadcasting standards, an image-based approach seems natural. In the following, some aspects are listed which have to be taken into account for image-based synchronization of mobile TV: First, packet errors usually occur while broadcasting digital TV to mobile receivers over terrestrial channels. As a consequence of block-based video coding, blocks, slices or frames are finally distorted in the decoding process (see Fig 1.). Image-based synchronization therefore has to cope with lost image information. Second, TV sequences broadcasted according to the mentioned standards typically have different image resolutions. Taking into account that DVB-H and T- DMB sequences are shown on mobile receivers Figure 1: Multi-broadcast reception scenario of mobile TV

2 . DVB-H or both DVB-T and T-DMB. The algorithm is generally introduced for a high-resolution sequence (HRS) and a corresponding low-resolution sequence (LRS) having similar image content but identical frame rates. In contrast to the HRS, the LRS is being considered error-free due to higher error protection. Based on a future realistic multi-broadcast reception scenario, we assume the LRS being delayed in comparison to the HRS. Figure 2: Image cropping in mobile TV with small displays, both transmission standards usually apply low spatial image resolutions like QVGA or CIF (see Fig. 1). As DVB-T supports higher spatial image resolutions like SDTV and HDTV, however, the resolutions have to be matched for image-based synchronization of mobile TV. Third, the particular image qualities of mobile received TV sequences vary significantly with the given data rate. DVB-T sequences usually have high image quality. However, DVB-H and T-DMB sequences are typically coded with a low bit rate and therefore have moderate image quality. As a consequence, imagebased synchronization must be robust against coding artifacts. Last, image cropping has to be considered for imagebased synchronization of mobile TV. Compared to DVB-T signals, DVB-H and T-DMB sequences have been possibly cropped at the image margins by the sender-site video generation process. Fig. 2 shows a resized and cropped image of the sequence crew on the left. The lost parts are marked red in the corresponding image on the right, which is just resized. Cropped TV sequences do not necessarily contain identical but similar images in a multi-broadcast reception scenario. Therefore, the synchronization method has to cope with cropped image parts. In general, synchronization of video sequences is a wellknown problem which has been studied in literature in recent years. In 2004, Tuytelaars and Gool presented a technique for the synchronization of arbitrarily moving cameras and general 3D scenes [3]. In the same year, Yan and Pollefeys proposed another synchronization method which is based on space-time interest points [4]. Whitehead et al. introduced a formalization of the video synchronization problem in 2005 [5]. One year later, Wolf and Zomet presented a synchronization algorithm in the context of reconstructing non-rigid scenes from two nonsynchronized but fixed video cameras [6]. Also in 2006, Ukrainitz and Irani described the synchronization of two video sequences containing a dynamic scene as finding the spatial temporal coordinate transformation [7]. Finally, Barkowsky et al. compared matching strategies for temporal alignment of two video sequences in 2006 [8]. However, to the best of our knowledge, image-based synchronization of two mobile received TV sequences being characterized by cropped image parts, lost image blocks, different image resolutions and deviant image qualities was not considered up to the present. We therefore propose a novel technique for image-based synchronization of two mobile received TV sequences either multi-broadcasted according to both DVB-T and SYNCHRONIZATION METHOD Problem Formulation In Fig. 3, a multi-broadcasted TV sequence is shown which is received as a HRS and a corresponding delayed LRS. t 0 is the starting point of the reception, d is the delay between the HRS and the LRS given in frames. At t 0, we take the first received frame of the HRS as reference frame A 0. Based on the image transformation method described below, we compare A 0 with several candidate frames BBc of the LRS to find the most similar frame of the LRS. The candidate frames B cb lie within the search range SR = [ tb, tb + L 1]. t B indicates the point in time when the first candidate frame BBc of the LRS arrives at the receiver. L is the length of the search range SR and defines the index c [ 0, L 1] of the candidate frames B cb As we assume the LRS being delayed in our case, t B is always greater or equal t 0 : t B t 0 (1) Image Transformation The image resolutions of the HRS and LRS differ and therefore have to be aligned first. In the proposed imagebased synchronization method, this is done by an image transformation process. Considering the sender-site video generation step, the LRS can be understood as a resized and possibly cropped version of the corresponding HRS. Therefore, the desired image transformation has to cope with scaling and shifting. Figure 3: Multi-broadcast reception scenario of mobile TV comprising a HRS and a delayed LRS

3 An image transformation capable of this is the affine transform introduced in [9]. It can handle translation, rotation, non-uniform as well as uniform scaling and shear. In general, an arbitrary vector x is mapped on a vector y by the affine transform: y = Ax + b (2) In our case, an arbitrary image B(r, of the LRS shall be mapped onto the corresponding image A(m,n) of the HRS, where r {1,...,R}, s {1,...,S}, m {1,...,M} and n {1,...,N} depict the pixel positions (M > R, N > S), respectively. Then, (2) can be rewritten as follows: m a1 a2 b1 r = n a3 a4 b2 s (3) As we only have to consider scaling and translation, the affine transform can be simplified by setting a 2 and a 3 to zero (a 2 = a 3 = 0). The transformation parameters a are then defined as follows: T [ a1 a4 b1 b2 a = ] (4) With respect to a given metric, the best-fitting transformation parameters have to be determined. The metric is introduced in (5) as the mean squared error (MSE) between A(m,n) and a transformed image B a ( m, n). For robustness, the mapping procedure only takes error-free samples into account which are indicated by error mask W(m,n). Image Registration We have now introduced a rule to map an arbitrary image B(r, onto an arbitrary image A(m,n) according to a given metric. To find the exact delay between the HRS and the corresponding LRS, we have to register A 0 (m,n) to the frame B( r, of the LRS which is most similar. Therefore the affine transform is performed for A0(m,n) and each candidate frame B( r, within the search range SR separately by minimizing the MSE in (5). In doing so, the image transformation method provides a minimum residual r for each frame B c ( r, of the LRS due to the given metric. B c (6) (7) Minimizing MSE(a) can be understood also as finding the best-fitting transformation parameters out of a set D for each frame B c ( r,. (7) is solved numerically as described below for each candidate frame ( r, within SR. Then, we can determine the frame which is most similar to A0(m,n) by finding the minimum residual of r( Bc ( r, ) and taking the argument. The delay d can be read off directly from the index of the resulting candidate frame ( r, : Once the correct delay d between the LRS and HRS is found, it just has to be recalculated in case of a varying delay. The synchronization of the HRS and the LRS can be finally achieved by delaying each frame of the HRS according to d. Amongst others, the overall complexity of the proposed image-based synchronization method is significantly determined by the number of candidate images in the image registration step. Therefore, the length L of search range SR should be chosen reasonably small and t B reasonably close to the actual delay due to prior knowledge. Additionally, several enhancements of the algorithm can be applied to reduce complexity besides simply shortening L. First, the number of candidate frames BBc(r, within the search range SR can be restricted by a feature-based selection prior to the synchronization step. Second, the affine transform can only be performed for characteristic image parts of A 0 (m,n) and B cb (r,. Optimization Technique As shown above, image registration is an important step of the proposed algorithm for synchronization of mobile TV sequences. According to (7), it is based on the minimization of MSE(a) in an affine image transformation process. As the unknown transformation parameters a lie in an unbounded set D, however, the minimization problem is computationally complex for each candidate frame B c ( r,. Therefore it seems natural to utilize a numerical optimization technique. Considering (5) and (7), the error function MSE(a) being minimized is expressed as the sum of squares of nonlinear functions. We use the Levenberg-Marquardt- Algorithm (LMA), here, as it has become a standard method for non-linear least-squares problems ([10], [11]). The LMA can be thought of as a blend of the gradient descent and Gauss-Newton method. It is a robust gradient method which converges with a high probability even for inaccurate start values. Adopting gradient methods for our algorithm, the minimum of the error function MSE(a) can be found by iteratively stepping down along the gradient of the function with step size λ. Intuitively, this leads to the parameter update of the gradient descent method where d is the negative gradient of the error function at a i. B d (8) (9) (5)

4 In general, the error function from (5) can be approximated by its Taylor series expansion where D is the Hessian matrix evaluated at a i and c denotes the function value at a i. (10) Assuming the MSE(a) being quadratic around a i, higher order higher order terms can be truncated. The corresponding rule for updating the transformation parameters a is given in [10] under consideration of gradient information and curvature of the MSE(a): (11) This Gauss-Newton iteration and the update rule for the gradient descent method given in (9) have complementary advantages which can be combined by the approach from Levenberg ([10]). (12) However, the Hessian matrix D is not taken into consideration for large λ. As a consequence, each component of the gradient is scaled according to the curvature by replacing the identity matrix I with the diagonal of the Hessian D. (13) Finally, the update rule from Levenberg and Marquardt in (13) has the desired characteristics: Low curvature leads to a large update step, high curvature results in a small update step. The LMA is iteratively performed in four steps: 1. Compute MSE(a i ) 2. Define a moderate λ 3. Update the parameter vector a i in (13) and evaluate the error function at the new parameter a i+1 4. If the error function has declined, it implies that the quadratic assumption was correct. Then, the influence of the gradient descent method has to be attenuated by decreasing λ (e.g. by a factor of 10, [10]). grown, increase λ (e.g. by a factor of 10, [10]). Go to step 3. In our algorithm for image-based synchronization of mobile TV, the LMA is executed in two stages as described in [2]. In the first stage, the LMA is applied to a subsampled version of frame A 0 (m,n). We define the start value a i for the optimization according to the LMA as ratio of the HRS and the LRS frame sizes (a i = [M/R, N/S, 0, 0] T ). The transformation parameters resulting from the first optimization stage are fairly coarse. They are taken as start values for the second optimization step, in which they are refined using the full resolution of A 0 (m,n). Performing the synchronization algorithm for consecutive frames A(m,n), temporal outliers of the transformation parameters a may be discarded by limiting the maximum deviation from the temporal mean. Outliers can occur if the degree of distortions is too high for a particular frame. EXPERIMENTAL RESULTS Simulation Setup The proposed algorithm for image-based synchronization of mobile received TV sequences was tested on the wellknown sequences crew, discovery city and shuttle. Crew is characterized by slow motion but sporadic flashlights. Discovery city contains fast motion and scene cuts. By contrast, shuttle has very little temporal activity. Some passages can be even considered as sequence of consecutive still images. The synchronization algorithm is independently performed for the first 70 frames of all three mentioned test sequences respectively. We use the YUV color space and consider only the luminance Y. Based on powerful synchronization of mobile received TV sequences in a future multi-broadcast-receiver for DVB-T, DVB-H and T-DMB, our focus is on a realistic simulation scenario. Possibly, the DVB-H and T-DMB signals are generated based on a DVB-T signal. Therefore, we assume the LRS being delayed in comparison to the HRS. Here, we simulate a constant delay d of 15 frames. The search range SR is set to [0, 20]. Due to powerful error protection schemes of DVB-H and T- DMB, the LRS is assumed to be error-free. Keeping in mind, that DVB-T was developed for stationary and portable reception, mobile adoption of DVB-T will usually lead to block and slice errors in the decoding process. Here, the HRS therefore contains randomly distributed block losses comprising 5%, 10%, 20% and 40% of each image. However, DVB-H and T-DMB sequences usually have poor image quality due to a limited data rate. We thus coded the LRS with the reference implementation of the H.264/AVC standard. The applied coding scheme is IBPBP. As the coding process of TV sequences is not part of the DVB-H or T-DMB standard, we simulate a wide quality range. The quantization parameter (QP) is varied from 15 to 45 in steps of 5. Furthermore, a GOP length of 16 is used. As DVB-T applies high image quality in general, the HRS stays uncompressed, here. Each image of the HRS has 720x576 samples, which typically is the spatial resolution of mobile TV according to the DVB-T standard. DVB-H and T-DMB signals, however, usually have format QVGA or CIF. LRS images therefore have either a size of 320x240 or 352x288 samples in our case. As mentioned before, the LRS is always a resized version of the HRS containing similar images. Possibly it is cropped at the image margins, additionally. Summing up, we consider the LRS being available in four versions in our simulations: format CIF with pure resizing (case 1), format CIF with resizing and cropping (case 2), format QVGA with pure resizing (case 3) and format QVGA with resizing and cropping (case 4). Considering the LMA, the stopping criteria for the optimization method are crucial parameters. A limited number of 100 iterations has been sufficient in our tests. Further aspects are the termination tolerance of the MSE and the termination tolerance of the affine transform parameters a which both have to be restricted reasonably. Finally, the resampling step in the optimization process has to be considered. It is performed as linear interpolation in our simulations. Some restrictions for image-based synchronization of

5 a) LRS case 1: CIF (resized) b) LRS case 2: CIF (resized & cropped) c) LRS case 3: QVGA (resized) d) LRS case 4: QVGA (resized & cropped) Figure 4: Correct detection rates of sequence crew plotted in percent against the quantization parameter of the LRS (5%, 10%, 20% and 40% of each HRS image are lost) mobile received TV sequences are introduced in this paper by the absence of frame drops and the assumption of identical frame rates for HRS and LRS. If the frame rates differ in practice, an up- or down-conversion technique has to be utilized prior to the algorithm. Performance Evaluation Depending on the application, accurate or approximate synchronization of mobile received TV sequences may be necessary. For performance evaluation of the proposed algorithm, we therefore distinguish between the correct detection rate (CDR) and the synchronization error (SE). The CDR shows the percentage of delays determined correctly in respect to the processed HRS frames. The SE is defined as mean absolute difference (MAD) in frames between the computed and the correct delay taking only inaccurate matches into account. Let us first consider exact image-based synchronization of two TV sequences. Fig. 4a-d shows the correct detection rates of sequence crew for all LRS cases. In each subfigure, the CDRs are plotted against the quantization parameter for an image loss of 5%, 10%, 20% and 40%. As can be clearly seen, the fundamental characteristics of the CDR are comparable for all considered LRS cases and are therefore depicted generally. Fig. 4a-d shows that the CDR is strongly connected with the image quality of the LRS. High image quality which is indicated by a low QP leads to a high CDR and vice versa. For the QP lying in an interval of 15 to 35, the CDR is relatively constant between 97% and 100% for all considered image loss percentages and LRS cases. Considering LRS cases 2 and 4, the CDR even is exactly 100% in the given interval. Coding crew according to H.264/AVC and a QP of 40 or 45 leads to an objective image quality of about 31 or 29 db in terms of PSNR Y. Looking at such a degraded image quality, the CDR decreases significantly. In LRS case 1 and 2, the CDR is around 50% to 60% for QP 45. Cases 3 and 4 even show a CDR of 35% to 40%. Using the LRS with resolution QVGA (case 3) and the HRS being characterized by an image loss of 20% as example (see Fig. 4c), it can be seen that the CDR is not always a strict monotonic function. Determining the minimum MSE between a frame of the HRS and a transformed candidate frame of the LRS according to (7) is not necessarily a convex problem. This aspect occurs in general due to the spatial characteristics of the considered frames and is aggravated by the presence of lost image parts and compression artifacts. Therefore, the LMA possibly converges towards a local minimum of the MSE. As a consequence, the minimum residual within search range SR may lead to a synchronization mismatch in (8). Summing up, the proposed algorithm for image-based synchronization of mobile TV can provide excellent results in terms of CDR for high and moderate image qualities. Just in case of poor image quality which is mostly not acceptable for the viewer, the CDR is not sufficient any more. Assuming a QP greater 45, the constant delay introduced by digital TV transmission is possibly varied by frame drops in the coding process. Then, a reasonable evaluation of the proposed synchronization method is not possible any more. As the LMA is a powerful optimization method which mostly converges even for inaccurate start values with a high probability, it can handle cropped image parts reliably (see Fig. 4b/d). The CDR is therefore relatively inde-

6 Table 1: Correct detection rate and synchronization error for sequences crew, discovery city and shuttle against the quantization parameter of the LRS regarding image losses of 5%, 10%, 20% and 40% and LRS cases 1-4 sequence measure QP = 15 QP = 20 QP = 25 QP = 30 QP = 35 QP = 40 QP = 45 crew CDR SE discovery city CDR SE shuttle CDR SE pendent of the applied LRS case. The same holds for the image loss percentage of the HRS. The optimization is robust against lost image blocks as it performs well even for 40% image loss. The corresponding CDR 40% differs only marginally from the CDR 5% determined for 5% image loss and is even higher in some cases (see Fig. 4b). Simulations show that the image content usually is not a crucial parameter for image-based synchronization of mobile TV. This becomes clear as the CDR is exactly or approximately 100% for high and moderate image quality in each LRS case (see Fig. 4a-d). In other words, an arbitrary frame of the HRS can be used in a future MBR for exact synchronization of the whole sequence assuming a constant delay. In case of a slightly changing delay, however, a periodic recalculation is required. As outlined before, the CDRs have a high correlation for all considered LRS cases and all image loss percentages. Therefore, a combined CDR is shown Tab. 1 for sequences crew, discovery city and shuttle regarding an image loss of 5%, 10%, 20% and 40% and LRS cases 1-4. Crew performs best in terms of CDR as the sporadic flashlights enhance exact synchronization due to the image-based algorithm. Discovery city has comparable results for high and moderate image quality. However, shuttle performs significantly weaker for all image qualities due to the low temporal activity which aggravates exact synchronization in combination with lost blocks. The SE was defined above as MAD between the determined and the correct delay only evaluated for inaccurately registered HRS frames. Tab.1 shows the SE averaged over all image loss percentages and all LRS cases. Unlike the CDR, the SE has a weak correlation with the QP indicating the image quality. Only for shuttle, the SE rises with decreasing image quality. It is further noticeable that the SE is below 4 on average for all sequences even in case of low image quality although the maximum could be 15 in our simulations. Considering crew at QP 45, the correct delay is not found in about 56% of all cases for instance. However, a mean error of 1.16 frames still might be acceptable depending on the application. SUMMARY AND CONCLUSIONS We proposed a new technique for image-based synchronization of two TV sequences which are received in a multi-broadcast scenario. The first TV sequence is characterized by lost blocks, a high spatial resolution and a high image quality. The second TV sequence is error-free and has low spatial resolution and moderate image quality typically. The presented synchronization method is based on an image transformation step which is numerically solved by an optimization method. Experimental results show that the algorithm is robust even in case of arbitrarily shaped errors and cropped image parts. REFERENCES [1] Garbas, J.-U., Fecker, U., Tröger, T., Kaup, A., 4D Scalable Multi-View Video Coding Using Disparity Compensated View Filtering and Motion Compensated Temporal Filtering, Intern. Workshop on Multimedia Signal Processing, pp , Victoria, Canada (2006). [2] Tröger, T., Heiber, H., Schmitt, A., Kaup, A., Inter- Sequence Error Concealment of High-Resolution Video Sequences in a Multi-Broadcast-Reception Scenario, Proc. Europ. Sign. Processing Conf., Lausanne, Switzerland (2008). [3] Tuytelaars, T., Van Gool, L., Synchronizing Video Sequences, Proc. IEEE Computer Society Conf. on Comp. Vision and Pattern Recog., pp , Washington, DC (2004). [4] Yan, J., Pollefeys, M., Video synchronization via space-time interest point distribution, Proc. of Advanced Concepts for Intelligent Vision Systems, Brussels, Belgium (2004). [5] Whitehead, A., Laganiere, R., Bose, P., Temporal Synchronization of Video Sequences in Theory and in Practice, Proc. IEEE Workshop on Motion and Video Computing, pp , Breckenridge, CO (2005). [6] Wolf. L, Zomet, A., Wide Baseline Matching between Unsynchronized Video Sequences, Intern. Journal of Computer Vision, Volume 68, No. 1, pp , Springer (2006). [7] Ukrainitz, U., Irani M., Aligning Sequences and Actions by Maximizing Space-Time Correlations, Lecture Notes in Comp. Science, Vol. 3953, pp , Springer (2006). [8] Barkowsky, M., Bitto, R., Bialkowski, J., Kaup, A., Comparison of Matching Strategies for Temporal Frame Registration in the Perceptual Evaluation of Video Quality, Proc. Intern. Workshop Video Processing and Quality Metrics, Scottsdale, AZ (2006). [9] Ohm, J.-R., Multimedia Communication Technology: Representation, Transmission and Identification of Multi-Media Signals, pp , Springer, Berlin (2004). [10] Press, W. H., Flannery, B.O., Teukolsky, S.A., Vetterling, W.T., Numerical Recipes in C: The Art of Scientific Computing, Cambridge University Press, pp , Cambridge, U.K. (1988). [11] Nocedal, J., Wright, S.J., Numerical Optimization, Springer, pp , New York, USA (1999).