An error resilient coding scheme for JPEG image transmission based on data embedding and side-match vector quantization q

Transcription

1 J. Vis. Commun. Image R. 17 (2006) An error resilient coding scheme for JPEG image transmission based on data embedding and side-match vector quantization q Li-Wei Kang, a,b Jin-Jang Leou a, * a Department of Computer Science and Information Engineering, National Chung Cheng University, Chiayi 621, Taiwan, ROC b Institute of Information Science, Academia Sinica, Taipei 115, Taiwan, ROC Received 1 October 2004; accepted 11 August 2005 Available online 10 October 2005 Abstract For an entropy-coded Joint Photographic Experts Group (JPEG) image, a transmission error in a codeword will not only affect the underlying codeword but also may affect subsequent codewords, resulting in a great degradation of the received image. In this study, an error resilient coding scheme for JPEG image transmission based on data embedding and side-match vector quantization (VQ) is proposed. To cope with the synchronization problem, the restart capability of JPEG images is enabled. The objective of the proposed scheme is to recover high-quality JPEG images from the corresponding corrupted images. At the encoder, the important data (the codebook index) for each Y (U or V) block in a JPEG image are extracted and embedded into another masking Y (U or V) block in the image by the odd even data embedding scheme. At the decoder, after all the corrupted blocks within a JPEG image are detected and located, if the codebook index for a corrupted block can be correctly extracted from the corresponding masking block, the extracted codebook index will be used to conceal the corrupted block; otherwise, the side-match VQ technique is employed to conceal the corrupted block. Based on the simulation results obtained in this study, the performance of the proposed scheme is better than those of the five existing approaches for comparison. The proposed scheme can recover high-quality JPEG images from the corresponding corrupted images up to a block loss rate (BLR) of 30%. Ó 2005 Elsevier Inc. All rights reserved. Keywords: Error resilient coding; JPEG image; Transmission error; Data embedding; Side-match vector quantization 1. Introduction To reduce transmission bit rate or storage capacity, many compression techniques have been developed for various applications, such as videophones, videoconferencing, World Wide Web (WWW), and multimedia q This work was supported in part by National Science Council and Ministry of Economic Affairs, Taiwan, ROC under Grants NSC E and 93-EC-17-A-02-S * Corresponding author. Fax: address: jjleou@cs.ccu.edu.tw /$ - see front matter Ó 2005 Elsevier Inc. All rights reserved. doi: /j.jvcir

2 L.-W. Kang, J.-J. Leou / J. Vis. Commun. Image R. 17 (2006) communications. Reliable transmission of compressed images/video over noisy channels, such as the Internet or mobile networks, is a challenging problem [1 3]. For an entropy-coded JPEG image, a transmission error in a codeword will not only affect the underlying codeword but also may affect subsequent codewords, resulting in a great degradation of the received image. To cope with the synchronization problem, many image/ video coding standards [4 6] insert synchronization markers (codewords) into the compressed image/video bitstream. For JPEG, if its restart capability is enabled, the eight unique restart markers in JPEG can be used as eight synchronization codewords [4,5]. After the decoder receives any restart marker, the decoder will be resynchronized regardless of the preceding slippage. Although, the propagation effect of a transmission error within a JPEG image can be terminated when any restart marker is correctly received, a transmission error may affect the underlying codeword and its subsequent codewords within the corrupted restart interval, as an illustrated example shown in Fig. 1. In general, error resilient approaches include three categories [2,3], namely: (1) the error resilient encoding approach [7 9], (2) the error concealment approach [10 18], and (3) the encoder decoder interactive error control approach [19]. The error resilient encoding approach can be further divided into three categories, namely, (1) the robust entropy coding approach [7], (2) the layered coding with unequal error protection approach [8], and (3) the multiple description coding approach [9]. Several robust entropy-coding approaches have been proposed to cope with the synchronization problem. Redmill and Kingsbury [7] developed a technique called error resilient entropy-coding (EREC), which divides the whole bitstream into blocks of variable-length coded data and then reorganizes the compressed data such that each block starts at a known position within the bitstream. For the layered coding with unequal error protection approach [8], an image/video bitstream is divided into a base layer and one or several enhancement layer(s). The base layer contains the more important information, such as the header and motion information, and provides a lower but acceptable image/video quality, whereas the enhancement layer(s) contain the remaining information, and incrementally improve the image/video quality. The base layer and enhancement layer(s) must be protected with unequal error protection, in which the base layer is protected more strongly. The multiple description coding approach [9] divides an image/video bitstream into several sub-bitstreams, known as descriptions. In contrast to the layer coding approach, the relationship between the descriptions is nonhierarchical and correlated, and the descriptions have similar importances. Any single description can provide a basic quality, and more descriptions together will provide an improved quality. The error concealment approach [10 18] conceals the corrupted (lost) information due to transmission errors in the compressed image/video bitstream at the decoder. In contrast to the error resilient encoding approach, the error concealment approach employs no additional bit rate, but adds computational complexity to the decoder. In terms of the information used for concealment, the error concealment approach can be classified into three categories: (1) spatial (spectral) [10 15], (2) temporal [16], and (3) hybrid [17,18]. For the spatial (spectral) error concealment approach [10 15], the information from correctly reconstructed and/or previously concealed blocks neighboring (eight-connected or four-connected) to a corrupted block is used to conceal the corrupted block. The corrupted block is usually concealed (reconstructed) by using Fig. 1. The error-free and corrupted JPEG images (the Y component) of the Lenna image with the block loss rate (BLR) = 10%: (A) the error-free image and (B) the corresponding corrupted image.

3 878 L.-W. Kang, J.-J. Leou / J. Vis. Commun. Image R. 17 (2006) boundary pixels surrounding the corrupted block, and the concealed block will satisfy the smoothness property of common image signals and produce a maximally smooth image among all those with the same coefficients and boundary conditions. The conventional spatial error concealment techniques are usually based on image/video interpolation in the spatial (pixel) domain [10 12] or transform coefficient recovery in the spectral DCT (discrete cosine transform) or DFT (discrete Fourier transform) domain [13 15]. For these techniques, it is usually assumed that the corrupted blocks are isolated or only few consecutive blocks are corrupted simultaneously. If only few correctly received neighboring blocks of a corrupted block are available, the recovered corrupted block may tend to be blurry and blocky. On the other hand, if a feedback channel can be set up from the decoder to the encoder [19], the decoder can inform the encoder about which parts of the compressed image/video bitstream are corrupted (due to transmission errors), and then the encoder can adjust its encoding operations accordingly to suppress or eliminate the effect of transmission errors. If the automatic repeat request (ARQ) function is supported, the corrupted (lost) packets can be retransmitted. But it is not suitable for real-time applications due to processing delays. The foregoing error resilient approaches concentrate on limiting error propagation and using the correctly received information to conceal corrupted image/video data. Recently, several error resilient coding approaches based on data embedding are proposed [20 27], in which some important data useful for error concealment performed at the decoder can be embedded into the compressed image/video bitstream, when they are encoded at the encoder. The embedded data should be almost invisible and cannot degrade the image/video quality greatly. At the decoder, if the corrupted blocks are detected and located, the important (embedded) data for the corrupted blocks will be extracted and used to facilitate error concealment performed at the decoder. Yu and Yin [20] proposed a multimedia data recovery approach, in which a content-associative signature of a block in an image is generated and inserted imperceptibly into another (remote) block of the image. At the decoder, the embedded content-associative signature for each corrupted block is extracted and employed to reconstruct (conceal) the corrupted block. Yin, Liu, and Yu [21] embedded the block type and edge direction index of a block within an image into the DCT coefficients of another block of the image by the odd even embedding scheme. At the decoder, the embedded data for each corrupted block are extracted and employed to conceal the corrupted block by bilinear interpolation. Song and Liu [25] proposed a data embedding scheme for error-prone channels, in which some redundant information used to protect motion vectors and coding modes of macroblocks in one video frame is embedded into the motion vectors in the next video frame. Based on the assumption that the next video frame of a corrupted video frame is correctly received, the decoder can recover the motion vectors of the corrupted group of blocks in the corrupted video frame. In this study, an error resilient coding scheme for JPEG image transmission based on data embedding and side-match vector quantization (VQ) is proposed. At the encoder, the important data (the codebook index) for each Y (U or V) block in a JPEG image are extracted and embedded into another masking Y (U or V) block in the image by the odd even data embedding scheme. At the decoder, after all the corrupted blocks within a JPEG image are detected and located, if the codebook index for a corrupted block can be extracted correctly from the corresponding masking block, the extracted codebook index will be used to conceal the corrupted block; otherwise, the side-match VQ technique is employed to conceal the corrupted block. This article is organized as follows. A brief overview of the JPEG image compression standard and the sidematch vector quantization technique is given in Section 2. The proposed error resilient coding scheme for JPEG image transmission is addressed in Section 3. Simulation results are included in Section 4, followed by concluding remarks. 2. JPEG image compression standard and side-match vector quantization A. JPEG image compression standard The JPEG compression standard has four modes of operation [4,5], namely, the sequential DCT-based, progressive DCT-based, sequential lossless, and hierarchical modes of operation. In this study, the sequential DCT-based mode of operation (the most popular mode of operation) is treated. At the encoder, an image is partitioned into equal-sized (8 8) and nonoverlapping blocks. After a block has been transformed by the forward DCT and quantized, all the 64 quantized DCT coefficients are entropy-encoded and output as part of the

4 compressed image data. After the quantized difference between the current quantized DC coefficient and that of the previous block is encoded, all the 63 quantized AC coefficients are converted into a zig-zag sequence. For a DCT-based JPEG codec, the data unit is an 8 8 block of samples and data units are assembled into groups called minimum coded units (MCUÕs). If the source image contains only one component and the data are noninterleaved, the MCU is one data unit. For interleaved data, the MCU is the sequence of data units defined by the sampling factors of the component [4,5]. Markers serve to identify the various structural parts of the compressed data formats. Most markers start marker segments containing a related group of parameters, whereas some markers stand alone. A marker segment consists of a marker followed by a sequence of related parameters. The first parameter is a two-byte length parameter, which encodes the number of bytes in the marker segment. The marker segments identified by the start of frame (SOF) and start of scan (SOS) marker codes are referred to as the frame header and the scan header, respectively. JPEG divides the compression sequence into frames and scans. For nonhierarchical mode of operation, the frame defines the basic attributes of the image, including size, number of components, precision, and entropycoding technique. For nonhierarchical encoding, only a single frame is allowed. In addition to the frame and scan headers, a number of other marker segments are needed to define entropy-coding tables, quantization tables, and parameters. As shown in Fig. 2, the high-level syntax of JPEG specifies the order of the high-level constituent parts of the interchange format for the nonhierarchical encoding processes. In Fig. 2, three markers can be identified: (1) SOI (start of image) indicates the start of a compressed image; (2) EOI (end of image) indicates the end of a compressed image; and (3) RST m (restart marker) is a marker placed between entropy-coded segments only if the restart capability is enabled. In JPEG, the eight unique restart markers (RST m, m = 0,1,2,...,7) repeating in sequence from 0 to 7 provide a modulo-8 restart interval count. In this study, we consider the case that a frame will contain one single scan with interleaved ordering, the restart capability is enabled, and a restart interval contains 32 MCUÕs, in which one MCU includes six data units (four Y blocks, one U block, and one V block), just like a GOB in the H.263 video compression standard [6]. B. Side-match vector quantization L.-W. Kang, J.-J. Leou / J. Vis. Commun. Image R. 17 (2006) Vector quantization (VQ) is an efficient approach to low bit rate image compression [28 31]. A vector quantizer Q is a mapping from a k-dimensional Euclidean space, R k, into a finite subset Y containing N reproduction points or codewords in R k. That is, Q: R k! Y, where Y ={y i j i = 1,2,...,N} is called the codebook and y i is the ith codeword. At the encoder, an image is first divided into several nonoverlapping blocks and each block represented by a vector x is compared with all the codewords (vectors) in the codebook Y to find the closest codeword y i and the index i (instead of y i itself) is used to represent x. At the decoder, the codeword y i is used to reconstruct the image block x. compressed image data [table / misc.] SOI frame EOI frame DNL frame header scan 1 [ segment ] [scan 2 ] [scan last ] [table / misc.] scan scan header [ECS0 RST 0 ECS last-1 RST last-1] ECS last entropy-coded segment0 entropy-coded segmentlast <MCU >, <MCU >, <MCU > <MCU >, <MCU >, <MCU > Fig. 2. The high-level syntax of JPEG for the sequential DCT-based mode of operation.

5 880 L.-W. Kang, J.-J. Leou / J. Vis. Commun. Image R. 17 (2006) x U (8,1) (8,2) (8,3) (8,4) (8,5) (8,6) (8,7) (8,8) (1,8) (1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) (1,8) (1,1) (2,8) (2,1) (2,8) (2,1) (3,8) (3,1) (3,8) (3,1) (4,8) (4,1) (4,8) (4,1) x L x (5,8) (5,1) (5,8) (5,1) x R (6,8) (6,1) (6,8) (6,1) (7,8) (7,1) (7,8) (7,1) (8,8) (8,1) (8,2) (8,3) (8,4) (8,5) (8,6) (8,7) (8,8) (8,1) (1,1) (1,2) (1,3) (1,4) (1,5) (1,6) (1,7) (1,8) x B Fig. 3. The relationship between an 8 8 image block x and its four (upper, left, bottom, and right) neighboring blocks, x U, x L, x B, and x R, respectively. Side-match VQ [29,30] uses the attribute of spatial contiguity across block boundary to establish the state, which is described as follows. For an input k-dimensional vector (image block) x (p, q), where k = n n and p,q = 1,2,...,n, its corresponding approximated closest codeword y i (p,q) in the codebook Y of size N can be found by using the side information of its neighboring image blocks. Assume that the four (upper, left, bottom, and right) neighboring blocks of x(p,q) are x U (p,q), x L (p,q), x B (p,q), and x R (p,q), respectively, as an illustrated example (n = 8 and k = 64) shown in Fig. 3. The side-match distortion d sm (x,y i ) between x(p,q) and y i (p,q) is defined as d sm ðx; y i Þ¼ Xn ½x U ðn; qþ y i ð1; qþš 2 þ Xn ½x L ðp; nþ y i ðp; 1ÞŠ 2 þ Xn ½x B ð1; qþ y i ðn; qþš 2 q¼1 p¼1 q¼1 þ Xn p¼1 ½x R ðp; 1Þ y i ðp; nþš 2. ð1þ If d sm ðx; y i Þ¼min d sm ðx; y j Þ, the codeword y i (p,q) in the codebook is decided to be the approximated closest j codeword of the image block x (p,q). Because, the blocks in an image are coded in a raster scan manner, the upper and left neighboring blocks are usually available in the encoding process, whereas the bottom and right neighboring blocks are usually ignored. 3. Proposed error resilient coding scheme Within the proposed scheme, the following issues will be addressed: (1) what kind of important data for the blocks within an image should be extracted and embedded, (2) where should the important data be embedded, (3) how to embed the important data into the corresponding masking blocks, and (4) how to extract and use the important data to conceal the corrupted blocks at the decoder.

6 A. Important data extraction and embedding L.-W. Kang, J.-J. Leou / J. Vis. Commun. Image R. 17 (2006) Similar to the VQ system [28 31], both the encoder and the decoder have an identical codebook, in which the dimension of each codeword is the same as the size (8 8) of a block. For each Y (U or V) block in JPEG images, there exists a closest codeword in its codebook with the smallest mean-squared error (MSE) between the block and the codeword in its codebook. For a codebook with size N, instead of a 64-dimensional vector, only dlog 2 Ne bits are required to represent the index of each codeword, where dlog 2 Ne denotes the ceiling of log 2 N. In the proposed scheme, the codebook index of each Y (U or V) block is extracted as its important data and its codebook is trained by training images using the fast algorithm provided in [31]. Because the human visual system is more sensitive to the luminance component Y than the chrominance components, U and V, the value of N Y for the Y component is usually larger than that of N U (=N V ) for the U (V) component, in which N Y, N U, and N V denote the codebook sizes for the Y, U, and V components, respectively. Additionally, to reduce the size of the important data for a Y (U or V) block, if the MSE between the closest codeword for the block in its codebook and the corresponding closest approximated codeword determined by side-match VQ is smaller than a threshold T V, the important data for the block is just one indicating bit 0 denoting that the block can be approximated by side-match VQ. Additionally, if the available embedding capacity for the codebook index for a block is not sufficient, the important data for the block is also replaced by one indicating bit 0 denoting that the block should be approximated by sidematch VQ. Because at the encoder, only the upper and left neighboring blocks of the current block are usually available, only two (upper and left) neighboring blocks are usually employed to find the closest approximated codeword for the current block by side-match VQ. When the MSE between the closest codeword for a block and the corresponding closest approximated codeword is smaller than a threshold T V, this case means that the block can be well-recovered by side-match VQ. For a block can be well-recovered by only two (upper and left) neighboring blocks at the encoder, if more (e.g., three or four) correctly received or well-concealed neighboring blocks for the block can be available at the decoder, the block will be wellrecovered by side-match VQ. For JPEG, different values of the quality factor M correspond to different compression ratios. When M is the default value 75, the compression ratio is about 18. In this study, three different M values (50, 75, and 90) are evaluated. The corresponding average compression ratios, average embedding capacities,n Y, N U, N V, and T V values for different M values are listed in Table 1. The average embedding capacity for an M value is determined by the numbers of nonzero quantized DCT coefficients (the embedding capacities for data embedding) for the blocks of all the training images encoded with the specified M value. When M is a large value, i.e., the larger number of nonzero DCT coefficients are available, the larger N Y, N U, N V, and T V values can be used, and vice versa. In this study, different N Y, N U, N V, and T V values for different M values are all determined empirically. In general, the larger the M value is, the larger the corresponding average embedding capacity is, i.e., the largern Y, N U, N V,andT V values are, and vice versa. For a small codebook containing few codewords, the difference between each codeword pair in the small codebook will be relatively large and the corresponding performance of side-match VQ will degrade accordingly. That is because only boundary pixels from the neighboring blocks of a block are employed to find the closest approximated codeword for the block in the small codebook, the corresponding closest approximated codeword may be not so good due to only a small number of codewords can be evaluated. Hence, for M = 50, the threshold T V for the Y component is set to 1 (a small value). Table 1 The average compression ratios, average embedding capacities, and the corresponding N Y, N U, N V, and T V values for different M values M Average compression ratio Average embedding capacity N Y N U N V T V Y U V

7 882 L.-W. Kang, J.-J. Leou / J. Vis. Commun. Image R. 17 (2006) The important data (codebook index) for an 8 8 Y (U or V) block in a JPEG image will be embedded into the DCT coefficients of another (remote) 8 8 Y (U or V) block, called the masking block, in the same image. A block and its masking block should be as far as possible so that the two corresponding blocks will be seldom corrupted simultaneously. To realize this idea, a block and its masking block should not be in the same restart interval, and the masking blocks of the blocks in a row should not be in the same row. To satisfy the above two constraints, in this study, for an 8 8 Y block, B(i,j), 0 6 i 6 63, 0 6 j 6 63, in a JPEG Y component image, its 8 8 masking block B(p,q) in the Y component image can be determined by 8 ði;i2þjþ2þmod 64 if 0 6 i 6 30; >< ðp;qþ¼ ði;ði 31Þ2þjþ2Þmod 64 if 31 6 i 6 61; ð2þ >: ði;ði 62Þ2þjþ2Þmod 64 if 62 6 i For an 8 8 U (or V) block, B(i,j), 0 6 i 6 31, 0 6 j 6 31, in a JPEG U (or V) component image, its 8 8 masking block B(p,q) in the corresponding U (or V) component image can be determined by 8 >< ði;i2þjþ2þmod 32 if 0 6 i 6 14; ðp;qþ¼ ði;ði 15Þ2þjþ2Þmod 32 if 15 6 i 6 29; ð3þ >: ði;ði 30Þ2þjþ2Þmod 32 if 30 6 i For an 8 8 block, B(i,j), in an N N JPEG component image, its 8 8 masking block B(p,q) in the corresponding N N component image can be determined in a similar way. For example, for an 8 8 block, B(i,j), 0 6 i 6 7, 0 6 j 6 7, in a JPEG component image, its 8 8 masking block B(p,q) in the corresponding component image can be determined by 8 >< ði;i2þjþ2þmod 8 if 0 6 i 6 2; ðp;qþ¼ ði;ði 3Þ2þjþ2Þmod 8 if 3 6 i 6 5; >: ði;ði 6Þ2þjþ2Þmod 8 if 6 6 i 6 7. The detailed relationship between any 8 8 block B(i,j) and its 8 8 masking block B(p,q) for an illustrated JPEG image component is shown in Fig. 4. The main advantage of this kind of relationship between an 8 8 block B(i,j) and its 8 8 masking block B(p,q) is that all the masking blocks of the blocks in the same restart interval will not be in the same restart interval. If some successive blocks within a restart interval are corrupted simultaneously, the masking blocks of the successive corrupted blocks within the same restart interval should be distributed over several different restart intervals, which will be seldom corrupted simultaneously. For example, referring to Fig. 4, the corresponding masking blocks B(p, q) of blocks B(i, j), 0 6 i 67, 2 6 j 6 3, i.e., blocks (0,2), (1,2), (2,2), (3,2), (4,2), (5,2), (6,2), (7,2), (0,3), (1,3), (2,3), (3,3), (4,3),(5,3), (6,3), and (7,3) within the second restart interval (containing 16 blocks) are blocks (0,4), (1,6), (2,0), (3,4), (4,6), (5,0), (6,4), (7,6), (0,5), (1,7), (2,1), (3,5), (4,7), (5,1), (6,5), and (7,7), respectively, which are not within the same restart interval. As a summary, the major design criteria for the relationship between a ð4þ i j 0 (0, 2) (1, 4) (2, 6) (3, 2)(4, 4)(5, 6) (6, 2) (7, 4) 1 (0, 3) (1, 5) (2, 7) (3, 3)(4, 5)(5, 7) (6, 3)(7, 5) 2 (0, 4) (1, 6) (2, 0) (3, 4) (4, 6) (5, 0)(6, 4)(7, 6) 3 (0, 5) (1, 7) (2, 1) (3, 5) (4, 7) (5,1)(6, 5)(7, 7) 4 (0, 6) (1, 0) (2, 2) (3, 6) (4, 0) (5, 2) (6, 6) (7, 0) 5 (0, 7) (1, 1) (2, 3) (3, 7) (4, 1) (5, 3) (6, 7) (7, 1) 6 (0, 0) (1, 2) (2, 4) (3, 0) (4, 2) (5, 4) (6, 0) (7, 2) 7 (0, 1) (1, 3) (2, 5) (3, 1) (4, 3) (5, 5) (6, 1) (7, 3) Fig. 4. The detailed relationship between any 8 8 block B(i,j) and its 8 8 masking block B(p,q) in an illustrated JPEG component image.

8 block and its masking block can be described as follows. (1) A block and its masking block should not be in the same restart interval, and (2) the masking blocks of the blocks in a row should not be in the same row. In general, if the two above-mentioned criteria are satisfied, the relationship between a block and its masking block (Eqs. (2) (4)) can be modified to be other similar manners, which will usually result in similar concealment results. To perform data embedding in JPEG images, the odd even embedding scheme [21] is employed and performed on the nonzero quantized DCT coefficients. If the data bit to be embedded is 0, the nonzero quantized DCT coefficient will be forced to be an even number, whereas if the data bit to be embedded is 1, the nonzero quantized DCT coefficient will be forced to be an odd number. That is, if the data bit to be embedded is b j (b j = 0 or 1), the quantized DCT coefficient C i of the odd even data embedding scheme is given by 8 C i þ 1 if C i mod 2 6¼ b j ; and C i > 0; >< C i ¼ C i 1 if C i mod 2 6¼ b j ; and C i < 0; ð5þ >: otherwise C i Data embedding for a JPEG image can be summarized as (1) Extract the important data, either 1 þdlog 2 N Y e bits (one indicating bit 1 and dlog 2 N Y e bits for the codebook index) or one indicating bit 0, for each 8 8 Y block B(i,j); (2) Determine the masking block B(p,q) for each Y block B(i,j); (3) Embed the important data for each block into its masking block; (4) Repeat (1) through (3) for the U and V components. Note that if N Y = 512, the important data for each 8 8 Y block is either 10 bits (one indicating bit 1 and 9 bits for the codebook index) or one indicating bit 0, and if N U = N V = 128, the important data for each 8 8U (or V) block is either 8 bits (one indicating bit 1 and 7 bits for the codebook index) or one indicating bit 0. Data embedding for different N Y, N U, and N V values can be similarly performed. For data embedding, the embedded data should be perceptually invisible, and should not degrade the image quality greatly. Actually, there is a tradeoff between data embedding capacity and error-free image quality degradation due to data embedding. B. Proposed error resilient decoding scheme for JPEG image transmission In this study, the corrupted blocks in a JPEG image are assumed to be detected and located first. For each corrupted Y (U or V) block, if its masking block is correctly received, the important data (the codebook index) for the corrupted block will be extracted from the masking block, and then the closest codeword in its codebook is used to conceal the corrupted block. On the other hand, if (1) the extracted data is only one indicating bit 0 denoting that no codebook index is embedded, or (2) the masking block of the corrupted block is also corrupted, the closest approximated codeword in its codebook (determined by side-match VQ) is used to conceal the corrupted block. Although corrupted blocks are usually concealed in a raster scan manner, if all the four neighboring blocks of a corrupted block are correctly received or well-concealed, side-match VQ can find the closest approximated codeword in its codebook with a more accurate manner. Here, if some of the four neighboring blocks of a corrupted block are also corrupted, and these corrupted neighboring blocks can be concealed with the codebook indices correctly extracted from their corresponding masking blocks, these corrupted neighboring blocks will be concealed first. Then, the corrupted block can be concealed by side-match VQ with more neighboring block information, resulting in the better concealed block. The proposed error resilient decoding scheme for JPEG image transmission can be summarized in Fig Simulation results L.-W. Kang, J.-J. Leou / J. Vis. Commun. Image R. 17 (2006) The proposed error resilient coding scheme for JPEG image transmission has been implemented on a Pentium-III 700 PC using the C++ programming language. Four images Lenna, Baboon,

9 884 L.-W. Kang, J.-J. Leou / J. Vis. Commun. Image R. 17 (2006) A corrupted block Masking block determination No Important data extraction Is the masking block corrupted? Indicating bit 0 Yes What kinds of extracted data? Codebook index Conceal the current corrupted block by its codebook index No Conceal the current corrupted block by side-match VQ Can the corrupted 4- connected neighboring blocks be concealed with their codebook indices? Yes Conceal the neighboring corrupted blocks of the current corrupted block first Fig. 5. The proposed error resilient decoding scheme for JPEG image transmission. Airplane, and Peppers with different block loss rates, denoted by BLR, are used to evaluate the performance of the proposed scheme (denoted by proposed). The former two images, Lenna and Baboon, are inside VQ codebook training, whereas the latter two images, Airplane and Peppers, are outside VQ codebook training. The block loss rate (BLR) is widely used in related researches [10,11,14,21], in which the locations of corrupted blocks can be either randomly selected or uniformly corrupted. In [10,11,14,21] the corrupted (missing) blocks are isolated ones. However, the corrupted (missing) blocks in JPEG are not just isolated ones, i.e., the successive blocks of a corrupted block within the same restart interval will be corrupted ones. In this study, as listed in Table 1, the quality factor M for scaling quantization tables used to adjust image quality in JPEG, ranged from 0 (the worst image quality) to 100 (the best image quality), is set to three typical values: 50, 75, and 90 [4,5]. As shown in Table 1, different threshold values of T V for the Y, U, and V components are specified for different M values. For example, when M = 75, the average embedding capacity of an 8 8 masking block is about 11 bits, and the actual average number of data bits needed to be embedded into an 8 8 masking block is about five bits, i.e., the important data for each block can be usually embedded into its masking block completely. The peak signal to noise ratio (PSNR) is employed in this study as the objective performance measure for the three components (Y, U, and V) of a JPEG image. The mean square error (MSE) between an original image and the corresponding reconstructed (concealed) image, denoted by MSE, is given by MSE ¼ð4MSE Y þ MSE U þ MSE V Þ=6; ð6þ where MSE Y, MSE U, and MSE V are the corresponding MSE values of the Y, U, and V components of a JPEG image, respectively. The PSNR of a JPEG image, denoted by PSNR, is given by PSNR ¼ 10 log 10 ð255 2 =MSEÞ. ð7þ To evaluate the performance of the proposed scheme, five existing error resilient coding and error concealment approaches for comparison [10,12,15,21] are implemented in this study. They are: (1) zero-substitution, which simply replaces all pixels in a corrupted block by zeros (denoted by Zero-S); (2) the best neighborhood matching (BNM) concealment algorithm using the blockwise similarity within an image, in which the information of both neighboring pixels and remote regions in the image is employed (denoted by BNM) [10]; (3) the spatial error concealment algorithm in H.264/AVC, in which each pixel value in a corrupted block

10 L.-W. Kang, J.-J. Leou / J. Vis. Commun. Image R. 17 (2006) is concealed by a weighted sum of the closest boundary pixels of the selected four-connected neighboring blocks (denoted by H.264) [12]; (4) the spectral DFT (discrete Fourier transform) domain error concealment algorithm, in which corrupted image data are concealed by frequency selective extrapolation. The weighted linear combination of two-dimensional DFT basis functions are employed for signal extrapolation (denoted by DFTEC) [15]. (5) the error resilient coding scheme based on data embedding, in which the block type and edge direction index of a block within an image are embedded into the DCT coefficients of another block in the same image by the odd even embedding scheme (denoted by ERDE) [21]. In terms of PSNR in db, the simulation results for the four test images with different block loss rates (BLR) and quality factors (M) of the four existing approaches for comparison (except Zero-S) and the proposed scheme are shown in Figs Here, each PSNR value is averaged over 50 simulation runs. In each PSNR (db) BNM H.264 DFTEC ERDE Proposed BLR (%) Fig. 6. The simulation results, PSNR (db), for the Lenna image with different BLRs and M = 50 of the four existing approaches for comparison and the proposed scheme. PSNR (db) BNM H.264 DFTEC ERDE Proposed BLR (%) Fig. 7. The simulation results, PSNR (db), for the Baboon image with different BLRs and M = 50 of the four existing approaches for comparison and the proposed scheme. PSNR (db) BNM H.264 DFTEC ERDE Proposed BLR (%) Fig. 8. The simulation results, PSNR (db), for the Airplane image with different BLRs and M = 50 of the four existing approaches for comparison and the proposed scheme.

11 886 L.-W. Kang, J.-J. Leou / J. Vis. Commun. Image R. 17 (2006) PSNR (db) BNM H.264 DFTEC ERDE Proposed BLR (%) Fig. 9. The simulation results, PSNR (db), for the Peppers image with different BLRs and M = 50 of the four existing approaches for comparison and the proposed scheme. PSNR (db) BNM H.264 DFTEC ERDE Proposed BLR (%) Fig. 10. The simulation results, PSNR (db), for the Lenna image with different BLRs and M = 75 of the four existing approaches for comparison and the proposed scheme. PSNR (db) BNM H.264 DFTEC ERDE Proposed BLR (%) Fig. 11. The simulation results, PSNR (db), for the Baboon image with different BLRs and M = 75 of the four existing approaches for comparison and the proposed scheme. simulation run, several blocks are randomly selected to be corrupted. If a block is selected to be corrupted, its successive blocks within the same restart interval will be all corrupted. This random selection process will be repeated until the number of corrupted blocks reaches the target number of corrupted blocks for the given BLR. As a subjective measure of the quality of the concealed images, the error-free, the error-free with data embedding, and concealed images (the Y component) of the four test images with different BLR and M values of the five existing approaches for comparison and the proposed scheme are shown in Figs The average processing time (second) per image with different BLR and M values of the five existing approaches for comparison and the proposed scheme is listed in Table 2.

12 L.-W. Kang, J.-J. Leou / J. Vis. Commun. Image R. 17 (2006) PSNR (db) BNM H.264 DFTEC ERDE Proposed BLR (%) Fig. 12. The simulation results, PSNR (db), for the Airplane image with different BLRs and M = 90 of the four existing approaches for comparison and the proposed scheme. PSNR (db) 42 BNM H DFTEC ERDE 38 Proposed BLR (%) Fig. 13. The simulation results, PSNR (db), for the Peppers image with different BLRs and M = 90 of the four existing approaches for comparison and the proposed scheme. Fig. 14. The error-free and concealed JPEG images (the Y component) of the Lenna image with BLR = 10% and M = 50: (A) the errorfree image, (B) the error-free image with data embedding; (C) (H) the concealed images by Zero-S, BNM, H.264, DFTEC, ERDE, and the proposed scheme, respectively. Based on the simulation results obtained in this study, several observations can be found. (1) Based on the simulation results shown in Figs. 6 17, the concealed results of the proposed scheme are better than those of the five existing approaches for comparison. (2) Compared with the error-free images (BLR = 0), the average

13 888 L.-W. Kang, J.-J. Leou / J. Vis. Commun. Image R. 17 (2006) Fig. 15. The error-free and concealed JPEG images (the Y component) of the Baboon image with BLR = 15% and M = 50: (A) the error-free image, (B) the error-free image with data embedding; (C) (H) the concealed images by Zero-S, BNM, H.264, DFTEC, ERDE, and the proposed scheme, respectively. Fig. 16. The error-free and concealed JPEG images (the Y component) of the Airplane image with BLR = 20% and M = 50: (A) the error-free image, (B) the error-free image with data embedding; (C) (H) the concealed images by Zero-S, BNM, H.264, DFTEC, ERDE, and the proposed scheme, respectively. image degradation of the images with data embedding is about 1 db, which is slightly larger than that reported in [21]. However, the embedded data are indeed perceptually invisible, as shown in Figs (3) When the larger quality factor (M) is used, the average performance gain (db) of the proposed scheme over the five existing approaches for comparison will increase accordingly, and vice versa. (4) Based on the simulation results shown in Figs. 6 17, whether a test image is inside VQ codebook training or not, the concealed results of the proposed scheme are better than those of the five existing approaches for comparison. This is because if a large number of training image blocks containing several different types of color, shape, edge, and texture information are employed for VQ codebook training, the trained codebook is suitable for most natural images. Here, training images, i.e., 81, training image blocks for the Y component, 20, training image blocks for the U component, and 20, training image blocks for the V component are employed. (5) Based on the simulation results shown in Table 2, the average processing time per image of the proposed scheme is much larger than those of the Zero-S, H.264, and ERDE approaches, comparable to that of the DFTEC approach, and much smaller than that of the BNM approach. This is because in the proposed scheme, if a corrupted block is decided to be concealed by side-match VQ, a full search in the codebook is required to find the corresponding closest approximated codeword.

14 L.-W. Kang, J.-J. Leou / J. Vis. Commun. Image R. 17 (2006) Fig. 17. The error-free and concealed JPEG images (the Y component) of the Peppers image with BLR = 25% and M = 50: (A) the error-free image, (B) the error-free image with data embedding; (C) (H) the concealed images by Zero-S, BNM, H.264, DFTEC, ERDE, and the proposed scheme, respectively. Table 2 The average processing time (second) per image with different BLR and M of the five existing approaches for comparison and the proposed scheme M BLR (%) Processing time (second) Zero-S BNM H.264 DFTEC ERDE Proposed Compared with the five existing approaches for comparison, the proposed scheme is more robust for noisy channels with higher block loss rates (BLRs). That is because, for lower BLR cases, more neighboring information of a corrupted block can be obtained, the corrupted block will be well-concealed by using the boundary pixels of the neighboring blocks. On the other hand, for higher BLR cases, the neighboring blocks of a corrupted block might also be corrupted, and the corrupted block cannot be well-concealed by using fewer neighboring information. Within the proposed scheme, if the codebook index for a corrupted block is extracted correctly, the corrupted block can be well-concealed by the extracted codebook index. When the codebook index for a corrupted block is not correctly extracted, if the neighboring blocks of the corrupted block can be well-concealed first by using their correctly extracted codebook indices and then the corrupted block can be concealed by side-match VQ with more neighboring information, resulting in the better concealed results. However, for a case with a small M value, such as M = 50, the corresponding concealed images may induce some blocking artifacts due to a small codebook (small N Y, N U, and N V values) is employed. In this situation, some blocking artifact reduction scheme can be employed to reduce blocking artifacts in the concealed images. Similar to the VQ system in [28 31], an identical codebook is assumed to be stored in advance at both the encoder and the decoder. Hence, the transmission bit rate of the proposed scheme will not be increased. At the decoder, if the codebook index for a corrupted block can be correctly extracted from the corresponding masking block, the extracted codebook index will be used to conceal the corrupted block, i.e., only one codebook access for a specified codebook index is required. Otherwise, side-match VQ is employed to conceal the corrupted block, i.e., a full search in the codebook is required.

15 890 L.-W. Kang, J.-J. Leou / J. Vis. Commun. Image R. 17 (2006) Concluding remarks In this study, an error resilient coding scheme for JPEG image transmission based on data embedding and side-match VQ is proposed. At the encoder, the important data (the codebook index) for each Y (U or V) block in a JPEG image are extracted and embedded into another masking Y (U or V) block in the image by the odd even data embedding scheme. At the decoder, after all the corrupted blocks within a JPEG image are detected and located, if the codebook index for a corrupted block can be correctly extracted from the corresponding masking block, the extracted codebook index will be used to conceal the corrupted block; otherwise, side-match VQ is employed to conceal the corrupted block. Based on the simulation results obtained in this study, the performance of the proposed scheme is better than those of the five existing approaches for comparison. The proposed scheme can recover high-quality JPEG images from the corresponding corrupted images up to a block loss rate (BLR) of 30%. References [1] Y. Wang, J. Ostermann, Y.Q. Zhang, Video Processing and Communications, Prentice Hall, New Jersey, [2] Y. Wang, Q.F. Zhu, Error control and concealment for video communication: a review, Proceedings of the IEEE 86 (5) (1998) [3] Y. Wang, S. Wenger, J. Wen, A.K. Katsaggelos, Error resilient video coding techniques, IEEE Signal Processing Magazine 17 (4) (2000) [4] CCITT Recommendation T.81, Digital compression and coding of continuous-tone still images (1992). [5] W.B. Pennebaker, J.L. Mitchell, JPEG: Still Image Data Compression Standard, Van Nostrand Reinhold, New York, [6] ITU-T Recommendation H.263: Video coding for low bit rate communication (1998). [7] D.W. Redmill, N.G. Kingsbury, The EREC: an error-resilient technique for coding variable-length blocks of data, IEEE Trans. on Image Processing 5 (4) (1996) [8] M. Gallant, F. Kossentini, Rate-distortion optimized layered coding with unequal error protection for robust Internet video, IEEE Trans. on Circuits and Systems for Video Technology 11 (3) (2001) [9] Y. Wang, S. Lin, Error-resilient video coding using multiple description motion compensation, IEEE Trans. on Circuits and Systems for Video Technology 12 (6) (2002) [10] Z. Wang, Y. Yu, D. Zhang, Best neighborhood matching: an information loss restoration technique for block-based image coding systems, IEEE Trans. on Image Processing 7 (7) (1998) [11] X. Li, M.T. Orchard, Novel sequential error-concealment techniques using orientation adaptive interpolation, IEEE Trans. on Circuits and Systems for Video Technology 12 (10) (2002) [12] Y.K. Wang, M.M. Hannuksela, V. Varsa, A. Hourunranta, M. Gabbouj, The error concealment feature in the H.26L test model, Proceedings, IEEE Int. Conference on Image Processing Rochester, NY, USA (2002) pp [13] W. Zhu, Y. Wang, Q.F. Zhu, Second-order derivative-based smoothness measure for error concealment in DCT-based codecs, IEEE Trans. on Circuits and Systems for Video Technology 8 (6) (1998) [14] F.G.B.D. Natale, C. Perra, G. Vernazza, DCT information recovery of erroneous image blocks by a neural predictor, IEEE Journal on Selected Areas in Communications 18 (6) (2000) [15] K. Meisinger, A. Kaup, in: Minimizing a Weighted Error Criterion for Spatial Error Concealment of Missing Image Data, Proceedings, IEEE Int. Conference on Image Processing, Singapore, 2004, pp [16] J. Zhang, J.F. Arnold, M.R. Frater, A cell-loss concealment technique for MPEG-2 coded video, IEEE Trans. on Circuits and Systems for Video Technology 10 (4) (2000) [17] Y.C. Lee, Y. Altunbasak, R.M. Mersereau, Multiframe error concealment for MPEG-coded video delivery over error-prone networks, IEEE Trans. on Image Processing 11 (11) (2002) [18] L.W. Kang, J.J. Leou, A hybrid error concealment scheme for MPEG-2 video transmission based on best neighborhood matching algorithm, Journal of Visual Communication and Image Representation 16 (3) (2005) [19] T. Stockhammer, H. Jenkac, C. Weiß, Feedback and error protection strategies for wireless progressive video transmission, IEEE Trans. on Circuits and Systems for Video Technology 12 (6) (2002) [20] H.H. Yu, P. Yin, Multimedia data recovery using information hiding, Proceedings, IEEE Int. Global Telecommunications Conference, San Francisco, CA, USA (2000) pp [21] P. Yin, B. Liu, H.H. Yu, Error concealment using data hiding, Proceedings, IEEE Int. Conference on Acoustics, Speech, and Signal Processing (2001) [22] C.Y. Lin, D. Sow, S.F. Chang, Using self-authentication-and-recovery images for error concealment in wireless environments, Proceedings SPIE 4518 (2001) [23] C.S. Lu, Wireless multimedia error resilience via a data hiding technique, Proceedings, IEEE Int. Workshop on Multimedia Signal Processing, St. Thomas, US Virgin Islands (2002) pp [24] L.W. Kang, J.J. Leou, Two error resilient coding schemes for wavelet-based image transmission based on data embedding and genetic algorithms, Proceedings, IEEE Int. Conference on Image Processing, Barcelona, Spain (2003) pp

16 L.-W. Kang, J.-J. Leou / J. Vis. Commun. Image R. 17 (2006) [25] J. Song, K.J.R. Liu, A data embedded video coding scheme for error-prone channels, IEEE Trans. on Multimedia 3 (4) (2001) [26] S.W. Lin, J.J. Leou, L.W. Kang, An error resilient coding scheme for H.26L video transmission based on data embedding, Journal of Visual Communication and Image Representation 15 (2) (2004) [27] L.W. Kang, J.J. Leou, An error resilient coding scheme for H.264/AVC video transmission based on data embedding, Journal of Visual Communication and Image Representation 16 (1) (2005) [28] Y. Linde, A. Buzo, R.M. Gray, An algorithm for vector quantization design, IEEE Trans. on Communications 28 (1980) [29] H.C. Wei, P.C. Tsai, J.S. Wang, Three-sided side match finite-state vector quantization, IEEE Trans. on Circuits and Systems for Video Technology 10 (1) (2000) [30] J.C. Tsai, C.H. Hsieh, T.C. Hsu, A new dynamic finite-state vector quantization algorithm for image compression, IEEE Trans. on Image Processing 9 (11) (2000) [31] S.J. Baek, B.K. Jeon, K.M. Sung, A fast encoding algorithm for vector quantization, IEEE Signal Processing Letters 4 (12) (1997) Li-Wei Kang was born in Taipei, Taiwan on December 26, He received the B.S., M.S., and Ph.D. degrees in computer science and information engineering in June 1997, July 1999, and January 2005, respectively, all from National Chung Cheng University, Chiayi, Taiwan. Since March 2005, he joined the Institute of Information Science, Academia Sinica, Taipei, Taiwan as a postdoctoral fellow. His current research interests include image/video processing, image/video communication, and multimedia database systems. Jin-Jang Leou was born in Chiayi, Taiwan, Republic of China on October 25, He received the B.S. degree in communication engineering in 1979, the M.S. degree in communication engineering in 1981, and the Ph.D degree in electronics in 1989, all from National Chiao Tung University, Hsinchu, Taiwan. From 1981 to 1983, he served in the Chinese Army as a Communication Officer. From 1983 to 1984, he was at National Chiao Tung University as a lecturer. Since August 1989, he has been on the faculty of the Department of Computer Science and Information Engineering at National Chung Cheng University, Chiayi, Taiwan. His current research interests include image/video processing, image/ video communication, pattern recognition, and computer vision.