Joint Optimization of Source-Channel Video Coding Using the H.264/AVC encoder and FEC Codes Digital Signal and Image Processing Lab Simone Milani Ph.D. student simone.milani@dei.unipd.it, Summer School on Information Enginnering
Outline Introduction to video coding for communications A brief overview of the H.264/AVC Techniques for robust transmission A Joint Source-Channel Coding Rate Control for the H.264/AVC coder Experimental results Conclusions and future work 2
The need for video coding Different video applications have to deal with different constraints that may be related to limited storage capacity available transmission bandwidth Video Storage Web TV and Video On Demand Services Video CODECs Videocommunication Here comes the need for video coding Video Surveillance Systems 3
Different stages of video transmission Video coding allows the transmission of the video sequence over a limited-bandwidth channel by formatting the input data according to the transmission capacity and channel reliability. Acquisition Compression/ Coding Original bit rate Display Transmission bit rate Decoding In order to fit the data to the transmission conditions, we adopt lossy coding schemes that are able to obtain a high compression gain at the expense of distorting the original sequence. For a given channel, the quality degradation depends on the coding standard and the parameters-optimization algorithm. 4
Choice of a video coding scheme The choice of a video coding algorithm is affected by different requirements Device-dependent requirements Storage Videocall Videoconferencing Rate control precision High video quality of reconstructed images Variations in reconstructed video quality Coding latency due to pre-analisys Computational complexity Channel-dependent requirements Storage Videocall Videoconferencing Rate control precision Efficiency at low bit rates Robustness to errors and data losses 5
Error and losses through the network Data may get lost in the network because of delays transmission errors discarding (network congestions, ) The end-user perceives the loss as a loss of visual information (i.e. a slice, a frame or more than one). At the receiver, error concealment techniques are able to estimate the lost part of the picture using temporal and spatial correlation. But. The recovered sequence may result quite different according to the input sequence 6
Error concealment: an example Coded with QP=32 GOP IPPP News (frame 4) Teeny (frame 20) The performance of error concealment is strictly dependent on the motion of the video sequence and the presence of new elements. Source coder and channel coder must consider this. 7
Robust transmission of video sequences over lossy channels Redundancy reduction has to be performed in a careful way taking into account the channel characteristics. The source coding unit has to be followed by a channel coding unit Reduces Increases redundancy redundancy Source Coding Channel Coding channel Channel Decoding Source Decoding Whenever dealing with multimedia transmission, A Joint Source-Channel (JSC) approach proves to be more efficient 8
As for source coding Different video standards have been proposed during the last decade. H.263 H.264/MPEG-4 AVC The recent video coding standard H.264/MPEG-4 AVC provides better performances than previous ones, and its design was mainly inspired by the need for video coders suited for power-limited hardware implementations. MPEG-4 H.264 Standard MPEG-4 H.264 PSNR(dB) 30.18 32.66 Sequence foreman GOP IPPP 60 frame 64 kbit/s @ 30 frame/s 9
The H.264 standard Source Coding Macroblock Frame format QCIF CIF Size 176 X 144 352 X 176 Input signal YUV 4:2:0 8 bit/sample H.264 standardizes a hybrid transform video coder with spatial and temporal prediction Macroblock 16 16 4 blocks 8 8 16 smaller blocks 4 4 10
Spatial Intra Prediction Unlike previous hybrid coding schemes, the H.264/MPEG-4 AVC coder includes a set of spatial predictors that reduces the spatial correlation of the input image. Intra 16x16 Intra 4x4 The patial prediction is performed either on 4x4 block basis or on a 16x16 block basis. The extended version of the H.264 coder (FRExt) also supports an Intra prediction on 8x8 blocks. 11
Temporal Inter Prediction Frame n-2 Frame n-1 Frame n Motion Vector Like many of the previous coding schemes, the H.264/MPEG-4 AVC coder predicts the current frame from the previous one through motion vectors. The standard improves the coding performance of motion estimation thanks to adopting Remind that a finer and more flexible prediction partitioning can be of the predicted block. bidirectional 16 16 8 8 16 8 8 1 1 2 8 1 1 2 8 4 4 8 4 4 1 2 1 1 Partitioning modes 2 2 3 4 4 4 1 3 2 4 12 8 8
H.264 bitstream coding Context Adaptive Variable Length Coder (CAVLC) Chooses a VLC Exp-Golom code table according to the number of non-zero DCT coefficients Context Adaptive Binary Arithmetic Coder (CABAC) N 0, 1 2, 3 4 7 > 8 Table for c_t Num-VLC0 Num-VLC1 Num-VLC2 FLC 13
Slice Partitioning According to the coded frame size and the slice partition, each picture is packetized in different manners. Input frame The slice partitioning can be performed in different ways Fixed number of MBs in a slice Nearly fixed number of bytes in a slice Regions of macroblocks with different sizes Pseudo-random groups of isolated MBs Fixed # of MBs Fixed # of bytes Regions of MBs 14
Packetization After coding the input picture, the bit stream is embedded into packets that are sent through the network. RTP packet.. Packets are sent across the network using the RTP protocol (Real-time Transport Protocol) Network Usually, 1 packet 1 slice In the simplest case, 1 packet 1 frame 15
As for robust transmission It is possible to increase the robustness of the transmission introducing some redundancy in the bit stream Two different approaches are now under investigation Multiple Description Coding (MDC) FEC coding at upper levels 16
Multiple Description Coding MDC represents a single information source with several chunks of data (i.e. descriptions) so that the source can be approximated from any subset of chunks [Goyal,2001] Application fields of Multiple Description: Multimedia transmission over unreliable MIMO channels Robust transmission across a multipath environment in a packet switched network Retrieval of multimedia contents from multiple sources Scalable video transmission 17
FEC Matrix codes FEC codes are widely used at the lowest OSI stack levels, but they can be adopted at application layer too. Packet Video Source It is possible to perform FEC filling a matrix with RTP packets ddding redundant data according to a FEC coding scheme decoding at the receiver whenever the number of losses is lower than a threshold FEC code packets Source code packets 18
JSC optimization Experimental results show that the visual quality perceived by the end user improves whenever redundancy allocation takes into account the characteristics of the input signal. In fact, whenever the lost information can be estimated from the data already received, protection can be reduced. We need a parameter to detect which part of a sequence is crucial for the visual quality in case it is lost. Lost section Lost section Low activity High activity 19
Parameters to drive UEP Adaptation algorithm current _ act avg _ act frame _ type frame _ type Matrix coder Rate current _ act avg _ act frame _ type frame _ type current _ act avg _ act frame _ type frame _ type Matrix coder Rate 70 The same performance can be obtained using the percentage (ρ) of non-null DCT coefficients called zeros with the advantage of providing an efficient model for the bit rate Rate/frame (kbits) Rate (Kbit) 60 50 40 30 20 10 0 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 ρ ρ 20
Rate allocation According to the available bandwidth and the previous bit allocation errors, a target amount of bits is given to the actual GOP. According to different criteria, the available bits are shared among all the frames to be coded in the current GOP. channel Rate control Available bits Criteria Bit allocation errors Image complexity Available bandwidth GOP structure and image type coded frames current frame future frames Target bit number T n The available number of bits T n is shared between the source coder and matrix-based FEC coder according to the parameter ρ. Source coder target bits FEC coder target bits 21
Joint Source-Channel Rate control scheme BW allocation algorithm Parameters Update Avg. QP and ρ computation H.264 coder FEC coder The BW allocation algorithm takes into account the data provided by both the H.264 coder and the FEC coder and tunes its parameters in such a way that the produced bit stream is kept within the available bandwidth R (ρ ) = m ρ + q Parameters update X i, i = I, P, B ρ frame_type,σ ρ 22
23 Target bit rate computation B P B P B P P I n n n K K K G T + + =,,, R B F n R G = B P B P P I P I X X K X X K ˆ ˆ ˆ ˆ,, = = For the n-th frame, the target amount of bits T n is computed as it follows where the complexity for each type of frame is computed as (i=i,p,b) i i i X X X + ) (1 ˆ ˆ α α ( ) C S QP i B B X + = / 6 2 B S B C G G where B S and B C are the number of coded bits prouced by the source coder and the channel coder respectively. Remind that the number of bits available for the current GOP is updated after coding each frame
Bandwidth sharing The target numberof bits is partitioned between the source and the channel coder T n = TS, n + TC, n At first, the target bit rate for the source coder is computed: this imply the estimation of ρ for the current frame. T = T ( 1 r ρ = S, n n ) T S, n m q (remember that R=m ρ+q) Then, the target bit rate for the channel coder is adjusted according to th following equations T C, n = Tn ( r + δr) + 0.1 + 0.05 δr = 0.05 0.1 if if if if ρ ρ < + 0.02 i ρ ρ < + 0.01 i ρ ρ > 0.01 i ρ ρ < 0.02 i 24
Experimental results (1) Simulations were carried on loss patterns obtained on an AWGN radio channel with E b /N r =4 db, measured BER 0.002. Transmission was performed in a packet-switched way with 200 bytes frames and a convolutional ECC (rate ½ memory 5) 25
Experimental results (2) Rate Control + Fixed FEC code Joint Source-Channel Rate Control news 500kbit/s @ 30 frame/s 26
Conclusions FEC codes channel coding permits transmitting a video sequence over lossy channel recovers lost packets increases the performance of the error concealment techniques The choice of optimal code depends on many time-varying factors such as the input video sequence the channel loss percentage quality or delay requirements An adaptive approach that controls the protection level according to the statistics of the input signal was presented. The signal statistics can be devised from the parameters activity ρ The ρ-adaptive UEP approach was included in a joint source-channel rate control algorithm that provides better performances with respect to a static solution. 27
Possible Future Work Adoption of packing algorithms to include in a more efficient way the packets in the matrix Adaptation of the ρ-based FEC tuning algorithm to different coding schemes (i.e. LDPC, turbo codes, etc ) 28