Video Over Mobile Networks Professor Mohammed Ghanbari Department of Electronic systems Engineering University of Essex United Kingdom June 2005, Zadar, Croatia (Slides prepared by M. Mahdi Ghandi) INTRODUCTION In the 20 th century, visual communications via portable handsets were just in the science fiction movies! At the turn of the 21 th century, it became a reality thanks to : Larger bandwidth in the 3 rd generation (3G) mobile networks and, Advances made in very low bit rate video coding. 1
Background 3G: Third Generation Year: 2002 Service: Voice, Data, Video, Multimedia System: Intelligent Signal Processing Frequency: 1800-2200 MHz Bandwidth: 128 kb/s 2 Mb/s Business View Unfortunately, 3G mobiles are NOT as successful as the 2G mobiles. Why? Mobile operators have paid too much for the licence but have not earned as expected. The quality of service is LOW, the cost of operation is HIGHT. Demand for video services is much less than the audio and text, but increasing. 2
Quality/Cost 37 ACCEPTABLE PRICE CURRENT PRICE QUALITY 33 29 DECENT QUALITY CURRENT QUALITY 25 10 20 30 40 50 60 PRICE CHALLENGES To deliver good quality of service, two main challenges in the wireless communication systems have to be dealt with: Bandwidth Limitation Wireless bandwidth is very scares (in contrast to almost unlimited optical bandwidth) Channel Noise Wireless channels are very prone to errors (normal error rate 10-4, compared to 10-12 for optical channels) 3
BANDWIDTH Video services are bandwidth and time thirsty (larger bandwidth and longer duration) Video services should not occupy much larger bandwidth than the existing voice like services. Raw video services contain a huge amount of data: Raw data rate of VHS quality video 37 Mb/s Therefore a sophisticated Video Compression tool is needed. Video coding standards Over the past 20 years the Video Coding Experts Group (VCEG) of ITU-T and the Motion Picture Experts Group (MPEG) of ISO/IEC have standardized many video codecs for various applications. They have worked either independently or Jointly. ITU-T Standards H.261 H.263 H.263+ H.263++ Joint ITU-T/ISO-IEC Standards H.262/ MPEG-2 H.264 (H.26L)/ MPEG-4v10 AVC ISO-IEC Standards MPEG-1 MPEG-4 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 4
Overview of H.264/AVC H.264/AVC is the latest video codec standard of the Joint Video Team (JVT) of ITU-T VCEG and ISO/IEC MPEG. It follows the generic standard codec, i.e. DCT, MC, Entropy Coding, etc. Compression efficiency H.264/AVC has achieved a significant improvement in compression efficiency over the previous standard video codecs. Foreman Test sequence, coded with four standard video codecs [Wiegand et al]. 5
Features that make H.264 attractive 4x4 integer transformation Advanced intra prediction modes Context based adaptive binary arithmetic coding In-loop deblocking filter Advanced inter prediction modes Quarter sample motion vectors Multiple reference selection Intra mode selection (4 4 intrapolation mode) 6
Inrta JVC/JPEG2000 (the intra frame prediction is even more powerful than the wavelet based image coding, used in JPEG2000) Y-PSNR (db) 47 45 43 41 39 37 35 33 31 29 27 25 H.264 QCIF H.264 CIF H.264 4CIF JP2K QCIF JP2K CIF JP2K 4CIF H.263 QCIF H.263 CIF H.263 4CIF 0 200 400 600 800 1000 Kbits Inter prediction Mode 7
Compression Performance H.264 versus H.263 which has a similar performance to Frame-based MPEG-4 H.263, 30 K-Bits/Sec H.264, 30 K-Bits/Sec Sensitivity to Errors Compressed video is very sensitive to errors. A single error, will spread both spatially and temporally damaging all the successive frames. The higher the Compression, the more sensitive is the compressed video to errors. Frame number 1 2 3 4 5... 8
CHALLENGES Noise There are several methods to decrease the impact of noise/errors on the compressed video: Error resilient Source Coding Channel Coding Error Concealment etc Error Resilience Several techniques have been tailored in the standard to make the bitstream robust to the errors: Resynchronization (slice) headers Data Partitioning (DP) Flexible Macroblock Ordering (FMO) Redundant slices Intra refresh Multiple reference selection 9
Layered coding Intra-Refresh and Multiple-Reference are effective, but need a back channel. In non-feedback video transmission, layered coding with transport prioritization is a very effective error resilience scheme. To improve the transmission efficiency, higher error protections are applied to the more important units of the coded data. Data Partitioning The only frame layering technique currently supported by H.264 is Data Partitioning (DP). In DP a slice is divided into three units according to the importance of the data. 10
40 Y-PSNR (db) 35 30 25 20 original error on frm 10 intra update at frm 11 data partitioned ref switching at frm 11 0 5 10 15 Frame Number (A) Loss at Frame 10 (B) Loss at Frame 10, updated at Frame 15 (C) DP: Loss at frame 10, and not corrected A B C 11
Unequal Error Protection (UEP) Higher priority (HP) bits should be better protected than the lower priority (LP) bits. This can be done with two methods: Turbo code: HP bits use stronger Forward Error Correcting (FEC) code than the LP bits. Modulation: HP bits use stronger SNR than the LP bits Hierarchical QAM In Hierarchical QAM HP symbols are further apart from each other than the LP symbols. (Example: HP/LP = 2/4 in a 64-HQAM) a α = b 12
Problem in UEP In HQAM the ratio of HP to LP symbols is constant. (e.g. HP/LP = 2/4 in a 64-HQAM) In DP, the ratio of HP to LP bits is not constant. For practical implementation this conflict must be resolved. Foreman@ 100 kb/s UEP With Channel Coding (Turbo code) The higher priority (HP) data should be protected by more redundancy (parity) bits than the lower priority (LP) parts. Since the HP/LP source bits ratio is variable, their Turbo coded rates, R H and R L, should vary accordingly. 13
With Multilevel HQAM, the distances between the symbols can vary to accommodate the HP/LP changes a α = b b β = c With Multilevel HQAM, α and β are adaptively adjusted according to the size of HP and LP. Relative Capacity Mode α β High Priority Low Priority 1. Non-hierarchical 1 1 6 bits (100%) 2. Hierarchical two-priorities 1 >1 4 bits (66%) 2 bits (33%) 3. Hierarchical two-priorities >1 1 2 bits (33%) 4 bits (66%) 14
Multilevel HQAM: BER vs. channel SNR Mode-2 (α=1, β=2) and Mode-1 Mode-3 (α:1.5, β:1) and Mode-1 UEP comparison between MHQAM/Turbo Code NOLAYER: QAM NOLAYER+TC: QAM + Turbo code for extra protection of entire data UEP ADPTTC: DP partitioned + adapting the parity bits in Turbo code + QAM UEP MHQAM+TC: DP partitioned + Multilevel HQAM in HP and LP + Turbo Code on entire data 36 Y PSNR (db) 31 26 21 16 NOLAYER NOLAYER+TC UEP ADPT TC UEP MHQAM+TC 15 20 25 30 35 40 45 50 CHANNEL SNR (db) 15
Simulation Results Increasing HP protection (e.g. 1/2 vs 4/7), will improve the performance at lower SNRs at the cost of lower performance at higher SNRs. 36 33 Y PSNR (db) 30 27 24 21 UEP ADPT TC RH=4/7 UEP ADPT TC RH=1/2 18 15 20 25 30 35 CHANNEL SNR (db) Scalability Another way of solving UEP problem of DP is scalable coding. Scalability is to partition a bitstream into layers such that the base layer gives a low quality video and the enhancement layer improves the quality. e.g. Quality (SNR), Spatial or Temporal resolutions. Although the first version of H.264 does not support scalability, it is listed on the work plan as an important tool that should be supported by the standard. 16
HP/LP changes in Scalability/DP HP bit rate, % total 80 70 60 50 40 30 20 Standard deviation data partitioning scalability, base 50% scalability, base 33% scalability, base 25% 10 0 50 100 150 200 overal bit rate (kbps) R-D performance of the non-scalable and the scalable coders for three values of dqp (= QPB-QPE). PSNR-Y (db) 41 40 39 38 37 36 35 34 33 32 31 30 Foreman QCIF@10Hz non-scalable scalable dqp:15 scalable dqp:10 scalable dqp:5 25 50 75 100 125 150 175 200 Bit Rate (kb/s) 17
PSNR vs. channel SNR, scalable and data-partitioned with equal error protection (QAM) and unequal error protection using hierarchical QAM α: 4 (H-QAM) 42 Foreman QCIF@10Hz, 200 Kb/s Y PSNR (db) 38 34 30 26 22 18 DP (QAM) Scal (QAM) DP (H-QAM) Scal (H-QAM) 18 21 24 27 30 channel SNR (db) Loss concealment If despite all these errors are still present, then they can be concealed (hidden). There are several ways to hide the error Copy from the adjacent pixels (intraframe) Copy from the previous frame (Interframe) Copy from motion compensated pixels, by an estimated motion vector Top or bottom MV Average of all MV Median of MV MV that gives the best continuity 18
CHALLENGES Noise Examples of some Error Concealment Methods; A Corrupted Sequence Concealed using simple copy Concealed using an advanced method CONCLUSIONS In the very early days of 21 th century, video transmission over the mobile handsets has become a reality! With continuing advances in video coding and reducing the transmission cost, usage of mobile video will become even more popular. 19