MPEG-2. Lecture Special Topics in Signal Processing. Multimedia Communications: Coding, Systems, and Networking

Transcription

1 1-99 Special Topics in Signal Processing Multimedia Communications: Coding, Systems, and Networking Prof. Tsuhan Chen Lecture 7 MPEG-2 1

2 Outline Applications and history Requirements New features Test Model 5 Bitstream syntax Profiles and Levels Scalability MPEG-2 MPEG-2 Video ISO/IEC (or ITU-T H.262) Broadcast TV, cable/satellite TV, HDTV, video services on networks (e.g., ATM) 4~9 Mbits/s, interlaced video, and scalable coding History Late 1990: started Nov 1991: competitive tests of video Collaborative phase Nov 1993: Committee Draft for video 2

3 Parts of MPEG-2 Parts: ISO/IEC 131-1: Systems ISO/IEC 131-2: Video ISO/IEC 131-3: Audio ISO/IEC 131-4: Compliance Testing ISO/IEC 131-5: Software ISO/IEC 131-6: DSM-CC ISO/IEC 131-7: NBC Audio ISO/IEC 131-: 10-Bit Video (dropped!) ISO/IEC 131-9: Real-Time Interface ISO/IEC : DSM-CC Conformance Requirements ITU-R 601 interlaced video with high quality at 4~9 Mbits/s Random access/channel switching, seek and play in FF/FR using access points Allow video coding higher chroma resolution formats, e.g., 4:2:2 and 4:4:4 Scalable video coding for multi-quality video applications System supporting audio-visual synchronized play/access for multiple streams Subset of the standard implementable as practical decoders 3

4 Additional Requirements Maximum interoperability/compatibility with MPEG-1 Support coding of non-interlaced and interlaced formats of many frame rates Support video formats of various aspect ratios Low overhead syntax while supporting above requirements for overall efficiency Subset of the standard permits real-time encoder of reasonable complexity New Features Allows 4:2:2 and 4:4:4 formats Frame-s and field-s Frame/field adaptive DCT Frame/field/dual-prime adaptive motion compensation Alternate scan for DCT coefficients New VLC table for DCT coefficients Profiles and levels Nonlinear quantization table Increased accuracy for small values 4

5 Additional New Features Chrominance samples horizontally co-sited as luminance samples Slices always start and end at the same row of macroblocks Concealment motion vectors for intra macroblocks Motion vectors always coded in half-pel units Display aspect ratio specified in bitstream Pel aspect ratio derived from it IDCT mismatch control Coefficient VLC table escape format not allowed if use of shorter VLC possible Slice and Macroblock GOP Slice Y Y Y Y C B C R Macroblock (MB) 5

6 Slice Structure general restricted Chrominance Sampling 4:2:0 4:2:2 4:4:4 6

7 Chrominance Sampling (cont.) top field bottom field top field bottom field tim e time time interlaced 4:2:0 interlaced 4:2:2/4:4:4 progressive Coding of Interlaced Video Frame-s and field-s Motion compensation Frame prediction for frame-s Same as MPEG-1 Field prediction for field-s Field prediction for frame-s Dual-prime for P-s field-s or frame-s 16 MC for field s 7

8 Field Prediction for Field-s Field s Ref Pred Field Prediction for Frame-s 16 field blocks Frame Macroblock 16 ME & MC Prediction from either field of the previous frame Good for fast motion

9 Dual-Prime Prediction ref pred PSNR at 4 Mbits/s Frame-s, M=1 Performance Sequence Frame MC Field MC Frame/Field MC Dualp MC Frame/ Fi el d/ Dual p MC Flowergarden (+0.34) 2.22 (+0.50) 2.39 (+0.67) 29.3 (+1.66) Mobile & Cal (+0.17) (+0.35) (-0.1) (+0.94) Football (+1.40) (+1.49) (+1.49) (+1.4) Bus (+1.27) (+1.44) (+1.71) (+2.32) Carousel (+1.30) (+1.40) (+1.32) (+1.6) Frame-s, M=3 Sequence Frame MC Field MC Frame/Field MC Flowergarden (+0.13) (+0.56) Mobile & Cal (-0.25) 2.27 (+0.16) Football (+0.47) (+0.5) Bus (+0.53) (+0.1) Carousel (+0.32) (+0.51) Field-s, M=1 Sequence Field MC 16x MC Field/16x MC Flowergarden (-1.05) 27.1 (+0.19) Mobile & Cal (-1.41) (+0.19) Football (-1.00) 35.9 (-0.1) Bus (-0.7) 29.3 (+0.20) Carousel (-1.01) (+0.12) 9

10 Frame/Field Adaptive DCT Organize 16x16 block as frame blocks or filed blocks Compute correlation in vertical direction in each case Choose the case that has higher correlation Frame blocks Field blocks PSNR at 4 Mbits/s M=1 Performance Sequence Frame DCT Field DCT Frame/Field DCT Flowergarden (-0.32) 29.3 (+0.02) Mobile & Cal (-0.79) (-0.03) Football (+0.41) (+0.50) Bus (-0.05) (+0.26) Carousel (+0.6) (+0.5) M=3 Sequence Frame DCT Field DCT Frame/Field DCT Flowergarden (-0.15) (+0.02) Mobile & Cal (-0.60) 2.27 (-0.07) Football (+0.37) (+0.45) Bus (+0.07) (+0.26) Carousel (+0.55) (+0.69) 10

11 DCT Coefficients Scan DC DC Zigzag scan inter DCT Alternate scan intra DCT Sequence Zigzag Scan Alternate Scan Flowergarden (+0.25) Mobile & Cal (+0.04) Football (+0.30) Bus (+0.22) Carousel (+0.11) EOB = 2 bit s R u n MPEG-1 subt a ble Ab s o lu t e Le ve l D VLC (Inter) MPEG-1 re m ainde r t able bit Fixed Leng t h Code s Es c ape ( 6b its ) +Run( 6bits) +Le v el(1 2b it s)

12 Intra subtable MPEG-1 remainder table EOB = 4 bits Absolute Level R u n D VLC (Intra) 24 bit Fixed Length Codes Escape(6bits)+Run(6bits)+Level(12bits) D VLC (cont.) FLC table for runs and levels Following the escape code of a VLC FLC codeword run FLC codeword signed_level reserved not allowed

13 Range of Differential DC (DIFFs) to to to to to to to to - -7 to -4-3 to to 3 4 to 7 to to to to to to to to 204 Range larger then MPEG1 VLC for Differential DC SIZE SIZE VLC Luminance 9*1 *1 0 7*1 0 6*1 0 5*1 0 4* *1 0 5*1 0 6*1 0 7*1 0 *1 0 9*1 SIZE VLC Chrominance 9*1 1 9*1 0 *1 0 7*1 0 6*1 0 5*1 0 4* *1 0 5*1 0 6*1 0 7*1 0 *1 0 9*1 0 9*1 1 VLIs 9*0 00 to 0 9*1 1 9*0 0 to 0 9*1 9*0 to 0 *1 *0 to 0 7*1 7*0 to 0 6*1 6*0to 0 5*1 5*0 to 0 4*1 4*0 to to to to to to 4*1 1 4*0 to 5*1 1 5*0 to 6*1 1 6*0 to 7*1 1 7*0 to *1 1 *0 to 9*1 1 9*0 to 9* *0 0 to 9*1 11 GOP with M=1 and 3 Test Model (TM) 5 Motion estimation Frame/field/dual prime Integer-pel full search followed by half-pel update Mode decision: MC/no MC, inter/intra Zigzag scan for inter; alternate scan for intra Quantization adaptation and rate control 13

14 Motion Estimation Motion estimation on luminance blocks Chrominance motion vectors by dividing luminance motion vectors and truncating Half-pel update on integer motion vectors 204 to pels for half-pel motion vectors Depending on motion modes and types: Frame motion vectors Field motion vectors Motion vectors in forward direction Motion vectors in backward direction Encoder Inter/Intra Classifier inter/ intra Quantization Adapter quantizer_scale mot type type mc/no mc mot mode video in - SW Prediction Frame/Field Formatter dct_type DCT inter/ intra Forw'd Quant Inverse Quant IDCT VLC Encoder and Multiplexer dct_type mv f, mv b quantizer_scale inter/ intra Buffer bits out inter/ intra cod type Frame/Field Unformatter "0" SW mot type mc/no mc mot mode Motion Type Classifier mot type mc/no mc mot mode Frame/Field/Dualprime Motion Compensated Predictor mv f, mv b Frame/Field/Dualprime Motion Estimator Future Store Previous Store type SW 14

15 Decoder type mc/no mc mot type mot mode inter/ intra mquant cod type bits in Buffer VLC Decoder and Demultiplexer Inverse Quant IDCT Frame/Field Unformatter video out mquant inter/ intra cod type mv f, mv b "0" SW inter/ intra type SW Next Store Previous Store mot type mc/no mc mot mode Frame/Field/Dualprime Motion Compensated Predictor mv f, mv b Syntax Syntax Header Sequence Group of s Slice Macroblock Block Functionality Definition of Entire Video Sequence Enables Random Access in Video Stream Primary Coding Unit Resyncrnization, Refresh and Errror Recovery Motion Compensation Unit Transform and Compression Unit Sequence Header Sequence Extension Extension and user Data GOP Header Extension and user Data Header Coding Extension Extension and user Data Data Sequence End Sequence Header MPEG-1 User Data GOP Header User Data Header User Data Data Sequence End Sequence Header 15

16 Syntax (cont.) Sequence Header sequence header code horiz size vertical size aspect ratio rate bit rate constr paramtr load intra matrix intra matrix load non intra matrix non intra matrix Sequence Extension extension start code extension start code id profile_ level id prog sequence chroma format hor_size ext vert_size ext bitrate ext marker bit Sequence Display Extension vbv_ buf_size ext low_delay frame_ rate_ext_n frame_ rate_ext_d extension start code id video format color description color primaries transfer charact. matrix coeffs display hor_size marker bit display vert_size GOP Header group start code time code closed GOP Flag broken link Flag Layer start code temporal reference coding type buffer fullness full pel forw forw f code full pel backw backw f code extra bit extra info User and Ext Data slice layer Syntax (cont.) Coding Extension extension start code extension start code id for_hor_ f_code for_vert_ f_code bac_hor_ f_code bac_vert_ f_code intra_dc_ prec. structure top_field _first frm_pred_ frm_dct conceal_ mot_vec qnt_scl_ type intra_ vlc_format alternate scan rpt_first_ field chroma_ 420_type prog_ frame composite display v_axis field sequence sub_ carrier burst_ amp. sub_ carrier phase Slice Layer (Incomplete/Incorrect) slice start code quant scale extra bit extra info macroblock layer Macroblock Layer (Incomplete/Incorrect) macroblock type dmvf dmvb coded block pattern block layer Block Layer (Incomplete/Incorrect) DCT coefficient end of block 16

17 Profiles and Levels Level HIGH HIGH MAIN LOW 720 pels/line 576 lines/frame 30 frames/s 10.4 Msample/s 15 Mbit/s 1920 pels/line 1152 lines/frame 60 frames/s 62.7 Msamples/s 0 Mbit/s 1440 pels/line 1152 lines/frame 60 frames/s 47.0 Msamples/s 60 Mbit/s 720 pels/line 576 lines/frame 30 frames/s 10.4 Msample/s 15 Mbit/s 352 pels/line 2 lines/frame 30 frames/s 3.04 Msamples/s 4 Mbit/s 720 pels/line 576 lines/frame 30 frames/s 10.4 Msample/s 15 Mbit/s for 2 layers 352 pels/line 2 lines/frame 30 frames/s 3.04 Msamples/s 4 Mbit/s for 2 layers 1440 pels/line 1152 lines/frame 60 frames/s 47.0 Msamples/s 60 Mbit/s for 3 layers 1920 pels/line 1152 lines/frame 60 frames/s Msamples/s * 100 Mbit/s for 3 layers 1440 pels/line 1152 lines/frame 60 frames/s Msamples/s * 0 Mbit/s for 3 layers 720 pels/line 576 lines/frame 30 frames/s Msamples/s * 20 Mbit/s for3 layers SIMPLE nonscalable 4:2:0 (no B- s) MAIN nonscalable 4:2:0 SNR scalable 4:2:0 SPATIAL scalable 4:2:0 HIGH nonscalable 4:2:2 scalable 4:2:0/4:2:2 * refers to refers to 4:2:2 Profile Data Partitioning SNR Scalability Spatial Scalability Temporal Scalability Hybrid Scalability Scalability Types 17

18 Temporal Scalability Example 1 structure for base layer with 2 B-frames. Enhancement layer uses either simple prediction or bidirectional prediction from the base layer. Stereoscopic scalability The base layer is the video for one eye. The enhancement layer is the video for the other eye Example 2 The base layer is a normal TV signal having 30 fps The enhancement layer provides a compatible upgrade to 60 fps. Both progressive and interlaced 60 fps are possible Bit Rate (Mbits/sec) MPEG Average Quality SIF-30 ~CVGA CCIR FPS ~VGA HDTV FPS HDTV 60 FPS ~SVGA 1.1 Mbs good poor 4.0 Mbs excellent good 9.0 Mbs excellent++ excellent 1.0 Mbs excellent++ good good 2.0 Mbs excellent excellent SIF-30 ~CGA CCIR FPS ~VGA HDTV FPS HDTV 60 FPS ~SVGA Pels Lines Uncompressed Bit Rates (Mbps)

19 References Joan L. Mitchell et al., MPEG Video: Compression Standard, Chapman & Hall, New York, NY Barry G. Haskell, Atul Puri, Arun N. Netravali, Sec 17.1, Digital Video : An Introduction to MPEG-2, Chapman & Hall, New York, NY 19