Lecture 2 Video Formation and Representation

Transcription

1 Wen-Hsiao Peng, Ph.D Multimedia Architecture and Processing Laboratory (MAPL) Department of Computer Science, National Chiao Tung University February 2008 Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

2 Video Signal Lecture 2 Video Formation and Representation Video Representation 2-D images projected from a 3-D scene onto image plane Artist Albrecht Durer's Perspective Projection Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

3 Color Video Camera Video Representation Digital Output Rate (13.5MHz) Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

4 Image Sensors Video Representation CCD/CMOS Sensors 25% R, 50% G, 25% B 33% R, 33% G, 33% B Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

5 Video Representation Progressive and Interlaced Scan K : 1 Interlaced Scan Adjacent lines are sampled in separate time Trade-o between Vertical and Temporal resolutions Analog (Progressive vs. Interlaced) Digital (Interlaced) Analog - consecutive lines captured at slightly dierent times Digital - pixels of a frame sampled at the same time Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

6 Progressive vs Interlaced Video Representation Progressive Interlaced Demo Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

7 Analog Video Raster Video Representation Frame Rate f s,t Line Number f s,y Horizontal Retrace Time T h Vertical Retrace Time T v Line Rate f l = f s,t f s,y Line Interval T l = 1/f l Frame Interval t = 1/f s,t Line Scanning Time Tl 0 = T l Active Lines fs,y 0 = ( t T v ) /T l T h 1-D Raster Signal Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

8 Video Representation Video Raster Spectrum Periodic (Period = T l ) because of similarity between adjacent lines Lobe width reects Vertical Spatial Bandwidth f max indicates Horizontal Spatial Bandwidth 1-D Raster Signal Spectrum Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

9 Video Representation Signal Bandwidth Overall Bandwidth f max Maximum Vertical Frequency f v,max f v,max = Kf 0 s,y 2 Maximum Horizontal Frequency f h,max f h,max = Kf 0 s,y 2 Maximum Bandwidth in 1-D Raster NTSC TV System - f max = 4.2MHz (Cycles/Height) Width Height (Cycles/Width) f max = f h,max Tl 0 (Hz) T 0 l = 53.5us, f 0 s,y = 483, K = 0.7, Width/Height = 4/3 Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

10 Analog Color TV Systems Video Representation Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

11 Video Representation Digital Video Frame Rate f s,t Line Number f s,y Samples per Line f s,x Bit Depth (Y) N b Image Aspect Ratio (IAR) Width/Height Pixel Aspect Ratio (PAR) x / y Physical area w.r.t. a pixel Ratio for proper render Vertical Interval y = Height/f s,y Horizontal Interval x = Width/f s,x Row Data Rate f s,t f s,y f s,x N b PAR = x / y = IAR f s,y /f s,x Example: IAR = 4/3, f s,y /f s,x = 720/480! PAR = 8/9 Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

12 ITU-R BT.601 (CCIR601) Video Representation Digital format for interlaced analog video signals Sampling Rate f s = f s,x f s,y f s,t = f s,x f l x y f s = IAR f 2 s,y f s,t = 11 (NTSC) or 13 (PAL) MHz Same f s for NTSC and PAL/SECAM f s = f s,x (NTSC ) f {z } l (NTSC ) = f s,x (PAL) f {z } {z } l (PAL) = 13.5 MHz {z }?= /60 eld/s?= /50 eld/s Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

13 Digital Video Format Video Representation Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

14 Video Representation Common Intermediate Format (CIF) QCIF 288 CIF 576 4CIF QCIF < CIF/SIF < 4CIF/BT.601 Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

15 High Denition (HD) Format Video Representation Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

16 MPEG Video Coding Standards MPEG Standards MPEG-1 Error free storage Progressive Video: Mbps Audio: MP3 192~256kbps MPEG-4 Media streaming, Frame-/Objectbased coding + Scalability MPEG-4 Part 10 Advanced video Coding (also known as H.264) MPEG-21 Multimedia Framework IP Management and Protection (IPMP) x Timeline MPEG-2 Broadcast TV Progressive/Interlaced Audio: AAC MPEG-7 Multimedia content description Not to define coding method MPEG-4 Amd. Scalable video Coding MPEG-4 Amd. Multi-View video Coding Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

17 Advances in Coding Eciency MPEG Standards HD in H.264/AVC Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

18 Lecture 2 Video Formation and Representation MPEG Standards MPEG Applications MPEG-4 AVC/H.264 Broadcasting Server Point-to-Point Transmission 128 kbps 64 kbps Wireless MPEG-1 Block-based Video Coding (VCD) 32 kbps 512 kbps 384 kbps Wireless Ethernet 256 kbps Router 1.5 Mbps 64 kbps Ethernet 3 Mbps Bandwidth Time MPEG-2 Block-based Video Coding with Interlaced tools (DVD) Scalable Video Coding MPEG-4 Object-based Video Coding Multi-View Video Coding for 3DTV Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

19 MPEG Standardization Activities MPEG Standards A. Exploration 1.The search for new technology 2.Seek Industry experts 3.Open seminars F. Amendment Adding new technology G. Corrigenda Corrective actions H. New subdivisions Add new non-compatible technology B. Requirements 1. Establish the scope of work 2. Call for Proposals E. Standardization 1. Ballots 2. National Body Comments C. Competitive phase 1. Do Homework 2. Response to CfP 3. Initial technology selection D. Collaborative phase 1. Core Experiments 2. Working Drafts Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

20 MPEG Standardization Activities MPEG Standards Activities 6-12 months 6-12 months 3-6 months Search for new technology Define requirements of spec. Initial tech. selection 1 year Call-for- Evidence WD: Working Draft CD: Committee Draft FCD: Final Committee Draft FDIS: Final Draft of International Standard IS: International Standard Core Experiments ~3 years Call-for- Proposals 3 months 3 months 3 months 3 months WD CD (PDAM) ~2 years FCD (FPDAM) FDIS (FDAM) IS (AMD) Time Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

21 MPEG Week Lecture 2 Video Formation and Representation MPEG Standards Work, Work, and Work! 9:00am noon Monday Tuesday Wednesday Thursday Friday Opening Plenary Mid-week Plenary Subgroup Plenary 6:00pm Subgroup Plenary Technical Sessions/Joint Meetings 2pm Closing Plenary Social Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

22 MPEG Week Lecture 2 Video Formation and Representation MPEG Standards MPEG is Good at Pruning Out Bad Ideas! Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

23 Color Coordinates Color Spaces Luminance and Chrominance Mixture RGB (Display) & CMY (Printing, Hard Copies) Separate Luminance and Chrominance XYZ - Fundamental Measurements YUV - PAL Color TV YIQ - NTSC Color TV 1 YCbCr - Digital Video HSI - Hue, Saturation, Brightness Nonlinearly related to tristimulus values 1 QAM: s(t) = I (t) cos 2πf c t + Q(t) sin 2πf c t = A(t) cos(cos 2πf c t θ(t)), A(t) = p I (t) 2 + Q 2 (t)/y approximates Saturation while θ(t) = tan 1 Q(t)/I (t) represents Hue (Better Protection with QAM) Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

24 Color Coordinates Color Space Transformation Rational Better representation for processing or data reduction (R, G, B) $ (C, M, Y ) 2 4 C M Y 3 5 = R G B R G B 3 5 = C M Y 3 5 Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

25 Color Space Transformation Color Coordinates (R, G, B)! (Y, B Y, R Y ) Y is Luminance Component B Y, R Y are Chrominance Components Y B Y 5 = R 3 G 5 R Y B (R, G, B)! (Y, U, V ) U = 0.492(B Y ), V = 0.877(R Y ) Y U V 5 = R G B 3 5 Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

26 Color Space Transformation Color Coordinates (R, G, B)! (Y, I, Q) 2 Y I Q 5 = R G B (Y, U, V )! (Y, I, Q) (Q, I ) is counterclockwise rotated version (33 ) of (V, U) Y Q 5 = Y 3 V 5 I U x 0 y 0 = cos θ sin θ sin θ cos θ x y (x 0, y 0 ) are the coordinates of a xed vector with respect to the rotated (counterclockwise) coordinate system while (x, y) are the original coordinates. Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

27 Color Space Transformation Color Coordinates (R, G, B)! (Y, Cb, Cr) 2 Y Cb 5 = Cr R G B R, G, B take integer values from 0 to 255 Cb, Cr are scaled, shifted versions of U, V Remarks Brightness contributions G > R > B Y = {z } R {z } G {z } B (2) (1) (3) Sum of rst row = 1 while sum of the other two rows = 0 If G = R = B = k, Then Y = k, Cb/U/I = 0, Cr/V /Q = 0 Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

28 RGB vs. YCbCr Color Coordinates R G B Y Cb Cr Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

29 Color Coordinates Chrominance Subsampling Human vision is less sensitive to color than to luminance 4:4:4 Progressive Interlaced Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

30 Color Coordinates Chrominance Subsampling 4:2:2 Progressive Interlaced 4:2:0 Progressive Interlaced Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

31 Chrominance Subsampling Color Coordinates 4:4:4 4:2:0 Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

32 Video Quality Measure Objective Measure Mean Squared Error (MSE) MSE = σ 2 e = 1 N (Ψ 1 (m, n, k) Ψ 2 (m, n, k)) 2 k m,n N is total number of pixels Mean Absolute Dierence (MAD) MAD = 1 N jψ 1 (m, n, k) k m,n Peak Signal-to-Noise Ratio (PSNR) Sequence Average PSNR PSNR = 10 log σ 2 e PSNR = 1 K PSNR k Ψ 2 (m, n, k)j PSNR = PSNR i MSE is the same for each frame Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

33 PSNR/MSE Lecture 2 Video Formation and Representation Video Quality Measure Excellent (>40), Good (30{40), Poor (<30) Original Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

34 PSNR/MSE Lecture 2 Video Formation and Representation Video Quality Measure Do not correlate well with human perception Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

35 Video Quality Measure Subjective Measure Mean Opinion Score Methods Cons Viewer's judgement of quality Double Stimulus Continuous Quality Scale (DSCQS) Reference vs. Impaired in a randomized order Relative quality of impaired and reference Double Stimulus Impairment Scale (DSIS) Reference vs. Impaired in a predened order Single Stimulus Continuous Quality Evaluation (SSCQE) Absence of an unimpaired reference Assess quality in a continuous manner Contextual eect - bias by two subsequent conditions Recency eect - bias by recent memory Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

36 Subjective Measure Video Quality Measure DSCQS DSIS SSCQE Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37

37 Video Quality Measure References 1 Yao Wang, et. al - Video Processing and Communications 2 D. Miras - On Quality Aware Adaptation of Internet Video, Ph.D Dissertation 3 T. Wiegand - Scalable Video Coding, JVT-W132 Wen-Hsiao Peng, Ph.D (NCTU CS) MAPL February / 37