Lecture 2 Video Formation and Representation

Transcription

1 Wen-Hsiao Peng, Ph.D. Multimedia Architecture and Processing Laboratory (MAPL) Department of Computer Science, National Chiao Tung University March 2013 Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

2 Video Signals Lecture 2 Video Formation and Representation Video Representation A sequence of 2-D images captured from the projection of a 3-D scene onto an image plane Artist Albrecht Durer s Perspective Projection Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

3 Color Video Camera Video Representation Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

4 Image Sensors Video Representation CCD/CMOS Sensors 25% R, 50% G, 25% B 100% R, 100% G, 100% B Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

5 Lecture 2 Video Formation and Representation Video Representation Progressive and Interlaced Sampling K : 1 interlaced sampling Adjacent lines are sampled at separate times Trade-o between Vertical and Temporal resolutions Analog (Progressive vs. Interlaced) Digital (Interlaced) Analog - consecutive lines captured at slightly di erent times Digital - pixels of a frame (or eld) sampled at the same time Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

6 Progressive vs Interlaced Video Representation Progressive Interlaced Demo Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

7 Analog Video Raster Video Representation Frame Rate f s,t Line Number f s,y Horizontal Retrace Time T h Vertical Retrace Time T v Line Rate f l = f s,t f s,y Line Interval T l = 1/f l Frame Interval t = 1/f s,t Line Scanning Time Tl 0 = T l Active Lines fs,y 0 = ( t T v ) /T l T h 1-D Raster Signal Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

8 Video Representation Video Raster Spectrum Periodic (Period = T l ) because of similarity between adjacent lines Lobe width re ects vertical spatial bandwidth f max indicates horizontal spatial bandwidth 1-D Raster Signal Spectrum Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

9 Video Representation Signal Bandwidth Overall bandwidth f max Maximum vertical frequency f v,max f v,max = Kf 0 s,y 2 (Cycles/Height) Maximum horizontal frequency f h,max f h,max = Kf 0 s,y 2 Maximum bandwidth in 1-D raster NTSC TV system - f max = 4.2MHz Width Height (Cycles/Width) f max = f h,max Tl 0 (Hz) T 0 l = 53.5us, f 0 s,y = 483, K = 0.7, Width/Height = 4/3 Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

10 Analog Color TV Systems Video Representation Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

11 Video Representation Digital Video Frame Rate f s,t Line Number f s,y Samples per Line f s,x Bit Depth (Y) N b Image Aspect Ratio (IAR) Width/Height Pixel Aspect Ratio (PAR) x / y Vertical Interval y = Height/f s,y Horizontal Interval x = Width/f s,x Row Data Rate f s,t f s,y f s,x N b The ratio of width to height of a physical, rectangular area for rendering a pixel PAR = x / y = IAR f s,y /f s,x Example: IAR = 4/3, f s,y /f s,x = 480/720! PAR = 8/9 Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

12 ITU-R BT.601 (CCIR601) Video Representation Digital format for interlaced analog video signals Sampling rate f s = f s,x f s,y f s,t = f s,x f l Similar horizontal & vertical sampling intervals, x y f s = IAR f 2 s,y f s,t = 11 (NTSC) or 13 (PAL) MHz Same f s for NTSC and PAL/SECAM f s = f s,x (NTSC ) f {z } l (NTSC ) = f s,x (PAL) f {z } {z } l (PAL) = 13.5 MHz {z }?= ?= Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

13 Digital Video Format Video Representation Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

14 Video Representation High De nition and Ultra High De nition Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

15 Color Space Lecture 2 Video Formation and Representation Color Coordinates Mixture of luminance and chrominance RGB & CMY primaries Separation of luminance and chrominance XYZ - fundamental measurements YUV - PAL YIQ - NTSC YCbCr - digital video HSI - hue, saturation, brightness Color space conversion better representation for processing or data reduction Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

16 Color Coordinates Color Space Conversion (R, G, B)! (Y, U, V ) Y : brightness U = 0.492(B Y ), V = 0.877(R Y ): color di erences 2 4 Y U V = Brightness contribution (G > R > B) 3 2 Y = {z } R {z } G {z } B (2) (1) (3) 5 4 R G B Coe s. of the rst row add up to 1, those of the other rows to 0 R = G = B ) gray images (U = V = 0) 3 5 Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

17 Color Space Conversion Color Coordinates Y, Cb, Cr: scaled, shifted versions of Y, U, V 2 Y Cb 5 = Cr R G B R, G, B take values in Y takes values in , Cb, Cr in Part of the BT.601 standard. Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

18 RGB vs. YCbCr Color Coordinates R G B Y Cb Cr Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

19 Color Coordinates Chrominance Subsampling Human vision is less sensitive to color than to luminance 4:4:4 Progressive Interlaced Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

20 Color Coordinates Chrominance Subsampling 4:2:2 Progressive Interlaced 4:2:0 Progressive Interlaced Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

21 Chrominance Subsampling Color Coordinates 4:4:4 4:2:0 Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

22 Objective Quality Measure Video Quality Measure Mean Squared Error (MSE) MSE = σ 2 e = 1 N (Ψ 1 (m, n, k) Ψ 2 (m, n, k)) 2 k m,n m, n pixel coordinates; k frame index; N total number of pixels. Peak Signal-to-Noise Ratio (PSNR) PSNR = 10 log 10 ψ 2 max σ 2 e (db) ψ max the maximum (or peak) signal value; 255 for 8-bit video. Average per-frame PSNR PSNR = 1 K k PSNR k k frame index; K number of frames compared. Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

23 PSNR & MSE Video Quality Measure Excellent (>40), Good (30 40), Poor (<30) Original Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

24 PSNR & MSE Video Quality Measure Both do not correlate well with human perception Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

25 Subjective Quality Evaluation Video Quality Measure Mean opinion score Human observers judgement of visual quality Methods (de ned in ITU-R BT ) Double Stimulus Continuous Quality Scale (DSCQS) Reference and impaired sequences graded in a randomized order. Scores converted into a number indicating the relative quality. Double Stimulus Impairment Scale (DSIS) Reference rst and then the impaired sequence. Grade the impaired sequence relative to its reference. Single Stimulus Continuous Quality Evaluation (SSCQE) Grade the impaired sequence in a continuous manner without a reference. Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

26 Subjective Quality Evaluation Video Quality Measure DSCQS DSIS SSCQE Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

27 MPEG Video Coding Standards MPEG Standards MPEG 1 Error free storage Progressive Video: 352x240@1.5 Mbps Audio: MP3 192~256kbps MPEG 4 Media streaming, Frame /Objectbased coding + Scalability MPEG 4 Part 10 Advanced video Coding (also known as H.264) MPEG 21 Multimedia Framework IP Management and Protection (IPMP) x Timeline MPEG 2 Broadcast TV Progressive/Interlaced Audio: AAC MPEG 7 Multimedia content description Not to define coding method MPEG 4 Amd. Scalable video Coding MPEG 4 Amd. Multi View video Coding Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

28 Advances in Coding E ciency MPEG Standards HD in H.264/AVC Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

29 Lecture 2 Video Formation and Representation MPEG Standards MPEG Applications MPEG-4 AVC/H.264 Broadcasting Server Point-to-Point Transmission 128 kbps 64 kbps Wireless MPEG-1 Block-based Video Coding (VCD) 32 kbps 512 kbps 384 kbps Wireless Ethernet 256 kbps Router 1.5 Mbps 64 kbps Ethernet 3 Mbps Bandwidth Time MPEG-2 Block-based Video Coding with Interlaced tools (DVD) Scalable Video Coding MPEG-4 Object-based Video Coding Multi-View Video Coding for 3DTV Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

30 MPEG Standardization Activities MPEG Standards A. Exploration 1.The search for new technology 2.Seek Industry experts 3.Open seminars F. Amendment Adding new technology G. Corrigenda Corrective actions H. New subdivisions Add new non compatible technology B. Requirements 1. Establish the scope of work 2. Call for Proposals E. Standardization 1. Ballots 2. National Body Comments C. Competitive phase 1. Do Homework 2. Response to CfP 3. Initial technology selection D. Collaborative phase 1. Core Experiments 2. Working Drafts Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

31 MPEG Standardization Activities MPEG Standards Activities 6 12 months 6 12 months 3 6 months Search for new technology Define requirements of spec. Initial tech. selection 1 year Call for Evidence WD: Working Draft CD: Committee Draft FCD: Final Committee Draft FDIS: Final Draft of International Standard IS: International Standard Core Experiments ~3 years Call for Proposals 3 months 3 months 3 months 3 months WD CD (PDAM) ~2 years FCD (FPDAM) FDIS (FDAM) IS (AMD) Time Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

32 MPEG Week Lecture 2 Video Formation and Representation MPEG Standards Work, Work, and Work! 9:00am noon Monday Tuesday Wednesday Thursday Friday Opening Plenary Mid week Plenary Subgroup Plenary 6:00pm Subgroup Plenary Technical Sessions/Joint Meetings 2pm Closing Plenary Social Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

33 MPEG Week Lecture 2 Video Formation and Representation MPEG Standards MPEG is Good at Pruning Out Bad Ideas! Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34

34 MPEG Standards References 1 Yao Wang, et. al - Video Processing and Communications 2 D. Miras - On Quality Aware Adaptation of Internet Video, Ph.D Dissertation 3 T. Wiegand - Scalable Video Coding, JVT-W132 Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March / 34