Wen-Hsiao Peng, Ph.D. Multimedia Architecture and Processing Laboratory (MAPL) Department of Computer Science, National Chiao Tung University March 2013 Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 1 / 34
Video Signals Lecture 2 Video Formation and Representation Video Representation A sequence of 2-D images captured from the projection of a 3-D scene onto an image plane Artist Albrecht Durer s Perspective Projection Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 2 / 34
Color Video Camera Video Representation Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 3 / 34
Image Sensors Video Representation CCD/CMOS Sensors 25% R, 50% G, 25% B 100% R, 100% G, 100% B Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 4 / 34
Lecture 2 Video Formation and Representation Video Representation Progressive and Interlaced Sampling K : 1 interlaced sampling Adjacent lines are sampled at separate times Trade-o between Vertical and Temporal resolutions Analog (Progressive vs. Interlaced) Digital (Interlaced) Analog - consecutive lines captured at slightly di erent times Digital - pixels of a frame (or eld) sampled at the same time Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 5 / 34
Progressive vs Interlaced Video Representation Progressive Interlaced Demo Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 6 / 34
Analog Video Raster Video Representation Frame Rate f s,t Line Number f s,y Horizontal Retrace Time T h Vertical Retrace Time T v Line Rate f l = f s,t f s,y Line Interval T l = 1/f l Frame Interval t = 1/f s,t Line Scanning Time Tl 0 = T l Active Lines fs,y 0 = ( t T v ) /T l T h 1-D Raster Signal Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 7 / 34
Video Representation Video Raster Spectrum Periodic (Period = T l ) because of similarity between adjacent lines Lobe width re ects vertical spatial bandwidth f max indicates horizontal spatial bandwidth 1-D Raster Signal Spectrum Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 8 / 34
Video Representation Signal Bandwidth Overall bandwidth f max Maximum vertical frequency f v,max f v,max = Kf 0 s,y 2 (Cycles/Height) Maximum horizontal frequency f h,max f h,max = Kf 0 s,y 2 Maximum bandwidth in 1-D raster NTSC TV system - f max = 4.2MHz Width Height (Cycles/Width) f max = f h,max Tl 0 (Hz) T 0 l = 53.5us, f 0 s,y = 483, K = 0.7, Width/Height = 4/3 Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 9 / 34
Analog Color TV Systems Video Representation Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 10 / 34
Video Representation Digital Video Frame Rate f s,t Line Number f s,y Samples per Line f s,x Bit Depth (Y) N b Image Aspect Ratio (IAR) Width/Height Pixel Aspect Ratio (PAR) x / y Vertical Interval y = Height/f s,y Horizontal Interval x = Width/f s,x Row Data Rate f s,t f s,y f s,x N b The ratio of width to height of a physical, rectangular area for rendering a pixel PAR = x / y = IAR f s,y /f s,x Example: IAR = 4/3, f s,y /f s,x = 480/720! PAR = 8/9 Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 11 / 34
ITU-R BT.601 (CCIR601) Video Representation Digital format for interlaced analog video signals Sampling rate f s = f s,x f s,y f s,t = f s,x f l Similar horizontal & vertical sampling intervals, x y f s = IAR f 2 s,y f s,t = 11 (NTSC) or 13 (PAL) MHz Same f s for NTSC and PAL/SECAM f s = f s,x (NTSC ) f {z } l (NTSC ) = f s,x (PAL) f {z } {z } l (PAL) = 13.5 MHz {z }?=858 15750?=864 15625 Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 12 / 34
Digital Video Format Video Representation Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 13 / 34
Video Representation High De nition and Ultra High De nition Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 14 / 34
Color Space Lecture 2 Video Formation and Representation Color Coordinates Mixture of luminance and chrominance RGB & CMY primaries Separation of luminance and chrominance XYZ - fundamental measurements YUV - PAL YIQ - NTSC YCbCr - digital video HSI - hue, saturation, brightness Color space conversion better representation for processing or data reduction Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 15 / 34
Color Coordinates Color Space Conversion (R, G, B)! (Y, U, V ) Y : brightness U = 0.492(B Y ), V = 0.877(R Y ): color di erences 2 4 Y U V 3 2 5 = 4 0.299 0.587 0.114 0.147 0.287 0.436 0.615 0.515 0.100 Brightness contribution (G > R > B) 3 2 Y = 0.299 {z } R + 0.587 {z } G + 0.114 {z } B (2) (1) (3) 5 4 R G B Coe s. of the rst row add up to 1, those of the other rows to 0 R = G = B ) gray images (U = V = 0) 3 5 Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 16 / 34
Color Space Conversion Color Coordinates Y, Cb, Cr: scaled, shifted versions of Y, U, V 2 Y 3 2 0.257 0.504 3 2 0.098 4 Cb 5 = 4 0.148 0.291 0.439 5 4 Cr 0.439 0.368 0.071 R G B 3 2 5 + 4 16 128 128 3 5 R, G, B take values in 0-255 Y takes values in 16-235, Cb, Cr in 16-240 Part of the BT.601 standard. Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 17 / 34
RGB vs. YCbCr Color Coordinates R G B Y Cb Cr Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 18 / 34
Color Coordinates Chrominance Subsampling Human vision is less sensitive to color than to luminance 4:4:4 Progressive Interlaced Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 19 / 34
Color Coordinates Chrominance Subsampling 4:2:2 Progressive Interlaced 4:2:0 Progressive Interlaced Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 20 / 34
Chrominance Subsampling Color Coordinates 4:4:4 4:2:0 Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 21 / 34
Objective Quality Measure Video Quality Measure Mean Squared Error (MSE) MSE = σ 2 e = 1 N (Ψ 1 (m, n, k) Ψ 2 (m, n, k)) 2 k m,n m, n pixel coordinates; k frame index; N total number of pixels. Peak Signal-to-Noise Ratio (PSNR) PSNR = 10 log 10 ψ 2 max σ 2 e (db) ψ max the maximum (or peak) signal value; 255 for 8-bit video. Average per-frame PSNR PSNR = 1 K k PSNR k k frame index; K number of frames compared. Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 22 / 34
PSNR & MSE Video Quality Measure Excellent (>40), Good (30 40), Poor (<30) Original 42.6 35.5 28.7 Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 23 / 34
PSNR & MSE Video Quality Measure Both do not correlate well with human perception Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 24 / 34
Subjective Quality Evaluation Video Quality Measure Mean opinion score Human observers judgement of visual quality Methods (de ned in ITU-R BT.500-11) Double Stimulus Continuous Quality Scale (DSCQS) Reference and impaired sequences graded in a randomized order. Scores converted into a number indicating the relative quality. Double Stimulus Impairment Scale (DSIS) Reference rst and then the impaired sequence. Grade the impaired sequence relative to its reference. Single Stimulus Continuous Quality Evaluation (SSCQE) Grade the impaired sequence in a continuous manner without a reference. Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 25 / 34
Subjective Quality Evaluation Video Quality Measure DSCQS DSIS SSCQE Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 26 / 34
MPEG Video Coding Standards MPEG Standards MPEG 1 Error free storage Progressive Video: 352x240@1.5 Mbps Audio: MP3 192~256kbps MPEG 4 Media streaming, Frame /Objectbased coding + Scalability MPEG 4 Part 10 Advanced video Coding (also known as H.264) MPEG 21 Multimedia Framework IP Management and Protection (IPMP) 1994 2001 2007 1992 1998 2003 200x Timeline MPEG 2 Broadcast TV Progressive/Interlaced Video:1280x720 @8Mbps Audio: AAC MPEG 7 Multimedia content description Not to define coding method MPEG 4 Amd. Scalable video Coding MPEG 4 Amd. Multi View video Coding Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 27 / 34
Advances in Coding E ciency MPEG Standards HD in H.264/AVC http://www.apple.com/quicktime/guide/hd/ Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 28 / 34
Lecture 2 Video Formation and Representation MPEG Standards MPEG Applications MPEG-4 AVC/H.264 Broadcasting Server Point-to-Point Transmission 128 kbps 64 kbps Wireless MPEG-1 Block-based Video Coding (VCD) 32 kbps 512 kbps 384 kbps Wireless Ethernet 256 kbps Router 1.5 Mbps 64 kbps Ethernet 3 Mbps Bandwidth Time MPEG-2 Block-based Video Coding with Interlaced tools (DVD) Scalable Video Coding MPEG-4 Object-based Video Coding Multi-View Video Coding for 3DTV Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 29 / 34
MPEG Standardization Activities MPEG Standards A. Exploration 1.The search for new technology 2.Seek Industry experts 3.Open seminars F. Amendment Adding new technology G. Corrigenda Corrective actions H. New subdivisions Add new non compatible technology B. Requirements 1. Establish the scope of work 2. Call for Proposals E. Standardization 1. Ballots 2. National Body Comments C. Competitive phase 1. Do Homework 2. Response to CfP 3. Initial technology selection D. Collaborative phase 1. Core Experiments 2. Working Drafts Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 30 / 34
MPEG Standardization Activities MPEG Standards Activities 6 12 months 6 12 months 3 6 months Search for new technology Define requirements of spec. Initial tech. selection 1 year Call for Evidence WD: Working Draft CD: Committee Draft FCD: Final Committee Draft FDIS: Final Draft of International Standard IS: International Standard Core Experiments ~3 years Call for Proposals 3 months 3 months 3 months 3 months WD CD (PDAM) ~2 years FCD (FPDAM) FDIS (FDAM) IS (AMD) Time Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 31 / 34
MPEG Week Lecture 2 Video Formation and Representation MPEG Standards Work, Work, and Work! 9:00am noon Monday Tuesday Wednesday Thursday Friday Opening Plenary Mid week Plenary Subgroup Plenary 6:00pm Subgroup Plenary Technical Sessions/Joint Meetings 2pm Closing Plenary Social Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 32 / 34
MPEG Week Lecture 2 Video Formation and Representation MPEG Standards MPEG is Good at Pruning Out Bad Ideas! Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 33 / 34
MPEG Standards References 1 Yao Wang, et. al - Video Processing and Communications 2 D. Miras - On Quality Aware Adaptation of Internet Video, Ph.D Dissertation 3 T. Wiegand - Scalable Video Coding, JVT-W132 Wen-Hsiao Peng, Ph.D. (NCTU CS) MAPL March 2013 34 / 34