Image and video encoding: A big picture. Predictive. Predictive Coding. Post- Processing (Post-filtering) Lossy. Pre-

Transcription

1 Lab Session 1 (with Supplemental Materials to Lecture 1) April 27, 2009 Outline Review Color Spaces in General Color Spaces for Formats Perceptual Quality MATLAB Exercises Reading and showing images and video sequences Getting familiar with color spaces Review 1 Image and video encoding: A big picture Predictive Image and video encoding: A big picture A/D Conversion Color Space Conversion Pre-Filtering Partitioning Predictive Differential Motion Estimation and Compensation Context-Based Pre- Lossy Lossless Processing Post- Processing (Post-Filtering) Pre- Lossy Lossless Processing Post- Processing (Post-filtering) Decoded Image and video decoding: A big picture Predictive Post- Lossy Lossless Processing Pre-Processing Processing Quantiation Transform Model-Based Entropy Dictionary-Based Run-Length

2 A oo of standards An example: MPEG-video codec MPEG-video encoding process Image coding standards JPEG JPEG2000 JBIG/JBIG2 JPEG XR (Microsoft HD Photo) Video coding standards MPEG-1/2 MPEG-4 Part2 H.264/MPEG-4 Part 10 AVC SMPTE VC-1 (Microsoft WMV9) International standardiation bodies ISO, IEC, ITU, SMPTE, ANSI, MPEG, JPEG, JVT, VCEG, Some terms Human visual system: Color perception Color Spaces Luminance = Brightness = Lightness = Intensity = Value Hue: Saturation = Colorfulness = Chroma = Vividness = Purity Chromaticity = Hue + Saturation It is a 2-tuple. Chromaticity Diagram: a 2-D diagram showing chromaticities Color = Luminance + Chromaticity So, it is a 3-tuple. Chrominance (Chroma) = Color Difference Color Space: a 3-D space of colors Color Mixing Systems (light) vs. Color Appearance Systems (perception) Gamut: range of colors (in a color space) JND: Just-Noticeable Difference (50% accuracy) Light Retinal neurons Anatomy of human retina Cone Rod

3 Shujun LI (李树钧): INF Multimedia Shujun LI (李树钧): INF Multimedia CIE 1931 XYZ color space Shujun LI (李树钧): INF Multimedia CIE 1976 uniform color spaces CIE = Commission internationale de l'éclairage = International Commission on Illumination CIE 1931 RGB color space Problem: CIE XYZ color space is nonlinear in term of perceptual color differences. b* a* Color matching functions of the CIE standard observer CIE 1931 RGB Color matching functions CIE 1931 xyy color space chromaticity diagram CIELUV (CIE 1976 L*u*v*) color space Shujun LI (李树钧): INF Multimedia CIE 1931 RGB color space chromaticity diagram CIELAB (CIE 1976 L*a*b*) color space Shujun LI (李树钧): INF Multimedia srgb color space (IEC :1999) Shujun LI (李树钧): INF Multimedia More RGB color spaces Adobe 1998 RGB color space 15 ProPhoto RGB color space (by Kodak) 14 HSV/HSL (HSI/HSB) color spaces HSV color space 16 HSL color space 17 3

4 Shujun LI (李树钧): INF Multimedia Shujun LI (李树钧): INF Multimedia Shujun LI (李树钧): INF Multimedia Munsell color system Gamma correction Color Spaces for y=xr Shujun LI (李树钧): INF Multimedia Luma (Y ) vs. Luminance (Y) Shujun LI (李树钧): INF Multimedia Shujun LI (李树钧): INF Multimedia Color conversion in image/video encoding Y UV/Y CbCr color space Luma = Luminance after gamma correction RGB R G B Y=kr*R+kg*G+kb*B Y =kr *R +kg *G +kb *B In practice, many people (even some experts!) don t know the difference between Y and Y. Roughly, U=Cb=B-Y, V=Cr=R-Y But, Y Y and Y UV Y CbCr Practically, YUV = YCbCr = Y UV = Y CbCr Y PbPr = Y CbCr for analog signal YCC = Y CbCr Pre-Processing g RGB R G B Color Space Conversion Y UV/ Y PbPr A/D Conversion Y CbCr Chroma Subsampling Other Parts of Encoding Process U-V plan (Y =0.5)

5 Color conversion: R G B => Y C b C r R,G,B [0,1],,, [,], k r, k b (0,1), (, k r+k b<1 Y =k r *R +(1-k r -k b )*G +k b *B [0,1] P b =0.5*(B -Y )/(1-k b ) [-0.5,0.5] P r =0.5*(R -Y )/(1-k r ) [-0.5,0.5] Values of k b and k r can be different! ITU-T BT.601 (SDTV): k r =0.229, k b =0.114 ITU-T T BT.709 (HDTV): k r =0.2126, k b = ANSI/SMPTE 240M-1995 (HDTV): k r =0.212, k b =0.087 Y P b P r Y C b C r (taking MPEG-2 as an example) Y =219*Y +16 [16,235] C b =224*P b +128 [16,240] C r =224*P r +128 [16,240] 24 Why Y C b C r? Human eyes are less sensitive to chrominance. We can sample chroma (C b C r ) less frequently than the luma (Y ). We achieve the goal of data compression. 25 Chroma subsampling formats A 4X2 Image P(1,1) P(1,2) P(1,3) P(1,4) P(2,1) P(2,2) P(2,3) P(2,4) Y C b C r Y Sampling Locations C b Sampling Locations C r Sampling Locations 4:4: ALL ALL ALL 4:2: ALL P(1,1) P(1,3) P(2,1) P(2,3) P(1,1) P(1,3) P(2,1) P(2,3) 4:2: ALL P(1,1) P(1,3) P(1,1) P(1,3) 4:1: ALL P(1,1) P(2,1) P(1,1) P(2,1) 26 Formats Spatial display resolutions (W H) Spatial resolutions: Images No standards For research purpose: 2 n 2 n, such as , , ,

6 Shujun LI (李树钧): INF Multimedia Shujun LI (李树钧): INF Multimedia Some test images for research Digital video formats Shujun LI (李树钧): INF Multimedia SIF = Source p Format ((since MPEG-1)) Planar formats Packed formats More info available at CIF (Full CIF = FCIF) = QCIF (Quarter CIF) = SQCIF (Sub Quarter CIF) = CIF = 4 CIF = CIF = 4 4CIF = /50 (TV: PAL/SECAM) = / /59.94 (TV: NTSC) = / Sub-SIF (Computers) = or YUV = I420 = IYUV (4:2:0) YV12 (4:2:0) YV16 (4:2:2) UYVY = UYNV = Y422 (4:2:2) YUY2 = YUNV = V422 = YUYV (4:2:2) 31 Shujun LI (李树钧): INF Multimedia YUV4MPEG2 format (.y4m) YUV video file format (.yuv/.cif/.qcif/.sif/) CIF = Common Intermediate Format ((since H.261)) 30 Shujun LI (李树钧): INF Multimedia 32 Shujun LI (李树钧): INF Multimedia Image and video coding: Quality issue Quality M t i Meterics File Header YUV4MPEG2 + a number of parameters + 0x0A Parameters Perceptual Quality (Mandatory) Width, height and frame rate: Wxxx Hyyy Fa:b (Optional) Interlacing: Ip (progressive), It (top field first), Ib (bottom field first), Im (mixed mode, detailed in frame headers) (Optional) Aspect ratio: Aa:b (Optional) Color space (Chroma format): C4xx... ( (Optional) i l) Comment: X. Frames Frame Header Frame (YUV planar format) image/video Decoded image/video Visual Quality Measurement Encoder Decoder image/video FRAME + a number of parameters (optional) + 0x0A

7 Visual quality measurement: Subjective DSIS (Double Stimulus Impairment Scale) DSCQS (Double Stimulus Continuous Quality Scale) SSCQE (Single Stimulus Continuous Quality Evaluation) Some measurement methods have been standardied: ITU-R BT.500, ITU-R BT.710, ITU-T P Visual quality measurement: Objective Two images: the original one f(x,y) and the decoded one f (x,y) MSE = Mean Squared Error 1 X (f 0 (x, y) f(x, y)) 2 MN x,y à P! x,y (f 0 (x, y)) 2 10 log 10 P x,y (f 0 (x, y) f(x, y)) 2 Ã! MNL 2 10 log 10 P x,y (f 0 (x, y) f(x, y)) 2 SNR = Signal-to-Noise Ratio PSNR = Peak Signal-to-Noise Ratio 37 Visual quality measurement: Objective The original image PSNR=32.7 db PSNR=37.5 db Research Question: PSNR is not good enough to perfectly reflect visual quality, more advanced metrics considering HVS are wanted. Some objective metrics are being standardied by the ITU-T. 38 MATLAB Exercises Reading and Showing Reading an image f=imread( test.bmp ); Showing an image imshow(f); Reading a frame from a YUV video f=yuvread( test.cif,1); Showing a video frame imshow(ycbcr2rgb(yuv4xx2yuv444(f))); Playing a YUV video yuvplay( test.cif ); 40 Getting familiar with color spaces Showing YUV planes of a video f=yuvread( test.cif',1); subplot(2,2,1:2); imshow(f.y); subplot(2,2,3); imshow(f.u); subplot(2,2,4); imshow(f.v); Trying to read an image and convert it to YUV color space. 41 7