Lab Session 1 (with Supplemental Materials to Lecture 1) April 27, 2009 Outline Review Color Spaces in General Color Spaces for Formats Perceptual Quality MATLAB Exercises Reading and showing images and video sequences Getting familiar with color spaces Review 1 Image and video encoding: A big picture Predictive Image and video encoding: A big picture A/D Conversion Color Space Conversion Pre-Filtering Partitioning Predictive Differential Motion Estimation and Compensation Context-Based Pre- Lossy Lossless Processing Post- Processing (Post-Filtering) Pre- Lossy Lossless Processing Post- Processing (Post-filtering) Decoded Image and video decoding: A big picture Predictive Post- Lossy Lossless Processing Pre-Processing Processing Quantiation Transform Model-Based Entropy Dictionary-Based Run-Length 3 4 5 1
A oo of standards An example: MPEG-video codec MPEG-video encoding process Image coding standards JPEG JPEG2000 JBIG/JBIG2 JPEG XR (Microsoft HD Photo) Video coding standards MPEG-1/2 MPEG-4 Part2 H.264/MPEG-4 Part 10 AVC SMPTE VC-1 (Microsoft WMV9) International standardiation bodies ISO, IEC, ITU, SMPTE, ANSI, MPEG, JPEG, JVT, VCEG, 6 7 8 Some terms Human visual system: Color perception Color Spaces Luminance = Brightness = Lightness = Intensity = Value Hue: Saturation = Colorfulness = Chroma = Vividness = Purity Chromaticity = Hue + Saturation It is a 2-tuple. Chromaticity Diagram: a 2-D diagram showing chromaticities Color = Luminance + Chromaticity So, it is a 3-tuple. Chrominance (Chroma) = Color Difference Color Space: a 3-D space of colors Color Mixing Systems (light) vs. Color Appearance Systems (perception) Gamut: range of colors (in a color space) JND: Just-Noticeable Difference (50% accuracy) Light Retinal neurons Anatomy of human retina Cone Rod 10 11 2
Shujun LI (李树钧): INF-10845-20091 Multimedia Shujun LI (李树钧): INF-10845-20091 Multimedia CIE 1931 XYZ color space Shujun LI (李树钧): INF-10845-20091 Multimedia CIE 1976 uniform color spaces CIE = Commission internationale de l'éclairage = International Commission on Illumination CIE 1931 RGB color space Problem: CIE XYZ color space is nonlinear in term of perceptual color differences. b* a* Color matching functions of the CIE standard observer CIE 1931 RGB Color matching functions CIE 1931 xyy color space chromaticity diagram CIELUV (CIE 1976 L*u*v*) color space Shujun LI (李树钧): INF-10845-20091 Multimedia CIE 1931 RGB color space chromaticity diagram CIELAB (CIE 1976 L*a*b*) color space 12 13 Shujun LI (李树钧): INF-10845-20091 Multimedia srgb color space (IEC 61966-2-1:1999) Shujun LI (李树钧): INF-10845-20091 Multimedia More RGB color spaces Adobe 1998 RGB color space 15 ProPhoto RGB color space (by Kodak) 14 HSV/HSL (HSI/HSB) color spaces HSV color space 16 HSL color space 17 3
Shujun LI (李树钧): INF-10845-20091 Multimedia Shujun LI (李树钧): INF-10845-20091 Multimedia Shujun LI (李树钧): INF-10845-20091 Multimedia Munsell color system Gamma correction Color Spaces for y=xr 20 18 Shujun LI (李树钧): INF-10845-20091 Multimedia Luma (Y ) vs. Luminance (Y) Shujun LI (李树钧): INF-10845-20091 Multimedia Shujun LI (李树钧): INF-10845-20091 Multimedia Color conversion in image/video encoding Y UV/Y CbCr color space Luma = Luminance after gamma correction RGB R G B Y=kr*R+kg*G+kb*B Y =kr *R +kg *G +kb *B In practice, many people (even some experts!) don t know the difference between Y and Y. Roughly, U=Cb=B-Y, V=Cr=R-Y But, Y Y and Y UV Y CbCr Practically, YUV = YCbCr = Y UV = Y CbCr Y PbPr = Y CbCr for analog signal YCC = Y CbCr Pre-Processing g RGB R G B Color Space Conversion Y UV/ Y PbPr A/D Conversion Y CbCr Chroma Subsampling 001110101001 Other Parts of Encoding Process U-V plan (Y =0.5) 21 22 23 4
Color conversion: R G B => Y C b C r R,G,B [0,1],,, [,], k r, k b (0,1), (, k r+k b<1 Y =k r *R +(1-k r -k b )*G +k b *B [0,1] P b =0.5*(B -Y )/(1-k b ) [-0.5,0.5] P r =0.5*(R -Y )/(1-k r ) [-0.5,0.5] Values of k b and k r can be different! ITU-T BT.601 (SDTV): k r =0.229, k b =0.114 ITU-T T BT.709 (HDTV): k r =0.2126, k b =00722 0.0722 ANSI/SMPTE 240M-1995 (HDTV): k r =0.212, k b =0.087 Y P b P r Y C b C r (taking MPEG-2 as an example) Y =219*Y +16 [16,235] C b =224*P b +128 [16,240] C r =224*P r +128 [16,240] 24 Why Y C b C r? Human eyes are less sensitive to chrominance. We can sample chroma (C b C r ) less frequently than the luma (Y ). We achieve the goal of data compression. 25 Chroma subsampling formats A 4X2 Image P(1,1) P(1,2) P(1,3) P(1,4) P(2,1) P(2,2) P(2,3) P(2,4) Y C b C r Y Sampling Locations C b Sampling Locations C r Sampling Locations 4:4:4 4 4 4 ALL ALL ALL 4:2:2 4 2 2 ALL P(1,1) P(1,3) P(2,1) P(2,3) P(1,1) P(1,3) P(2,1) P(2,3) 4:2:0 4 1 1 ALL P(1,1) P(1,3) P(1,1) P(1,3) 4:1:1 4 1 1 ALL P(1,1) P(2,1) P(1,1) P(2,1) 26 Formats Spatial display resolutions (W H) Spatial resolutions: Images No standards For research purpose: 2 n 2 n, such as 128 128, 256 256, 512 512, 1024 1024 28 29 5
Shujun LI (李树钧): INF-10845-20091 Multimedia Shujun LI (李树钧): INF-10845-20091 Multimedia Some test images for research Digital video formats Shujun LI (李树钧): INF-10845-20091 Multimedia SIF = Source p Format ((since MPEG-1)) Planar formats Packed formats More info available at http://www.fourcc.org/yuv.php CIF (Full CIF = FCIF) = 352 288 QCIF (Quarter CIF) = 176 144 SQCIF (Sub Quarter CIF) = 128 96 4CIF = 4 CIF = 704 576 16CIF = 4 4CIF = 1408 1152 625/50 (TV: PAL/SECAM) = 352 288/360 288 525/59.94 (TV: NTSC) = 352 240/ 360 240 Sub-SIF (Computers) = 320 288 or 384 288 YUV = I420 = IYUV (4:2:0) YV12 (4:2:0) YV16 (4:2:2) UYVY = UYNV = Y422 (4:2:2) YUY2 = YUNV = V422 = YUYV (4:2:2) 31 Shujun LI (李树钧): INF-10845-20091 Multimedia YUV4MPEG2 format (.y4m) YUV video file format (.yuv/.cif/.qcif/.sif/) CIF = Common Intermediate Format ((since H.261)) 30 Shujun LI (李树钧): INF-10845-20091 Multimedia 32 Shujun LI (李树钧): INF-10845-20091 Multimedia Image and video coding: Quality issue Quality M t i Meterics File Header YUV4MPEG2 + a number of parameters + 0x0A Parameters Perceptual Quality (Mandatory) Width, height and frame rate: Wxxx Hyyy Fa:b (Optional) Interlacing: Ip (progressive), It (top field first), Ib (bottom field first), Im (mixed mode, detailed in frame headers) (Optional) Aspect ratio: Aa:b (Optional) Color space (Chroma format): C4xx... ( (Optional) i l) Comment: X. Frames Frame Header Frame (YUV planar format) image/video Decoded image/video Visual Quality Measurement Encoder Decoder image/video FRAME + a number of parameters (optional) + 0x0A 33 35 6
Visual quality measurement: Subjective DSIS (Double Stimulus Impairment Scale) DSCQS (Double Stimulus Continuous Quality Scale) SSCQE (Single Stimulus Continuous Quality Evaluation) Some measurement methods have been standardied: ITU-R BT.500, ITU-R BT.710, ITU-T P.910 36 Visual quality measurement: Objective Two images: the original one f(x,y) and the decoded one f (x,y) MSE = Mean Squared Error 1 X (f 0 (x, y) f(x, y)) 2 MN x,y à P! x,y (f 0 (x, y)) 2 10 log 10 P x,y (f 0 (x, y) f(x, y)) 2 Ã! MNL 2 10 log 10 P x,y (f 0 (x, y) f(x, y)) 2 SNR = Signal-to-Noise Ratio PSNR = Peak Signal-to-Noise Ratio 37 Visual quality measurement: Objective The original image PSNR=32.7 db PSNR=37.5 db Research Question: PSNR is not good enough to perfectly reflect visual quality, more advanced metrics considering HVS are wanted. Some objective metrics are being standardied by the ITU-T. 38 MATLAB Exercises Reading and Showing Reading an image f=imread( test.bmp ); Showing an image imshow(f); Reading a frame from a YUV video f=yuvread( test.cif,1); Showing a video frame imshow(ycbcr2rgb(yuv4xx2yuv444(f))); Playing a YUV video yuvplay( test.cif ); 40 Getting familiar with color spaces Showing YUV planes of a video f=yuvread( test.cif',1); subplot(2,2,1:2); imshow(f.y); subplot(2,2,3); imshow(f.u); subplot(2,2,4); imshow(f.v); Trying to read an image and convert it to YUV color space. 41 7