In the name of Allah the compassionate, the merciful
Digital Video Systems S. Kasaei Room: CE 307 Department of Computer Engineering Sharif University of Technology E-Mail: skasaei@sharif.edu Webpage: http://sharif.edu/~skasaei Lab. Website: http://mehr.sharif.edu/~ipl
Acknowledgment Most of the slides used in this course have been provided by: Prof. Yao Wang (Polytechnic University, Brooklyn) based on the book: Video Processing & Communications written by: Yao Wang, Jom Ostermann, & Ya-Oin Zhang Prentice Hall, 1 st edition, 2001, ISBN: 0130175471. [SUT Code: TK 5105.2.W36 2001].
Chapter 1 Introduction & Basics of Video
Outline Color perception & specification Video capture & display Analog raster video Analog TV systems Digital video Kasaei 6
Digital Video Processing A video signal is a sequence of 2- D images, projected from a dynamic 3- D scene onto the image plane of a video camera. The color value at any point in a video frame records the emitted (or reflected) light at a particular * 3- D point in the observed scene. * * * n-1 n n+1 Kasaei 7
Color Perception & Specification Light -> color perception Human perception of color Type of light sources Trichromatic color mixing theory Specification of color Tristimulus representation Luminance/Chrominance representation Color coordinate conversion Kasaei 8
Eye Anatomy From http://www.stlukeseye.com/anatomy.asp Kasaei 9
Eye vs. Camera Camera components Lens Shutter Film Cable to transfer images Eye components Lens, cornea Iris, pupil Retina Optic nerve send the info to the brain Kasaei 10
Human Perception of Color Retina contains photo receptors: Cones: day vision, can perceive color tone: Red, green, & blue cones. Different cones have different frequency responses. Tri-receptor theory of color vision [Young1802]. Rods: night vision, perceive brightness only. Color sensation is characterized by: Luminance (brightness). Chrominance: Hue (color tone). Saturation (color purity). From: http://www.macula.org/anatomy /retinaframe.html Kasaei 11
Human Perception of Color Fig. 1: Simplified diagram of a cross section of the human eye. Kasaei 12
Human Perception of Color Fig. 2: Cross section of the eye. Kasaei 13
Human Perception of Color Light consists of an electromagnetic wave, with wavelengths in the range of 380-780 nm, in which the human eye is sensitive. The energy of light is measured in flux, with the unit of the watt. Light is defined as the electromagnetic radiation that stimulates our vision response. It is expressed as a spectral energy distribution. Kasaei 14
Human Perception of Color Fig. 3: The electromagnetic spectrum. Kasaei 15
Human Perception of Color Fig. 4: Visible wavelengths. Kasaei 16
Human Perception of Color Green Red Blue Fig. 5: Typical relative luminous efficiency function of human eye. Kasaei 17
Human Perception of Color The luminance of an object is independent of the luminance of the surrounding objects. The (apparent) brightness of an object is the perceived luminance & depends on the luminance of the surround. Kasaei 18
Human Perception of Color Fig. 6: Simultaneous contrast. Top: Middle squares have equal luminance but do not appear equally; Bottom: Middle squares appear almost equally bright, but their luminance are different. Kasaei 19
Human Perception of Color Human visual system is most sensitive to midfrequencies (3~10 cycles/degree) & least sensitive to high frequencies. Contrast sensitivity also depends on orientation of the grating (max for horizontal & vertical grating). Kasaei 20
Human Perception of Color Angular sensitivity variations are within 3dB (Max. deviation at 45 degree). Spatial frequency components, separated by about one octave, can be detected independently by observers. Thus, visual system contains a number of independent spatial channels, each tuned to a different spatial frequency & orientation angle. Kasaei 21
Color Representation Use of color is not only more pleasing but it also enables us to receive more visual information. While human can perceive only a few dozen gray levels, have the ability to distinguish between thousands of colors. Kasaei 22
Color Representation Fig. 7: Visible color spectrum. Kasaei 23
Color Representation Fig. 8: Visible wavelengths. Kasaei 24
Color Representation The perceptual attributes of colors are brightness, hue, & saturation. Brightness presents the perceived luminance. Hue refers to its redness, greenness,... Saturation is that aspect of perception that varies most strongly as more white light is added. Kasaei 25
Color Representation Fig. 9: Hue representation. Kasaei 26
Color Representation Fig. 10: Hue representation. Kasaei 27
Color Representation Fig. 11: HSV color model representation. Kasaei 28
Color Representation Fig. 12: HSV color model representation. Kasaei 29
Color Representation Fig. 13: HSV color model representation. Kasaei 30
Color Representation Fig. 14: HSV color model. Kasaei 31
Color Representation Fig. 15: HIS color model. Kasaei 32
Color Representation Fig. 16: HIS color model. Kasaei 33
Color Representation Fig. 17: HIS color model. Kasaei 34
Color Representation For monochromatic light sources, differences in hues are manifested by the differences is wavelengths. These definitions are somewhat imprecise. Hue, brightness,& saturation all change when either the wavelength, the intensity, the hue, or amount of white light in a color is changed. Kasaei 35
Illuminating & Reflecting Light Illuminating sources: emit light (e.g., the sun, light bulb, TV monitors). perceived color depends on the emitted freq. follows additive rule: R+G+B=White. Reflecting sources: reflect an incoming light (e.g. the color dye, matte surface, cloth). perceived color depends on reflected freq (=emitted freqabsorbed freq.). follows subtractive rule: R+G+B=Black. Kasaei 36
Frequency Responses of Cones from [Gonzalez02] Ci = C( λ) ai ( λ) dλ, i = r, g, b, y Kasaei 37
Frequency Responses of Cones C i : spectral (radial intensity) response, C(λ) : spectral energy distribution (complete incoming light spectrum), a i (λ) : absorption spectra, (spectral sensitivity curves, frequency responses). Kasaei 38
Frequency Responses of Cones & the Luminous Efficiency Function Relative sensitivity 100 80 60 40 Blue 20 Luminosity function Red Green 20 0 400 500 600 700 Wavelength Ci = C( λ) ai ( λ) dλ, i = r, g, b, y Kasaei 39
Color Hue Specification Kasaei 40
Trichromatic Color Mixing Trichromatic color mixing theory: Any color can be obtained by mixing three primary colors with a right proportion. C = TkCk, Tk : Tristimulus values k = 1,2,3 Primary colors for illuminating sources: Red, Green, & Blue (RGB). Color monitor works by exciting red, green, & blue phosphors using separate electronic guns. Kasaei 41
Trichromatic Color Mixing Primary colors for reflecting sources (also known as secondary colors): Cyan, Magenta, & Yellow (CMY). Color printer works by using cyan, magenta, yellow, & black (CMYK) dyes. (the dye acts like a narrow-band filter) Kasaei 42
RGB vs CMY Kasaei 43
red Green Blue Kasaei 44
Color Representation Models Specify the tristimulus values associated with the three primary colors: RGB. CMY. Specify the luminance & chrominance: HSI (Hue, saturation, & intensity). YIQ (used in NTSC color TV). YCbCr (used in digital color TV). Amplitude specification: 8 bits for each color component (or 24 bits total for each pixel). Total of 16 million colors. A true RGB color display of size 1Kx1K requires a display buffer memory size of 3 MB. Kasaei 45
Color Coordinate Conversion Conversion between different primary sets are linear (3x3 matrix). Conversion between primary and XYZ/YIQ/YUV are also linear. Conversion to LSI/Lab are nonlinear. LSI & Lab coordinates: coordinate Euclidean distance is proportional to actual color difference. Conversion formulae between many color coordinates can be found in [Gonzalez92]. Kasaei 46
Color Coordinate Conversion Table 1: Color coordinate systems [Commission Internationale de L Eclairage (CIE)]. Kasaei 47
Color Coordinate Conversion Table 1: Color coordinate systems (Cntd). Kasaei 48
Color Coordinate Conversion Table 1: Color coordinate systems (Cntd). Kasaei 49
Color Coordinate Conversion Table 2: Transformation from NTSC Receiver Primary to other coordinate systems. Kasaei 50
Color Coordinate Conversion Kasaei 51
Color Coordinate Conversion Kasaei 52
Color Coordinate Conversion Kasaei 53
Color Spaces CIE XYZ. Kasaei 54
Color Spaces CIE XYZ chromaticity diagram. Kasaei 55
Color Spaces CIE XYZ chromaticity diagram. Kasaei 56
Color Spaces CIE XYZ chromaticity diagram. Kasaei 57
Color Spaces CIE XYZ chromaticity diagram. Kasaei 58
Color Spaces The RGB safe-color cube. Kasaei 59
Color Spaces CIE Lab color models. Kasaei 60
Color Spaces color copier. Kasaei 61
Color Image Processing RGB color model. Kasaei 62
Color Image Processing Pseudo color for detection. Kasaei 63
Color Image Processing Pseudo color example. Kasaei 64
Color Image Processing Color manipulation. Kasaei 65
Video Capture & Display Light reflection physics Imaging operator Color capture Color display Component vs. composite video Kasaei 66
Video Capture For natural images we need a light source? (λ: wavelength of the source). E(x, y, z, λ): incident light on a point (x, y, z world coordinates of the point) Each point in the scene has a reflectivity function. r(x, y, z, λ): reflectivity function Light reflects from a point and the reflected light is captured by an imaging device. c(x, y, z, λ) =E(x, y, z, λ) r(x, y, z, λ): reflected light. Courtesy of Onur Guleryuz Kasaei 67
More on Video Capture Reflected light to camera: Light intensity distribution in 3-D world: ψ ( X, t) = C( X, t, λ) ac ( λ) dλ Camera spectral absorption function: a Projected image from 3-D to 2-D (video): The projection operator is non-linear: X x P ψ ( P( X), t) = ψ ( X, t) Perspective projection. Othographic projection. or ψ ( x, t) = ψ ( P 1 ( x), t) Kasaei 68
Perspective Projection Model Y X Y X 3-D point Z X Z x y F C x y x Camera center X x = F, y = Z Y F Z Image plane 2-D image The image of an object is reversed from its 3-D position. The object appears smaller when it is farther away. Kasaei 69
How to Capture Color Single chip color charge-coupled devise (CCD). Kasaei 70
How to Capture Color Needs three types of sensors. Complicated digital processing is incorporated in advanced cameras. ( f s,1 ) CCDs f s,1 f s,1 Image 2fs,1 Rate enhancer conv. 2f s,1/f s,2 Digital CN output Lens R B G Analog process A/D Pre-process Interpolation Color corrector Nonlinear processing Matrix & encoder 2f s,1 13.5 MHz D/A D/A Analog CN & CS output Viewfinder output Figure 1.2 Schematic block diagram of a professional color video camera. Reprinted from Y. Hashimoto, M. Yamamoto, and T. Asaida, Cameras and display systems, IEEE (July 1995), 83(7):1032 43. Copyright 1995 IEEE. Kasaei 71
Video Display Monitor phosphor. Kasaei 72
Video Display A human observer perceives color through the stimuli of 3 different pigmented cones. Typical absorption spectra of cons in the retina, as a function of wavelength. Kasaei 73
Video Display A weighted sum of primaries produces a color that cannot be distinguished by an observer from the color of the spectrum. Additive color model Kasaei 74
Video Display Primary & secondary colors of light & pigments. Kasaei 75
Video Display Cathode ray tube (CRT) vs liquid crystal display (LCD). CRT needs three light sources projecting red, green, & blue components, respectively. The depth of a CRT needs to be about the same as the width of the screen, for the electrons to reach the side of the screen ( flat CRT). LCD charges the optical properties & consequently the brightness (or color) of a liquid crystal by an applied electric field ( plasma). Kasaei 76
Analog Video Video raster Progressive vs. interlaced raster Analog TV systems Kasaei 77
Raster Scan Real-world scene is a continuous 3-D signal (temporal, horizontal, vertical). Analog video is stored in the raster format: Sampling in time: consecutive sets of frames. To render motion properly >=30 frame/s is needed. Kasaei 78
Raster Scan Sampling in vertical direction: a frame is represented by a set of scan lines. Number of lines depends on maximum vertical frequency & viewing distance (525 lines in the NTSC system). Video-raster: 1-D signal consisting of scan lines from successive frames. Kasaei 79
Progressive & Interlaced Scans Progressive Frame Horizontal retrace Interlaced Frame Field 1 Field 2 Vertical retrace Interlaced scan is developed to provide a trade-off between temporal & vertical resolution, for a given, fixed data rate (number of line/sec). Kasaei 80
Waveform & Spectrum of an Interlaced Raster Horizontal retrace for first field Vertical retrace from first to second field Vertical retrace from second to third field Blanking level Black level T h White level T l t 2 T T t (a) ( f ) f 0 f Kasaei l 2f l 3f l f max 81 (b)
Color TV Broadcasting & Receiving RGB ---> YC1C2 Luminance, Chrominance, Audio Multiplexing Modulation YC1C2 ---> RGB De- Multiplexing De- Modulation Kasaei 82
Why not using RGB directly? R,G,B components are correlated: Transmitting R,G,& B components separately is redundant. More efficient use of bandwidth is desired. RGB->YC1C2 transformation: Decorrelating: Y, C1 & C2 are uncorrelated. C1 & C2 require lower bandwidths. Y (luminance) component can be received by B/W recivers. Hue is better retained than saturation. Kasaei 83
Why not using RGB directly? YIQ in NTSC I: orange-to-cyan. Q: green-to-purple (human eye is less sensitive). Q can be further bandlimited than I. Phase = Arctan(Q/I) = hue. Magnitude = sqrt (I^2+Q^2) = saturation. Kasaei 84
Color Image Y image I image (orange-cyan) Q image (green-purple)
I & Q on the Color Circle Q: green-purple I: orange-cyan Kasaei 86
Conversion between RGB & YIQ RGB -> YIQ Y = 0.299 R + 0.587 G + 0.114 B I = 0.596 R -0.275 G -0.321 B Q = 0.212 R -0.523 G + 0.311 B YIQ -> RGB R =1.0 Y + 0.956 I + 0.620 Q, G = 1.0 Y - 0.272 I -0.647 Q, B =1.0 Y -1.108 I + 1.700 Q. Kasaei 87
TV Signal Bandwidth f, : # of active lines. s f, /2 : cycles/picture-height. s y y Maximum frequency that can be rendered properly is lower than this theoretical limit. K=07: Kell factor (attenuation factor). f v,max : Maximum vertical frequency. f h,max : Maximum horizontal frequency. T : Line scanned time (seconds). l f max : Maximum frequency in 1-D raster signal. Kasaei 88
TV Signal Bandwidth Luminance: Maximum vertical frequency (cycles/picture-height): black & white lines interlacing: f = v, max Kf ' s, y / 2 Maximum horizontal frequency (cycles/picture-width): f h f, max = v,max IAR Corresponding temporal frequency (cycles/second or Hz): For NTSC: f max = f h,max / T ' l = IAR Kf ' s, y /2T ' l f max = 4.2 MHz Kasaei 89
TV Signal Bandwidth Chrominance: Can be bandlimited significantly. I: 1.5 MHz, Q: 0.5 MHz. Kasaei 90
Bandwidth of Chrominance Signals Theoretically, for the same line rate, the chromiance signal can have as high frequency as the luminance signal. However, with real video signals, the chrominance component typically changes much slower than luminance. Furthermore, the human eye is less sensitive to changes in chrominance than to changes in luminance. Kasaei 91
Bandwidth of Chrominance Signals The human eye is more sensitive to the orange-cyan range (I) (the color of face!) than to green-purple range (Q). The above factors lead to: I: bandlimitted to 1.5 MHz. Q: bandlimitted to 0.5 MHz. Kasaei 92
Multiplexing of Luminance & Chrominance Chrominance signal can be bandlimited. It usually has a narrower frequency span than the luminance & the human eye is less sensitive to high frequencies in chrominance. The two chrominance components (I and Q) are multiplexed onto the same sub-carrier using QAM. The upper band of I is limited to 0.5 MHz to avoid interference with audio. Kasaei 93
Multiplexing of Luminance & Chrominance Position the bandlimited chrominance at the high end spectrum of the luminance (where the luminance is weak) but still sufficiently lower than the audio (at 4.5 MHz=286 f l ). The actual position should be such that the peaks of chrominance spectrum interlace with those of the luminance: f c 455 f / 2 ( = = l 3.58 Hz for NTSC) Kasaei 94
Spectrum Illustration (f ) Luminance Chrominance 0 f l 2f l 3f l 225f l 226f l 227f l 228f l 229f l 230f l f f c (Color subcarrier) Kasaei 95
Multiplexing of Luminance, Chrominance & Audio (Composite Video Spectrum) 1.25 MHz 6.0 MHz 4.5 MHz 4.2 MHz 3.58 MHz Luminance I I and Q Audio f p f c f a f Picture carrier Color subcarrier Audio subcarrier Kasaei 96
Quadrature Amplitude Modulation (QAM) A method to modulate two signals onto the same carrier frequency, but with 90 o phase shift. s 1( t ) cos( 2 πf 1 t ) m (t ) m (t ) cos( 2 πf 1 t ) LPF s 1( t ) s 2 ( t ) sin( 2πf1t ) sin( 2πf1t ) LPF s 2 ( t ) QAM modulator QAM demodulator Kasaei 97
Adding Color Bursts for Synchronization For accurate regeneration of the color sub-carrier signal at the receiver, a color burst signal is added during the horizontal retrace period. Figure from From Grob, Basic Color Television Principles and Servicing, McGraw Hill, 1975, http://www.ee.washington.edu/conselec/ce/kuhn/ntsc/95x417.gif. Kasaei 98
Multiplexing of Luminance & Chrominance Y(t) LPF 0-4.2 MHz I(t) LPF 0-1.5 MHz Q(t) LPF 0-0.5 MHz -π/2 Σ BPF 2-4.2MHz Σ Composite Video Acos(2πf c t) Gate Color Burst Signal Kasaei 99
Demultiplexing of Luminance & Chrominance Composite Video Comb Filter 0-4.2 MHz Y(t) _ + Σ LPF 0-1.5 MHz I(t) Horizontal Sync Signal Gate 2Acos(2πf c t) -π/2 LPF 0-0.5 MHz Q(t) Phase Comparator Voltage Controlled Oscillator Kasaei 100
Luminance/Chrominance Separation In low-end TV receivers, a low-pass filter with cut-off frequency at 3MHz is typically used to separate the luminance & chrominance signals. The high frequency part of the I component (2 to 3 MHz) is still retained in the luminance signal. The extracted chrominance components can contain significant luminance signal in a scene with very high frequency (luminance energy is not negligible near f c ). These can lead to color bleeding artifacts. For better quality, a comb filter can be used, which will filter out harmonic peaks correspond to chrominance signals. Kasaei 101
What will a monochrome TV see? The monochrome TV receiver uses a LPF with cutoff at 4.2 MHz, & thus will get the composite video (baseband luminance plus the I & Q signals modulated to f c =3.58 MHz). Because the modulated chrominance signal is at very high frequency (227.5 cycles per line), the eye smoothes it out mostly, but there can be artifacts. The LPF in Practical TV receivers have wide transition bands, & the response is already quite low at f c. Kasaei 102
Color TV Broadcasting & Receiving RGB ---> YC1C2 Luminance, Chrominance, Audio Multiplexing Modulation YC1C2 ---> RGB De- Multiplexing De- Modulation Kasaei 103
Transmitter in More Details Audio FM Modulator 4.5 MHz R(t) G(t) B(t) RGB to YIQ Conversion Y(t) I(t) Q(t) LPF 0-4.2 MHz LPF 0-1.5 MHz LPF 0-0.5 MHz Acos(2πf c t) -π/2 Σ Gate BPF 2-4.2 MHz Color Burst Signal Σ VSB To Transmit Antenna Vestigial sideband modulation Kasaei 104
Receiver in More Details BPF, 4.4-4.6MHz Composite video BPF, 0-4.2 MHz VSB Demodulator From Antenna Gate Comb Filter 0-4.2 MHz 2Acos(2πf c t) Phase Comparator Horizontal Sync Signal FM Demodulator + _ Σ Voltage Controlled Oscillator LPF 0-1.5 MHz Kasaei 105 -π/2 LPF 0-0.5 MHz Y(t) I(t) Q(t) YIQ to RGB Conversion Audio R(t) G(t) B(t) To Speaker To CRT
Matlab Simulation of Mux/Demux We will show the multiplexing/demultiplexing of YIQ process for a real sequence ( mobile calendar ): Original Y,I, Q frames. Converted Y,I, Q raster signals & their respective spectrums. QAM of I & Q: choice of f c, waveform & spectrum. Multiplexing of Y & QAM(I+Q): waveform & spectrum. Kasaei 106
Matlab Simulation of Mux/Demux What will a B/W TV receiver see: W/o filtering vs. with filtering. What will a color TV receiver see: Original & recovered Y,I, & Q. Original & recovered color image. Spectrum & waveforms. Kasaei 107
Spectrum of Y, I, & Q 10 6 Y Spectrum 10 6 I Spectrum 10 6 Q Spectrum 10 5 10 5 10 5 10 4 10 4 10 4 10 3 10 3 10 3 10 2 10 2 10 2 10 1 10 1 10 1 10 0 10 0 10 0 10-1 0 5 10 x 10 5 10-1 0 5 10 x 10 5 10-1 0 5 10 x 10 5 Spectrum of Y, I, & Q components, computed from first two progressive frames of mobilcal, 352x240/frame. Maximum possible frequency is 352x240x30/2=1.26 MHz. Notice bandwidths of Y, I, & Q components are 0.8,0.2,0.15 MHz, respectively, if we consider 10^3 as the cut-off magnitude. Kasaei 108
QAM of I & Q: Waveform 80 I Waveform 80 Q Waveform 80 QAM multiplexed I & Q 60 60 60 40 40 40 20 20 20 Gray Level 0 Gray Level 0 Gray Level 0-20 -20-20 -40-40 -40-60 -60-60 -80 0 0.5 1 1.5 Time x 10-4 -80 0 0.5 1 1.5 Time x 10-4 -80 0 0.5 1 1.5 Time x 10-4 Line rate f l =30*240; Luminance f max =30*240*352/2*0.7=.89 MHz. The color subcarrier f c =225*f l /2=0.81MHz.. M(t)=I(t)*cos(2πf c t)+q(t)*sin (2πf c t) Kasaei 109
QAM of I & Q: Spectrum 10 6 I Spectrum 10 6 Q Spectrum 10 6 QAM I+Q Spectrum 10 5 10 5 10 5 10 4 10 4 10 4 10 3 10 3 10 3 10 2 10 2 10 2 10 1 10 1 10 1 10 0 10 0 10 0 10-1 0 5 10 x 10 5 10-1 0 5 10 x 10 5 Spectrum of I, Q, & QAM multiplexed I+Q, fc=225*fl/2=0.81 MHz. 10-1 0 5 10 x 10 5 Kasaei 110
Composite Video: Waveform 250 Y Waveform 250 Composite Waveform 200 200 150 150 Gray Level 100 Gray Level 100 50 50 0 0 0 0.5 1 1.5 0 0.5 1 1.5 Time x 10-4 Time x 10-4 Waveform of the Y signal Y(t) & the composite signal V(t)=Y(t)+M(t). 1 line. Kasaei 111
Composite Video: Spectrum 10 6 Y Spectrum 10 6 Composite Video Spectrum 10 5 10 5 10 4 10 4 10 3 10 3 10 2 10 2 0 2 4 6 8 10 12 0 2 4 6 8 10 12 x 10 5 x 10 5 Kasaei 112
Blown-up View of Spectrum 10 6 Composite Spectrum (beginning) Luminance peaks 10 6 Composite Spectrum (near f c ) Chrominance peaks 10 5 10 5 10 4 10 4 10 3 10 3 Luminance peaks 10 2 0 5 10 15 x 10 4 10 2 7.5 8 8.5 9 x 10 5 Notice that the harmonic peaks of Y & M interleaves near fc. Kasaei 113
Composite Video Viewed as a Monochrome Image w/o filtering Original Y Composite Signal as Y On the right is what a B/W receiver will see if no filtering is applied to the baseband video signal. Kasaei 114
Low-Pass Filter for Recovering Y Frequency response Impulse response (filter coefficients) Magnitude (db) 50 0-50 -100 0.6 0.5 0.4-150 0 2 4 6 Frequency (Hz) 8 10 12 x 10 5 0 0.3 0.2 Phase (degrees) -500-1000 -1500 0 2 4 6 Frequency (Hz) 8 10 12 x 10 5 0.1 0-0.1 0 5 10 15 20 25 f_lpf=30*240/2*150=0.54mhz; fir_length=20; LPF=fir1(fir_length, f_lpf/(fs/2)); Kasaei 115
Recovered Y with Filtering Original Y Recovered Y On the right is what a B/W receiver will see if a lowpass filter with cutoff frequency at about 0.75 MHz is applied to the baseband video signal. This is also the recovered Y component by a color receiver if the same filter is used to separate Y & QAM signal. Y (t)=conv(v(t),lpf(t)) Kasaei 116
Y Waveform Comparison 250 Y Waveform 250 Composite Waveform 250 Y from Composite using LPF 200 200 200 Gray Level 150 100 Gray Level 150 100 Gray Level 150 100 50 50 50 0 0 0 0 0.5 1 1.5 0 0.5 1 1.5 0 0.5 1 1.5 Time x 10-4 Time x 10-4 Time x 10-4 Kasaei 117
Demux Y & QAM (I,Q) 80 QAM Waveform 80 Demultiplexed QAM 60 60 40 40 20 20 Gray Level 0 Gray Level 0-20 -20-40 -40-60 -60-80 -80 0 0.5 1 1.5 0 0.5 1 1.5 Time x 10-4 Time x 10-4 M (t)=v(t)-y (t) Kasaei 118
QMA Modulation & Demodulation Modulated signal: M(t)=I(t)*cos(2πf c t)+q(t)*sin (2πf c t) Demodulated signal: I (t)=2*m(t)*cos(2πf c t), Q (t)=2*m(t)*sin(2πf c t). I (t) contains I(t) at baseband, as well as I(t) at 2f c & Q(t) at 4f c. A LPF is required to extract I(t). s 1( t ) cos( 2 πf 1 t ) m(t) m(t) cos( 2 πf 1 t ) s 1 ( t) LPF s 2 ( t) sin( 2πf1t ) sin( 2πf1t ) LPF s 2 ( t) QAM modulator QAM demodulator Kasaei 119
Low-Pass Filter for Extracting QAM (I+Q) Frequency response Impulse response Magnitude (db) Phase (degrees) 50 0-50 -100 0 2 4 6 8 10 12 Frequency (Hz) x 10 5 0-200 -400-600 -800 0 2 4 6 8 10 12 Frequency (Hz) x 10 5 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0-0.02 0 5 10 15 20 25 f_lpf=0.2mhz; fir_length=20; LPF=fir1(fir_length,f_LPF/(Fs/2)); Kasaei 120
QAM Demodulation: Waveform 80 Original I 80 Demodulated I 80 Demodulation+LPF I 60 60 60 40 40 40 20 20 20 Gray Level 0-20 Gray Level 0-20 Gray Level 0-20 -40-40 -40-60 -60-60 -80 0 0.5 1 1.5 Time x 10-4 -80 0 0.5 1 1.5 Time x 10-4 -80 0 0.5 1 1.5 Time x 10-4 I (t)=2*m(t)*cos(2πf c t) I (t)=conv(i (t),lpf(t)) Kasaei 121
QAM Demodultion: Spectrum 10 6 I Spectrum 10 6 Extracted I Spectrum w/o LPF 10 6 Extracted I Spectrum after LPF 10 5 10 5 10 5 10 4 10 4 10 4 10 3 10 3 10 3 10 2 0 5 10 x 10 5 10 2 0 5 10 x 10 5 10 2 0 5 10 x 10 5 Kasaei 122
original I original Q 50 100 150 200 50 100 150 200 100 200 300 Recovered I 100 200 300 Recovered Q 50 100 150 200 50 100 150 200 100 200 300 100 200 300 Kasaei 123
Original color frame Recovered color frame Kasaei 124
Different Color TV Systems Parameters NTSC PAL SECAM Field Rate (Hz) 59.95 (60) 50 50 Line Number/Frame 525 625 625 Line Rate (Line/s) 15,750 15,625 15,625 Color Coordinate YIQ YUV YDbDr Luminance Bandwidth (MHz) 4.2 5.0/5.5 6.0 Chrominance Bandwidth (MHz) 1.5(I)/0.5(Q) 1.3(U,V) 1.0 (U,V) Color Subcarrier (MHz) 3.58 4.43 4.25(Db),4.41(Dr) Color Modulation QAM QAM FM Audio Subcarrier 4.5 5.5/6.0 6.5 Total Bandwidth (MHz) 6.0 7.0/8.0 8.0 Kasaei 125
Who uses what? From http://www.stjarnhimlen.se/tv/tv.html#worldwide_0 Kasaei 126
Digital Video Digital video by sampling/quantizing analog video raster BT.601 video. Other digital video formats & their applications. Kasaei 127
Digitizing A Raster Video Sample the raster waveform = Sample along the horizontal direction. Sampling rate must be chosen properly: For the samples to be aligned vertically, the sampling rate should be multiples of the line rate. Horizontal sampling interval = vertical sampling interval. Total sampling rate equal among different systems: f s = l l 858 f (NTSC) = 864 f (PAL/SECAM) = 13.5 MHz Kasaei 128
BT.601* Video Format 858 pels 864 pels 720 pels 720 pels 525 lines 480 lines Active Area 625 lines 576 lines Active Area 122 pel 16 pel 132 pel 12 pel 525/60: 60 field/s 625/50: 50 field/s * BT.601 is formerly known as CCIR601 Kasaei 129
RGB <--> YCbCr Y_d = 0.257 R_d + 0.504 G_d + 0.098 B_d + 16, C_b = -0.148 R_d - 0.291 G_d + 0.439 B_d + 128, C_r = 0.439 R_d -0.368 G_d - 0.071 B_d + 128, R_d = 1.164 Y_d + 0.0 C_b + 1.596 C_r, G_d = 1.164 Y_d - 0.392 C_b -0.813 C_r, B_d = 1.164 Y_d + 2.017 C_b + 0.0 C_r, Y_d =Y_d -16, C_b =C_b-128, C_r =C_r-128 Kasaei 130
YCbCr Format RGB: 0-255 Y: 16-235 Cb & Cr (scaled versions of B-Y & R-Y): 16-240 Cr=240 or R=255, G=B=0 (red) Cr=16 or R=0, G=B=255 (cyan) Cb=240 or B=255, G=R=0 (blue) Cb=16 or B=0, G=R=255 (yellow) Kasaei 131
Chrominance Subsampling Formats 4:4:4 For every 2x2 Y Pixels 4 Cb & 4 Cr Pixel (No subsampling) 4:2:2 For every 2x2 Y Pixels 2 Cb & 2 Cr Pixel (Subsampling by 2:1 horizontally only) 4:1:1 For every 4x1 Y Pixels 1Cb & 1 CrPixel (Subsampling by 4:1 horizontally only) 4:2:0 For every 2x2 Y Pixels 1Cb&1CrPixel (Subsampling by 2:1 both horizontally and vertically) Y Pixel Cb and Cr Pixel Kasaei 132
Some Abbreviations ITU-R: International Telecommunications Union- Radio Sector ITU-T: International Telecommunications Union- Telecommunications Sector ISO: International Standards Organization SMPTE: Society of Motion Picture & TV Engineers ISDN: Integrated Service Digital Network CIE: Commission Internationale de L Eclariage Kasaei 133
Some Abbreviations HDTV: High Definition Television MPEG: Motion Picture Export Group SIF: Source Intermediate Format CIF: Common Intermediate Format IAR: Image Aspect Ratio PAR: Pixel Aspect Ratio DVD: Digital Video Disk VCR: Video Cassette Recorder VOD: Video-on-Demand Kasaei 134
Some Abbreviations CCD: Charge-Coupled Device CRT: Cathode Ray Tube LCD: Liquid Crystal Display ATM: Asynchronous Transfer Mode LAN: Local Area Network WAN: Wide Area Network IP: Internet Protocol URL: Universal Resource Locator QoS: Quality of Service Kasaei 135
Digital Video Formats Video Format Y Size Color Sampling Frame Rate (Hz) Raw Data Rate (Mbps) HDTV Over air. cable, satellite, MPEG2 video, 20-45 Mbps SMPTE296M 1280x720 4:2:0 24P/30P/60P 265/332/664 SMPTE295M 1920x1080 4:2:0 24P/30P/60I 597/746/746 Video production, MPEG2, 15-50 Mbps BT.601 720x480/576 4:4:4 60I/50I 249 BT.601 720x480/576 4:2:2 60I/50I 166 High quality video distribution (DVD, SDTV), MPEG2, 4-10 Mbps BT.601 720x480/576 4:2:0 60I/50I 124 Intermediate quality video distribution (VCD, WWW), MPEG1, 1.5 Mbps SIF 352x240/288 4:2:0 30P/25P 30 Video conferencing over ISDN/Internet, H.261/H.263, 128-384 Kbps CIF 352x288 4:2:0 30P 37 Video telephony over wired/wireless modem, H.263, 20-64 Kbps QCIF 176x144 4:2:0 30P 9.1 Kasaei 136
Digital Video Applications CIF for video conferences (about half resolution of BT.601 4:2:0). QCIF for videophone (half resolution of CIF). ITU-T H.261 for transport over ISDN (px64 kbps, p=1,,30). ITU-T H.263 for videophone over 28.8 kbpp modem line. Typically: H.263@20 kbps = H.261@64 kbps. Kasaei 137
Digital Video Applications SIF for video games and CD movies (a quarter the size of active area in BT.601, about the same as CIF). MPEG-1 for movies on VCDs, compresses SIF 30 mbps to 1.1 mbps (quality the same as VHS VCR). MPEG-2 for high quality broadcasting or on DVDs (HDTV 20 mbps). Kasaei 138
Video Terminology Component video Three color components stored/transmitted separately Use either RGB or YIQ (YUV) coordinate New digital video format (YCrCb) Betacam (professional tape recorder) use this format Composite video Convert RGB to YIQ (YUV) Multiplexing YIQ into a single signal Used in most consumer analog video devices S-video Y and C (QAM of I and Q) are stored separately Used in high end consumer video devices High end monitors can take input from all three Kasaei 139
Image Fidelity Criteria There are two types of fidelity criteria: subjective & quantitative. Subjective criteria use rating scales such as goodness scales & impairment scales. Quantitative criteria includes: average LSE, MSE, average MS, SNR, PSNR, & frequency weighted MS. Kasaei 140
Subjective Criteria Table 3: Image goodness scales. Kasaei 141
Subjective Criteria Table 4: Image impairment scales. Sk: score, nk: # observers, n: # grades. Kasaei 142
Quantitative Criteria Kasaei 143
Quantitative Criteria Kasaei 144
Quantitative Criteria Kasaei 145
Quantitative Criteria Kasaei 146
Quantitative Criteria Kasaei 147
Homework Reading assignment: Chap. 1. Problems: Prob. 1.5. Prob. 1.6. Prob. 1.7. Prob. 1.8. Prob. 1.9. Prob. 1.10 Prob. 1.11 Kasaei 148
The End