Midterm Review Yao Wang Polytechnic University, Brooklyn, NY11201 yao@vision.poly.edu
Yao Wang, 2003 EE4414: Midterm Review 2 Analog Video Representation (Raster) What is a video raster? A video is represented as consecutive sets of frames and each frame in turn consists of successive scan lines Representing a 3D signal as a 1D signal What determines necessary frame rates and line rates? Maximum temporal frequency (how fast object moves) Maximum vertical frequency (fine details in a picture) Sensitivity of the human visual system (frame merging and line merging beyond certain frame rate and line rate) How does the spectrum of a raster video looks like? Adjacent lines are similar -> fundamental frequency=line rate Harmonic peaks at multiples of the line rate Peak values gradually reduces How to estimate the bandwidth of a raster video signal? Maximum number of samples per second / 2 * Kell Factor
Yao Wang, 2003 EE4414: Midterm Review 3 Progressive vs. Interlaced Scan Progressive scan: Each frame consists of consecutive lines Interlaced scan: Each frame consists of two fields Each field consists of alternating lines Why using interlacing? At the same overall line rate (lines/second), interlaced scan enables Faster temporal refresh (field_rate=2*frame_rate), But lower vertical resolution (field_line_spacing=2* frame_line_spacing) A smart way to realize 60 Hz temporal sampling frequency (necessary for rendering fast moving objects) with the available display/capture technology at the time of TV system development (late 1930 s) But do have notorious interlacing artifacts for special scenes
Yao Wang, 2003 EE4414: Midterm Review 4 Basic Modulation Techniques for Analog B/W TVs What are the three major modulation techniques AM, FM, PM What are the differences between DSB, SSB, VSB? All are variation of AM DSB: retain both USB and LSB require twice the baseband bandwidth, but simple SSB: retain only USB or LSB Bandwidth efficient, but difficult to implement (need sharp BPF) VSB: retain a small portion of LSB in addition to USB Good compromise between DSB and SSB, need shaping and equalizing filters Used to modulate the video portion of a TV signal onto a carrier frequency
Yao Wang, 2003 EE4414: Midterm Review 5 Migration to Color TV How to be backward compatible with B/W TV? Use luminance+chrominance color coordinate (YIQ or YUV) Modulate chrominance components to high end of the luminance spectrum, using QAM Why not using RGB for transmission? R,G, B are correlated, not efficient for transmission Y forms the B/W signal to be extracted by B/W TV receiver I and Q have lower bandwidth than Y Should know how to convert between RGB and YIQ QAM Modulate two signals with similar bandwidth into the same USB and LSB, bandwidth efficiency = SSB Used to multiplex I and Q into the upper band of the Y spectrum in NTSC How to choose the modulation frequency for QAM? Should understand block diagram of QAM modulator and demodulator
Yao Wang, 2003 EE4414: Midterm Review 6 Quadrature Amplitude Modulation (QAM) A method to modulate two signals onto the same carrier frequency, but with 90 o phase shift cos( 2 πf 1 t ) cos( 2 πf 1 t ) s 1( t ) m (t ) m (t ) LPF s 1 ( t ) s 2 ( t ) sin( 2πf1t ) sin( 2πf1t ) LPF s 2 ( t ) QAM modulator QAM demodulator
Yao Wang, 2003 EE4414: Midterm Review 7 QAM for Multiplexing Y,I,Q (f ) Luminance Chrominance 0 f l 2f l 3f l 225f l 226f l 227f l 228f l 229f l 230f l f f c (Color subcarrier)
Yao Wang, 2003 EE4414: Midterm Review 8 Multiplexing of luminance, chrominance and audio (Composite Video Spectrum) 1.25 MHz 6.0 MHz 4.5 MHz 4.2 MHz 3.58 MHz Luminance I I and Q Audio f p f c f a f Picture carrier Color subcarrier Audio subcarrier (b)
Yao Wang, 2003 EE4414: Midterm Review 9 Color TV Broadcasting and Receiving RGB ---> YC1C2 Luminance, Chrominance, Audio Multiplexing Modulation YC1C2 ---> RGB De- Multiplexing De- Modulation
Yao Wang, 2003 EE4414: Midterm Review 10 Transmitter in More Details Audio FM modulator 4.5MHz R(t) G(t) B(t) RGB to YIQ conversion Y(t) I(t) Q(t) LPF 0-4.2MHz LPF 0-1.5MHz LPF 0-0.5MHz Acos(2πf c t) -π/2 Σ Gate BPF 2-4.2MHz Color burst signal Σ VSB To transmit antenna
Yao Wang, 2003 EE4414: Midterm Review 11 Receiver in More Details BPF, 4.4-4.6MHz Composite video BPF, 0-4.2 MHz VSB Demodulator From antenna Gate Comb Filter 0-4.2MHz 2Acos(2πf c t) Phase comparator Horizontal sync signal FM demodulator + _ Σ Voltage controlled oscillator -π/2 LPF 0-1.5MHz LPF 0-0.5MHz Y(t) I(t) Q(t) YIQ to RGB conversion Audio R(t) G(t) B(t) To speaker To CRT
Yao Wang, 2003 EE4414: Midterm Review 12 Analog Color TV Systems Parameters NTSC PAL SECAM Field Rate (Hz) 59.95 (60) 50 50 Line Number/Frame 525 625 625 Line Rate (Line/s) 15,750 15,625 15,625 Color Coordinate YIQ YUV YDbDr Luminance Bandwidth (MHz) 4.2 5.0/5.5 6.0 Chrominance Bandwidth (MHz) 1.5(I)/0.5(Q) 1.3(U,V) 1.0 (U,V) Color Subcarrier (MHz) 3.58 4.43 4.25(Db),4.41(Dr) Color Modulation QAM QAM FM Audio Subcarrier 4.5 5.5/6.0 6.5 Total Bandwidth (MHz) 6.0 7.0/8.0 8.0
Yao Wang, 2003 EE4414: Midterm Review 13 Digitizing Analog Raster Digitization = Sampling + Quantization Sample the raster waveform = Sample along the horizontal direction What are the factors considered when determining the sampling rate for Y? Know the format of BT.601 video Why can we sample the chrominance components at lower rates? What are some of the color sub-sampling formats? Digital color coordinates: YCbCr Know conversion between RGB and YCbCr (=YUV)
Yao Wang, 2003 EE4414: Midterm Review 14 BT.601* Video Format 858 pels 864 pels 720 pels 720 pels 525 lines 480 lines Active Area 625 lines 576 lines Active Area 122 pel 16 pel 132 pel 12 pel 525/60: 60 field/s 625/50: 50 field/s Pixels in non-shaded areas correspond to samples taken during horizontal/vertical retrace * BT.601 is formerly known as CCIR601
Yao Wang, 2003 EE4414: Midterm Review 15 Chrominance Subsampling Formats 4:4:4 For every 2x2 Y Pixels 4 Cb & 4 Cr Pixel (No subsampling) 4:2:2 For every 2x2 Y Pixels 2 Cb & 2 Cr Pixel (Subsampling by 2:1 horizontally only) 4:1:1 For every 4x1 Y Pixels 1Cb& 1 CrPixel (Subsampling by 4:1 horizontally only) 4:2:0 For every 2x2 Y Pixels 1Cb&1CrPixel (Subsampling by 2:1 both horizontally and vertically) Y Pixel Cb and Cr Pixel I,Q raster are sampled at the same rate as Y I,Q raster are sampled at ½ rate I,Q raster are sampled at ¼ rate Vertical downsampling from 4:2:2
Yao Wang, 2003 EE4414: Midterm Review 16 Video Format Conversion What are some of the commonly required conversion problems? Interlaced -> progressive (AKA: deinterlacing) NSTC <-> PAL SDTV <-> HDTV What is deinterlacing? What are some simple deinterlacing method (should be able to do calculation on example images and understand matlab scripts) Field merging, field averaging Line averaging Field and line averaging
Yao Wang, 2003 EE4414: Midterm Review 17 Deinterlacing Problem From [Wang02]
Yao Wang, 2003 EE4414: Midterm Review 18 Block Matching Algorithm for Motion Estimation MV Search Region Frame t-1 (Reference Frame) Frame t (Predicted frame)
Yao Wang, 2003 EE4414: Midterm Review 19 Exhaustive Block Matching Algorithm (EBMA) For each MB in a new (predicted) frame Search for a block in a reference frame that has the lowest matching error Using sum of absolute differences (SAD) between corresponding pels Search range: depends on the anticipated motion range Displacement between the current MB and the best matching MB is the MV Current MB is replaced by the best matching MB (motioncompensated prediction or motion compensation) Widely used for motion compensated prediction in video coding Because its simplicity and optimality in minimizing prediction error
Yao Wang, 2003 EE4414: Midterm Review 20 Integer- vs. Half-Pel EBMA Integer-pel EBMA Move the candidate block by 1 pixel to the left or down each time Half-pel EBMA: Move the candidate block by ½ pel to the left or down each time Need to interpolate the reference frame by factor of 2 both horizontally and vertically, using bilinear interpolation More accurate than integer-pel EBMA, but also more complex! (4x) Should know the effect of block size, search range, search accuracy (integer vs. half-pel) on prediction accuracy and computation time Should be able to write a pseudo code (C or MATLAB style) for implementing integer-pel EBMA Pseudo code for half-pel EBMA is not required!
Yao Wang, 2003 EE4414: Midterm Review 21 Hierarchical BMA (HBMA) Problems with EBMA: blocking artifacts Requires a lot of computation HBMA overcome these problem by using multi-resolution search Build a pyramid of both reference (target) frame and the current (anchor) Search at the top level of the pyramid first Interpolate the resulting motion field and use as initial motion field for the next level Can use reduced search range and accuracy at higher levels Should know the principle of HBMA Implementation (pseudo-code) not required!
Yao Wang, 2003 EE4414: Midterm Review 22 Key Ideas in Video Coding Predict a new frame from a previous frame and only specify the prediction error (INTER mode) Code the prediction error using an image coder (e.g., DCT-based as in JPEG) Prediction errors have smaller energy than the original pixel values and can be coded with fewer bits Those regions that cannot be predicted well will be coded directly using DCT-based method (INTRA mode) Use motion-compensated temporal prediction to account for object motion Work on each macroblock (MB) (16x16 pixels) independently for reduced complexity Motion compensation done at the MB level DCT coding of error at the block level (8x8 pixels) Block-based hybrid video coding
Yao Wang, 2003 EE4414: Midterm Review 23 Encoder Block Diagram of a Typical Block-Based Hybrid Coder From [Wang02]
Yao Wang, 2003 EE4414: Midterm Review 24 Decoder Block Diagram From [Wang02]
Yao Wang, 2003 EE4414: Midterm Review 25 Different Coding Modes Intra: coded directly; Predictive: predicted from a previous frame; Bidirectional: predicted from a previous frame and a following frame. Can be done at the block or frame level. From [Wang02]
Yao Wang, 2003 EE4414: Midterm Review 26 What should you know about video coding? What are the principle steps in a video coder? What are the three types of information coded? You should be able to draw the block diagram of a typical block-based video codec (encoder and decoder) using motioncompensation and know the function of each step Why do we use motion-compensated prediction? What are the difference between I, B, and P modes? Why do we use different modes? What may be the problem if we use P-modes only (except the first frame)? What are the basic steps in DCT-based coding? How to apply it to I and P and B blocks, respectively? Why is error-resilience and error-concealment important in video encoder and decoder design? What is scalable coding? What are the benefits and trade-offs?
Yao Wang, 2003 EE4414: Midterm Review 27 Video Coding Standards H.261: First video coding standard, targeted for video conferencing over ISDN Uses block-based hybrid coding framework with integer-pel MC and loop filtering H.263: Improved quality at lower bit rate, to enable video conferencing/telephony below 54 bkps (modems, desktop conferencing) Half-pel MC and other improvement MPEG-1 video Video on CD and video on the Internet (good quality at 1.5 mbps) Half-pel MC and bidirectional MC Use GOP structure to enable random access MPEG-2 video (H.262) SDTV/HDTV/DVD (4-15 mbps) Focus on motion estimation and DCT coding for interlaced video Different scalability modes You should know the major differences between above standards in both targeted applications and techniques used
Yao Wang, 2003 EE4414: Midterm Review 28 Not Required for Midterm But Good to Know! TV system: Cable TV VCR Digital video basics Other format conversion techniques except deinterlacing Motion estimation HBMA Video coding Error resilience issue and techniques Scalable coding techniques Standards Actual coder block diagrams and syntaxes not required
Yao Wang, 2003 EE4414: Midterm Review 29 Logistics Time: 10/21 (Tuesday) 11-12:50 Closed book, 1 sheet of notes (both sides) OK Office hour: 10/16 (Thursday) afternoon 10/20 (Monday) By email What to prepare for: Should be able to answer all questions in HW1-HW4 without referring to lecture notes/handouts, except those problems corresponding to NOT required categories For complex block diagrams, you don t need to memorize the actual block diagrams, but you should understand the functionality of each block, if a block diagram is given.