Sampling Issues in Image and Video

Sampling Issues in Image and Video Spring 06 Instructor: K. J. Ray Liu ECE Department, Univ. of Maryland, College Park Overview and Logistics Last Time: Motion analysis Geometric relations and manipulations Today: 2-D sampling at Rectangular grid Lattice theory for multidimensional sampling at non-rectangular grid Sampling and resampling for video ENEE631 Digital Image Processing (Spring'06) ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [2] Sampling: From 1-D 1 D to 2-D 2 D and 3-D3 Review: 1-D 1 D Sampling Time domain Multiply continuous-time signal with periodic impulse train Frequency domain X(ω) x(t) T p(t) = Σ k δ ( t - kt) Review Oppenheim Sig. & Sys Chapt.7 (Sampling) Chapt.3,4,5 (FS,FT,DFT) Duality: sampling in one domain tiling in another domain FT of an impulse train is an impulse train (proper scaling & stretching) 2π/T x s (t) X s (ω) ω P(ω) = Σ k δ ( ω -2kπ/T) *2π/T 2π/T ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [3] ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [4] 1

Review: 1-D 1 D Sampling Theorem 1-D Sampling Theorem A 1-D signal x(t) bandlimited within [-ω B,ω B ] can be uniquely determined by its samples x(nt) if ω s > 2ω B (sample fast enough). Using the samples x(nt), we can reconstruct x(t) by filtering the impulse version of x(nt) by an ideal low pass filter Sampling below Nyquist rate (2ω B ) cause Aliasing X s (ω) with ω s > 2ω B Perfect Reconstructable X s (ω) with ω s < 2ω B Aliasing Extend to 2-D 2 D Sampling with Rectangular Grid Bandlimited 2-D signal Its FT is zero outside a bounded region ( ζ x > ζ x0, ζ y > ζ y0 ) in spatial freq. domain Real-word multi-dimensional signals often exhibit diamond or football shape of support With spectrum normalization, we will get spherical shape of support -ωs ω B ω s =2π/T ω B ω s =2π/T Jain s Fig.4.6 ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [5] ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [6] 2-D D Sampling (cont d) 2-D Comb function comb(x,y; Δx, Δy) = Σ m,n δ ( x - mδx, y - nδy ) ~ separable function FT: COMB(ζ x, ζ y ) = comb(ζ x, ζ y ; 1/Δx, 1/Δy) / ΔxΔy Sampling vs. Replication (tiling) Nyquist rates (2ζ x0 and 2ζ y0 ) Aliasing Jain s Fig.4.7 2-D D Sampling: Beyond Rectangular Grid Sampling at nonrectangular grid May give more efficient sampling density when spectrum region of support is not rectangular Sampling density measured by #samples needed per unit area E.g. interlaced grid for diamondshaped region of support equiv. to rotate 45-deg. of rectangular grid spectrum rotate by the same degree From Wang s book preprint Fig.4.2 ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [7] ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [8] 2

General Sampling Lattice Lattice Λ in K-dimension space R K From Wang s book preprint Fig.3.1 A set of all possible vectors represented as integer weighted combinations of K linearly independent basis vectors K K Λ = x R x = n jv j, nk Z j= 1 Generating matrix V (sampling matrix) V = [v 1, v 2,, v k ] => lattice points x = V n e.g., identity matrix V ~ square lattice Voronoi cell of a lattice A unit cell of a lattice, whose translations cover the whole space Consists of vectors that are closer to the origin than to other lattice points cell boundaries are equidistant lines between surrounding lattice points Sampling density d(λ) = 1 / det(v) det(v) measures volume of a cell; d(λ) is # lattice points in unit volume ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [9] Example of Lattices 1 V1 = 0 V2 = 0 1 3 / 2 1/ 2 0 1 Sampling Density: d1 = 1 d2 = 2 / 3 (rectangular) (hexagonal) From Wang s book preprint Fig.3.1 ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [10] Frequency Domain View & Reciprocal Lattice Reciprocal lattice Λ # for a lattice Λ (with generating matrix V) Generating matrix of Λ # is U = (V T ) -1 Basis vectors for Λ and Λ # are orthonormal to each other: V T U= I Denser lattice Λ has sparser reciprocal lattice Λ # : det(u) = 1 / det(v) Frequency domain view of sampling over lattice Sampling in spatial domain Repetition in freq. Domain Repetition grid in freq. domain can be described by reciprocal lattice Aliasing and prefiltering to avoid aliasing Aliasing happens when Voronoi cell of reciprocal lattice overlapped Sampling Efficiency From Wang s book preprint Fig.4.2 & 3.5 Consider spherical signal spectrum support Most real-world signals have symmetric freq. contents in many directions The multi-dim spectrum can be approximated well by a sphere (with proper scaling spectrum support) Voronoi cell of reciprocal lattice need to cover the sphere to avoid aliasing Tighter fit of the Voronoi cell to the sphere requires less sampling density What lattice gives the best sphere-covering capability? Sampling Efficiency ρ = volume(unit sphere) / d(λ) prefer close to 1 ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [11] ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [12] 3

Sampling Lattice Conversion Original From Wang s book preprint Fig.4.4 Recall: 1-D 1 Upsample and Downsample Intermediate Targeted From Crochiere-Rabiner Multirate DSP book Fig.2.15-16 ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [13] ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [14] General Procedures for Sampling Rate Conversion From Wang s book preprint Fig.4.1 ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [15] Example: Frame Rate Conversion Video sampling: formulate as a 3-D sampling problem Note: different signal characteristics and visual sensitivities along spatial and temporal dimensions (see Wang s Sec.3.3 on video sampling) General Approach to frame rate conversion Upsample => LPF => Downsample Interlaced 50 fields/sec 60 fields/sec Analyze in terms of 2-D sampling lattice (y, t) Convert odd field rate and even field rate separately do 25 30 rate conversion twice not fully utilize info. in the other fields Deinterlace first then convert frame rate do 50 60 frame rate conversion: 50 300 60 Simplify 50 60 by converting 5 frames 6 frames each of output 6 frames is from two nearest frames of the 5 originals weights are inversely proportional to the distance between I/O May do motion-interpolation for hybrid-coded video ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [16] 4

Case Studies on Sampling and Resampling in Video Processing Reading Assignment: Wang s s book Chapter 4 From Wang s book preprint Fig.4.3 ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [17] ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [18] Video Format Conversion for NTSC PAL Require both temporal and spatial rate conversion NTSC 525 lines per picture, 60 fields per second PAL 625 lines per picture, 50 fields per second Ideal approach (direct conversion) 525 lines 60 field/sec 13125 line 300 field/sec 625 lines 50 field/sec 4-step sequential conversion Deinterlace => line rate conversion => frame rate conversion => interlace From Wang s book preprint Fig.4.9 ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [19] ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [20] 5

Simplified Video Format Conversion 50 field/sec 60 field/sec Simplified after deinterlacing to 5 frames 6 frames Conversion involves two adjacent frames only 625 lines 525 lines Simplified to 25 lines 21 lines Conversion involves two adjacent lines only ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [21] ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [22] Interlaced Video and Deinterlacing Interlaced video Odd field at 0 Even field at Δt Odd field at 2Δt Even field at 3Δt Deinterlacing Merge to get a complete frame with odd and even field De-interlacing: Practical Approaches Spatial interpolation Vertical interpolation within the same field (1-D upsample by 2) Line averaging ~ average the line above and below D=(C+E)/2 Temporal interpolation 2-frame field merging => artifacts 3-frame field averaging D=(K+R)/2 fill in the missing odd field by averaging odd fields before and after Spatial-temporal interpolation Line-and-field averaging D=(C+E+K+R)/4 Examples from http://www.geocities.com/lukesvideo/interlacing.html ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [23] ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [24] 6

Motion-Compensated De-interlacing Stationary video scenes Temporary deinterlacing approach yield good result Scenes with rapid temporal changes Artifacts incurred from temporal interpolation Spatial interpolation alone is better than involving temporal interpolation Switching between spatial & temporal interpolation modes Based on motion detection result Hard switching or weighted average Motion-compensated interpolation Summary of Today s s Lecture Sampling and resampling issues in 2-D and 3-D Sampling lattice and frequency-domain interpretation Sampling rate conversion Next Lecture: Introduction to digital watermarking for image and video Readings Wang s book: Sec. 3.1-3.3, 3.5; Chapter 4 Computer Graphics Chapter 5 (Hearn-Baker, Prentice- Hall, 2 nd Ed) ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [25] ENEE631 Digital Image Processing (Spring'06) Lec24 2-D and 3-D Sampling [26] 7