1 Essence of Image and Video Wei-Ta Chu 2009/9/24
Outline 2 Image Digital Image Fundamentals Representation of Images Video Representation of Videos
3 Essence of Image Wei-Ta Chu 2009/9/24 Chapters 2 and 6 of Digital Image Procesing by R.C. Gonzalez and R.E. Woods, Prentice Hall, 2 nd edition, 2001
Image Sensing and Acquisition 4 Collect the incoming energy and focus it onto an image plane.
A Simple Image Formation Model 5 Denote an image by a 2D function Characterized by two components: Illumination:, determined by the illumination source Reflectance:, determined by the characteristics of the imaged objects.
Image Sampling and Quantization 6 Sampling Quantization Digitizing the coordinate values Digitizing the amplitude values
Image Sampling and Quantization 7 Continuous image projected onto a sensor array Results of image sampling and quantization
Digital Image Representation 8 Dynamic range The number of discrete gray levels allowed for each pixel Due to processing, storage, and sampling hardware considerations, the number of gray levels typically is an integer power of 2: We refer to images whose gray levels span a significant portion of the gray scale as having a high dynamic range.
Digital Image Representation 9 Image size For a square image with width(height) is N, the total number of bits required to represent the image:
Spatial Resolution 10 Sampling is the principal factor determining the spatial resolution of an image. 1024x1024 32x32: Downsampled by a factor of 2
11 Spatial Resolution 1024x1024 Resample 512 x 512 to 1024 x 1024 From 256x256 From 128x128 From 64x64 From 32x32
Gray-Level Resolution(L=256,128,,4,2) 12 256 128 16 8 64 32 2 4
Histogram 13 The histogram of an image with gray level in the range [0,L-1] is a discrete function Normalized histogram
Histogram 14 Useful image statistics Image processing applications Image enhancement Image compression Image segmentation
Color Fundamentals 15 Color spectrum: violet, blue, green, yellow, orange & red Each color in the spectrum blends smoothly into the next The color perceived in an object are determined by the nature of the light reflected from the object
Color Fundamentals 16 Cones can be divided into three principal sensing categories Due to the absorption of the human eyes, colors are seen as variable of three primary colors (red, green, blue) Approximately 65% of all cones are sensitive to red light, 33% to green light, 2% to blue light.
Color Fundamentals 17 Secondary colors of light Magenta (R + B) Cyan (G + B) Yellow (R + G) The primary color of pigments subtract a primary color of light and reflects the other two.
Color Fundamentals 18 Brightness Embodies the chromatic notion of intensity Hue Attribute associated with the dominant wavelength in a mixture of light waves Dominant color as perceived by an observer Saturation The relative purity of the amount of white light mixed with a hue Less saturated: e.g. pink (red+white), lavender (violet+white) Hue and saturation taken together as called chromaticity.
Specifying Colors 19 The amounts of red, green, and blue needed to form any particular color are called the tristimulus values and are denoted X, Y, Z, respectively. A color is then specified by its trichromatic coefficients, defined as Using CIE chromaticity diagram, which shows color composition as a function of x (red) and y(green)
Specifying Colors 20 The point marked green has approximately 63% green and 25% red content. The composition of blue is approximately 13%.
Color Models (Color Spaces) 21 A color model is a specification of a coordinate system and a subspace within that system where each color is represented by a single point. Hardware-oriented & application-oriented RGB color monitor, color video cameras CMY (cyan, magenta, yellow) color printing CMYK (cyan, magenta, yellow, black) color printing HSI (hue, saturation, intensity) closely matching with human perception
The RGB Color Model 22 Based on Cartesian coordinate system Different colors are points on or inside the cube Full color image: 8 bits for each component, total 24 bits
23 The RGB Color Model
The CMY and CMYK Color Models 24 When a surface coated with cyan pigment is illuminated with white light, no red light is reflected from the surface. Cyan subtracts red light Most devices that deposit colored pigments on paper require CMY data input or perform RGB to CMY conversion. Equal amounts of CMY pigments should produce black.
The HSI Color Model 25 RGB/CMY color systems are suited for hardware implementations. RGB system matches nicely with the fact that the human eye is strongly perceptive to red, green, and blue primaries. But RGB and CMY are not well suited for describing colors for human interpretation.
The HSI Color Model 26 We describe a color object by its hue, saturation, and brightness. Hue: color attribute that describes a pure color Saturation: degree of pure color diluted by white light Brightness: measured by intensity HSI color model decouples the intensity component from the color-carrying information
The HSI Color Model 27 Take the RGB cube, stand on the black vertex, with the white vertex above it. The intensity (gray scale) is along The intensity (gray scale) is along the line joining these two vertices.
The HSI Color Model 28 The dot is an arbitrary color point. The angle from the red axis gives the hue, and the length of the vector is the saturation. The intensity of all colors is given by the position of the plane on the vertical intensity axis.
HSI 29 HSI is also known as HSL, HLS HSV color space
Converting colors from RGB to HSI 30 RGB values have been normalized to the range [0,1] The angle θis measured with respect to the red axis of the HSI space.
The LAB (CIELAB) Color Models 31 CIELAB (L * a * b * ) color space L*: lightness dimension a*,b*: two chromatic dimensions that are roughly red-green and blue-yellow. L*a*b* color is designed to approximate human vision http://en.wikipedia.org/wiki/lab_color_space http://coatings.specialchem.com.cn/tc/color/index.aspx?id=cielab
Other Color Models 32 YUV, YIQ, YCbCr color spaces YCbCr is widely used in video/image compression schemes such as MPEG and JPEG Please refer to http://en.wikipedia.org/wiki/color_space
Color Histogram 33 A representation of the distribution of colors in an image. Discretize colors into a number of bins, and counting the number of pixels with colors in each bin. http://rsb.info.nih.gov/ij/plugins/color-inspector.html
Nonuniform Quantization 34 An example in HLS (HSI) space Considering human perception Lee, et al. Spatial color descriptor for image retrieval and Video summarization, IEEE Trans. on Multimedia, 2003.
Characteristics of Histogram 35 The color histogram of an image represents the global statistics (color distribution) of pixels colors Histogram is one of the most useful feature to describe images or be the basis for similarity measure
Histogram-based Difference 36 Bin-wise histogram difference between Image I 1 and I 2
Short Introduction to Image Features 37 Color features Color histogram Color moments Color coherence vectors (CCV) Color correlogram Ma, et al. Benchmarking image features content-based image retrieval, Record of the 32nd Asilomar Conf. on Signals, Systems & Computers, vol 1., 1998.
Short Introduction to Image Features 38 Texture features Tamura features (coarseness, directionality, contrast) Multi-resolution simultaneous auto-regressive model Canny edge histogram Gabor texture feature Pyramid-structured wavelet transform (PWT) feature Tree-structured transform (TWT) feature Ma, et al. Benchmarking image features content-based image retrieval, Record of the 32nd Asilomar Conf. on Signals, Systems & Computers, vol 1., 1998.
Color Moments 39 Containing only the dominant features instead of storing the complete color distributions. Store the first three moments of each color channel of an image in the index. Average Variance Skewness
Color Moments 40
Color Moments 41 Distance between two images I 1 and I 2 Diff. of average Diff. of variance Diff. of skewness
Color Correlogram 42 A color correlogram expresses how the spatial correlation of parts of colors changes with distance. The histogram of an image is defined as The colors in are quantized into The notation is synonymous with and Huang, et al. Image indexing using color correlograms, CVPR, 1997.
Color Correlogram 43 Let a distance be a fixed a priori. Then the correlogram of is defined for, This value gives the probability that a pixel at distance away from the given pixel is of color., The autocorrelogram of captures spatial correlation between identical colors only
44 Essence of Video Wei-Ta Chu 2009/9/24
Constitution of Digital Video Data 45 A natural video stream is continuous in both spatial and temporal domains. In order to represent and process a video stream digitally it is necessary to sample spatially and temporally. Spatial domain Temporal domain
Video Stream 46 Natural scene Camera RGB to YC 1 C 2 Monitor Processing, Storage, Transmission YC 1 C 2 To RGB
Video Data Representation 47 RGB is not very efficient for representing real-world images, since equal bandwidths are required to describe all the three color components. E.g. 8 bits per component, then 24 bits per pixel Human eye is more sensitive to luminance. Many image coding standards and broadcast systems use luminance and color difference signals. YUV and YIQ for analog television standards, YCbCr for their digital version.
Color Models in Video 48 Largely derive from older analog methods for coding color for TV. Luminance is separated from color information. YIQ is the color space used by the NTSC color TV system, employed mainly in North and Central America, and Japan. In Europe, video tape uses the PAL and SECAM codings, which are based on TV that uses a matrix transform called YUV. Digital video mostly uses a matrix transform called YCbCr that is closely related to YUV.
TV Encoding System 49 PAL, short for Phase Alternating Line, is a color encoding system used in broadcast television systems in large parts of the world. SECAM, French for "Sequential Color with Memory"), is an analog color television system first used in France. NTSC is the analog television system in use in the United States, Canada, Japan, South Korea, Taiwan, the Philippines, Mexico, and some other countries
The YUV Color Model 50 The YUV model defines a color space in terms of one luma (brightness) and two chrominance components. The YUV color model is used in the PAL, NTSC, and SECAM composite color video standards. YUV signals are created from an original RGB source. The weighted values of R, G, and B are added together to produce a single Y signal.
The YUV Color Model 51 The U signal is then created by subtracting the Y from the blue signal, and then scaling; V is created by subtracting the Y from the red, and then scaling by a different factor. Y U V
The YCbCr Color Model 52 YCbCr is a family of color spaces used in video and digital photography systems. Y is the luma component and Cb and Cr are the blue and red chroma components. Recommendation 601 specifies 8-bit coding: Y C b C r
Chroma Subsampling 53 4:2:2 indicates horizontal subsampling of the Cb, Cr signals by a factor of 2. Of four pixels labeled as 0 to 3, all four Ys are sent, and every twocb sthe two Cr sare sent. (Y0,Cb0) (Y1,Cr0) (Y2,Cb2) (Y3,Cr2) 4:2:0 subsamples in both the horizontal and vertical dimensions by a factor of 2.
Examples 54 Given image resolution of 720x576 pixels represented with 8 bits each component, the bit rate required is: 4:4:4 resolution: 720x576x8x3 = 10 Mbits/frame 4:2:0 resolution: (720x576x8) + (360x288x8)x2 = 5 Mbits/frame
Motion Estimation 55 Successive video frames may contain the same objects (still or moving). Motion estimation examines the movement of objects in an image sequence to try to obtain vectors representing the estimated motion.
Motion Estimation 56 The Essence of Image and Video Compression, by A.C. Kokaram http://www.mee.tcd.ie/~ack/teaching/1e8/lecture3.pdf
Three Typical Types of Coded Picture 57 I frame (intraframe) Intraframe encoded without any temporal prediction P frame (forward predicted frame) Interframe encoded using motion predition from the previous I or P frame B frame (bidirectionally predicted frame) Interframe encoded using interpolated motion prediction between the previous I or P frames and the next I or P frames.
Motion Prediction 58 A typical Group of Picture (GOP) in MPEG-2
Short Introduction to Video Features 59 Motion-based features Camera motion, object motion Motion activity/magnitude Moving object detection Shot-based features Average shot length/shot change frequency Scene-based features
Motion Type 60 Camera motion (global motion) Zoom-in/Zoom-out Pan Tilt Object motion
Motion Activity/Magnitude 61 Attributes: Intensity of activity Direction of activity Spatial distribution of activity Indication of the number and size of active regions Temporal distribution of activity Variation of activity over the duration of a video segment or shot
62 Average Shot Length/Shot Change Frequency A statistical measurement which divides the total length of the film by the number of shots. Average duration of a shot between cuts Directors often change shots frequently (shorter ASL) to attract the audience E.g. commercials Video segments with longer ASLs usually present peaceful scenes.
Next Week 63 Video syntax analysis