ELEG5502 Video Coding Technology

Transcription

1 ELEG5502 Video Coding Technology Ngan King Ngi 顏慶義 Room 309, Ho Sin Hang Engineering Building Department of Electronic Engineering, CUHK

2 Objectives After completing this course, you would know how video images are formed, perceived by sensors and represented in various formats; understand how the video is characterized in space and time, sampled and modeled using different physical models; have learnt the general principles of video compression and coding; and have been introduced the latest video coding standards, such as MPEG-4, H.264, and HEVC. 2

3 Resourses Tutor: Mr. Cheung Chi Ho Website: References [1] Yao Wang, Jorn Ostermann, Ya-Qin Zhang: Video Processing and Communications, Prentice-Hall, [2] Iain E.G. Richardson: H.264 and MPEG-4 video compression, Wiley, [3] Iain E.G. Richardson: The H.264 Advanced Video Compression Standard, Wiley, [4] K.R. Rao, J.J. Hwang: Techniques and Standards for Image, Video, and Audio Coding, Prentice Hall,

4 Assessment Final Examination: 50% 2 labs: 20% (10% each) You are to perform 2 software lab experiments. 1 mini-project: 20% You are to write/modify computer programs to implement some basic video processing algorithms. 5 quizzes: 10% (2% each) 4

5 Course Outline 1. Video Formation, Perception and Representation 2. Video Characterization, Sampling and Modeling 3. Basic Principles of Video Coding 4. Video Coding Standards MPEG-4 H.264 HEVC 5

6 Video Formation, Perception and Representation Color Perception and Specification Video Capture and Display Analog Video Raster Analog Color Television Systems Digital Video 6

7 Color Perception and Specification Light and Color (1) Light consists of an electromagnetic (EM) wave, with wavelengths with a range of nm, to which the human eye is sensitive. The energy of light in terms of the rate at which energy is emitted is measured by flux. The unit of flux is watt. The radiant intensity of a light (which is directly related to our perception of brightness of the light) is defined as the flux radiated into a unit solid angle in a particular direction, measured in watt/solid-angle. The radiant intensity distribution of a light is C(X, t, λ), which specifies the light intensity at wavelength λ, at spatial location X = (X, Y, Z) and time t. 7

8 Color Perception and Specification Light and Color (2) The perceived color of a light depends on its spectral content or wavelength composition. In general, a light that has a very narrow bandwidth is referred as a spectral color. A light that has equal energy across the entire visible spectrum appears white, and is said to be achromatic. There are two types of light sources: illuminating source which emits an EM wave, and reflecting source which reflects an incident wave. Illuminating light follows additive rule [R+G+B=White], while reflecting light obeys subtractive rule [White (R+G+B)=Black]. 8

9 Color Perception and Specification Human Perception of Color (1) There are two types of photoreceptors located in the retina: cones, which function under bright light and can perceive color tone; and rods, which work under low ambient light and can only extract luminance information. There are three types of cones: red ( 570 nm), green ( 535 nm), and blue ( 445 nm). The responses of these receptors to an incoming light distribution C(λ), can be described by: C = C( λ) a ( λ) dλ, i r, g, b, i i = where a r (λ), a g (λ), a b (λ), are the frequency responses of the absorption functions of the red, green, and blue cones. 9

10 Color Perception and Specification Fig. 1.1 Frequency responses of the three types of cones in the human retina and the luminous efficiency function. [1] 10

11 Color Perception and Specification Human Perception of Color (2) The combination of these three colors, rather than the complete light spectrum enables a human to perceive any color. This is known as the tri-receptor theory of color vision. There are two attributes that describe the color sensation of a human being: luminance and chrominance. Luminance refers to the perceived brightness of the light, which is proportional to the total energy in the visible band. Chrominance describes the perceived color tone of a light, which depends on the wavelength composition of the light. Chrominance is in turn characterized by two attributes: hue and saturation. 11

12 Color Perception and Specification Human Perception of Color (3) Hue specifies the color tone, which depends on the peak wavelength of the light. Saturation describes how pure the color is, which depends on the bandwidth of the light spectrum. The human visual system (HVS) converts the three color values into one value that is proportional to the luminance and two other values that are responsible for the perception of chrominance. This is known as the opponent color model of HVS. It has been found that the same amount of energy produces different sensation of brightness at different wavelengths, which is characterized by a relative luminous efficiency function, a y (λ). 12

13 Color Perception and Specification Human Perception of Color (4) The parameter a y (λ) is essentially the sum of the frequency responses of all three types of cones as shown in Fig The luminance Y is related to the incoming light spectrum by Y = C( λ) ay ( λ) dλ 13

14 Color Perception and Specification The Trichromatic Theory of Color Mixture (1) Most colors C can be produced by mixing three properly chosen primary colors, i.e., C = T k C k where T k are the amount of the three primary colors required to match the color C. The T k are known as the tristimulus values. The most popular primary set for illuminating light sources contains red, green and blue colors, known as RGB primary. The most common primary set for reflecting light sources contains cyan, magenta and yellow colors, known as CMY primary. RGB primary and CMY primary are complement to each other. k = 1,2,3 14

15 Color Perception and Specification The Trichromatic Theory of Color Mixture (2) The tristimulus values, T i, for any color with a spectrum C(λ) can be obtained by first determining the color matching functions, m i (λ), for the primary colors, C i, i = 1, 2, 3, by visual experiments under controlled viewing conditions. T i = C( λ) mi ( λ) dλ, i = 1, 2, 3. To record the color of an incoming light, a camera must have three sensors that have frequency responses similar to the color matching functions of a chosen primary set. To display a color picture, the display device must emit three optical beams of the chosen primary set with appropriate intensities, as specified by the tristimulus values. 15

16 Color Perception and Specification Color Specification by Tristimulus Values For the RGB primary color set, the three tristimulus values are denoted by R (red), G (green) and B (blue). A color is then specified by its tristimulus coefficient, defined as x = R G ; y R + G + B = R + G + B ; B z = R + G + B ; and x + y + z =1. The exact color is specified on a CIE chromaticity diagram, shown in Fig

17 Color Perception and Specification Fig. 1.2 Chromaticity diagram. 17

18 Color Perception and Specification Color Specification by Luminance and Chrominance Attributes It is desirable to describe a color in terms of its luminance and chrominance content separately, to enable more efficient processing of the color information. In the CIE XYZ primary coordinate, Y directly measures the luminance, whereas X and Z characterize hue and saturation. The CIE XYZ coordinate is related to the CIE RGB coordinate by: X Y Z = R G B Other color coordinates such as YIQ and YUV are derived from the XYZ coordinate. 18

19 Video Capture and Display Principles of Color Video Imaging (1) A video (a sequence of moving images) records the emitted and/or reflected light intensity from the objects in a scene that is observed by a video camera. The image function ψ(x, y, t) captured by the camera at any time t is the projection of the light distribution in the 3-D scene onto a 2-D image plane. In general, a video signal has spatial and temporal dimensions. The spatial dimension (x, y) depends on the viewing area; and the temporal dimension (t) depends on the duration for which the scene is captured. A point in the image plane is called a pixel or pel. 19

20 Video Capture and Display Principles of Color Video Imaging (2) If the camera has only one luminance sensor, ψ(x, y, t) is a scalar function representing the luminance of the projected light. Luminance image is also known as grey scale image. On the other hand, if the camera has three separate sensors, each tuned to a chosen primary color, the signal is a vector function that contains three color values at every point. A black and white image strictly has two colors: black and white. A monochrome image is one which consists of colors of a narrow band of wavelengths. 20

21 Video Capture and Display Video Cameras There are basically two types of video cameras: (1) tube-based cameras such as vidicons, plumbicons, or orthicons; and (2) solidstate cameras, e.g., charge-coupled devices (CCDs). Many cameras of today are CCD-based because they are much smaller and lighter to acquire images of the same spatial resolution. To capture color, there are usually three types of CCD sensors, each with a frequency response that is determined by the color matching function of the chosen primary color. 21

22 Video Capture and Display CN Component video CS Composite video Fig. 1.3 Schematic block diagram of a professional color video camera. [1] 22

23 Video Capture and Display Video Displays (1) In the past, the most common device to display a video is the cathode ray tube (CRT). In a CRT monitor, an electron gun emits an electron beam across the screen, exciting phosphors with intensities proportional to the intensity of the video signal at corresponding locations. To display color, three beams are emitted by three separate guns, exciting red, green and blue phosphors with the desired intensity combination at each location. The CRT display is bright but bulky. Nowadays, flat-panel displays such as the liquid crystal display (LCD) has been developed. 23

24 Video Capture and Display Video Displays (2) The LCD makes use of an applied electric field to change the optical properties and consequently the brightness or color of the liquid crystal. The electric field can be generated by an array of activematrix thin-film transistors (TFTs). LCD panel needs backlight behind it to be visible as it does not itself emits light. A plasma display panel (PDP) is a type of flat panel display where many tiny cells between two panels of glass hold a mixture of noble gases. The gas in the cells is electrically turned into a plasma which then excites phosphors to emit light. Plasma technology eliminates the need for TFTs and makes large screen displays possible. Source: Wikipedia 24

25 Video Capture and Display Video Displays (3) An LED display is a flat panel display, which uses light-emitting diodes as a video display. There are two types of LED panels: conventional (using discrete LEDs) and surface-mounted device (SMD) panels. Most outdoor screens and some indoor screens are built around discrete LEDs, also known as individually mounted LEDs. A cluster of red, green, and blue diodes is driven together to form a full-color pixel, usually square in shape. These pixels are spaced evenly apart and are measured from center to center for absolute pixel resolution. Source: Wikipedia 25

26 Video Capture and Display Video Displays (4) Most indoor screens on the market are built using SMD technology - a trend that is now extending to the outdoor market. An SMD pixel consists of red, green, and blue diodes mounted in a single package, which is then mounted on the driver PCB (printed-circuit board). The individual diodes are smaller than a pinhead and are set very close together. The disadvantage is that the maximum viewing distance is reduced by 25% from the discrete diode screen with the same resolution. The so-called LED TV is actually LCD panel backlit by LEDs replacing the old CCFL (Cold Cathode Fluorescent Lamps) technology making the LCD panel thinner, and having a higher contrast ratio because LEDs can be turned on and off during display. Source: Wikipedia 26

27 Video Capture and Display Video Displays (5) An OLED (organic light-emitting diode) is a LED in which the emissive electroluminescent layer is a film of organic compound which emits light in response to an electric current. This layer of organic semiconductor is situated between two electrodes; typically, at least one of these electrodes is transparent. OLEDs are used to create flexible digital displays in devices such as television screens, computer monitors, portable systems such as mobile phones, handheld game consoles and PDAs. An OLED display works without a backlight; thus, it can be thinner and lighter than a LCD. In low ambient light conditions (such as a dark room), an OLED screen can achieve a higher contrast ratio than an LCD, regardless of whether the LCD uses CCFL or an LED backlight. Source: Wikipedia 27

28 Video Capture and Display Video Displays (6) LG flexible OLED display Samsung OLED TV 28

29 Video Capture and Display Composite and Component Video A component video is specified by a tristimulus color representation or a luminance-chrominance representation. The three color components can be multiplexed into a single signal to form a composite video. The composite format is used in analog TV systems where the chrominance signals are modulated to a higher frequency than the luminance and adding the resulting modulated chrominance signals to the luminance signal. A filter is used to separate the luminance and chrominance signals for display in a color monitor. With a grey-scale monitor, only the luminance signal is extracted for display. 29

30 Video Capture and Display Gamma Correction (1) The output signals from most cameras are not linearly related to the actual color values: v c = B c where B c represents the actual light brightness and v c the camera output voltage. The γ c value ranges from 1.0 for most CCD cameras to 1.7 for a vidicon camera. Similarly, most display devices also suffer from such a non-linearity between the input voltage value v d and the displayed color intensity B d i.e., where γ d typically has a value of γ B d = v d γ c d 30

31 Video Capture and Display Gamma Correction (2) To equalize the captured light intensity B c and the displayed light intensity B d, a gamma-correction factor 1/γ c γ d is applied: v d = v c 1/γ γ In most TV systems, a value of γ c γ d = 2.2 is used. c d 1/γ cγ d v c v d = v c v d B c B d Fig. 1.4 Gamma correction. 31

32 Analog Video Raster Progressive and Interlaced Scan (1) In a raster scan, a camera captures a video sequence by sampling it in both temporal and spatial directions. The resulting signal is stored in a continuous 1-D waveform. The video signal consists of a series of frames separated by a regular frame interval. Each frame consists of a consecutive set of horizontal scan lines, separated by a regular vertical spacing. The format is known as progressive scan. In the interlaced scan, each frame is scanned in two fields, separated by the field interval. Each field thus contains half the number of lines in a frame. The field containing the first line is called the top field; and that containing the second line is the bottom field. 32

33 Analog Video Raster Progressive and Interlaced Scan (2) Field 2 Field 1 Progressive scan 2:1 Interlaced scan Fig. 1.5 Progressive and interlaced scan. 33

34 Analog Color Television Systems Fig. 1.6 Field synchronizing signal (PAL). 34

35 Analog Color Television Systems Fig. 1.7 Line sync levels and pulse times (PAL). 35

36 Analog Color Television Systems An analog color TV system must be compatible with the previous monochrome TV system. The bandwidth of a color TV signal has to fit within that allocated for a monochrome TV signal. All color signals must be multiplexed into a single composite signal in such a way that a monochrome TV receiver can extract from it the luminance signal. There are three different analog TV systems worldwide: NTSC system used in the North America, Japan and Taiwan; PAL system used in Western Europe and most of Asia; and SECAM used in the former Soviet Union, Eastern Europe, France and Middle East. 36

37 Analog Color Television Systems Fig. 1.8 A typical analog TV system. 37

38 Analog Color Television Systems Table 1.1 Parameters of analog color TV systems. 38

39 Analog Color Television Systems Fig. 1.9 Interleaving of luminance and chrominance components in NTSC TV system. [1] 39

40 Analog Color Television Systems Fig Spectrum of NTSC composite signal. [1] 40

41 Analog Color Television Systems Color Coordinates (1) To reduce the bandwidth requirement and to be compatible with monochrome TV systems, the video signal is transmitted as a luminance/chrominance (YUV or YIQ) coordinate. In PAL system, the YUV coordinate originates from the XYZ coordinate, and from the relationship between XYZ and RGB coordinates, YUV can be computed given the RGB values, i.e. Y U V = R G B where R, G, B are normalized gamma-corrected values, so that ( R, G, B) = (1,1,1) corresponds to the reference white color. 41

42 Analog Color Television Systems Color Coordinates (2) and R G = B Y U V The NTSC uses the YIQ coordinate, where the I and Q components are rotated 33 relative to the U and V components. The YIQ values are related to the NTSC RGB coordinate by: Y I Q = R G B 42

43 Analog Color Television Systems Color Coordinates (3) and R G = B Y I Q I corresponds to colors in the orange-to-cyan range and Q the green-to-purple range, and since the human eye is less sensitive to changes in the green-to-purple range, the Q component can be transmitted with less bandwidth. With the YIQ coordinate, tan 1 ( Q / I) approximates the hue, and I Q / Y corresponds to the saturation. In an NTSC composite video, the I and Q signals are multiplexed into one signal, such that the phase of the modulated signal is tan 1 ( Q / I), whereas the magnitude equals I Q / Y. 43

44 Analog Video Recording Video format Tape format Horizontal lines Composite Table 1.2 Analog videotape formats. VHS, 8 mm Umatic SP Luminance bandwidth 3.0 MHz 4.0 MHz Applications Consumer Professional S-video S-VHS, Hi MHz High quality consumer Component Betacam SP MHz Professional 44

45 Digital Video Terminology ψ(x, y, t) - analog video signal; ψ(k, m, n) - digital video signal; f s,t - frame rate t = 1/ f s,t - frame interval; f s,y - number of lines per frame y = picture-height / f s,y - pixel-height; f s,x - number of pixels per line x = picture-width / f s,x - pixel-width; N b - number of bits per pixel; R - data rate; R = f s,t f s,y f s,x N b bits per second; PAR (pixel aspect ratio) = x / y = IAR (image aspect ratio) f s,y / f s,x where IAR = picture-width/pictrue-height. 45

46 ITU-R BT.601 Digital Video Spatial Resolution (1) A digital video can be obtained either by sampling a raster scan, or directly from a digital video camera, such as the CCD camera. The International Telecommunications Union Radio Sector (ITU- R) developed the BT.601 standards for digital video formats to represent different analog TV video signals for both 4:3 and 16:9 aspect ratios. In the BT.601 standards, the sampling rate f s is chosen to satisfy two constraints: 1) the horizontal sampling resolution should match the vertical sampling resolution as closely as possible; 2) the same sampling rate should be used for the NTSC and PAL/SECAM systems. 46

47 ITU-R BT.601 Digital Video Spatial Resolution (2) The sampling rate is given by: The first criterion calls for PAR 1 f f f f = s s, x s, y s, t s, x l samples/second 2 f s, x IAR fs, y fs IAR fs, y fs, t which leads to f s 11 and 13 MHz for the NTSC and PAL/SECAM systems, respectively. To satisfy the second criterion, a number closest to both the above numbers is then chosen, which is f f s = 858 f l (NTSC) = 864 f l (PAL/SECAM) = 13.5 MHz f 47

48 ITU-R BT.601 Digital Video Spatial Resolution (3) The number of pixels per line are f s,x = 858 for NTSC and 864 for PAL/SECAM. The formats are known as 525/60 and 625/50 signals, respectively. The number of active lines are, respectively, f s, y = 480 and 576 in the 525- and 625-line systems, but the number of active pixels per line are the same, i.e., f s, x = 720 pixels, for both systems, as illustrated in Fig With the BT.601 standard, the pixel width-to-height ratio is not 1, i.e., the pixel area is not square since PAR = x / y = IAR f s, y / f s, x = 8/9 for 525/60 and 16/15 for 625/50 signals. To display a BT.601 signal, the display must have a proper PAR, otherwise the image will be distorted. 48

49 ITU-R BT.601 Digital Video 858 pixels 864 pixels 720 pixels 720 pixels 525 pixels 480 pixels Active Area 625 pixels 576 pixels Active Area 122 pixels 16 pixels 525/60: 60 fields 132 pixels 12 pixels Fig BT.601 video formats. 625/50: 50 fields 49

50 ITU-R BT.601 Digital Video Color Coordinate (1) The BT.601 standard also defines a digital color coordinate, known as YCbCr where Y, Cb, and Cr components are scaled and shifted versions of the analog Y, U, V components. Assuming the range of RGB values is (0-255), the transformation matrix is: Y Cb Cr = R G B

51 ITU-R BT.601 Digital Video Color Coordinate (2) The inverse transformation is: R G B = Y Cb Cr 128 In the relation above, R = 255 R, G = 255G, B = 255B, are the digital equivalent of the normalized RGB primaries, R, G, B. Y reflects the luminance and is scaled to have a range (16-255), whereas Cb and Cr are scaled versions of the color differences (B Y) and (R Y), respectively, and have a range of (16-240). 51

52 ITU-R BT.601 Digital Video Chrominance Subsampling Chrominance is usually subsampled at a reduced rate than the luminance, resulting in different formats as shown in Fig In 4:2:2 format, the chrominance is subsampled at half the sampling rate of luminance, i.e. f s,c = f s / 2. This leads to half the number of Cb & Cr pixels per line. To further reduce sampling rate, the chrominance is subsampled by a factor of 4 along each line, resulting in 4:1:1 format. However, this format yields asymmetrical resolutions in the horizontal and vertical directions. To solve the above problem, the chrominance components are subsampled by half along both the horizontal and vertical directions in the 4:2:0 format. 52

53 ITU-R BT.601 Digital Video Y pixel Cb & Cr pixel 4:4:4 For every 2 2 Y pixels 4 Cb & 4 Cr pixels (no subsampling) 4:2:2 For every 2 2 Y pixels 2 Cb & 2 Cr pixels (subsampling by 2:1 horizontally only) Fig BT.601 chrominance subsampling formats. 53

54 ITU-R BT.601 Digital Video Y pixel Cb & Cr pixel 4:1:1 For every 4 1 Y pixels 1 Cb & 1 Cr pixel (subsampling by 4:1 horizontally only 4:2:0 For every 2 2 Y pixels 1 Cb & 1 Cr pixel (subsampling by 2:1 both horizontally and vertically) Fig BT.601 chrominance subsampling formats. 54

55 Digital Video Formats Video format Y size Color sampling SMPTE 296M SMPTE 295M BT.601 BT.601 Table 1.3 Digital video formats for different applications / /576 4:2:0 4:2:0 4:4:4 4:2:2 Frame rate 24P/30P/60I 24P/30P/60I 60I/50I 60I/50I Raw data (mbps) 265/332/ /746/ Applications HDTV over air, cable, satellite, MPEG-2 video mbps Video production, MPEG-2, mbps BT /576 4:2:0 60I/50I 124 High-quality video distribution (DVD, SDTV), MPEG-2, 4-8 mbps SIF /288 4:2:0 30P/25P 30 Intermediate-quality video distribution (VCD, WWW), MPEG-1, 1.5 mbps CIF :2:0 30P 37 Videoconferencing over ISDN/Internet, H.261/H.263, kbps QCIF :2:0 30P 9.1 Video telephony over wired/wireless modem, H.263, kbps 55

56 Digital Video Recording Table 1.4 Digital video tape formats. Tape format Video format Source rate (mbps) Compressed rate (mbps) Compressed method Intended application Uncompressed formats SMPTE D1 SMPTE D2 SMPTE D3 SMPTE D5 BT.601 4:2:2 BT.601 compatible BT.601 compatible BT.601 4:2:2 (10 bits) N/A N/A N/A N/A N/A N/A N/A N/A Professional Professional Professional/ consumer Professional Compressed formats Digital Betacam Betacam SX DVCPRO50 DVCPRO25 (DV) BT.601 4:2:2 BT.601 4:2:2 BT.601 4:2:2 BT.601 4:1: Frame DCT MPEG-2 (I & P mode) frame/field DCT frame/field DCT Professional Consumer Professional Consumer 56

57 Video Quality Measures The most commonly used objective quality measures between two images ψ 1 and ψ 2 are the MSE and PSNR, defined as follows: MSE 1 N 2 = σ e = k m, n PSNR = 10log 10 ψ σ 2 max 2 e [ ψ ( m, n, k) ψ ( m, n, k) ] db where N is the total number of pixels and ψ max is the peak intensity value of the video signal. For most 8-bit/color video, ψ max = It is well known that MSE and PSNR do not correlate well with subjective quality in terms of perceptual distortion between images, they are widely used partly because of their mathematical tractability and partly because there is a lack of other alternatives

58 Video Quality Measures 58

59 Light and Color Gamma-ray X-ray Ultra-violet Infra-red Microwave Radio Visible spectrum Wavelength (cm) (nm) Wavelengths comprising the visible range of the electromagnetic spectrum. 59

60 Additive and Subtractive Colors Additive Color Process The additive color process produces and presents color to the naked eye by engaging light directly emitted from a source, and then mixing up three primary colors from this emitted light to form all the other colors. The additive color process engages the three primary colors Red, Green and Blue. The color model employed in the additive color process is called the RGB (Red Green Blue) color model. Mixing any one of these three primary colors in equal quantities with another primary color produces what are known as the secondary colors Cyan, Magenta and Yellow. 60

61 Additive and Subtractive Colors Subtractive color Process The subtractive color process produces and presents color to the naked eye by absorbing (amounting to subtraction) some light wavelengths and reflecting some other. The working of this color process can be understood by looking at dyes, inks, paints and natural colorants. Each color printed by these on any media can be seen by the naked eye because the media absorbs some part of the light wavelengths and reflects the others. The subtractive color process begins with white light for example, that is reflected by white paper. When the colors are placed on the paper, two layers are formed the colors and the white light reflected from the paper. When the viewer views the printed paper, the inks placed on the paper absorb and subtract certain light wavelengths from the white and present the corresponding color to the eye. 61

62 Human Visual System The human eye The retina 62

63 Opponent Color Model The opponent process is a color theory that states that the human visual system interprets information about color by processing signals from cones in an antagonistic manner. The three types of cones have some overlap in the wavelengths of light to which they respond, so it is more efficient for the visual system to record differences between the responses of cones, rather than each type of cone's individual response. The opponent color theory suggests that there are three opponent channels: red versus green, blue versus yellow, and black versus white (the latter type is achromatic and detects light-dark variation, or luminance). Responses to one color of an opponent channel are antagonistic to those to the other color. 63

64 RGB & CMY Primaries RGB primary and CMY primary are complement to each other means: white minus red gives cyan; white minus green gives magenta; white minus blue gives yellow; red + green + blue = white (additive); and cyan + magenta + yellow = black (subtractive). 64

65 RGB & CMY Primaries 65