MULTIMEDIA SIGNALS AND SYSTEMS

MULTIMEDIA SIGNALS AND SYSTEMS

THE KLUWER INTERNATIONAL SERIES IN ENGINEERING AND COMPUTER SCIENCE

MULTIMEDIA SIGNALS ANDSYSTEMS Mrinal Kr. Mandal University of Alberta, Canada SPRINGER SCIENCE+BUSINESS MEDIA, LLC

Additional material to this book can be downloaded from http://extra.springer.com. Library of Congress Cataloging-in-Publication Data Mandal, Mrinal Kr. Multimedia Signals and Systems / Mrinal Kr. Mandal. p.cm.-(the Kluwer international series in engineering and computer science; SECS 716) lncludes bibliographical references and index. ISBN 978-1-4613-4994-5 ISBN 978-1-4615-0265-4 (ebook) DOI 10.1007/978-1-4615-0265-4 1. Multimedia systems. 2. Signal processing-digitial techniques. I. Title. II. Series. QA76.575.M3155 2002 006.7--dc21 2002034047 Copyright 2003 by Springer Science+Business Media New York Originally published by Kluwer Academic Publishers in 2003 Softcover reprint of the hardcover lst edition 2003 AH rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photocopying, record ing, or otherwise, without the prior written permission of the publisher. MATLAB is a registered trademark ofthe MathWorks, Inc. Printed an acid-free paper.

Table of Contents 1. INTRODUCTION 1.1. Development of Multimedia Systems 1.2. Classification of Media 1.3. Properties of Multimedia Systems 1.4. Multimedia Computing 1.5. Different Aspects of Multimedia References Questions 1 2 3 5 6 9 9 Part I: MULTIMEDIA SIGNALS 2. AUDIO FUNDAMENTALS 11 2.1 Characteristics of Sound 11 2.2 The Human Auditory System 14 2.3 Audio Recording 18 2.4 Audio Signal Representation 23 2.4.1 Wavefonn method 23 2.4.2 Musical Instrument Digital Interface 24 References 30 Questions 31 3. THE HUMAN VISUAL SYSTEM AND PERCEPTION 33 3.1 Introduction 33 3.2 The Human Visual System 34 3.2.1 Relative Luminous Efficiency 36 3.2.2 Weber's Law 37 3.2.3 Modulation Transfer Function 38 3.2.4 HVS Model 42 3.3 Color Representation 42 3.3.1 Three-Receptor Model 42 3.3.2 Color Matching 44 3.3.3 Tristimulus Value 45 3.3.4 Chromaticity Diagram 47 3.3.5 Color Models and Transformation of Primaries 48

VI Multimedia Signals and Systems 3.3.5.1 NTSC Receiver Primary 49 3.3.5.2 NTSC Transmission System 50 3.3.5.3 1960 CIE-UCS Color coordinates 53 3.3.5.4 CMYModel 54 3.4 Temporal Properties of Vision 54 References 55 Questions 56 4. MULTIMEDIA DATA ACQUISITION 57 4.1 Sampling of Audio Signals 57 4.2 Sampling of Two-Dimensional Images 63 4.3 Anti-Aliasing Filters 67 4.4 Digitization of Audio Signals 70 4.4.1 Analog to Digital Conversion 71 4.4.2 Audio Fidelity Criteria 75 4.4.3 MIDI versus Digital Audio 78 4.5 Digitization of Images 79 4.5.1 Visual Fidelity Measures 79 References 81 Questions 81 Part ll: SIGNAL PROCESSING AND COMPRESSION 5. TRANSFORMS AND SUBBAND DECOMPOSITION 83 5.1 I-D Unitary Transfonn 84 5.2 I-D Discrete Fourier Transfonn 85 5.3 I-D Discrete Cosine Transfonn 90 5.4 Digital Filtering and Subband Analysis 93 5.4.1 Digital Filters 93 5.4.2 Subband Analysis 97 5.4.3 Transfonns and Digital Filtering 103 5.5 I-D Discrete Wavelet Transfonn 104 5.6 2-D Unitary Transfonn 109 5.7 2-D Discrete Fourier Transfonn 111 5.8 2-D Discrete Cosine Transfonn 114 5.9 2-D Discrete Wavelet Transfonn 116 References 118 Questions 119

Table of Contents vii 6. TEXT REPRESENTATION AND COMPRESSION 121 6.1 Text Representation 121 6.2 Principles of Text Compression 124 6.3 Statistical Redundancy 124 6.3.1 Probability Density Function and Entropy 125 6.3.2 Shannon's Noiseless Source Coding Theorem 127 6.3.3 Huffman Coding 129 6.3.4 Arithmetic Coding 133 6.4 Dictionary-based Compression 137 6.4.1 LZ77 Technique 138 6.4.2 LZ78 Technique 140 6.5 Summary 143 References 143 Questions 144 7. DIGITAL AUDIO COMPRESSION 145 7.1 Audio Compression Principles 145 7.1.1 Rate Distortion Function 147 7.2 Statistical Redundancy 148 7.2.1 Companding and Expanding 149 7.3 Temporal Redundancy 151 7.4 Perceptual Audio Coding 156 7.5 Audio Compression Standards 158 7.6 MPEG-l Audio Compression Standard 159 7.7 MPEG-2 Audio Compression Standard 162 7.8 AC Audio Compression Standards 163 7.9 Comparison of Compression Algorithms 165 7.10 Audio Formats 166 References 166 Questions 167 8. DIGITAL IMAGE COMPRESSION TECHNIQUES 169 8.1 Principles oflmage Compression 169 8.2 Low Complexity Compression Techniques 170 8.2.1 Entropy Coding 170 8.2.2 Run-length Coding 171 8.2.3 Predictive Coding 173 8.3 Transform Coding 175

VIII Multimedia Signals and Systems 8.3 Transfonn Coding 175 8.3.1 Unitary Transfonn 176 8.3.2 Block Transfonn 177 8.3.3 Wavelet Coding 179 8.3.4 Comparison ofdct and Wavelets 180 8.4 Other Coding Techniques 182 8.4.1 Vector Quantization 183 8.4.2 Fractal Image Compression 184 8.5 Image Compression Standards 185 8.6 The JPEG Image Compression Standard 186 8.6.1 Baseline Sequential Mode 186 8.6.2 Other JPEG Modes 192 8.7 The JPEG 2000 Standard 193 8.8 Image Fonnats 199 References 200 Questions 201 9. DIGITAL VIDEO COMPRESSION TECHNIQUES 203 9.1 Principles of Video Compression 203 9.2 Digital Video and Color Redundancy 204 9.3 Temporal Redundancy Reduction 207 9.4 Block-based Motion Estimation 209 9.4.1 Fast Motion Estimation Algorithms 214 9.5 Video Compression Standards 221 9.5.1 Motion JPEG 222 9.5.2 The MPEG-I Video Compression Standard 222 9.5.3 The MPEG-2 Video Compression Standard 224 9.5.4 The MPEG-4 Video Compression Standard 226 9.5.4.1 Video Coding Scheme 228 9.5.5 The H.261 Video Compression Standard 231 9.5.6 H.263, H.263+ and H.26L Standards 231 9.5.7 Comparison of Standard Codecs 232 References 235 Questions 236 10. DIGITAL AUDIO PROCESSING 239 10.1 Audio Filtering Techniques 10.2 Audio Equalization 10.3 Audio Enhancement 10.3.1 Noise Suppression by Digital Filtering 239 241 245 246

Table of Contents IX 10.3.2 Spectral Subtraction Method 248 10.4 Editing MIDI Files 252 10.5 Digital Audio and MIDI Editing Tools 254 References 255 Questions 256 11. DIGITAL IMAGE AND VIDEO PROCESSING 257 11.1 Basic Image Processing Tools 11.1.1 Image Resizing 11.1.2 Cropping 11.2 Image Enhancement Techniques 11.2.1 Brightness and Contrast Improvement 11.2.1.1 Contrast Stretching 11.2.1.2 Histogram Equalization 11.2.2 Image Sharpening 11.3 Digital Video 11.3.1 Special Effects and Gradual Transition 11.3.1.1 Wipe 11.3.1.2 Dissolve 11.3.1.3 FadeIn/Out 11.3.2 Video Segmentation 11.3.2.1 Camera Operations 11.4 Image and Video Editing Softwares 11.5 Summary References Questions 257 257 260 261 261 262 265 266 267 269 269 272 272 273 279 280 280 281 282 Part m: MULTIMEDIA SYSTEMS 12. ANALOG AND DIGITAL TELEVISION 12.1 Analog Television Standards 12.2 Raster Scanning 12.3 Color Space for TV Transmission 12.3.1 NTSC System 12.3.2 PAL System 12.4 NTSC Television System 12.4.1 Channel Assignment 283 283 285 286 288 289 291 291

x Multimedia Signals and Systems 12.4.2 NTSC Encoder and Decoder 293 12.5 Component and S-Video 295 12.6 Digital Television 296 12.6.1 Grand Alliance HDTV Standard 299 References 303 Questions 304 13. CONTENT CREATION AND MANAGEMENT 305 13.1 Multimedia Authoring 305 13.1.1 Authoring Steps 306 13.2 Multimedia Authoring Tools 308 13.2.1 CardlPage-Based Tools 309 13.2.2 Icon-Based Tools 310 13.2.3 Time-Based Tools 311 13.2.4 Object Oriented Tools 312 13.3 Multimedia Documents 313 13.4 Hepertext and Hypermedia 315 13.4.1 Nonlinear Information Chain 315 13.4.2 Hypertext and Hypermedia Systems 316 13.4.3 Mark-up Languages 317 13.4.4 HTML 318 13.4.5 XML 321 13.5 Web Authoring Tools 323 13.6 Multimedia Standards 325 13.6.1 The MPEG-7 Standard 326 13.6.2 The MPEG-21 Standard 328 13.6.3 The MHEG Standard 329 13.7 Summary 329 References 330 Questions 330 14. OPTICAL STORAGE MEDIA 333 14.1 Physical Medium 334 14.1.1 Cross-section ofa CD 334 14.1.2 Digital Versatile Disc (DVD) 335 14.1.3 Physical Formats and Speeds 336 14.1.4 Playback of CD and DVD 337 14.1.5 CD-ROM, CD-R, and CD-RW 340 14.1.6 Advantages of Optical Technology 342 14.2 CD and DVD Standards 342

Table of Contents xi 14.2.4 Video CD and DVD-Video Standards References Questions 15. ELECTRONIC DISPLAYS 15.1 Important Display Parameters 15.2 Cathode-Ray Tube 15.3 Field Emission Display 15.4 Plasma Display 15.5 Liquid Crystal Display 15.6 Digital Micromirror Display References Questions APPENDIX: About the CD-ROM INDEX 348 350 350 351 351 353 355 357 360 363 365 366 367 371

PREFACE Multimedia computing and communications have emerged as a major research and development area. Multimedia computers in particular open a wide range of possibilities by combining different types of digital media such as text, graphics, audio and video. The emergence of the World Wide Web, unthinkable even two decades ago, also has fuelled the growth of multimedia computing. There are several books on multimedia systems that can be divided into two major categories. In the first category, the books are purely technical, providing detailed theories of multimedia engineering, with an emphasis on signal processing. These books are more suitable for graduate students and researchers in the multimedia area. In the second category, there are several books on multimedia, which are primarily about content creation and management. Because the number of multimedia users is increasing daily, there is a strong need for books somewhere between these two extremes. People with engineering or even non-engineering background are now familiar with buzzwords such as JPEG, GIF, W A V, MP3, and MPEG files. These files can be edited or manipulated with a wide variety of software tools. However, the curious-minded may wonder how these files work that ultimately provide us with impressive images or audio. This book intends to fill this gap by explaining the multimedia signal processing at a less technical level. However, in order to understand the digital signal processing techniques, readers must still be familiar with discrete time signals and systems, especially sampling theory, analog-todigital conversion, digital filter theory, and Fourier transform. The book has 15 Chapters, with Chapter 1 being the introductory chapter. The remaining 14 chapters can be divided into three parts. The first part consists of Chapters 2-4. These chapters focus on the multimedia signals, namely audio and image, their acquisition techniques, and properties of human auditory and visual systems. The second part consists of Chapters 5-11. These chapters focus on the signal processing aspects, and are strongly linked in order to introduce the signal processing techniques step-by-step. The third part consists of Chapters 12-15, which presents a few select multimedia systems. These chapters can be read independently. The objective of including this section is to introduce readers to the intricacies of a few select frequently used multimedia systems.

XIV Preface including this section is to introduce readers to the intricacies of a few select frequently used multimedia systems. The chapters in the first and second parts of the book have been organized to enable a hierarchical study. In addition to the Introductory Chapter, the following reading sequence may be considered. i) Text Representation: Chapter 6 ii) Audio Compression: Chapters 2, 4,5,6, 7 iii) Audio Processing: Chapters 2, 4, 5, 10 iv) Image Compression: Chapters 3, 4, 5, 6, 7, 8 v) Video Compression: Chapters 3, 4, 5, 6, 7, 8, 9 vi) Image & Video Processing: Chapters 3, 4, 5, 11 vii) Television Fundamentals: Chapters 3, 4, 5, 6, 7, 8, 9, 12 Chapters 13-15 can be read in any order. A major focus of this book is to illustrate with examples the basic signal processing concepts. We have used MATLAB to illustrate the examples since MATLAB codes are very compact and easy to follow. The MATLAB codes of most examples, wherever appropriate, in the book are provided in the accompanying CD so that readers can experiment on their own. Any suggestion and concern regarding the book can be emailed to the author at the email address:mandai@ee.ualberta.ca. There would be a follow-up website (http://www.ee.ualberta.ca/-mandallbook-multimedia!) where future updates will be posted. I would like to extend my deepest gratitude to all my coworkers and students who have helped in the preparation of this book. Special thanks are due to Sunil Bandaru, Alesya Bajoria, Mahesh Nagarajan, Shahid Khan, Hongyu Liao, Qinghong Guo, and Sasan Haghani for their help in the overall preparation. I would also like to thank Drs. Philip Mingay, Bruce Cockburn, Behrouz Nowrouzian, and Sethuraman Panchanathan (from Arizona State University) for their helpful suggestions to improve the course content. Jennifer Evans and Anne Murray from Kluwer Academic Publishers have always lent a helping hand. Last but not least, I would like to thank Rupa and Geeta, without whose encouragement and support this book would not be completed. August 2002 Mrinal Kr. Mandai