Digital Media Daniel Fuller ITEC 2110
Daily Question: Video How does interlaced scan display video? Email answer to DFullerDailyQuestion@gmail.com Subject Line: ITEC2110-26
Housekeeping Project 4 is assigned
Video it s big
Video it s big: How do you deal with it? Playback degradation (graceful degradation) Compression
Video it s big: Handling Playback Transport or playback not fast enough to keep up with the story? Something s got to give There s too much data to either transport or display Some players just freeze the image and halt the audio This kills the ability to tell the story Some players (like quicktime) make attempts to degrade gracefully
Graceful Degradation Graceful degrading allows the story to continue Some players drop frames First showing as a slide show while continuing to play the audio Then holding the last image while continuing to play the audio stream This effectively loses the illusion of motion but continues the story as an audio stream Some players play lower resolution images while remaining synched to the audio stream This continues the illusion of motion (at a lower resolution) and continues the story with the audio stream
Compression On the capture side Digitization & compression can be carried out by hardware to be fast Can be done in the camera (hardware) Can be done in the computer (software)
Hardware vs Software compression Hardware compression User has no control over it It can be in the camera It can be in the video card Software conversion Computationally expensive A slow process Provides for the most flexibility since it can be changed Can use different software coder-decoders (codec), picking and choosing what fits your needs better
Hardware Compression Our cameras? Mini DV format Compress each captured image into a jpeg image This is called intra-frame compression Present a digital stream of bits to the computer over a firewire connection With compression you get artifacts
More about Analog vs Digital An analog signal to the computer is susceptible to noise corruption Digital signal is not What s the big deal? Consider compressing a video of a wall painted a solid color Analog noise will cause small fluctuations from pixel to pixel RLE can t compress it because each pixel is a bit different
Comparing cameras isight to MiniDV isight (or a webcam) is built into the Macs in this room Presents an analog signal to the computer Subject to analog noise The cameras we can check out from the library are Mini DV format and record on tape Presents a digital signal to the computer
Our video cameras compress using jpeg analog signal!!!noise!!! webcam minidv digital signal computer video capture card compression imovie the scene compression
Converting Film to Video
Film to Video Problematic (interleaved) Film is 24 frames per second Video is 30 frames per second How do you make 30 frames from 24? One way: The 3-2 pull down Ratio of 24 to 30 is 4:5 So stretch every 4 frames into 5
Film to Interlaced Video: http://en.wikipedia.org/wiki/telecine
Digital vs Analog TV
TV Broadcast: Digital Replaces Analog Why Digital Broadcast? Reduced spectrum use Greater capacity Multiple programs on one frequency Better quality picture HDTV Can use compression Allows multiple HD signals on one frequency Allows user interaction
TV Broadcast Standards Analog (old school) NTSC (north America) PAL (western europe) SECAM (eastern europe) Digital ATSC (see map on wikipedia)
TV Broadcast Difference with poor reception Analog As signal gets weaker: Image gets less distinct Ghosts (white shadows) appear Gracefully degrades Digital With digital, you either have signal or you don t have signal Lose signal: Everything goes black Audio stops Ungraceful degrading
Digital Video Standards
Digital Video Standards Standard definition High definition Digital Television
The Term DV Common simple abbreviation for digital video DV compression and DV format: specific types of digital video compression and format respectively In the textbook DV refers to the specific types of digital video DV is NOT used to abbreviate digital video
Standard Definition DV25 Format Pixel Dimensions 720 480 (NTSC) Frame Aspect Ration 4:3 16:9 Pixel Aspect Ratio 0.9 1.2 Data Rate Total (video + audio + control information): Audio Setting Video data only: 3.6 megabytes per second (MB/s), i.e. about 4.6 minutes of video per gigabyte of storage space 25 megabits per second (Mbps); compressed at a fixed rate of 5 :1 Color Sampling Method YUV 4:1:1 Sampling rate and bit depth: Two options: 48 khz, 16-bit 32 khz, 12-bit
Standard Definition DV25 Format Pixel Dimensions 720 480 (NTSC) Frame Aspect Ration 4:3 16:9 Pixel Aspect Ratio 0.9 1.2 Data Rate Total (video + audio + control information): Audio Setting Video data only: 3.6 megabytes per second (MB/s), i.e. about 4.6 minutes of video per gigabyte of storage space 25 megabits per second (Mbps); compressed at a fixed rate of 5:1 Color Sampling Method YUV 4:1:1 Sampling rate and bit depth: Two options: 48 khz, 16-bit 32 khz, 12-bit
YUV
YUV A luminance-chrominance color model A color model that digital video uses to represent the color of each pixel Y: luminance (brightness) component U: a chrominance (color or hue) component V : a chrominance (color or hue) component
Advantages Using YUV over RGB The human eye is more sensitive to changes of the luminance than it is to chrominance changes. Thus, with YUV, we can assign fewer bits to store the chrominance components. Fewer bits means smaller file size.
Loss of Color Information Some chrominance data are discarded in order to reduce bits. Chroma subsampling or color subsampling
The Three Numbers in YUV Ratio of numbers of samples of Y : U : V for each group of 4 pixels 4:4:4 4:2:2 4:2:0 4:1:1
Chrominance Sub-sampling http://dougkerr.net/pumpkin/articles/subsampling.pdf
Chrominance Sub-sampling http://en.wikipedia.org/wiki/chroma_subsampling#sampling_systems_and_ratios
YUV 4:4:4 YUV No. of samples of Y for each group of 4 pixels No. of samples of U for each group of 4 pixels No. of samples of V for each group of 4 pixels Total no. of YUV samples of for each group of 4 pixels Saving in storage 4:4:4 4 4 4 4+4+4=12 None (no compression, i.e., no subsampling)
YUV 4:2:2 YUV No. of samples of Y for each group of 4 pixels No. of samples of U for each group of 4 pixels No. of samples of V for each group of 4 pixels Total no. of YUV samples of for each group of 4 pixels Saving in storage 4:2:2 4 2 2 4+2+2=8 Reduced 4 samples out of 12 samples, i.e. 33% compression Used in Digital Betacam
YUV 4:2:0 YUV No. of samples of Y for each group of 4 pixels No. of samples of U for each group of 4 pixels No. of samples of V for each group of 4 pixels Total no. of YUV samples of for each group of 4 pixels Saving in storage 4:2:0 4 2 0 4+2+0=6 Reduced 6 0 2 samples out of 12 samples, i.e. 50% compression Used in HDV, MPEG-1, DVD MPEG-2, PAL DV
YUV 4:1:1 YUV No. of samples of Y for each group of 4 pixels No. of samples of U for each group of 4 pixels No. of samples of V for each group of 4 pixels Total no. of YUV samples of for each group of 4 pixels Saving in storage 4:2:0 4 1 1 4+1+1=6 Reduced 6 samples out of 12 samples, i.e. 50% compression Used in NTSC DV
HDV
High Definition HDV Format Picture Format 720/25p, 720/30p, 720/50p, 720/60p 1080/50i 1080/60i Pixel Dimensions 1280 720 1440 1080 Frame Aspect Ration 16:9 Pixel Aspect Ratio 1.0 1.33 Data Rate Video data only: approx. 19 Mbps approx. 25 Mbps Audio Setting Color Sampling Method YUV 4:2:0 Sampling rate and bit depth: Bit rate after compression: 48 khz 16-bit 384 kbps
HDV Picture Format Notation 1080 / 60 i "i": interlaced "p": progressive frame/field frequency frame height
Resolution Comparison
Frame Size (Resolution) Comparison between Standard Definition and High Definition By viewing frame size
Frame Size (Resolution) Comparison between Standard Definition and High Definition By pixel dimensions
Frame Size (Resolution) Comparison between Standard Definition and High Definition A frame from a 1080i video
Frame Size (Resolution) Comparison between Standard Definition and High Definition Same frame as 720p
Frame Size (Resolution) Comparison between Standard Definition and High Definition Same frame as standard definition DV wide-screen (16:9)
Frame Size (Resolution) Comparison between Standard Definition and High Definition Same frame as standard definition DV standard 4:3
Digital Television
Digital Television (DTV) Signals of DTV are broadcast and transmitted digitally Need a digital TV set to watch
Digital Television (DTV) Standard definition 704 480, 16:9 and 4:3, progressive and interlaced 640 480, 4:3, progressive and interlaced High definition 1920 1080, 16:9, progressive and interlaced 1280 720, 16:9, progressive and interlaced MPEG-2
Digital Video File Types
Common Video File Types File Type Acronym For Originally Created By File Info & Compression.mov QuickTime movie Apple Also audio-only Can be streamed "Fast start" Common compression methods: H.264, Sorenson Video, Animation.avi.mpg.mpeg Audio Video Interleave MPEG Intel Motion Picture Experts Group Common compression methods: Microsoft RLE, Intel Indeo Video For DVD-video High definition HDV Platforms Apple QuickTime player, which is available for Mac and Windows Primarily used on Windows but Apple QuickTime player can play AVI files Cross-platform
Common Video File Types File Type Acronym For Originally Created By File Info & Compression.flv Flash Video Adobe Progressive download Can be streamed Common compression methods: H.264, Sorenson Spark, On2 VP6.rm Real Video Real System Can have very high degree of compression Choose the compression level based on network connection speed Can be streamed Platforms Cross-platform Requires Adobe Media Player to play Cross-platform Requires Real player to play.wmv Windows Media Microsoft Requires Windows Media Player to play
Considerations for File Type File size restriction Intended audience Future editing
File Size Restriction For Web: High compression Streaming video CD-ROM or DVD-ROM playback: Use data rate that can be handled by your target audience's computer DVD-video: MPEG-2
Intended audience Multiple platforms Cross-platform formats: Apple QuickTime, MPEG, Flash video, Real Video How your target audience is going to watch your video?
Future Editing If the video will be used as a source for future editing: Lower compression level Choose uncompressed if The frame size is small The video duration is extremely short You have enough disk space
Digital Video File Size Optimization Video tends to have very large file size compared to other media. Why should we care file size optimization? A large file requires more disk space. A large file takes longer to transfer. Data transfer can be expensive (because data plans are not unlimited) High data rate may cause choppy playback of the video.
Digital Video File Size
Digital Video File Size To get a feel of the file size of uncompressed video, let's compute the file size of an uncompressed 1-second video with the same resolution and frame rate as HDV 1080i/p video.
The 1-second Video 1440 1080 pixels 24-bit color 30 fps 1 second long Audio: stereo (2 channels) Audio: 48 khz, 16-bit
File Size Calculation The picture component: Total pixels in each frame: 1440 1080 pixels = 1,555,200 pixels/frames File size of a frame: 1,555,200 pixels/frames 24 bits/pixel = 37,324,800 bits/frame File size of 30 frames (1 second): 37,324,800 bits/frame 30 frames = 1,119,744,000 bits
File Size Calculation 1,119,744,000 bits = 1,119,744,000 bits / (8 bits/bytes) = 139,968,000 bytes 133 MB (for just the picture component)
File Size Calculation Audio: Sampling rate length of the audio bit-depth number of channels = 48,000 samples/sec 1 second 16 bits/sample 2 = 1,536,000 bits = 1,536,000 bits / (8 bits/byte) = 192,000 bytes 188 KB
File Size Calculation Total file size of this 1-second uncompressed video = size of the picture component + audio size 133 MB + 188 KB = 133 MB 10 seconds would be 1.33 GB!
File Size Calculation Total file size of this 1-second uncompressed video = video size + audio size 133 MB + 188 KB = 133 MB Note that the audio size is insignificant compared to the picture component of the video. 10 seconds would be 1.33 GB!
Data Rate
Data Rate Amount of video data to be processed per second Data Rate = File Size Duration of Video (seconds)
Data Rate Example Data rate of a 48x-speed CD-ROM drive: 7 MB/s Previous video file size example: 1-second uncompression video: 133 MB Data rate = 133 MB / 1 second = 133 MB/second Way too high for the 48x-speed CD-ROM drive! Very choppy playback on 48x-speed CD- ROM drive
Effect of File Size vs. Data Rate on Video Playback Data rate: If high: choppy playback Amount of data to be processed per second Larger file size can have a low data rate if long video Smaller file size can have a high data rate if short video File size: If high: Requires larger storage space Not unnecessary choppy playback The impact of file size on smoothness of playback also depends on the video duration.
Finding out Data Rate in QuickTime Window > Show Movie Info
Finding out Data Rate in QuickTime Compressed with 100% picture quality.
Finding out Data Rate in QuickTime Compressed with 70% picture quality. Lower data rate, but no discernible degradation of the image in this example.
General Strategies for Reducing Video Data Rate Basic ideas: A video is a sequence of images + audio Apply strategies for reducing digital image and audio file size.
Reducing Digital Video File Size
General Strategies for Reducing Video File Size General Strategies for reducing digital image file size Reduce frame size Reduce frame rate Choose a video compressor that allows higher compression Choose the lower picture quality option Reduce duration of the video so you have less frames Not always possible Will not impact data rate
Why General Strategies for Reducing Digital Image File Size Work for Video Reduce frame size because: There will be less pixels for each frame Reduce frame rate because: There will be less frames Video compression with high compression because: Some data will be discarded Lower picture quality option because: Some data will be discarded
Most Common Choices of Compressor H.264 Sorenson Video 3
Picture Quality Option Example
Strategies Least Used for Reducing Video Data Rate Reduce bit depth Not all video formats support lower bit depth Live videos need 24-bit color to look natural Some compressors do not support lower bit depth Reduce sampling rate, bit depth, and channel numbers of the audio Size of the audio is insignificant compared to that of the picture component in a video
Digital Video Compression
Compression Basic idea: Represent the same content with less data
Compression and Decompression An Analogy Compression: Packing a suitcase Packing your clothes neatly: More compact Takes more time Decompression: unpacking a suitcase Unpack or even iron your clothes before you wear them How you unpack often depends on how the clothes are packed
Compression and Decompression Compression: To reduce file size Takes time Often takes more time for higher compression Decompression: A compression video file must be decompressed before it is played. The decompression method or algorithm depends on how it is originally compressed.
Compression and Decompression Compression and decompression always go together as a pair. Codec: compressor/decompressor
Digital Video Compression Methods
Types of Compression Methods Spatial compression Temporal compression Lossless vs. lossy compression Symmetrical and asymmetrical compression
Spatial Compression Compact individual frames as if they are independent digital images Examples of algorithms: Run-length encoding (RLE) JPEG compression Example codecs: QuickTime Animation QuickTime PlanarRGB Microsoft RLE
Spatial Compression Types of video that spatial compression is good for: Video that contains large areas of solid colors, such as cartoon animation Disadvantage: Less compressed, i.e. relatively large file size compared to other types of compression
Temporal Compression Exploits the repetitious nature of image content over time in video Saving more information for selected frames, i.e. less compressed. These are called key frames. All other frames stores only the difference from the previous key frame, instead of full frame Advantage: Effective if the change between a frame and its previous key frame is small
Temporal Compression Compressed well for: Video that contains continuous motion e.g. Security cameras spend most of their time seeing the same thing all night long Not compressed well for: Video with frequent flickering and scene changes Example codecs that use temporal compression: H.264 Sorenson Video
Temporal Compression: Differencing Captured Frame 1 Captured Frame 2 Saved Frame 1 Saved Frame 2
Temporal Compression: Vectoring Captured Frame 1 Captured Frame 2 Stored Background Stored Object Movement Vector
More on Differencing The differencing can happen in a forward manner and a backward manner It might be more economical (in data size) to create a frame from a frame that follows it instead of a frame that precedes it More on this when we get to MPEG-2
Lossy Compression Usually much smaller file size than lossless compression Lower picture quality Often try to maintain perceptual quality when deciding what data are to be discarded Discarded data cannot be recovered
Lossless Compression Usually much larger file size than lossy compression Example codecs: QuickTime Animation PlanarRGB (set at the maximum quality setting)
Symmetrical and Asymmetrical Compression Symmetrical codec: Same amount of time in compression and decompression Asymmetrical codec: Amount of time to compress and decompress are significantly different Preferable: Fast decompression so less wait time to play back the video
Ways of Playing Video
Two Ways of Playing Video Play from disk Play over a network
Play from Disk An entire clip needs to be on disk before it can be played Played from hard drive, CD, or DVD
Play over a Network The video can be played while it is being downloaded Can be played from disk Streaming video Progressive download
Streaming Video Play video as soon as enough data has arrived Examples: Streaming QuickTime Real Video Window Media Video (WMV)
Streaming Video Require a streaming server to stream video Allow saving several different compression levels of a video in a single file The server chooses the compression level to match the speed of network connection Buffering: Wait time depends on network speed
Progressive Download Play video as soon as enough data has arrived Does not require special servers Example: QuickTime fast-start Created by saving the QuickTime movie as selfcontained using QuickTime Pro
Questions?
Homework Finish reading Ch. 6 Work on Project 4