CS A490 Digital Media and Interactive Systems Lecture 8 Review of Digital Video Encoding/Decoding and Transport October 7, 2013 Sam Siewert
MT Review Scheduling Taxonomy and Architecture Traditional CPU SMP vs. NUMA Uni-processor (Peaked in 2000, Going Away ) Multi-core, Many-core Fixed Prio vs. Dynamic Prio. Scheduling Offload and Vector Processing Flynn s Taxonomy Single/Multiple Instruction x Single/Multiple Data SIMD (Vector Processing) SPMD (GP-GPU), Single Program, Multiple Data Sam Siewert 2
MT Review RT Theory, Pthreads Fundamentals of Real-Time Theory and Analysis RT Correctness Before Deadline and Correct Result Best Effort, Hard RT, Isochronal RT, and Soft RT Utility Curves for each of the above CPU, I/O, and memory resource space (CPU-bound, I/O-bound, memory/cpu trade-offs) Basic Timing Diagrams RM Least Upper Bound Lehoczky, Shah, Ding If deadline(s) are met over longest period (or better yet, LCM) from C.I., then system is feasible Fixed priority, preemptive, run-to-completion scheduling Services and interference (interrupts and higher priority services) Necessary and Sufficient feasibility means? Soft Real-Time Compared to Hard Real-Time Linux NPTL (Native POSIX Threads Library) Scheduling Classes (FIFO, RR, OTHER), 1-to-1 Pthread/Kernel Mapping pthread_create pthread_join pthread_detach Sam Siewert 3
MT Review SRT Dynamic Scheduling HRT - Rate Monotonic Theory RM priority assignment policy (optimality is?) Liu and Layland Assumptions C = WCET T=D Critical Instant Others? RM LUB Feasibility test is? Why is it Most Often NOT used for Digital Media? SRT - Dynamic Priority Theories EDF Least Laxity Timing Diagrams Harmonic vs. Non-harmonic Service Requests Overlapping Requests for Same Service (T D) Common in On-Demand Sam Siewert 4
MT Review DM Encoding Digital Sampling for Encoding Color Models (RGB Cube vs. HSV) Luminance, Chrominance RGB YCrCb Digital Audio PCM (Pulse Code Modulation), A-to-D Resolution and Frequency Audio Playback: DACs, Speakers, Channels Digital Video Encoding Pixel (POINT) - Encoding and Color Sub-sampling Frame (XY Pixel Map) - Resolution, AR, Frame Rate Sequence of Frames (Group of Pictures) Digital Media Systems Encoding, Transport, Decoding Digital Cable (Quadrature Amplitude Modulation), HFC Networks IPTV (Packet Switched MPEG TS encapsulated in UDP or RTP) ATSC OTA 8 level Vestigal Sideband Modulation, 6MhZ channel, 19.39 Mbps, Reed-Solomon Error Correction MPEG Basics and Standards Order of Operators for Encoding (Compression) Program Streams Transport Streams MPTS (PID, PSI data, Video Elementary, Audio Elementary) Sam Siewert 5
MPEG-2: Order Of Operators #1 #2C #2B #2A #3 #1: POINT (Pixel) Encoding #2 A-C: Macro-Block Lossy Intra-Frame Compression #3: Motion-Based Compression in Group of Pictures Sam Siewert 6
DCT Fundamentals Small Losses Due to DCT, idct Numerical Truncation Large Losses Due to H.O.T. Truncation Sam Siewert 7
Step #2A: Macro-block Discrete Cosine Transform 8x8 Pixel Block Macro-block SD NTSC 720x480 (90x60 Macro-blocks), 3:2 Aspect Ratio HD 720 1280x720 (160x90 Macro-blocks), 16:9 AR HD 1080 1920x1080 (240x135 Macro-blocks), 16:9 AR Sam Siewert 8
Step #2B: Macro-block Quantization (Lossy) Apply Weighting and Scaling 8x8 to DCT Produces Lots of Repeated Values (and Zeros) Compared to Original Sam Siewert 9
Decode Process for #2A-B Sam Siewert 10
Details on Q8 Weighting and Scaling Q=8 Quantization is DCT / (Weighting[i,j] x 8) 1) Example Macroblock 101 100 94 102 97 91 88 83 101 99 98 103 93 93 107 110 98 97 97 97 103 101 94 100 97 98 99 100 103 105 101 96 99 100 104 104 100 107 109 89 99 101 105 105 116 113 87 58 94 69 66 66 79 70 40 26 59 30 33 33 32 37 45 41 DCT MPEG DCT Weighting 8 16 19 22 26 27 29 34 16 16 22 24 27 29 34 37 19 22 26 27 29 34 34 38 22 22 26 27 29 34 37 40 22 26 27 29 32 35 40 48 26 27 29 32 35 40 48 58 26 27 29 34 38 46 56 69 27 29 35 38 46 56 69 83 W 2) Example DCT (use cvdct to Compute) 5561 207-133 205 7 61 17 14 966-162 70-208 6-34 -65-43 -857 89 139 21 118 35-2 -5 459 192-238 75-86 -4-18 2-219 -133 124-53 -5 62-42 -22-62 198-129 105 31-82 49 13 34-38 -72 25-22 -33-3 -5-83 54 22 13 16-3 18 5 Q8 = DCT[Ii,j] / (Weight[i,j] x 8) 3) q=8 from DCT / (Weighting x q=8) trunca 86 1 0 1 0 0 0 0 7-1 0-1 0 0 0 0-5 0 0 0 0 0 0 0 2 1-1 0 0 0 0 0-1 0 0 0 0 0 0 0 Sam Siewert 11
Details on Q8 Weighting and Scaling Inverse Q=8 Inverse Quantization is Q8 * (Weighting[i,j] x 8) MPEG DCT Weighting 8 16 19 22 26 27 29 34 16 16 22 24 27 29 34 37 19 22 26 27 29 34 34 38 22 22 26 27 29 34 37 40 22 26 27 29 32 35 40 48 26 27 29 32 35 40 48 58 26 27 29 34 38 46 56 69 27 29 35 38 46 56 69 83 W 5) Recovered Macroblock (cvdct INVERSE) 90 94 99 100 96 91 88 86 96 96 97 96 95 96 98 100 100 96 92 91 93 99 105 108 102 98 93 93 96 100 101 100 106 102 101 101 104 101 93 85 102 96 95 95 99 95 82 69 80 71 65 65 72 72 62 51 57 43 31 31 41 41 44 36 idct 3) q=8 from DCT / (Weighting x q=8) trunca 86 1 0 1 0 0 0 0 7-1 0-1 0 0 0 0-5 0 0 0 0 0 0 0 2 1-1 0 0 0 0 0-1 0 0 0 0 0 0 0 DCT recovered = Q8 x (Weight[i,j] x 8) 4) q=8 INVERSE is q8 * Weighting * 8 5504 128 0 176 0 0 0 0 896-128 0-192 0 0 0 0-76 352 176-208 0 0 0 0 0-176 0 0 0 0 0 0 0 Sam Siewert 12
MT Review DV Encoding MPEG Encoding Order of Operations Pixel/Color Encoding in YUV (e.g. YCrCb 4:2:2) Macro block encoding (DCT, Quantization, Zigzag, Huffman) Group of Pictures (I-Frame, B-Frame, P-Frame) Deeper Dive in 13818-1 and 13818-2 13818-1: Transport Streams for Video & Audio Container for Program Streams (188 Byte Packets) Multiplexed Video and Audio Elementary Streams PSI Program Specific Information System Clock (PCT, PTS/DTS) 13818-2: Elementary Video Stream Encode/Decode Video DCT Macro-blocks Color Format GoP (I-Frame, B-Frame, P-Frame) Motion Compensation and Vector Quantization Sam Siewert 13
MPEG2 vs MPEG4 Differences Between MPEG-2 and MPEG-4 MPEG-2 Leverages MPEG-1 (Motion Picture Experts Group 1988) Widely Used for Digital Video Digital Cable TV, DVD Transport Stream designed for Broadcast (Lossy, No Beginning or End of Stream) MPEG-4 Leverages MPEG-2 Standards for Program/Transport, Encode/Decode Better Compression Rates (Macroblock), MPEG-4 Part-10 (H.264), e.g. Blu-Ray Extensions for Digital Rights Management Advanced Audio Encoding Becoming More Widely Deployed for HD and Because of Lower Bit-Rate Transport Streams Sam Siewert 14
MT Review - MPEG2 Transport Stream vs. Program Stream PCR, PTS/DTS What are they for? Multi-program Transport Streams Sam Siewert 15