Video Video October 6,
Video Event-based programs read() is blocking server only works with single socket audio, network input need I/O multiplexing event-based programming also need to handle time-outs, connection requests all events (mouse clicks, windows, etc.) handled by event loop do wait for event(s) handle event (hopefully short) forever harder to maintain state, recursion alternative : interrupts (signals) called at any time alternaitve : threads (separate scheduling, same address space) October 6,
Video 3 Multiplexing with select() int select(int nfds, fd_set *readfds, fd_set *writefds, fd_set *exceptfds, struct timeval *timeout) block until >= file descriptors have something to be read, written, or an exception, or timeout set bit mask for descriptors to watch using FD SET returns with bits for ready descriptors set check with FD ISSET cannot specify amount of data ready October 6,
Video Audio timing Need to write block of audio to speaker every t ms (t =... ms). timer OS overhead may not be accurate error accumulation (time between timers) clock may differ from audio sampling clock. use audio input: for every block read, write one audio block stay in sync 3. but: doesn t work for half-duplex audio cards October 6,
Video 5 Analog video black & white (RS7) + color burst subcarrier 5,75 Hz horizontal scanning system where lines (disp.) size f/s bw NTSC US, Japan 55 (83) :3 3 6 MHz PAL Europe 65 (576) :3 5 8 MHz Scan line 3 5 7 9 The next field start here Scan line painted on the screen 3 Time 5 83 Horizontal retrace... Vertical retrace reduce flicker interlace scanning: fields/frame October 6,
Video 6 Digital video format lines pixels SCIF 8 7 CIF 88 35 QCIF 76 motion video: 5 to 3 f/s camera Red, Green, Blue chrominance, luminance YUV: Y = luminance, UV: color difference (red, blue Y) October 6,
Video 7 Digital video eye more sensitive to luminance subsample chrominance: ::, ::, :: RGB Y I 3 8 8 Block 8-Bit pixel treat chrominance and luminance differently (a) -Bit pixel Block 799 (b) Q October 6,
Video 8 Video coding lossless (entropy): (X ray images!) run-length, statistical encoding (Huffman coding,...) lossy: exploit spatial redundancy transform: frequency domain, higher quantization steps for higher frequencies vector quantization: map NxN block into N -dimensional space and find closest in codebook model-based: geometric description very low bit-rate October 6,
Video 9 Images X-ray digitization: pixels x lines, at bits/pixel slide: -5 dpi for 6.75 x.5 7 mm movie: lines 35 mm film: 753 lines slide film: lines/mm (5 lines/inch) HDTV: 5 lines fax: lines/inch October 6,
Video Video coding: JPEG (Joint Photographic Experts Group) individual (still) pictures lossless or lossy typically about Mb/s for video stream discrete cosine transform (DCT): samples blocks (6x6 Y, 8x8 UV) frequency domain D matrix, (,) = DC ( ) ( ) (i +)uπ (j +)vπ B u,v (i, j) = cos cos 6 6 with the transformed image F (u, v) = N C(u)C(v) 7 i= 7 f(i, j)b u,v (i, j) j= October 6,
Video Video coding: JPEG code frequency sampled with different resolution Q y DCT Fy x Fx October 6,
Video Video coding: JPEG (Joint Photographic Experts Group) DC or average (,) value: difference to previous block DCT Coefficients Quantized coefficients 5 9 5 8 75 38 8 3 36 6 6 8 6 7 5 9 6 3 8 75 9 8 3 3 Quantization table 8 6 3 8 6 3 8 6 3 8 6 3 8 8 8 8 8 6 3 6 6 6 6 6 6 3 3 3 3 3 3 3 3 October 6,
Video 3 JPEG zig-zag scan: 5 8 9 75 8 3 6 9 3 3 run-length coding: group all zeros Huffmann coding See also http://www.cs.columbia.edu/ hgs/video October 6,
Video JPEG bandwidth Size [Bytes] 3 5 5 Noise Face Face White Picture Black Picture 5 5 5 5 3 Q-Factor October 6,
Video 5 MPEG (Motion Picture Experts Group) Audio and video: Audio signal Audio encoder Clock System multiplexer MPEG output Video signal Video encoder JPEG + motion compensation H.6, MPEG video MPEG-:. Mb/s fixed rate (CD ROM) MPEG-: higher resolutions (HDTV), scaling image prediction: intra (I), forward (P), bidirectional (B) IBBPBBPBBPBBIBBPBBPB need I frames for error resiliency, joining movie in the middle October 6,
Video 6 MPEG I, P, B I B P P 3 frames B frames need to wait for next frame. October 6,
Video 7 MPEG: motion compensation transmit motion vectors to account for panning and zooming hard to find (must try lots), easy to decode October 6,
Video 8 H.6 video codec ISDN (n kb/s) conferencing lower delay conditional replenishment: only transmit blocks that are different motion vectors for each 6x6 macroblock: ±5 integer pixels GOB: macroblocks H, 3 macroblocks V, marked by start code October 6,
Video 9 H.63 video codec sub-qcif (8 96), CIF (7x576), 6CIF (8x5) motion prediction outside frame advanced prediction mode: vectors for each 8x8 block advanced intra prediction within picture slice-structured mode: non-overlapping rectangles scalability: temporal (B frames), SNR, spatial October 6,
Video HDTV subset of MPEG- video compression, Dolby AC-3 audio compression vestigial sideband modulation (8-VSB) of 9 Mb/s or 6-VSB for two channels in CATV formats: 8 x 7, 3, 6 Hz progressive scan 9 x 8, 3 progressive, 6 Hz interlaced MPEG- transport stream: fixed-length 88-byte packets (x7 ATM cells) one channel = one or more programs October 6,
Video Multiplexing Pack multiple streams into a single lower layer IETF: MIME, RTP (later) ITU: H. (synchronous, 8x8) MPEG: elementary, transport, program file formats: AVI, QuickTime October 6,
Video Characteristics of digital audio and video audio video rate 5.3...... 5 kb/s...5...9mb/s loss tolerance 5% 5...% packet size small large traffic constant + silences variable bit rate October 6,
Video 3 Audio traffic models talkspurt: constant bit rate: one packet every... ms mean:.67 s silence period: usually none (maybe transmit background noise value).3 s for telephone conversation, both roughly exponentially distributed double talk for hand-off may vary between conversations... only in aggregate October 6,
Video Video traffic models easy case: fit into constant bit rate quality, f/s, size buffer occupancy video codec network smoothing buffer alternative: variable rate mux gain of...6 short time scales: packets within slice or frame medium time scales: I, B, P packet pattern longer time scales: scene changes (every few seconds) higher rate looks similar at all time scales, long-term correlation, heavy-tailed distribution self-similar but: for short queues, long-term correlations don t matter October 6,