A320 Supplemental Digital Media Material for OS Lecture 1 - Introduction November 8, 2013 Sam Siewert
Digital Media and Interactive Course Topics Digital Media Digital Video Encoding/Decoding Machine Vision (For Interactive Aspects) Gesture Recognition Augmented Reality Graphics Post-Production CGI With Blender and Pixie (RenderMan) for Ray-Trace Rendering Linux-based Labs Sam Siewert 2
The World of Interactive Digital Media Sam Siewert 3
Scalable Linux Systems From Game Consoles to Super-Computing PS3 Blue Gene GPGPU Tianhe-1 Pflop http://www.nscc-tj.gov.cn/en/ From Android Mobiles to GIS and Digital Video Services Huge Value in Open Source Drivers, Tools, and Applications Speeds Up Time to Market Focus on Leveraging Linux for Desktop and Embedded Systems for Machine Vision and Graphics Sam Siewert 4
Mobile Linux NDK Layer Digital Video Exposed in AOS/iOS Handled in OS layer AOS and ios for Java Apps that Manage DM and User data Still Need NDK OS drivers (E.g. Beagle xm OpenCV) Sam Siewert 5
Digital Media Systems Embedded Media Devices Set-Top Boxes (Linux) Mobile Media Systems: Smart Phones, Tablet Computing, Readers, Notebooks, DVD Players, ipods, etc. Digital Camera Systems (SD, HD, HD-SDI, 2K, 4K, 6K) Resolutions/Formats - http://en.wikipedia.org/wiki/file:vector_video_standards2.svg Game Consoles: X-box, PS3, Etc. Gesture Recognition, Augmented Reality SD, HD Cameras and Interfaces: Composite, S-Video, Component, DVI, HDMI Scalable Digital Media Server Systems (Head End) Post Production for Digital Cinema, TV, Web 2K, 4K, 6K Streams from Digital Cameras Frame/Color Editing, CGI (Computer Generated Imagery), Soundtrack, Write to Distribution Media Digital Cinema: HD Digital Projectors, 3D Digital Projectors Closed Circuit Security Systems: Multi-Camera NTSC/HD Sam Siewert 6
Digital Media Focus Scalable Post Production Pipelines Multi-core Threading with POSIX Threads (CPU scaling) I/O Scheduling and Tuning (Disk bottlenecks and Capacity Scale) Post-Production HD Frame Transformations Color Transformations, Edge Enhancement Integrated Graphics & Video (Interactive TV, Anmiation) Green Screen Replacement, Video Morphing Segmentation, Recognition, Tagging Ray-Trace Rendering Pixie for Renderman Blender Sam Siewert 7
Old School Moving Picture Media NTSC OTA (1941, 1953 color, 2009 dead) Analog, Interlaced, Continuous OTA Broadcast Transmission Tuner with Immediate CRT Display No Buffers, No Routing, No De-mux No Compression Analog Cable AM/FM OTA Film Projectors Sam Siewert 8
New Digital Media Digital Cable QAM 256, 30+ Mbps, 10+ MPEG Programs per 6Mhz Channel Minimal Buffering (In Set-top Box for Digital Tuning and On-Demand) Dedicated Coaxial RF Carrier (Hybrid Fiber to Coaxial Networks) On-Demand, Trick-Play, Start-Over DOCSIS for Internet and Return Path (Streaming Control) ATSC Digital OTA Supports HD 1080p or Multiple SD Programs per 6Mhz Channel Digital Modulation (8VSB) at 19+ Mbps per Channel Digital Cinema 1080p, 2K, 4K Resolutions Automated Digital Delivery and Projection IPTV, IP Radio and Mobile Media Routed, Buffered, Compressed Multiplexed Video/Audio Transport Streams File Download or Network Streaming Streaming over UDP or RTP/UDP with RTSP Most Often, No Retransmission Sam Siewert 9
Differences Analog vs Digital Encoding for Transmission NTSC Frequency Modulation on Channels Broadband QPSK, QAM, 8VSB OTA Baseband Packet Switched Networks (Optical, Ethernet) Routed (Diversely?) Buffered Compressed Multiplexed (Shares Transmission Carrier) Transported by IP (Large Packets) QoS? Continuous Transmission with Instant Tuning vs. Digital Network Streaming vs. Download and Playback (e.g. YouTube) Sam Siewert 10
NTSC (Analog TV) AM Video to CRT FM Audio Chroma Added Later Odd/Even Lines (Interlaced) 29.97 FPS (30 before color) Vertical Blanking (CRT Retrace Time, Closed Captioning) 525 Lines, 262.5 per Field, 60 Fields per Second Sam Siewert 11
MPEG2 Fundamentals Basic Head-End Broadband MPEG2 System Broadcast VoD Services Config & Playlist Video Services Control Interface MPTS Playback SPTS Playback QAM Driver Server Bit-streams Pre-mux Tools DVB-ASI PCI DVB-ASI Analyzer IP Network QAM-RF QAM-SA STBs PRO-1000 Quad Sam Siewert 12
MPEG Encoding / Decoding The Motivation is Compression for Transport Over Networks and Efficient Storage Sam Siewert 13
The Key Step is DCT for I-Frames Convolution of Image with Discrete Cosine See http://www.cse.uaa.alaska.edu/~ssiewert/a490dmis_code/example-dct1/ De-convolved to restore image from Convolved Image DCT Inverse DCT Sam Siewert 14
Linux in Digital Media Common in Digital Cable Set-Top Boxes Common in Android Mobile Media Used in Digital Video VoD Head-Ends Used in Post Production Common for IPTV Used with Newer H.264 MPEG-4 for Mobiles and CV HD-SDI (Broadcast) NTSC (Old OTA/Coaxial Broadcast, Current CCTV), ATSC (New OTA Broadcast) Camera Link (http://en.wikipedia.org/wiki/camera_link) Sam Siewert 15
Digital Transport QoS Latency To Tune in a Program, Turn-on To Deliver a Video Frame or Audio PCM Sample To Start, FF, REW, Start-Over, Pause Bandwidth Resolution, Lossy/Lossless Compression, High Motion Pixel Encoding for Color Frame Rate Constant Bit-rate Transport? Variable Bit-rate Transport and Encoding? Jitter Decode and Presentation Rates Elasticity in Decode to Presentation Buffering Necessary Sam Siewert 16
Linux Desktop Options Native Linux Installation Ubuntu Logitech C200 Camera ffmpeg GLUT Using Amazon Free Micro-Instances Requires a Credit Card Account with Amazon Can Do Spot-Instance Bids for Large Nodes and Clusters Can Do Spot-Instance Bids for GPGPU Nodes Virtual-Box Ubuntu Installation Sam Siewert 17
Linux Embedded Options Beagle xm with Angstrom or Ubuntu Linux Logitech C200 Camera Leopard HD Camera Sam Siewert 18
Final Project! You will Produce your Own Short DVD (Movie) DVD Plus Write-Up Code, Methods to Produce, and Design Using ffmpeg to Encode SD/HD Digital Video from Frames you have Modified or Rendered At least 180 Seconds in Length at 30Hz, So 5400 frames Original Creative Work Using Graphics, Machine Vision, and/or Digital Video Processing Methods Learned in Class Sam Siewert 19
Linux Digital Video Processing Skills Introduction Session August 24, 2004 Sam Siewert
What are NPTL POSIX Threads? Example Code Posted on Blackboard Start looking at this code and playing with it Thread Creation and Scheduling Policy Thread Parameters Thread Entry Points and Functions Thread Completion Intro to First Lab Check Blackboard for Labs Sam Siewert 21
Let s Look at Some Code Threading and Image Segmentation for Many-Core and GPGPU Example Published in IBM Paper Example Code Used in First Lab Threading!! Grids for Threading, Step One toward Real-Time Processing Examples of Video/Still-Frame Enhancement Sam Siewert 22