Computer and Machine Vision Lecture Week 3 Part-1 January 27, 2014 Sam Siewert
Outline of Week 3 Processing Images and Moving Pictures High Level View and Computer Architecture for it Linux Platforms for Computer/Machine Vision I/O, Memory and Processing Challenges Sam Siewert 2
Old School Moving Picture Media and Cameras NTSC OTA (1941, 1953 color, 2009 dead) Analog, Interlaced, Continuous Broadcast Transmission or CCTV (Closed Circuit TV) Coax Cable or Tuner with Immediate CRT Display No Buffers, No Routing, No De-mux No Compression Analog Cable AM/FM OTA Film Projectors Sam Siewert 3
Modern Digital Cameras Camera Link High Frame Rates High Data Rates and Resolutions Industry Standard for Machine Vision Automation E.g. Inspection Systems E.g. Sony, IDT, National Instruments SD-SDI and HD-SDI Standard and High Definition Synchronous Digital Interface Standard for Studios, Broadcast Digital Cinema Red Camera 1080p, 2K, 4K Resolutions and Much Higher Automated Digital Delivery and Projection Webcams and Mobile Phone Cameras Very Low Cost Proprietary Performance Varies Dramatically Sam Siewert 4
Differences Analog vs Digital Encoding for Transmission Digital Allows for Image Processing Adds Latency Requires Compression for Packet Switched Networks and Storage Routed (Diversely), Buffered Compressed (MPEG, JPEG) to Lower Bit-rates Multiplexed (Shares Transmission Carrier for Audio, Video, Channels) Transported by IP (Large Packets) Continuous Transmission Analog or Constant Bit-Rate / Frame-Rate Sam Siewert 5
E.g. UAV Latency and Jitter Verification of Video Frame Latency Telemetry for UAV Systems Using A Secondary Optical Method, Sam Siewert, Muhammad Ahmad, Kevin Yao Sam Siewert 6
NTSC (Analog TV) http://en.wikipedia.org/wiki/file:ntsc_channel.svg AM Video to CRT FM Audio Chroma Added Later Odd/Even Lines (Interlaced) 29.97 FPS (30 before color) Vertical Blanking (CRT Retrace Time, Closed Captioning) 525 Lines, 262.5 per Field, 60 Fields per Second Sam Siewert 7
Linux in Computer Vision Embedded Solutions Texas Instruments OMAP (Beagle xm, Bone) Numerous ARM SoCs (NVIDIA, Qualcomm, Broadcomm, ) Scalable Solutions Multi-Core (Xeon Phi) Vector Processing CUDA, OpenCL GPU and GP- GPU Computer and Machine Vision is I/O, Memory and Processing Intensive Sam Siewert 8
Camera Interfaces CCD (Charge Coupled Device) or CMOS (Common Metal Oxide Substrate) Detector Integration Time for Photo-sensitive Elements in Array (to Build up Charge) Read-out Time to Sample Elements in Array Luminance and Chroma Analog to Digital Conversion Double Buffer for Read-out + Processing Frame Capture http://www.cse.uaa.alaska.edu/~ssiewert/a485_doc/frame- Capture-Chips/ Host Interface over PCI Bus or USB Sam Siewert 9
Digital Video Transport QoS Latency To Tune in a Program, Turn-on To Deliver a Video Frame or Audio PCM Sample To Start, FF, REW, Start-Over, Pause Bandwidth Resolution, Lossy/Lossless Compression, High Motion Pixel Encoding for Color Frame Rate Constant Bit-rate Transport? Variable Bit-rate Transport and Encoding? Jitter Decode and Presentation Rates Elasticity in Decode to Presentation Buffering Necessary Sam Siewert 10
Linux System Options (Linux for Image Processing, Camera Interfacing and Computer Vision) January 27, 2014 Sam Siewert
Processing Outline Many-Core Linux Host(s) Intel Atom ARM Xeon GP-GPU Vector Processing PCI-E Co-Processors NVIDIA Tesla/Fermi AMD ATI NPTL Native POSIX Threads Library NPTL Example Code Walkthrough Sam Siewert 12
Conceptual View of RT Resources Memory-Utility CPU-Utility IO-Utility Three-Space View of Utilization Requirements CPU Margin? IO Latency (and Bandwidth) Margin? Memory Capacity (and Latency) Margin? Upper Right Front Corner Low-Margin Origin High-Margin Mobile Must Consider Battery Life Too (Power) Sam Siewert 13
Processing Initial Focus Processing and Scaling Frame Transformation, Encode, Decode is Critical Memory for Buffering (Frame Transformations, CPU Integrated or GPU Offloaded e.g. Linux VDPAU) I/O for Networking (Transport) I/O for Storage (On-Demand, Post, Non-Linear Editing) Sam Siewert 14
Flynn s Computer Architecture Single Data Taxonomy Single Instruction SISD (Traditional Uniprocessor) Multiple Instruction MISD (Voting schemes and active-active controllers) Multiple Data SIMD (e.g. SSE 4.2, GP- GPU, Vector Processing) MIMD (Distributed systems (MPMD), Clusters with MPI/PVM (SPMD), AMP/SMP) GPC has gone MIMD with SIMD Instruction Sets and SIMD Offload (GP-GPU) NUMA vs. UMA (Trend away from UMA to NUMA or MCH vs. IOH) SMP with One OS (Shared Memory, CPU-balanced Interrupt Handling, Process Load Balancing, Mutli-User, Multi-Application, CPU Affinity Possible) MIMD - Single Program Multi-Data vs. Multi-Program Multi-Data Sam Siewert 15
Computer and Machine Vision Treated as a Real-time and/or Interactive System Requires Predictable Response (By Deadline) Rate Monotonic Earliest Deadline First Least Laxity First Covered in Depth in OS and Digital Media (Just be Aware for Computer Vision) Sam Siewert 16
CPU Scheduling Taxonomy Execution Scheduling Global-MP Local-Uniprocessor Dynamic Static Preemptive Non-Preemptive Symmetric (SMP OS) SMT (Micro-Paralell) Asymmetric (AMP ) Distributed Fixed-Priority Batch (Preemptive, Non-Preemptive Subtree Under Each Global-MP Leaf) Hybrid Rate Monotonic Deadline Monotonic FCFS SJN Dynamic-Priority Cooperative Dataflow Heuristic EDF/LLF RR Timeslice (desktop) Multi-Frequency Executives Co-Routine Continuation Function Sam Siewert 17
C i WCET Input/Output Latency Interference Time Response Latency Response Time = Time Actuation Time Sensed (From Release to Response) Event Sensed Interrupt Dispatch Preemption Dispatch Completion (IO Queued) Actuation (IO Completion) Interference Time Input-Latency Dispatch-Latency Execution Execution Output-Latency Sam Siewert 18
Resource Scaling Processing Co-Processors GPU CUDA, OpenCL Many-Core E.g. Intel Xeon Phi MICA FPGA E.g. Altera Stratix Ideally Camera Interface I/O High Rate Transport HD-SDI, Camera Link, GigE/10GE Memory SSD, PCIe Nand, NVM FusionIO, Micron, Intel Memristor (Future) Sam Siewert 19