Efficient FPGA-based Video Systems Aaron Behman Xilinx
Agenda About Xilinx Trends in Professional Video FPGAs in the Video Value Chain Video System Architecture FPGA Video System Software Defined Video Infrastructure Conclusion
About Xilinx Pioneer of the FPGA $2.3B in revenue Largest player in the $5B PLD industry Serve 20K customers globally 3,000 employees Strategically focused on six megatrends Video / Vision ADAS 5G Industrial IoT Cloud Compute SDN/NFV
Consumers Want To Be Immersed Faster Pixels Better Pixels More Pixels 240Hz/fps 120Hz/fps 60Hz/fps Wider Color Gamut Higher Dynamic Range Deeper Color Depth SHV 8K UHD 4K New Way HDMI 2.1 FHD 1080P HD 720P Old Way HDMI 1.4 HDMI 2.0 DP 1.2 DP 1.3 SD DP 1.1 24G-SDI SDI HD-SDI 3G-SDI 12G-SDI 270 Mbps 1.5 Gbps 3 Gbps 12 Gbps 24 Gbps MSOs and Terrestrial OTT LIVE & VOD OVER BB or LTE
Advertising From New Channels Video 20% Network Traffic 2 #1 Broadcaster 1 by 2016 2018 OTT > TV 3 by 2019 Opportunities and Realities: Computational demands growing exponentially Broadcasters seek Moore s Law effects in content production Traditional broadcast business being threatened
Must Scale To Remain Relevant SDI SDI + IP All IP COTS Server Video Accelerated Server HD Era 4K Era 8K Era Hyper Immersive Dedicated Hardware Selling Products Software Enabled Selling Services
FPGAs in the Video Value Chain Acquisition Contribution Distribution Consumption
Video System Architecture Image Sensor Processing Video Display Processing Graphics Input Interfaces Framework / Infrastructure Output Interfaces Codecs Analytics Encryption Input/Output Interfaces: HDMI, DisplayPort, SDI (incl. 6G & 12G), MIPI, LVDS, SMPTE ST 2022, Eth AVB Image Sensor Processing(ISP): Image processing pipeline algorithms: De-Bayer, image stats, color correction, pixel correction, noise reduction High Level Synthesis (HLS) Video Display Processing: Scaling, deinterlacing, on-screed display (OSD)
Video System Architecture Image Sensor Processing Video Display Processing Graphics Input Interfaces Framework / Infrastructure Output Interfaces Codecs Analytics Encryption Codecs: JPEG 2000, TICO, VC-2, H.264, HEVC/H.265 Graphics: 3D graphics engine (soft) OpenGL ES 1.1., ARM Mali 400 Analytics: HOG, Optical Flow, Deep Learning (CNN) Encryption: HDCP Framework / Infrastructure (HLS, SDAccel & SDSoC): AXI4 interconnect for control plane, memory management and streaming
FPGA Video System Input/Output Interfaces: SDI (6 & 12G), HDMI 2.0, DisplayPort 1.2, HDCP Video Display Processing: Support for up to 4096 x 2160 @ 120 Hz Key algorithms: deinterlacer, scalar, on-screen display Graphics: Qt graphics running in ARM processing system Framework / Infrastructure: AXI4 for memory management, processor control plane, Linux-based web UI
Video Display Porcessing Multichannel Resize & Overlay Aspect Ratio Conversion & Crop Colour Correction Motion Adaptive Deinterlacing Low Angle Detection & Processing Jaggies removed Gamma Correction Text over Film Processing Tearing removed Moire pattern Noise removed Noise Reduction Cadence Detection & Handling removed
FPGA Video System (Con t) Functional Block Diagram Target Hardware (OZ745)
Software Defined Video Infrastructure Input/Output Interfaces: SDI, SMPTE ST 2022-1 Codecs: JPEG 2000 Framework / Infrastructure: AXI4 for memory management, streaming interconnect between logic blocks, provide API access to the parameters of the cores (over AXI4 lite) HLS tool used to develop new apps in very short order (RFC4175 & OSD)
Software Defined Video Infrastructure Functional Block Diagram Target Hardware
Conclusion FPGAs & FPGA-based SOCs very capable of solving computationally intensive problems like video processing more efficiently than alternatives High level synthesis tools and heterogeneous computing software defined environments for SOCs and FPGAs with CPUs open up FPGA technology to a wider audience, e.g. software engineering teams Video and image processing algorithms are an ideal use case for these software defined environments