Bring out the Best in Pixels Video Pipe in Intel Processor Graphics Victor H. S. Ha and Yi-Jen Chiu Graphics Architecture, Intel Corp.
Legal INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS. Intel may make changes to specifications and product descriptions at any time, without notice. All products, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice. Intel processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request. Any code names featured are used internally within Intel to identify products that are in development and not yet publicly announced for release. Customers, licensees and other third parties are not authorized by Intel to use code names in advertising, promotion or marketing of any product or services and any such use of Intel's internal code names is at the sole risk of the user. Intel product plans in this presentation do not constitute Intel plan of record product roadmaps. Please contact your Intel representative to obtain Intel s current plan of record product roadmaps. Ultrabook products are offered in multiple models. Some models may not be available in your market. Consult your Ultrabook manufacturer. For more information and details, visit http://www.intel.com/ultrabook Performance claims: Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance Copyright 2013 Intel Corporation. All rights reserved. Intel, Intel Inside, the Intel logo, Intel Core, and Ultrabook are trademarks of Intel Corporation in the United States and other countries. *Other names and brands may be claimed as the property of others.
Contents Intel Processor Graphics Video Processing Pipe Video Processing Modules Control Panel Adaptive Processing Summary Q&A 3
4
Command Streamers Vertex Processing Rasterize Multi-Format Codec Media Processing Instruction Cache EU EU EU Array of Unified Execution Units EU EU EU Video Quality Engine Display Media Sampler Texture Sampler Pixel Ops 4 th Generation Intel Core Microprocessor, built on 22 nm process technology Next Generation Intel HD Graphics with Microsoft * DirectX * 11.1, OpenGL * 4.0, OpenCL* 1.2 support Three Simultaneous Display, HDMI, DisplayPort *, with high-resolution up to 4Kx2K Significant 3D and Media performance improvement 5
SD, FHD, UHD 4K Interlaced / Progressive Chroma Format: 420/422/444 JPG, MPG, H.264, VC1 WiFi WebCam BT Ethernet/LAN HDD USB DVD/BD Source Format, Content, Quality Intel Graphics Media Processing Network Mobility VGA Display Port DVI/HDMI WiDi 6
Bring out the Best in Pixels 0101010001001011101010101 Multi-format Codec Decode MPEG2 / VC1 AVC SVC MVC JPEG / MJPEG Denoise Deinterlace Contrast Saturation Skin-tone Gamut Color Correction Stabilization VQE Media Sampler Scaling Sharpness 7
Contents Intel Processor Graphics Video Processing Pipe Video Processing Modules Control Panel Adaptive Processing Summary Q&A 8
Multi-Format Codec Video Decode and Encode High Performance Parallel Engine Video Quality Engine Video Processing Color Processing Command Streamers Front-End Thread Dispatch/Management Multi - format Codec Media FrontEnd Video Processing Pipeline VQE Introducing Video Quality Engine 3D Geometry Processing EU Array of Execution Units EU EU EU Media Samplers Texture Samplers Media Accelerators $L3 Cache Rasterizer EU EU EU EU Pixel Ops Media Optimized Execution Units 9
Video Processing Pipe Dedicated video processing on newly designed Video Quality Engine (VQE) Support for an extensive suite of functions for higher quality video at lower power Denoise Deinterlace Skin-tone Gamut Compression Contrast Saturation ProcAmp Sharpness Scaler Image Stabilization Gamut Expansion CSC 10
Contents Intel Processor Graphics Video Processing Pipe Video Processing Modules Control Panel Adaptive Processing Summary Q&A 11
De-noise Spatial and Temporal De-noise Filter Global noise level measurement Content-adaptive spatiotemporal filtering of noise Motion history-based blending of spatial and temporal filter results Noisy Images Clean Images Block Interface Input: YCbCr 420/422 Output: YCbCr 420/422 12
De-interlace Convert interlaced contents to progressive format Edge-adaptive Spatial Interpolation result + Motion-compensated Interpolation result Motion-adaptive Blending with Temporal Filtering results based on Spatial Temporal Motion Measure (STMM) Detection of most common cadences Chroma Upsampling from 420 to 422 13
Skin-tone Processing Per-pixel Enhancement of Skin-tone Pixels Reproduce the natural skin colors on the display screen Skin Tone Detection identifies pixels with skin-like colors with per-pixel indicator Skin Tone Enhancement modifies the Saturation and Hue of the skin-tone pixels Original Input Skin-tone Definition Skin-tone Detection Skin-tone Enhancement Skin-tone Indicator Skin-tone Enhanced Output Block Interface Input: YCbCr444 Output: YCbCr444 with modified CbCr components; Per-pixel skin tone indicator 14
Contrast Enhancement Automatic Contrast Enhancement: Per-pixel mapping of luma to enhance contrast 1. Histogram of luma Y pixel values is generated for the input video frame 2. Piece-Wise Linear Function (PWLF) is generated from luma histogram 3. Pixel values are modified according to the PWLF Input Image Histogram Computation PWLF Mapping Enhanced Image Block Interface Input: YCbCr 444 Output: YCbCr 444 with modified Y 15
Saturation Enhancement Per-Pixel Saturation Enhancement Utilize 6 basic colors as primaries/anchors (Red, Green, Blue, Magenta, Yellow, Cyan) Adjust colorfulness (saturation) of pixels while maintaining their color (hue) Input Image Saturation Enhanced Image Block Interface Input: YCbCr 444 Output: YCbCr 444 with modified CbCr components 16
Color Correction Display proper colors on display screen 1. Inverse gamma correction via PWLF 2. 3x3 matrix multiplication with input/output offset 3. Forward gamma correction via PWLF Input Image Color Corrected Image Block Interface Input: RGB Output: RGB 17
Image Stabilization Stabilize shaky video contents captured by handheld camera devices Usage models Real-time playback: Watch real-time playback of video streams Offline processing: Stabilize shaky video and save/transmit for video sharing Block Interface Input: YCbCr 420 Output: YCbCr 420 18
Scaler Adaptive Video Scaler Advanced scaling method for adaptive scaling mode Content-adaptive per-pixel blending of polyphase filtering and bilinear filtering reduces ringing Programmable coefficient tables available for Y / CbCr channels in horizontal and vertical directions Support for up to 16:1 downscaling and max output picture size of 16Kx16K for YUV420/422/444/RGB input picture 19
Sharpness Enhancement Enhancement of Image Sharpness and Details Content adaptive spatial filtering applied to luma channel User control of sharpness enhancement strength = [0, 63] Adaptive to Skin Tone Pixels Skin tone information is utilized Enhance detail information without over-sharpening of the skin regions Interface Input: YCbCr 444 Output: YCbCr 444 20
Contents Intel Processor Graphics Video Processing Pipe Video Processing Modules Control Panel Adaptive Processing Summary Q&A 21
Control Panel Standard Color Correction Brightness Contrast Hue Saturation 22
Control Panel Input Range Limited Full Total Color Correction Saturation Enhancement Adjustment = [0, 255] 23
Control Panel Sharpness Driver Custom Settings = [0, 64] Skin Tone Enhancement On = [0, 9] Off 24
Control Panel Noise Reduction Luma only = [0, 64] Luma and Chroma = [0, 64] Contrast Enhancement On/Off Film Mode Detection On/Off 25
Contents Intel Processor Graphics Video Processing Pipe Control Panel Video Processing Modules Adaptive Processing Summary Q&A 26
Hardware Statistics Statistics are gathered on Per-block (16x4) basis for encoder stats Per-frame basis for other stats 16 bytes of encoder statistics data for per-block statistics is available Temporal Variances De-noise: Sum of block noise estimates and Number of blocks per frame A variety of per-frame data is stored in a linear surface Skin-tone: Ymax = Max luma value, Ymin = Min luma value, Counter = Number of skin pixels Contrast: Luma histogram with 256 bins and 24-bits per bin Gamut Compression: Sum of distances of out-of-gamut-range pixels clipped to 32 bits, Number of out-of-gamut-range pixels in 29 bits 27
Adaptive Processing & Control The goal of adaptive processing is to deliver optimized video experience Hardware statistics and adaptive control logic allow end-to-end optimization and effective sharing of data and information across all processing blocks 28
APC Example Noise Level Estimator Global Noise Level Noise Reduction Contrast Enhancement Skin-tone Detection Saturation Enhancement Sharpness Enhancement Global Motion Indicator Motion Detection Brightness Information Luma Histogram Skin-tone Indicator 29
Contents Intel Processor Graphics Video Processing Pipe Video Processing Modules Control Panel Adaptive Processing Summary Q&A 30
Summary Intel is building Media Solutions for Quality, Power, and Scalable Performance Dedicated HW acceleration for codecs, video and image processing, and analytics Media processing in 4 th Generation Processor Graphics delivers: VQE: A new feature with a full-set of video processing pipe Adaptive Processing & Control for integrated and optimized video pipe behavior Ready for various application scenarios and network/mobility environments Looking Ahead Future generations of Intel media processing will address further HW acceleration in video processing, camera image processing, and perceptual computing/machine vision applications Improvement in power and performance fitted to exciting new form-factors 31
Contents Intel Processor Graphics Video Processing Pipe Video Processing Modules Control Panel Adaptive Processing Summary Q&A 32