Update on Super HDTV Decoder Project

Transcription

1 Update on Super HDTV Decoder Project Youn-Long Lin 林永隆 Department of Computer Science National Tsing Hua University IC-DFN 27, Rizhao

2 More Pixels YLLIN NTHU-CS 2

3 NHK Proposes UHD TV Broadcast Super HiVision 768x432 pixels at 6 fps (16XHDTV) Baseband signal is 24 Gbps. Using 16 MPEG-2 encoding chips, the signal was compressed to 25 Mbps for transmission. HDTV signals at present are 1.5 Gbps for baseband and 2 Mbps for compressed signals. High Performance compression / decompression and transmission / storage are needed for 24 Gbps 3 Mbps YLLIN NTHU-CS 3

4 768x432 UHD TV 384x216 QFHD TV 192x18 HDTV SDTV YLLIN NTHU-CS 4

5 Applications QCIF CIF D2 72HD 18HD YLLIN NTHU-CS 5 QFHD

6 Video Coding Technology Trend H.264 5% 69% YLLIN NTHU-CS 6

7 Features of Video Coding Standards Standard MPEG-1 MPEG-2 MPEG-4 H.264/AVC MB size 16*16 16*16(frame) 16*16 16*16 Block size 8*8 8*8 16*16, 8*8 16*16, 16*8, 8*16, 8*8, 8*4, 4*8, 4*4 Transform DCT DCT DCT/ Wavelet 4*4 int transform Entropy coding VLC VLC VLC VLC, CAVLC and CABAC ME, MC Yes Yes Yes 41 MVs per MB Pixel accuracy ½ pel ½ pel ¼ pel ¼ pel Reference frames One frame One frame One frame Multiple (5) frames Picture type I, P, B I, P, B I, P, B I, P, B Transmission rate Up to 1.5 Mbps 2-15 Mbps 64kbps2Mbps 64kbps 15Mbps YLLIN NTHU-CS 7

8 Not all H.264/AVC systems are equal Relative Computational Complexity #Ref Frames Search Range Video Coding with H.264/AVC: Tools, Performance and Complexity, J. Ostermann et al, IEEE CAS Mag., Q1 24. YLLIN NTHU-CS 8

9 Quality vs Bit-rate vs Decoding Throughput Decoding Capability of a 6MHz CPU QP Bit Rate (Kbps) Fps H.264/AVC Baseline Profile Decoder Complexity Analysis, M. Horowitz, IEEE T-CSVT, July 23 YLLIN NTHU-CS 9

10 Our Target Single-Chip Decoder for QFHD (384x216) H.264/AVC High Profile Video.264 bitstream Video output CABAD Advanced Entropy Coder Mixed 4x4/8x8 Transform Commodity External Memory Platform-Based Design YLLIN NTHU-CS 1

11 Resolution vs. Needed Frequency 66 % Frequency Saving Resolution QFHD 16 VGA 18 HD 72 HD 4 x larger frame size Lin [3],Chen[32], Chien [46], Peng[48] Lin [31], Liu[42] Conexant [36] C&S [39] Kawakami [44] Clock Frequency (MHz) YLLIN NTHU-CS 11

12 Frequency Budget Resolution Size Clock Frequency Application SQCIF (128 x 96) 1..4 MHz Video phone QCIF (176 x 144) 2..8 MHz CIF (352 x 288) MHz Mobile TV D2 (72 x 48) MHz Car TV Surveillance 72HD (18 x 72) MHz Home theater 18HD (192 x 188) MHz QFHD (384 x 216) MHz Digital signage Medical video Satellite image Space exploration YLLIN NTHU-CS 12

13 Essential Issues Memory Tradeoff Between the Size of Internal Memory and Bandwidth of External Access Massive Parallelism (Pipelining) Macroblock Decoding Scheduling Power YLLIN NTHU-CS 13

14 NTHU H.264 Decoder Architecture CPU Display Memory Controller Ethernet AHB MAU & AMBA Interface Translator Parser CAVLD/ CABAD coeff mvdinfo IQ & IT MVG residual mv & ridx IPRED INTERP BSG recon bs DF para & predinfo H.264 Video Decoder YLLIN NTHU-CS 14

15 Memory

16 Memory Size (Bytes) size vs. b/w in ME D Full HD 3fps, # of rf =1, SRV=SRH=64 Level A : 24 Bytes, MB/s Level B : 12 Bytes, 1516MB/s Level C: 4977 Bytes, 317MB/s Level D: 124,929 Bytes, 62 MB/s 4977 C B 12 A Memory Bandwidth (MB/s) YLLIN NTHU-CS 16

17 rf mem rf1 mem CB mem rf AG CB AG IME block diagram rf router rf reg array CMB reg CMB reg CMB reg CMB reg comparator comparator comparator comparator MVGen rf MVGen rf MVGen rf MVGen rf MVGen rf MVGen rf MVGen rf MVGen rf YLLIN NTHU-CS 17 MV AG MV mem

18 Memory Size (Bytes) size vs. b/w in ME D C 4977 B ours A Memory Bandwidth (MB/s) YLLIN NTHU-CS 18

19 Reference-data Pre-fetch System No redundant fetching Collecting several MB s motion vectors, and read the same place by only one single operation Minimize the number of burst initials On average, 2 burst initials per MB (1 for luma, 1 for chroma) : a group of sequentially read (burst read) YLLIN NTHU-CS 19

20 Reference-data Pre-fetch System (Cont) CABAD.... MB1 MB9 MB8 MB7 R7 Reference Region & Index Register R6 MB6 MB7 MB7 Region Information MB7 MV R5 MB4 MB5 MB6 MB7 Translator Motion Vector Generator R4 MB4 MB5 R3 MB2 MB3 MB4 R2 MB1 MB2 R1 MB MB1 MB2 R MB R2 Information Region Analyzer / Searcher OES manager MAU Interface Buffer R2 Information R R1 R2 R2 Data from SDRAM R/R1 Data MB7 Information MB7 MB6 MB5 MB4 MB3 MB2 MB1 MB Interp YLLIN NTHU-CS 2

21 Massive Parallelism

22 IQ/IDCT Timing Diagram t coeflag_mem read coeff_mem read 16 luma ac luma ac_14_15 4 dc 15 chroma ac chroma ac_6_7 IQ stage 1 16 luma ac luma ac_14_15 15 chroma ac chroma ac_6_7 IQ stage 2 16 luma ac luma ac_14_15 15 chroma ac chroma ac_6_7 IDCT stage IDCT stage residual_mem write YLLIN NTHU-CS 22

23 Deblocking Filter Timing Diagram YLLIN NTHU-CS 23

24 Dual Pipelined Edge Filter Stage 1 Read Pixels L L1 L2 L3 M M1 M2 M3 R R1 R2 R3 Stage 2 Strong filter (Bs=4)/ Left delta calculation R21 delta calculation L1 L11 L12 L13 Left delta M1 M11 M12 M13 R21 delta R1 R11 R12 R13 Stage 3 Left Weak Filter (Bs<4) Right delta calculation R21 filter L2 L21 L22 L23 M2 M21 M22 M23 Right delta R2 R21 R22 R23 Stage 4 Right Weak filter (Bs<4) L31 L3 L32 L33 M3 M31 M32 M33 R3 R31 R32 R33 Stage 5 Write Pixels YLLIN NTHU-CS 24

25 System-Level Optimization Cyclic-Queue-Based IP Interface

26 Main Controller CPU Display Memory Controller Ethernet AHB H.264 Video Decoder Decoder Controller MAIN CONTROL FSM PARSER FSM CABAD FSM CAVLD FSM IQ/IT FSM MVG FSM INTERP FSM IPRED FSM BSG FSM DF FSM MFU FSM PARSER CABAD CAVLD IQ/IT MVG INTERP IPRED BSG DF MFU YLLIN NTHU-CS 26

27 Performance Gap in Elastic Pipeline About 25% performance drop between the actual and the ideal situation Pattern: pedestrian QP: 28 Resolution:72*48 GOP: III Frame #: Actual Ideal 4 2 Whole CABAD IQ/IT IPRED BSG DF Idle Cycles/MB Procesing Cycles/MB YLLIN NTHU-CS 27

28 Elastic Pipeline Decoder Timing Diagram (I Frame) PARSER CABAD Bubble cycles 1 to 1 cycles per MB IQ/IT BSG IPRED DF Header information decode Initial context table and condition offset MB-Level decode YLLIN NTHU-CS 28 (time)

29 Elastic Pipeline Decoder Timing Diagram (I Frame) PARSER CABAD Bubble cycles IQ/IT BSG IPRED DF Header information decode Initial context table and condition offset MB-Level decode YLLIN NTHU-CS 29 (time)

30 Timing Diagram after ASAP Scheduling PARSER CABAD However, bubble cycles still exist Reduced bubble cycles IQ/IT BSG IPRED DF Header information decode Initial context table and condition offset MB-Level decode YLLIN NTHU-CS 3 (time)

31 Timing Diagram after ASAP Scheduling PARSER Remaining bubble cycles CABAD IQ/IT BSG IPRED DF Header information decode Initial context table and condition offset MB-Level decode YLLIN NTHU-CS 31 (time)

32 Timing Diagram after Cyclic Queue Insertion PARSER CABAD Reduced remaining bubble cycles Total reduced bubble cycles IQ/IT BSG IPRED DF Header information decode Initial context table and condition offset MB-Level decode YLLIN NTHU-CS 32 (time)

33 Elastic Pipeline Decoder Timing Diagram (P/B Frame) PARSER CABAD IQ/IT MVG INTERP BSG DF Header information decode Initial context table and condition offset MB-Level decode (time) YLLIN NTHU-CS 33

34 Cyclic Queue Decoder Timing Diagram (P/B Frame) PARSER CABAD Reduced Bubble Cycles IQ/IT MVG INTERP BSG Reduced Processing Cycles DF Header information decode Initial context table and condition offset MB-Level decode (time) YLLIN NTHU-CS 34

35 Comparison of Different Scheduling Methods (Cycles/ MB) KB Sequential Elastic Pipeline ASAP Ping-Pong ASAP Cyclicqueue SRAM Usage Turnaround Cycle Processing Cycle YLLIN NTHU-CS 35 Test Pattern: pedestrian Resolution: 72*48 QP: 28 GOP: III Frame #: 3

36 Clock Gating Manual clock gating assign gclk = clk & ip_en; clk cabac_en picrec_en idct_en ipred_en df_en mc_en cabac picrec idct ipred df mc YLLIN NTHU-CS 36

37 Two Clock Gating Methods # of clock gating elements # of gated registers # of un-gated register Viability Register-based (91.1%) 137(8.9%) No, Malfunction Module-based (67.72%) 379(32.28%) Yes, Function YLLIN NTHU-CS 37

38 Verification Environment H264 filelist tbench fpga_lib rtl_sim asic_lib mfu amba_wrap top lm_wrap main_ctrl Easy Bug Tracing gate_sim mvg bsg parser mem cabad vn idct ipred nlint interp netlist df Sub IP jm11. hd_amba syn def filelist tbench rtl_sim xilinx_mem altera_mem artisan_mem rtl syn vn nlint gate_sim YLLIN NTHU-CS 38

39 Verification Flow SW Designer Sys Designer IP Designer Golden (1) Test Data Test data extraction SW profiling modify Reference SW Design Spec IP Spec Sys Spec C model SW Spec IP design IP testbench design Sys design Sys testbench design SW design No No IP linting no error? func. currect? Yes Yes RTL code (2) RTL code (3) RTL-Sim (1) Coverage Analysis (1) (2) (2) (3) IP/Sys Integration (1) (3) IP/Sys RTL-Sim (4) (5) RTL code (4) RTL code (5) IP/Sys Synthesis & Gate-Sim (1) (4) HW image Compilation C code SW image (1) No meet Yes IP Synthesis Yes Sys No func. (3) Sys building criterion? & Gate-Sim fail? currect? No Yes No func. Yes Yes IP Delivering IP No HW image currect? YLLIN fail? NTHU-CS Prototype Delivering 39

40 A Multimedia SOC Platform CPU Accelerator (FPGA) USB(PHY) Daughter Board ROM/ Flash Memory SRAM SDRAM FPGA VIC USB 2. Static memory SDRAM Controller(4-CH) High-Speed Bus JPEG Codec DMA SRAM PWM WDT TIMER APB Bridge Capture Display Controller Peripheral Bus DAI SSI SD SM UART GPIO 12C Audio Codec I2S Flash memory with SSI Video-In CCIR61 TV/LCD YLLIN NTHU-CS 4 Flash Card Button LED

41 Summary Super High Definition Video Capturing, Delivery and Display are on the Horizon Massive Parallelism is Essential for Making Consumer Applications Possible Tradeoff Among Memory Usage, Bandwidth and Logic Has Profound Impact on the Overall System Performance System Design Should Be Adaptable to Content, Quality Variation YLLIN NTHU-CS 41

42 Thank You!!