Pre-Workshop Tutorial HD Transport, Timing and Compression

Transcription

1 Pre-Workshop Tutorial HD Transport, Timing and Compression Paul Hightower CEO Instrumentation Technology Systems May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 1

2 HD Video New Concerns Resolution is a more complex subject More formats High transport data rates Large volumes of data for archive Alias artifacts in images Benefits Order of magnitude improvement in imagery Wider color gamut Recording and reproduction can be identical to raw video Single Frames approach snap shot quality Metadata space built in to the SMPTE frames May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 2

3 Translating between NTSC (analog) & HD-SDI (digital) Video NTSC video -color bar test pattern May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 3

4 Translating between NTSC (analog) & HD-SDI (digital) Video At the SDI source At the end of a 100 meter cable May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 4

5 HD-SDI Imagery Chain ITU-R-BT-xxx Image sampling standards 709 includes 4:2:2 SMPTE 296/274 HD Format Sampling Standards (720/1080) SMPTE 291 HD Metadata Standard SMPTE 292/424 HD-SDI Serialization Standards (720p & 1080i/1080p-3G May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 5

6 Translating between NTSC (analog) & HD-SDI (digital) Video Attribute Analog Video SDI Digital Video Visible Scan Lines Resolution/Line Color Sampling NTSC 480/frame PAL 576/frame This depends on the source and signal quality but ranges to the equivalent of 300 to 720 pixels Continuous time domain signal, intensity swings are limited by the available 1.5 MHz bandwidth 480 SD 576 (PAL) SD 720 HD (750 total/frame) 1080 HD (1125 total/frame) SD 720 pixels HD 1280 (1650 total/line) pixels and 1920 pixels ( 2200 total/line) pixels YUV encoding samples intensity every pixel and color differently depending on the encoding chosen. 4:2:2 is most frequently used Sample rate = 74MHz (720p/1080i) Color BW 6 10 MHz Sample rate = 148MHz (1080p) Color BW 20 MHz May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 6

7 Translating between NTSC (analog) & HD-SDI (digital) Video Attribute Analog Video SDI Digital Video Raw Video Complex AM, FM and phase modulated signal requiring 6 MHz bandwidth Serial encoded bit stream at bit rates from 270 Mbits/s to 3000 Mbits/sec (12 GB for 4K) Sync Blanking Active Video Frame/Field Rate Pedestal and color burst sync areas scaled generally below the black level A predetermined voltage level in the video signal An AM signal with overlaid phase modulated color information RS Hz /30 Hz Field/Frame RS170A (NTSC) (60/1.001) CCIR 50Hz /25Hz field/frame Progressive and Interlaced A reserved bit patterns (SAV & EAV) defined by SMPTE in the image frame ANC space between EAV & SAV is Horizontal blanking data space, ANC space between SAV and EAV from first line to line 40 (1080) or 25 (720) is Vertical blanking data space. A stream of video samples between SAV & EAV the number of which and format varies with resolution, sampling scheme and color depth Many from to 60 Hz and beyond Progressive, segmented and Interlaced 7 May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 7

8 Quick Idea About Subsampling 4:2:2 subsampling causes two luma samples to share one pair (Cr and Cb) of color samples Graphic from Chrominance Subsampling in Digital Images, by Douglas Kerr May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 8

9 Video Data 4:2:2 10 Bit Color Sample Color Sample Active Video Data Stream Cr 0-1 Y 0 Cb 0-1 Y 1 Cr 2-3 Y 2 Cb 2-3 Y 3 Sample Pair Sample Pair Sample Pair Sample Pair Pixel 0 Pixel 1 Pixel 2 Pixel 3 Each Y sample represents a pixel > 1920 pixels on a line=1920 Y samples Each Y sample is coupled to ½ of a color sample as shown Each Pixel then is comprised of 1 Y and ½ 10-Bits each Each pixel is 20 bits as a way of thinking about it A 1920x1080 frame = 20x1920x1080 bits =41,472,000 bits or 5,184,000 bytes (image only) Add in the EAV, SAV, Line Count, CRC, HANC and VANC space; Frame = 2200 S x 1125 L 2200 samples x 1125 lines = 20x2200x1125/8=6,187,500 bytes/frame At 60 FPS; Data rate = 371,250,000 bytes/second Archive 1 minute of video = 22,275 Mbytes Frame Rate only affects Data Rate Cb = blue color value Cr= red color value Y = luma value LN= line count ANC is ancillary data space CA = color channel Y = Luma channel Ancillary data is placed in Luma channel first until full May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 9

10 Image Frame Organization The Samples build a frame 720 Frame Parameters Frame size = [750 lines x 1650 samples x 20 bits/pixel ]/(8 bits/byte) = 3,093,750 bytes/frame HANC Space 370 Samples x 750 VANC Space 25 Lines * 1280 samples = 32K Cinema & broadcast generally only use lines for scene switch & closed captioning EAV Line Count CRC >2000 data items Example Packets 16 channel audio snippets Film Codes Payload data Workflow data Copyright data V-chip data Billing data Logging Info SAV EAV Buffer Space Lines HANC 1280 luma samples 1650 luma samples May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 10

11 SDI = Frame Data Serialization 4:2:2 Multiplexed Encoded SDI Video Stream Byte level stack up SDI Serial Stream IAW SMPTE 292 or 454 When serialized 720p/60 =3,093,750 bytes/frame * 60 fps =185,625,000 bytes/second 185,625,000 bytes/second *8 = 1,485,000,000 bps. May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 11

12 Imagery Movies Frame Rate (pictures per second) Set to give the illusion of smooth motion; beyond persistence of vision frequency. Rates above 16 images/second yield smooth motion 24 fps is used in movies 25/30 fps used in TV (PAL/NTSC) Illumination Rate (flashes per second) Flicker fusion is the frequency that pulsing light looks steady Illumination rate is pushed high enough to achieve flicker fusion Film generally uses 48 Hz flicker rate TV targets 60 Hz flicker rate Interlace TV scanning is 2x the frame rate; Odd lines then even lines Progressive 30 FPS TV displays the same frame 60 Hz Progressive 60 FPS displays each 1/60 th of a second May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 12

13 NTSC National Television System Committee (NTSC) Did set the standards for analog TV format Color upgrade for TV in the 50s required B&W TVs to receive Color Signal 60Hz scan rate slowed to make room for color burst signal = flicker frame rate Digital Video on modern displays don t need this Legacy rate is predominant and remains May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 13

14 Imagery May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 14

15 Imagery Resolution is 3 dimensional Line Count Pixel Count/Line Bit Depth/Pixel Lines Pixels May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 15

16 Imagery More Bits = More Shades = More detail May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 16

17 Imagery 2 bits = 4 colors All pictures have the same number of pixels 4 bits = 16 colors 8 bits = 256 colors 24 bits = 16 million colors According to humans can distinguish up to 10 million colors Pictures from May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 17 17

18 Imagery Rendering Effects of Pixels and Bit Depth Same Image Few pixels Many shades Many pixels Few shades Many pixels Many shades May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 18 18

19 Imagery Sampling RGB; the primary colors Intensity; luminescence, (luma), the black & white information How many samples per image? (pixel count) How many shades per sample of each color? (bit depth) Many Bits/Color One Bit/Color May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 19

20 Imagery 4:2:2 Subsampling is not new Analog Color TV has a color content bandwidth of 1.5 MHz, BW info is 3.0 Mhz. Human physiology More sensitive to changes in intensity than changes in color 4:4:4 Sampling Equivalent to RGB In motion imagery, 4:2:2 sampling is not discernable from 4:4:4 Bit/Pixel Benefit 8 bits/color =24 bits/ pixel 10 bits + ½ 20 bit color sample = 20 bits 16.6% savings in data with no visual motion imagery degradation May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 20

21 Imagery Subsampling Created to reduce bandwidth to transport video Takes advantage of human physiology where The eye far more sensitive to intensity changes vs. color changes In Analog Intensity (Luma) plus TWO of the color (chroma) components is used Phase modulation used to encode color is about 1.5 MHz (vs for Intensity) In Digital One color sample per luma = 4:4:4 equivalent to RGB Bits/frame = 1080*1920*24 (16 million colors) = 49,766,400 One color sample per two luma = 4:2:2, equivalent to broadcast analog TV Bits/Frame = 1080*1920*(10 -bit luma + 10-bit Red or Blue sample) = 41,472,000 (16% reduction) Side effect: Saves storage capacity requirements as well No effect on human perception of the imagery EXCEPT there is a color alias at edges when the edge is between luma samples with shared color sample. May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 21

22 Imagery Subsampling Effects May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 22

23 Imagery Color Alias when intensity changes on a constant color Y-b-r = g Y-Δ b-r = g Δ Changes the ratio of RGB which changes the color, not just the brightness Good Side effect for overlay Natural surround May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 23

24 Pixels & Bit Count & Data Rate Key Points to Remember Each pixel = a Y (luma) sample Vertical blanking space adds lines; e.g. 45 in lines, plus blanking = 1125 lines/frame Horizontal blanking space adds samples; e.g. 280 Y samples per line in 1080/ visible pixels Y H blanking samples = 2200 pixels/line Each pixel in 4:2:2 sampling is 20 bits deep 10 bits of luma (Y) and 1 of the color components (Cr or 10 bits = 20 bits Interlace Video two types Segmented Frame delivers ½ the image in one field and the other half in a second field True Interlace One image odd lines, a second image (16 ms later) using even lines. Progressive Video delivers a complete frame per scan Frame rate = field rate (e.g. 1080p/60) Frames may repeated at field rate (e.g. 1080p/30) October 26, 2011 May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 24

25 Pixels & Bit Count, Data Rates, Resolutions & Specs SDTV HDTV Interlaced (I) Progressive (P) Active lines per frame Total lines per frame Active Luma samples per Line Luma Samples in Blanking Area Total Luma Samples (Pixels) Sampling Frequency (MHz) Aspect Ratio Frame Rate (Hz) SMPTE SDI Bit Format Digitizing Specification Bit Rate (MBit/Sec) SDTV I : M ITU-R BT HDTV P :9 60 or 60/ M ITU-R BT HDTV (PAL) P : M ITU-R BT HDTV I :9 30 or 30/ M ITU-R BT HDTV (PAL) I : M ITU-R BT HDTV P : M ITU-R BT May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 25 25

26 Transport & Storage May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 26

27 Metadata Storage Metrics SMPTE 296; 720 Frame Parameters 3.1 MB/frame 1 minute of 720p/60 video = 11,160 MB SMPTE 274; 1080 Frame parameters Data Rate Metrics 6.2 MB/frame 1 minute of 1080p/30 video =11,160 MB 1 minute of 1080i/60 video =11,160 MB 1 minute of 1080p/60 video =22,360 MB 720p/60 = Gbits/sec 1080i/60 =1.485 Gbits/sec 1080p/30=1.485 Gbits/sec 1080p/60=2.970 Gbits/sec May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 27

28 Transport Issue How do you get raw video (SDI) from source to destination? Analog NTSC any channel with a 6 MHz bandwidth will work SD-SDI Requires a channel capable of passing 270 M Bit data rate (1.5 GHz for 7 th harmonic) HD-SDI Requires 1.5 Gbit channel for 720p/1080i and 3 GHz for 1080p (10 GHz channel) Choices Direct Connect (copper) SMPTE Specs HD-SDI be capable of operating to 100 meters of 75 Ω Coax (e.g Belden 1694A) ; LOW LOSS, tight tolerance These do require line equalizers and drivers; Reclocking is generally needed to properly decode Short runs can use Cat 6A/Cat 7 copper cable for short runs (10 m); CAT 5 and standard CAT 6 will not work May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 28

29 Transport Issue Choices (cont.) Ethernet Fiber Radio At 1G Ethernet, ED-SDI can work At 1 G Ethernet, 720p/1080i will not work 10G Fiber Only full duplex only 100G Fiber Only, full duplex, still evolving 10GBASE-ER single-mode fiber supports 10.3 Gbit/sec up to Km Next level down -LR can support this rate up to 220 meters Where would the band and bandwidth exist? Transfer of Video as Data over GigE No less than 0.07 seconds/1080 frame; 1 minute of 60 FPS video = 5 minutes to download Requires use of full capacity of GigE channel Assumes no lost/corrupted packets May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 29

30 Storage Issue 1 Hour of 1080p/60 = terabytes of storage Cinema Industry captures 100s of petabytes per day While cost/mb is going down, the real estate, power and maintenance is huge May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 30

31 Compression Tool May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 31

32 Compression Tool Compression would never exist if infinite bandwidth and zero cost storage existed Compression is a tool to reduce data rate & storage needs Transport Alternative to whole new infrastructures Typical Compression Ratios that maintain excellent image quality H.263 and MPEG-2 ; 30:1 MJPG 2000; 20:1 to 40:1 H.264/MEG-4 part 10; 50:1, 100:1 Compression Issues Interframe prediction (MPEG) vs. frame-to-frame image compression (M-JPG) MPEG is motion sensitive M-JPG can generate rings at the harsh image edges Trade off between image quality and frame rate/frame dropping Latency Storage Gigabytes/clip become megabytes/clip May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 32

33 Compression Tool Compression destroys pixels Lossless preserves all of the essence Lossy loses varying degrees of the essence MPEG, H.264, H.265 all divide each image into a mosaic Macro-pixel 16x16 pixels Change in Color > Change in Color > Average Intensity 32 number values in a matrix of values > 256 values replaced by 32 = 8:1 May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 33

34 Compression Tool Differential Frames Reference Frame All values in the matrix Differential Frames Duplicate Matrix elements replaced with a reference to its duplicate in the reference frame Group of Frames Can achieve 1000:1 average compression or more Reference Frame All values in the matrix May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 34

35 Compression Tool Reference Frame Interframe 1 Options: Set the number of frames in a group, image quality/frame rate preference More frames, higher compression ratio More frames, the more buffering needed to reconstruct Motion prediction reduces aliasing Set data rate Group of Frames Closes loop on data rate; auto adjusts GOP, bit depth mosaic dimensions, motion comp, frame rate as needed Set Frame Rate, Image Quality all have different effects Interframe n May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 35

36 Compression Tool Compressor is Closed Loop A FIFO equivalent is used to monitor transfer rate into the transport channel FIFO content grows = more compression needed FIFO content decreases = less compression needed FIFO can t empty (not data to send) FIFO can t overflow (lost frames) Initial buffering required to pre-load the FIFO Consequence of FIFO underflow The video chain must resync, delays could be many frames Consequence of FIFO overflow Dropped frames Compressor Options Increase the GOP Increase the macro-pixel dimension Reduce the sampling Reduce bit depth May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 36

37 Compression Tool Consequences of Increasing the GOP A corrupted reference frame can lose more differential frames Dropped frames, sync loss More pre-buffer is needed, more decoder buffering is needed Consequences of Increasing the macro-pixel dimension Fewer macro-pixels per image; loss of resolution Consequences of Reducing the sampling Compressors can alter sampling to 4:2:0 Loss of detail and increased color aliasing Consequences of Reducing the bit depth Loss of detail May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 37

38 Compression Tool Compressor is Closed Loop A FIFO equivalent is used to monitor transfer rate into the transport channel FIFO content grows = more compression needed FIFO content decreases = less compression needed FIFO can t empty (not data to send) FIFO can t overflow (lost frames) Initial buffering required to pre-load the FIFO Consequence of FIFO underflow The video chain must resync, delays could be many frames Consequence of FIFO overflow Dropped frames Compressor Options Increase the GOP Increase the macro-pixel dimension Reduce the sampling Reduce bit depth May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 38

39 Transport Compression Tool MJPEG Every frame independently compressed Good Bad Less buffering to reconstruct (less end-to-end latency than MPEG) Each image is independently reconstructable Wavelet exhibits ringing at sharp edges Compression ratio limited Complex but static scenes don t compress very well no differential frames used to take advantage of this May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 39

40 Compression Tool Compressor is Closed Loop A FIFO equivalent is used to monitor transfer rate into the transport channel FIFO content grows = more compression needed FIFO content decreases = less compression needed FIFO can t empty (not data to send) FIFO can t overflow (lost frames) Initial buffering required to pre-load the FIFO Consequence of FIFO underflow The video chain must resync, delays could be many frames Consequence of FIFO overflow Dropped frames Compressor Options Increase the GOP Increase the macro-pixel dimension Reduce the sampling Reduce bit depth May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 40

41 Compression Tool Compressor is Closed Loop FIFO Driven just as MPEG Compressor Options Increase the macro-pixel dimension Reduce the sampling Reduce bit depth Each frame may only be compressed 8:1 to 16:1 without loss Wavelet is lossy but can only achieve 30:1 to 40:1 compassion with good image quality Frame to frame compression varies depending on the complexity of the scene Without the use of differential frames, no further average compression is available Benefits of no use of differential frames No lost frames with corruption of a reference frame Each frame may be rendered independently Less pre-buffering is required at the destination to being rendering video May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 41

42 Compression Tool Latency decode (sdi-image stream) + compression + xmit latency + buffer time + decompress + decode for display SDI Source Decode Compress xmit Display Decompress Buffer Receive Compression Many factors including image content, motion between frames, hardware speed Buffer Time Decompression requires a complete data set and enough buffered data to ensure every frame is reconstructed at the full expected frame rate 1300 ms to complete a 60 frame 100 MB MPEG compression, 1080p/60 (2 reference frames) 1300 ms to complete 60 frame 100 MB 500:1 compression; 720p/ ms to complete a 2 frame buffer 40:1 compression May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 42

43 Transport Compression Tool SMPTE Standard Video Type Example Formats Bit Rates (Mbits/s) Bit Rates MJP2000 Bit Rates MPEG-2 30:1 Bit Rates MJP2000 Bit Rates H Stream Video over Ethernet 259M SD-SDI 480i, 576i Base T 344M ED-SDI 480p, 576p Base T 292M HD-SDI 720p, 1080i Base T 424M 3G-SDI 1080p Base T SDI & Ethernet SMPTE :2012 STD for SDI over Internet Protocol As bandwidth continues to increase, SDI over IP will become practical Need to compress, at least lossy compression will dissipate. Specs are being developed for clean switch points and other work flow issues Good survey of the subject, SMPTE Motion Imaging Journal, March 2014 May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 43

44 Timing May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 44

45 Timing Analog Video 6 MHz bandwidth Can be transported in native form over long distances Timing compensation is predictable and could be calibrated to microsecond resolutions Timestamps can be anywhere in the transport stream and be accurate to a few microseconds. HD Video Needs a bandwidth > 6 GHz for reliable transport Raw video is data Raw video may only be transported over coax 300 feet; packetized over 10G Ethernet Compression is needed to transport over general networks and radio links Introduces Latency Latency is scene, network, codec hardware and frame rate dependent Latency is a variable with average values as high as 10 seconds Latency introduces the need to timestamp at the video source May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 45

46 Timing Timestamping at the source Overlay Compression can blur fractional seconds Metadata Great solution MISB adopted this; an example = Microsecond Timestamp Many encoders strip VANC space May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 46

47 Metadata May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 47

48 Metadata Metadata is not new Analog video has limited metadata in vertical blanking A few lines are used each holding perhaps bytes Edge encoding is another technique used mostly by the Ranges The standard puts 1 bit per line for about 250 lines/field (30 bytes/field) Vertical Blanking Space A few lines above the visible image Contains VITC Closed Captioning Teletext Billing data Copy protection & V-chip data 240 bytes per frame Edge Encoding (non-broadcast) Steals a bit of each video scan line Supports 1 bits/line 60 bytes per frame May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 48

49 Metadata Analog Data Spaces Vertical Blanking Space A few lines above the visible image Contains VITC Closed Captioning Teletext Billing data Copy protection & V-chip data 240 bytes per frame Edge Encoding (non-broadcast) Steals a bit of each video scan line Supports 1 bits/line 60 bytes per frame May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 49

50 Metadata HD-SDI Vertical Ancillary (VANC) Space samples per line ( vertical interval ) lines Totals 30K 76K bytes per frame Horizontal Ancillary (HANC) Space samples per line ( left edge but out of view) lines Totals up to 273K -313K bytes per frame May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 50

51 Metadata HD Data Spaces Interlaced (I) Progressive (P) Active lines per frame Total lines per frame V Blanking Are Active Luma samples per Line Luma Samples in H Blanking Area Total Luma Samples (Pixels) Aspect Ratio Frame Rate (Hz) SMPTE SDI Bit Format Digitizing Specification Bit Rate (MBit/Sec) HDTV HDTV P :9 HDTV (PAL) 60 or 60/ M P : M HDTV I :9 HDTV (PAL) 30 or 30/ M I : M HDTV P : M ITU-R BT.709 ITU-R BT.709 ITU-R BT.709 ITU-R BT.709 ITU-R BT VANC data space HANC data space May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 51

52 Metadata Metadata Spaces are Designed into HD 720 Frame Parameters VANC Space 25 Lines * 1280 samples = 32K Cinema & broadcast generally only use lines for scene switch & closed captioning HANC Space 370 Samples x 750 EAV Line Count CRC >2000 data items Example Packets 16 channel audio snippets Film Codes Payload data Workflow data Copyright data V-chip data Billing data Logging Info SAV EAV Buffer Space Lines HANC 1280 luma samples 1650 luma samples May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 52

53 Metadata 1080 Frame Parameters VANC Space 40 Lines * 1920 samples = 76.8K Cinema & broadcast generally only use lines for scene switch & closed captioning HANC)Space 280 samples * 1125 EAV Line Count CRC >2000 data items Example Packets 16 channel audio snippets Film Codes Payload data Workflow data Copyright data V-chip data Billing data Logging Info SAV EAV Buffer Space Lines HANC 1920 luma samples 2200 luma samples May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 53

54 Metadata Uses of Horizontal Ancillary (HANC) space HANC space used by broadcast & Cinema As many as 3000 data types could be present Audio (AES); 16 Channels Payload (format, frame rate, etc.) Advertisers, FCC logging, etc. Uses of Vertical Ancillary (VANC) Space VANC space is not used much by broadcast Line 14 is scene switch point Closed Captioning (mostly line 15) GOV uses it MISB has >900 data types defined All are KLV type 02 Anyone can use the KLV Structure Form valid SMPTE 291M packet Insert desired data Detect Packet Extract Data May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 54

55 Metadata Many Metadata Types Defined by SMPTE Dictionary Example Keys 16 Channels of sound bytes, source data, airing time, editing workflow SMPTE time code SAP, film codes on and on Data content Format Metadata Elements Dictionary RP210 Version 13 of this registry contains more than 3000 data types. May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 55

56 Metadata Why is VANC Metadata Important Sparsely used Can collect data permanently synchronized to the imagery Can collect data without burning it in to the imagery Can display some or all of the data later Can extract data from video in synchronism with the imagery Data never interferes with imagery Overlay never compromised by compression artifacts Overlay color can be changed during display time Overlay position can be changed during display time MISB has many VANC space keys reserved May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 56

57 Metadata KLV VANC Pack is a SMPTE Structure SMPTE Standard 291M type 02 Wrappers (ADF signature bytes) Type identifiers (DID and SSID) Length (DC) MISB uses the Type 02 KLV SMPTE Structure Key, Length Value Key = data type identifier (group of variables) Length = number of bytes of the value Value = data itself Size of KLV = Data Count (1-255 Bytes) Message ID (1 byte) Program Segment Counter (2 bytes) Key (16 bytes) Length (1 byte) Value ( L number of bytes); any data UDW May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 57

58 Metadata Value Element is Memory Space Bytes may be assigned to ASCII Binary blocks Values: integers, floating point Value The Key Number identifies the organization of the memory Memory Location Bytes Memory Location Memory Location Memory Location Memory Location May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 58

59 Metadata How many KLV packs are needed? Data have a resolution of 24 bits = 3 bytes [59 parts/billion or 59*E-9] More than 74 data items of 24-bit resolution can fit in a single KLV pack Example: ITS sample Instrumentation Pack 41 data items Pointing Angles up to 24 bit resolution Ranges as SP values of ± E±16 64 bit time stamps Five 10 ASCII character fields 170 Bytes TOTAL! Test ID Encoding DAS Time Image mode Geodetic Datum Integration time Run Number Trigger time Classification Time offset Temp Timestamp mode Pressure Frame Rate humidity Lens ID Object(target) ID wind speed Zoom factor Tsens AZ direction Units Tsens EL Mount ID Focus setting Tsens Range Camera ID azimuth Tx Camera placement elevation Ty H resolution Mount angles time Tz V Resoluiton Range Tspeed Image bit depth Range timestamp T Heading Encoding Object(target) ID Ttemp Image mode Tsens AZ May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 59

60 Metadata How do you use KLV packs MISB Fixed keys No general purpose software available Most keys have specialized purposes Private Keys SMPTE 291 compliant (MISB foundation) Up to 2 packs can be inserted per frame After the MISB Microsecond Timestamp (line 9 of each frame) KLV Tool Kit Free with KLV upgrade to any inserter XL template to design the memory space (235 bytes of V) GUI to Program ITS Inserters GUI to simulate & test key designs May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 60

61 Metadata Non-Registered Keys Custom Uses Special purpose designs Fuse your data sampled at the frame rate to each image Must conform to SMPTE 291M Type 02 structure Can pass through all commercial equipment without harm Using equipment capable of recognizing customs keys can: Pass unobstructed, unaltered imagery while carrying image relevant data Extract previously inserted custom key data Overlay Output customer data to files May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 61

62 Metadata MISB used the type 02 KLV SMPTE structure > 900 register keys MISB Standard 0807 lists the registered private keys All MISB keys are 6.0E.2B.34.xx.xx.xx.xx.0E.0y.xx.xx.xx.xx.xx.xx Y=01 or 02 or 03 Most Derived for UAV uses Structured to result in continuous pulse streams for PCM systems Microsecond Time Stamp Found in SMPTE RP210, and MISB 0807 Key = 06.0E.2B E Must be start on 1 st sample after SAV of Line 9 Status Value = 1 Byte; Locked/unlocked source, valid, etc. Time Value = 8 Bytes; UNIX Epoch > µsec since Jan 1, 1970 May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 62

63 Custom Keys and UDWs Design a Private Key Create a key number (16 byte;32 HEX digits) Name Fields Select a data type Signed/unsigned integers Scaled singed/unsigned integers Single and Double Precision Floating Point Numbers ASCII strings Binary blocks of data Locate data items and allocate space Offset byte and number of bytes Define Scale & Decimal Point location Key (HEX) 57414A19360A00017F 44A0904F 013AAB 0 OK Length (Bytes) 200 Est DisplayProcess Time Field Ref Name Field Start Input Len Pad Format Number Byte LEN Range Tes t ID AS C II DAS Time Binary Geodetic Datum Binary R un Number Binary C las s ification AS C II Temp UI MAX Pressure UI MAX h idit UI MAX bytes unsigned = 16,777,216 full-scale, scaled to 360 yields a resolution of 21*10-6 Specify 1,2 or 3 decimal places! Field Ref Name Pad Field Start Len Full Scale Decimal Before Format Input LEN Number Byte Range Value Places This Byte Display Model Focus setting UI MAX azimuth UI MAX elevation SI MAX / May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 63

64 Custom Keys and UDWs Assign a Key number Design the Value Key template calculates the length UDW May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 64

65 KLV = Game Change ITS Insertion Engine w/klv Option Adds up to two full KLV packs after MISB Timestamp Up to 470 bytes of YOUR data per frame; Data rate to 28.2KB/sec Line 9 SAV Pack 0 Microsecond Timestamp Option Pack 1 Data block Option Pack 2 Data block Can turn off Fixed MISB Format Always first pack Can turn on or off Can be any data from bytes Optional basic decoder ; ASCII, Integer, decimal Can turn on or off Can be any data from bytes Optional basic decoder ; ASCII, Integer, decimal May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 65

66 ITS KLV Tool Kit Use a MISB key or Create with the ITS KeyTemplate Excel worksheet Export design to CSV Distribute design to all stakeholders Build a 16 byte key Key (HEX) 57414A19360A00017F 44A0904F 013AAB 0 OK Length (Bytes) 200 Est DisplayProcess Time Create Fields & Assign Data Types Field Start Input Len Field Ref Name Pad Format Number Byte LEN Range Plan what fields would be displayed Resolution Time Qualified Plan to Budget Length Show (us) Y 20 8 Y 20 1 Y 20 1 Y Y N N N N Tes t ID AS C II DAS Time Binary Geodetic Datum Binary R un Number Binary C las s ification AS C II Temp UI MAX Pressure UI MAX h idit UI MAX 1 14 Choose how fields would be displayed Field Ref Name Pad Field Start Len Full Scale Decimal Before Format Input LEN Number Byte Range Value Places This Byte Display Model Focus setting UI MAX azimuth UI MAX elevation SI MAX / May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 66

67 ITS KLV Tool Kit Data Types Built Into KeyTemplate Types used to parse incoming data in monitor mode Types used to parse data when decoded from KLV packs Types used by Key Test to form data blocks Types used by Key Read to display extracted KLV data Pack Definable Data Types Data Type Max Bytes that May be Assigned Description Binary 235 ASCII 235 UI-NZ 4 UI-WZ 4 SI-NZ 4 SI-WZ 4 UI-MAX 4 SI-MAX 4 SP 4 DP 8 Binary pass-through format, L=number of bytes ASCII (printable characters only), field length = number of bytes L. unsigned integer, no leading zeros, field length = number of digits of highest value (e.g. 1 byte = 0 to 255, 2 bytes = 0 to 65535, etc.; max = 4 bytes), right justified. ;±N optional to shift decimal. unsigned integer, with leading zeros, field length = number of digits of highest value (e.g. 1 byte = 000 to 255, 2 bytes = to 65535, etc.; max = 4 bytes). ;±N optional to shift decimal. signed integer, no leading zeros, field length = number of digits of highest value (e.g. 1 byte = ±0 to ±127, 2 bytes = ±0 to ±32767, etc. where sign immediately precedes the first number, max = 4 bytes), right justified. ;±N optional to shift decimal. signed integer, with leading zeros, field length = number of digits of highest value (e.g. 1 byte = ±000 to ±127, 2 bytes = ±00000 to ±32767, etc.). ;±N optional to shift decimal. unsigned integer maximum scaled, no leading zeros, field length = number of bits of resolution (e.g. 1 byte = 8 bits, 2 bytes = 16 bits, etc.; max = 4 bytes), right justified. ;N;MMM required. N= number of fractional digits to display. MMM=scale factor. signed integer maximum scaled, no leading zeros, field length = number of bits of resolution (e.g. 1 byte = 8 bits, 2 bytes = 16 bits, etc.; max = 4 bytes), right justified. ;N;MMM required. N= number of fractional digits to display. MMM=scale factor. Binary 32 single precision FP; fixed byte length (4) number range ± E±16 Binary 64 double precision FP; fixed byte length (8); number range ± E±16 May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 67

68 ITS KLV Tool Kit Inserter internal fields (IF) Leave locations (offset/byte lengths) in key design for them Link them to your key with the Configuration Utility Inserters will place bytes where specified for each field Key can be parsed and data will be displayed and extracted Field Name L atitude Longitude Height/Altitude MG R S Time Zone Offset Time Sync S ource Video Format Video Frame R ate Measured Latency IP Address Camera Sync Delay Timestamp E vent Timestamp offs et KTM AZ KTM EL KTM Range KTM Range Units ITS Inserter Internal Field Specifications Data Type Byte Number Decimal Availability Max Value Length Range places 6980G HD 6055C ng HD 6055C nhd IF # S I MAX 3 ± Y Y N 1 S I MAX 3 ± Y Y N 2 SI NZ/SI WZ 3 ±8,388,608 NA 0 Y Y N 3 AS C II 11 NA NA Y Y N 4 SI NZ/SI WZ 2 ±12.00 NA 2 Y Y N 5 AS C II 1 NA NA Y Y Y 6 AS C II 5 NA NA Y Y Y 7 UI NZ/UI WZ NA 2 Y Y Y 8 S I NZ/S I WZ 4 ± NA Y Y Y 9 AS C II 21 NA NA Y Y Y 10 UI NZ/UI WZ NA N Y N 11 UI NZ/UI WZ 1 0,1,2 NA N Y N 12 SI NZ/SI WZ 2 ±16,000 NA N Y N 13 S I MAX 3 ± N Y N 14 S I MAX 3 ± N Y N 15 UI NZ/UI WZ NA N Y N 16 AS C II 1 NA NA N Y N 17 degrees /10000 degrees /10000 Meters LLL######## ±##.## (Hrs). (1/4 "I"=IR IG, "G "=GPS e.g. "0720P ", "108 ###.##; 23.97, 24,25, e.g. ±####### µse Requires incoming M metadata, else 0000 IP addres s & P ort "###.###.###.###; ##### (µsec) Only w/c S y option ( 1=TTL camera s y, 2=The delayed s Only w/c S option ±##### (µsec) Only w/cs Option Requires K T optio Requires K T optio Requires K T optio Requires KT optio 2,4,6 channel units May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 68

69 Key Development, Debug & Troubleshooting Tools Key Read Select the previously saved data file The header will show Key length What video channel the data was extracted from What pack number of the channel The Time Flag (1=locked, 0= not locked) The key (16 byte value) in hex The Time Status microseconds Each field of the Key Design will be shown along with the data associated with the frame from which it came Indicated by the microsecond value Visual Basic Source Code supplied at not cost Global Const MAX_PACK_FIELD = 64 Type BROWSE_INFO hwndowner As Long pidlroot As Long pszdisplayname As Long lpsztitle As Long ulflags As Long lpfncallback As Long lparam As Long iimage As Long End Type Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" (ByRef lpdest As Any, ByRef lpsrc As Any, ByVal ByteCount As Long) Declare Function SHBrowseForFolder Lib "shell32" (lpbi As BROWSE_INFO) As Long Declare Function SHGetPathFromIDList Lib "shell32" (ByVal pidlist As Long, ByVal lpbuffer As String) As Long Declare Function lstrcat Lib "kernel32" Alias "lstrcata" (ByVal lpstring1 As String, ByVal lpstring2 As String) As Long Global g_ifilenum As Integer Global g_ioutfilenum As Integer Global g_sinput As String Global g_sklvtoolkitpath As String Public Function IEEE32toDouble(ByVal x1 As Integer, ByVal x2 As Integer) As Double Dim nint(1) As Integer Dim ftemp As Single Dim stemp As String 'Convert 32 bit IEEE754 value to floating point nint(0) = x2.. May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 69

70 Insert into the HD video stream at the source Timestamp accurately Sample data at the frame rate See and Extract from HD video stream at the Destination Overlay some or all data Extract data to a file May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 70

71 6520 End-To-End Solutions Artel Fiber Artel Video Capture SDI Live Video Monitor LE4 SDI GUI (from Matrox SDK) GUI captures, displays, imagery and KLV metadata Your Video Source Ethernet Data Concentration for Metadata Insertion Control, Insert, Extract Webserver & GUI Download Raw video data and metadata KLV Data Concentrator GUI Metadata Uncompressed Image Your Data KeyRead May 3, 2016 Sources ITEA 20 th Test Instrumentation Workshop Sheet 71

72 VANC KLV Metadata Check List KLV packs in SDI are a game changer KLV Packs can Transport data Move cipher blocks Enable recording of clean video Maintain alignment of imagery and data Video Encoders/Decoders Must Preserve VANC end-to-end SDI Recorders must Preserve VANC at record time Restore VANC at playback time Video Archiving must preserve VANC ITS software toolkit SMPTE can support this capability Create KLV Insert your data Monitor your data Display your data Test your KLV design May 3, 2016 ITEA 20 th Test Instrumentation Workshop Sheet 72