Ultra HDTV technologies for APAN cultural and medical demonstrations

Transcription

1 Ultra HDTV technologies for APAN cultural and medical demonstrations Sven Ubik (CESNET) - talk Boncheol Goo (KAIST), Aleš Friedl, Martin Kolbe, Jiří Navrátil (CESNET), Jaroslav Hrb (HAMU), Pavel Pečiva (MNUL), Alain Baumann (Koniclab), Justin Trieger (NWS) - team APAN 38 meeting Nantou, 12 August /27

2 Outline APAN 38 transmissions Other example transmissions Technology Future directions 2/27

3 Surgery transmissions Doctors have observed surgeries for decades in operating rooms or through domes Computer vision in modern surgery systems allows camera signal to be transmitted for educational purposes 3D HD and 4K formats are now common Photo: justnurses.com 3/27

4 4K surgery from Prague to Taiwan Prague 4K Gateway 4K content Taiwan 4K Gateway SDI/HDMI konvertor 4K TV 4/27

5 4K surgery transmission for APAN 38 Surgery: Vestibular schwannoma, Radical resection via the retrosigmoid approach, Petr Vachata MD Performed in Masaryk Hospital in Usti nad Labem, CZ Pre-recorded by two 4K cameras Transmitted by 4K Gateway devices Live surgery can be done, the same technology, but more difficult to arrange timing of the surgery vs. conference session, often multiple hospitals contribute to one conference 5/27

6 Various past surgery transmissions 3D HD from da Vinci Surgical System with live commentary, Banská Bystrica, Slovakia 3D HD live eye surgeries, multiple screens, pictures from microscopes and lasers, Zlin, CZ Parallel view of robotic and laparoscopic surgeries Multiple live surgeries from different hospitals 6/27

7 Cyber performance Presented by the e-culture WG of APAN To show how performing artists in different countries and of different cultures can exchange their inventions, collaborate and create a piece of art making use modern networking and audiovisual technologies Trying to find the best solution to overcome obstacles in performing arts over networks, such as a relation between stages, actors due to network delay, eye contact, music synchronization etc. Picture: dq13 Dancing beyond TIME at Daejeon (KR) in the 36th APAN meeting 7/27

8 Cyber performance between TW, CZ, ES, US Prague Miami Barcelona Taiwan Live music in Prague (classical trio) Improvised music in Miami Dancers in Barcelona and Nantou 8/27

9 Picture design (projections) CZ Trio CZ US US Cellist Normal Background BC Dancer Black Background TW Final Screen ES TW Dancer Composed Picture Picture: B. Goo slides 9/27

10 Audio-video sync for music Test Signal 4K Recorder CZ audio JackTrip JackTrip V+A(embedded) 4K GW (b) US UltraGrid (a) HD V+A(embedded) ES 4K GW HDP2 BNC audio delayed audio(cz) MOTU mk3 audio(us) audio HDP BNC Audio Video V+A Network BNC UltraStudio Mini Recorder USB MAX HDMI DVI DVI UltraGrid UltraGrid BNC Thunderbolt HDP DVI TW video(bc) delayed video(cz) video(us) Picture: B. Goo slides 10/27

11 Remote collaboration with low-latency media sharing In medicine In music and art: playing together remote voice training remote examinations distributed concerts and performances In science and engineering: machinery models (e.g., automotive industry) architecture chemistry biology 11/27

12 Example: architecture model discussion Real-time sharing of architecture model CAVE and 3D projection 12/27

13 Example: Remote access to museum artefacts Langweil model of Prague, created in by Antonin Langweil paper model, scale 1:480, approx houses significant historical value approx. half of structures no longer exist All photographs are reproductions of a collection item administered by the City of Prague Museum, the author of the item Antonin Langweil. 14/27

14 Example: Remote access to museum artefacts Calit2, UCSD Univ. of Hawaii Interactive remote navigation through the model allows remote model study 15/27

15 Video requirements HD (1920 x 1080) 2K (2048 x 1080) UHDTV1 (3840 x 2160) 4K (4096 x 2160) UHDTV2 / Super Hi-Vision (SHV) (7680 x 4320) 8K (8192 x 4320) Resolution: HD, 2K, 4K, 8K Frames per second: 24, 25, 30, 50, 60 Multiple channels: 3D, multiple screens Uncompressed bitrate = resolution x fps x colors x color depth x channels Inputs and outputs: physical interfaces (SDI, HDMI, DVI) easier with HW screen capture, SAGE walls, CAVE immersible environments easier with SW 16/27

16 Delay requirements Format Music playing together Interactive remote work without feeling distance Remote work possible by accommodating delay in thinking Video contribution and distribution Bandwidth < approx.30 ms < approx. 150 ms RTT < approx. 500 ms RTT commonly up to several seconds Jitter: must be accumulated by receiver buffer current networks have very low jitter 17/27

17 Sound delay in music 10 ms no perceived effect 12.5ms approx. threshold of perception 15 ms comfortably acceptable for playing together 30 ms approx. limit of playing together, affects music, feels like sport event Remote reproduced sound should have similar dynamics (volume, range), space and reverberation as the locally produced sound Non-acoustic cues are important (feeling of other musicians) 18/27

18 4K Gateway Hardware based on Xilinx Virtex FPGA Developed by CESNET, manufactured by Infivision Industry standard 6G/3G/HD-SDI and 1G/10G ethernet All HW path between inputs and outputs 8x HD/2K full-duplex uncompressed 4x HD/2K or 4K60p JPEG 2000, 4:4:4, 12-bit Patented techniques for receiver rate adaptation to minimize buffering and allow flexible inter-channel synchronization Added latency from 2 ms uncompressed (can be set higher for jitter resiliency), ~3 frames for JPEG 2000, subframe JPEG Q /27

19 Ultragrid Software for Linux, Mac, Windows Originally developed by Colin Perkins, Ladan Gharai (U. of Glasgow) Enhancements by CESNET, Masaryk Univ. and others Up to 8K transmissions, uncompressed and compressed (DXT, MJPEG) Supports various capture cards (e.g., Blackmagic) and GPU cards for acceleration Most widely used software for video transmissions on a PC platform 20/27

20 Ultragrid-ext Based on libyuri library An extension with object-oriented structures for easier modules programming, code reduce, dynamic application creation and high scalability to CPUs and threads Each module typically lines Approx. 50 modules available for video input, output and transformations Application as a graph: 21/27

21 Describe graph in XML, application created dynamically <app name="test"> <variable name="debayer" default="1"/> <general> <parameter name="run_limit">-1</parameter> </general> <node class="read_pcap" name="pcap"> <parameter </node> <node class="mvtp_receiver" name="mvtp"> <parameter name="debug">2</parameter> </node>... Command-line arguments Nodes <link name="input" source="pcap:0" target="mvtp:0"/> <link name="input0" source="mvtp:0" target="conv:0"/> Links <link name="input1" source="conv:0" target="render:0"/> Load XML, create app... yuri2 4k_decklink_file_25_4k.xmlpath=apan38.mp4 audio=audio.wav 22/27

22 LoLa Software for low-latency audio and video transmissions on a PC Developed by GARR and Conservatorio Tartini Uncompressed video from analog camera + Bitflow grabber card Uncompressed audio from RME audio card Suppressing interrupt coalescing in Ethernet adapter driver Latency for audio approx. 6 ms 23/27

23 JackTrip Microphone CZ JackTrip Audio Interface USB Terminal 1 & 2./jacktrip s./jacktrip s o10 US Terminal./jacktrip c xxx.xxx... CZ IP address Terminal ES CZ IP address./jacktrip c xxx.xxx... -o10 Picture: B. Goo slides Connects inputs and outputs of audio adapters between remote PCs For contribution, adds significant latency 24/27

24 Video convertors and audio (dis-)embedders Interface convertors (e.g., SDI/HDMI), without fps conversion without frame buffer, delay 1-5 µs, suitable for playing together audio (dis-)embedders dedicated or included Format convertors with fps conversion with frame buffer, delay ~1 frame, unsuitable for playing together some have built-in sample generators useful for testing 25/27

25 Further development 4K multi-point connections as a cloud-based service MCU-like nodes in the network Lightweight low-latency compression for playing together Low compression rate, but low added latency (~1 line) session recorder for LCD walls (SAGE) 26/27

26 Thank you for your attention 27/27