Video Technologies for Next Generation Immersive Media Mauricio Alvarez-Mesa CEO, Spin Digital Berlin, Germany
Introduction Spin Digital Video Technologies GmbH (Spin Digital) develops highperformance video codecs for the next generation of ultra-high-quality video applications. Based in Berlin, working with customers and partners in market areas such as UHD-TV (8K), Large Screen Display, and Immersive Media. Dr. Mauricio Alvarez-Mesa is the CEO and co-founder of Spin Digital. He has more than 15 years of experience in the area of video codecs and high performance computing. He has been part of multiple international projects combining the media industry and the ICT sector. He holds a PhD degree in Computing Engineering from Polytechnic University of Catalonia.
Immersive Media Source: Cambridge English Dictionary
Outline 1. Immersify Project 2. Next Generation Video Technlogies 3. Selected Case Studies
Immersify Project Audiovisual Technologies for Next Generation Immersive Media October 2017 March 2020 EU H2020 Innovation Action Total funding: 2.5 m
Project Partners Poznan Supercomputing and Networking Center - PSNC (Poznan, PL) Spin Digital Video Technologies GmbH (Berlin, DE) Ars Electronica Futurelab (Linz, AT) Marché du Film Festival de Cannes (Cannes, FR) Visualization Center C (Norrköping, SE)
Project Objectives Video codecs: improve quality using advanced compression technology Multi-Display: enable immersive media for multiple display environments Interactive: provide tools for personalized and interactive nonlinear storytelling Content and Tools: promote immersive media in the creative industries
Outline 1. Immersify Project 2. Next Generation Video Technologies 3. Selected Case Studies
Next Generation Video: Beyond Resolution SDR HDR VR 360 Vide o FHD 4K 8K UHD HD Dynamic Range Color Gamut Next Generat ion Video Bit Depth 8 bit 10 bit Resolution Frame rate 30 fps 60 fps 120 fps
Quality Requirements for Next Gen VR 4K 8K 12K 16K Per eye resolution 1Kx1K 2Kx2K 3Kx3K 4Kx4K 360 2D total resolution 4Kx2K 8Kx4K 12Kx6K 16Kx8K 360 3D total resolution 4Kx4K 8Kx8K 12Kx12K 16Kx16K 8K per eye is the ideal resolution for full immersive virtual reality Palmer Lucky, co-founder of Oculus Rift 16K and 240 fps are needed for true VR Raja Koduri, head of GPU at AMD
Data Requirements for Next-gen Video Uncompressed Bitrate [Gbit/s] 900.00 800.00 700.00 600.00 500.00 400.00 300.00 200.00 100.00 0.00 UHD-4Kp60 8Kp60 8Kp120 16Kp60 16Kp240 unpacked format packed format
Video Codecs for Immersive Media HEVC Optimized decoders for high performance 8K and 16K decoding on a single PC High bitrate support Optimized encoders for high-res high-quality video HEVC with RExt (4:2:2, 4:4:4, 12-bit) 800 700 600 500 400 300 200 100 0 Compressed Bitrates - HEVC 4Kp60 8Kp60 8Kp120 16Kp60 16Kp240
Current Results and Challenges High performance HEVC decoder and media player 8Kp60 on compact PC platforms: Intel Core i9 12 cores 16Kp60 on workstations: Intel Xeon Scalable 2x24 cores High quality HEVC encoder for 8K 34% bitrate reduction compared to x265 for equal quality and encoding speed
High-end Video Processing on PC Platforms
Outline 1. Immersify Project 2. Next Generation Video Technologies 3. Immersive Content Creation
Immersive Content Creation Approaches Time-lapse photography Real-life 8K footage: 2D & 3D CGI and animation 3D laser scanning & point cloud Panoramic and 360 video
Timelapse Photography Pano LA 10K by Joe Capra 10K timelapse video of Los Angeles 10k@23.967fps, 2 min 30 sec Shot using 2 synced DSLR cameras (Canon 5DIII) side by side
Real Life 8K Footage 2D and 3D PSNC 8K Media Laboratory Modified Sony F65 cameras 8K short clips Palm house 8K Poznan racetrack Karkonosze mountains
CGI - Animation Prima Materia 8K by NOHlab stereoscopic piece taking the audience on an audiovisual journey Format is side by side stereoscopic 3D Rendered 8Kx8K @60 fps for each eye, 7 min 28 sec
3D Laser Scanning and Point Clouds PSNC Cathedral Church Faro laser scanner 2 billion points Modified CloudCompare for 16K rendering 8K rendering for video playback Spin Digital HEVC Encoder & Player 8Kp60, HEVC, 1.5 Gbit/s
High-res 360 Video Pandarama by ARRI 5 ALEXA Mini (360 x 60 ) Fraunhofer HHI OmniCam rig 12k x 8k resolution, 24 fps Spin Player full resolution playback with 360 player
Conclusions Highly immersive media is emerging Requires next generation video: 8K, 16K, high fps, high bidepth, 3D, 360,.. Massive data rates beyond UHD TV formats Immersify European consortium working on tools and content Advanced compression based on HEVC Media players for 8K and 16K Content creators are exploring multiple techniques Timelapse, 8K footage, CGI, laser scanning & point clouds, high-res 360 video
http://immersify.eu: Visit us: Spin Digital 1.F11 Intel 5.B65 This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 7620799 PSNC - Poznan Supercomputing and Networking Center Spin Digital Video Technologies GmbH Ars Electronica Linz GmbH & Co KG Marché du Film Festival de Cannes Visualization Center C