Taming he Beas in Mankind Telecommunicaions in he 21s Cenury InerComms alked o Ecma TC32-TG22 s Convenor and Swissaudec s CEO Clemens Par abou he 21s cenury s broadcasing and communicaion means Clemens Par, CEO, Swissaudec Clemens Par pursued parallel sudies in conducing a Hochschule Mozareum in Salzburg and mahemaics a ETH Zurich. Besides his arisic projecs, e.g. as a painer, auhor, musician, presener and execuive producer for ARD, ORF and Schweizer Radio DRS, his scienific work focuses on inverse problems in audio engineering and invarian heory. Resuls have been sandardized as he world s firs 3D audio sandard ECMA-407 in June 2014. Clemens Par received he WIPO Award 2009 for his IP. He is CEO of Swissaudec (a Swiss enerprise specialized in 3D audio coding echnologies), Exper of ISO/IEC JTC1/SC29/WG11 (MPEG), and Convenor of Ecma TC32-TG22. ECMA-407, MPEG-H and ATSC are currenly shaping he fuure of UHD TV up o 8K resoluion and NHK 22.2 audio. A low-delay profile for ECMA-407, crafed by Swissaudec, however, may revoluionize omorrow s mobile elecommunicaion and eleconferencing means offering NHK 22.2 capabiliies down o 48kb/s wih a laency of one frame. Q: Broadcasing and elecommunicaions indusries have led parallel lives so far. You made a very firm echnological saemen a IMTC 20h Anniversary Forum abou fuure echnology opions in his field. Could you share your vision of 21s cenury s mulimedia scenario wih our readers? A: The very visionaries are no found in echnology, hey are primarily found in lieraure. Transgressing echnology limis of one s own ime has lead o he considerable oeuvre of Jules Verne; more criical approaches may, for insance, be found wih Aldous Huxley. Boh largely have become realiy. Unlimied visualizaion means and immersive audio are visionary imaginaions and invenions of he pas. The firs 3D broadcas via elephone lines occurred in he 19h cenury in Paris. Three-dimensional imaging, hough pained, even daes back o he 18h cenury. Swiching on a screen and communicaing like in realiy nowadays has become a muli-billion marke. Neverheless, broadcasing and elecommunicaion indusries have gone separae ways, due o hisoric reasons: birae budge for elecommunicaion would hen have compromised broadcasing qualiy. The sigma has remained despie he fac ha we have brillian echnologies a hand, which can equally fulfil boh needs: UHD TV and advanced elecommunicaion. Q: Teleconferencing rooms already go immersive. Is your saed resricion of qualiy sill valid? A: I have been invied o visi a well-known eleconferencing company s sie wih screens of several square meers. Even he room s colour was opimized for video capure, whils cheap plasic able microphones wih mono ransmission capured audio. Even in modern imes, people are no supposed o appropriaely lisen o heir fellow-beings in a realisic acousic environmen. I is all visual appearance. The faul is parly cinemaic: 3D cinema sound is overrealisic and evidenly reflecs bad ase. In silenly quoing www.inercomms.ne InerComms 1
Paul Valéry: people have aesheically grown deaf and seem o perceive no need for advanced eleconferencing audio means. We are far apar lierary science ficion where people wished o explore, learn and discourse. Social media feed his innae need oday, however, on echnologically lowes level. Telecommunicaion indusries, however, refrain from kissing his sleeping beauy: i s a ragic fairy ale! science communiy, were never applied o audio coding. So far, spaial properies had o be exraced by means of a Fourier ransform and o be ransmied as opulen side informaion. This mehod is called parameric coding. Conrarily, given an exising spaial audio signal, you may calibrae he inverse problem in such way ha spaial audio may be resriced o an, occasionally ransmied, parameer se packe. For insance, a spaial daa packe of ECMA-407 wih NHK 22.2 requires less han 100 byes and may las for several minues. Wih parameric coding, a leas 40kb/s are required in he low birae range wih parameric coding. Given equal performance of he used base audio codec, ECMA-407 may achieve equal performance a lowes biraes wih, for insance, MPEG-H, however, wih an unresriced number of oupu channels up o NHK 22.2 a all biraes. Figure 1: NHK 22.2 represens he sae-of-he-ar loudspeaker seup for 3D audio conen producion and delivery. I will be broadcased over saellie by Japanese ARIB from 2016 onwards wih AAC and is complemened by 8K HEVC video coding. Q: You have been a researcher for more han a decade in he field of mulimedia audio wih numerous discoveries, which have been normalized as he world s firs 3D audio sandard ECMA-407 in June 2014. Could you give our readers a lile overview? A: I am no very much an engineering guy (laughs). My love has always been music, ars, pure science and is unforeseen applicaion for he benefi of human welfare. As a professionally rained musician I have a naural affiniy for beauiful soundscapes. I sared as a painer wih my firs exhibiion in Paris in 1989/90. Visual ars exend our imaginaion above he level of he curren calamiy show (as modern film producion was called by Arnold Schoenberg s pupil David Raksin). My professor Rudolf E. Kálmán laid he foundaions for my early ineres in sysems heory, which, hough having very eminen applicaions, is nohing bu pure hough. My discoveries are hreefold: I solved he firs inverse problem in audio in a raher playful and unplanned way in 2002 when rying o find a mahemaical subsiue for a sereo microphone for my privae sudio. Inverse problems were unknown in hese imes in his field and, as Michael Dickreier in his Handbuch der Tonsudioechnik saed, could no even be solved. However, Michael A. Gerzon, a very eminen mahemaician a Oxford Universiy, already had worked ino his direcion. My final soluion eliminaes frequency as a degree of freedom and says wihin he realm of ime and level in providing sufficien degrees of freedom. I has become he basis of my firs key echnology paened in 2008 and he basis of inernaional sandard ECMA- 407. Models of his kind, now called inverse coding in he Figure 2: ECMA-407 encoder. The normaive signal analysis, which may be based on invarian heory and hen works in real-ime, provides he lowes spaial biraes ever achieved. Since 2009, I have been in consan exchange wih Rudolf E. Kálmán who awakened my ineres in invarian heory, hen a discipline of pure mahemaics. David Hilber is well known for Hilber space or Hilber ransform. I, however, is no common knowledge ha his eminen German mahemaician from his hesis onwards made his foremos discoveries in invarian heory. Invarians are coefficien funcions, which, like cerain baceria, essenially survive ransforms and, as Hilber proved in 1893, luckily form a field - arousing he scienific suspicion ha such invarians exised wih Gaussian (random) processes! However, hese algebraic objecs never were isolaed. In 1903 Grace and Young published a book on invarians, which capured my aenion in 2010 wih respec o apolariy behaviour (a sae when invarians vanish). Apolariy proved o be key o solve his problem. This soluion has been paened in 2010 for Gaussian signals. Invarians hen became par of applied mahemaics. Our invarian-driven inverse encoder wihin he framework of ECMA-407 makes use of hese resuls. Invarians are much faser han is saisical analysis; hey require very lile known daa. This is why our ECMA-407 implemenaions work in real-ime and make equally way for UHD TV and elecommunicaion indusries. 2 InerComms www.inercomms.ne
Figure 3: A roaing verical plain is he answer o a one-cenuryold problem: is associaed invarians represen a sound real-ime alernaive o saisical analysis. Inverse coding works in ime domain and requires almos no compuaional effor. My hird scienific goal was o find an equivalen o his echnology in he Fourier field, wih lowes possible compuaional complexiy and minimal laency. My research was likewise successful in his field, which means ha up o he double number of oupu channels can be creaed from a downmix. Conrarily o parameric coding in frequency domain, his mehod requires no side informaion a all, which makes i fully conforman wih inernaional sandard ECMA-407. Compuaion essenially is relaed o going from ime domain o frequency domain and back. When using his echnology, laency can be resriced o one frame, as wih mos speech coders. Figure 4: ECMA-407 decoder. Is normaive S5 upmix performs in realime, according o he conveyed inverse coding parameer daa. Q: You are he Convenor of Ecma TC32-TG22, and likewise an MPEG Audio Exper since January 2012, primarily working in he field of UHD TV audio ranspor. In Augus 2014, your company launched he firs ECMA- 507 UHD TV es carrier in co-operaion wih France Télévisions and SES Asra. Wha is your personal forecas regarding UHD TV ransmission? A: MPEG-H 3D audio has been primarily been driven by Japanese broadcaser NHK. We already know 8K recording and HEVC coding equipmen complemened wih NHK 22.2 audio and AAC, afer NHK s years long scienific promoion of MPEG-H 3D audio under Kimio Hamasaki. Conrarily, MPEG-H and ATSC boh know pre-defined proprieary reference qualiy encoders, which, unlike ECMA-407, are no complian wih AAC or HE-AAC. Japanese ARIB has announced is firs UHD TV es saellie broadcass for he Olympic games. I may be expeced ha UHD TV ses, which are capable o render 3D audio, will be launched by he Japanese and Souh Korean indusry abou his ime. Rendering NHK 22.2 is far from being rivial, paricularly on loudspeakers, as wavefield synhesis or cross-alk cancellaion have o ake place inside he device iself no average consumer will ever se up a complicaed, properly measured NHK 22.2 lisening room! Virualizaion means like wavefield synhesis and crossalk cancellaion boh work decenly; loudspeakers can hen be direcly mouned wihin he UHD TV s frame. NHK demonsraed such a sysem nex o our booh in IBC 2014 s Fuure Zone. Real NHK 22.2 loudspeaker reproducion only happens in he sudio and in he laboraory. MPEG-H may be expeced o become inernaional sandard in 2015, while ATSC sandardizaion is sill on is way. Dolby, DTS and Barco have already launched 3D sound in cinema and have sirred public awareness of his subjec. Broadcasers currenly explore 3D audio producion and ranspor. However, neiher he American nor he European markes are prepared for he broad launching of 3D broadcass. The same siuaion is valid for Souh Korea, anoher echnology driving marke. Given he fac ha UHD TV has been announced by ARIB o be broadcased regularly from 2020 onwards, UHD TV may expeced o be launched firs in Souh Korea, followed by Europe and by Norh America. Swissaudec s curren focus wih is ECMA-407 implemenaions, however, lies on webcass and elecommunicaions, suppored by is auomaic 2D o 3D upmix as a subsiue o genuine conen producion. (Similar echnologies are likewise applied in 2D wih HD TV for he auomaic conversion of sereo o 5.1 Surround.) Invarian heory plays a key role in 2D o 3D conversion; i allows us o deermine he mos suiable spaial model for he upmix in real-ime up o NHK 22.2. UHD TV is a muli-billion marke, which sill requires susainable invesmens from he side of indusry. We may anicipae his huge leap in mulimedia conen delivery by caering 3D audio conen up o NHK 22.2 over he Inerne. Q: There is no common awareness of 3D audio rendering echniques. Could you give a shor explanaion how 3D audio may be consumed on a mobile device? A: Primary consumpion is via binaural means. Wih Smarphones, ables and compuers, we have experienced he headphone boom and he rising of brands like Beas or Sennheiser. Binaural 3D audio was already commercially launched in he sevenies wih lile commercial success he Walkman simply had no ye been invened! Markes are evidenly ou of phase : broad 3D audio consumpion could have been made possible weny years earlier! www.inercomms.ne InerComms 3
Binaural 3D audio echnologies are echnologically nohing exciing. Every mammal brain is used o adap is percepion of localisaion o differences in ime, ampliude and frequency caused by is head s exerior anaomy. If you record sound wih a dummy head you anicipae wha my friend Günher Theile calls inverse Filerung: when such signals are played back on headphones he said differences in ime, ampliude and frequency auomaically represen localisaion cues o he human brain. Ineresingly enough, such arificial head signals sound very bad when being played back on sereo loudspeakers. Inverse Filerung in he brain includes an equalizaion sep for resiuing he original frequency response. No sound processor currenly equals such amazing capabiliy! In he sric sense his likewise imposes an inverse problem. As Günher Theile proved, he brain can be fooled wih respec o frequency response and is complemenary localizaion cues. If I now record sweeps in an NHK 22.2 laboraory for each loudspeaker wih a dummy head, I may subsequenly convolve each channel of a 3D audio signal wih such measuremens. All of a sudden, my headphones become an NHK 22.2 lisening environmen! This is wha 3D rendering on mobile devices is all abou. A boom in 3D audio may herefore be expeced in parallel wih UHD TV, which sirs public awareness for immersion. UHD TV unconsciously educaes o inerpre complex audio localizaion cues ogeher wih video. The same visual cues, now caered by he raher small smar device s screen, all of a sudden become virual realiy hrough rained imaginaion hey are all sirred by immersive 3D sound! Swissaudec expecs he smar device segmen, wih or wihou wearable hardware, o become he primordial marke segmen for 3D audio from 2016 onwards. Is ECMA- 407 implemenaions are in he sweespo wih respec o available bi budge from 48kb/s o 128kb/s (currenly YouTube is consumed a an average of 48kb/s wih HE-AAC in sereo). Sereo, however, is no immersive as i presens erroneous cues o he human brain: sound sources are localized in he head a common, however, fully unnaural experience. Do you expec he conducor ogeher wih his big orchesra perform Bruckner in your basal ganglions? This currenly happens in six billion devices equipped wih AAC and HE-AAC. The only sandard complian wih hese codecs is ECMA- 407 - capable of making immersive audio immediaely happen on hese six billion devices! Figure 5: Immersive 3D audio soluions on mobile devices. Swissaudec in parallel addresses he muli-billion UHD TV and elecommunicaions markes. Q: Wha is your marke prognosis regarding immersive audio for mobile devices in he near fuure? A: Mobile devices currenly face severely declining sales, see, for insance, Samsung s alering revenue forecass for Smarphones, due o marke sauraion. The consumer now has a sufficien level of funcionaliies and hird pary applicaions a hand. Unique selling proposiions on highes echnological level herefore are of growing imporance. The foremos USP evidenly is a severely enriched personal experience, which is synonymous wih immersive audio. Due o screen size consrains, visual immersion on a smar device is wishful hinking unless complemened by complex wearable hardware. Conrarily, creaing an immersive experience via headphones is easy! Q: You have primarily answered in erms of fuure UHD TV. However, wha is he direc impac of ECMA-407 for he elecommunicaion indusry? A: Seing up an audio group in an IT sandardizaion body like Ecma Inernaional, known for opical sorage or ECMAScrip (beer known as JavaScrip), would have been a bold enerprise 10 years ago. Now he world has grown smaller wih smar devices wih sufficien memory and high compuaional power and has urned ino an amazing mulimedia world where communicaion perfecly fis in! Communicaion, however, is a conservaive marke: for he reasons already saed, we sill live in he mono sone age of highly opimized speech coders, which are fully agnosic of ambien non-verbal communicaion in he broad sense. The essenial lesson is augh by shared video conen on social neworks wha people wish o express by means of images and sounds goes far beyond verbal expression runcaed by speech coders. The fuure of eleconferencing is via smar devices sharing a virual and ye real environmen over he globe wih inelligen and secure communicaion sysems. In my opinion, securiy plays a key role no company or professional organizaion wishes o share confidenial informaion o a hidden public in he wires, and he added value wih respec o virual realiy only remains a long-erm business case if securiy is properly addressed. ECMA-407 is perfecly complian wih virual realiy communicaions, as lowes delay may be achieved wih highly demanding formas like NHK 22.2. You may now ask he quesion why such a complex forma may be ineresing for elecommunicaions according o Günher Theile, 4 InerComms www.inercomms.ne
hese channels are primordially perceived as poin sources. You may hus allocae muliple speakers o heir precise posiion in space wih curren speech coders hey are all unnaurally siing in your basal ganglions! There is a precise psychoacousic reason for advanced elecommunicaion means eiher for ransporing nonverbal ambiance or for creaing a naural inerlocuory environmen wih eleconferencing. Q: Is your echnology proposal o adap 3D codecs o fuure enhanced communicaion environmens and o qui he world of speech coders? A: ECMA-407 has been designed his way mosly by public broadcasers who wish his codec o serve he needs of UHD TV. I simulaneously is a elecommunicaion means because of is low complexiy, low delay capabiliies. An ECMA-407 decoder currenly requires 33.7 MOPS per second in C++ for UHD TV and can be seamlessly swiched o low delay for elecommunicaions. I s an all-in-one ool for he elevision se or seop box, for he able and for he Smarphone. Didn you ever wish o alk o people on he screen in your living room simply via he Inerne and probably even share mulimedia conen you are jus waching? There is a desperae need for a dynamic Facebook subsiue in mulimedia which curiously enough has been never adequaely me by indusry! I have been leading a sudy on his segmen ogeher wih oher well-known manufacurers and researchers whils sandardizing ECMA-407 as an open and base-coder-agnosic concep. Coninuous one-way communicaion is never appreciaed when alernaives are a hand. My kids spend more ime in social neworks han I would ever do in my lifeime. As a grown-up you may hink of his phenomenon in erms of decadence. The ruh is ha grown-ups have only been condiioned o oneway communicaion hrough heir elevision ses, which populaed homes from he fifies onwards. I personally grew up wihou elevision se, only wih my books, my friends and my pes. I seems ha I am neiher condiioned for one-way communicaion nor for social neworks over he Inerne (laughs). broadcasing indusry perfecly knows o combine in a hree-dimensional environmen we have broadcased such sunning conen by France Télévisions over saellie! The same is valid for images. Auomaized compuer processing, based on prior arisic knowledge, creaes his virual world, Jules Verne has been dreaming of in he 19h cenury. As for audio, we know how o ranspor such an enire scene a biraes as low as 48kb/s wih ECMA-407, and HEVC likewise poins owards he fuure of video compression and ranspor. My conclusion is ha he fuure is here righ now bu ha conservaive business models in elecommunicaion impede rue innovaion. We would be mos happy o provide such experience wih ECMA-407 already in 2015! Q: You will be presen in NAB Labs Fuures Park a he 2015 NAB Show in Las Vegas wih an UHD TV premiere for ECMA-407 in Norh America. Do you see a realisic chance o reconcile he broadcasing and elecommunicaions worlds on shor erms? A: Visionaries hae shorsighed realiy. We evidenly will likewise showcase our low delay ECMA-407decoder wih UHD TV, and i is our hope ha he same number of engineers will be aware of his echnology revoluion as was he case in IBC 2014 s Fuure Zone : we welcomed more han 1 500 visiors! Some of hem were acive in elecommunicaions, and i is my hope ha hey will spread he word for a visionary ineracive mulimedia concep, which will help people around he globe o undersand heir world in a beer way. Q: Wha is your echnical conclusion for he design of a fuure elecommunicaion sysem? A: High video resoluion on a smar device is already feasible. When looking for a perfec microphone, you already find i in he MEMS world: omnidirecional, capable of beamforming and wih excellen sound qualiy. Local ransmission and opical fibres complemen such a sysem in combinaion wih a cenral processor, which is capable of creaing a soundscape from he muliple worlds of inerlocuors. Inerlocuor 1 is in he New York Ciy subway and he hammering of a passing rain can be heard in he background. Inerlocuor 2 alks o Inerlocuor 1 whils somebody is playing piano in he background. Inerlocuor 3 is lying on he beach, and all enjoy he waves background ambiance. This is communicaion on highes level, which Figure 6: ECMA-407 in IBC 2014 s presigious Fuure Zone wih an ECMA-407 saellie es carrier and ECMA-407 on mobile devices. If, for insance, such mulimedia communicaion were enabled beween Africa, Europe and Norh America, people in he firs world would have been confroned wih he rude consequences of Ebola. Human empahy is developed hrough our sensory capabiliies, which are highly resriced by curren elecommunicaion means. If such communicaion were possible beween Ukraine and he whole world, including Russia, he observed muual poliical isolaion would probably never have happened. www.inercomms.ne InerComms 5
In such conex, rue echnology progress is desirable. This has been Jules Verne s iniial vision. This has been he foremos hope and despair of Paul Valéry, he very roo of his culural pessimism. He knew abou he beas in mankind, only amed by beauy or ruh. Telecommunicaions may serve he spreading of boh: beauy and ruh! ECMA-407 is he world s firs 3D audio sandard approved in June 2014. I is primarily based on inverse problems, as formulaed by Russian-Armenian asrophysicis V. Hambardzumyan in 1929, which enjoy high populariy in fields like physics or omography wih hree scienific journals. In audio coding, inverse problems severely reduce he amoun of spaial daa, which needs o be conveyed. ECMA-407 describes a scalable mulichannel coding sysem for spaial audio daa compression, which can be applied o provide 3D audio experience wih lile overhead. Such sysem may incorporae a wide range of sae-of-hear audio codecs like AAC, HE-AAC, Ogg or USAC. By using an audio codec, which may offer encapsulaion capaciy for exernal daa, he enire ECMA-407 bisream may be carried wihin he audio coder sream wih lile overhead and mainain a compaible bi sream synax. ECMA-407 hus becomes invisible even wih highly complex 3D audio formas like NHK 22.2. ECMA-407 specifies he base S5 encoder and decoder in erms of configuraion daa, downmix, inverse coding parameer daa and upmix and provides reference and guidance on how o incorporae furher componens. See hp://www.ecma-inernaional.org/publicaions/ sandards/ecma-407.hm. 6 For more informaion please visi: www.ecma-inernaional.org/publicaions/sandards/ecma- 407.hm www.swissaudec.com InerComms www.inercomms.ne