Experiments in Digital Television

Transcription

1 EUROGRAPHICS 99 / P. Brunet and R. Sopigno (Guest Editors) Volume 18 (1999), Number 3 Experiments in Digital Television Philipp Slusallek, Milton Chen, Brad Johanson Computer Siene Department, Stanford University Abstrat Television has been around for many deades now and has beome ubiquitous. Analog TV tehnology, however, whih was developed before the 1940 s, has remained essentially the same over the last 50 years. Alternative digital transmission standards have been developed reently and are in use for satellite and able transmissions. This tehnology is now being deployed for terrestrial TV as well. This high-speed infrastruture for digital broadasts of video, audio, and other data, whih will be build over the next few years, suggests that we look more losely at this tehnology and explore its use from the standpoint of omputer graphis. This paper provides an overview of the tehnology of digital television, disusses its potential and hallenges, and presents several experimental appliations whih demonstrate how omputer graphis an provide input and influene the future of digital television. These appliations inlude a virtual VCR, interative video overlays, tehnology for the leture of the future, and interative video panoramas. 1. Introdution Television (TV) has beome a signifiant part of our life over the last few deades. Available in most households, it is not only a ommodity item, whih is ontrary to a omputer easily used by anyone, but also a primary soure of entertainment and information. One of the major harateristis of TV is its broadast model of data distribution. In ontrast to the Internet, TV is mainly used for providing ontent to large audienes. On aount of this, prodution values are high and therefore ontent is expensive to reate. Broadast deisions are entralized and are only weakly influened by the viewers through feedbak on their hannel seletions. Interativity in TV is strongly limited and requires a separate bak hannel usually a telephone. All this is in strong ontrast to the Internet and the World Wide Web. The Internet provides peer-to-peer ommuniation, where anyone an author his or her own ontent with interation only limited by the available bandwidth. All this is attrating the interest of an inreasingly large part of the publi, although time spent using the Internet for entertainment and information retrieval is still tiny ompared to the number of hours spent in front of TVs. The Internet is not, however, well suited to the simultaneous distribution of large amounts of data to many plaes. Experiments with video-on-demand have mostly been failures, and the distribution of the Starr Report on the web almost brought the Internet to its knees. While we an expet new tehnologies, (suh as multi-asting) and higher bandwidth, to alleviate some of these problems, the broadast model of TV seems better suited to the delivery of large data sets to many different onsumers (suh as the morning news, stok tikers, et.) than the Internet. With the introdution of digital TV (DTV) a high bandwidth digital broadast infrastruture (20-60 Mbit/s per DTV hannel) is being set up in many ountries around the world and will beome available over the next few years. While it is designed for sending real-time video and audio, this digital infrastruture is equally well suited to other kinds of digital data transmission that are ompatible with the broadast model. In this ontext it is prudent to look into the possibilities and limitations the new medium offers to the omputer graphis ommunity and to explore where we an ontribute to its development. DTV and the broadast infrastruture is an interesting new tehnology from a omputer graphis point of view for several reasons: TV has always provided a partiular suessful form of Virtual Reality. While not tehnially immersive like a CAVE 6 (in the sense of 3D immersion), TV allows the viewers to imagine themselves in another loation by pro- Published by Blakwell Publishers, 108 Cowley Road, Oxford OX4 1JF, UK and 350 Main Street, Malden, MA 02148, USA.

2 Slusallek, Chen, Johanson / Experiments in Digital Television viding images and sounds that suggest the larger remote loation. Certainly TV has been providing immersive experienes to more people than any other VR equipment to date, mainly by providing highly realisti images and onvining story telling. We believe that omputer graphis an learn a lot from these onepts. The reent wave of image-based tehniques promises to narrow the gap between purely image-based TV and omputer graphis. Computer graphis offers many tools and tehniques for extending the model of TV from real-time and isohronous video and audio transmission to other models for reating user experienes (suh as textured 3D models, panoramas, animated sprites, et.). Wathing TV has always been a purely passive ativity (aside from hannel hopping). With the advent of DTV, new forms of interation beome possible. This is an area where omputer graphis has a long tradition and an provide valuable ontributions. These omputer graphis tehniques, however, need to be reviewed in the ontext of the mono-diretional broadast medium that is provided by DTV. TV sets and omputers will tehnologially onverge. DTV reeivers require onsiderable omputational power to reeive, deode, and display the DTV program. It seems reasonable that some of these yles an and will be used for other tasks suh as displaying web pages or running interative appliations. On the other hand, omputers are already able to at as DTV reeivers (see Setion 5.1). It remains to be seen exatly how this onvergene will proeed, but both TV tehnology and omputer graphis should be able provide signifiant ontributions in any ase. In this paper we explore some of the options for DTV from a omputer graphis perspetive. The first part of this paper provides the basi bakground of DTV. We desribe the basi tehnology and disuss urrent standards and systems. In the seond part, we present four appliations: virtual VCR, video overlays, leture of the future, and video panoramas. These appliations explore several aspets of the large design spae that DTV offers suh as surround video, interativity, non-isohronous display, enhaned information display, and mixed-rate video streams. All appliations have been developed as part of the ongoing Stanford Immersive Television Projet and are running on experimental DTV- PCs. Our main intent with this paper is to spark interest for this interesting new area of researh and to provide ideas and suggestions for further work. 2. Analog TV Tehnology The ability to ommuniate with other people over long distanes has been a desire throughout human history. Not long after Morse invented the eletri telegraph, people suggested that images ould also be transmitted eletrially. The development of television was a long struggle starting with Figure 1: John L. Baird, one of the TV pioneers, and his mehanial TV in It provides only a few san lines and ould only transmit rude outlines, but it already used the basi priniples of analog TV transmission. the first experiments around However, it was not until early this entury that inventors overame the immense tehnial problems (see Figure 1). It required a tremendous sientifi, tehnologial, and industrial effort before the first, reliably working eletroni television ameras and reeivers were built in the 1930s and the first TV standard (NTSC) was defined in Essentially, a television amera sans an image using horizontal lines, onverting the light intensities along the way into eletrial signals. In blak and white TV this signal is diretly modulated on an RF arrier after it has been augmented with synhronization information. On the reeiver side the signal is demodulated, and displayed by synhronously sanning the sreen phosphor with an eletron beam repliating the observed intensity pattern. Even for blak and white TV, ompression by interlaed sanning had to be used to squeeze the immense data stream into a narrow RF band. Interlaed sanning allows the refresh rate to be doubled to the ergonomi 60 Hz by sending only every other line per sreen update. The addition of olor to TV in the 1950 s introdued even more bandwidth problems. They were solved by signifiantly ompressing the olor signal without too muh loss in olor fidelity by exploiting limitations in human visual pereption. The ombined blak and white and olor signals still had to fit into the same small RF bandwidth of 6-8 MHz, leaving little room for any further extensions or improvements to the standard. As a result, the tehnology of the initial TV standard, while brilliant for its time, severely limited the quality of TV signals for many years to ome. The NTSC standard in partiular suffered from this tehnology limitation. Sine the 1940 s the basi tehnology of TV has remained essentially the same, although the equipments themselves have

3 Slusallek, Chen, Johanson / Experiments in Digital Television improved dramatially. Given the newness of the tehnology and the dramati ompetition and pressure under whih early TV was standardized and marketed, it is amazing that we have been able to get by with the same TV tehnology ever sine. Nonetheless, the (lak of) quality of video and audio has been a onstant nuisane and many improvements have been suggested (suh as the many forms of High-Definition TV (HDTV)). The main problem for all of them and one of the major reasons why little has hanged for TV for so long is the analog nature of TV. Any improvement to the image and sound quality would have required higher bandwidth, whih was very diffiult to arrange given the huge existing TV infrastruture. Analog HDTV systems that would offer signifiantly better image and audio quality have been proposed in many variants over the last two deades. They always suffered from the bandwidth problem, beause of the diffiulty in effiiently ompressing the analog signal. It took onsiderable time for the (often stritly analog) TV ommunity to realize that only digital proessing ould provide the neessary ompression ratios to make HDTV pratial. 3. Digital TV Tehnology A major limitation of analog TV proessing is the onversion of the video image into a serial stream by horizontal sanning. This makes it hard to perform any vertial proessing on the signal. In order to ompress a video image well, both horizontal as well as vertial proessing is required. Video ompression also relies on taking temporal oherene into aount, by omparing suessive frames and only sending the differene information. All of those omputations are greatly simplified by breaking the piture into bloks of 8 8 or pixels eah, whih an easily be done in a digital representation. For digital broadasts, the video and audio streams are separately ompressed, split into pakages of a fixed length, and then multiplexed into a single digital MPEG transport stream Before the DTV transport stream is broadasted, it is augmented with forward error orretion and is modulated on an RF arrier. Depending on the DTV and transmission system, eah DTV hannel offers a payload bandwidth of roughly 20 Mbit/s for terrestrial, 30 Mbit/s on able, and Mbit/s using satellite broadasting. This is enough for distributing at least one HDTV or 5-8 standard TV programs. In addition to audio and video data, the MPEG transport stream an also ontain other data streams. Most important is the program information stream that tells a DTV deoder The Japanese HiVision/MUSE standard atually uses a number of DSPs to perform signal proessing on the analog TV signal. whih video and audio streams form one program and should be deoded and rendered together. Auxiliary streams suh as multi-language audio, losed aption, teletext, and other arbitrary data an also be inluded in a DTV stream. The main benefits of a DTV system are: Image and Sound Quality Due to digital error protetion and reovery, images ontain no snow or other analog noise. However, in ases of signifiant ompression or paket loss, bloking artifats and ringing an our (well known from JPEG-ompressed images). Similar effets apply to sound. Higher Resolution DTV allows the transmission of higher resolution images (up to interlaed). It turns out that the inreased image quality at standard resolution (480 lines of video for NTSC) in progressive san mode (i.e. without the interlaing artifats) is already onsidered a signifiant quality improvement by most viewers 19. It remains unlear for now whih resolution is suffiient for being onsidered HDTV. One drawbak of HDTV programs is that it requires the onsumer to buy expensive high-resolution displays to take full advantage of the higher resolution. Extendibility Due to the very general MPEG transport stream, any new servies an be easily added to a DTV system without ompromising existing DTV reeivers. Also, better video and audio ompression tehnology installed on the head ends promises to transparently make more bandwidth available to other servies in the future. New developments in salable video oding even make it possible to inrementally add the missing pixels of HDTV to existing low resolution video streams Data Casting A digital TV system has the other important advantage in that essentially any data an be broadasted (data asting) video and audio being only the two most obvious examples. Examples of data asting are simple broadasting of data files, auxiliary data to a TV show (like overlays and event-driven sripts ontrolling their appearane, see Setion 5.3), and olletions of video and audio streams that together make up a new TV experiene (e.g. video panoramas, see Setion 5.5). One very simple extension of DTV would be to use any exess bandwidth (e.g. at off prime time hours), or even whole DTV hannels to broadast data suh as web pages, news, software updates, et. DTV reeivers with disks ould store or ahe some of the data depending on user preferene or subsription and make it available on demand. It seems that data asting will beome an important aspet of DTV networks, although it is yet unlear whih of these servies will be ommerially suessful. Although proprietary DTV systems have been in use for some time in speifi satellite systems (e.g. DigiCipher II), the two major DTV systems now are the European DVB (Digital Video Broadast) 8 and the Amerian ATSC (Advaned Television Systems Committee) systems 2. DTV was first introdued for satellite transmission and is only now

4 Slusallek, Chen, Johanson / Experiments in Digital Television also being deployed for terrestrial broadasting where it is supposed to replae analog TV over the next deade or so The DVB System DVB has been developed as a onsistent base standard for the DTV stream that is augmented with standards that speify hannel oding and forward error orretion suitable to the speifi harateristis of transmission hannels suh as satellite (DVB-S), able (DVB-C), and terrestrial broadasting (DVB-T). In partiular, the modulation sheme (COFDM) for terrestrial broadasting supports mobile reeption and is well suited to diffiult signal onditions (e.g. ghosting) requiring only small settop antennas. Early on the DVB onsortium inluded failities for data broadasting, interative servies, and onditional aess into their standards. For the DVB onsortium HDTV had low priority on the list of features and has only reently been added to the standard 17. The DVB onsortium plans to simulast separate standard and high-definition video streams instead of requiring eah reeiver to be able to deode and down-onvert HDTV streams. Beause onditional aess is well-defined for DVB, the same reeiver boxes an be used for free hannels as well as for pay-per-view servies by inserting a speifi deryption PCMCIA/PC-Card into the box. Many subsidized DTV reeivers in the UK are now distributed at lowost ( $300) by pay-per-view ompanies making DTV affordable to the onsumer. DVB already is the major standard for most satellite systems and many able TV systems. Terrestrial broadasting using DVB started in the UK in last November, with Sweden and Australia soon to follow. Other ountries still rely on satellite distribution of DTV, but most plan to introdue terrestrial DTV in the near future The ATSC System The Amerian ATSC system started off by onentrating on terrestrial broadasting only. Beause HDTV has been a major motivation for the introdution of DTV in the US, its distribution is atually mandated by the US government. However, stations plan on broadasting at most 3% of the daily programming in HDTV. Interestingly, the FCC adopted ATSC standard does not define a speifi video format or resolution, leaving this hoie to the TV networks and reeiver manufaturers. The onsumer, however, is left with the possibility that his new reeiver may not atually be able to deode all available programming. It still remains unlear how well DTV reeivers will be aepted by onsumers, sine their pries in the US are likely to be high, mainly sine eah box needs to be able to deode and possibly down-sample HDTV programming. It appears that some manufatures will ignore HDTV in favor of low ost reeivers. Although both standards are based on similar onepts, there are enough differenes that make DVB and ATSC inompatible: Modulation Eah system uses its own modulation and error orretion sheme for terrestrial broadasting. Program Information The data streams that desribe the available programs are ompletely inompatible, so that neither system would find the other s programs even though both understand the same MPEG transport stream and video ompression syntax. Audio Different audio enoding shemes are being used. DVB uses MPEG Layer 3, while ATSC uses the Dolby AC3 standard, although both standards allow the same 5.1 hannel, CD quality surround sound Video Although the MPEG-2 ompression sheme 10 is the same, both standards have hosen different MPEG parameters. This may render the video streams inompatible with deoders that are speifially designed for one system or the other. Due to market onstraints it urrently seems unlikely that the two standards will eventually onverge, although manufaturers might build dual standard reeivers in the future. Unfortunately, Japan seems to be determined to develop yet another DTV standard ISTV (Integrated System TV) in addition to its urrent analog HiVision/MUSE system. This may introdue yet another standard into the mix. Many TV stations and networks see DTV as an opportunity to gain new market share. The situation is partiularly problemati in the US, beause the introdution and ontinued support of an HDTV apable ATSC system has ost US TV stations billions of dollars for new equipment and distribution infrastruture. The stations are now looking for new soures of revenues to make up for this loss. Although for eah analog hannel a TV station reeived an additional free DTV hannel if it promised to transmit HDTV programming, it seems highly unlikely that a larger number hannels an over the osts of DTV. As a result, many TV networks and stations are looking into adding simple data asting to DTV and augmenting their DTV program with new user experienes through interative omponents and enhaned visual presentations. 4. DTV and Computer Graphis In the following setion we explore some of the interesting dimensions of the DTV design spae, suh as the time dimension where the presentation time an be deoupled from the transmission time, the presentation of additional on-sreen information linked to the DTV program, new user experienes through extended and semi-immersive visual presentation, and finally the interative dimension of the previously passive medium of TV. In partiular we onentrate on areas where omputer graphis offers tehniques and experiene for future DTV appliations. Conrete appliation

5 Slusallek, Chen, Johanson / Experiments in Digital Television senarios and example implementations are then desribed in Setion Time Warping A DTV reeiver with enough disk spae ould easily at as an intelligent VCR or a video buffer, that an automatially or on ommand save your favorite shows and programs to disk if you happen not to be at home. Current 18 GB disks drives an provide up to 10 hours of video depending on ompression. The strit isohronous nature of wathing TV an be eliminated by temporarily buffering the inoming video stream of a program on a hard disk, while you pik up the phone or prepare a snak. The paused program an then be ontinued from the stopped loation running off the hard disk, while the inoming video ontinues to be buffered until you have a hane to ath up by skipping the next ommerial (see also Setion 5.2). This new tehnique requires little support from a omputer graphis perspetive exept for very simple and intuitive user interfaes, but offers a basi servie for many of the higher level appliations desribed below Video Overlays and Insets There are many oasions where a viewer might want to get more information than is normally available in the video stream. A partiularly good example is sports programming, where some people would like to see more detailed information about a game or a player (biography, statistis, related events, merhandise, et.). This information an be provided in video overlays, whih an pop up automatially or on the request of a user. The user an then hoose to browse the information further, adjust its display, or dismiss it. The overlays an be animated or stati, linked to hot spots on the sreen (e.g. to where a partiular player is urrently visible), or ould be textured insets attahed to objets visible in the sene (e.g. advertisements on the floor of a basketball ourt). These latter forms of display require non-trivial image-based tehniques both from the field of omputer vision and omputer graphis to properly operate and synhronize with the program Extended Video Presentation In a traditional TV program every detail of the presentation is predefined by the TV station or network. This is a diret result of the hosen representation as a fixed, full-sreen video stream. As soon as we extend the representation of video, we offer more options for the user to ustomize the presentation aording to his requirements and preferenes. One general option that is urrently being standardized as part of MPEG-4 is to split the video program into multiple Video Objets (VOB), whih need not have retangular shape and may be partly transparent. The VOBs are transmitted separately aross the same DTV stream together with a default presentation arrangement, whih may be threedimensional. Depending on the parameters, the user may then be able to interat and rearrange this default presentation or substitute his own. Using these priniples we have implemented a video panorama, whih is desribed in Setion 5.5. Another opportunity that is offered by this framework is the adaptive oding of the different video streams, whih may have ompletely different oding and update requirements. As an example, when transmitting a leture the image of a blakboard requires very high resolution to be readable, but only ertain regions need to be update at frequent intervals (see Setion 5.4) Interativity The video presentation outlined in the previous setion already allows the user a limited form of interation with the DTV program, but more degrees of freedom are ertainly desirable. A major stumbling blok to interativity is the strit broadast model of DTV, whih does not offer an integrated bak hannel. Adding Internet aess solves this issue and offers all the usual interation apabilities. However, it is interesting to explore the available options without adding a bak hannel. In this mode, only loal interativity an be provided. This interation is limited to an environment predefined by the reator of the program. Examples are downloaded sripts that reat to ertain interations by displaying, hiding, or modifying previously downloaded media presentations and program state. An example appliation is given in Setion 5.3, where overlays an reat to user events as well as time events oming in from the transmission hannel Image-based Modeling and Rendering While traditional geometry-based approahes an be applied to DTV, image-based algorithms seem to be partiularly well suited to the task. We an often diretly use input video streams and proess them appropriately to produe new and enhaned DTV experienes as in the panorama example (see Setion 5.5). Sending a video stream of range images or downloading a layered-depth images 18 would allow a lient to perform loal interations with the DTV ontent Exeution Environment An important aspet for developing new DTV appliations is the environment that is provided by a DTV reeiver for exeuting downloaded software. The vendor neutral DAVIC onsortium is working on API interfaes for interative TV based mainly on the DVB standards, but other ompeting solutions exist as well.

6 It still remains unlear what support the exeution environment will provide to potential DTV appliations. How muh CPU power and graphis support are required and how muh an we possibly expet to see in a DTV reeiver in the future? How losely will a TV set be integrated with the omputer or will they eventually be the same server devie with multiple display aross the house? It seems that as we ome up with new servies, the TV set of the future will need to adjust and expand aordingly and may need to be periodially upgraded as is neessary with omputers today. Slusallek, Chen, Johanson / Experiments in Digital Television 5. Experiments in DTV In the following setions we desribe ongoing work on a number of experimental DTV appliations that explore the design spae of DTV. In hoosing what to persue we foused more on developing new experienes for the potential user than on diret appliability in todays DTV environment. Nonetheless, all the appliations have been implemented within an experimental DTV infrastruture and have been tested in a broadast setup Infrastruture The infrastruture for our experiments onsists of several 350 MHz Pentium-II PCs running Windows 95 with video boards that provide olor spae onversion. In addition eah system has an ATSC reeiver ard (see Figure 2), whih has been designed by INTEL s MiroComputer Researh Lab. This ard provides the basi funtionality for reeiving an ATSC modulated RF signal off the air via a standard rooftop antenna and perform hannel tuning, demodulation, and error orretion. The DTV hardware delivers the raw 19.2 Mbit/s MPEG-2 transport stream into system memory. Any further proessing is performed in software. The DTV hardware is aessed through an experimental API that shields the appliation software from the hardware details. At the lowest level this API demultiplexes the transport stream and delivers pakets of hosen audio, video, or data streams. Layered on top of this interfae are servies for deoding and/or displaying a video stream, as well as reeiving data objets (essentially files with identifiation information). Sine the hardware only provides the raw MPEG-2 transport stream, all demultiplexing and MPEG deoding is performed in software. Our systems allow us to reeive and display a standard TV video stream ( , interlaed) in real time. Deoding of HDTV streams is an area of ongoing development. An experimental ATSC transmitted set up by INTEL some 10 miles from Stanford has been used to provide the DTV streams. During development the DTV reeiver ard an simulate the reeption of a DTV stream by reading the transport stream from a loal file. Figure 2: Photo of digital/analog TV reeiver ards build by Intel s MiroComputer Researh Lab. The top board an reeive NTSC signals while the bottom board handles digital ATSC signals Virtual VCR As part of the support infrastruture we have implemented a video buffer that supports VCR-like operations suh as slow motion, pause, replay, and fast forward. The buffer ould be used in appliations suh as personal sports replays, selfpaed letures (see Setion 5.4), flexible guided tours (see Setion 5.5), or for downloading a movie during the night that you want to wath tomorrow. To implement the VCR-like interative features, a ahe manager is required that is able to buffer a large amount of data and supply it to the deoder in real-time. Sine the memory requirement an be huge, a main-memory-only buffering approah may be prohibitively ostly. The alternative is using a memory-disk integrated ahe (MEDIC) 3. Sine the per-byte disk ost is about one hundredth of the per-byte memory ost, MEDIC is eonomially attrative. MEDIC arefully alloates a limited amount of memory to ompeting tasks, i.e. to reeiving new data from network hannels, to writing data to disk as memory fills up, to reading data from disk as needed, and to holding data for deoding and playbak. Sine data is onurrently written to the disk ahe and read from the disk ahe, MEDIC must intelligently issue IOs to maximize throughput and to avoid undue onflits. With less than 8 MB of RAM our video buffer an support the interative operations disussed above without ausing any jitter. This result onfirms the theoretial study 3. Figure 3 and the aompanying video shows an appliation of video buffering for implementing a Virtual VCR.

7 Slusallek, Chen, Johanson / Experiments in Digital Television Figure 3: A sreen shot of the Virtual VCR, where the top window shows the deoded video. The bottom window shows the ontrol panel, the urrent memory and disk usage, and the reeption and deoding progress. Figure 4: A sreen shot of the video overlay projet. The feature movie is displayed in the enter of the sreen. A list of the available overlays (movie desription, main atress biography, tuxedo advertising, and ring advertising) are shown on the bottom of the sreen. The ring advertising overlay has just arrived (displayed in lower right orner). This ontext sensitive advertising is triggered by the movie sene that shows a similar ring Video Overlays The value of video overlays is readily demonstrated by the stok tikers, hannel logos, and sports sore boards that are already found in today s analog TV programs. In ontrast to analog TV, DTV allows these overlays to be omposited in the viewers homes rather than in the prodution studios, thus allowing flexibility in the storage and presentation of the ontents as well as ustomization of the viewing experiene. The goal of the overlay projet is to reate an infrastruture for experimenting with overlays as well as example appliations. Our overlays are desribed by a markup language similar to HTML. In addition to standard fields, eah objet desriptor ontains a time field. This allows the speifiation of not only where and how objets should appear but also when. The degree to whih viewers an hange the visual properties of the overlays an also be speified. For example, an overlay s position and size an be either fixed or ustomizable by the viewer. Eah overlay objet as well as the markup language is sent through a separate DTV stream. The overlay reeiver handles four tasks: reeiving objets, parsing markup language, reating to viewer input, and displaying the overlays. When a ustomizable overlay is reeived, an ion is displayed in a sliding window at the bottom of the display. The user an then selet this overlay to show or hide it, as well as position and resize it. An important new use of overlays is ontext based advertising. In our video, we show two examples where the information for produts in a feature film is made available to the viewers at an opportune moment. Suh advertising, oupled with the Virtual VCR tehnology, will allow the viewers to pause the movie, onsider and possibly purhase the produt, and then enjoy the rest of the movie. Suh advertising is potentially less distrating, more enjoyable, and more appropriate that urrent advertisements that ompletely interrupt the program. See Figure 4 and the aompanying video for an example of video overlays for DTV Leture of the Future Stanford University has a long tradition of transmitting many of its letures on a speialized network known as the Stanford Instrutional Television Network to whih many of the nearby Silion Valley ompanies subsribe. Most transmitted letures onsist of videos that are swithed between showing the leturer, the blakboard, and slides or other material along with audio. There are several problems with this style of presentation: The viewer an only see what the amera operator hooses to transmit. It is impossible to look at some material in more detail. The resolution of the blakboard image is often less than adequate, rendering some text unreadable. A higher resolution blakboard image would be of great help. Sine its ontents do not hange very often, even high-resolution images would require little bandwidth. For long periods of the leture, programs only show the talking leturer. A lot of bandwidth is wasted sine the bakground is usually not hanging.

8 In our projet we addressed these problems from multiple diretions applying the enhaned apabilities of DTV and image-based tehniques. First of all, we have hosen to separately proess and transmit the image of the leturer, the room, the blakboard, and any additional material. Eah of these soure materials have very distint video harateristis that we plan to utilize. In the following we give a short overview of the whole projet, but onentrating on the extration of a high-resolution blakboard image. Slusallek, Chen, Johanson / Experiments in Digital Television On the head-end we need to apture multiple video streams of the room with the leture and the blakboard. We then need to segment and broadast the different objets (muh in the spirit of MPEG-4 s video objets 14 ). On the reeiver side we need to reompose the different video streams into a single presentation, but are now able to give the user the ability to ustomize it aording to his preferenes. This allows him, for instane, to onentrate longer on a blakboard image or review some earlier slides again. We start by reating a geometri model of the leture hall, whih is augmented with projetive textures extrated oasionally from a video stream. As a result we an save onsiderable bandwidth by only transmitting this model and the textures infrequently instead of sending the image of the bakground for eah video frame. This approah allows a viewer to freely move within the room and view the lassroom from whatever loation he prefers, not just the angle hosen by the amera operator. In order to display the leturer within this model, we need to segment him from the bakground. We urrently use a simple segmentation algorithm that uses the known olors of the bakground to distinguish it from the leturer. Using the known amera position and the geometry of the room, we roughly estimate the position of the leturer in front of the blakboard and plae his video image as a 3D billboard into the sene. Although this is a simple tehnique it already provides a surprisingly realisti view of the leture while using only a fration of the bandwidth a full video transmission of the leture would require. In order to display the blakboard with high enough resolution to be readable, it is neessary to use several ameras to form a running image of the blakboard that is updated in real time. Sine a single amera annot apture the entire board with suffiient resolution, we use ameras that pan and zoom to areas of interest and integrate that data into the running high-resolution image of the board. A single fixed amera is used to obtain a low resolution referene image of the entire board to aid in the integration of the image streams from the higher resolution ameras. It also insures that something an be said about all of the board in the ase that an area has not yet been sanned by one of the high-resolution ameras. The leturer is segmented from the low resolution amera s input to obtain the running referene image of the Figure 5: Interior view of a panorama shortly before ompletion of the painting 15. board, without the leturer obsuring the view. The segmentation problem here is simpler than the previous one, sine we are eliminating the leturer, rather than extrating him and a onservative algorithm an be used that might also remove a small border around him. The remainder of the frame is then opied over the running low resolution image. To detet the leturer, we threshold the intensity differene of the running image and the next video frame based on the fat that the blakboard will stay nearly the same. This simple tehnique an fail and is therefore augmented with a more robust but slower algorithm that analyzes the olor distribution for those areas that have not been updated for a while (beause we might have wrongly identified a piee of blakboard as a leturer). To deide where to point the pan and zoom ameras, we maintain a uriosity bitmap. It is marked when we see a large enough differene in the orresponding pixel in the low resolution ontrol image (whih is updated in real time, regardless of the positions of the mobile ameras). The moving amera will then sweep out that area of the image, take high-resolution images, and lear the appropriate areas in the uriosity bitmap. When integrating a high-resolution image from a highresolution amera into the output stream, we ompute the mapping between the amera s image spae and board spae by identifying markers on the board. After reprojetion of the high-resolution image into the blakboard image it is masked with the leturer and opied into the output image. Having a referene stream with the entire board is ruial to integrating the high-resolution images taken from arbitrary positions.

9 Slusallek, Chen, Johanson / Experiments in Digital Television Figure 6: The amera rig used for apturing the video panoramas. We urrently use a modified MPEG enoder to ompress the high resolution blakboard image using a signifiantly lower frame rate. Ideally, we would like to only transmit those areas that have hanged, but MPEG already has a fairly small overhead for oding these unhanged regions Video-Panorama In the late 18th and 19th entury, panoramas were a popular form of mass entertainment. Panoramas were invented by Robert Barker in 1792, as a means to show omplete 360 degree views of interesting environments (suh as landsapes) to the interested and paying publi 15. Panoramas in these days were reated in speially build irular buildings with a small viewing platform in the middle and a painting on a large ylindrial anvas around it. These panorama anvases reahed heights of up to 18m and measured up to 130m in irumferene. In order to ahieve the orret perspetive view, the painting (whih often took months to omplete) had to be warped aordingly (see Figure 5). Due to the large highly realisti drawings at a large distane the viewer, limited vertial viewing, and limited movements on the viewing platform, the human senses were triked into believing they were seeing the real landsape. This experiene was often enhaned by plaing real 3D objets in front of the image that merged into the bakground (providing otherwise non-existent parallax information) and by adjusting the lighting onditions aordingly. In that sense, panoramas provided a early form of virtual reality experiene and aused dramati responses by the viewers of its time. Animated video panoramas were first reated by Raoul Brimoin-Sanson (Cineoramas) in 1897 and are still used today, e.g. in Disneys CirleVision installations. 3D IMAX and OMNIMAX 11 are limited forms of panoramas but also inlude stereo effets. Figure 7: The arrangements of the polygons on to whih the eight DTV streams get projeted. By adjusting the position, orientation, and brightness of the polygons, we alibrate the panorama to show a seamless surround view. Panoramas reently got a new push in the ontext of image-based rendering by the work of Chen and Williams 5 4. Here, a ylindrial panorama is reated and a small view an be displayed and moved interatively on a monitor, providing some sense of the surround view of a real panorama. These panoramas, however, are limited to stati views points. Using the infrastruture of DTV, we deided to takle the problem of video panoramas by providing the viewer with an interative view of a moving panorama. The ontext of this appliation would be a guided tour of the Stanford ampus. At eah instant the video panorama would provide a 360 degree surround view from the urrent loation. As the loation or surroundings hanges, new panoramas beome visible. The user should also be able to interatively move the urrent view within the full panorama, so he an be wathing a partiular objet longer than the guide antiipated or he an simply enjoy the surrounding ampus. In order to reate the 360 degree panorama we had to apture live video from eight CCD ameras arranged on a irular platform. In order to manage the immense video bandwidth of roughly 240 Mbit, two quad video mergers are used that ombined the analog signals from the eight ameras into two NTSC video signals, eah divided into quadrants showing one of the ameras. These two video signals are aptured at 752x480 resolution using Motion-JPEG apture boards on two separate PCs. Finally, the eight video streams are extrated from the aptured footage, enoded using MPEG-2, and multiplexed into a single ATSC DTV hannel. On the reeiver side, a pipeline of programs work together to display this transport stream. The DTV stream is reeived on a PC, where it is retransmitted over 100 Mbps Ethernet to an SGI Onyx 2 with eight CPUs. The standard referene

10 Slusallek, Chen, Johanson / Experiments in Digital Television Figure 8: This image shows a single frame from the video panorama projet. The video panorama onsists of a tour of Stanford ampus. At eah point in time a full 360 degree panorama is reeived and the user is free to browse around, but a default view is provided. The display also shows a map of the ampus with a pointer indiating the urrent position. By liking on a loation on the map the urrent viewing diretion rotates and loks onto this feature allowing a user to wath it while the tour ontinues. MPEG-2 deoder software has been modified so that eight opies an be run synhronously delivering frames in the YUV format. A master proess ontrols the eight deoders, and requests the frames that are needed to onstrut the urrent view. These frames are then passed on to the panorama viewer. We have hosen to always deode all eight video streams due to the startup delay of about half a seonds when deoding MPEG-2 streams with large GOPs (group of pitures). This delay would have severely restrited the speed at whih viewers ould have hanged their view. The viewer program is an OpenGL appliation that texture maps the video frames on to eight retangular polygons whih form an overlapping otagon around the virtual amera (see Figure 7). The spherial aberration of the apture ameras is orreted by suitable warping of the input video on to eah display polygon. During display, the viewer appliation determines the frames that are needed based on the urrent view diretion and requests them from the MPEG-2 deoder system. The images are loaded as OpenGL textures, and a olor transformation matrix is used to onvert them from YUV format into RGB in hardware. For a given amera rig, the exat arrangement of the display polygons needs to be adjusted so that the frames appear stithed together into a seamless panorama. Currently, this is aomplished by manually adjusting their positions, but automati alibration is planned for a future version of the appliation. In addition to the basi panorama viewer, several additional features were added to make the video panorama more useful (see Figure 8 and the video lip). Sine the aptured material was a tour of Stanford ampus, a ampus map is displayed in addition to the video panorama window. This map displays the position and diretion of the urrent view. The user an point on the map to indiate a new viewing diretion, e.g. toward a ertain objet, whih is then traked. This is useful for getting ones bearings as the tour progresses. Finally, on ommand or in ases when the user stops interations, the view drifts bak towards a predefined view sequene hosen by the tour guide. Thus, lazy viewers are guaranteed to be looking in the diretion of the objet urrently being disussed by the guide on the assoiated audio trak. At any time, the user has the freedom to override the tour guide or again join his tour, very muh like in reality. Combining the urrent appliation with the Virtual VCR support (see Setion 5.2) would also allow a user to stay behind and enjoy a partiular view before athing up or resuming the tour. This is planned as a future extension. 6. Conlusions In this paper we explored the emerging digital television tehnology and its interation with omputer graphis. DTV is supposed to provide high bandwidth broadast onnetions to every household and offers high-quality, highresolution video streams. Most interesting from a omputer graphis perspetive is the ability to transmit any other data with the DTV stream, whih allows us to extend DTV in a number of different diretions. In the previous setions we presented several appliations as examples from the large DTV design spae. We provided the ability to deouple the display time from the transmission time through a real-time video buffer, whih is an important base tehnology that we applied to a number of the other appliations. We showed how to provide additional information onto the live video sreen in the form of video overlays. The overlays are manipulated by small sripts that ontrol their appearane and allow the user to interatively browse their information. In the leture of the future projet, we onentrated on using image-based tehniques to san and ombine small high-resolution images of a blakboard and transmit them as a separate video stream to enhane the key leture part. Finally, we presented a virtual reality appliation, using real-time video panoramas that provide new visual experienes and interativity in the ontext of DTV transmissions. In onlusion, we hope to have demonstrated that omputer graphis offers onsiderable know-how and tehnology in areas suh as image-based rendering, interative appliations, and virtual reality, that an diretly ontribute to reating exiting new DTV appliations. As digital television tehnology beomes more widely available over the next few month and years, we expet to see many new developments that may forever hange our notion of television.

11 Slusallek, Chen, Johanson / Experiments in Digital Television Aknowledgments First of all, we would like to thank Pat Hanrahan and Serge Rutman (INTEL) for sharing their experiene. Many thanks are also due to all the students who spent many hours in designing and implementing the presented projets. In partiular we would like to thank Edward Chang, Ben Mowery, Hareesh Kessavan, Brad Nelson, Chuk Fraleigh, Rustan Eklund, Bryn Forbes, and Jin Hian Lee for providing diret input to this paper. We also want to thank Tamara Munzer for helping with the aompanying video. Finally, we like to thank Intel, Sony, and Interval Researh for funding the Stanford Immersive Television Projet, whih inludes most of this work. Referenes 1. Albert Abramson. The History of Television, 1880 to MFarland, ATSC homepage Edward Chang and Hetor Garia-Molina. Effetive memory use in a media server. In Proeedings of the 23rd Very Large Data Base (VLDB) Conferene, pages , Athens, Greee, August Shenhang Eri Chen. Quiktime VR - an image-based approah to virtual environment navigation. Computer Graphis (SIGGRAPH 95 Proeedings), pages 29 38, Shenhang Eri Chen and Lane Williams. View interpolation for image synthesis. Computer Graphis (SIG- GRAPH 93 Proeedings),, 27: , August Carolina Cruz-Neira, Daniel J. Sandin, and Thomas A. DeFanti. Surround-sreen projetion-based virtual reality: The design and implementation of the CAVE. Computer Graphis (SIGGRAPH 93 Proeedings), pages , Gary Demos. Temporal and resolution layering in advaned television. demografx/layered.html, DVB homepage David E. Fisher and Marshall Jon Fisher. The Tube The Invention of Television. Harourt Brae, Barry G. Haskell, Atul Puri, and Arun N. Netravali. Digtal Video An Introdution to MPEG-2. Chapman & Hall, Nelson L. Max. Computer graphis distortion for IMAX and OMNIMAX projetion. Niograph 83 Proeedings, pages , Deember MPEG audio MPEG-2 generi oding of moving pitures and assoiated audio information. online available at 2/mpeg-2.htm, July ISO/IEC MPEG-4 standard, version 1. available at Otober Stephan Oettermann. The Panorama History of a Mass Medium. Zone Books, Charles A. Poynton. A Tehnial Introdution to Digital Video. Wiley, Charlie Sandbank and Ken MCann. DVB and HDTV. In Proeedings IBC 97, Sydney, Australia, Deember online available at Jonathan W. Shade, Steven J. Gortler, Li wei He, and Rihard Szeliski. Layered depth images. Computer Graphis (SIGGRAPH 98 Conferene Proeedings), pages , July Alvy Ray Smith. Digital TV postgame wrapup. NAB%20Postgame%20Wrapup.htm, August 1998.