Why Take Notes? Use the Whiteboard Capture System

Why Take Notes? Use the Whteboard Capture System L-we He Zhengyou Zhang and Zcheng Lu September, 2002 Techncal Report MSR-TR-2002-89 Mcrosoft Research Mcrosoft Corporaton One Mcrosoft Way Redmond, WA 98052

Why Take Notes? Use the Whteboard Capture System L-we He, Zcheng Lu, Zhengyou Zhang Mcrosoft Research One Mcrosoft Way, Redmond, WA, USA {lhe, zlu, zhang}@mcrosoft.com ABSTRACT Ths paper descrbes our recently developed system whch captures both whteboard content and audo sgnals of a meetng usng a dgtal stll camera and a mcrophone. Our system can be retroft to any exstng whteboard. It computes the tme stamps of pen strokes on the whteboard by analyzng the sequence of captured snapshots. It also automatcally produces a set of key frames representng all the wrtten content on the whteboard before each erasure. Therefore the whteboard content serves as a vsual ndex to effcently browse the audo meetng. It s a complete system whch not only captures the whteboard content, but also helps the users to vew and manage the captured meetng content effcently and securely. Keywords: Whteboard capture, meetng archvng, vdeo analyss, audo/vdeo ndexng, mage classfcaton, corporate knowledge base, dgtal lbrary. 1 INTRODUCTION The Whteboard Capture System s a part of our Dstrbuted Meetngs project, the goal of whch s to dramatcally mprove the meetng experence of knowledge workers usng ubqutous computng technologes. The work presented n ths paper focuses on the partcular meetng scenaros that use whteboard heavly such as: branstormng sessons, lectures, project plannng meetngs, patent dsclosures, etc. In those sessons, a whteboard s ndspensable. It provdes a large shared space for the partcpants to focus ther attenton and express ther deas spontaneously. It s not only effectve but also economcal and easy to use -- all you need s a flat board and several dry-nk pens. Whle whteboard sessons are frequent for knowledge workers, they are not perfect. The content on the board s hard to archve or share wth others who are not present n the sesson. People are often busy copyng the whteboard content to ther notepads when they should spend tme sharng and absorbng deas. Sometmes they put Do Not Erase sgn on the whteboard and hope to come back and deal wth t later. In many cases, they forget and content s left on the board for days. Even f they do come back, they are often confused by the sketchy handwrtngs, unorthodox acronyms, and wonder how on earth a result s derved. Our system s an attempt to allevate meetng partcpants the mundane tasks of note takng by capturng whteboard content automatcally along wth the audo. We use a hghresoluton dgtal stll camera to capture one mage of the whteboard about every 5 seconds. We desgned an algorthm to dstll a small set of key frame mages from the captured mage sequence. Tme stamps of the pen strokes contaned n the key frames are also computed. The users can vew the key frame mages, prnt them as notes, or cut and paste them nto documents. If the users want to fnd more about the dscusson on a partcular topc, our browsng software allows them to clck some pen stroke assocated wth that topc and brng up the audo at the moment when the stroke was wrtten. From our ntal deployment, we found our system useful for the meetng partcpants to revew the meetng at a later tme, and also for users who dd not attend the meetng to understand the gst of the meetng n a fracton of meetng tme. We are also pleasantly surprsed by some creatve ways that partcpants use to take advantage of the newfound capablty n the meetng room. The paper s organzed as follows: Secton 2 dscusses related works. Secton 3 explans the desgn choces that we made and presents the archtecture of our Whteboard Capture System. Secton 4 explans the techncal challenges we encountered whle buldng the system. Secton 5 reports the detals of the deployment process of our system. We dscuss the lmtatons of our system n Secton 6 and conclude n Secton 7. Note that the drawngs n ths paper were done on a whteboard and were captured by our system. 2 RELATED WORKS 2.1 Whteboard Capture Devces Many technologes exst to capture the whteboard content automatcally. One of the earlest, the whteboard coper, s a specal whteboard wth a bult-n coper. Wth a clck of a button, the whteboard content s scanned and prnted. Once the whteboard content s on paper, t can be

photocoped, faxed, put away n the fle cabnet, or scanned nto dgtal form. Recent technologes all attempt to capture the whteboard nto dgtal form from the start. They generally fall nto two categores. The devces n the frst category capture mages of the whteboard drectly. NTSC-resoluton vdeo cameras are often used because of ther low cost. Snce they usually do not have enough resoluton for a typcal conference room sze whteboard, several vdeo frames must be sttched together to create a sngle whteboard mage. The ZombeBoard system [14], deployed nternally at Xerox PARC, uses a Pan-Tlt vdeo camera to scan the whteboard. The Hawkeye system from SmartTech opts for a three-camera array that takes mages smultaneously. Another devce n ths category s the dgtal stll camera. As hgh resoluton dgtal cameras get cheaper, takng snapshots of the board wth a dgtal camera becomes a popular choce. To clean-up the results, people use software (e.g. Adobe Photoshop or Pxd Whteboard Photo) to crop the non-whteboard regon and color-balance the mages. Our system dfferentates from the above systems n that we compute the tme stamps of the pen strokes and the key frames by performng analyss on the captured mages. The key frames provde the summary notes to the meetng. The tme stamps and key frames are effectve ndces to the recorded audo. Devces n the second category track the locaton of the pen at hgh frequency and nfer the content of the whteboard from the hstory of the pen coordnates. Mmo by Vrtual Ink s one of the best systems n ths category. Mmo s an add-on devce attached to the sde of a conventonal whteboard and uses specal cases for dry-nk pens and eraser. The pen emts ultrasonc pulses when pressed aganst the board. The two recevers at the add-on devce use the dfference n tme-of-arrval of the audo pulses to trangulate the pen coordnates. Snce the hstory of the pen coordnates s captured, the content on the whteboard at any gven moment can be reconstructed later. The user of whteboard recordng can play back the whteboard lke a move. Because the content s captured n vector form, t can be transmtted and archved wth low bandwdth and storage requrement. Electronc whteboards also use pen trackng technology. They go one step further by makng the whteboard an nteractve devce. For example, the SMARTBoard from SmartTech s essentally a computer wth a gant touch-senstve montor. The user wrtes on the montor wth a specal stylus whch s tracked by the computer. The computer renders the strokes on the screen wherever the stylus touches the screen -- as f the nk s deposted by the stylus. Because the strokes are computer generated, t can be edted, re-flowed, and anmated. The user can also ssue gesture commands to the computer and show other computer applcatons on the same screen. However, electronc whteboards currently stll have lmted nstall base due to ther hgh cost and small szes (the sze of an electronc whteboard rarely exceeds 6 feet dagonal). Furthermore, those pen-trackng devces have the followng dsadvantages: 1) If the system s not on or the user wrtes wthout usng the specal pens, the content cannot be recovered by the devce; 2) Many people lke to use ther fngers to correct small mstakes on the whteboard n stead of the specal eraser. Ths common behavor causes extra strokes to appear on the captured content. Regardng Mmo, we can menton two more ponts: 3) People have to use specal dry-nk pen adapters, whch make them much thcker and harder to press; 4) Imprecson of pen trackng sometmes causes ms-regstraton of adjacent pen strokes. 2.2 Multmeda Recordng Systems Studes have shown that capturng multmeda experence such as lectures or meetngs s useful ether for later revewng by the partcpants or for the people who dd not attend [1,5,6,8,11,16]. A lot of research has been done on the capture, ntegraton, and access of the multmeda experence. People have developed technques and systems that use handwrtten notes, whteboard content, sldes, or manual annotatons to ndex the recorded vdeo and audo for easy access [1,4,8,10,11,15,16,17,18]. Our work s partly nspred by Lsa Stfelman s work on Audo Notebook, a prototype notepad combnng pen-andpaper and audo recordng [11,13]. Audo Notebook provdes structure to the recorded audo based on the synchronzaton of the key ponts marked by pen on paper. Another related work s the Classroom2000 project as reported by Abowd et al. [1,2,3]. In the Classroom2000 project, Abowd et al. used an electronc whteboard to tmestamp the nk strokes so that the vewer (students) can use the nks as the ndexes to the recorded vdeo and audo. They also computed key frames (called pages) based on the erasng events provded by the electronc whteboard. The man dfference between ther work and ours s that we use an ordnary whteboard whle they requre an electronc whteboard. Wth electronc whteboard, t s trval to obtan the tme stamps of the pen strokes and know when the strokes are erased. But there are many dsadvantages from the end user s pont of vew. Frst of all, most of the offces and meetng rooms do not have electronc whteboards nstalled. Secondly, as reported n [1], people found that t s much more natural to use a regular whteboard than an electronc whteboard. Thrdly, mages captured wth a camera provde much more contextual nformaton such as who was wrtng and whch topc was dscussng (usually by hand pontng). 3 SYSTEM DESIGN The desgn goals of the Whteboard Capture System are: 1. Work wth any exstng whteboard. 2. Capture the whteboard content automatcally and relably. 3. Use the whteboard content as a vsual ndex to effcently browse the recorded meetng. In desgnng the system, we had several desgn questons: 1) Whch capture devce to use? 2) How do users record meetngs? 3) How do we present the browsng user

nterface? 4) How do we keep the recorded meetng data secure? The desgn s an teratve process. In order to collect feedback early n the desgn phase, we nstalled capture systems n three conference rooms nearby. Not only our team members use t frequently as the system evolves, we also encourage other groups to use the system. 3.1 Capture Devce One mportant desgn goal of our system s to work wth any exstng whteboard. Even though Mmo system works wth exstng whteboard, t requres people to use specal pens and erasures, and there are varous problems as mentoned n the prevous secton. Therefore we choose to use a drect capture devce. Wthout requrng specal pens and erasers makes the nteracton much more natural. Snce t s takng pctures of the whteboard drectly, there s no ms-regstraton of the pen strokes. As long as the users turn on the system before erasng, the content wll be preserved. Our Whteboard Capture System uses a 4-mega pxel dgtal stll camera to capture the whteboard content (see Fgure 1). The camera provdes mages that are 2272 pxels by 1704 pxels -- equvalent to 31.6 dp for a 6 by 4 board. Usng a mass market consumer product as opposed to propretary hardware can potentally lower the cost. smply want the system to fgure out the key frames automatcally. Ths UI requrement adds sgnfcant complexty to our analyss algorthm because we have to reconstruct the whteboard content from dfferent snapshots. A descrpton of our algorthm s detaled n Secton 4. 3.3 Browsng Interface Snce most people probably do not want to lsten to the recorded meetng from start to end, we provde two browsng features to make non-lnear accessng of the recorded nformaton very effcent. 1. Key Frames: Key frame mages contan all the mportant content on the whteboard and serve as a summary to the recordng. They can be cut and pasted to other documents or prnted as notes. 2. Vsual Indexng: We provde two levels of nonlnear access to the recorded audo. The frst s to use the key frame thumbnals. The user can clck a thumbnal to jump to the startng pont of the correspondng key frame. The second s to use the pen strokes n each key frame. Together wth the standard tme lne, these two levels of vsual ndexng allow the user to browse a meetng n a very effcent way. Current Strokes Raw Image Key Frame Thumbnals Fgure 1: A typcal Whteboard Capture Installaton. Note that ths mage was captured from one of our whteboard sessons. VCR & Tmelne Control Future Strokes 3.2 Capture Interface Another mportant feature dfferentatng our system from others s that people are not requred to move out of the camera s feld of vew durng capture as long as they do not block the same porton of the whteboard durng the whole meetng. In our ntal mplementaton, we asked the meetng partcpants to press a Key Frame button on the sde of the whteboard every tme before they want to erase a porton of the whteboard. Ths desgn smplfes the computaton n two ways: 1) The button s located off the sde of the whteboard, pressng button makes the whteboard temporarly fully exposed to the camera; 2) The button press events nform the analyss process whch mage frames are un-obscured key frames. Wth the assumpton that key frames contan only and all pen stokes, computng the tme stamp of each pen stroke becomes straghtforward: start from the key frame, smply search forward for the frst frame when the same mage pattern appears. However after a short tral perod, the users of the system found the nterface restrctve. They dd not want to remember pressng the button before each erasure. They Fgure 2: Browsng nterface. Each key frame mage represents the whteboard content of a key moment n the recordng. The man wndow shows a composton of the raw mage from the camera and the current key frame mage. The pen-strokes that the partcpants are gong to wrte n the future (Future Strokes) are shown n ghost-lke style. 3.4 Securty Meetng partcpants are usually apprehensve about recordng the meetng because senstve nformaton mght be vewed by unntended people. For them, keep the recorded data secure s a concern. To address ths concern, we developed a smple token-based access model. We ask meetng partcpants to regster wth the capture software at begnnng of the recordng. They can ether fll n ther emal alases n a dalog box or, to speedup the process, nsert ther corporate d cards nto a smartcard reader to regster. All the recorded sessons resde on a web server. If no one regsters, the meetng s posted on a publc accessble web page. If at least one partcpant regstered, an access token s generated after the analyss. The token s a long strng randomly generated wth the unque meetng d. The URL contanng the token s emaled to the regstered partcpants. The recpents can clck the URL to launch the

web browsng software to revew the meetng. They can also forward the URL to people who have not attended the meetng. Ths smple Securty-by-Obscurty model seems to work well durng our ntal deployment although more securty and prvacy measures are needed n a productzed system. 3.5 System Archtecture Conceptually, the overall Whteboard Capture System conssts of three components: the capture unt, the processng server, and the browsng software (see Fgure 3). whteboard and cast shadow on t. Wthn a sequence, there may be no frame that s totally un-obscured. We need to deal wth these problems n order to compute tme stamps and extract key frames. Fgure 4: Selected frames from an nput mage sequence. The sesson lasts about 16 mnutes and contans 195 frames. Fgure 3: The system conssts of three components: the capture unt, the analyss server, and the browsng software. 1. Capture unt s nstalled n the room where meetngs take place. It ncludes a 4 mega-pxel dgtal camera, a mcrophone, and a PC. 2. Analyss server s located n a central place and stores the recorded data. An analyss program s launched automatcally after the user stops the recordng n the capture unt. After processng, emals contanng the URL to meetng recordng are sent to the regstered partcpants. 3. Browsng software s a web plug-n nstalled by the users who wsh to vew the recordngs. Once nstalled, the users can clck the URL to launch the software to access the data on the analyss server. 4 TECHNICAL CHALLENGES The nput to the Whteboard Capture System s a set of stll dgtal mages (see Fgure 4). We need to analyze the mage sequence to fnd out when and where the users wrote on the board and dstll a set of key frame mages that summarze the whteboard content throughout a sesson. Compared to the sensng mechansm of devces lke Mmo or electronc whteboard, our system has a set of unque techncal challenges: 1) The whteboard background color cannot be pre-calbrated (e.g. take a pcture of a blank whteboard) because each room has several lght settngs that may vary from sesson to sesson; 2) Frequently, people move between the dgtal camera and the whteboard, and these foreground objects obscure some porton of the 4.1 Image Acquston Currently we use the Canon PowerShot G2 camera wth 4 mega pxels. One mportant reason that we choose G2 s the avalablty of a Software Development Kt (SDK) whch allows us to wrte customzed software solutons to control the camera from a PC. Our software can specfy vrtually all the camera parameters on a per-shot bass. The camera s mounted at ether the sde or the back of a meetng room. We zoom the camera as close to the whteboard as possble to maxmze the effectve resoluton. The camera s statonary after the nstallaton and we assume the whteboard does not move, so the whteboard mages are statonary throughout the captured sequence. Snce the G2 camera has only auto focus mode, the whteboard mght become out-of-focus f an object n front of the whteboard trggers the attenton of the auto focus mechansm of the camera. We mtgate ths problem by dong two thngs: 1) Algn mage plane of the camera as parallel to the whteboard as possble to mnmze scene depth; 2) Mnmze the aperture to ncrease the depth of feld. In practce, only 1-2% of the frames are out-of-focus. The camera takes the pctures as fast as t can and transfers the mages to the PC va USB. Usng the G2 camera, we are able to get one JPEG mage about every 5 seconds. We keep the exposure and whte-balance parameters constant. Assumng the lght settng does not change wthn one sesson, the color of whteboard background should stay constant n a sequence. We found slghtly under exposed the mages gve better color saturaton, whch makes the stroke extracton process more accurate. A color-balance step s performed at the end to make the graysh whteboard mages more appealng (see Secton 4.2.7). 4.2 Image Sequence Analyss Snce the person who s wrtng on the board s n the lne of sght between the dgtal camera and the whteboard, he/she often obscures some part of the whteboard and casts shadow on other part. We need to dstngush among strokes, the foreground object, and the whteboard. Once

we know the classfcaton results, we can produce the key frame mages and an ndex to be used by the browsng software. Rather than analyze the mages on a per-pxel level, we dvde the whteboard regon nto rectangular cells to lower the computatonal cost. The cell sze s roughly the same as what we expect the sze of a sngle character on the board (about 1.5 by 1.5 nches n our mplementaton). Snce the cell grd dvdes each frame n the nput sequence nto cell mages, we can thnk of nput as a 3D matrx of cell mages. corners, a smple b-lnear warp s performed for each mage n the sequence usng b-cubc nterpolaton. 4.2.2 Computng the Whteboard Color For the classfcaton of the cells, we need to know for each cell what the whteboard color s (that s, the color of the whteboard tself wthout anythng wrtten on t). The whteboard color s also used for whte-balancng n producng the key frames, so t needs to be estmated accurately to ensure the hgh qualty of the key frame mages. Fgure 6: Whteboard color extracton results. The left mage s the result of the 1 st strategy, the mddle mage s the result of the 2 nd strategy, and the rght mage shows the actual blank whteboard mage. Fgure 5: The mage sequence analyss process. Here s an outlne of the algorthm that we use. 1. Rectfy the whteboard regon of every mage n the sequence. 2. Extract the whteboard background color. 3. Cluster the cell mages throughout the sequence for the same cell. If two cell mages are consdered to be the same, they are clustered n the same group. 4. Classfy each cell mage as a stroke, a foreground object, or the whteboard. 5. Flter the cell mages both spatally and temporally to refne the classfcaton results. 6. Extract the key frame mages usng the classfcaton results. 7. Color-balance the key frame mages. In the followng, we use the runnng example as shown n Fgure 4 to llustrate our algorthm. 4.2.1 Rectfy the Whteboard Images Before feedng the mage sequence to the stroke extracton process, we crop the non-whteboard regon and rectfy the mages. Because the lens of the G2 camera has farly low radal dstorton, we only need to specfy the four corners of the whteboard. Ths s currently done manually by clckng a captured mage durng the one-tme calbraton step, although ths could be done automatcally. Wth the four We have expermented wth two strateges. The frst s based on the assumpton that the whteboard cells have the brghtest lumnance over tme and have small varance (.e., almost unform wthn each cell). Ths s reasonable snce the color of the strokes (red, green, blue or black) wll lower the lumnance. Ths, however, may produce holes n the fnal whteboard color mage, for example, f a cell ether contans a stroke or s blocked by a foreground object throughout the sequence. To fll a hole lke ths, we search ts neghborhood, and set ts whteboard color to that of the nearest cell whch s not a hole. Ths strategy usually works qute well, but t fals when a person wears a whte T-shrt and/or holds a pece of whte paper. The left mage of Fgure 6 shows the result of the whteboard color mage computed from the nput sequence n Fgure 4 where a person was holdng a whte paper n some of the frames. We can see that the computed whteboard color s corrupted by the whte paper. The second strategy s more sophstcated. The assumpton s that a sgnfcant porton of the pxels n each cell over tme belongs to the whteboard. By buldng a hstogram of the lumnance, the color correspondng to the peak wth a hgh lumnance value s very lkely the color of the whteboard. The frst step s therefore to compute an ntal whteboard color n ths way. It works even f a cell contans a stroke throughout the sequence, but t fals n the case when a person wears a whte T-shrt and/or holds a pece of whte paper, or when a cell s always hdden by people or other objects. In such cases, the computed whteboard color mage contans outlers. The second step s to detect those outlers. The outler detecton s based on a robust technque called least-medan-squares [11]. Assumng the color vares smoothly across the whteboard, a plane s ft n the lumnance Y or RGB space by mnmzng the medan of the squared errors. The cells whose color does not follow ths model are consdered to be outlers and consequently rejected,.e., they are marked as holes. The nterested reader s referred to the Appendx for the detals of ths technque. The thrd step s to fll the

holes by usng the same procedure as n the frst strategy. Fnally, to further mprove the result, we flter the whteboard color mage by locally fttng a plane n the RGB space. The nterested reader s agan referred to the Appendx for detals. The result obtaned wth ths new technque on the same example s shown n the mddle mage of Fgure 6. We see clear mprovements over the result obtaned wth the frst strategy as shown n the left. We also show the actual blank whteboard n the rght mage for comparson. 4.2.3 Clusterng Cell Images over Tme Durng the meetng, the content of each cell usually changes over tme. For each cell, we would lke to cluster all the cell mages n the sequence nto groups, where each group contans the cell mages whch are consdered to be the same. We use a modfed Normalzed Cross-Correlaton algorthm to determne f two cell mages are the same or not. In the followng, we descrbe the Normalzed Cross- Correlaton technque usng one component of the mage, but t apples to all RGB components. Consder two cell mages I and I. Let I and I ' be ther mean colors and σ and σ ' be ther standard devatons. The normalzed cross-correlaton score s gven by 1 c = ( I I)( I' I') where the summaton s over Nσσ ' every pxel and N s the total number of pxels. The score ranges from -1, for two mages not smlar at all, to 1, for two dentcal mages. Snce ths score s computed after the subtracton of the mean color, t may stll gve a hgh value even two mages have very dfferent mean colors. So we have an addtonal test on the mean color dfference based on the Mahalanobs dstance [7], whch s gven by d = I I' ( σ + σ '). In summary, two cell mages I and I are consdered to be dentcal and thus should be put nto the same group f and only f d < Td and c > T. In our c mplementaton, T = 2 and T = 0. 707. d 4.2.4 Classfyng Cells Ths step s to determne whether a cell mage s a whteboard, a stroke, or a foreground object. We use the followng heurstcs: 1) a whteboard cell s unform n color and s grey or whte (.e., the RGB values are approxmately the same); 2) a stroke cell s mostly whte or grey wth one or two prmary colors mxed n; 3) a foreground object does not have the characterstcs above. The classfcaton s therefore to determne whether the color dstrbuton of the current cell mage and the whteboard color dstrbuton are the same, or not the same but havng strong overlap, or totally dfferent. Agan, we use the Mahalanobs dstance [7] as descrbed below. Notce that the whteboard color has already been computed as descrbed n Sect.4.2.2. Agan, we use one component of RGB as an example. Let I w be the whteboard color and σ be the standard devaton (t s a small value snce a w c whteboard cell s approxmately unform). Let I and σ be the mean and standard devaton of the current cell mage. The cell mage s classfed as a whteboard cell f and only f I I w ( σ + σ ) < T and σ / σ < w T ; as a stroke σ w w cell f and only f I I w ( σ + σ w) < T and w σ / σ w T ; otherwse, as a foreground object cell. In our σ mplementaton, T = 2 andt = 2. w σ 4.2.5 Flterng Cell Classfcaton Notce that the above classfcaton algorthm only uses the color nformaton n a sngle cell. More accurate results can be acheved by utlzng spatal and temporal relatonshp among the cell groups. Spatal flterng. The basc observaton s that foreground cells should not appear solated spatally snce a person usually blocks a contnuous regon of the whteboard. So we perform two operatons on every sngle whteboard mage. Frst, we dentfy solated foreground cells and reclassfy them as strokes. Second, we reclassfy stroke cells whch are mmedately connected to some foreground cells as foreground cells. One man purpose of the second operaton s to handle the cells at the boundares of the foreground object. Notce that f such a cell contans strokes, the second operaton would ncorrectly classfy ths cell as a foreground object. But fortunately, the followng temporal flterng wll correct such potental errors. Temporal flterng. The basc observaton s that t s vrtually mpossble to wrte the same stroke n exactly the same poston after t s erased. In other words, f for any gven cell, the cell mages of two dfferent frames contan the same stroke, then all the cell mages n between the two frames must have the same stroke unless there s a foreground object blockng the cell. Ths observaton s very useful to segment out the foreground objects. Consder the example n the prevous secton where a stroke cell at the boundary of the foreground object s ncorrectly classfed as a foreground cell. At the temporal flterng step, ths cell wll be classfed as a stroke as long as t s exposed to the camera before and after the foreground object blocks t. Fgure 7: Samples of the classfcaton results. The mages above correspond to the mages n Fgure 4 after croppng and rectfcaton.

Fgure 7 shows the classfcaton results for the sample mages n Fgure 4, where the strokes are n green, the foreground s n black, and the whteboard s n whte. 4.2.6 Key Frame Image Extracton Key frame mages are the summary of a whteboard sesson. The user would expect the key frame mages to have the followng propertes: 1) They should capture all the mportant content on the board; 2) The number of the key frames should be kept to a mnmum; 3) They should only contan the pen strokes and the whteboard, but not the person n front; 4) They should have unform whte background and saturated pen colors for easy cut-and-paste and prntng. The key frame extracton algorthm uses the cell mages classfcaton results from the prevous step. The algorthm frst decdes whch frames n the sequence should be selected as key frames; t then reconstructs the key frame mages. 1000 900 800 700 600 500 400 300 200 100 0 Key Frame 1 1 21 41 61 81 101 121 141 161 181 Chapter 1 Chapter 2 Key Frame 2 Fgure 8: Key frame extracton. A plot of number of strokes vs. tme for the sequence n Fgure 4. The correspondng two key frame mages are shown n Fgure 3 and Fgure 5. Key frame selecton. There s no unque soluton n selectng the key frames -- just as there s no sngle way to summarze a meetng. In our system, we frst dvde the meetng nto several chapters (topcs) then create a key frame mage representatve of the whteboard content for that chapter. An erasure of a sgnfcant porton of the board content usually ndcates a change of topc so we use t as a dvder of the chapters. We choose the frame just before an erasure starts as the key frame, whch ensures that the content s preserved n those frames. The detaled algorthm works as follows: 1. Count the number of stroke cells for each frame n the sequence. Note that one stroke cell mage may span multple frames t s ncluded n the count for each of those frames. Fgure 8 shows the stroke cell count plotted aganst frame number n the example sesson (Fgure 4). A rse n the plot ndcates more strokes are wrtten on the board, where a dp n the plot ndcates some strokes are erased. As we can see the graph s qute nosy. There are two reasons: 1) The user s constantly makng small adjustments on the board; 2) The classfcaton results contan small errors. 2. If we produce a key frame at each dp, we wll get dozens of key frames. In order to keep the number of key frames to a mnmum, the data s fltered to retan only the sgnfcant erasure events. Our algorthm gnores the fluctuaton n the data unless the dfference between the adjacent peak and valley exceeds a certan threshold. We use 20% of the maxmum stroke count n our system. 3. The valleys n the data dvde the sesson nto chapters. The frame contanng the peak wthn a chapter s chosen to be the key frame representng the chapter. Image reconstructon. Once the frames are selected, we need to reconstruct the mages correspondng to what the whteboard looks lke at these ponts of tme. But we cannot smply use the raw mages from the nput sequence because they may contan foreground objects. We reconstruct the mage by gatherng the cell mages n the frame. There are three cases dependng on the cell classfcaton: 1. If a cell mage s whteboard or stroke, ts own mage s used. 2. If the foreground cell mage s wthn the span of a stroke (.e., the person s obscurng the strokes on the board), we replace ths cell mage wth the stroke cell mage from the neghborng frames. 3. Otherwse, a foreground object must be coverng the whteboard background n ths cell, and we smply fll t wth ts whteboard color computed n Secton 4.2.2. 4.2.7 Key Frame Color Balance The reconstructon process removes the person from the whteboard mages, but the mages stll look lke the raw mages from the nput sequence: graysh and nosy. They need to be color balanced. The process conssts of two steps: 1. Make the background unformly whte and ncrease color saturaton of the pen strokes. For each cell, the whteboard color computed n Secton 4.2.2, I w, s used to scale the color of I each pxel n the cell. n I out = mn( 255, 255) Iw 2. Reduce mage nose. We remap the value of each color channel of each pxel n the key frames accordng to an S-shaped curve. The fnal mages from our runnng example (Fgure 4) are Fgure 3 and Fgure 5 whch we used earler n ths paper. The begnnng and endng tmes of the chapters and the fle names of ther key frame mages are saved n the ndex along wth the tme stamps of the strokes produced n Secton 4.2.5. 4.3 Browsng Experence After the analyss server processed the mage sequence and produced the ndex and key frame mages, t sends emals to the sesson partcpants wth the URL to the processed

recordng. The users can clck the URL to launch the browsng software (see Fgure 2). The goal of the browsng software s to allow users to vew the key frame mages and quckly access the audo assocated wth a partcular topc. 4.3.1 Image Vewng Mode The thumbnals of the key frame mages are lsted n the rght pane. Clckng one of the thumbnals brngs the correspondng key frame mage to the man wndow at the left and takes the applcaton to the mage vewng mode, where the user can zoom n and out, read the text and dagrams n the mage, or cut and paste a porton of the mage to other documents. 4.3.2 Audo Playback Mode When the cursor s hoverng over a pen stroke cell, the cursor s changed to a hand symbol ndcatng that t s clckable. Double clckng the cell brngs the applcaton to the audo playback mode. The playback starts from the tme of the sesson when the clcked stroke cell was wrtten. The user can stll clck other stroke cells to jump to other part of the sesson. 4.3.3 Whteboard Content Vsualzaton In the audo playback mode, how do we vsualze the whteboard content n the man wndow? Gven the key frame mages and the tme stamp nformaton, we can reconstruct an mage that corresponds to the whteboard content at any gven tme. If we render the mage every frame accordng to the audo playback tme, the man wndow playbacks the whteboard content lke a move. Usng ths approach, the users wll have both the aural and vsual context to the sesson. But they cannot clck any pen stroke that takes them forward n tme (Future Strokes) because these strokes have not yet been rendered n the man wndow. In our ntal mplementaton, we borrowed the approach n Classroom2000 [2]: show the future strokes n a washed out mode. However after a short tral perod, the users of the browser often confuse the future strokes wth the strokes that are not cleanly erased. Another complant about the nterface s that although the users lke the whteboard mages wthout the person n front, they sometmes want to know who wrote the strokes. After a few desgn teratons, we decde on the followng vsualzaton that addresses all those concerns: 1. Render the current whteboard content usng the key frame mage of the current chapter and tme stamp nformaton. 2. Render the Future Strokes, convert the results to grey scale, and blur them usng a Gaussan flter. 3. Add mages from Step 1 and Step 2. 4. Alpha-blend the mage from Step 3 wth the rectfed mage from the nput sequence. The user can control the alpha value wth a GUI slder from 0, showng only the rendered whteboard mage, to 1, showng exactly the orgnal mage. See Fgure 2 for an example of such a vsualzaton wth alpha=0.8. 5 SYSTEM PERFORMANCE AND USAGE We have equpped three conference rooms wth the Whteboard Capture System. Informaton about those three rooms s lsted n Table 1. The szes of whteboards n those rooms vary and so do the qualtes of the key frame mages produced. As we can see from the sample mages, the wrtngs on a 12 x5 board are fuzzer than the ones on the other two boards because the resoluton s maxed out for a 4 mega-pxel nput mage. Nevertheless, they are stll qute legble. Several selected frames from a sesson usng a 12 x5 whteboard and the key frame mages are shown n Fgure 9. Table 1: Informaton about the three nstallaton stes. Room 1 Room 2 Room 3 Board Dmenson (ft) 4x3 8x5 12x5 Key Frame Image Dmenson (pxel) 1200x900 2400x1500 2400x1000 Resoluton (dp) 25 25 16.7 Sample Images (80x80 pxels, approx. 96 pt font on the board) Fgure 9: A sesson usng the 12'x5' whteboard. Ths sesson s 1 hour and 5 mnutes long. Shown are 6 sample mages and 2 extracted key frame mages.

The analyss server runs on a Pentum III 800MHz dual CPU PC. The analyss process takes about 20 mnutes for every hour of sesson tme. The storage requrement for the 16 bt 11 KHz mono audo takes about 15 Mb per hour usng MP3 encodng. The nput mage sequence requres about 34 Mb per hour usng Moton JPEG compresson. The systems nstalled n three conference rooms are used frequently not only by our own team members but also by colleagues from neghborng groups. Over the course of 6 months, we recorded 108 sessons totalng 48 hours -- averagng 27 mnutes per sesson and 4.5 sessons per week. The average number of key frames per sesson s 2.7. The key frame mages are saved n JPEG format. The average mage sze s 51.8 Kb. The szes range from 17 Kb to 150 Kb. Because the JPEG compresson works extremely well on the unform whte background, the mage sze s more related to how much the users wrte on the board than the mage dmenson. All users of our system beleve that our system s very useful for meetngs that use whteboard extensvely: The key frame mages and the vsual ndexng capablty not only allow the partcpants to revew a meetng at a later tme, but also allow the users who dd not attend the meetng to understand the gst of the meetng n a fracton of the actual meetng tme. I would de to have such a system n all of our dsclosure meetngs, a patent attorney clamed after usng the system to browse the dsclosure meetng he just had. We are pleasantly surprsed by some users who found new ways to use the system whch we dd not ntend ntally. Take an example of status meetngs whch usually dd not requre wrtng on whteboard. People stll turned on the whteboard capture system. When t was someone s turn to speak, the manager wrote hs/her name on the board so that the speech segments could be easly found later n the recorded audo by clckng on the names n the key frame mage. Another example s durng a branstorm sesson, when someone thought of a good dea, he wrote a star on the sde of the board and sad t aloud. The audo can then be retreved later by clckng on the star. 6 LIMITATIONS OF OUR CURRENT SYSTEM Under some very specal crcumstances, our system may fal to produce the desred results. For example, f a person stands perfectly stll n front of the whteboard for an extended perod, our system would not be able to determne that t s a person. The cells covered by the person would be ether treated as strokes or whteboard dependng on the textures of hs/her cloth. If a regon of the whteboard s never exposed to the camera, our system would not be able to fgure out the content n that regon. Currently, our analyss algorthm requres the color of the whteboard background to reman constant n an nput sequence. The requrement assumes constant lghtng and constant camera exposure settng throughout a meetng whch seems to work n our ntal deployment, but a more flexble approach mght be to nstall a known color patch above the top of the whteboard where nobody can obscure from the camera. The software can then adjust the camera exposure parameters for dfferent lghtng condtons on a per-frame bass. The frame rate of our system s lmted by the frame rate of the commercally avalable stll cameras. To acheve hgher frame rate, one possblty s to use a hgh resoluton vdeo camera such as HDTV cameras at hgher cost. Another possblty s to use super-resoluton technques to ncrease the resoluton of a regular vdeo camera. 7 CONCLUDING REMARKS Meetngs consttute a large part of knowledge workers workng tme. Makng more effcent use of ths tme translates to a bg ncrease n ther productvty. The work presented n ths paper, the Whteboard Capture System, focuses on the meetng scenaros that use whteboard heavly: branstormng sessons, lectures, project plannng meetngs, patent dsclosures, etc. Our system allevates the partcpants of those meetngs the mundane note-takng task, so they can focus on contrbutng and absorbng deas durng the meetngs. By provdng key frame mages that summarze the whteboard content and structured vsual ndexng to the audo, our system helps the partcpants to revew the meetng at a later tme. Furthermore, the users who dd not attend the meetng can often understand the gst of the meetng n a fracton of meetng tme. From our ntal deployment, we found the meetng partcpants started to change ther behavors when the recordng and ndexng capabltes of the system are dscovered. We beleve that as the partcpants become more and more famlar wth the system, the summares and ndces produced by the system wll become more useful. Appendx: Plane-based whteboard color estmaton We only consder one component of the color mage, but the technque descrbed below apples to all components (R, G, B, or Y). Each cell s defned by ts mage coordnates (x, y ). Its color s desgnated by z (z=r, G, B, or Y). The color s computed as descrbed n Secton xxx, and s therefore nosy and even erroneous. From our experence wth the meetng rooms n our company, the color vares regularly. It s usually much brghter n the upper part and becomes darker toward the lower part, or s much brghter n one of the upper corners and becomes darker toward the opposte lower corner. Ths s because the lghts are nstalled aganst the celng. Therefore, for a local regon (7x7 cells n our case), the color can be ft accurately by a plane; for the whole mage, a plane fttng s stll very reasonable, and provdes a robust ndcaton whether a cell color s an outler. A plane can be represented by ax + by + c z = 0. We are gven a set of 3D ponts {( x, y, z ) = 1,..., n} wth T nose only n z. The plane parameters p = [ a, b, c] can be estmated by mnmzng the followng objectve functon:

2 F = f, where f = ax + by + c z. The leastsquares soluton s gven by p = ( A A) A z, T 1 T where x1 y1 = L L x n y n 1 L 1 z = z,, A and [ ] T 1 K z n. Once the plane parameters are determned, the color of the cell s replaced by zˆ = ax + by + c. The least-squares technque s not robust to erroneous data (outlers). As mentoned earler, the whteboard color we ntally computed does contan outlers. In order to detect and reject outlers, we use a robust technque to ft a plane to the whole whteboard mage. We use the least-medansquares [11], a very robust technque that s able to tolerate near half of the data to be outlers. The dea s to estmate the parameters by mnmzng the medan, rather than the 2 sum, of the squared errors,.e., mn medan f. We frst draw m random subsamples of 3 ponts (3 s the mnmum number to defne a plane). Each subsample gves an estmate of the plane. The number m should be large enough such that the probablty that at least one of the m subsamples s good s close to 1, say 99%. If we assume that half of the data could be outlers, then m = 35, therefore the random samplng can be done very effcently. For each subsample, we compute the plane parameters and the medan of the squared errors f 2. We retan the plane parameters that gve the mnmum medan of the squared errors, denoted by M. We then compute the so-called robust standard devaton σ = 1.4826 M (the coeffcent s used to acheve the same effcency when no outlers are present). A pont s consdered to be an outler and dscarded f ts error f > 2.5σ. Fnally, a plane s ft to the good ponts usng the least-squares technque descrbed earler. The color of an outler cell s replaced by zˆ = ax + by c. + p 5. Chu, P., Kapuskar, A., Retmeer, S., and Wlcox, L. Meetng capture n a meda enrched conference room. Proceedngs of CoBuld '99. Sprnger-Verlag LNCS 1670, pp. 79-88. 6. Chu, P., & Wlcox, L. D., A dynamc groupng technque for nk and audo notes. Proceedngs of UIST 98. 7. Duda, R.O., Hart, P.E. and Stork, D.G. Pattern Classfcaton, Second Edton, John Wley & Sons, New York, 2001. 8. Ju, S.X., Black, M.J., Mnnerman, S. & Kmber D. Analyss of Gesture and Acton n Techncal Talks for Vdeo Indexng. In IEEE Trans. on Crcuts and Systems for Vdeo Technology. 9. Moran, T. P., Palen, L., Harrson, S., Chu,, P., Kmber, D., Mnneman, S., Melle, W. v. & Zellweger, P., ""I'll Get That Off the Audo": A Case Study of Salvagng Multmeda Meetng Records," n Proceedngs of CHI '97, Atlanta, GA, 1997. 10. Pedersen, E., McCall, K., Moran, T. P., & Halasz, F., Tvol: An electronc whteboard for nformal workgroup meetngs. Proceedngs of INTERCHI 93. pp391-389. 11. Rousseeuw, P. and Leroy, A. Robust Regresson and Outler Detecton, John Wley & Sons, New York, 1987. 12. Stfelman, L., The audo notebook. Ph.D. Thess, MIT Meda Laboratory, September, 1997. 13. Stfelman, L.J., Arons, B., Schmandt, C. & Hulteen, E.A. VoceNotes: A Speech Interface for a Hand-Held Voce Notetaker. Proc. INTERCHI 93 (Amsterdam, 1993), ACM 14. Saund, E. Image Mosacng and a Dagrammatc User Interface for an Offce Whteboard Scanner. Techncal Report, Xerox Palo Alto Research Center, 1999. 15. Weber, K. & Poon, A., Marquee: A tool for real-tme vdeo loggng. Proceedngs of CHI 94. pp 58-64. 16. Wlcox, L. D., Schlt, B. N. & Sawhney, N., Dynomte: A dynamcally organzed nk and audo notebook. Proceedngs of CHI 97. pp 186-193. 17. Whttaker, S., Hyland, P. & Wley, M., Flochat: Handwrtten notes provde access to recorded conversatons. Proceedngs of CHI 94. pp 271-276. 18. Wolf, C., Rhyne, J. & Brggs, L., Communcaton and nformaton retreval wth a pen-based meetng support tool. Proceedngs of CSCW 92. pp 322-329. REFERENCES 1. Abowd, G. D, Atkeson, C. G., Jason A., Brotherton, J. A., Enqvst, T., Gulley, P. & Lemon, J., Investgatng the capture, ntegraton and access problem of ubqutous computng n an educatonal settng. In the Proceedngs of CHI '98, pp. 440-447, May, 1998. 2. Brotherton, J. A. & Abowd, G. D, Rooms Take Note: Rooms take notes! Workng Papers of AAAI 98 Sprng Symposum, March 1998. 3. Brotherton, J. A., Abowd, G. D. & Truong, K. N., Supportng Capture and Access Interfaces for Informal and Opportunstc Meetngs, Georga Tech Techncal Report, GIT-GVU-99-06. (1998). 4. Chu, P., Kapuskar, A., Retmeer, S., and Wlcox, L. NoteLook: Takng notes n meetngs wth dgtal vdeo and nk. Proceedngs of ACM Multmeda '99. ACM, New York, pp. 149-158.