3Book: A 3D Electronic Smart Book

Transcription

1 3Book: A 3D Electronic Smart Book Stuart K. Card, Lichan Hong, Jock D. Mackinlay, Ed H. Chi PARC 3333 Coyote Hill Road, Palo Alto, CA 94304, USA {card, hong, mackinlay, echi}@parc.com ABSTRACT This paper describes the 3Book, a 3D interactive visualization of a codex book as a component for various digital library and sensemaking systems. The book is designed to hold large books and to support sensemaking operations by readers. The book includes methods in which the automatic semantic analysis of the book s content is used to dynamically tailor access. Author Keywords 3D books, 3D UI, spreading activation, sensemaking, ebooks, Electronic Publishing ACM Classification Keywords H.5.2 [Information Interfaces and Presentation]: User Interfaces Graphical User Interfaces; I.7.4 [Document and Text Processing] Electronic Publishing; H.5.4 [Information Interfaces and Presentation]: Hypertext/Hypermedia Navigation; User Issues. General Terms: Design, Human Factors. THE 3D CODEX BOOK Computational forms of documents, document collections, and information workspaces are currently undergoing rapid evolution. Kilgour in his study of the book calls these transformations more extreme than any since the original creations on clay and papyrus in the third millennium B.C. [8]. These transformations seek to increase the accessibility of documents, to decrease their costs, and generally to respond to the glut of information, which is estimated by a Berkley study to be doubling about every three years [9]. One of the more awkward points of the information revolution has been the relationship between paper and electronic media. While once it seemed conceivable that electronic systems would result in the paperless office, what has actually happened is that the role of paper documents has been altered instead of eliminated. In the foreseeable future it seems likely that there will be a coexistence of paper, abstract electronic documents, and digital media forms that are paper-like [14]. One of these forms is the 3D codex book, bound on one side with pages that turn. There are at least three sets of circumstances under which this form is attractive. First is the case of documents that already exist in a physical form. The British Library, for example, has developed a museum-oriented digital book that allows a feel for turning the pages of historical parchment codex manuscripts, including the Lindisfarne Gospels and Leonardo s Notebook [7]. The pages can be turned by finger (in the Library itself) or by mouse in a reduced-size Web version. Carnegie Mellon University has announced the Million Book Project to scan and make available for free 1,000,000 out-of-copyright books [11]. Both Zinio 1 and News Stand 2 have begun to make available to readers electronic versions of magazines and newspapers that specifically resemble the print editions, even though web format electronic versions of these publications already exist on line. They argue that readers prefer the familiar format and the highly-evolved graphic design of a form that resembles the print version. Second is the case of the 3D electronic workspaces that are being experimented with as possible successors to the common desktop metaphor. Examples include Task Gallery at Microsoft [12], Project Looking Glass at Sun 3, the Intel 3D Workspace Project, the Workscape project at Maya [2], and the Web Forager [5]. These workspaces are naturally compatible with a 3D codex representation of a book or other document. A third case is the utilization of 3D virtual books as design elements in their own right. A 3D book can be used as an aggregation structure, a visually compact object with a strong metaphor, whose multiple surfaces and metaphorical expectations can be unfolded into a set of affordances for access and manipulation. An example is the WebBook [5] (Figure 1), where a 3D book metaphor is used to store Web pages. Affordances for access to the Web pages are derived from manual page turning, tabs on pages, and hyperlinked page turning. Closed books are spatially compact, especially when miniaturized by perspective transformation or being stored on bookshelves, but retain identification as the same object

2 reader can turn individual pages or by touching the fore edge of the book, turn many pages as in Figure 3. This gives a physical look and feel to the book. Figure 1. WebBook and Web Forager Figure 2. 3Book open to double page. THE 3BOOK This paper describes the design of a 3D codex book form that evolves beyond the WebBook [5]. In another paper [3] we describe one goal of the 3Book, to be able to handle books of any scale and compare our approach with other attempts to create virtual books. Whereas the WebBook would slow significantly for books more than about 40 pages and one version of the British Museum book required 304 MB for only 20 pages of the book [7], the 3Book has been engineered to handle books indefinitely large. In this paper, we describe the features of the 3Book built around a simple model of the reader s task. Reading a book is more than simply turning the pages and processing the words linearly [14]. We are particularly interested in features that aid the reader in an important class of information-intensive task called knowledge crystallization or sensemaking [4, 10, 13]. In knowledge crystallization tasks, a person forages for information, reads it, organizes and analyses it, then authors some product (or sometimes just acts on it). An example would be accessing literature to write a paper. In our simple model of sensemaking, we include features in the book to aid in foraging, reading, organizing, and writing. We now describe a prototype 3D electronic codex called the 3Book4. To test it against a typical real book, we created a 3Book by scanning in the paper book Information Visualization: User Vision to Think [4] (Figure 2) in addition to several other books. Information Visualization is a challenging test case 700 large-format pages, about 700,000 words. Foraging The fundamental properties of a codex are the abilities to turn to the desired page of text and to read the text. Figure 2 shows the 3Book open to a double page of text. The 4 If the 3 in the name is turned backwards, it spells EBook. Figure 3. 3Book page during turn. Smart Access While retaining a physical look and feel, an electronic book can be augmented beyond what is possible in a physical book, such as by smart indexing. By giving the electronic book the ability to processes its own semantic content, we can build smart indexes and tables of content that dynamically reorganize according to the reader s changing degreeof-interest derived from reading patterns or expressed information needs. For this purpose, we perform semantic computation over the contents of the book as described by Figure 4. First, the images of the book pages are scanned, then OCR ed to produce recognized words. The page images are sent as texture maps to the renderer to be displayed as part of the book. Coordinates of bounding boxes around words are extracted from the OCR results. These bounding boxes enable us to highlight any word on the page. The table of content (TOC) and the index are both also parsed from the book pages and extracted into a XML-format. The text is fed through a word co-occurrence computation that calculates how frequently words occur with each other within a 40-word window. This word co-occurrence com-

3 putation gives us a word symmetric semantic association matrix. Spreading activation on such matrices can be used to model the relative semantic distance between words [1]. More details of the use of this method for making smart indexes and tables of content are discussed in [6]. Page Textures focus+context. Green bars in the margin indicate lines where there is a strong information scent. Clicking on a page number turns the pages of the book to that page. The smart degree-of-interest index speeds readers search by putting relevant entries on a single page and by suggesting relevant associations. These techniques can also be applied to the table of contents (see Figure 6). scan sample Renderer Page Images Text OCR extract parse compute Words + Locations Book TOC Semantic Association Matrix DOI TOC parse Book Index DOI Index Figure 4. Preparation of 3Book contents starting from paper book. Degree-of-Interest Index One of the most useful innovations for books was the invention of the index. This device amplified the virtues of the codex by making it possible to locate names or concepts wherever they might be in the text and to access them using the numbered pages. In contrast to a simple Find command, an index provides context for a term. For example, looking up the term fisheye views in Information Visualization finds index entries Fisheye views, See also focus + context algorithms, Alternative methods, 313 Automatic multi-foevial fisheye, 329 Bifocal lens vs., etc. This two-level index give additional information scent on which pages will be relevant to some purpose. In a paper book, the index is bulky. For example, in Information Visualization, the index is more than 16 pages, 3 columns per page. We can construct a single-page version of the index by allowing the reader to type in terms indicating her interest. A degree-of-interest computation computes which index entries are shown. The method uses spreading activation to compute expanded keyword vectors of the index entries and compare these expanded vectors to the expressed information need vector. Based on this computation, we rank the index entries and prune the low scoring entries until the index entries fit on a single page. During this process, we ensure that index entries with exact keyword hits are activated. These exact hits are highlighted in red, semantic associates are underlined (See Figure 5). For additional details see [6]. As shown in Figure 5, if the reader is looking for fisheye, the returned index will have lines containing the term fisheye, but also entries for related terms bifocal lens and Figure 5. Degree-of-Interest index page. Figure 6. Degree-of-Interest Table-of-Contents page. Reading For a typical 6 by 9 book, the reader can read the book simply by turning the pages and reading content. We provide additional aids for larger books or readers who need large print. Information Visualization is 11 by 17 when

4 opened, but an LCD screen of a large laptop is 9 x 12, only 80% of physical book height, shrinking 10pt type to 8 pt. In fact, the situation is even worse, because room must be left on the screen for other parts of the interface, and even with a resolution of 1600 x 1200 pixels, coarseness of the pixel grid (especially if the book is at a slight angle as shown) further degrades the book image. Organizing Highlighting and Annotation Highlighting. We provide a tool that allows the reader to highlight a rectangular region (called a snippet) in a certain color, as shown in Figure 9. The highlighting color can be interactively modified by the reader. For example, red may be used to highlight all of the paragraphs related to a specific topic. In the figure, the reader has highlighted a snippet commenting on the fisheye method. Bookmarks. Highlighting text automatically creates a bookmark, which the reader can assign a name. Bookmarks enable rapid access back to the highlighted snippet. Figure 9. Highlighting and bookmarks. Figure 7. Rocker page with verso page bent back and recto page in reading position. Our solution is to allow rapid zooming of the image by the reader, when needed, into a rocker page form (see Figure 7) where about 2/3 of the height of a page (for a book the size of our target book) is large and readable. If there is not enough horizontal room, the other page is bent back, allowing all of its page width to be visible. The reader can rock between the two pages (for example, to consult a figure). The reader can also move the focus forward down the page (with a single click) for sequential reading (See Figure 8). Front Cover Zoom Forward Backward Zoom Book Book Book Book Book Book Back Cover Figure 8. States transitions changing pages and views during access and reading of 3Book. Figure 10. Slide-out pages. Multi-page Comparison Sensemaking often requires comparison, and this is especially awkward across pages in a codex. We therefore enable pages to be slid out of the book and compared. Slide-out pages. The reader can slide out copies of pages and place them at the top of the book, allowing the reader to keep them in mind, even though other pages may be visited. Figure 10 shows that pages discussing fisheye and bifocal lens have been slid out from the book. By clicking the mouse over one of the slide-out pages, the reader can turn the book to the location where the slide-out page originally came from. A slide-out page can also be reordered or slid back into the book Page comparison. Slide-out pages allow comparison and access of pages from different places in the book. Comparison of pages creates an even more severe challenge for readability and navigation. To facilitate comparison of the slide-out pages, we implement a multi-focus overview + detail tool as shown in Figure 11. Switching between the overview and this comparison view is carried out in animations. Using this tool, the reader can easily read and com-

5 pare passages in the book, for example passages explaining fisheye and bifocal lens visualization techniques By moving independently, the two focused portion (in white) up and down on the enlarged slide-out pages, the reader obtains a legible view of the two focused areas at the left side of the window. The book is scaled to fit in the upper right corner so as to maintain a constant overview. Figure 11. Comparison viewer. Writing Snippet exporting. To facilitate the communication with applications for authoring or organizing, we provide a cutand-paste extraction tool. Using this tool, the reader can interactively specify a rectangular area on a page, the text in which is subsequently cut and sent to the system clipboard. The text can then be pasted from the clipboard to other applications such as Microsoft Word. CONCLUSIONS Our goal was to develop a 3D codex book component in which it was practical to hold a full-scale book and with features to support basic sensemaking. The 3Book accomplishes this purpose. It can contain a large format book with large number of pages, yet run at interactive speeds on systems relatively modest even by today s standards. 3Book supports foraging through page turning and search, or using the semantics of the content text to create on the fly one-page indexes and tables of content. It supports reading directly by automatic zooming, focus+context display, and reading-order following for large-format books. It supports organization thorough bookmarks, highlighting, annotation, comparison pages, and page slide-outs. It supports writing through the export of text snippets. Finally, we have enhanced the subject index and the table of contents of the books to enable on-the-fly conceptual reorganization of the indexes. It should be noted that the 3Book supports these functions, even when it only starts with the scanned image of a text, not words. We believe the 3Book is promising as a component of digital libraries, e-book commerce, and information workspaces. Further enhancements are possible as 3Book is used in conjunction with its surrounding workspace [15]. REFERENCES 1. Anderson, J. R. and Pirolli, P. L. (1984). Spread of activation. Journal of Experimental Psychology: Learning, Memory and Cognition, 10, Ballay, J. M. (1994). Designing Workscape: an interdisciplinary experience. CHI 94, ACM Conference on Human Factors in Computer Systems (Boston). New York: ACM Press. 3. Card, S. K., Hong, L., Mackinlay, J. D., and Chi, E. H. (2004). 3Book: A scalable 3D virtual book. CHI 2004, ACM Conference on Human Factors in Software. New York: ACM Press. 4. Card, S. K. Mackinlay, J. D. and Shneiderman, B. (1999). Readings in Information Visualization: Using Vision to Think. San Francisco. San Francisco: Morgan-Kaufmann. 5. Card, S. K. Robertson, G, G., and York, W. (1996). The WebBook and the Web Forager: An information workspace for the World-Wide Web, CHI'96, ACM Conference on Human Factors in Computing Systems, New York: ACM Chi, Ed H., Hong, L., Heiser, J., Card, S. K. (2004). ebooks with Indexes that Reorganize Conceptually. CHI 2004, ACM Conference on Human Factors in Software. ACM Press. 7. Chu, Y-C., Witten, I. H., Lobb, R., and Bainbridge, D. (2003). How to turn the page. JCDL 03, Kilgour, F. (1998). The Evolution of the Book. New York: Oxford University Press. 9. Lyman, Peter and Hal R. Varian, "How Much Information", Retrieved from on 29 Feb Pirolli, P. and Card, S. K. (1999). Information foraging. Psychological Review 106(4): Reddy, Raj and St. Clair, Gloriana (2002). The Million Book Project. Project proposal to NSF. Pittsburgh, PA: Carnegie Mellon University. 12. Robertson, G. R. van Dantzich, M, Robbins, D., Czerwinski, M., Hinckley, K., Risden, K., Thiel, D., Gorokhovsky, V. (2000). The task gallery: A 3D window manager. CHI 2000, ACM Conference on Human Factors in Software (1-6 April 2000), Russell, D. M., Stefik, M. J., Pirolli, P., and Card, S. K. (1993). The cost structure of sensemaking. INTERCHI 93, ACM Conference on Human Factors in Computing Systems. 14. Sellen, Abigail J. and Harper, Richard H. R. (2002). The Myth of the Paperless Office. Cambridge, MA: MIT Press. 15. Woodruff, A., Gossweiler, R., Pitkow, J., Chi, E. H. and Card, S. K. (2000). Enhancing a Digital Book with a Reading Recommender. CHI 2000, ACM Conference on Human Factors in Computing Systems (The Hague, The Netherlands), New York: ACM Press.