Viewer-Adaptive Control of Displayed Content for Digital Signage

Transcription

1 A Thesis for the Degree of Ph.D. in Engineering Viewer-Adaptive Control of Displayed Content for Digital Signage February 2017 Graduate School of Science and Technology Keio University Ken Nagao

2 Thesis Abstract Viewer-Adaptive Control of Displayed Content for Digital Signage Digital signage, which is a large display set in a public area, is becoming increasingly common due to the recent developments of underlying hardware technologies and improvements in installing environments. Digital signage has several advantages, as its content can be movies, easily replaced, and controlled over a network. Its content can also be controlled based on viewers requests by accompanying information input devices. Common examples include a mouse, a keyboard, and a touch-panel display. Nowadays, in addition, information input in non-contact ways, such as gesture recognition, has begun to be utilized, thanks to the development of motion capture devices, and interactive digital signage systems whose content can be controlled through gestures, such as swiping one s hands, have been developed. On the other hand, in terms of viewers understanding of displayed content, few studies have been done on the effectiveness of utilizing motion capture devices for controlling displayed content. The aim of this thesis is to analyze the pros and cons of previous utilization of motion capture devices for controlling displayed content to explore effective ways to utilize interactions for digital signage and to propose novel methods to control displayed content by adapting it to viewers. More specifically, first, through a laboratory experiment, it was demonstrated that the previous utilization of motion capture devices to control displayed content can have negative effects on viewers understanding of the displayed content although the utilization can also have other positive effects. Then, the previous utilization was categorized as a content control based on viewers voluntary behaviors and a novel aspect of content control based on viewers involuntary behaviors was proposed by referring to it as viewer-adaptive control. Based on this, a method to adjust the geometry of the displayed content and a method to make the presentation of the displayed content adaptive were developed. In this thesis, their effectiveness was demonstrated by applying the methods to digital signage systems for displaying advertisement and academic posters and conducting laboratory experiments. In addition, the role of a large display in the future when display technologies such as AR glasses are developed was considered through the development of an immersive visualization system.

3 Acknowledgments Firstly, I would like to express my special appreciation to my adviser Prof. Issei Fujishiro for the continuous support of my Ph.D study and research. He has been a tremendous mentor for me, and his guidance has been always valuable. Without his permanent help, this thesis would not have been possible. My grateful thanks also go to my adviser in the Graduate School of Business and Commerce, Prof. Ikuo Takahashi, for giving me valuable opportunities and advices both on research as well as on my career. He made my experience in the Graduate School of Business and Commerce invaluable. I owe my deepest gratitude to Prof. Kwan-Liu Ma for welcoming me to the ViDi lab in the University of California, Davis and it was a great honor for me to study in the ViDi lab under his enthusiastic support. He also gave me constructive comments and warm encouragement as my committee member which made my research more detailed and useful. I would like to express the deepest appreciation to Prof. Michita Imai, Prof. Maki Sugimoto, and Prof. Yoshimitsu Aoki, who gave me insightful comments and helping me widen my research from various perspectives as my committee members. I would like to offer my special thanks to RAs, mentors, and professors in the Program for Leading Graduate School for Science for Development of Super Mature Society for their invaluable and warm supports. I would particularly like to thank every member of Fujishiro Laboratory, Takahashi Seminar, and the ViDi lab for encouraging me and helping me in various ways. Finally, I would like to show my greatest appreciation to my family for their unconditional supports and encouragements.

4 Contents 1 Introduction Digital Signage Advantages of Digital Signage Important Points in Digital Signage Research Area Research Aim Related Work Contact Interaction Technologies in Digital Signage Non-Contact Interaction Technologies in Digital Signage Issues to Be Addressed Effectiveness of Non-Contact Interactions for Digital Signage Previous Research into the Effectiveness of Non-Contact Interactions Hypothesis Hypothesis A: Effects on Viewers Exposures to the Content Hypothesis B: Effects on Viewers Understandings of the Content Hypothesis C: Effects on Viewers Attitudes Toward the Content Experiments Setting Results Discussions Proposal of a New Aspect for Controlling Displayed Content Utilization of Involuntary Behaviors Standpoint of the Research i

5 5 Methods for Adapting Displayed Content to Viewers Approach Method for Situation A Overview Background Algorithm Method for Situation B Overview Background Algorithm Applications and Evaluations of the Methods Applications of the Method for Situation A Settings of the System Evaluation Discussions Applications of the Method for Situation B Settings of the System Evaluation Discussions Conclusion Future Extensions of the Research Future Research Directions Future of Digital Signage Utilization of Large Displays and HMDs for Immersive Visualization Conclusion Publications 101 References 103 ii

6 List of Figure 1.1 Ways of controlling the displayed content and the focus of this thesis The schematic illustration of a contact interaction The schematic illustration of a non-contact interaction The experimental environment with the digital signage advertisement The results of the covariance structure analysis for Hypotheses C1 and C2 (Upper: DS1, Lower: DS2, p < 0.01) The standpoint of the research Two types of situations between viewers and digital signage Hardware configuration of the proposed methods An example illustrating how differences in angle toward the display influence the geometry of the display content in viewers line of sight An executed example of the large digital signage system using the proposed method An example of adapting geometry of the displayed content using the proposed method Processing flow of the proposed method for Situation A Conceptual diagram that shows the separation of sections, each of which has individual target passenger(s) Conceptual diagram of calculating the adapted geometric shape A snapshot of the digital signage system with the proposed method for Situation B in actual use An example of an adaptive conference poster displayed by the proposed method which utilizes the idea of GA iii

7 5.11 Processing flow of the proposed method for Situation B Schematic illustration for describing the processing flow of the proposed method in Figure 5.11 and an example of recognizing viewers involuntary behaviors Example of a gathering evaluation. The system uses Gaussian distributions and accumulates each evaluation by viewers until displaying the content design finishes. The viewer s behavior determines whether the evaluation is unconvincing, and the degree of interest affects the radius and value of the distribution. The other heightmap indicates the attracting evaluation Examples of mutation and crossover of displayed contents Geometric parameters of a section Difference between Case A and Case B mapped by the positions of the passenger. Larger difference means the distortion at the position is less in Case B compared to that in Case A Difference between Setting A and Setting B An example set of displayed contents used for evaluations Prototype implementation and test setting System configuration How the finger tracking method works. The left image shows the extracted depth data of a user s hand, and the right image shows the edge detection and estimation of the position of the user s index finger A user can use his hand to interact with the data by selecting buttons, changing slider values, and moving his open hand for rotating local view.. 99 iv

8 List of Table 1.1 The chapter titles and the relationships between the chapters and the publications The results of the regression analysis for Hypothesis A (DS1 & DS2) The results of the regression analysis for Hypothesis B (DS1 & DS2) The results of analyzing distortions via simulation The results of t-test Results of the user evaluation with the three designs Displayed contents A, B and C, whose examples are shown in Figure 6.4. These results show how many subjects thought Displayed content C is better than A, and Displayed content C is better than B in terms of attractiveness, convincingness and layout v

9 Chapter 1 Introduction

10 Chapter 1 Introduction 2 Digital signage, which is a large electronic display set in the public area, is becoming more and more common due to the recent development of underlying hardware technologies and improvements in installing environments. Digital signage has advantages compared to traditional paper signage, particularly since its displayed content can be controlled based on viewers requests by accompanying information input devices. In addition, information input in non-contact ways, such as gesture recognition, is now starting to be utilized. The aims of this thesis are to discuss the pros and cons of the previous utilization of interaction technologies to control displayed content, to develop new methods for controlling the displayed content to solve the current problems, and to discuss the effectiveness of the methods. In Chapter 1, the introduction and background of the thesis are discussed, and its position and possible contributions are clarified.

11 Chapter 1 Introduction Digital Signage Advantages of Digital Signage In addition to TV and magazines, signage has been utilized to deliver information to the public, including advertisements and news. Traditionally, signage is made of paper, wood, or metal, and the content of signage is a static image. Currently, thanks to the development of the underlying technologies, the types of signage have become more diverse. In particular, there exists a new type of signage in which the content is displayed onto an electronic display, called digital signage. This type of signage has been drawing attention as a new type of information tool to deliver information to the public, in addition to traditional TV or magazines. Digital signage has been considered to have advantages compared to traditional signage. One such advantage is that the contents of digital signage can be movies. In traditional types of signage, the contents are only static images. In contrast, because digital signage consists of an electronic display, its contents can be both static images and movies. Dynamic movies are considered attractive to viewers [55], and thus this is considered one of the advantages of digital signage [36]. The second advantage is that displayed content of digital signage can be replaced easily. To change the content of traditional signage, a provider of the signage not only needs to prepare the content, but also needs to go to the location of the signage to replace it, which takes time. On the other hand, in the case of digital signage, a provider of the signage only needs to prepare the content and send it to the digital signage system over the network. Most digital signage systems are connected to a server over the network so that the provider can control the content using the server. Not only can the provider replace the entire content, but the provider can also modify some parts of the displayed content remotely. The change can be done without requiring time, and thus, digital signage allows the provider to change its displayed content at any time, for example, daily. Moreover, some digital signage systems are connected to each other, and in such cases, the systems can collaboratively change their displayed contents. The third advantage is that digital signage allows a viewer to interact with it. Its content can be controlled based on viewers requests by accompanying information input devices. Such interaction technologies have been studied over the years [53] [98] [11] [14]. In the case of traditional signage, as mentioned before, the content of the signage is a

12 Chapter 1 Introduction 4 static image and cannot be changed even with viewers requests. On the other hand, the content of digital signage can be changed instantly, and the change can be controlled not only by the provider, but also by viewers. A variety of information input devices and interaction technologies for digital signage have been introduced over the years, each with pros and cons. Because traditional display technologies required high costs and large space, digital signage was not common in the past. In contrast, recent developments in hardware technologies have made such costs low and digital signage systems themselves slim. Simultaneously, the environment for installing digital signage has improved, thanks to redevelopments of public spaces, such as stations. Because of these changes, there exist more and more digital signage systems in the public area. According to one survey [43], the size of the global digital signage market was about 15 billion US dollars in 2015 and it is estimated that the market size will grow to around 30 billion US dollars by While we can see digital signage systems in many areas including stations, hospitals, schools, and other places, the survey [43] argued that around 40% of digital signage systems are used in retail environments, mainly for advertising. At the same time, recent developments in display technologies have reduced the cost of large displays. The effectiveness of large displays has been addressed in various fields [95] [16] [74], and the size of digital signage is getting larger and larger. In accordance with the dissemination of digital signage systems and their increasing display sizes, studies on the effectiveness of digital signage have been done, particularly from the aspect of digital signage as an advertising tool. For example, there are several studies on the effectiveness of digital signage in selling environments [66] [40] [83], and some focus on its advertising effects [100]. Moreover, the effects of digital signage on shoppers behavior have been studied [35]. In addition to the aspect of digital signage as an advertising tool, digital signage can be considered from another aspect. For instance, one study indicated that through interactive digital signage, a user model of the city where the digital signage is set can be captured [71].

13 Chapter 1 Introduction Important Points in Digital Signage Digital signage is used for various purposes, including displaying advertisement and city information, and sometimes, for increasing the social interactions of citizens. Digital signage is often referred to as public display, especially when it is used for the last purpose. However, digital signage is signage in nature. Thus, most importantly, digital signage needs to offer content with an appealing design to deliver the message and make viewers understand the content. Therefore, it is important to control the displayed content to make it more attractive and easy-to-understand for the viewers. For such control, there are two important steps: the first is to estimate viewers evaluations of the displayed content, and the second is to improve the displayed content based on such effect measurements so that the content can be attractive and easily understood. When digital signage is used as an advertisement, the first step is often referred to as advertisement effect measurement. In traditional signage, measurements by humans, such as interviews of viewers, have been common. In general, the interviews are done by asking pedestrians how they feel about the signage. Nowadays, thanks to developments of network technologies, online surveys have been also enabled, and the measurements can be made more easily through the Internet. Moreover, currently, other measurement methods that use peripheral hardware for signage have been introduced. These methods require fewer human resources than traditional methods, and one of the common methods among them is the utilization of video cameras. Technologies utilizing video cameras to capture and detect human faces have been widely researched, and such technologies allow the video camera to detect viewers of the signage, including re-visits of viewers [41]. There are some technologies that not only detect viewers of the signage, but also estimate the age, sex, and viewing time of each viewer [45] [73] [56]. Such utilization of video cameras is common in digital signage. One of the general methods for the second step involves improvements made by providers, such as redesign by designers. While these improvements by manual procedures have been common over the years, as mentioned before, digital signage enables the displayed content to be replaced easily, controlled remotely, and interacted with viewers, which have made the improvement procedures more efficient and diverse. For example, similar to the first step, recent developments of network technologies allowed providers to improve the displayed content remotely through, e.g., schedulers and online interface for

14 Chapter 1 Introduction 6 providers [90] [38] [37]. In particular, the last-mentioned advantage in the previous section allows viewers to voluntarily request their ideal content to digital signage. Such interactivity is realized with accompanying information input devices, such as a mouse, a keyboard, and a touch panel display. For example, there are some digital signage advertisements with touch panel displays that can change their displayed content to more detailed content if a viewer touches the corresponding display area. In this way, these types of digital signage can change their displayed content to match viewers preferences without effect measurements, and this reduces labors of the providers. This means that the two important steps mentioned above can be accomplished at the same time by utilizing the interaction technologies. In addition, recent developments of interaction technologies enabled viewers to interact with the digital signage in non-contact ways, such as through gestures. These interaction technologies mainly utilize motion capture devices to detect the movements of viewers, and interactive digital signage in which content can be controlled through gestures, such as swinging one s hands, has been developed. The details of the related work will be discussed in Chapter 2.

15 Chapter 1 Introduction Research Area Based on the descriptions in the previous sections, the ways of controlling the displayed content of digital signage are shown in Figure 1.1. Figure 1.1: Ways of controlling the displayed content and the focus of this thesis. First, the ways of controlling the displayed content of digital signage can be categorized based on whether the control comes from providers or viewers. Approaches originating from providers are mainly by humans, but some of them utilize supporting technologies. On the other hand, viewer-initiated approach mainly utilizes interaction technologies. Traditionally, the interaction technologies were contact interaction technologies. However, thanks to the development of motion capture devices, there are more and more digital signage systems with non-contact interaction technologies. In this context, this thesis will not focus on provider-initiated approach, as it is estimated that there will be more and more digital signage systems that do not require such an approach from providers because this can reduce the labor of providers. Another reason is that while approaches originating from viewers are relatively new, those from providers have been used for a long time and there have been many studies on them.

16 Chapter 1 Introduction Research Aim The aim of this research is three-fold. The first aim is to study the pros and cons of current interactions. While there are more and more digital signage systems with interaction technologies, the effectiveness of these technologies has not been extensively studied. In particular, because non-contact interaction technologies are new technologies, in terms of making the displayed content more attractive and easy to understand for the viewers, studies into the effectiveness of utilizing motion capture devices for digital signage have not been conducted and their effectiveness remains unclear. The second aim is to develop methods to solve the current problems. Here,viewer-adaptive control will be suggested. This approach utilizes viewers involuntary behaviors. The third aim is to consider the applications and evaluations of the methods. There are a variety of applications of the methods, and the examples of the applications will be discussed. Using some examples, the effectiveness of the methods will be evaluated. The remainder of this thesis is organized as follows: First, in Chapter 2, previous digital signage systems are surveyed with a particular focus on the interaction technologies they utilized. Here, the interaction technologies are considered from the aspects of contact interactions and non-contact interactions, and possible usefulness and current issue of non-contact interactions for digital signage are discussed. In Chapter 3, the effects of noncontact interactions for digital signage in terms of viewers voluntary interactions will be studied. In Chapter 4, a new approach to controlling displayed content will be proposed. In Chapter 5, based on the new approach, the methods for adapting displayed content to viewers will be discussed. In Chapter 6, the applications utilizing these methods will be demonstrated and the methods will be evaluated with the example applications. In Chapter 7, with the summary of the research, the conclusion of the research is discussed, also referring to its future work. The chapter titles and the relationships between the chapters and the corresponding publications are summarized in Table 1.1.

17 Chapter 1 Introduction 9 Table 1.1: The chapter titles and the relationships between the chapters and the publications. Chapter number Chapter 1 Chapter 2 Chapter 3 Chapter 4 Chapter 5 Chapter 6 Chapter 7 Chapter title and corresponding publications Introduction Related Work Effectiveness of Non-Contact Interactions for Digital Signage (K. Nagao et al. [7]) Proposal of a New Aspect for Controlling Displayed Content (K. Nagao et al. [1]-[4], [6]) Methods for Adapting Displayed Content to Viewers (K. Nagao et al. [1]-[4], [6]) Applications and Evaluations of the Methods (K. Nagao et al. [1]-[4], [6]) Conclusion (K. Nagao et al. [5])

18 Chapter 2 Related Work

19 Chapter 2 Related Work 11 In this chapter, related works of digital signage are summarized, with an emphasis on interaction technologies for controlling the displayed content of digital signage. Interaction technologies for digital signage can be considered from various aspects, such as the utilization of touch, external devices, tangible objects, and body, as suggested by Ardito et al. [14]. As discussed in Chapter 1, in this thesis, the interaction technologies will be divided based on whether they involve contact or non-contact. First, previous contact interaction technologies for digital signage will be discussed, and then non-contact interaction technologies for digital signage will be discussed from the aspect of the utilization of motion capture devices for controlling its displayed content. Then considering the related works, current issues in controlling displayed content for digital signage will be discussed.

20 Chapter 2 Related Work Contact Interaction Technologies in Digital Signage Contact interactions are interactions that require viewers to touch the input devices of digital signage systems. Figure 2.1. The schematic illustration of a contact interaction is given in Figure 2.1: The schematic illustration of a contact interaction. Common examples of the traditional interaction technologies for digital signage involving contact are a mouse and keyboard, and there are many digital signage systems, or public display systems, that involve such input devices [86] [99]. The problem lies in that extra space is required. Although environments for installing digital signage systems have been improved, such extra space should not be required considering that digital signage systems will be installed in public areas. In addition to a mouse or keyboard, a touch screen has been utilized for interaction with digital signage, and there are many examples of the digital signage systems with a touch screen [76] [9] [11]. In the case of a touch screen, the digital signage system does not require any other peripheral hardware except its display. Thus, a digital signage system with a touch screen does not require any extra space for input devices, and it also allows a viewer to interact with its displayed content in more intuitive ways. On the other hand, when a digital signage system does not have any peripheral hardware, it becomes difficult for a viewer to realize that the digital signage system can be interactive. In such cases, the digital signage system has to imply that it can be interactive in some way, such as

21 Chapter 2 Related Work 13 displaying text making this clear. From this aspect, a touch screen has the disadvantage of display blindness [47] [64]. In addition, such contact interactions require viewers voluntary actions because contact interactions require viewers approach to the digital signage system. Considering that viewers mainly approach a digital signage system when they have some interest in the displayed content, these contact interactions may not be used when the viewers do not have much interest in the displayed content. In addition, most importantly, such a touch screen is not suitable for a large display [15]. In the case of a touch screen, viewers have to approach the screen. On the other hand, viewers would prefer to look at the overview of the displayed content by standing back from the display. Moreover, the cost of a touch screen will be higher in accordance with its display size.

22 Chapter 2 Related Work Non-Contact Interaction Technologies in Digital Signage In addition to contact interaction technologies, thanks to the developments of underlying technologies, a variety of non-contact interaction technologies have been developed. The schematic illustration of a non-contact interaction is provided in Figure 2.2. Figure 2.2: The schematic illustration of a non-contact interaction. A common way to realize such non-contact interactions is to make viewers use some specific devices for non-contact interaction. One example is the interactive public display demonstrated by Robertson et al. [81]. In their research, a Wii remote controller [69], which is originally the controller for TV games, has been utilized for non-contact interactions. For the interaction, viewers are required to have a Wii remote controller for the non-contact interactions.

23 Chapter 2 Related Work 15 Other researchers have attempted to utilize more common devices as input devices for non-contact interaction of digital signage. One example is the utilization of mobile phones for public display, as demonstrated by Ballagas et al. [17]. Ballagas et al. proposed a public display system that allows viewers to interact with the content of the public display using their mobile phones. In the system, the mobile phones of viewers can be used to select items in the content of the public display. They utilize visual codes for communications between their mobile phones and the public display system. Another example is the study done by Boring and Baur [25], which enabled viewers to interact with public displays using mobile phones regardless of their distance from or angles to the displays. Some studies have focused on the utilization of mobile phones for more specific situations, such as in selling environments [58]. While these systems allow viewers to interact with the content in non-contact ways, they still require viewers to have hand-held devices, such as Wii remote controllers or mobile phones. In addition to these methods which require viewers to have hand-held devices, another type of non-contact interaction technology has recently been researched. One example is the utilization of a web camera to allow viewers to interact with digital signage through eye movements [101]. In addition to the utilization of web cameras, there are more and more systems with RGB-D cameras to realize interactions between viewers and a digital signage system without requiring any handheld devices for viewers. An RGB-D camera can capture viewers movements and by segmenting the movement data into gestures, which technologies have been discussed widely [19], and the systems can realize non-contact interactions with the gestures. Microsoft Kinect [59] is common for such RGB-D cameras. Microsoft Kinect is affordable hardware and its validity has been supported through various studies [32] [29]. There are various digital signage systems with such technologies [94] [23] [63], and it is estimated that there will be more digital signage systems with such technologies. In such digital signage systems, viewers can change the content of the digital signage through body gestures, such as swiping their hands. Moreover, researchers have tried to utilize such non-contact interactions to make passengers aware of the display [42], which can be one solution to the problem of display blindness.

24 Chapter 2 Related Work Issues to Be Addressed In Chapter 1, it was discussed that it is important to control the displayed content to make it more attractive and easy to understand for the viewers, and as discussed in this chapter, there are various studies into interaction technologies for controlling the displayed content of digital signage. On the other hand, what current interaction technologies lack is consideration of their effectiveness, which is particularly the case in non-contact interaction technologies from the aspect of viewers mental process. One of few studies is the study by Alt et al. [10], the validity of which will be considered in Chapter 3.

25 Chapter 3 Effectiveness of Non-Contact Interactions for Digital Signage

26 Chapter 3 Effectiveness of Non-Contact Interactions for Digital Signage 18 In this chapter, the pros and cons of utilizing non-contact interactions for digital signage will be discussed. Currently, the main application of digital signage is advertisement and non-contact interactions for digital signage are mainly used for interactions with advertisement content. The examples of non-contact interactions for digital signage advertisement include swiping one s hands to change an advertisement s content and body gestures to move the advertisement s content for fun. However, as previously discussed in Chapter 2, while many digital signage systems with such non-contact interactions have been introduced in the past, there has not been much consideration of the effects of non-contact interactions for digital signage. Thus, in this chapter, the effectiveness of non-contact interactions is considered by using the advertisements as the displayed content and conducting laboratory experiments and hypothesis testing. This chapter is based on the study in one publication [7]. The study will provide an evidence to support the importance of the new aspect of vieweradaptive control, which will be proposed in this thesis later in Chapter 4.

27 Chapter 3 Effectiveness of Non-Contact Interactions for Digital Signage Previous Research into the Effectiveness of Non-Contact Interactions While there are some research findings on the effectiveness of contact interactions such as touch interactions for digital signage [11], there have been few research attempts to consider the effectiveness of non-contact interactions for digital signage particularly from the aspect of viewers mental process. One of the known attempts is by Alt et al. [10], where they deployed an interactive digital signage system that can be interacted with by touching the screen or body gestures, and through their experiments, they suggested that interactions for digital signage have positive effects on viewers in terms of their understanding of the content of digital signage. This research finding gives important insights about the effectiveness of non-contact interactions to the study in this chapter. However, they focused on both contact and non-contact interactions and did not discuss the effectiveness of non-contact interactions deeply. In addition, the content of digital signage was not advertisement, and thus, it is inappropriate to simply apply the results to the study in this chapter.

28 Chapter 3 Effectiveness of Non-Contact Interactions for Digital Signage Hypothesis The aim of this chapter is to investigate the effects of non-contact interactions for digital signage advertisement. The aim is achieved through hypothesis testing from the aspect of consumer behavior. There are a variety of consumer behavior theories to consider viewers mental process in response to advertisement [93]. One of the traditional theories is the AIDMA model proposed by Hall [46]. In the model, it is suggested that in response to an advertisement, viewers engage in the following steps: attention, interest, desire, memory, and action. Another traditional theory is the DAGMAR model proposed by Colley [33]. The model is similar to the AIDMA model, as the model considers that the viewers response to the advertisement is stepwise. In the DAGMAR model, the response is considered to follow the steps of unawareness, awareness, comprehension, conviction and action. In addition, some studies focused on other aspects of viewers response, such as their affective responses [60] [87] and information processing [22]. Through the theories, four common steps of viewers mental process toward advertisement are picked for the hypotheses: exposures to the advertisement content, understandings of the advertisement content, attitudes toward the advertisement content, and attitudes toward the product in the advertisement. From these aspects, the effectiveness of non-contact interactions in digital signage advertisement is considered.

29 Chapter 3 Effectiveness of Non-Contact Interactions for Digital Signage Hypothesis A: Effects on Viewers Exposures to the Content According to the research by Alt et al. [10], non-contact interactions in digital signage have positive effects on viewers understandings of the content. However, as described previously, there are several limitations to the study. For example, they did not consider the unique effectiveness of non-contact interactions deeply. Most importantly, they only used simple icons as the content of digital signage. In the case of advertisements, more complex content should be considered, such as a catch phrase of the advertisement and details of the product in the advertisement. In such a case, it would be more difficult for viewers to understand the content compared to simple icons, and the effectiveness of the interactions would be different. Therefore, the interactions might not have direct positive effects on viewers understandings of the content. Before a viewer understands the content, he needs to be exposed to the content. In this context, it is considered that non-contact interactions in digital signage advertisement have positive effects on exposures to the content, instead of understandings of the content. The effect can be stronger when a viewer interacts more. Thus, Hypothesis A can be stated as follows: Hypothesis A: The more a viewer interacts with digital signage advertisement in non-contact ways, the more the viewer is exposed to the content.

30 Chapter 3 Effectiveness of Non-Contact Interactions for Digital Signage Hypothesis B: Effects on Viewers Understandings of the Content In non-contact interactions, such as body gestures, a viewer is required to move dynamically rather than in traditional contact interactions, such as a touch display. In such a case, viewers need to pay much more attention to moving their bodies. This means that moving bodies can be a cognitive load for viewers compared to traditional contact interactions, such as a touch display, which require less movements for viewers. Many related studies that support this idea have been done. Bettman [22] proposed the information processing theory of consumer behavior, and in the theory, Bettman pointed out the limitations and capacity of information processing of a human. This means that a human cannot focus on everything and selectively focus on something, which is also supported by other previous research [77] [88]. Based on this idea, because non-contact interactions can be a cognitive load for a viewer, when a viewer is focusing on non-contact interactions, a viewer may not pay attention to the content and understand the content, even if the viewer is exposed to the content. In particular, in the case of advertisement, as previously discussed, viewers may need to pay much more attention for understanding the content. Hypothesis B can be stated as follows: Hypothesis B: The more a viewer focuses on non-contact interactions in digital signage advertisement, the less the viewer understands the content in the advertisement.

31 Chapter 3 Effectiveness of Non-Contact Interactions for Digital Signage Hypothesis C: Effects on Viewers Attitudes Toward the Content As demonstrated by Microsoft Kinect for TV games, a moving body itself can have positive effects on an individual s mood. Thus, while non-contact interactions can be a cognitive load for a viewer, non-contact interactions themselves can have positive effects on the viewer s mood, or when the interaction is not interesting, it can have negative effects on the viewer s mood. This means that a viewer can have his own attitude towards interactions, which is the idea proposed by this thesis. While there have been few research efforts on consideration of attitudes toward interactions, there have been some related research findings. One study on the effect of attitude toward the media [62] suggested that the viewers attitudes toward the media will influence their attitudes toward advertising, the advertisement, and the brand in the advertisement. Boushra [26] researched the effects of interactivity in websites and suggested that their attitudes toward a website have positive effects on their attitudes toward the content of the website and toward the brand of the website. In this context, the effects of viewers attitude toward interactions are considered in the two hypotheses C1 and C2: Hypothesis C1: A viewer s attitude towards the non-contact interactions in digital signage advertisement has positive effects on the viewer s attitude towards the content of the digital signage advertisement. Hypothesis C2: A viewer s attitude towards the non-contact interactions in digital signage advertisement has positive effects on the viewer s attitude towards the product in the digital signage advertisement.

32 Chapter 3 Effectiveness of Non-Contact Interactions for Digital Signage Experiments To verify the four Hypotheses A, B, C1 and C2, a laboratory experiment was done, using two digital signage advertisement systems with non-contact interactions with Microsoft Kinect V2. One was digital signage with few interactive contents (DS1) and the other had many interactive contents (DS2). Both digital signage systems were made for this laboratory experiment. In the study by Alt et al. [10], two digital signage systems were used for their experiment, one with and the other without interactive contents. In the study in this chapter, by preparing two digital signage systems with different amounts of interactive contents, the influences of the amount of interactive contents can be considered. In the laboratory experiment, subjects were asked to view one of the digital signage systems (DS1 or DS2), and evaluations from the subjects were gathered Setting A virtual food product was used as the content of digital signage advertisement. The reason why a virtual food product was used for the experiment is to remove the influences of subjects varying levels of knowledge of the advertising product. For DS1, left-hand swipe gesture recognition was implemented to interact with the content. For DS2, lefthand swipe gesture, right-hand swipe gesture, and body up-and-down gesture recognitions were implemented to interact with the content. The digital signage advertisement includes several sentences and pictures regarding the food product. The subjects were 104 undergraduate students and graduate students. Among them, 46 subjects saw DS1, and 58 subjects saw DS2. As the independent values of each subject, the viewing time of each subject, the times each subject interacted with, and the answers to the questionnaire are measured. The former two values are captured through the video camera, which was set in the experiment environment to record the movements of subjects. The questionnaire includes the three categories of questions on subjects attitudes toward the interactions (1-1: evaluations of the suitability of the interactions, 1-2: evaluations of the enjoyment of the interactions, 1-3: evaluations of the usability of the interactions), the three categories of the questions for attitudes toward the content (2-1: evaluations of the quality of the advertisement content, 2-2: emotions regarding the content, 2-3: evaluations of the comprehensibility of the content), and the two categories of the questions for attitudes toward the product in the advertisement content (3-1: evaluations of the quality

33 Chapter 3 Effectiveness of Non-Contact Interactions for Digital Signage 25 of the product, 3-2: feelings to the product). These questions were based on previous research [28] and were on a seven-point Likert scale. Figure 3.1 shows the experimental environment. Figure 3.1: The experimental environment with the digital signage advertisement.

34 Chapter 3 Effectiveness of Non-Contact Interactions for Digital Signage Results Firstly, manipulation check for DS1 and DS2 was done. On average, subjects interacted with the digital signage system 5.93 times (DS1) and 9.71 times (DS2). The t-test results were as follows: t = 3.96, p < It can be confirmed that DS1 and DS2 were set separately. The regression analysis was used to test Hypothesis A through IBM SPSS Statistics Version 22. As a dependent value, the viewing time of each subject, which indicates exposure to the content of each subject, was used. As an independent value, the number of times each subject interacted with was used. Standardized coefficients were 0.62 (DS1, t = 5.28, p < 0.01) and 0.68 (DS2, t = 6.89, p < 0.01). The details are shown in Table 3.1. Hypothesis A is supported both in DS1 and DS2. As for Hypothesis A, the regression analysis was used to test Hypothesis B. As a dependent value, the number of items in the advertisement content each subject can remember, which indicates each subject s understanding of the content, was used. As an independent value that describes each subject s focus on the non-contact interactions in digital signage advertisement, the following value of each subject was used: the viewing time of each subject divided by the number of times each subject interacted with. When the value is low, the viewer is concentrating upon the interactions, and when the value is high, the viewer is not concentrating upon the interactions. Standardized coefficients were 0.31 (DS1, t = 2.17, p < 0.05) and 0.27 (DS2, t = 2.11, p < 0.05). The details are shown in Table 3.2. While the effects were limited to a certain amount, Hypothesis B is supported both in DS1 and DS2. To test Hypotheses C1 and C2, covariance structure analysis was used through IBM SPSS Amos Version 22. Values from the questionnaire were used. Cronbach s α for the questions for attitudes toward the interactions (1-1, 1-2, 1-3) is 0.868, Cronbach s α for the questions for attitudes toward the content (2-1, 2-2, 2-3) is 0.721, and Cronbach s α for the questions for attitudes toward the product in the advertisement content (3-1, 3-2) is These values indicate a certain validity of the analysis [70]. The detailed results of the covariance structure analysis are depicted in Figure 3.2, where ek (k = 1,, 10) mean error variables of the corresponding values. The goodness of fit index was and the adjusted goodness of fit index was These values indicate a certain fitness of the analysis. The results show that while Hypothesis C1 is supported both in DS1 and DS2,

35 Chapter 3 Effectiveness of Non-Contact Interactions for Digital Signage 27 Hypothesis C2 is supported only in DS2. Table 3.1: The results of the regression analysis for Hypothesis A (DS1 & DS2). Unstandardized coefficients Standard error Standardized coefficients (Constant) & & 6.55 The number of times each subject interacted with 4.11 & & & 0.68 t-value p-value (Constant) 5.57 & & The number of times each subject interacted with 5.28 & & Table 3.2: The results of the regression analysis for Hypothesis B (DS1 & DS2). Unstandardized coefficients Standard error Standardized coefficients (Constant) 1.77 & & 0.38 Viewer s non-concentration upon the non-contact interactions 0.06 & & & 0.27 t-value p-value (Constant) 5.17 & & Viewer s non-concentration upon the non-contact interactions 2.17 & & 0.040

36 Chapter 3 Effectiveness of Non-Contact Interactions for Digital Signage 28 Figure 3.2: The results of the covariance structure analysis for Hypotheses C1 and C2 (Upper: DS1, Lower: DS2, p < 0.01).

37 Chapter 3 Effectiveness of Non-Contact Interactions for Digital Signage Discussions Because Hypothesis A is supported, it can be suggested that when an advertisement provider wants a viewer to be exposed to the content, the provider should prepare the digital signage system with non-contact interactivity content that induces many interactions from a viewer. On the other hand, the results of testing Hypothesis B suggest that when an advertisement provider wants a viewer to remember the content in detail, the provider should prepare the digital signage system with limited interactive content when utilizing non-contact interactions. The reason why Hypothesis C2 is not supported in DS1 may be when digital signage has minimal interactive content, a viewer cannot form an opinion or attitude toward the interactions. The results suggest that in the case of utilizing non-contact interactions, a digital signage system with limited interactive content should not be made for the purpose of influencing a viewer s attitude toward the product. In contrast, when a digital signage system has many interactive contents, a viewer s attitude towards the advertising product is directly influenced by his attitude towards the non-contact interactions, and thus a provider should prepare interaction designs that give an impression similar to the impression of the product the provider wants a viewer to have. In particular, non-contact interactions may give a viewer a good impression of modern technology. This means that these types of interactive digital signage systems are suitable for advertising innovative products. In this chapter, it was demonstrated that there are the pros and cons in utilizing non-contact interactions for controlling the displayed content. In particular, although some positive effects of non-contact interactions are supported, non-contact interactions can be a cognitive load for viewers and can have negative effects on viewers understanding of the displayed content.

38 Chapter 4 Proposal of a New Aspect for Controlling Displayed Content

39 Chapter 4 Proposal of a New Aspect for Controlling Displayed Content 31 In Chapter 3, the effectiveness of non-contact interaction for digital signage was empirically studied. Based on the results of the study, it is suggested that non-contact interaction for digital signage can be useful for making viewers to be exposed to the content of digital signage and encouraging viewers to have a positive attitude toward the content, and in some cases, toward the product of the advertisement. At the same time, it is also suggested that non-contact interaction results in viewers poorer understanding of the content of digital signage. This is because non-contact interaction requires viewers to move their body, which is a cognitive load for the viewers. In Chapter 2, it was also discussed that contact interaction for digital signage has some disadvantages. Thus, it can be said that both the utilization of contact interaction and utilization of non-contact interaction have some difficulty in achieving one of the main goals of digital signage, which is to make its displayed content attractive or easy to understand. In this chapter, to cope with the problem, a new aspect for controlling the displayed content will be addressed. This chapter is based on the study in five publications [1]-[4], [6].

40 Chapter 4 Proposal of a New Aspect for Controlling Displayed Content Utilization of Involuntary Behaviors To date, the utilization of interaction technologies by viewers, in other words, the utilization of viewers voluntary interactions has been discussed. On the other hand, not only voluntarily, but also involuntarily behave viewers in front of the digital signage system. For example, if the displayed content of digital signage is attractive, a viewer may stop to read the content, and he may change his head angle to read each part of the displayed content. If a certain part is convincing, he may nod or if it is unacceptable, he may shrug his shoulders. In these ways, there are many contexts from which we can understand viewers mood toward the contents. In fact, there have been several studies utilizing viewers involuntary behaviors for controlling the displayed content. One example of these digital signage systems is the public display system developed by Vogel and Balakrishnan [94], which captures viewers standing points when considering ideal display contents. For example, when the viewer is standing far from the public display, it changes its content to overview content, and when the viewer is standing nearby, it displays more detailed content. This public display system is one example of the utilization of viewers involuntary behaviors, and there are several similar studies [97] [31]. In addition, Schiavo et al. [85] developed a system to estimate how much attention viewers are paying to the public display system by capturing viewers locations and angles toward the display. Ogi et al. proposed a digital signage system that changes its displayed content by recognizing approaching viewers [72]; the content to change is determined based on the viewers interests, which are pre-defined by the viewers. However, most of these studies are only replacing the displayed contents of digital signage systems with pre-defined contents. On the top of that, most of the studies focus only on fundamental elements of viewers involuntary behaviors, such as viewers position and angles, and do not focus on more complicated behaviors, while many of viewers involuntary behaviors may have important meanings. These viewers involuntary behaviors can be captured mainly by utilizing noncontact interaction technologies, such as Microsoft Kinect. This means that by accompanying such non-contact interaction devices with digital signage systems, the systems can consider those viewers involuntary behaviors as their evaluations toward the displayed content. For example, when they are merely roaming around the displayed content, this

41 Chapter 4 Proposal of a New Aspect for Controlling Displayed Content 33 means they may not have interest in the displayed content, and when they remain standing in front of the digital signage and make some gestures, they may have specific feelings toward the displayed content. Based on these, the systems can modify the displayed content for each situation to make the displayed content more attractive and easy to understand. In the case of utilizing viewers involuntary behaviors, the problem of a cognitive load does not exist because what a digital signage system captures are involuntary behaviors. Another advantage of utilizing viewers involuntary behaviors is that the system does not need to wait for viewers voluntary actions or an explicit approach from providers. In other words, the system can control the displayed content autonomously. At the same time, because it utilizes non-contact interactions, it is suitable for a large display. It is affordable and can consider viewers behaviors regardless of whether they are standing close to digital signage or far from it.

42 Chapter 4 Proposal of a New Aspect for Controlling Displayed Content Standpoint of the Research Based on the idea, a new category of the ways to control the displayed content can be considered. It can be referred to as a system-initiated approach. The related studies mentioned in the previous section can be categorized as fundamental studies of a systeminitiated approach. The categorization and the standpoint of the research are summarized in Figure 4.1. Based on the categorization and the standpoint, as a system-initiated approach which utilizes non-contact interaction technologies, this thesis focuses on the viewer-adaptive control of the displayed content for digital signage. Figure 4.1: The standpoint of the research. In Chapter 1, it was mentioned that the first aim of this research is to study the pros and cons of current interactions, and in Chapter 2 and 3, by referring to previous studies and conducting the laboratory experiment, it was discussed that a viewer-initiated approach has some difficulty in achieving one of the main goals of digital signage, which is to make its displayed content attractive or easy to understand. In Chapter 1, it was also mentioned that the second aim of this research is to develop methods to solve the current problems, and in this chapter, a system-initiated approach which is considered to be useful for achieving the goals was proposed. As a system-initiated approach which can

43 Chapter 4 Proposal of a New Aspect for Controlling Displayed Content 35 achieve the goal of digital signage, from Chapter 5, new methods will be proposed for the viewer-adaptive control of the displayed content.

44 Chapter 5 Methods for Adapting Displayed Content to Viewers

45 Chapter 5 Methods for Adapting Displayed Content to Viewers 37 As discussed in Chapter 4, this thesis focuses on the viewer-adaptive control of the displayed content for digital signage. In this chapter, a novel way to utilize non-contact interaction technologies is proposed, which automatically adapts its displayed content to viewers so that viewers can be attracted to the displayed content and can easily understand the content. This chapter is based on the study in five publications [1]-[4], [6].

46 Chapter 5 Methods for Adapting Displayed Content to Viewers Approach The proposed methods utilize viewers involuntary behaviors. When a digital signage system utilizes viewers involuntary behaviors, the system does not require any specific movements of viewers. In such a case, because viewers do not necessarily consider the interactions, their cognitive load is small, which makes them easily understand the displayed content. As discussed, while the possibility of utilizing involuntary behaviors is one of the major advantages of non-contact interactions compared to contact interactions, in the previous utilization of involuntary behaviors for digital signage, most of the studies are only replacing displayed content with pre-defined content. However, it is possible for a digital signage system to automatically adapt its displayed content to viewers on site so that the displayed content becomes attractive and viewers can easily understand it. This can be realized by utilizing motion capture devices to detect viewers behaviors and developing methods for adapting the displayed content to viewers. In particular, by using more complicated behaviors, which most prior studies have not focused on, it is estimated that the digital signage can gather viewers evaluation in more varied ways. To develop the methods, two types of situations between viewers and digital signage are considered: a situation in which a viewer is merely roaming around the digital signage and a situation in which a viewer remains standing in front of the digital signage. In the former situation, a viewer would pass by the digital signage Figure 5.1 (a), while in the latter situation, a viewer would look at the digital signage content in detail Figure 5.1 (b). From now on, the former situation will be referred to as Situation A and latter one as Situation B. In fact, Situation B would happen after Situation A. In this study, by separating the situations and considering a specific method for each situation, the effectiveness of each method can be measured, while the digital signage system is capable of utilizing the two methods at the same time. Both methods are using the same hardware configuration, which is illustrated in Figure 5.2. In the next sections, methods to adapt the displayed content of the digital signage system to viewers in each situation will be detailed.

47 Chapter 5 Methods for Adapting Displayed Content to Viewers 39 (a) Situation A, in which a viewer is merely roaming around the digital signage. (b) Situation B, in which a viewer is standing in front of the digital signage. Figure 5.1: Two types of situations between viewers and digital signage.

48 Chapter 5 Methods for Adapting Displayed Content to Viewers 40 Figure 5.2: Hardware configuration of the proposed methods.

49 Chapter 5 Methods for Adapting Displayed Content to Viewers Method for Situation A When a viewer does not have interest in the displayed content, the viewer will merely roam around and pass by the displayed content. In such a case, contact interactions would be useless because the viewer would not approach the displayed content. At the same time, non-contact interactions with voluntary gestures would not be useful because these require viewers to be involved in the displayed content in the same way as contact interactions. In this context, what is important for making viewers understand the displayed content in the situation is to make them easily see the displayed content. In fact, in the case of utilizing a large display for digital signage, there is a problem of the distortion of the displayed content, which makes it difficult for viewers to understand the displayed content. The problem cannot be solved with the previous methods, which only replace the displayed content with pre-defined ones. In this section, for the situation in which a viewer is merely roaming around the digital signage, the method to adapt its displayed content will be proposed. In the method, viewers movements of positions are considered as viewers involuntary behaviors. The utilization of more complicated behaviors will be discussed in the next section as the method for Situation B.

50 Chapter 5 Methods for Adapting Displayed Content to Viewers Overview Recent developments of underlying technologies have enlarged a display. Because a large display makes it possible to transmit visual information in a wide angle, most digital signage systems utilize a large display. On the other hand, when visual information can be transmitted in a wide angle with a large display, it causes problems. One of the problems is that the angle to the large display varies based on the position of a viewer. When an angle to a large display is different, the shape of the large display content in the viewer s sight is distorted. One example is depicted in Figure 5.3. In particular, when a viewer does not have interest in the displayed content, he will pass by the digital signage. In this case, while passing by the signage, the shape of the displayed content in the viewer s sight is distorted most of the time. Here, when the large display content is seen in a distorted shape, a viewer has difficulty in understanding its content. Such an issue, which is particularly conspicuous in the case of a large display, has been discussed in previous studies. In one study, it is suggested that to solve the problem, the main approaches are to make the viewer move or allow him to control the displayed content so that the distortion would be reduced [24]. It is clear that these approaches can solve the problem, but the viewer is required to take voluntary actions in both cases. On the other hand, in the situation discussed in this section, a viewer may not have interest in the displayed content. Thus, the solution suggested by the previous study cannot be applied to this situation. In addition, when a viewer does not have interest in the displayed content, he will pass by the digital signage in a few seconds. This means that the digital signage system needs to offer displayed content that can be easily understood in a few seconds to make the best use of the time when the viewer glances at the displayed content while passing by. To solve this problem, a novel method to reduce the distortion of the shape of the displayed content in the viewer s sight is proposed. The method does not require viewers voluntary movements and automatically reduces the distortion. Specifically, the digital signage system automatically detects the viewers in front of the digital signage with the accompanying RGB-D camera and modifies the shape of the displayed content so that for the viewer s sight, it looks as if it rotated to face the viewer. This type of method is referred to as anamorphosis, and it has been used traditionally in the field of art [34]. An executed example of the proposed method is depicted in Figure 5.4, and the

51 Chapter 5 Methods for Adapting Displayed Content to Viewers 43 comparison with the original displayed content is shown in Figure 5.5. It is clear that in the viewer s sight, the distortion of the displayed content is reduced. This reduction allows the digital signage system to instantly offer images of the displayed content in which distortion is reduced even if the viewer does not have interest in the digital signage, which will result in improvements of viewers understanding of the displayed content. Figure 5.3: An example illustrating how differences in angle toward the display influence the geometry of the display content in viewers line of sight.

52 Chapter 5 Methods for Adapting Displayed Content to Viewers 44 Figure 5.4: An executed example of the large digital signage system using the proposed method. (a) Original displayed content. (b) Adapted displayed content to the viewer in Figure 5.4. Figure 5.5: An example of adapting geometry of the displayed content using the proposed method.

53 Chapter 5 Methods for Adapting Displayed Content to Viewers Background Digital signage mainly uses a large display as its display device and there have been studies to address the problems of the utilization of a large display [80]. For example, several researchers have indicated the problems associated with displaying the details [12] [74] and with the shape of a large display [96]. In addition, Endert et al. analyzed how visual encodings influence the effectiveness of visualizations with large displays [39]. On the other hand, while the distortion problems resulting from the viewers positions have been also discussed [24], few studies have suggested a solution to the problem. The solutions suggested by the previous studies are to make viewers move in front of the digital signage, and there are a few studies making viewers move in front of the digital signage, for example by using visual cues [8]. These solutions cannot be simply applied to viewers who do not have interest in the digital signage. In the case of digital signage, when a viewer does not have interest in the displayed content, he will pass by the digital signage and see the displayed content obliquely for the majority of the time. Regarding display systems that consider viewers looking at the content obliquely, there have been several studies. Pastoor et al. [75] suggested an interaction technique that utilizes the movement of viewer s gaze point. In addition, there are studies realizing three-dimensional display with motion parallax by modifying its displayed content based on a viewer s face position [91] [48]. However, these studies do not aim to reduce the distortion problem, while the aim of this thesis is to reduce it and make viewers easily understand the displayed content. There are several studies of the user interface in terms of reducing the distortion of the displayed content [65] [78]. As for digital signage, few studies have utilized the idea of anamorphosis. One of the few studies is by Suto et al. [92]. They tried to utilize the idea of anamorphosis to improve viewers attention to the digital signage. On the other hand, in their approach, anamorphosis is applied only to the whole displayed content, which results in an extra blank area in the display, and only one viewer is considered. However, the displayed content can be considered to have some structure and can be divided into a number of sections. In our approach, by applying anamorphosis to each section, the system can make full use of the display area and the displayed content can be applied to multiple viewers.

54 Chapter 5 Methods for Adapting Displayed Content to Viewers Algorithm In this subsection, the proposed method is detailed. The aim of this method is to reduce the distortion of the shape of the displayed content in the viewer s sight. In particular, the method targets viewers who have little interest and pass by the digital signage. Hereafter, such viewers will be referred to as passengers. The hardware configuration of the proposed method is depicted in Figure 5.2, and the processing flow of the proposed method is illustrated in Figure 5.6. First, the provider of the digital signage needs to prepare the original content for the digital signage. The original content needs to be separated into a number of sections. In addition, the position of the RGB-D camera to the large display needs to be inputted. Based on the pre-processing, in three steps, the digital signage system reduces the distortion of the displayed content in the passengers sights. The three steps are as follows: the recognition of passengers positions, calculations of the modified shape of each section, and the decision of the layout. The proposed digital signage system recognizes the passengers positions through utilizing the RGB-D camera. Subsequently, based on the recognized positions, the system calculates how the geometry of each section should be modified. Based on the results of the calculation, the system adapts the shape, scale, and position of each section and decides the overall layout. In this context, when some of the passengers stop in front of the digital signage and start to look at the displayed content in other words, when some of the passengers become viewers adapting the displayed content for other passengers would have a negative influence on the passengers. Thus, when some of the passengers stop in front of the digital signage, the system hibernates the adaptation and gradually restores the original-displayed content. In the proposed method, displaying content can be both static and dynamic images, but in this thesis, static images, such as text and pictures, are mainly considered as displaying content, the reason for which will be mentioned in Chapter 6.

55 Chapter 5 Methods for Adapting Displayed Content to Viewers 47 Figure 5.6: Processing flow of the proposed method for Situation A.

56 Chapter 5 Methods for Adapting Displayed Content to Viewers 48 (1) Pre-processing First, the position of the RGB-D camera to the digital signage system needs to be inputted by the provider. The position will be used to calculate the positions of passengers when the digital signage system is set in the public area. In this method, the right-handed coordinate system is used. In the coordinate system, the origin is the center of the large display. The x axis is the right direction when facing the display. The y axis is the vertical upper direction. The z axis is the direction moving farther from the display. Based on the coordinate system, the system considers the inputted camera position as the coordinate of C(C x, C y, C z ). Next, the provider needs to separate the content into sections based on the context. Each section should be a rectangular area. Based on this separation, the system adapts the shape and layout of its displayed content. An example of the separation can be seen in Figure 5.7. Hereafter, the total number of the sections will be referred to as N. For instance, in Figure 5.7, N equals 4. For each section, the corresponding section number k (k = 1,, N) will be assigned.

57 Chapter 5 Methods for Adapting Displayed Content to Viewers 49 (2) Recognizing Passengers Positions Next, using the data from the RGB-D camera that accompanies the digital signage, the system recognizes passengers positions. This method uses Microsoft Kinect V2 as the RGB-D camera, and through Microsoft Kinect SDK V2.0, the system detects the positions of passengers faces. Because the system detects only faces that look at the RGB-D camera, the system will not detect passengers who are turning their backs or not near the digital signage. The positions of passengers the system detected can be referred to as V vc (V vcx, V vcy, V vcz ). Let v denote the passenger number, which will be assigned to each passenger (v = 1,, M). Here, in the coordinate system, the position of the passenger v is V v = C + V vc. Simultaneously, the system detects whether there is a passenger who stops passing by.

58 Chapter 5 Methods for Adapting Displayed Content to Viewers 50 (3) Calculation of Modified Geometry of Each Section Figure 5.7: Conceptual diagram that shows the separation of sections, each of which has individual target passenger(s). Based on the recognized positions of passengers, the system calculates the modified geometry of each section. First, for each section, the system calculates two parameters of angles and two parameters of distances. The two parameters of angles mean how obliquely each passenger is looking at the corresponding section. One parameter is the angle between the passenger and the left edge position of the corresponding section. The other is the angle between the passenger and the right edge position of the section. Here, the former parameter for section k, passenger v can be considered Akvl ( Akvl < π/2), and the latter parameter can be considered Akvr ( Akvr < π/2). In contrast, the parameters of distance mean from how far the passenger is looking at the corresponding section. The two parameters are the distance between the passenger s position and the position of the left edge of the section, and the distance between the passenger s position and the position of the right edge of the section. The former parameter for section k and passenger v can be considered Dkvl, and the latter parameter can be considered Dkvr. Next, using the parameters, each section determines to which passengers the section will adapt. More specifically, when the section number is k, the system calculates all passengers who satisfy max( Akvl, Akvr ) < Alimit and max(dkvl, Dkvr ) < Dlimit, where Alimit and Dlimit are the parameters that can be determined by the provider and each means the limitation of the modification. For example, when Dlimit is two meters, this means that the section will not adapt to the passengers who are more than two meters

59 Chapter 5 Methods for Adapting Displayed Content to Viewers 51 away from the digital signage. After deciding to which passengers the section will adapt, the system calculates the average position of the passengers to which the system will adapt the corresponding section. The average position of the section k is considered P k. Using the calculated P k, the system calculates the two parameters regarding angles for the section k again. One parameter is the angle between P k and the left edge position of the corresponding section and considered A kl ( A kl < π/2). The other is the angle between P k and the right edge position of the section, and considered A kr ( A kr < π/2). Using these parameters of angles, even if there are a number of passengers, the system can calculate the modified geometry of each section. Figure 5.7 gives a conceptual diagram which indicates that each section has each adaptation target of passengers and each modified geometry. The round numbers mean the passenger numbers, and the round numbers on the sections mean the adaptation target passengers of each section at the moment. The target passengers can be changed through the movements of the passengers. In this method, the modified geometry can be calculated by assuming the situation in which each section is rotated so that it faces the viewer. For this, first, the system calculates the coordinates of each section that was virtually rotated to face the viewer. Here, the rotation angle is min(a kl, A kr ) Bias, where Bias is the value that indicates how largely the modification to the shape will be performed, and can be determined by the provider. The concept is depicted in Figure 5.8. The original rectangular geometry of the section is drawn as the red area. The coordinates of the points of the rectangular geometry are referred to as R k1, R k2, R k3, and R k4, which will be assigned from the upperright point in clock-wise order. The virtually-rotated rectangular geometry is drawn as the orange area, and the coordinates of the corresponding points are referred to as R k1, R k2, R k3, R k4. When min(a kl, A kr ) equals A kl, the coordinates of each point can be calculated as follows: cos(a kl Bias) 0 sin(a kl Bias) R k1 =R k4 + (R k1 R k4 ) (5.1) sin(a kl Bias) 0 cos(a kl Bias) R k2 =R k3 + (R k2 R k3 ) cos(a kl Bias) 0 sin(a kl Bias) sin(a kl Bias) 0 cos(a kl Bias) (5.2)

60 Chapter 5 Methods for Adapting Displayed Content to Viewers 52 R k3 = R k3 (5.3) R k4 = R k4 (5.4) The coordinates of each point can be calculated in the same manner when min(a kl, A kr ) equals A kr. Next, the system considers the lines between P k and R k1, P k and R k2, P k and R k3, and P k and R k4. Accordingly, the system calculates the intersection points between these lines and the surface of the display. These intersection points will be considered as the modified geometry of each section in the display, which is depicted as the yellow area in Figure 5.8. When each intersection point is referred to as R k1, R k2, R k3, R k4, the coordinates of each point can be calculated as follows: R k1 = P k + P kz P kz R k1z (R k1 P k) (5.5) R k2 = P k + P kz P kz R k2z (R k2 P k) (5.6) R k3 = R k3 (5.7) R k4 = R k4 (5.8) In the equations, P kz, R k1z and R k2z mean the z value of the P k, R k1 and R k2. Finally, by considering the correspondences between R k1 and R k1, R k2 and R k2, R k3 and R k3, R k4 and R k4, the system calculates the parameters for the projective transformation. By performing the projective transformation for each entire section, the system calculates the modified geometry of each section. The above calculations are performed when each passenger is passing by and does not stop. When there are passengers who stop in front of the digital signage, the system gradually changes the modified geometry of each section to the original one. These changes are performed by gradually decreasing the value of Bias. The value of Bias also decreases when there are no passengers to adapt.

61 Chapter 5 Methods for Adapting Displayed Content to Viewers 53 (a) Original geometry R k (red area). (b) Virtually rotated geometry in real world coordinate R k (orange area) and adapted geometry R k (yellow area). Figure 5.8: Conceptual diagram of calculating the adapted geometric shape.

62 Chapter 5 Methods for Adapting Displayed Content to Viewers 54 (4) Decision of the Layout As depicted in Figure 5.8, the modified geometry of each section will be larger than the original geometry of each section. Thus, when the geometries of all sections are modified, there may be a section with points outside the display area, or over the other sections. To make all points of all sections inside the display area and not overlapping the other sections, the system shrinks the sections until all points of all sections become so. More specifically, when R k center is the center of modified geometry of each section, the system calculates as follows: R kp scaled = R kcenter + (R kp R k center) scale, (5.9) where R kp scaled (k = 1,, N, p = 1, 2, 3, 4) are the shrunk points of each section. The system calculates the overall scale by considering the largest scale ensuring that all the R kp scaled are inside the display area and not overlapping other sections. Next, the system enlarges the sections that have target passengers and shrinks the sections that do not. Through these processes, the system can make the sections that have target passengers easier to see for passengers. (5) Continuous Modification of Geometries When the above modification of geometries is performed discontinuously, it will be disturbing for passengers. To prevent this, in addition to the above processes, there are two additional processes. First, P k should be changed gradually. When the new target passengers to adapt appear, or some passengers become non-targets, P k will be changed discontinuously. To avoid this, the system limits the amount of changes of P k so that it changes gradually. The next is the gradual change of R k position. By considering R kposition as the initial value of R k position in next frame, the system limits drastic changes of R k position.

63 Chapter 5 Methods for Adapting Displayed Content to Viewers Method for Situation B So far, the situation in which a viewer is merely roaming around and does not have interest in the displayed content has been discussed. In contrast, viewers could have interest in the displayed content, especially when the digital signage is not limited to advertisement, but for other purposes, such as displaying some specific information to viewers. In such case, a viewer would not pass by the digital signage, and he would stay in front of the digital signage and look at the displayed content. Considering such situations, voluntary contact or non-contact interactions can be used, as the viewer has some interest in the displayed content and may voluntarily take actions. On the other hand, as discussed in the previous chapter, especially when utilizing voluntary behaviors for non-contact interactions, the voluntary behaviors would affect viewers understanding of the content. As previously discussed, one of the major advantages of non-contact interactions is that the interaction technology can also utilize viewers involuntary behaviors. In the previous section, utilizing viewer s involuntary behaviors, the method for adapting displayed content was discussed especially in terms of reducing the distortion of the displayed content in the viewer s sight. In this method, movements of viewers positions are utilized as one element of viewers involuntary behaviors. This is because in the case in which viewers do not have interest in the displayed content, they only pass by the digital signage and do not behave in other ways. On the other hand, when viewers have interest in the displayed content and when they approach the digital signage, they behave in more diverse ways rather than just passing by. In this section, for the situation in which viewers have interest in the displayed content, a novel method for adapting displayed content to viewers will be proposed considering the utilization of more varied involuntary behaviors.

64 Chapter 5 Methods for Adapting Displayed Content to Viewers Overview When a viewer has an interest in the displayed content, the viewer will behave in a variety of ways. For example, the viewer may change his head angle to read each part of the content. If that part was convincing, he may nod, or if it was unacceptable, he may shrug his shoulders. Such involuntary behaviors come directly from his motion, and therefore if the digital signage system can capture and recognize them, rather than just utilizing movements of viewers positions in the manner discussed in the previous section, the system can automatically estimate the viewer s mood toward each part of the displayed content. Here, a novel method is proposed that estimates viewers feelings toward the displayed content from their involuntary behaviors and uses their feelings for the automatic adaptation of the displayed content. The adaptation can even create new presentations of displayed content that the provider has not prepared in advance. In particular, while the method for the situation in which a viewer does not have an interest in the displayed content is for the real-time adaptation of the displayed content for viewers who are simultaneously in front of the digital signage, this method focuses on more global adaptation of the displayed content, such as modifying the presentation of the displayed content, and the modification does not have to be done in real-time. This is because when the viewer has an interest in the displayed content and is looking at it, it should not be changed to avoid interrupting the viewer. Instead, the method focuses on adapting the displayed content to local viewers where the digital signage is set, as such local tastes can differ in accordance with the location. The adaptation method for the displayed content utilizes the idea of a genetic algorithm (GA). The system can gather evaluations from a number of viewers without any need to wait for their voluntary interactions. Additionally, those behaviors come directly from their actual feelings toward the displayed content, and thus it is not necessary to check the validity of the evaluation. The method focuses primarily on viewers specific gestures, such as folding their arms and nodding. The digital signage system with the proposed method in actual use can be seen in Figure 5.9, and Figure 5.10 shows an example of an adaptive conference poster in Japanese. Figure 5.11 illustrates the processing flow of the proposed method, and Figure 5.12 shows the schematic illustration of the method and how the viewers behaviors are recognized, which will be discussed later in this chapter.

65 Chapter 5 Methods for Adapting Displayed Content to Viewers 57 Figure 5.9: A snapshot of the digital signage system with the proposed method for Situation B in actual use. (a) A randomly-generated initial displayed content in the first generation. (b) An evolved design in the seventh generation. Figure 5.10: An example of an adaptive conference poster displayed by the proposed method which utilizes the idea of GA.

66 Chapter 5 Methods for Adapting Displayed Content to Viewers Background As previously discussed in Chapter 4, several researchers have considered viewers involuntary behaviors. However, most only focus on fundamental elements of viewers involuntary behaviors, such as viewers position and angles, and do not focus on more complicated behaviors. Regarding the adaptation of the displayed content, most modifications of the previous systems only display displayed content pre-defined by the provider and do not focus on seeking other better presentations of the content. The question regarding how to improve the presentation is difficult because it involves many subjective issues ranging from psychology to other disciplines. There have been some research attempts to address this problem. For example, Singh and Bhattacharya proposed a method to improve the aesthetics of a Web interface utilizing a GA [89], using more than ten geometrical features of the page layout as the evaluations. However, the approach does not take users evaluation of the Web interface into primary account, and they concluded that it is difficult to define the cases in which the approach works well. In addition, in the case of digital signage, it should be considered that local tastes toward the content may differ depending on where it is installed.

67 Chapter 5 Methods for Adapting Displayed Content to Viewers Algorithm The hardware configuration of the proposed method is depicted in Figure 5.2, which is the same as the method for Situation A. As shown in Figure 5.11, the main processing flow in the method is composed of two steps: evaluation and modification. Repeating these steps, more adapted displaying contents can be evolutionarily produced, and the method automatically reaches an ideal displaying content, as illustrated in Figure Here, ideal means that it fits local tastes, with an assumption that different local viewers are likely to have similar moods toward similar displaying contents. The processes terminate the evolution when many similar displaying contents appear or when a large number of viewers attend and they become less likely to behave in a negative way. Our experiments show that around the seventh generation is sufficient, but the detailed conditions of the termination can be determined by the provider. The details of the evaluation of the method will be discussed in Chapter 6. Figure 5.10 illustrates one example of evolved displaying content. In the evolved displaying content, each of the figures in the poster is independently adjusted in terms of size, and each section is repositioned based on its relationships with the others. In addition, some sections are modified for its attractiveness in the evolved displaying content. Considering academic posters as examples of contents of the digital signage, detailed methods of each step will be described in order as follows: defining the content, evaluation, and modification.

68 Chapter 5 Methods for Adapting Displayed Content to Viewers 60 Figure 5.11: Processing flow of the proposed method for Situation B. Figure 5.12: Schematic illustration for describing the processing flow of the proposed method in Figure 5.11 and an example of recognizing viewers involuntary behaviors.

69 Chapter 5 Methods for Adapting Displayed Content to Viewers 61 (1) Defining the Content First, the provider is required to define what he wants to show to the public, dividing the content into smaller sections. For example, an academic conference poster can generally be divided into title, authors, section titles, content of sections, and reference figures. Each section is required to have two kinds of properties: content properties and graphical properties. Content properties are what the provider wants to tell to the public in the corresponding section. For example, in the authors section, content properties are specified as the names of the authors. On the other hand, graphical properties indicate the decoration of the section, such as font, font color, frame color, and the size of the section. Based on these properties, the digital signage system with the method randomly creates some displaying contents, which constitute the first generation of displaying contents. When the total number of sections is n, each section is given one section number from 1 to n, respectively. Here, the provider defines what values the content properties can take because there are various ways to express the message. On the other hand, graphical property values are given to each section by the system automatically. Giving random values to the graphical properties and randomly selecting the predefined values for the content properties of each section, the system creates a fixed number of initial displaying contents. The layout of each displaying content is also randomly set, while there is a constraint that requires the sections title and authors to be set at the top so that each displaying content satisfies the minimal requirements for a layout of an academic conference poster. These generated displaying contents constitute the first generation of displaying contents. The provider can define the total number N of the displaying contents of the same generation. Here, the system gives each displaying content one content design number from 1 to N, respectively.

70 Chapter 5 Methods for Adapting Displayed Content to Viewers 62 (2) Evaluation Step First, the system starts to display one displaying content. Then, the displaying content will switch to another displaying content of the same generation when a fixed time duration ends and at that point no viewer is looking at it. In this way, the system gathers evaluations for each displaying content of the same generation. For a finite period of time, it is estimated that a number of viewers look at the digital signage one after another. Here, the system recognizes various behaviors of the viewers individually for the evaluation. The system secretly captures these behaviors by utilizing the motion capture device, Microsoft Kinect. Here, two types of behaviors are defined: behaviors indicating viewers attention point (attention pointer) and behaviors indicating viewers feeling (feeling indicator). Attention pointers are behaviors such as changing one s head angle and pointing gestures, which can be used for estimating viewers gaze points. On the other hand, feeling indicators are behaviors such as folding one s arms, nodding, and shrugging shoulders, which are deeply related to the viewers emotions. Extensive research has been done to find the relationships between these gestures and feelings [44]. The system can learn the meaning of the behaviors referring to such a kind of database of our own that maintains the relationships. Here, for the system, fifteen varieties of behaviors are categorized. When recognizing these behaviors, the system needs to take cultural differences into account. The meanings of viewers behaviors may differ depending on cultures. Considering that previous research on the detailed meaning of gestures has a limited capability because human affection is a subjective matter, as mentioned before, the system herein focuses only on one culture, i.e., Japanese, and then classifies the meanings according to whether the behavior has positive or negative meaning. Based on these gathered data of behaviors, the system surveys and analyzes the gathered evaluations of each section in two aspects: how attracting and unconvincing the section is. The former is defined as the total of weighted time duration when viewers are looking at the section, no matter how they behave. We will refer to this evaluation as the attracting evaluation of the section. The latter is defined as the total of weighted time duration when the viewers are looking at the section with a negative feeling. We will refer to this evaluation as the unconvincing evaluation of the section. Note that the system considers every behavior the viewer takes and that the negative behavior is only

71 Chapter 5 Methods for Adapting Displayed Content to Viewers 63 considered for the time the behavior is taken. These values are accumulated based on the individual viewers gaze points and distributed to each section finally. More specifically, attracting/unconvincing evaluations are approximated by Gaussian kernels separately. Here, the viewer s gaze points in the displaying content, which can be estimated from the attention pointer, are regarded as the center of Gaussian distribution. Then, the evaluation value and radius of the Gaussian distribution are determined by the degree of interest. When the viewer is getting closer, the radius is set as small and the value is set as large. On the other hand, when the viewer is staying back from the display, the radius is large while the value becomes small. Figure 5.13 illustrates how viewers movements affect the Gaussian distribution. Finally, based on the meaning of behaviors, which can be estimated from the feeling indicator, the system determines whether the evaluation is attracting or unconvincing. Figure 5.13: Example of a gathering evaluation. The system uses Gaussian distributions and accumulates each evaluation by viewers until displaying the content design finishes. The viewer s behavior determines whether the evaluation is unconvincing, and the degree of interest affects the radius and value of the distribution. The other heightmap indicates the attracting evaluation. As shown in Figure 5.13, the system gives values to the displaying content based on the distributions and accumulates the values until the displaying content is changed. When evaluations are finished, each displaying content has two heightmaps: one indicates the attracting evaluation and the other the unconvincing evaluation. Then, in each heightmap,

72 Chapter 5 Methods for Adapting Displayed Content to Viewers 64 the system distributes and normalizes the evaluation values to each section based on the sizes and positions of the sections. Here, let E j,k be the total amount of attracting evaluations that has been given to the k-th section of the j-th displaying content. In a similar way, let U j,k be the total amount of unconvincing evaluations that has been given to the k-th section of the j-th displaying contents. So far, it is discussed that the system determines whether each section is attracting or unconvincing. In addition, the system automatically estimates the relationships between the sections of the content, and this is not required for the provider to define it. The reason is that there may be an ideal structure rather than the one the provider originally thought the content has. For example, in the case of academic conference posters, if there were many professional people in the field of the content near the digital signage, they would find the flowchart image as the reference image of the section system overview, while if most attendees were not conversant with the field so much, they would find the execution example images as the reference images of the section. An appropriate structure depends on a situation where digital signage is set, and the system needs to have an allowance for this. Because a viewer changes his gaze point based on the roles and meanings of the sections, the system surveys the sectional structure utilizing the trajectory of a viewer s gaze point. Specifically, Let F v x (x = 1 n) be a value that indicates how long a viewer has been focusing on the x-th section, where v indicates the identifier of the viewer given by the system. When the viewer v is looking at the f-th section, F v f increases according to the viewing time, and when the section is not looked at, it decreases slowly until it comes to zero as time passes by. In this way, the system memorizes which sections the viewer was looking at before. On the other hand, each section has values that indicate the relationships with the other sections. Let Rx j,k (x = 1,, n) be the values that the k-th section of the j-th displaying content has. When some viewer is looking at the f-th section when the j-th displaying content is displayed, R j,f 1, Rj,f 2,, Rj,f n increase correspondingly based on F 1, F 2,, F n the viewer has, and accumulate the values by every viewer. From these values, the system estimates the degrees of the relationships among sections. For example, when displaying the number j displaying content is finished, if max{r j,f 1,, Rj,f n } is Rm j,f, this means that the f-th section has the closest relationship with the m-th section in the j-th displaying contents.

73 Chapter 5 Methods for Adapting Displayed Content to Viewers 65 (3) Modification Step After all displaying contents are evaluated, they are sorted in an evaluation order based on the weighted time duration each displaying content was viewed without unconvincing feelings. Then, using the above evaluations as a guide, the system generates the next generation of displaying contents in two ways: mutation and crossover, also inheriting the current top-ranked displaying contents. Here, because the weight of information each section has can be different, our system makes modifications to sections with heavy information, such as making the title smaller than other sections with minimal information, keeping information inertia in mind. These amounts of information can be pre-defined by the provider. Figure 5.14 shows examples of the breeding. Figure 5.14: Examples of mutation and crossover of displayed contents. In this step, the system breeds some new displaying contents by changing the property values of each section in the highest-ranked displaying content. This type of breeding is referred to as mutation. Since a section that obtained a high attracting evaluation value is a section viewers found worth looking at, it is important to emphasize this section to make the display-

74 Chapter 5 Methods for Adapting Displayed Content to Viewers 66 ing content more attractive. Therefore, the system changes its graphical properties to increase its attractiveness. Note here that the system simply randomly changes its graphical properties, as the unattractive displaying contents will be exterminated later and only the attractive ones will naturally survive. A section that obtained a high unconvincing evaluation value is a section whose content property is thought to be unconvincing by the viewers, and thus the content properties of the section should be changed to make the content easier to understand, especially in the case in which the section obtained a high value attracting evaluation. In the same manner as before, the system only needs to randomly select the value from the pre-defined values of the content properties, as unconvincing displaying contents will be exterminated later. Specifically, let p j,k be the probability that the modification of graphical properties happens at the k-th section of the j-th displaying content. This corresponds to the calculated attracting evaluation value of each section: p j,k = E j,k max{e j,1, E j,2,, E j,n aggressiveness, (5.10) } where aggressiveness means how aggressive modifications are applied. This value can be defined by the provider. On the other hand, considering both attracting and unconvincing evaluations, a probability that the modification of content property is applied to the section (let p j,k be the probability in same way as p j,k ) can be formulated as follows: p j,k = U j,k p j,k max{u j,1, U j,2,, U j,n }. (5.11) Additionally, the system modifies the layout of displaying content based on the analysis about the relationships of sections. The system puts up a layout where mutuallyrelated sections are set closer. In Figure 5.14, system overview (the section at the lower left) is emphasized because it gathered considerable attention from the viewers. On the other hand, the sentences in approach (the section at the lower right) are replaced with other pre-defined sentence patterns because the previous sentences were thought to be unconvincing by the viewers. Furthermore, the sections that were thought to have the relationships were set closer. In addition to by mutation, the system breeds new displaying contents by crossover. From highly-ranked displaying contents, the system selects two displaying contents for crossover and breeds new displaying contents by combining the property values of each

75 Chapter 5 Methods for Adapting Displayed Content to Viewers 67 section in the two displaying contents. In Figure 5.14, the system breeds a new displaying content whose properties come from the two previous displaying contents. More specifically, for crossover, the system compares each value in the same sections in the two selected displaying contents. The possibility of output values is based on the attracting and unconvincing evaluations of each displaying content so that attracting and convincing sections are selected for the crossover. For example, when the system crossovers two displaying contents, a and b, in the section k, the probability that the graphical property in each displaying content is used is the ratio of E a,k to E b,k.

76 Chapter 6 Applications and Evaluations of the Methods

77 Chapter 6 Applications and Evaluations of the Methods 69 The two novel methods to utilize motion capture devices were proposed in the previous chapter, and the two methods automatically adapt the displayed content of the digital signage system to viewers so that it becomes attractive and viewers can easily understand the content. The two methods allow for two situations of viewers: Situation A, in which a viewer is merely roaming around and Situation B, in which a viewer is staying in front of digital signage. In this chapter, each of the methods will be evaluated with a practical application. This chapter is based on the study in five publications [1]-[4], [6].

78 Chapter 6 Applications and Evaluations of the Methods Applications of the Method for Situation A As previously discussed, viewers may not have interests in the displayed content especially when the content of digital signage is not important for them and they may just roam around the digital signage. As the applications with the method for such situation, a variety of contents can be considered for the digital signage system. One example is advertisement, as previously discussed. Another example is city information. Recently, there are more and more digital signage systems which shows such information to the public as weather, events, and news. In this section, by applying the method for Situation A and using advertisement as the content of the digital signage system, a laboratory experiment of the method will be performed. In addition, the method will be evaluated and discussed. As mentioned in Chapter 5, static images are considered as the displaying content in this thesis, although the method has the capability of displaying dynamic images. It is because when dynamic images are used for evaluation, distortions of the displayed content that the passengers may perceive will differ in accordance with the timing of the dynamic images, as they may contain some frames in which images tend to be perceived as less distorted or considerably distorted. To exclude such time-dependent differences and prepare a fair evaluation environment, only static images are considered in this thesis.

79 Chapter 6 Applications and Evaluations of the Methods Settings of the System A new digital signage system is developed for the evaluation. The digital signage system displays an advertisement, and the displayed content will be modified based on the method discussed as the one for Situation A. The system is developed with Unity Version f1 and written in C. The system runs in the laptop PC connected to a large display, and its execution environment is as follows: OS:Windows 8 64bit CPU:Core i7-3540m (3.00GHz, 4 CPUs) GPU:NVIDIA GeForce GT 640M LE (2GB) Memory:12GB In addition, Microsoft Kinect V2 is used as the accompanying motion capture device with its SDK.

80 Chapter 6 Applications and Evaluations of the Methods Evaluation the important point to evaluate the method is how the method can reduce the distortion of the displayed content. In addition, when the application of the digital signage is advertisement, its advertisement effect should be considered. The laboratory experiment for the evaluation will be discussed in the following order: the evaluation of the position recognition, the reduction of distortion, and the effectiveness of the advertisement. (1) Evaluation of the Position Recognition First, a laboratory experiment was conducted to evaluate the accuracy of the position recognition with Microsoft Kinect V2. A subject was asked to stand at a random position in front of the digital signage within two and a half meters from the Microsoft Kinect V2, and the real position and the position the system recognized were compared. The real position of the subject was measured by a tape measure. The experiment was repeated 60 times. As the result, the max difference was 98mm, the minimum difference was 10mm, and the average of the differences was 42mm. As 42mm equals 3.4% of the 55-inch display, this result suggests that the differences in the position recognitions have little influence on the method, and it can be concluded that the accuracy of the position recognition is sufficient for the method.

81 Chapter 6 Applications and Evaluations of the Methods 73 (2) Evaluation of the Reduction of Distortion As mentioned, the distortion of displayed content in a viewer s sight has a negative influence on the viewer s understanding of the content. Next, the effectiveness of reducing distortion using the proposed method will be discussed to consider whether the method can make the displayed content easier to understand. For this purpose, a virtual digital signage system and virtual passengers were prepared in a virtual world. By simulating passengers sights and the distortion of the displayed content in their sight, we can determine how the method can reduce the distortion. The reason why the virtual digital signage system and virtual passengers are used is by using them, the distortion based on passengers positions can be calculated accurately, and at the same time, a variety of conditions of passengers can be prepared for the evaluation. As the virtual digital signage system, a digital signage advertisement with a 55-inch display was set in the virtual world. Virtual passengers were randomly set in front of the digital signage system. Specifically, they were set randomly in a 3.5m 0.8m rectangular area 0.5m in front of the digital signage. The size and position of the area were set through the assumption that normally passengers will walk in the area in front of the digital signage and the distortion would be greater in the area. To evaluate the method, three cases of digital signage systems and passengers were considered: Case A: without the method Case B: with the method and one passenger Case C: with the method and from two to five passengers In each case, the simulation and calculation of the distortion of the displayed content in passengers sights were conducted 500 times. The distortion of the displayed content is measured from two aspects: aspect-ratio and horizontal keystone parameters. The aspect-ratio parameter means how different the aspect-ratio of the sections in the displayed content in the passenger s sight is compared to the original aspect-ratio of the sections. In contrast, the horizontal keystone parameter means the rate between the lengths of the left edge and the right edge of the sections in the passenger s sight. Let h be the original height of the section and w be the original width of the section, and let h l be the height of the left edge of the section in the viewer s sight, h r be the height of

82 Chapter 6 Applications and Evaluations of the Methods 74 the right edge of the section in the viewer s sight, w be the width of the section in the viewer s sight. Figure 6.1 illustrates the definitions of the variables. The aspect-ratio and horizontal keystone parameters can be calculated as follows: Aspect ratio parameter = w max(h l,h r ) w h Horizontal keystone parameter = min(h l, h r) max(h l, h r) (6.1) (6.2) (a) The geometry of a section viewed in front of the display. (b) The geometry of a section viewed obliquely. Figure 6.1: Geometric parameters of a section. Table 6.1: The results of analyzing distortions via simulation. Geometric parameters Case A Case B Case C Aspect-ratio parameter Horizontal keystone parameter Average Standard deviation Average Standard deviation The calculation results are summarized in Table 6.1, where the larger the geometric parameters are, the better the aspect-ratio and horizontal keystone distortions are; in other words, the aspect-ratio and horizontal keystone distortions are closer to the original.

83 Chapter 6 Applications and Evaluations of the Methods 75 Table 6.2: The results of t-test. Geometric parameters Aspect-ratio parameter Horizontal keystone parameter Case B and Case A Case C and Case A t-value Degree of freedom p-value t-value Degree of freedom p-value The average number of passengers was 3.3 in Case C. From Table 6.1, it is clear that in Case B and Case C, both geometric parameters are larger than those in Case A. To validate the differences between Case B and Case A and between Case C and Case A, t-tests were conducted. The results are summarized in Table 6.2, where in both cases, there were significant differences. Therefore, by applying this method, regardless of whether there is one passenger or multiple passengers simultaneously, changes in the aspect-ratio and horizontal keystone distortions can be improved. At the same time, the change in the perceived-scale of sections in passengers sight was measured. The scale was calculated as follows: scale = (h l + h r)/2 (6.3) As the result, the average scale in Case B is 85.2% of that in Case A, and the average scale in Case C is 93.8% of that in Case A. On the other hand, a previous study [24] indicates that a reduction in the scale does not affect the understanding of the content until the reduction is large. Next, to analyze the results in detail, one specific section is referred for considering the relationships between the geometric parameters and passengers position. The differences in the geometric parameters of the section between in Case A and Case B are visualized in Figure 6.2. The differences are calculated by subtracting the values in Case A from the values in Case B. Thus, when the difference is larger, this means that Case B has larger values compared to Case A. In other words, the geometric parameters are closer to the original. As shown in Figure 6.2, when passengers are on the left or right side of the digital signage, the proposed method has a significant positive effect on the geometric parameters.

84 Chapter 6 Applications and Evaluations of the Methods 76 (a) Aspect-ratio parameter. (b) Horizontal keystone parameter. Figure 6.2: Difference between Case A and Case B mapped by the positions of the passenger. Larger difference means the distortion at the position is less in Case B compared to that in Case A. In the situation in which viewers do not have interest in the displayed content, they merely roam around and pass by the digital signage. Therefore, in such cases, the method appears to work effectively.

85 Chapter 6 Applications and Evaluations of the Methods 77 (3) Evaluation of Advertising Effect The method s effectiveness for reducing distortion was demonstrated through the evaluation in the virtual world. As the next evaluation, its effectiveness for digital signage advertisement will be discussed. A real digital signage advertisement system is prepared for the evaluation. The digital signage advertisement system has two settings as follows: Setting A: Digital signage advertisement system without the proposed method Setting B: Digital signage advertisement system with the proposed method To consider the displayed content in the passenger s line of sight in front of the digital signage advertisement, a first-person view of the passenger was captured through a video camera for both Settings A and B. The two videos recorded the changes in the geometry of the displayed content in the viewer s sight in Setting A and Setting B. The reason why the videos were used for the evaluation was to reduce the difference in subjects eyesights and walking speeds and to prepare fair environments for the evaluation. The content of the digital signage advertisement was selected from the following website: where new products are reviewed. Specifically speaking, because subjects of the evaluation were students in the field of information engineering, as the content, a new laptop PC was selected from the website. The digital signage advertisement includes several sentences and pictures regarding the laptop which were picked up from the official website of the laptop. The digital signage advertisement system consists of a 55-inch large display and the accompanying computer and Microsoft Kinect V2. The two videos were recorded with a passenger who passed by 0.5 meters in front of the digital signage advertisement. The subjects of the evaluation were first told that they would see the video of passing by a digital signage advertisement, and the two videos were shown to each of them. The order of the videos they saw is at random. After they saw the two videos, they were asked to answer a questionnaire, including questions of how easy the content was to understand, how attractive the content was, and how much they were motivated to buy the product in the advertisement as follows: Question A: Comparing Setting A and Setting B, in which was it easier to understand the advertisement content?

86 Chapter 6 Applications and Evaluations of the Methods 78 Question B: Comparing Setting A and Setting B, which do you think is more attractive as an advertisement? Question C: Comparing Setting A and Setting B, which motivated you to buy the product? The total number of the subjects was 41, and their evaluations of Settings A and B were gathered. The evaluation consisted of five-point Likert scale items, and the results are summarized in Figure 6.3. (a) Question A: In terms of the ease of understanding the content. (b) Question B: In terms of the attractiveness of the content. (c) Question C: In terms of stimulation to buy the product. Figure 6.3: Difference between Setting A and Setting B.

87 Chapter 6 Applications and Evaluations of the Methods 79 In response to Question A, about 56% of the subjects answered as B or Slightly B, while about 5% of the subjects answered A or Slightly A. Compared to Setting A, there were more subjects who preferred Setting B. Some of them said, the content was easy to see obliquely, the content was impressive, and so on. Thus, thanks to the distortion reduction as a result of the method, the displayed content became easier to understand for the subjects. As for Question B, about 82.9% of the subjects answered B or Slightly B, while no subjects answered A or Slightly A. On the other hand, as for Question C, about 31.7% of the subjects answered B or Slightly B, while about 68.3% of the subjects answered No difference. Some of the subjects said that while the displayed content was attractive, they are not sure whether the content motivates them to buy the product. Though it is obvious that the method can improve the advertising effect of the digital signage, its effects on viewers motivations to buy the product should be considered from other aspects, such as consideration of each viewer s preference.

88 Chapter 6 Applications and Evaluations of the Methods Discussions In this section, by using advertisements as the content of the digital signage system and applying the method to Situation A, a laboratory experiment of the method was performed and an evaluation was conducted. For making passengers understand the displayed content, it was clear that the method can be useful based on the evaluation. On the other hand, there are several limitations of the method as will be discussed below, as well as recommended directions for future research. As shown in Figure 6.2, while the method is effective for those who pass by the digital signage, the effectiveness of the digital signage can be both larger and smaller depending on in which direction the passengers are walking. For example, when every passenger is walking in the same direction, the method will be effective for each of them. Moreover, it is possible to make the digital signage system adapt to the direction by weighting P k for the direction. In contrast, when each passenger is walking in different directions and there are many passing by persons simultaneously, the method will be less effective. To deal with such cases, the digital signage system should record all movements of passengers to consider the most frequent patterns of movements and adapt to the patterns. In addition, the moving speed of passengers will influence the method s effectiveness. In particular, when passengers are running so fast that Microsoft Kinect cannot recognize, the digital signage system cannot adapt the geometry to the passengers. For considering a variety of passengers and their movements, the digital signage system should be set for the public area for the evaluation. In particular, the environment, such as the space in front of the digital signage and lighting near the digital signage, will be different depending on the place where the digital signage is installed. Such environmental effects should be considered for the real use of the digital signage system. For the questionnaire, a five-point Likert scale was used while in the experiment in Chapter 3, a seven-point Likert scale was used. This is because the analysis in Chapter 3 was the analysis into viewers mental process, and the analysis methods require wide-ranging answers to the questions so that the results can be validated. On the other hand, the aim of the evaluations of the method is to find the differences between digital signage system without and with the method. Therefore, to avoid vague answers to the questionnaire from subjects, rather than using a seven-point Likert scale, the questionnaire used a fivepoint Likert scale and the questions emphasized the comparison of the system with and

89 Chapter 6 Applications and Evaluations of the Methods 81 without the method. As the next step of the study, for the evaluation in the public area, the questionnaire should be modified so that it can help analyze viewers mental process deeper. For a large display, a wide-screen display was used for the evaluation. Though the effectiveness of the method was supported in the setting, the aspect ratio is considered to influence the effectiveness. For example, if the aspect ratio was wider, the method would work in a more effective way, while when the aspect ratio were narrower, the effectiveness of the method would be smaller. In addition, the size of the display could have a major influence on the effectiveness. In the evaluation, a 55-inch display was used and the method s effectiveness was supported. When the size of the display were larger, the effectiveness could be both larger and smaller. Basically, when the size of the display were larger, the space for the modification of geometries would be larger, and thus passengers could easily see the displayed content in less distorted geometries. On the other hand, there can be a problem regarding viewers depth perception due to the unnatural three-dimensional space offered by the adaptation with the method. When the display size were larger and the width of the section were wider, the distance between the left side of the section and the viewer and the distance between the right side of the section and the viewer could be quite different, which might influence viewers depth perceptions. Because a recorded video was used for the evaluation, the problem was excluded in the evaluation experiment. On the other hand, to consider the actual usage of the method for a much larger display, it is required for providers to divide the content into multiple sections so that the width of each section is not so long that it influences viewers depth perceptions.

90 Chapter 6 Applications and Evaluations of the Methods Applications of the Method for Situation B In particular, when the content of the digital signage is not an advertisement and the digital signage offers information that is useful for viewers, viewers may have interest in the displayed content. For example, as the content of the digital signage in such cases, academic posters can be considered. In such cases, viewers will come to see the content, and most will not pass by in front of the digital signage. Another example is digital signage as a menu board. To make their choices, viewers take a close look at the content. Digital signage as a floor guide is another example of applications of the method for Situation B. Viewers need to look at the content of digital signage to find their destinations. In this section, by using an academic poster as the content of the digital signage system and applying the method to Situation B, a laboratory experiment of the method will be carried out. In addition, the experiment results are evaluated. In the same manner as the application of the method for Situation A, displayed content is assumed to consist of static images.

91 Chapter 6 Applications and Evaluations of the Methods Settings of the System For the evaluation, a new digital signage system was developed. The displayed content of the digital signage system is an academic poster. The content was modified based on the method used for Situation B. The system was developed with C++. The system runs on a laptop PC connected to a large display, and the execution environment is the same as the system that was previously discussed.

92 Chapter 6 Applications and Evaluations of the Methods Evaluation While the main point of the method in Section 6.1 was how the method could reduce the distortion of the displayed content, the method discussed in this section is focusing on the adaption of displayed content to viewers by utilizing a wider variety of involuntary behaviors than the previous method. To evaluate it, the other laboratory experiment for the evaluation will be discussed in the following order: the evaluation of behavior recognition and the evaluation of evolved displayed contents. (1) Evaluation of Behavior Recognition First, two aspects of the accuracy of behavior recognition through Microsoft Kinect were evaluated: the accuracy of viewers behavior recognition and gaze estimation. A subject was asked to perform one indicated behavior in front of the digital signage and to look at one indicated section. The experiment was done 100 times, and as a result, the system recognized the behavior correctly in 91% of the tests, which shows the validity of the behavior recognition. Additionally, the system recognized the gaze point correctly in 88% of the tests. This tells us the sufficient accuracy of gaze estimation, while the accuracy rate can be improved utilizing eye movements through the further development of the resolution of the capture device.

93 Chapter 6 Applications and Evaluations of the Methods 85 (2) Evaluation of Evolved Displayed Contents To evaluate the proposed method empirically, some sets of three displayed contents were made: Displayed content A: One from the first generation Displayed content B: Manually created displayed content Displayed content C: One from the seventh generation where the total number N of the displaying contents of the same generation was six and the time interval of each displayed content was ten minutes For producing Displayed content C which is an evolved content, five to ten Japanese students in the field of information engineering who are not professionals in the field of digital signage attended as viewers. The reason why N is small is that displayed contents of academic conference posters are confined within a variety of designs and there could be similar designs even if N was large. Figure 6.4 shows example results of Displayed contents A, B and C, where the academic conference posters include several sentences and images. To compare the attractiveness, convincingness, and layout of Displayed contents A, B and C, these produced displayed contents were shown to 25 Japanese subjects. As was the case with the viewers, these subjects were general students in the field of information engineering, who have an ability to understand the content. As mentioned above, there are different sets of Displayed contents A, B and C, and thus these people did not always see the same set of Displayed contents A, B and C. After this, they were asked how they feel about the three displayed contents. The results in Table 6.3 show that, in every aspect, about 70-85% of the subjects thought evolved displayed contents are better than the initial displayed contents. This suggests most of the displayed contents were improved through the method. Moreover, in every aspect, more than half of the subjects thought that evolved displayed contents were better than the manually-designed contents. Regarding attractiveness, what is interesting is that the number of subjects who thought Displayed content C is better than A and that of subjects who thought Displayed content C is better than B were nearly identical although Displayed content A is randomly generated while Displayed content B is manually set one. This may be because

94 Chapter 6 Applications and Evaluations of the Methods 86 Table 6.3: Results of the user evaluation with the three designs Displayed contents A, B and C, whose examples are shown in Figure 6.4. These results show how many subjects thought Displayed content C is better than A, and Displayed content C is better than B in terms of attractiveness, convincingness and layout. Displayed content C is better than A Displayed content C is better than B Attractiveness 69% 69% Convincingness 80% 57% Mutually-related layout 84% 50% the attractiveness of the manual design reflects the provider s ability, and the individual who created the design in this test was not a professional designer. This makes us reacknowledge how difficult it is to seek an attractive design. However, since about 70% of subjects thought Displayed content C is better than A and B, the system did seem to improve the attractiveness, which validates the effectiveness of the method. On the other hand, as for mutually-related layout, the difference between Displayed content C is better than A and Displayed content C is better than B is large, and while about 85% of subjects thought Displayed content C is better than A, only about 50% of subjects thought Displayed content C is better than B. This means while layouts were considerably improved through the evolutionary method, the improved layout is still as good as the manual layout in most cases.

95 Chapter 6 Applications and Evaluations of the Methods 87 (a) Example of Displayed content A: One from the first generation. (b) Example of Displayed content B: Manually created displayed content. (c) Example of Displayed content C: One from the seventh generation. Figure 6.4: An example set of displayed contents used for evaluations.

96 Chapter 6 Applications and Evaluations of the Methods Discussions Similar to the evaluations of the method for Situation A, there were some differences between the questionnaire used in Chapter 3 and the evaluations of the method for Situation B. This is because the aim of the evaluations of the method is to determine the differences between the use of the method or its absence. To examine the effectiveness of the method in detail, such as its effects on the mental process of viewers, the digital signage system should be set in a public area, and the questionnaire should be similar to the one used in Chapter 3, although it is not necessary for considering the method s effectiveness. The results indicate that the method s effectiveness was acceptable, while there are some allowances for extensions. Most importantly, viewers mood can be considered from other contexts. One point is the interactions between viewers. In the method, each viewer is regarded as an individual. However, when viewers are looking at digital signage, in many cases, they interact with each other, inviting others to look at the content, discussing the content, and so on. It is important for the method to consider the mutual influences between individuals to ensure deeper understanding of viewers on-site mood as hardware technologies develop. Another issue to consider is individuality. In the method, there is an assumption that there are a reasonable number of people who have similar tastes, especially in the case in which the digital signage is set at a specific location, such as a conference, and the results show pragmatic effectiveness. However, when there is a small number of people, or when we extend the method to more generalized applications, the system can be made more efficient by considering individual viewers differences, by using a database managing individual differences and modifying the displayed content based on individual viewers preferences. Additionally, by combing other areas of information engineering, the system can be extended. For example, with natural language analysis, the system can automatically estimate and analyze the amounts of information of the sections. Moreover, if the system can clearly understand whether viewers gestures are caused by its content or visual properties, the system can modify the displayed content in a more effective way by dealing with the semantics of behaviors more plausibly.

97 Chapter 7 Conclusion

98 Chapter 7 Conclusion 90 Thus far, two methods have been introduced to make the displayed content more attractive and easy to understand. In addition, the applications and the evaluations of the two methods have been discussed. In this chapter, regarding future digital signage, the future extension of the research will be discussed by introducing the substantial study [5] which indicates the possible future of the digital signage system. Finally, the conclusion of the research will be mentioned.

99 Chapter 7 Conclusion Future Extensions of the Research Although the effectiveness of the two methods has been supported by the experiments, further research needs to be done. In this section, in addition to the discussion in Chapter 6, future works of the two methods will be discussed. In addition, the study which indicates the possible future of digital signage will be introduced Future Research Directions One of possible directions for future research on the proposed methods is to modify them to specific display content and/or situations. While the proposed methods are applicable to both static and dynamic images, only static images are used for the evaluations. Because one of the advantages of digital signage is that it is capable of dynamic images, evaluations of the methods with dynamic images should be made although there remain several difficulties, which were mentioned in Chapter 6. In addition, static images are considered to consist of several sentences and pictures. Here, the effectiveness of the proposed methods can be different by whether the images mainly consist of texts or pictures, and such difference among contents should be taken into account. In the method for Situation B, it is discussed that weight of information each section has can be different and that keeping information inertia in mind is important. This aspect should be considered more deeply for modifying the proposed methods to specific display contents. Moreover, the two methods were developed mainly for two situations; however, there are a variety of other possible situations. The methods can be modified for each of the situations. Another future area of application of these methods involves the consideration of other behaviors of viewers. In front of digital signage, viewers involuntarily behave in diverse ways. Compared to previous studies, more behaviors have been considered in this research. However, there are other behaviors, such as interactions between viewers, that still need to be studied. Viewers can be categorized by whether they are individuals or groups. Adapting the displayed content to groups is one of the underlying challenges in the research in this thesis. In Chapter 2, the point of display blindness was discussed. Because the methods discussed in this thesis do not require any voluntary behaviors of viewers, the problem of display blindness was small. On the other hand, there can be additional affordances for digital signage. For example, as discussed in Chapter 5, several studies have been done

100 Chapter 7 Conclusion 92 for making viewers move in front of the digital signage by using visual cues [8]. Such aspects should also be considered for the proposed methods. In addition, the proposed aspect of viewer-adaptive control can be utilized simultaneously with provider-initiated approach and viewer-initiated approach. By combing these approaches, the digital signage system can consider viewers involuntary behaviors before, during and after his voluntary interactions with the system, as in Chapter 3 it was demonstrated that such voluntary interactions with the system also have some positive effects to viewers. Another important issue to be discussed is the influences of installing environments of digital signage. In this thesis, the evaluations were made through laboratory experiments, while there exist studies on the effectiveness of the spatial layout around digital signage [20]. Moreover, when a digital signage system is set outdoor, surrounding environment s influence on the digital signage system will be larger. For example, the sunlight has big impact on the brightness of the display of the digital signage system, and to deal with the problem, a brightness sensor should be attached to the system so that the system can automatically adapt the brightness of the display to the surrounding brightness. In terms of installing environments, there is the study into the relationships between viewers behaviors and interactive cylindrical screens [23]. The shape of digital signage also influences the effectiveness of the methods.

101 Chapter 7 Conclusion Future of Digital Signage Moreover, it is important to consider the future of digital signage to discuss the future extensions of the research, particularly in terms of related hardware technologies. One of the surveys [43] points out the importance of innovative display hardware, such as 4K and 3D displays for digital signage. The proposed methods are applicable to both 4K and 3D displays, while in the case of 3D displays, the effectiveness of the method applied to Situation A should be researched from the aspect of viewers perceived 3D space, which was discussed in Chapter 5. In addition, current digital signage consists of one display. This means viewers are sharing only one display. However, many portable display devices have recently been developed. This suggests that in the future, each viewer will have an individual display as well. The individual display may be an HMD or other forms of display. In fact, even in such cases, to have one large shared display is effective, especially for the case in which there are many viewers. To demonstrate their possible effectiveness, a new system which utilizes both large displays and HMDs will be proposed in the following sub-section. The study focuses on the immersive visualization, but suggests the usefulness of the combination. The study is based on one publication [5].

102 Chapter 7 Conclusion Utilization of Large Displays and HMDs for Immersive Visualization (1) Introduction Numerical simulation is an important tool for conducting scientific research. Large-scale volumetric data is generated from the simulations of 3D physical phenomena and complex chemical processes. To validate and understand the simulation results, scientists rely on visualization and analysis tools. While many advanced flow visualization and analysis methods have been introduced over the years, the growing scale and complexity of data out of the state-of-the-art scientific simulations continuously demand new, enhanced ways for scientists to depict and dissect data. In particular, as high-resolution large displays and VR devices are now becoming more affordable, it is timely to realize and exploit immersive visualization and the analysis of complex data. New opportunities are for both visualization researchers and users. The value of VR to 3D flow visualization has been demonstrated by many researchers since the 1990 s. One of the early systems is CAVEvis system [49], which utilizes CAVE system as an immersive environment for interactive visualization of 3D data and can display isosurface and cutplane of the data in stereoscopic images. The other system to be mentioned is the Virtual Wind Tunnel [27], which is a virtual reality system displaying the 3D information of complex fluid flows on a stereoscopic head-tracked display with various visualization techniques including isosurface and streamlines. VR has not been extensively used in scientific visualization and analysis tasks because it requires high speed, high resolution, high quality rendering solutions and easy input methods to be truly useful. Large-scale data visualization tasks simply make the problem more challenging. However, rendering hardware requirements have become more manageable when HMDs are used together with a large display instead of CAVE like environments. Intuitive and easy-to-use input methods for interacting with the data and specifying intent for analysis tasks are not generally available. Wireless input devices and touch interfaces have been used, but full immersion would require a more natural interface based on simple gesture input. Recent motion capture devices such as Microsoft Kinect enable body gesture inputs, but effective way to utilize body gesture inputs for particular scientific analysis tasks remained to be developed.

103 Chapter 7 Conclusion 95 It is possible to address some of the challenges by utilizing the increasingly affordable see-through HMD devices together with a low-cost tiled HDTV display to provide a usable immersive analytics solution for studying large-scale complex flow field data. With the large display to provide a high-resolution overview of the data, the user can freely choose a small area to explore and analyze using a see-through HMD in 3D stereoscopically with gesture input. During such local exploration and detail analysis, the user can apply a newly derived visualization parameter setting to the large display for a new overview. In this way, computational costs become more manageable because realtime rendering and response are only required to cover a small screen space and a subset of the data. In this sub-section, for considering future of digital signage, the preliminary design and results of the system are demonstrated, which prototype implementation can be seen in Figure 7.1. Figure 7.1: Prototype implementation and test setting. (2) Background As previously discussed in this thesis, large displays are becoming more and more common, and used for a variety of purposes [68] not only for digital signage. With its large size display area and high resolution, large displays allow users to gain both an overview and the detail of the displayed content [13]; in particular, it is pointed out [16] [67] that search and comparison tasks can be effectively done with large displays to support visualization and analysis tasks. One of the well-known large display systems for data visualization is

104 Chapter 7 Conclusion 96 SAGE2 [57], a complete redesign and reimplementation of SAGE [50]. Using a large high-resolution tiled display for interactive visualization of large-scale data demands high-performance software and hardware solutions [61] [54]. To support stereoscopic viewing the computing requirements and cost could become many times higher. As demonstrated by Reda et al. for the immersive visualization of large-scale atomistic simulation [79], a scalable solution is possible but at the cost of custom software development and dedicated hardware setting. A variety of technologies for interacting with a large display have been introduced in the past, as previously discussed in Chapter 2 from the aspect of digital signage. In the same way in Chapter 2, these technologies can be divided into two categories: in contact way or in non-contact way. Hand-held devices such as mobile phones have been used to interact with a large display in non-contact way [17] [84]. In addition, the possibility of attaching sensors to hand-held devices, such as pencils, has been discussed [18]. On the other hand, some research results indicate that hands-free devices can encourage more frequent interactions of users than hand-held devices [52]. A touch display is one such hands-free device for contact interactions, but the cost can grow quickly with its display size, which was also discussed in Chapter 2 from the aspect of digital signage. Putting costs into perspective, inexpensive motion capture sensors is quite usable and have been utilized for interacting with a large display in non-contact way, not only for digital signage. In the system by Johnson et al. [51], together with other devices such as joysticks and mobile phones, Microsoft Kinect is used so that users can intuitively interact with the visualization by gestures. However, effective way to utilize such body gestures for analysis tasks remains to be developed. There are data visualization systems using a large display together with portable display devices for supporting analysis tasks. For example, in [30], mobile phones are used for displaying details of data visualization as well as for interaction. With their increasingly affordable prices, HMDs also become attractive. HMDs enable stereoscopic viewing for data visualization when the screen of the large display is not 3D capable [21]. In the work by Rodrigue et al. [82], the potential of coupling a large display with mobile devices is discussed from the aspect of its high versatility for mixed reality environment. (3) Designs The overall system configuration is depicted in Figure 7.2.

105 Chapter 7 Conclusion 97 Figure 7.2: System configuration. The proposed system consists of a client-side system and a server-side system, as shown in Figure 7.2. A large tiled display is connected to the server, and HMDs are connected to clients. The server and clients can communicate with each other through a wireless router. For example, a client sends new visualization parameters to the server, and the server responds to it by updating the visualization. The server delivers stereoscopic images of selected views to the client, and monocular overview image to the large tiled display if requested. The view of the client can be selected through utilizing the image captured by RGB camera on the HMD of the client. It is noted that a client does not render any visualization. Rendering is completely done by the server. Each user can interact with the data visualization using his hand. The method is as follows. Firstly, the client gets raw depth data from its depth sensor attached to the see-through HMD, and then from the data the client excludes every pixel whose distance from the HMD is greater than the pre-defined threshold so that the client can get the

106 Chapter 7 Conclusion 98 depth data of the user s hand. Next, the client applies edge detection to the depth data. Specifically, the client uses the Sobel operator for the edge detection method. Among the detected edges the client regards the topmost one as the position of the user s index finger. Figure 7.3 shows how the finger tracking method works. At the same time, the client determines whether the hand is open or not by calculating the number of edges next to the position of the user s index finger. Through the hand tracking by the client, a user can use his hand to interact with the data using the interface in the HMD, such as by selecting buttons, changing slider values, and moving his open hand for rotating local view. Examples of the interactions are depicted in Figure 7.4. Figure 7.3: How the finger tracking method works. The left image shows the extracted depth data of a user s hand, and the right image shows the edge detection and estimation of the position of the user s index finger. (4) Tests and Discussion A prototype system is created based on the design. As shown, a variety of small details can be easily found with the high-resolution large display. Simultaneously, local data exploration with HMDs in stereoscopic view is achieved in acceptable rendering rates (10-20 frames per second), allowing users to apply a newly derived visualization parameter setting to the large display for a new overview through hand gestures. This study suggests that the utilization of the affordable see-through HMD devices together with a large display can play an important role for analysis tasks. It can contribute to viewers immersive experiences and also allow each viewer to have individually-adapted view together with one shared view. Still, the idea can be also applied to other contents, including digital