The Uncanny Valley: Effect of Realism on the Impression of Artificial Human Faces

Transcription

1 Jun ichiro Seyama* Department of Psychology Faculty of Letters University of Tokyo Hongo, Bunkyo-ku Tokyo , Japan The Uncanny Valley: Effect of Realism on the Impression of Artificial Human Faces Ruth S. Nagayama Department of Humanities and Social Sciences Shizuoka Eiwa Gakuin University Abstract Roboticists believe that people will have an unpleasant impression of a humanoid robot that has an almost, but not perfectly, realistic human appearance. This is called the uncanny valley, and is not limited to robots, but is also applicable to any type of human-like object, such as dolls, masks, facial caricatures, avatars in virtual reality, and characters in computer graphics movies. The present study investigated the uncanny valley by measuring observers impressions of facial images whose degree of realism was manipulated by morphing between artificial and real human faces. Facial images yielded the most unpleasant impressions when they were highly realistic, supporting the hypothesis of the uncanny valley. However, the uncanny valley was confirmed only when morphed faces had abnormal features such as bizarre eyes. These results suggest that to have an almost perfectly realistic human appearance is a necessary but not a sufficient condition for the uncanny valley. The uncanny valley emerges only when there is also an abnormal feature. 1 Introduction Roboticists have attempted to construct humanoid robots whose physical appearance is indistinguishable from real humans (e.g., Kobayashi, Ichikawa, Senda, & Shiba, 2003; Minato, MacDorman, et al., 2004; Minato, Shimada, Ishiguro, & Itakura, 2004). However, Mori (1970) warned that robots should not be made too similar to real humans because such robots can fall into the uncanny valley, where too high a degree of human realism evokes an unpleasant impression in the viewer (see also Norman, 2004; Reichardt, 1978). To summarize his informal observations and predictions of how a robot s degree of realism in physical appearance (or humanlikeness) can affect a human observer s impression of the robot, Mori introduced a hypothetical graph of the impression of pleasantness as a function of the degree of realism (Figure 1). Although the degree of realism was defined as the robot s physical similarity to real humans, the impression of pleasantness for the ordinate of the graph was not clearly defined. One definition consistent with Mori s conjecture is that the ordinate numerically represents degrees of any pleasant impressions (e.g., attractive, pretty, and fascinating) in the positive range and any unpleasant impressions (e.g., unattractive, ugly, and uncanny) in the negative range. Presence, Vol. 16, No. 4, August 2007, by the Massachusetts Institute of Technology *Correspondence to Lseyama@mail.ecc.u-tokyo.ac.jp. Seyama and Nagayama 337

2 338 PRESENCE: VOLUME 16, NUMBER 4 Figure 1. Mori s (1970) uncanny valley. Human observers impression of artificial objects is plotted hypothetically as a function of the degree of realism (the artificial objects similarity to real humans). Mori assumed that the level of impression would drop suddenly at a relatively higher degree of realism, and called this dip the uncanny valley. Regions A and B are realism ranges related to discussion in Experiment 2. Modified after Mori (1970). In Mori s graph, a positive impression of pleasantness of a robot increases with an increasing degree of realism. For example, Honda ASIMO (Sakagami et al., 2002) may be more attractive than industrial robots. Mori claimed, however, that human observers have exceptionally unpleasant impressions of robots that have an almost, but not perfectly, realistic human appearance. This effect is shown in Mori s graph as a negative peak at a relatively high level of realism (Figure 1). Mori called this negative peak the uncanny valley, by analogizing the shape of his graph to a mountain. Along the abscissa of Mori s graph, not only robots but also various kinds of humanlike artificial objects (e.g., dolls and prosthetic hands) were sorted in subjective order of degree of realism. Thus, Mori s hypothesis is not limited to robots but is also applicable to any type of artificial humanlike object. The physical appearance of robots that are supposed to communicate, cooperate, and coexist with humans should be designed with due consideration of the emotional and psychological impact on human observers. The same holds true for the physical appearances of agents in virtual reality and characters in computer graphics movies. Mori s hypothesis has been adopted as a guideline for designing the physical appearance of robots (Cañamero & Fredslund, 2001; DiSalvo, Gemperle, Forlizzi, & Kiesler, 2002; Fong, Nourbakhsh, & Dautenhahn, 2003; Hara, 2004; Hinds, Roberts, & Jones, 2004; Minato, MacDorman, et al., 2004; Minato, Shimada, et al., 2004; Woods, Dautenhahn, & Schulz, 2004) and agents in virtual reality (Aylett, 2004; Fabri, Moor, & Hobbs, 2004; Wages, Grünvogel, & Grützmacher, 2004). According to Mori s hypothesis, designers should seek a moderate level of realism (e.g., within the range B in Figure 1) for the physical appearance of robots and virtual reality agents in order to avoid falling into the uncanny valley. However, the validity of the uncanny valley has not been confirmed with psychological evidence. Thus it is uncertain whether the uncanny valley actually emerges at certain realism levels. In the present study we measured observers ratings of pleasantness of facial images with varying degrees of realism. The degree of realism was manipulated by morphing between images of artificial and real human faces: the degree of realism was represented as a morphing percentage (% of real human). The artificial face images used in this study were photographs of dolls and computer graphics images of human models. Participants in each experiment rated their impressions of the pleasantness of the morphed images on a five-point scale. The rated scores were plotted against the degree of realism in order to empirically validate Mori s hypothetical graph. In hypothesizing about the emergence of the uncanny valley, Mori assumed that the impression of pleasantness is zero (i.e., neutral impression) when the realism of robots is extremely low (e.g., industrial robots) and highest for a perfectly realistic human appearance (Figure 1). In other words, he assumed that impressions of pleasantness increase monotonically with an increasing degree of realism, except for the uncanny valley. However, this assumption may not be valid. In fact, some robots, dolls, and human characters in computer graphics films seem highly pleasant although very unrealistic, while others are unpleasant. Similarly, humans

3 Seyama and Nagayama 339 vary in impressions of pleasantness although all are realistic. Thus, the realism-pleasantness graph might not necessarily show a monotonic increasing trend. However, in the present study, we focus only on whether the negative peak, that is, the uncanny valley, actually emerges in an empirically obtained realism-pleasantness graph. 2 Method 2.1 Participants The experiments were web-based. Each participant accessed a web page using a web browser. They were first guided to pages where they read instructions written in Japanese, and then they decided whether or not to participate in this study. Although most of the participants were students at the University of Tokyo, Shizuoka Eiwa Gakuin University, or Tokyo National University of Fine Arts and Music, some of them were identified only by their self-reported age and gender. 2.2 Stimuli The stimuli were frames of image sequences in which an artificial face was gradually morphed into a real face. Examples of the stimuli are shown in Figures 2, 3, 5, 7, and 9. To define the correspondence between the two source images (i.e., artificial and real facial images), landmarks on each face were manually chosen. For the stimuli used in Experiments 1 3, the number of the landmarks was 9 for each eye, 8 for each eyebrow, 9 for the nose, 14 for the mouth, and 26 for facial contour. For the stimuli used in Experiment 4, the number of the landmarks varied from 222 to 325 depending on the details of the image content. Although the internal facial features and the face lines in the lower halves of the faces corresponded as precisely as possible, the outer contours of the upper halves of the faces (i.e., hair shapes) and the hairlines corresponded only roughly, because they were drastically different. Morphing software transformed positions and/or pixel values between corresponding points of the artificial and real faces. Morphing ratios, which controlled the magnitude of the Figure 2. A screenshot of a browser window during Experiment 4. morphing, were defined as the percentages of real human. Thus, an image with a morphing ratio of 0% would correspond to an unmorphed image of an artificial face, and an image with a morphing ratio of 100% would be a perfectly realistic human face image. Details of the morphing procedure differed among the experiments, which will be described later. We obtained written consent to use the photographs of the human faces for academic purposes. Each face was presented on a uniform square background with a height and width of 256 pixels. The size and orientation of each face were normalized so that the eyes were aligned along a horizontal line, and the distance between the left and right pupils was 60 pixels. The actual size of the stimuli measured in visual angles is not known, because it depended on the viewing condition of each participant when he/she accessed the web page for the experiments. 2.3 Procedure Each participant executed JavaScript programs in a web browser. In each trial a stimulus image and five buttons were presented in the web browser window (Figure 2). Each of the five buttons showed a Japanese

4 340 PRESENCE: VOLUME 16, NUMBER 4 Figure 3. Examples of stimuli used in Experiment 1. (a) Morphing sequence from Doll A to Human A, (b) Morphing sequence from CG A to Human B, and (c) Morphing sequence from CG B to Human A, with morphing percentages of 0, 30, 50, 70, and 100. phrase corresponding to extremely unpleasant, unpleasant, difficult to decide (uncertain), pleasant, or extremely pleasant. These buttons represented a five-point scale ranging from 2 (extremely unpleasant) to 2 (extremely pleasant). Participants were instructed to interpret the word pleasant as representing any emotionally positive adjective such as attractive, pretty, natural, healthy, intimate, or elegant, and the word unpleasant as representing any emotionally negative adjective such as unattractive, fearful, ugly, abnormal, sick, or inelegant. Each participant rated the pleasantness of the presented image by clicking the corresponding button. After the button was clicked, the stimulus image was replaced with a uniform black field for 1 s, and then the next trial started. The images for each experiment were presented in random order. Participants performed two practice trials at the beginning of each experiment. At the end of each experiment, participants sent the data, as well as their age and gender, to the authors. At this stage, participants were able to decide whether or not to send the data. The number of participants who did not send the data is not known. 2.4 Data analysis The pleasantness scores for each experiment were averaged across participants and submitted to repeatedmeasures ANOVAs. When necessary, Bonferroni s multiple comparisons were performed. Statistical tests for the other analyses will be specified as necessary. 3 Results and Discussion 3.1 Experiment 1 Let us hypothesize that an almost, but not perfectly, realistic human appearance possesses an exceptional perceptual significance and that for some reason the human visual system generates an unpleasant impression for such a special degree of realism. This hypothesis predicts that the uncanny valley would always emerge at some point when the degree of realism is increased approaching the most realistic level. To test this, we showed the participants images from three types of morphing sequences, in each of which an

5 Seyama and Nagayama 341 Figure 4. Pleasantness scores averaged across participants (Experiment 1) for the Doll A Human A sequence (circles), for the Doll B Human G sequence (squares), for the CG A Human B sequence (triangles), and for CG B Human A sequence (diamonds). Error bars are 95% confidence intervals. artificial face was gradually morphed into a real human face (Figure 3). The morphing ratio varied from 0% to 100% in increments of 10%. Forty-nine participants (mean age 26.5 years, 20 female) rated the pleasantness of the face images. The percentage of real human (i.e., the degree of realism) was significantly related to the pleasantness score for all three morphing sequences (Fs(10, 480) 5.96, ps.001). However, the empirically obtained realism-pleasantness graphs did not show negative peaks (Figure 4). In one morphing sequence (Figure 3a), the face of a doll (Doll A, Pongratz Puppen) was morphed into a 19-year-old Japanese female s face (Human A). This morphing sequence produced a positive rather than a negative peak at 80% real human (Figure 4, open circles). The second morphing sequence, in which the face of another doll (Doll B, BP053-1, CITITOY) was morphed to a one-year-old Japanese female s face (Human G), yielded the highest score at 60% real human without producing a negative peak (Figure 4, filled squares). The images based on Doll B are not shown in Figure 3 due to copyright considerations. For the third morphing sequence (Figure 3b), in which a computer graphics (CG) image of an adult female face (CG A, Poser2 model bundled in Poser4, Curious Labs Inc.) was morphed into a 21-yearold Japanese female s face (Human B), the pleasantness score increased monotonically with an increasing degree of realism (Figure 4, open triangles). Thirty-seven participants (mean age 24.9 years, 15 female) rated the pleasantness scores for another morphing sequence in which CG B (Aiko 3.0, DAZ Productions, Inc., Figure 3c) was morphed into Human A. To improve the precision of the data analysis, the step size of the morphing ratio was halved (i.e., 5%), and sixty-eight landmarks were used for the eyes to achieve smoother morphing. In spite of these improvements, however, there was no clear indication of a negative peak corresponding to the uncanny valley (Figure 4, open diamonds). Although the pleasantness scores were significantly influenced by the percentage of real human (F(20, 720) 2.48, p.001), a multiple comparison revealed that a significant difference was obtained only between the scores for 0% real human and 70% real human (p.05). Although the morphing sequences produced different tendencies, none of the four types of morphing sequences showed that an almost perfectly realistic human appearance was a sufficient condition for the uncanny valley to emerge. 3.2 Experiment 2 In Mori s (1970) hypothetical graph (Figure 1), robots were supposed to have no resemblance to real humans at the lowest level of realism (e.g., industrial robots equipped with only manipulators). On the other hand, the stimuli used in Experiment 1 had reasonably realistic human appearances even at 0% real human. The morphing ratio of 0% did not imply that the face had no resemblance to real humans; it only indicated that the image was the same as the original image of an artificial face. Thus, one may argue that Experiment 1 failed to detect the uncanny valley because we tested only a limited range of realism, denoted as A in Figure 1. Furthermore, the images were somewhat unrealistic even at the morphing ratio of 100%, because they were low-resolution images presented on computer displays. So it is possible that Experiment 1 instead may only have tested the range of realism denoted as B in Figure 1, where the

6 342 PRESENCE: VOLUME 16, NUMBER 4 Figure 5. Examples of stimuli used in Experiment 2. (a) Morphing sequence from Doll A to Human A, where the eyes were morphed first and then the head. Morphing ratios were 0 0, 60 0, 100 0, , and (eyes % head %). (b) Sequence where the head was morphed first. Figure 6. Pleasantness scores averaged across participants (Experiment 2) for the eyes first sequence (squares), and for the head first sequence (triangles). Error bars are 95% confidence intervals. uncanny valley was not involved. In Experiment 2, we tested whether the failure to detect the uncanny valley in Experiment 1 was due to an inappropriate range of realism in the stimuli. Forty-five participants (mean age 23.6 years, 22 female) observed images from morphing sequences where the eyes and head (i.e., facial regions other than the eyes) were asynchronously morphed. In the eyes first sequence (Figure 5a), only the eyes of Doll A were morphed into those of Human A while the head was unchanged, resulting in realistic human eyes in an artificial head. Next, the artificial doll head was morphed into that of Human A, resulting in a wholly realistic human face. Participants gave the lowest pleasantness score when the eyes were 100% real human and the head was 0% real human. This morphed image had higher realism than the unmorphed image of Doll A because of its real human eyes, and lower realism than the unmorphed image of Human A because of its artificial head. In the head first sequence (Figure 5b), the head was morphed first and then the eyes. Participants gave the lowest pleasantness score when the head was 100% real human and the eyes were 0% real human. This face also had higher realism than the unmorphed image of Doll A because of its real human head, and lower realism than the unmorphed image of Human A because of its artificial eyes. Thus, in these two types of morphing sequences, the negative peaks were found where the degrees of realism were between those of Doll A and Human A. As shown in Figure 6, the percentage of real human significantly influenced the pleasantness score (Fs(10, 440) 28.0, ps.001), and the lowest scores were significantly lower than those for the unmorphed images of Doll A and Human A (ps.001). The negative peaks found in Experiment 2 give empirical evidence for the emergence of the uncanny valley within a range comparable to the range tested in Experi-

7 Seyama and Nagayama 343 Figure 7. Examples of stimuli used in Experiment 3. (a) Morphing sequence from Doll A to Human A with a manipulation of eye size (Doll A, Doll A with eyes scaled to 150%, 50% morph between Doll A and Human A both with 150% eyes, Human A with 150% eyes, and Human A). (b) Morphing sequence from CG B to Human B. ment 1. Therefore, the failure to detect the uncanny valley in Experiment 1 is not attributable to the limited range of realism. The uncanny valley emerged when the eyes and the head showed the largest mismatch in the degree of realism. Such mismatched realism was not presented in Experiment 1, where the facial features were morphed synchronously and the uncanny valley did not emerge. This suggests that mismatched realism may be a necessary condition for the uncanny valley s emergence. However, we suspect that abnormalities in the stimuli, rather than the mismatched realism per se, were the direct cause of the emergence of the uncanny valley. Doll A s face is abnormal if it is viewed as a real human face, in the sense that its features remarkably deviate from a real human. Nevertheless, the pleasantness scores for the unmorphed image of Doll A were not significantly lower than zero (one-tailed t tests, Experiment 1, t(48).60, p.05; Experiment 2, t(44).17, p.05). This suggests that the deviation may have been viewed as an artistic representation rather than abnormality, although Doll A s potential attractiveness for children (Zeit, 1992) may have been underestimated by adult participants. However, the faces at the bottom of the uncanny valley may have been judged abnormal due to the mismatched realism between the eyes and the head. If the eyes were judged based on the head s realism, Human A s eyes in Doll A s head may have been judged abnormal as a doll s eyes (Figure 5a) and Doll A s eyes in Human A s head may have been judged abnormal as real human eyes (Figure 5b). Such judgments about the eyes, with reference to the head, are consistent with past findings that visual features of the head can influence the perceptual processing of the eyes (Hietanen, 1999; Kontsevich & Tyler, 2004; Langton, 2000; Seyama & Nagayama, 2002, 2005). The abnormalities induced by the mismatched realism may have produced unpleasant impressions, which in turn produced the uncanny valley. The same reasoning holds true for the judgments of the abnormality of the head based on the realism of the eyes. However, it is still uncertain whether the perceptual processing of the head is influenced by the appearance of the eyes. 3.3 Experiment 3 If the uncanny valley reflects unpleasant impressions of abnormalities, then any type of abnormality besides mismatched realism should also produce the uncanny valley. Among various factors that can make faces bizarre (see, e.g., Murray, Rhodes, & Schuchinsky, 2003), we tested the effect of abnormal eye size using two morphing sequences in Experiment 3 (Figure 7).

8 344 PRESENCE: VOLUME 16, NUMBER 4 Figure 8. Pleasantness scores averaged across participants (Experiment 3). Upper abscissa: eye size scaling factor. Lower abscissa: degree of realism. Circles: Doll A to Human A. Triangles: CG B to Human B. Error bars are 95% confidence intervals. Forty participants (mean age 22.5 years, 33 female) rated images from the Doll A Human A sequence (Figure 7a), and thirty-nine participants (mean age 24.2 years, 19 female) rated images from the CG B Human B sequence (Figure 7b). In each sequence, the eye size of an artificial face first increased from the original size (100%) up to 150%; however, the artificial faces did not produce unpleasant impressions (i.e., negative scores) even when their eyes were scaled to 150% (Figure 8, left plot area). The eye size did not significantly influence the pleasantness score for Doll A (F(5, 195) 1.49, p.05). Although the eye size significantly influenced the pleasantness score for CG B (F(5, 190) 7.91, p.001), the score for 150% eye size was not significantly different from that for 100% eye size (p.05). After the eye size was scaled to 150%, each artificial face was morphed into a real human face with 150% eyes. Pleasantness scores for the faces with 150% eyes decreased with increasing degree of realism (Figure 8, middle plot area). The Doll A Human A sequence produced the lowest score for 100% real human face with enlarged eyes. Although the CG B Human B sequence produced the lowest score for 80% real human, the scores for % were not significantly different from one another (ps.05). Finally, the enlarged eyes of the real human face were reduced to their original size. The pleasantness scores increased with decreasing eye size (Figure 8, right plot area). For each image sequence, the eye size significantly influenced the pleasantness score (F(5, 195) for Human A; F(5, 190) for Human B; ps.001), and the score for 150% eye size was significantly different from 100% eye size (ps.001). As a result, the uncanny valley emerged around the real human faces with 150% eyes. For the faces with varying degree of realism, the eyes scaled to 150% and the head were morphed synchronously in each morphing sequence. Thus, the abnormalities induced by the mismatched realism do not account for the results. The results suggest that an interaction between the abnormal eye size and realism produced negative peaks that are comparable to Mori s (1970) uncanny valley only for the faces with higher degrees of realism. It should be pointed out, however, that in each morphing sequence the degree of realism covaried with other facial characteristics such as gender, age, and expression. Thus, it is possible that the factor that interacted with the abnormality in Experiment 3 was a covarying factor rather than the realism. To test this possibility, we presented the face stimuli with 100% eye size to 24 naive raters (undergraduates at Tokyo National University of Fine Arts and Music and Rikkyo University), and asked them to judge the realism, gender, facial expression, and age of each face. For the realism judgment, raters compared two faces presented side by side, and all raters consistently judged that the human faces were more realistic than the artificial faces (ps.0001, binomial tests). This suggests that the realism in the two morphing sequences increased from left to right along the abscissa of Figure 8 (middle plot area). The raters judgments about gender and facial expression were not consistent between the two morphing sequences. In the gender judgment, 18 of the 25 raters judged Doll A as male, and all raters judged Human A as female (ps.05, binomial tests). For CG B and Human B, all raters judged both faces as female (p.0001). For the facial expression judgment, the raters chose the most suitable expression for each face among happy, sad, surprised, angry, disgusted, fearful (six basic

9 Seyama and Nagayama 345 Figure 9. Examples of stimuli used in Experiment 4. From left to right columns, dolls with 100% eye size, dolls with 150% eye size, 50% morphs between doll and human faces with 150% eye size, humans with 150% eye size, and humans with 100% eye size. emotions; Ekman & Friesen, 1975) and neutral expressions. The overall patterns of the expression choice were significantly different between Doll A and Human A (p.0001, Fisher s exact test), but were not significantly different between CG B and Human B (p.79). Therefore, neither gender nor facial expressions consistently explain the results for the two morphing sequences. For the judgments about age, the raters consistently judged that the human faces were older than the artificial faces. The raters observed two faces presented side by side, and chose the older one. All raters judged that Human A was older than Doll A, and 22 of the 25 raters judged that Human B was older than CG B (ps.001, binomial tests). The raters also estimated the age in years for each face. The mean estimated ages for Human A (24.0 years, SD 1.7) and Doll A (5.6 years, SD 2.2) were significantly different (t(24) 35.0, p.001). Also the mean estimated age for Human B (24.5 years, SD 4.1) and CG B (20.6 years, SD 4.7) were significantly different (t(24) 4.15, p.001). Thus, one may argue that what produced the uncanny valley in Figure 8 was the age rather than the realism interacting with abnormal eye size. It should also be pointed out that, even if the realism played a major role in Experiment 3, there still remains a possibility that the realism interacted with confounding factors other than abnormal eye size. The possible influences of confounding factors were investigated further in Experiment Experiment 4 To minimize the influence of confounding factors, we used images of artificial faces that were as similar as

10 346 PRESENCE: VOLUME 16, NUMBER 4 Figure 10. Pleasantness scores for faces with eyes scaled to 150% (Experiment 4). Upper abscissa: eye size scaling factor. Lower abscissa: degree of realism. Squares: Doll C to Human C. Diamonds: Doll D to Human D. Triangles: Doll E to Human E. Circles: Doll F to Human F. Error bars are 95% confidence intervals. possible to real human faces. The artificial face images were produced by mapping the textures of artificial faces onto real human faces (Figure 9). In other words, shapes of artificial face images were warped into those of real human faces without changing their textures. As is shown later, most raters judged these stimuli to be artificial faces in spite of the realistic humanlike shapes. In producing the stimuli, four human face images (2 male and 2 female Japanese, age 21 to 22 years; Humans C, D, E, and F) were paired with four doll face images (Dolls C, D, E, and F). Doll C was a classic Japanese doll photographed by the authors, and Dolls D F were images of masks obtained from a commercially available photo collection (Mask by Corel, SKU:CPH480). Based on these four Doll Human pairs, stimuli were produced in a manner similar to Experiment 3. For simplicity, the eye size was scaled only to 100%, 125%, and 150%, and the doll faces with 150% eye size were morphed into the human faces with 150% eye size in only five steps (0, 25, 50, 75, and 100% real human). In addition, morph images between the artificial and real human faces with 100% eye size were also produced to measure the pleasantness scores for normal eye size. Forty-seven participants (mean age 20.6 years, 26 female, 20 male, one did not report his/her gender) rated their impressions of the pleasantness of these stimuli. Figure 11. Pleasantness scores for faces with 100% eye size (Experiment 4). Squares: Doll C to Human C. Diamonds: Doll D to Human D. Triangles: Doll E to Human E. Circles: Doll F to Human F. Error bars are 95% confidence intervals. Figure 10 shows the mean pleasantness scores in the same manner as Figure 8. As shown in the left plot area, the effect of eye size was not consistent across the four dolls. The eye size significantly influenced the pleasantness score only for Dolls C and E (Fs(2, 92) 5.88, ps.01). In contrast, the eye size influenced the pleasantness score in a consistent manner across the four human faces (Figure 10, right plot area). For the human faces, the pleasantness score decreased with increasing eye size (Fs(2, 92) 27.8, ps.001), and the eyes scaled to 150% produced the most unpleasant impression. Although the results of Experiment 4 did demonstrate how the impression of faces with a 150% eye size varied as a function of the morphing percentage (Figure 10, middle plot area), the influence of various confounding factors must be removed from these results to isolate the effects of abnormal eye size and realism. Figure 11 shows the pleasantness scores for faces with 100% eye size. These scores may also reflect the influence of confounding factors. Thus, by subtracting the scores in Figure 11 from those for 150% eye size presented in Figure 10 (middle plot area), the influence of confounding factors can be eliminated (calibrated scores). For all four morphing sequences (solid lines in Figure 12), the calibrated scores decreased with increasing morphing percentage (Fs(4, 184) 20.71, ps.001).

11 Seyama and Nagayama 347 Figure 12. Calibrated scores (Experiment 4) obtained by subtracting the scores for 100% eye size from those for 150% eye size. Filled squares: Doll C to Human C. Diamonds: Doll D to Human D. Triangles: Doll E to Human E. Open circles: Doll F to Human F. Error bars are 95% confidence intervals. Filled circles and open squares connected with broken lines represent the scores for Doll A Human A sequence and CG B Human B sequence, respectively. Figure 12 also shows the calibrated scores for the Doll A Human A and CG B Human B sequences (broken lines). For the Doll A Human A sequence, the calibrated scores were obtained by subtracting the results of Experiment 1, scores for 100% eye size (Figure 4), from those of Experiment 3, scores for 150% eye size (middle plot area in Figure 8). For the CG B Human B sequence, the calibrated scores were obtained based only on the results of Experiment 3. In Experiment 3, the pleasantness scores for 100% eye size were not measured at intermediate morphing percentages. Thus, the calibrated scores for this morphing sequence were yielded only at 0 and 100% real human. As can be seen in Figure 12, the calibrated scores for the artificial faces (0% real human) were close to zero, indicating that the 100% eye size and the 150% eye size yielded similar pleasantness scores. This suggests that the scaling of the eyes from 100% to 150% had only a weak influence on the impression created by artificial faces. On the other hand, for the human faces (100% real human), all six morphing sequences produced the lowest calibrated scores, which were lower than zero. This suggests that the abnormally scaled eyes decreased pleasantness scores the most when the faces were the most realistic. Although the interpretation of the ordinate of Figure 12 (i.e., the calibrated score) is not confounded, there still remains the possibility noted earlier, that the abscissa may represent variation in a confounding factor other than realism. To test this possibility, 45 naive raters (undergraduates at Tokyo National University of Fine Arts and Music) judged the gender, expression, age, and realism of the stimuli in the same manner as Experiment 3. All raters judged the gender correctly for Humans D (male), E (female), and F (male), and only three were incorrect for Human C (female) (ps.0001, binomial tests). The gender judgment for Human D was not significantly different from that for Doll D, for which 43 raters judged as male (p.49, Fisher s exact test). Thus, gender does not account for the results for the Doll D Human D sequence. On the other hand, the genders of the other doll faces were judged less consistently, and the male/female judgment ratios were significantly different between the doll and human faces (ps.01, Fisher s exact tests). The Doll C Human C sequence and the Doll E Human E sequence showed increasing femininity, and the Doll F Human F sequence showed increasing masculinity. In spite of these opposite variations in gender, all doll human sequences yielded the same trends for impressions of pleasantness (Figure 12). Therefore, the factor of gender does not explain the results. For the facial expression judgment, most raters chose either a happy or neutral expression for each face, and the overall patterns of the expression choices were not significantly different between each of the paired doll and human faces (ps.39, Fisher s exact test) except between Doll C and Human C (p.05). As noted earlier, the facial expression judgments in Experiment 3 were significantly different between Doll A and Human A, but not significantly different between CG B and Human B. Thus, among the six morphing sequences tested in Experiments 3 and 4, the factor of facial expression does not explain the results for the four morphing sequences. For Doll C, 30 of the 45 raters chose the neutral expression, but this ratio was reduced to 16/45 for Human C, suggesting that the human face appeared to be more expressive than the artificial face.

12 348 PRESENCE: VOLUME 16, NUMBER 4 In contrast, the neutral expression was chosen by only 5 of the 25 raters for Doll A, but 21 of the 25 raters for Human A, suggesting that the artificial face was more expressive than the human face. Because of this inconsistency between the two morphing sequences, it is difficult to explain the results shown in Figure 12 based on the facial expression. For the age judgment, the raters judged that Human C was older than Doll C (p.001, binomial test). For the other doll human pairs, they judged that the dolls were older than the humans (ps.001). Thus, among the six morphing sequences tested in Experiments 3 and 4, human faces were perceived to be older than artificial faces in three morphing sequences, but the opposite was true in the other three morphing sequences. Despite these inconsistent age judgments, the six morphing sequences yielded similar results for the calibrated scores as shown in Figure 12. Therefore, the factor of age does not solely explain the results. In contrast to the inconsistent judgments of gender, age, and facial expression, the raters consistently judged that the human faces were more realistic than the artificial faces in all six morphing sequences tested in Experiments 3 and 4 (ps.0001, binomial tests). Thus, among the four factors considered here, only realism can consistently explain the results as the factor that interacted with the abnormal eye size. Therefore, we may interpret the abscissa of Figure 12 as representing the degree of realism. 4 General Discussion It is assumed that the human visual system involves sophisticated mechanisms for processing facial information (e.g., Haxby, Hoffman, & Gobbini, 2000). Such mechanisms seem to be broadly tuned to faces with various degrees of realism. Real human faces, artificial faces of dolls and robots, computer generated facial images, schematic line drawings of faces, and even simple face-like patterns consisting of simple geometric shapes (e.g., Robert & Robert, 2000; Turati, 2004) are all accepted as faces. Past studies showed that facial images with different degrees of realism often yielded comparable experimental results (e.g., Driver et al., 1999 and Friesen & Kingstone, 1998; Wilson, Loffler, & Wilkinson, 2002; Yin, 1969). Nevertheless, in daily life people rarely confuse artificial faces with real human faces; people do not ask a mannequin in a store window for directions to a train station. This suggests that the visual system has sensitivity to the degree of realism of faces. The present study investigated an effect of the degree of realism on the impression of pleasantness of artificial human faces, and in particular investigated the uncanny valley hypothesis proposed by Mori (1970). The results of our experiments showed that the uncanny valley actually emerged as Mori (1970) had predicted. However, our results also showed that the uncanny valley emerged only when the face images involved abnormal features. Thus, to fully understand the nature of the uncanny valley, we need to consider the effects of both the realism and the abnormality of artificial human appearance. For example, if human observers have unpleasant impressions of the faces of avatars in virtual reality or robots, the unpleasantness should not be attributed solely to the degree of realism. Probably, the physical appearance of such avatars and robots may involve certain abnormal visual features. Thus, improving the degree of realism of such avatars and robots without removing the abnormal features may simply lead to an exaggeration of the human observers unpleasant impressions of the artificial faces. Although the degrees of abnormality investigated in Experiments 3 and 4 (i.e., scaling of the eyes to 150%) were identical for artificial and real faces, its impact was greater for faces with higher realism. Participants may have judged that the eyes scaled to 150% were too large for real human eyes, but such eyes were acceptable as artificial human eyes. This implies that the judgment criterion for real faces was different from that for artificial faces. The human visual system may have knowledge about how eye size varies among humans (i.e., data of the statistical distribution of the size of real human eyes) from past experience, and such knowledge can serve as a judgment criterion of abnormality. If the eye size on a face deviated from the center of the statistical distribution of normal eye sizes, such a face may be

13 Seyama and Nagayama 349 judged abnormal. On the other hand, knowledge about how eye size varies among artificial faces may constitute another judgment criterion of abnormality. In fact, artificial faces can have arbitrary eye sizes depending on the designers intentions, and past experience in observing such artificial eyes may constitute statistical knowledge that is different from that about real human faces. It should be pointed out that the participants tolerance of the huge eyes on the artificial faces might have reflected their cultural background. Since most of the participants were Japanese, their judgment criterion may have been formed through their experience in watching Japanesestyle animation moves, computer games, and comics (manga), in which huge eye size designs have been frequently employed. Otherwise, the uncanny valley may indicate that perceptual sensitivity to facial features was higher for real faces than for artificial faces, and the higher sensitivity for real faces produced unpleasant impressions of abnormality while the lower sensitivity for artificial faces did not. Sensitivity to facial features is known to be better for familiar faces than for unfamiliar faces (Walker & Tanaka, 2003). Thus, the results of Experiments 3 and 4 may reflect the fact that participants were more familiar with real faces than with artificial faces. Although we tested only static images, Mori (1970) noted that robots motion would also influence the uncanny valley. If the judgment criterion of an abnormality in motion is different for real and artificial human appearances (Hodgins, O Brien, & Tumblin, 1998), then abnormality in motion would produce the uncanny valley depending on the degree of realism. Further studies are necessary to unveil other aspects of the effect of realism on facial perception and cognition. Such studies will provide clues to further understanding human responses to artificial human-like objects (e.g., Arita, Hiraki, Kanda, & Ishiguro, 2005; Breazeal, 2003; Garau, Slater, Pertaub, & Razzaque, 2005; Hinds et al., 2004; Minato, MacDorman, et al., 2004; Minato, Shimada, et al., 2004; Shinozawa, Naya, Yamato, & Kogure, 2005). One of our unanswered questions is how the human visual system extracts the information of realism from the visual features of face images. Lacking this knowledge made it difficult to effectively manipulate only the degree of realism in our stimuli. Since we simply morphed artificial faces into real human faces, various confounding factors covaried with the realism. Although we showed that the realism explains the results of the present study better than the confounding factors of gender, age, and facial expression, there still remains a possibility that an interaction between abnormality and an untested confounding factor better explains the results. The participants task in the present study can be interpreted as a judgment of facial attractiveness (or unattractiveness). Researchers have shown that the facial attractiveness is influenced by various factors, such as averageness, youthfulness, symmetry, hairstyle, and skin smoothness (see Rhodes & Zebrowitz, 2002 for review). The influences of such factors on the results of the present study are still unclear. The distinction between realism and abnormality is not so straightforward. We have operationally defined the degree of realism as the morphing percentage; that is, the similarity of a morphed image to the photograph of a real human face that was used as a source image for the morphing sequence. In the actual definition of realism employed by the human visual system, the similarity may be measured between an observed (artificial) face and a certain standard face. The average face of real humans (e.g., Rhodes et al., 2001; Rhodes & Zebrowitz, 2002) may serve as the standard face, since unrealistic artificial faces are supposed to have visual features that deviate considerably from those of the average face of real humans. It should be noted, however, that the degree of abnormality (or normality) may also be defined based on similarity to the average (or normal) face, since abnormal faces are supposed to have deviant visual features. In spite of the similarity between realism and abnormality, the results of the present study suggest that the human visual system processes realism and abnormality as separate perceptual dimensions. In other words, the human visual system may define realism and abnormality based on different visual features.

14 350 PRESENCE: VOLUME 16, NUMBER 4 Acknowledgments We thank S. Akamatsu for assistance in producing the morphed facial images; M. Nomura, K. Hasegawa, I. Ito, K. Taki, and K. Enokida for assistance in recruiting participants; A. Tanaka and D. Norman for comments; and E. Pongratz for permission to use the photo of Doll A. The preparation of this manuscript was supported by MEXT.KAKENHI ( ). References Arita, A., Hiraki, K., Kanda, T., & Ishiguro, H. (2005). Can we talk to robots? Ten-month-old infants expected interactive humanoid robots to be talked to by persons. Cognition, 95(3), B49 B57. Aylett, R. S. (2004). Agents and affect: Why embodied agents need affective systems. Lecture Notes in Computer Science, 3025, Breazeal, C. (2003). Emotion and sociable humanoid robots. International Journal of Human-Computer Studies, 59(1 2), Cañamero, L., & Fredslund, J. (2001). I show you how I like you: Can you read it in my face? IEEE Transactions on Systems, Man, and Cybernetics Part A: Systems and Humans, 31(5), DiSalvo, C. F., Gemperle, F., Forlizzi, J., & Kiesler, S. (2002). All robots are not created equal: The design and perception of humanoid robot heads. Proceedings of the DIS Conference, Driver, J., Davis, G., Ricciardelli, P., Kidd, P., Maxwell, E., & Baron-Cohen, S. (1999). Gaze perception triggers reflexive visuospatial orienting. Visual Cognition, 6, Ekman, P., & Friesen, W. V. (1975). Unmasking the face: A guide to recognizing emotions from facial clues. Englewood Cliffs, NJ: Prentice-Hall. Fabri, M., Moor, D., & Hobbs, D. (2004). Mediating the expression of emotion in educational collaborative virtual environments: An experimental study. Virtual Reality, 7(2), Fong, T., Nourbakhsh, I., & Dautenhahn, K. (2003). A survey of socially interactive robots. Robotics and Autonomous Systems, 42(3 4), Friesen, C. K., & Kingstone, A. (1998). The eyes have it! Reflexive orienting is triggered by nonpredictive gaze. Psychonomic Bulletin & Review, 5(3), Garau, M., Slater, M., Pertaub, D.-P., & Razzaque, S. (2005). The responses of people to virtual humans in an immersive virtual environment. Presence: Teleoperators and Virtual Environments, 14(1), Hara, F. (2004). Artificial emotion of face robot through learning in communicative interactions with humans. Proceedings of the 2004 IEEE Workshop on Robot and Human Interactive Communication, Haxby, J. V., Hoffman, E. A., & Gobbini, M. I. (2000). The distributed human neural system for face perception. Trends in Cognitive Sciences, 4(6), Hietanen, J. K. (1999). Does your gaze direction and head orientation shift my visual attention? NeuroReport, 10(16), Hinds, P. J., Roberts, T. L., & Jones, H. (2004). Whose job is it anyway? A study of human-robot interaction in a collaborative task. Human-Computer Interaction, 19(1 2), Hodgins, J. K., O Brien, J. F., & Tumblin, J. (1998). Perception of human motion with different geometric models. IEEE Transactions on Visualization and Computer Graphics, 4(4), Kobayashi, H., Ichikawa, Y., Senda, M., & Shiba, T. (2003). Realization of realistic and rich facial expressions by face robot. Proceedings of the 2003 IEEE International Conference on Intelligent Robots and Systems, Kontsevich, L. L., & Tyler, C. W. (2004). What makes Mona Lisa smile? Vision Research, 44(13), Langton, S. R. H. (2000). The mutual influence of gaze and head orientation in the analysis of social attention direction. Quarterly Journal of Experimental Psychology, 53A(3), Minato, T., MacDorman, K. F., Shimada, M., Itakura, S., Lee, K., & Ishiguro, H. (2004). Evaluating humanlikeness by comparing responses elicited by an android and a person. Proceedings of the 2nd International Workshop on Man-Machine Symbiotic Systems, Minato, T., Shimada, M., Ishiguro, H., & Itakura, S. (2004). Development of an android robot for studying humanrobot interaction. Lecture Notes in Computer Science, 3029, Mori, M. (1970). Bukimi no tani [The uncanny valley]. Energy, 7(4), Murray, J. E., Rhodes, G., & Schuchinsky, M. (2003). When is a face not a face? The effects of misorientation on mechanisms of face perception. In M. A. Peterson & G. Rhodes (Eds.), Perception of faces, objects, and scenes: Analytic and

15 Seyama and Nagayama 351 holistic processes (pp ). New York: Oxford University Press. Norman, D. A. (2004). Emotional design: Why we love (or hate) everyday things. New York: Basic Books. Reichardt, J. (1978). Robots: Fact, fiction, and prediction. Harmondsworth, Middlesex: Penguin Books Ltd. Rhodes, G., Yoshikawa, S., Clark, A., Lee, K., McKay, R., & Akamatsu, S. (2001). Attractiveness of facial averageness and symmetry in non-western cultures: In search of biologically based standards of beauty. Perception, 30(5), Rhodes, G., & Zebrowitz, L. A. (Eds.). (2002). Facial attractiveness: Evolutionary cognitive and social perspectives. Westport, CT: Ablex. Robert, F., & Robert, J. (2000). Faces. San Francisco: Chronicle Books. Sakagami, Y., Watanabe, R., Aoyama, C., Matsunaga, S., Higaki, N., & Fujimura, K. (2002). The intelligent ASIMO: System overview and integration. Proceedings of the 2002 IEEE/RSJ International Conference on Intelligent Robots and Systems, Seyama, J., & Nagayama, R. S. (2002). Perceived eye size is larger in happy faces than in surprised faces. Perception, 31(8), Seyama, J., & Nagayama, R. S. (2005). The effect of torso direction on the judgment of eye direction. Visual Cognition, 12(1), Shinozawa, K., Naya, F., Yamato, J., & Kogure, K. (2005). Differences in effect of robot and screen agent recommendations on human decision-making. International Journal of Human-Computer Studies, 62(2), Turati, C. (2004). Why faces are not special to newborns: An alternative account of the face preference. Current Directions in Psychological Science, 13(1), 5 8. Wages, R., Grünvogel, S. M., & Grützmacher, B. (2004). How realistic is realism? Considerations on the aesthetics of computer games. Lecture Notes in Computer Science, 3166, Walker, P. M., & Tanaka, J. W. (2003). An encoding advantage for own-race versus other-race faces. Perception, 32(9), Wilson, H. R., Loffler, G., & Wilkinson, F. (2002). Synthetic faces, face cubes, and the geometry of face space. Vision Research, 42(27), Woods, S., Dautenhahn, K., & Schultz, J. (2004). The design space of robots: Investigating children s views. Proceedings of the 2004 IEEE International Workshop on Robot and Human Interactive Communication, Yin, R. K. (1969). Looking at upside-down faces. Journal of Experimental Psychology, 81(1), Zeit, U. (1992). Künstler machen Puppen für Kinder: von Marion Kaulitz bis Elisabeth Pongratz. Duisburg, Germany: Verlag Puppen und Spielzeug.