Usability testing of an Electronic Programme Guide and Interactive TV applications Pedro Concejero, Santiago Gil, Rocío Ramos, José Antonio Collado, Miguel Ángel Castellanos Human Factors Group. Telefónica Investigación y Desarrollo Emilio Vargas, 6 E-28043 MADRID (SPAIN) Tel: +34 91 337 3943 Fax: +34 91 337 4491 E-mail: pedroc@tid.es Abstract This paper describes the usability testing of the Electronic Programme Guide (EPG) and some Interactive TV applications developed by Telefónica I+D for the Spanish satellite-tv operator Vía Digital. Fifty six people participated in the tests. In order to have a representative sample of the general population, participants were sampled following sex and age quotas from a total of 346 people randomly contacted by telephone. The paper also presents the results of the small telephone survey about TV viewing habits and the opinions about the advances in TV technology. The environment where the trials were made was a domestic setting, and people were asked to use a real working prototype, and to perform typical user tasks, as, e.g., searching for a particular program in the EPG. After each trial, the experimenter took note of the problems faced by the user, and whether the task was successfully completed, and then asked the subject to fill a short questionnaire about the usability of that particular application, including questions about difficulty, presentation speed and aesthetic appearance. At the end of the tests, subjects were also briefly interviewed to obtain the global opinion about the system, and including questions about some marketing issues of digital TV applications. Detailed results about the usability of the Programming Grid, and of the Theme and Channel Search functions are provided. After these results, design guidelines and user interface alternatives were produced, especially for the Programming Grid, which was considered by most subjects the most difficult part of the EPG. Two interactive TV applications, Tele-banking and Weather Channel, were also tested. These applications presented different difficulty levels for the users, and in particular, the two user interface approaches of the two interactive TV applications produced important differences in the usability of the system.
1 Introduction Background This paper describes the results and conclusions from the usability tests of the Electronic Program Guide (EPG) developed for the Spanish satellite TV operator Vía Digital by Telefónica Investigación y Desarrollo (Telefónica I+D). These tests were carried out in November 1997. Telefónica I+D was formed in 1988 to contribute to the technological innovation of the parent company Telefónica de España, by performing research and development activities. Over the last few years, the line of work of Telefónica I+D has evolved to conform with the objective of becoming a services creation lab. This objective is driven by the emergence of interactive services which are increasingly becoming a mayor force in the telecommunications market. Telefónica owns a significant share in the Spanish satellite TV operator Vía Digital, for whom Telefónica I+D is developing many applications. The Electronic Program Guide (EPG) Figure 1: First level menu of the EPG Figure 2: The usability lab for the tests One of the applications being developed is the EPG, which is the user interface for selecting the channels and for getting information about the programming on each of them (ETSI ETS 300 707, TR 101 288). Satellite TV provides many more channels than traditional TV, thus making troublesome for the user to find the desired programs. Satellite TV provides also an excellent platform to provide simple interactive applications, as, e.g., telebanking, weather forecast information, traffic conditions, etc. The main concern in the development of these applications has always been their usability, since the target population is very general, and especially because it adds new functions to a currently very simple system, the traditional TV. Therefore, the stress was put on the representativeness and the validity of the results. This is the main reason for the complex way in which subjects were chosen, using a quota sampling procedure carried out by a marketing research bureau. This was also used to get some knowledge of people's attitudes towards TV in general, and satellite TV in particular. A small report on the results of this survey is presented here.
Another added research objective was obtaining commercial information which could be useful for the marketing of the services. Nowadays, these issues are closely related to the usability issues, and the trials represented a good chance of collecting this information. 2 Materials and procedures 2.1 The usability lab The trials were carried out in the usability lab of Telefónica I+D, where a normal sitting room was simulated as best as possible. This setting is depicted in Figure 1. Subjects were sitting on an armchair in front of a TV set, at a distance 5 times the height of the display. The lab was illuminated as in a domestic setting, using normal lights. There was a light behind the TV set to improve the perceived image quality. The experimenter was sitting close to the subject, as in the figure, and took notes or helped him or her whenever required. The trials were video-taped using a device which split one single image into four sources of video from the trials. These are depicted in figure 2. Clockwise from the upper left corner, the images present: a global view of the lab, the TV screen, an image from the back of the subject and last, the image taken from a mini-camera on top of the subject, and which recorded his or her hands using the remote control. 2.2 Description of the trials Figure 3: Image from the videotape with the four views. The trials begun with a short briefing to the subject, to inform him or her about the aims of the trials, and a training phase, explaining the general operation of the remote control and the basic components and functions in the Vía Digital EPG. It was emphasized during the briefing that the aim of the trial was collecting the subjective opinion of the usability of the user interface and the components of the system, and not a detailed performance measurement, so that the subject felt easy during the tests and operated the system in a normal way. After this introduction the subject performed an example task, in which he or she used all of the basic functions, and filled the first page of a questionnaire, to get used with this tool (code named part A in the trial). The part of the EPG involved in this training was not used
during the main trials. It was the option "In this moment" of the main menu, which was the easiest of all the possible functions in the system. After finishing every trial task, the subject had to fill in the questionnaire its subjective opinion about some task characteristics (these were the variables for the analysis): B1. The Program Grid This component presents the timetable of programs for each channel in the current day. User has to use the scrolling arrows in the remote control to move for and backwards, up and down in the grid. In the inferior bar, the menu functions are displayed. User had to search for the ending time of a football match in the sports channel. He or she had to use vertical and horizontal scrolling functions to find this information. B2. Thematic Search The user finds here a classification of channels as a function of themes (e.g., films, news, sports, children, etc.). Once chosen the theme, the system presents all channels in the category, and then displays the program grid for each of them. The tasks proposed was to search for the name of the film that it is going to be shown tonight in one channel (called "Solo cine"). B3. Channels Menu There is a channels menu in a top bar. Once selected a category in this bar, the channels are displayed in the box at the right side. Information about the contents of that channel is displayed in a box at the right side. The user had to use the option channels from the first level of the EPG, and then select the news channels, and watch the channel called "24 Hours". C1. Calendar This application presents a calendar so that the user can select a particular day, and then view the programming for that day. Apart from the traditional calendar, the interface provides icons for different parts of the day: morning, afternoon, evening and night. Subject had to use a calendar to find out the programming of another day.
C2. Weather Forecast This is an application which provides information about the weather in Spain. It provides this information for the whole country, for regions and for capital cities. The day of the forecast can be chosen. The user had to get the information about the weather in Vitoria (a Spanish city) tomorrow. C3. Telebanking Application (BBV) This is an application which allows the user to compute the rates for loans, among other things. The interface is far more simple than for the other applications and components, offering only a few options. Subject had to compute the monthly fee for a loan to repair a house. He or she had to introduce some specific data about the loan (total amount, time to cancel, etc.) Table 1: Tasks and screenshots used during the trials. The different tasks were devised so that they were representative of a person's normal way of using TV. The tasks were divided into two parts: part B presented basic components of the EPG, like channel selection, and getting information about the programming; part C presented interactive applications, like weather forecast and telebanking. To avoid interference between the two types of applications, and in order to avoid order effects in the measurements, parts B and C were counterbalanced separately. Table 1 presents the tasks used, and screenshots of the interfaces. During the trials, a number of subject's actions were registered by the experimenter in a register sheet, especially if the user needed help or not, which was the most important measure we wanted to analyze in the trials. After ending each task, subject filled a questionnaire to evaluate five attributes of the system, using a rating scale (0 nothing at all to 10 completely): - Facility of the task. - Utility to achieve the goals. - Speed of the system response, and whether it was considered acceptable or not. - The aesthetic appearance of the screens. - Help for the navigation offered in all the screens. After all the trials finished, the subject was asked to fill a separate questionnaire to obtain his or her opinion on the whole system. These global features were essentially the same as those asked for the particular components, and including a rank order of preference for them. Finally, after the subject had completed this questionnaire, the experimenter made a structured interview, asking commercial questions about this type of TV system 2.3 Subjects and sampling The objective of the trials were having a representative sample of the general customer population for this kind of service, therefore a quota sampling procedure was chosen. The
objective was then having the same number of participant in each of the following categories: 3 age groups (14-30, 31-44 and 45-55 years old) and sex (male/female). Sex / Age (years old) 14-30 31-44 45 55 Total Men 9 9 9 27 Women 9 9 9 27 Total 18 18 18 54 Table 2: Distribution of participants by sex and age. Sampling quotas Subjects participating in the trials were chosen by means of a sampling procedure, by means of a telephone interview. This was made based on a structured questionnaire, including questions on the use of TV and on the knowledge of the different companies operating the service being tested. The information was taken from 13 th to 17 th November 1997, with the target of the general population of Madrid (Spain), age 14 to 55 years. Result of the telephone call Number of telephones Completed Interviews 346 Do not answer 80 Wrong number 83 Negative 384 Not completed 26 Out of quota 608 TOTAL 1.527 Table 3: Results of the telephone interview for the total number of calls made. Out of the total of completed interviews (the 22.7% of the total), 163 accepted to participate in the trials. This represents the 47% of the interviewees, but only a 10.6% of the total number of telephone numbers contacted for the trials. Only 1 or 2 calls were needed for the 90% of interviews within the quotas. Interviews took, as an average, 3.5 minutes, and it is very seldom that they took 7 minutes or more. Subjects were paid 5000 pts. for their participation in the trials. 3 Results 3.1 Tasks performance One of the most interesting results for these trials was whether participants were able to perform the proposed tasks by themselves, or on the contrary they required the experimenter s help, or consulting an external source of help, apart from the own TV system, to achieve the goal. This was registered by the experimenter in a sheet, and also registered in the viewing of the video-tapes from the trials. Global results can be seen in Figure 4. The most important result is that less than one third of the participants were able to end the task for the Programming Grid by themselves, i.e., without the experimenter's help. The rest of the applications have better usability, and the telebanking application is that with the highest percentage of success. There are no differences as a function of sex or age groups in the performance for the programming grid, but there are significant differences in performance for task B2 (thematic
search) as a function of age group, (χ 2 = 18.94, 2 d.f., p<0.001): only 25% of the third age group was able to finish the task without the experimenter s help. There are also differences in the performance for the task C1 (Calendar) as a function of the age group. While 80% of age group 1 and 62% of age group 2 finished the task without problems, 26% of the age group 3 did so. No person of the age groups 1 and 2 required any help to finish the telebanking application task, and only 3 people from age group 3 required this help. 9 70.00 60.00 50.00 8 7 40.00 6 30.00 20.00 10.00 0.00 Telebank Theme Weather Channels Calendar Grid Average Rating Grid Theme Search Channel Search Calendar Weather Telebank 5 4 Required help (percentage) Telebank Theme Weather Channels Calendar Grid 6.25 29.17 31.25 33.33 41.67 68.75 Grid Theme Search Channel Search Calendar Weather Telebank Average Rating 5 7.04 7.51 6.6 6.7 8.26 Figure 4: Results of performance for the different tasks Figure 5: Average rating of the tasks in subjective facility 3.3 Subjective evaluation of the applications and components The most interesting result of the variables included in the questionnaire is that about the subjective facility for the tasks. These results are graphically represented in Figure 5. There are significant differences between the tasks/components in a repeated measures ANOVA (F 6,222 =18.40, p<0.01; see Kirk, 1982), but there are no significant differences between sex and age groups, nor their combinations. Subjective rating results follow the same trend as those for the performance, the programming grid is the most difficult, and the telebanking application the easiest. Preference rank order results are, however, different: the most preferred component by most participants is the theme search, followed by the channel search and the grid. This rank order is the same for all age groups. This is an important result: although some of these components are the most difficult to use, they are considered as very useful by the users. It is also most significant that the telebanking application was rank ordered in the last position (except by the age group between 45-55) by the participants. The most likely reason is the limited utility provided by this application. It is clear from these results that in the usability evaluation of this kind of applications, the trade-off between usefulness and facility is perhaps the key issue. 3.4 Attitudes towards TV and telecommunications There are results about attitudes coming from two parts of the trials. First, the telephone survey provided results about people's attitudes towards television. Watching TV is a daily activity for nearly all the interviewed people in the survey. However, attitudes towards technological advances in digital TV are different between different age groups. While people of less than 30 years old show some interest for these advances, an important
percentage of people older than 30 years old think that in 10 years they will use the same TV systems as now. The second results come from the interview after the usability trials. In this interview people were asked about different new telecommunication systems, including, e.g., cable TV and Internet. One of the most interesting results of these interviews is the effect of media in the people's knowledge of the new telecommunication services. One example: although very few of the participants had any experience using Internet, nearly the 75% of them state that they know what it is. The same happens with cable TV, even though very few people could say where it was available, many were aware of that possibility. 4 Discussion The main objection which can be made to the validity of these results is the fact that they were obtained in a laboratory situation. There was not any possibility to make the trials in a more realistic situation, since the system was still a prototype. Of course, it can be assumed that in a real situation, the user will read the system documentation first, so the starting point could be different, and perhaps the performance results of basic components (e.g., the Programming Grid) would be different. But we have the opinion, as Nielsen (1993), that documentation or help are not the most important usability issues in a system: "it is always better if users can operate the system without having to refer to a help system" (ibidem, p. 16). This is especially true in forthcoming TV systems, since people are used to have a very simple interface. As a conclusion, we believe that these results are valid, perhaps only for the first moments of user interaction, just after unwrapping and installing the system, but provide useful insight into user preferences and opinions. 5 Conclusion The most important result obtained is the difference between age groups in some of the variables tested. The main reason for these differences is that older people are not used to explore a new interface. These people clearly prefer guided, hierarchical procedures, although they are slower and offer less usefulness. It is clear that the Programming Grid is the interface component needing more changes to make it more usable. The results of the tests made it possible to produce a comprehensive set of 68 design guidelines, which were classified in 10 categories, following the classification of usability principles from authors like Gerhardt- Pawals (1997) and Shneiderman (1997). References Nielsen, J. (1993): Usability Engineering. London: Academic Press. ETSI ETS 300 707 (1997): Electronic Programme Guide (EPG); Protocol for a TV Guide using electronic data transmission. Sophia Antipolis, France: ETSI. ETSI TR 101 288 (1997): Television systems; Code of practice for an Electronic Programme Guide (EPG). Sophia Antipolis, France: ETSI. Gerhardt-Pawals, J. (1997): Cognitive engineering principles for enhancing human-computer performance. International Journal of Human Computer Interaction 8 (2) 189-211. Kirk, R.E. (1982): Experimental Procedures for the Behavioural Sciences. Monterey (California): Brooks Cole. Shneiderman, B. (1997): Designing the User Interface. 3 rd Ed. Reading, Massachusetts: Addison Wesley.