ACCURATE INTERACTION WITH COMPUTER BY EYE MOVEMENT TRACKING

Transcription

1 ACCURATE INTERACTION WITH COMPUTER BY EYE MOVEMENT TRACKING B. Estrany, P. Fuster, A. Garcia, Y. Luo University of Balearic Islands, Spain {tomeu.estrany, pilar.fuster, dpsagm0, Keywords: Human-Computer Interaction, EOG (Electro- Oculography), eye movement. Abstract The paper presents a state of the art interface system between human and computer by tracking the eye movements on the screen. The user eye EOG (electrophysiological) signal is detected, amplified and adjusted real time to give precise position of the cursor on the screen to control the computer. To better capture the EOG signal, we built a prototype electrode s fixation device to attach on the user s head. A software application for the digitalization and processing of the signal has also been designed and implemented. Key issues of stabilizing the signal and remove noise signal to provide reliable cursor control have been solved. The results show that the eye movement tracking for human computer interaction is feasible and can be achieved. 1 Introduction Using eye movement to control a computer has been a long time topic of many researchers in the world. The advantage of this approach of interaction is obvious which is a kind of hand free interaction. It can be for handicapped people or normal people with their hands busy with other more important tasks. The use of this type of channel for communication has been studied in early systems [7][8][9][10][11][12][13]. However, they did not reach for interaction with computers directly. There are many ways to track the eye movement. We can classify them as optical, electrical and magnetic. Among the electrical approaches, the EOG (Electro-Oculography) signal tracking system provides an inexpensive, relatively reliable solution [4][5][6]. However, due to the degeneration of the signal after a small period of use and other difficulties, the practical use of the system for accurate cursor control on the screen has not been possible. The work presented in this paper has overcome the major obstacle and make the human computer interaction to be accurate enough to control the applications on computer systems. It will open up a new way of human machine interaction and for many application areas such as smart environments, ambient intelligence, just to name a few. 1.1 The EOG The metabolic activities of the retina can generate a potential in corneal-retinal which can be used for tracking the movement of the eyes. This potential difference can be measured in the cornea and, under normal circumstances, is within a constant range. However, the potential can change due to the ambient light and the condition of the eyes. Further difficulty of tracking is to overcome the interference to the signal. The EOG signal is at a very low voltage (50 to 3500 microvolts) and is easily affected by any other electrical signals. There is also large interference that comes from biological activities and skin sensor contacts such as eye blinking and sweating. 1.2 Our approach A careful study of the electrophysiological characteristic of the human eye system was performed first which gives us a clear theoretical base about the source of the EOG signal. The mechanism of obtaining reliable EOG signal on a computer was then targeted. We designed an experimental paradigm to serve this objective. By experiments on different type of users and recording the EOG signal from numerous experiments, the factors that affect obtaining reliable, stable EOG signals were identified. The results of the experiments were then under a careful study and analysis. From there, an improvement of hardware and software elements of the experimental system was undertaken. Much better results were obtained from new experiments. Analysis on these new results lets us improve the experiment system. This iterative process eventually led to a our design of a new human-computer interaction scheme. In this paper, we will not discuss the human eye system which can be found in references[1][2][3]. We will describe our experiment system and the experiments we performed. The analysis of the experiment results will then be described. The design of the human-computer interaction system will be presented and the key technical issues that lead to the success of the system will be discussed. 2 The experiment system

2 To capture the EOG signal and analyze the factors that affect the EOG signal and master its control, an experimental system was designed. The system setup is shown in the Figure 1. and applying smoothing filters to take the interference away. A comparison is then made to compare the recorded signal with the original perfect signal. Scaling factor and offsets can be applied to align the two signals. Figure 4 shows such a process. Figure 1. The experimental system The objective of the experiment is to use a reverse engineering approach. The experiments use known patterns and create them on the screen. The user will follow the patterns by looking at the cursor movement on the screen. The generated EOG signal will then be recorded for analysis. A module for creating the patterns on the screen is designed. The detection of EOG signal is via a mask that the user wears during the experiment, see Figure 2. The signal is then amplified and the interference signal will be filtered away. Figure 3 Experiment patterns Figure 4. Comparison of recorded and generated signals Figure 2. The mask for EOG detection We chose the following patterns in the experiment: horizontal, vertical, diagonal top-down, bottom-up diagonal, heart function, cosine function, circular, saw tooth and rectangular pulse etc. The speed, smoothness and duration of movement parameters are also part of the pattern that we can control. See Figure 3 for some of these patterns. We developed a software application for the experiment. The application can control the type of the generated pattern and their parameters. It can record the EOG signal when the user s eyes follow the movement of the cursor on the screen. The recorded signal will then be processed The experimental tests helped us establish some of the design criteria that should be taken into account when we design the user interface to control the computer by EOG signals. Below are the major findings in most of our test users. Comparing the horizontal and vertical channels in the recorded signals, we found that the horizontal component needs much more adjustment than the vertical component. At the same time, the noise and interference affect much more the vertical component than the horizontal component. The range of eye movement can be calculated depending on the size of the rectangle in which the stimuli generated and the distance from the eyes to the monitor. We found out that the behaviour of eye dipole is linear within our measurement range.

3 We also found out that movements of fovea persecution system can generate a signal that interrupts the system. Such interruption may be caused by the loss of concentration of the user or some un-uniform speed of pattern generation. Drift and high-frequency noise can also be observed in the recorded signal. For checking the behaviour of them, further experiments were performed and the we found the parts that produced the high frequency noise. Filters were applied to remove such noises. At the same time, we observed that the higher sensibility we set the more precise EOG signal can be recorded but the drift effect will be stronger. A series of experiments have been made to find out the optimal drift factors. Another important finding is to use the EOG signal to control the computer practically, we have to solve the flickering in the signal EOG caused the eye blinking of the user. 3 Analysis of experiment results The analysis of experiment results helps us not only finding general problems in using EOG as control mechanism for interacting with the computer, but also the specific problems due to our hardware settings. It helps us in compensating and adjusting the recorded signal to obtain a more reliable stable EOG signal. 3.2 The linearity From the experiment results, we have identified the linear range of the eye dipole behaviour. The range of eye movement can be calculated depending on the size of the rectangle formed by the generated stimuli and the distance from the eyes to the monitor. In the case of a 17-inch monitor, this rectangle is by cm. The distance from the user to the screen was about 30 cm. The range of horizontal movement is ± 27 and vertical movement is ± 19. Therefore, practically, the behaviour of eye dipole is linear in this range. We also calculate the mean square error on both x and y directions. We found out that the lowest values of the mean square error occur when the DC channel has little drift. The coefficient applied to each channel can be interpreted as an indicator of the amount of information provided by the channel in the reconstruction of this component eye movement. The following figures are some of the recorded signals for our linearity analysis and calculation (Figure 7). 3.1 The comparison of x and y channel signal A specific but very important problem we found from the experiment results is the importance of the electrodes being in the dipole axis of the eye. This results in y channels signal is much weaker than the x channel signal in almost all the tests. As a result, noise and drift were affecting the signal to a greater extent. In Figure 5, the higher magnitude signal is from the x channel while the lower one is the y channel. This has helped us in a redesign of the location of the electrodes on the mask. A set of masks were designed and built during our project until a satisfactory mask was made. Figure 7. Recorded EOG signal for analysis of linearity 3.3 The movement of fovea persecution Figure 5. The comparison of x and y channel signal Figure 6. Development of EOG detection masks This set of results is recorded with the intention of provoking a stimulus in the fovea persecution system of the test user. However the saccadic system can interrupt the fovea persecution system and generate a pattern of movement known as saccadic persecution. We have to eliminate these patterns. Different range of Inter Stimuli Interval (ISI) has been applied in the experiment combined with different range of x and y. The analysis results can be classified in five cases. Case 1: a small ISI (10ms). In this case the cursor exceeds the speed of fovea persecution system and initiates a saccadic movement for the purpose of focusing the target again in the fovea. Case 2: the combination of a small x and a large ISI. This causes the cursor to stop for a long moment of time and

4 then suddenly appear in a position away so that it triggers a saccadic movement of the eyes. Case 3: a sudden change in the path of the cursor xy regardless of the ISI. This was observed in some rectangular pulse patterns. It is possibly due to loss of concentration during the record time. Case 4: ISI variations due to overloading of the system as mentioned in the last section. The effect is a variable speed, with random stops and rapid accelerations, the cursor movement causing continuous drawn on the subject. Case 5: The beginning of the cursor movement caused a small saccadic movement in the tester s eyes because of the experimental nature of the fovea persecution system. This occurs despite the fact that the movement is preceded by an audible signal to alert the subject of the initiation of the movement. For this reason, it is advisable to remove the top of EOG records in measuring the linearity. However, it is interesting enough that these results incidentally captured the information of the eye searching system. All these cases were taken into account when analysing the EOG records in our further testing in the experiment iteration. 3.4 Drift and high-frequency noise Using the adhesive electrodes in our experiment reduced significantly the drift in the recorded EOG signal. However, the drift does not disappear completely, nor predictably. To deal with the drift problem, we set up a series of experiments with the intent to assess the drift and sources of high-frequency noise. In this series, the electrodes and the reference electrode were only placed horizontal. The records were made with the long-term pattern in the horizontal circular pattern. We used three channels polygraph with the sensitivity of 500, 200 and 100 V/mm for the same electrodes. That is, using the same input for all channels record and varying only the sensitivity. When comparing the different channels of registration, it was noted, that the signal with lower sensitivity contains high-frequency noise. Therefore, we can assume that after the signal processed by the polygraph, the cable connection between the polygraph and the board introduced the noise into the EOG signal. It seems that a greater sensitivity can obtain more accurate records. However, in return, the signal is lost more easily because the effect of drift also increases and, consequently, the amplifier can be saturated. Moreover, the signal with greater sensitivity still contained some noise that could be produced by any of the sources. To reduce it, we added a medium frequency filter which removes the noise very effective. This results in an increase in sensitivity. In analysing the drift, we measured it on the signal generated by circular patterns and horizontal long-term signals. An interval of 60 seconds the level of drift can raise up to a range of 0.2 volt, almost half of the EOG reference signal. The effect of drift depends on many factors. The measurements give an idea of their magnitude. Figure 8. The drift in EOG signal 3.5 Detection of eye blinking It is important to detect flickering in the signal EOG due to the involuntary blinking of the user eyes during the testing. If we apply a high-pass filter on the vertical component of the signal the detection of the blinking can be simplified. By experiment we used a filter with a cut-off frequency of 8 Hz, The detection is then reduced to just establish the lower and upper thresholds to a given interval that we consider to be a flickering. This method has obtained satisfactory results. See the left part of Figure 9 for the result of applying the high-pass filter. Figure 9. Applying high-pass filter for blinking detection The linear relationship of the horizontal component with respect to the movement pattern indicates that it is possible to use EOG signal for an eye positioning device. The major obstacle is the drift. Some researchers [7][10] have opted to remove the drift using a high-pass filter with a cut-off frequency close to 0 Hz, and using the amplitude and direction of the saccadic movement as the information source in their systems. However, in applying this filter, an important part of the information on the eye position is lost. On the other hand, if the control device is designed with a narrow set of commands, the use of amplitude and direction of the saccadic movement will allow the classification control commands to be very reliable. This is provided that the saccadic movement control commences from a neutral eye position. Other researchers [11][2] restricted the EOG use by classifying it as direction, amplitude, or other parameters. Some used an algorithm based on fuzzy sets that use EOG characteristics as speed, acceleration and changes of direction for separating eye movements from the drifting on the EOG. However, this algorithm only uses saccadic movements that some other information was not used to make better results.

5 In our case, we decide to try to reduce the drift in the signal EOG. We also developed a special algorithm dealing with the drift problem after our analysis. We think that it is possible to improve the EOG signals if we adjust the settings properly such as the amplification range, low-pass filter parameters, offsets and reduce the external interference both inductive and capacitive interference. To confirm these assumptions, we decide to design and build up our own system. The next sections describe the design process and the results achieved. 4. System design for accurate cursor control by EOG signal Based on the analysis described in the above section, we have enough information to design a system for human machine interaction using the EOG signal produced by the user eye movement. The system scheme and the interactive process can be found in Figure 10. This is a real-time process of interactive control signal recorded from the user to position the cursor on the screen which will work exactly like the mousemovement. The computer is then controlled by eye movement. Figure 10. The eye movement control cursor positioning The system, an interface for human and machine, can be divided into two parts. One part consists of the user and the other the hardware and associated software. See Figure 10for such a division. On the user part, the system has been reduced to a single element the user eye system, but actually consists of several human and machine independent sub-systems such as reflexion system, saccadic system and fovea persecution system. The system formed by the user takes the controllable objects on the screen as input which are the stimuli for the ocular system. The response is generated from this stimulus, willingly or unwillingly, in the form of an eye movement as output. The eye movement can be generated directly by the ocular system or saccadic system and fovea persecution system, or indirectly by the system as a reflex response to a voluntary movement of the head. An EOG signal is then produced which will be used for the cursor movement relocation control. The other part of the system, the machine, is formed by the mask with EOG signal recorder, the amplifier, the digitalization card, and other components. A new set of software modules are developed to control different aspects of the interaction. The machine, seen as a whole, receives the EOG input information due to eye movements through its electrodes sensors. It then processes this information in different stages: amplification, offset, filtering, digitalization, software positioning. It generates output information for controlling the cursor dependent on the information input and adjusted by a set of configuration parameters. The exchange of information between the two systems, the user and the machine is done iteratively, in sequence and real time. From a functional point of view, in each iteration, the two systems perform a set of operations that will be described below. The system formed by the machine performs the operation to update the cursor position according to the received EOG signal and eliminate the noise produced by the drift taking the advantage of the set of points on which the cursor can be placed is discrete. This means that the cursors can only be at a set of fixed positions on the screen, not an arbitrary position. We use this as a basic means to restrict the cursor s position at each movement. The error on positioning the cursor will not be accumulated. The user, depending on the situation, has to do some of the following: - Position setting: Adjust the sitting position and the head position to position the cursor in the central area of the eye sight. - Control setting: The user should try to move the cursor in a particular direction. To achieve the goal, the user can choose between two options: to make a saccadic move proportional to the distance between the origin and destination and in the same direction, or start a movement of fovea persecution by slightly moving his head in the opposite direction to the movement while keeps an eye fixed on the cursor. Although the setting operations to be carried out by the user may seem complicated, if the positioning system is properly adjusted, it is very intuitive and easy to learn. 5. Usage of the system As the developed system is a prototype, to use the system for interacting with the computer by eye movement, the user has to pass a learning phase. At the same time, the environment conditions and the system parameters should be adjusted. 5.1 The user learning phase To manage the system, the user will need a learning phase. When settings are adequately set, this learning phase can be done in a few minutes during the first session. During this period, the user learns how to perform certain operations that can improve the control of the system including: focus the cursor in the center of the

6 fovea through small movements of the head, change the reference value of the center coordinates through an eye movement beyond the limits of the screen or make small movements of the head to compensate the miscalculation on the positioning of the cursor. The variation of any of the settings may require new learning. 5.2 Environment adjustment The use of the system will preferable be in a nice environment, proper temperature with less possibility of external inferences. The positioning system requires a few minutes of self adjustment to reach a stable usable state. This is the time needed for the electrodes attach to the face skin, and for the electronic components reach the working temperature. The higher level of ambient light, the greater the range of signals EOG that can give precise control over the cursor. However, a low level of ambient light can be compensated by adjusting the configuration parameters such as the scaling factor. The ambient light is actually not an important factor due to the latency of the eye system responding to it. 5.3 Parameter adjustment Most of the parameters for the system are configured from long time experiment results. The optimal values and the effect thereof for a given user can be adjusted as discussed below. - Regression coefficients (Xx, Xy, Yx, Yy): The regression coefficients are calculated automatically. These values are calculated only once before starting the interaction and it is not necessary to change them during the whole session. - Sampling frequency (scans/sec): This determines the response time of the system and we adjust it to be between 50 and 70 samples per second. If a higher frequency is used, the responding time can be shorter. But this may cause the user to lose the synchronization with the system, and will become chaotic for the user. - Scaling factors (X, Y scale): The scaling factors adjustment is empirically and their values depend on the level of ambient light and the size of the screen. - Precision: number of points in the buffer and cut-off points which controls the smoothness of the movement. - Sensibility: This variable, which contains the activation threshold of the readjustment of the coordinate axes, amends the feeling sensitivity of the system that the user sees. It should be adjusted in combination with the precision and the number of points of the buffer. Either value amends its effect on the system. The lower its value, the greater sensitivity will be felt. However, the effect of drift is also controlled by this variable. Therefore, a value suitable for both should be found by trading-off. Its effect was nullified by assigning a zero sensitivity. If a higher value is assigned, the cursor tends to stay in its position and gives the feeling that the system loses the sensitivity to eye movements. 6 System evaluation and testing results We used the following parameters in our system evaluation testing. - Screen resolution: 1024x768 and 800x Accuracy: 8 pixels at the resolution of 1024 x 768 and 4 pixels for a resolution of 800 x 600 pixels. - Scaling factors: between 0.45 and 0.9 depending on the level of environmental brightness. - Points in buffer: Between 10 and 15 points. - Cut-off: Between 20% and 30% of the points in buffer. - Sensitivity: to The system successfully interacts with a set of applications on the screen. See Figure 11 for some of these applications. The system's behaviour is acceptable during the interaction. It allows the user communicate with the graphical interface of the operating system and the software that it controls (Figure 11, Figure 12) through eye movement. The user can do so even with a laptop. Actually, in the case that the cursor has to move a large distance on the screen, the eye movement works faster and more effective than traditional manual mouse movements. In the case of precise movements, we need a greater concentration and control is not as effective as the mouse. The future system will increase the size of the cursor buttons which could solve this problem. Figure 11. Sample applications that our system successfully interacts The control capability of our prototype system depends on the nature of the interaction of the graphical objects. Bear in mind that resources graphics interaction that uses the operating system and programs that run on it, were designed for interaction of traditional manual mouse positioning system. If the future application development takes this new interaction mechanism into their design consideration, a much better interaction can definitely be achieved. Interaction of our system with the pull-down menus (Figure 12) is simple and fast; although it will be greatly

7 enhanced if we introduce a small delay in the open and close of the menus.. will benefit more for those who can not interact with computer with their limitation today. Acknowledgements The acknowledgement for the support funding from the Ajuntament de Sa Pobla, Mallorca is highly appreciated. We would also like to thank all the individuals that have helped this work to be possible. References Figure 12. Eye movement interacts with pull-down menus The ability of operation with buttons depends on its size. It is difficult to place the cursor over small areas such as those commonly used in the windows to control it. To perform these operations is necessary to reduce the screen resolution to 640x480 or use a large monitor. The operations of move the content within the windows ( "scrolling") is a bit complicated by eye movement control if only the sliders are used just as the small buttons. But if we change the behaviour of the slider, linking their position to the value of the signal EOG of the respective component, interaction would be more ergonomic than the current system. In fact, some other positioning systems include a physical button to perform such operations. 7 Conclusion and future works The paper presents a human and computer interactive system by tracking the eye movements on the screen using EOG signals. The results show that the eye movement tracking for human computer interaction is feasible and can be achieved. A direct application is for handicapped people and patient assistance. It can also apply to many application areas such as smart environments, ambient intelligence. There is still a lot to achieve before the prototype system going to practical use. There are many issues should be solved such as automatic adjustment for different users; implementation of applications suitable for this type of interaction etc. For example, a direct and immediate extension is to implement virtual keyboards, adding manipulators on the screen easy for such interactive mechanism etc. However, we believe that the interactive mechanism presented in the paper can be a new source in the interaction with multimedia display and virtual reality environments etc. We are encouraged by the results we achieved so far. We believe that the interactive technique we are developing [1] Carpenter, R. H. S.. The Eye Movement. London: The Mcmillan Press Ltd. (1988). [2] Newman N.M.. Supranuclear eye movement systems and their substrates. En Newman N.M. (1992). [3] Neuro-Ophthalmology: A practical text. Norwalk, Connecticut: Appleton & Glance. [4] Qiuping Ding Kaiyu Tong Guang Li. Development of an EOG (Electro-Oculography) Based Human- Computer Interface, proceedings of 27th Annual International Conference of the Engineering in Medicine and Biology Society, IEEE-EMBS [5] Kaufman, A.E.; Bandopadhay, A.; Shaviv, B.D. Title of the article, Proceeding IEEE 1993 Symposium on Research Frontiers in Virtual Reality, Oct 1993 Page(s): [6] Kumar, D. Poole, E., Classification of EOG for human computer interface, Proceedings of the Second Joint EMBS/BMES Conference, [7] Barea, R. (2001). Interfaz Usuario-Máquina basado en electrooculografía. Aplicación a la ayuda a la movilidad. Tesis doctoral. Madrid: Universidad de Alcalá de Henares. [8] Gips, J. et al.. Using EagleEyes-an Electrodes Based Device for Controlling the Computer with Your Eyes. Klaus, J et al.(eds.). Interdisciplinary Aspects on Computers Helping People with Special Needs. Viena: R, (1996). [9] Knapp, R. B.; Lusted, H. y Patmore, D. W. (1995). Using the electrooculogram as a means for computer control. Proc. of RESNA [10] Kuno, Y.; Yagui, T. y Uchikawa, Y.. Development of Eye-gaze input interface. Proc. of 7th International Conference on Human Computer Interaction. Volumen 1, 44. (1997) [11] LaCourse, J. R. y Hludik, F. C. An eye movement Communication-Control System for the Disabled. IEEE Transactions on Biomedical Engineering. Volumen 37 (12), (1990). [12] Lusted, H. S. y Knapp, R. B. Controlling computers with neural signals. Scientific American. Volumen 275 (4), (1996). [13] Yamada, M. y Fukuda, T. Eye Word Processor and Peripheral Controler for the ALS Patient. IEEE Proceedings A. Volumen 19 (6), (1987).