ERP recording with stimulus delivery and experimental control (SDEC) software

Transcription

1 ERP recording with stimulus delivery and experimental control (SDEC) software David L. Woods 1,2, Peter A. Pebler 2, Kristi Geraci 2, And Turken 1, and E. William Yund 1 1 Human Cognitive Neurophysiology Lab, UC Davis and VANCHCS, Martinez, CA ( 2 Neurobehavioral Systems Inc., 828 San Pablo Ave., Albany, CA ( DISCLAIMER: The first three authors are affiliated with Neurobehavioral Systems, developers of Presentation SDEC software. NOTE: Preliminary draft written for ERP Boot Camp, 8/9/07: comments and suggestions welcome. Introduction Despite vast increases in computer speed, timing precision has become more problematic with advances in computer technology. Although computer hardware has greatly improved over the years, operating systems (OSs) have changed in a manner that complicates the design of software for stimulus delivery and experimental control (SDEC). Older OSs such as DOS or specialized version of UNIX could run in single-tasking mode. This assured that your experiment had complete control of the OS and continuous access to hardware resources. Hence, timing precision was limited only by the speed of the computer and the precision of the computer hardware. In contrast, contemporary OSs use multitasking in order to improve overall performance. In normal conditions, this assures rapid responses to multiple programs as they compete for CPU resources. Indeed, as this document is being edited the Windows Task Manager reveals that 7 applications, 75 processes (including approximately 50 system processes) and 664 threads (independently running tasks) are currently active on the computer. These tasks are not processed in parallel 1. Rather, the appearance of multitasking is created by running each program sequentially for brief periods. The switching between programs occurs so rapidly under most conditions that users have the impression that different computer operations are running in parallel. However, the time sharing underlying multi-tasking OSs becomes more apparent when CPU intensive processes (e.g., performing Matlab calculations) are run in parallel with a foreground process, such as document editing. Under these conditions, editing changes may require hundreds of ms or even seconds to take effect. Similar performance decrements may be noted at computer start-up because many processes are competing for CPU resources. Similar system interruptions (SYSINTs) also occur during ERP experiments. Most multitasking OSs are designed to minimize SYSINTs of durations (e.g., 50 ms) that would be noticeable to the computer user in typical applications. However, SYSINTs of shorter durations can critically affect the results of ERP experiments. Consider an ERP experiment examining the detection of target stimuli of 25 ms duration in a high-rate rapid serial visual presentation (RSVP) task with interstimulus intervals of 100 ms (Figure 1). The stimulus-delivery computer is connected to an ERP recording system so that a trigger event (arrow) is sent to the ERP recording system to indicate the time of stimulus delivery as each stimulus is presented. In order to deliver stimuli at the requested times, the stimulus delivery and experiment control (SDEC) 1 Newer processors may have multiple CPUs that can perform several processes at the same time. The integration of this multi-threading capability in OS design is a significant ongoing challenge for software developers. 1

2 RT RT MS Figure 1. System interruptions (SYINTs, red triangles) can introduce timing errors in ERP experiments running on multitasking operating systems if the software controlling stimulus delivery is interrupted for brief periods by other tasks. Black = standard stimulus, Red = target, arrow = event pulse sent to ERP recording system. * NO EFFECT DELAYED RT EARLY TRIGGER LENGTHENED ISI LONG STIMULUS software must have access to the central processing unit (CPU) in order to read the computer clock to determine when stimuli should be delivered. However, when other tasks take over the CPU, SDEC code execution is halted. During such times, no SDEC operations can be initiated, modified, or completed As a result, SYSINTs can alter the desired pattern of stimulus delivery and response monitoring. In this simplified example assume that each SYSINT lasts for 25 ms. If the SYSINT occurs at the beginning of the interstimulus interval (ISI) there may be no alteration in the experiment. This is because the scheduler will have returned control to the SDEC program in time to read the clock and execute the commands necessary to deliver the stimulus. However, If a response occurs during the SYSINT, the processing of the response event will be delayed (discussed in more detail below). If the response is stored, when the SDEC program restarts execution, it will note that a response has occurred and register its latency by reading the clock. However, the latency will be delayed by the duration of the SYSINT. Alternatively, if the response is brief and not stored (e.g., transient events on the parallel port), it may no longer be detectable when the SDEC software restarts execution. As a result, the response may be missed altogether. In rare cases (and depending on SDEC software design) the SYSINT may occur between the time the event code is sent to the ERP system and the instant that the stimulus is delivered. In this case, the event trigger will be sent to the ERP recording system but the stimulus will be delayed. More frequently, the SYSINT may occur shortly before the stimulus is scheduled to be delivered. As a result, the SDEC software will miss the desired presentation time. It will attempt to deliver the stimulus as soon as it can thereafter, but the preceding ISI will be lengthened. Finally, if the interruption occurs during the period when the stimulus is being delivered the SDEC software may not be able to turn off the stimulus at the desired time. As a result, the stimulus duration will be increased. In general, SYSINTs of 25 ms magnitude will occur frequently only if CPU-intensive programs are competing for computer resources. This rarely happens in laboratory situations, unless your research computer becomes infected with spyware or your research assistant decides to watch YouTube during an experiment. Nevertheless, even the minimal system processes that are running concurrently with your SDEC software will produce SYSINTs although these will usually be of much shorter duration. In addition, system processes invoked by the SDEC software itself may alter stimulus timing. For example, experiment-related SYSINTs (EXP-SYSINTs) may occur during periods when the SDEC software retrieves large stimuli from disk, synthesizes visual or auditory stimuli, or decompresses video images. EXP-SYSINTs may have 2

3 * RT * RT NO EFFECT LENGTHENED ISI DELAYED RT Figure 2. Interruptions due to processing required intermittently by the experimental task (red triangles) can introduce timing errors and result in the delayed detection of response events. Black = standard stimulus. Red = stimulus digitally filtered during the experiment. relatively long durations, and, because they occur at nonrandom intervals during the experiment, they can potentially bias experimental results. Some examples of EXP- SYSINTs in a hypothetical experiment are shown in Figure 2. Stimuli consist of bitmaps (black squares) that are subjected to differing amounts of digital filtering (red squares) depending on subject performance. The subject s task is to respond to target objects (asterisks) in filtered or unfiltered images. The digital filtering computations are performed in Matlab and produce significant EXP-SYSINTs (red triangles). These will occur predictably based on the position of the filtered images in the experiment scenario. For example, the ISI of the filtered trials may be systematically delayed (center), or reaction times (RTs) to targets on trials preceding the filtered stimulus may be prolonged (bottom). EXP-SYSINTs become more likely and more difficult to detect -- as experiments become more complex. For example, randomized ISIs and randomized complex stimulus sequences will both make the occurrence of EXP-SYSINTs more likely and more difficult to detect. In addition to SYSINTs, timing precision will be influenced by the hardware used for stimulus delivery (e.g., video card, monitor, and sound card) and response detection (e.g., keyboard, mouse, or joystick). The remainder of this manuscript will discuss in more detail the nature and consequences of the timing errors that may result, identify their sources, and describe the tools needed for their detection. Consequences of timing error on ERPs. As discussed above, SYSINTs introduce timing delays that vary depending on when the interruption occurs relative to the stimuli and responses that occur in the experiment. SYSINTs will introduce timing delays and increase timing variance. In general, the magnitude of ERP distortion will depend on the relative size of time-jitter in comparison with the duration of the ERP component. These effects are simulated in Figure 3 using monophasic and biphasic square wave pulses to simulate ERPs and time-jitter in a rectangular distribution with magnitudes that correspond to 1, 2, and 5 times the duration of the component duration. Monophasic ERPs are Optimizing system configuration for timing precision. Competing processes should be reduced to a minimum when an experiment is run. In addition to terminating all competing user applications, in most multitasking OSs it is possible to terminate many of the inessential system processes that can interfere with SDEC software performance. On Windows this can be accomplished by creating a Hardware Profile to be used when running experiments (see Unless your experiment requires network access to function correctly, you should also disable network connections or unplug the network, as processing network traffic uses system interrupts that can adversely affect timing performance. smeared by time-jitter with increases in peak latencies equal to half of the range of the time-jitter imposed. For example if there is up to 10 ms of time-jitter distributed according 3

4 500% 200% 100% to a rectangular distribution, the average delay will be 5 ms. For monophasic responses, the area of the true ERP (no jitter) and the time-jittered ERP that would be obtained from signal averaging remain the same, but peak amplitudes of the jittered ERP will be reduced when jitter magnitude exceeds the component width. The results are similar to low-pass filtering the ERP. Thus, in the frequency domain the power spectrum of the jittered ERP would show increased power in low frequency bands, and reduced power at high frequencies in comparison with the true (unjittered) ERP. More complex alterations occur with polyphasic ERPs of the sort encountered in actual ERP recordings. As the magnitude of jitter increases, the negative components evoked on one trial will be aligned with the positive components evoked by the same MONOPHASIC BIPHASIC Figure 3. Effects of time-jitter between trigger pulse sent to ERP recording system and the time of stimulus delivery on the morphology of ERPs. ERPs were simulated using monophasic and biphasic square waves. Time-jitter was simulated with a rectangular distribution with width equal to 1, 2 or 5 times the width of the ERP component. The amplitudes and time base are arbitrary stimulus on the next trial, resulting in delays and reductions in amplitude of both components during signal averaging. Concurrently, the latency difference between the true negative and positive peaks of the response will increase in the jittered average ERP. An analysis of the power spectrum of the average jittered ERP would show reduced power in both low and high frequency bands in comparison with the true response. Therefore, single-trial spectral analysis will show an increased percentage of the ERP signals that will appear as induced activity : i.e., activity that is not phase-locked to the stimulus. Time-jitter results from delays between the event trigger as detected by the ERP recording system and the time the stimulus is actually delivered to the subject. These delays can occur in either the SDEC computer or in the EEG analysis system. However, as the trigger events sent to the ERP recording system and stimulus delivery usually occur on successive commands in the SDEC program, asynchronies tend to be small. Delays in acquiring the event codes on the EEG recording 4

5 system will introduce timing variance. This additional timing variance will reflect both the digitization rate of the EEG signals and the software used to combine stimulus event times with the digital EEG record. EEG signals are usually captured using hardware A/D converters that permit continuous EEG signal acquisition. However, unless the trigger events are also captured by the A/D (analog-to-digital) converter, delays can also occur in detecting ERP triggers if the EEG acquisition program is interrupted by SYSINTs. In these cases, the stimulus may be presented before the ERP trigger is detected by the recording system. Time jitter is generally most critical for short latency ERPs with correspondingly brief durations. One extreme example is Wave I of the auditory brainstem evoked potential which has a mean latency of about 1.8 ms in response to 70 db clicks, and a component width of about 0.5 ms. Time jitter of 1 ms would result in abnormal Wave 1 latency values and significant reductions in Wave 1 amplitudes. Time-jitter control is also important when recording phasic ERPs, for example those recorded directly from the surface of the exposed cortex. Since significant time-jitter may be introduced by some SDEC software and hardware (particularly sound cards, see below), high frequency components at the rising and falling portions of phasic ERPs may be significantly reduced, with a corresponding increase in the magnitude of induced high-frequency activity in the high-gamma frequency band. Visual stimulus delivery. Before turning to a discussion of visual stimulus delivery it is necessary to review the technology that underlies the delivery of video stimuli. For many years, video stimuli were delivered with cathode ray tube (CRT) monitors. The images are drawn on the CRT by an electron beam that illuminates pixels on the screen in a series of horizontal scans that begin in the top left corner of the screen and sweep from left to right, line by line, down the monitor. Monitor resolution determines the number of pixels that will be illuminated. For example, on a monitor with 1280x1024 pixel resolution the electron beam will sweep over each of the 1024 vertical lines (each containing 1280 pixels) in sequence. The refresh rate of the monitor determines how frequently each pixel is illuminated. For example, at a 60 Hz refresh rate, each video frame (one full screen) requires one 60 th of a second (16.67 ms). During this time about 16 ms is spent illuminating the pixels on the screen. The remaining time (~0.67 ms) is used to move the electron beam to the top left of the screen and prepare to display the next frame in the display. The illumination of each pixel depends on numerical values stored in video memory on the video card. The video card sends appropriate information for each pixel to the CRT during the display. Once the electron beam hits the phosphors on the screen, light is produced whose color depends upon the phosphor composition. The phosphors used in computer CRT displays generally have fast response times (<.1 ms) and limited persistence (reduced by 90% in 10 ms). Thus, each pixel must be refreshed continuously for a stable image to be perceived. Because of the fast phosphor decay, when a stimulus on the bottom of the screen in being illuminated, the top of the screen will be dark. The fact that the CRT monitor is illuminated sequentially produces measurable delays between stimuli presented at the top and bottom of the screen. Thus, a visual ERP evoked by a stimulus at the top of the screen will have ~10 ms shorter latency (depending on size and display parameters) than an ERP elicited by the same stimulus at the bottom of the screen. Similar differences are found for RTs to stimuli in different vertical locations. SDEC software control of the delivery of video stimuli can be divided into a sequence of operations. 5

6 Preparing the video image in system memory. If the stimulus is stored in compressed form (e.g., JPG) on disk, this includes copying the file from disk to memory and decompressing the file so that it is ready to copy into memory on the video card for display. When feasible, it is advisable to decompress and preload video stimuli into memory before the experiment begins. Otherwise, EXP-SYSINTs may occur because of stimulus retrieval and decompression. Stimuli can also be generated using mathematical formula using such programs as Matlab. Depending on the complexity of the image and the design of the software, stimulus synthesis can introduce significant timing delays. For example, if the program calls a routine in Matlab or Visual Basic, response events may not be detected during the ms period when the synthesis of a complex, full screen stimulus is occurring. Developing tools for the high-speed synthesis of video images is therefore an important challenge. Copying the image from system memory to video memory. The video image is loaded into a portion of video RAM that is not currently being displayed. This step can be performed very rapidly. Monitoring the clock until the requested time for the display of the image occurs. When the clock indicates that the image should be presented on the next video frame, the video card is instructed to switch the video buffer it is using on the next vertical scan to a new buffer containing the stimulus. This switch, or page flip, can be restricted to only occur before a new vertical scan, so that the video display is never changed in the middle of a vertical scan. After this, scan line information from the video card driver can be monitored to determine when the vertical scan containing the new image begins.. Sending a trigger pulse to the ERP recording system. The design of the SDEC code that sends the event pulse to the EEG recording system determines how precisely the trigger reflects actual stimulus delivery. In some software, the event pulse is sent at the time the stimulus should be presented. In others, the raster position is monitored until the stimulus actually begins to display before a trigger pulse is sent. Monitoring the duration of the stimulus display. When the clock indicates that the image has been presented for the requested duration, the previous steps must all be performed again for a new image display (possibly a blank screen.) Temporal granularity of video stimulus delivery. The temporal resolution of video stimulus delivery is dependent on the refresh rate of the display monitor. For example, for a monitor running at 60 Hz, the shortest possible stimulus duration is 16.7 ms (although a small stimulus may only be illuminated for a portion of this time). Similarly, the minimal ISI would be one video frame: i.e., 16.7 ms. This leads to a temporal granularity of visual stimulus delivery. For example, if an experimenter desires to display a single video frame at an ISI of 54 ms on a monitor with 60 Hz refresh rate, the SDEC software will send a page flip command shortly after the preceding video frame starts to display at 50 ms. However, the page flip will not actually occur until the frame display has completed at 66.7 ms. The synchronization of page flips with the vertical scan described above avoids tearing of the video image. Tearing results if the video display is changed in the middle of a vertical scan. This can result in a mismatch between the top and bottom of the display. It could also result in a failure to display the stimulus altogether. For example, if the stimulus were small and located near the top of the display, shifting the video memory in mid-frame would display only the lower portion of the video frame containing the stimulus. Hence, the stimulus might never appear, even though its frame was presented. 6

7 This example also illustrates the importance of sending the event pulse to the EEG recording system only after the stimulus has actually begun to display. If the event pulse were sent at the time specified by experimenter (54 ms) rather than the time the stimulus actually occurred (66.7 ms), the ERP latency on that trial would be artifactually increased. In general, if event codes were sent at arbitrary requested times rather than at the actual times of stimulus delivery, time jitter would occur between triggers and stimulus events with a rectangular distribution equal to video frame duration (e.g., 16.7 ms for 60 Hz). The granularity of visual stimulus timing can also amplify the effects of small SYSINTs. Consider the case where an ISI of 16 ms is requested on a monitor with a 60 Hz refresh rate. A brief SYSINT (e.g., 1 ms) would result in the page flip request being delayed until after the video card had begun to display another video frame. In such a case, the actual ISI would be 33.4 ms; i.e., a 1 ms SYSINT would introduce a 16.7 ms stimulus delay. This problem can be reduced by using timing parameters (ISIs and stimulus durations) that result in page flip requests being sent in the middle of video frames. For example, if stimulus duration of 8 ms were requested, SYSINTs would need to exceed 8 ms in order to introduce errors in the ISI. In contrast, SYSINT-induced errors usually have minimal direct effect on RTs or fmri activations because the variance introduced by stimulus timing imprecision is significantly less than the intrinsic trial-to-trial variance of the response. More troubling is the possibility that systematic EXP-SYSINTs may alter timing differentially for different categories of stimuli. This could occur if small SYSINTs systematically lengthened or shortened the duration of display of certain stimuli. For example, if one category of stimuli (e.g., deviants ) were large compressed bitmaps retrieved from disk and another category of stimuli (e.g, standards ) were small bitmaps resident in memory, systematic between-category difference might be produced in ISIs or stimulus durations. For example, the ISI preceding the deviants might be randomly prolonged relative to the ISIs preceding the distractors. This confound would alter ERPs and, if large enough, might permit subjects to bias responses based on the ISI rather than the perceptual analysis of the deviant stimuli. Several types of video stimuli deserve special comment. Text. Display of text requires the creation of an image using the font-display capabilities of the SDEC. These are scaled to the appropriate size and a bitmap is created for display. These operations are normally very fast. However, the exact generation time will depend on the system and should be measured if generation is taking place during critical intervals. This display of characters used in non- European languages requires Unicode support where each text character is be coded in two bytes (rather than one byte used in traditional systems). Bitmaps. Bitmaps are generally stored as compressed images (e.g.,.jpg,.png, etc.) that must be converted into uncompressed images before display. This operation should be performed before the experiment begins to minimize extraneous demands on CPU resources. The expansion in the amount of memory required for each image can be substantial. For example, a black square in the center of a uniform background can be greatly compressed without distortion requiring slightly more than 400 kilobytes. However, if this display is prepared for a monitor with 1024 x 1280 pixels, it will require approximately 1.3 million pixels. Since each pixel typically requires one byte of information to specify each of the three (RGB colors) and one byte of information to specify transparency, one video image at this resolution will normally require about 5 megabytes of video memory. Streaming video. Displays of compressed videos present a particular challenge to ERP researchers. Traditional video display pipelines decompress and display the 7

8 video but largely bypass the SDEC software. This can introduce significant ER- SYSINTs that may degrade timing performance. In addition, the standard streaming video pipeline provides limited information on the timing of any particular video frame. Thus, the time of delivery of a particular frame in the streaming video (e.g., the frame where a target object first occurs in the video) cannot be established with precision and frames can be dropped if the video display falls behind its desired rate. Moreover, it may be impossible to combine other types of visual stimuli (e.g., text or bitmaps) with the video. An alternative approach is to decompress the video and store each frame as a bitmap on disk. Unfortunately, this process may generate too many files to be conveniently pre-loaded into memory. It is also possible to develop a streaming video pipeline optimized for SDEC applications so that bitmaps from the video stream are captured and displayed individually by the SDEC software as if they had been read directly from disk. This provides precise timing of each video frame and assures that no frames are dropped during display. Moreover, other video signals (e.g., bitmaps, text, 3D objects, etc.) can be mixed into the video stream at precise times. 3D objects. Displaying a 3D object in a visual scene requires intensive calculations to determine the luminance, color and intensity of light emitted or reflected from each point on the object s surface. Fortunately, software and hardware innovations required to support realistic 3D displays have occurred rapidly, driven by the computer gaming industry. The modern video card can therefore perform the calculations needed for realistic, real time displays even if large numbers of objects are present in each video frame. These capabilities can be exploited by SDEC software to present simple 3D objects and even complex 3D scenes of the sort found in computer games. Because the video card handles the rendering, the temporal precision of a complex 3D displays is generally as accurate as that of simple 2D bitmaps. In theory, ERP studies of 3D displays can provide insight into aspects of visual and cognitive processing that are inaccessible to experiments using simpler 2D representations. Stereo displays. Stereo displays involve transmitting separate video images to the two eyes to create the binocular disparity that is needed for stereoscopic depth perception. This can be accomplished by LCD shutter glasses synchronized to the graphics card that are illuminated alternately to project slightly different images to the two eyes. The control pulses needed for LCD goggles can be generated automatically by video cards from 3D image specifications. Stereoscopic DLP projectors incorporating LCD shutter technologies and 120 Hz refresh rates have entered the home-theater market with prices in the $4,000 range. 3D movies (including feature movies) generally use two superimposed polarized images projected onto special silver-coated screens and polarizing filters that transmit separate images to each eye. Video card technology. The quality of the video card is an important determinant of visual stimulus delivery capabilities. Video card performance has increased more than hundredfold over the past decade due to improvements in graphics processing units (GPU), video memory, and video card connections to the CPU. Modern GPUs are designed for the complex floating-point matrix operations required for the display of 3D images and so can easily handle the task of displaying simple stimuli. They can also perform much more complex operations including shading pixels on 3D objects, calculating the light reflected from objects, calculating reflectance and occlusion, antialiasing to counter line distortion, texture mapping with fine surface detail, and determine how objects should appear in reflective or semi-transparent surfaces 8

9 Monitor calibration. The relationship between luminance and the intensity of the signal sent to the electron gun of a CRT is not linear but is rather expressed as a power function with an exponent of gamma. The gamma for PC video displays is approximately 2.2, so that to double the luminance of a pixel, the intensity of the electron beam must be increased by a factor of about 5. In order to approximate linearity, PC images are compressed with a gamma of 0.45 and decoded with a gamma of 2.2. Adjusting a monitor s brightness and contrast settings will alter the gamma. Monitor calibration utilities allow you to set gamma individually for each color to calibration your monitor. A problem can arise when obtaining psychophysical measurements of faint stimuli when standard monitor settings are used, because the resolution of luminance information of each color (8-bit or 10-bit for some newer video cards) is significantly less than the dynamic range of the eye. Some companies (e.g., CRS, Ltd,) provide special hardware with up to 14 bits of dynamic range. Alternatively, if measures are obtained over a narrow range (e.g., at perceptual thresholds), it is possible to alter the minimum and maximum luminance values for each color in the monitor settings so that the 8 bits of dynamic range available in the standard video card are allocated to different portions of the luminance function. For example, perceptual thresholds can be measured more precisely (with the equivalent of 14 bits of resolution on a 10-bit monitor) by altering the intensity range by a factor of 16. Some SDEC software allows gamma settings to be altered dynamically during the experiment to permit precise measurements near threshold to be obtained with standard monitors. Quantifying timing precision. The careful experimenter will want to evaluate the timing precision of each stimulus presented during an experiment. Such measurements should include both the requested and actual times of stimulus delivery as well as the requested and actual stimulus durations. These timing precision measures should be obtained for each experiment since the hardware or software of the computer may be changed over the course of the experiment. Moreover, the particular stimuli and responses that show delays will not be constant from one experiment to the next since they will reflect the random status of other processes that are running concurrently on the computer. Although most SDEC programs report a single time of event occurrence, it is actually impossible to measure the exact time at which an event occurs in a multitasking system. This is because the operations of checking for the occurrence of an event and reading the computer clock occur in sequence. Of course, if there were only a single processing thread locked in memory (as was possible with older OSs) delays between the delivery of each stimulus and the reading of the clock would be constant. However, such precision cannot be achieved with multi-tasking systems where the time of stimulus occurrence can only be estimated. An estimate of the accuracy of the time measurement of stimulus delivery on a multi-tasking system can be derived from a comparison of times that were definitely before and definitely after the detection of an event. These times establish a time interval during which the event was detected. The duration of this interval represents the temporal uncertainty of the time measurement of the event. These estimates can be obtained by constantly reading the computer clock in a tight loop that includes a check as to whether or not an event has occurred. For example, in Figure 4, no event was detected on the first and second successive iterations through the eventchecking loop, while on the third iteration the event was detected before the clock is read. Clearly, the event had not occurred at time T1, and had definitely occurred before time T3. However, we cannot be sure whether or not the event was delivered before or 9

10 after time T2. This is because each processing operation requires a minimal amount of time. For example, in a visual experiment the video card may begin to display the stimulus (asterisk) after event-occurrence was checked in the T2 loop, but before the clock was read to provide the T2 time estimate. Thus, the event may have actually occurred before T2. Hence, the T3-T1 interval is required to define a time range wherein the event definitely occurred. In cases where the SDEC thread is continuously executed, the T1-T3 range of timing uncertainties is extremely small ( ms) 2. Many SDEC programs report T3 as the time of the event, but provide no measures of temporal uncertainty. This makes it impossible to estimate the range of times over which the event may actually have occurred. For example, if the SDEC thread were interrupted for 25 ms by a SYSINT after T2, the T3 value will be increased by the duration of the interruption. If only the T3 time were reported, no problem would be evident. However, the T3-T1 measure would warn the investigator that the timing of that particular event could not be established with certainty. Some SDEC software is not designed to perform the tight timing loops that are needed to measure timing uncertainties and capture external responses with precision. An alternative method of measuring RTs is to use a separate response box with a built-in timer that is synchronized to the PC clock. The external box captures the time at which the response occurs and can transmit an accurate RT to the PC when it is polled at a later time. However, unless the response box is directly connected to the ERP recording system, the response trigger produced by the SDEC software will be delayed by the length of the SYSINT. Since timing delays between stimulus display and event code output are of critical concern in ERP recording, timing loops can be designed to include tests for both event occurrence and trigger output in the same loop. This design guarantees that the timing uncertainty will include any interruptions that may have occurred between the event trigger output and the stimulus itself. Thus, the T3-T1 interval will include both the delivery of the stimulus and the transmission of the event trigger to the ERP recording system. It is important to note that this kind of uncertainty measurement is a measure of the software s ability to assign an accurate time to an event detection by the software. Thus, while it reflects the software s timing performance within a multi-tasking operating system environment, the uncertainties do not necessarily indicate the actual uncertainty in the time of a physical experimental event. To obtain complete information about the quality of an event time measurement, one must have additional information about the relationship between the means used by the software to detect an event and the actual physical events. Calibrating timing uncertainties throughout the experiment. Depending on the design of the SDEC software, the timing loops shown in Figure 4 can be performed at a very high speed (e.g khz). These timing loops can measure of SYSINTs throughout an experiment by quantifying the delays between the successive values reported from the clock. By setting the experiment to run repeatedly, it is possible to quantify the frequency and magnitude of timing interruptions that occur for a particular experiment using a particular configuration of hardware. This will include the timing uncertainties associated with the preparation and delivery of stimuli as well as timing uncertainties that might affect 2 It is important to note that this technique only controls for timing uncertainties introduced by the OS or other software on the system. The relationship between the detection of an event by the SDEC software and an actual experiment event will depend on the details of the hardware involved as well as device drivers and other software controlling that hardware. This is discussed in a following section. 10

11 response events occurring during ISIs. Hardware calibration of PC timing. If temporal uncertainty measures are available, these will provide a precise measure of the timing of events as recorded by the internal clock. However, delays can also occur between the time the PC reports that a stimulus occurred and the actual time that it was presented.. In the case of responses, there also may be a delay between the physical response and the time the SDEC software detects that response. For visual stimuli, these differences are usually negligible because page flipping and raster monitoring permits the precise identification of the particular frame containing the stimulus. Therefore, hardware calibration of the time of visual stimulus delivery is usually unnecessary. However, hardware measurements with photometers are needed to calibrate the color and luminance of the monitor, and the temporal response of the display device. For example, older LCDs may require several refreshes before the stimulus luminance becomes detectible and therefore need temporal response calibration as well. For auditory stimuli, hardware-specific delays occur in the actual presentation of the sound, thus requiring physical calibration. Hardware calibration of response devices is also needed because these have different intrinsic latency delays. Physical stimulus calibration may be the only way to obtain information about the timing precision of a stimulation system considered in its entirety the SDEC software combined with all attached hardware devices. Special hardware calibration systems such as the Black Box Toolkit ( have been developed for this purpose. The Toolkit uses photodiodes to measure the time of visual stimulus delivery by attaching the photodiode to the monitor and including a bright box at the photodiode location in the stimulus being presented. In addition, the Toolkit can calibrate response devices (such as a mouse) by attaching contacts to detect button closure and can capture sounds using a microphone. Homemade circuits permit similar timing calibrations. Photodiode output can be digitized on a standard sound card running on a second PC. Analysis of the digitized recordings can then be used to quantify the ISI between stimuli and their duration. Similarly, in order to calibrate the timing of response devices a programmable relay can be used to simulate button closure with its time recorded by the sound card. The response event is detected by the SDEC program it can send an output pulse to the second audio channel on the sound card, permitting a measure of latency delays between the two events. To analyze asynchronies between event pulses sent to the EEG recording system and stimulus delivery, both channels on the sound card can be used in a similar manner, one channel recording record stimulus delivery using the photodiode and the other recording the associated event pulse sent from the SDEC system. Since video cards provide rapid and highly reliable display control that is not subject to SYSINTs, once a video image has been loaded into the video memory for display it will appear at the precise time determined by the video refresh rate. Insofar as the event pulses sent to the ERP system are tightly time locked to actual stimulus delivery, timing calibration of the visual display is unnecessary, provided that the display speed characteristics of the display device hardware are known. Unfortunately, the measurement of timing precision during actual experiments is difficult unless timing uncertainty measures can be obtained from within the SDEC software. An alternative solution is to compare event timing measures obtained with hardware calibration systems (such as the Black Box Toolkit) and timing measures reported by the SDEC software. This solution requires that video stimulus frames include a white box (to permit detection with the photodiode). Monitoring the timing of responses and auditory stimuli require connections through relays, and the trigger outputs to the ERP system should also be captured. A careful comparison of the timing of events on the 11

12 SDEC system and actual time of event occurrence on the hardware calibration system would then reveal timing errors. In ERP experiments, the ERP recording system can also provide independent estimates of timing delays and uncertainties. Test scenarios can be designed to send event codes to the ERP recording system (set at a fast digitization rate) at regular intervals (e.g., 100 ms). An examination of the SDEC logging file will indicate the time that the SDEC software reported sending event pulses, and an analysis of the ERP event logging file will indicate the time that they were received. Problems would be indicated if either logging file showed inter-event intervals that deviated from those requested. Variations in the ERP event log alone would suggest that variable event transmission delays were occurring, and further testing would be needed to determine their source and minimize their impact. Auditory stimulus timing. Auditory stimulus delivery is more complex and variable than the delivery of video stimuli because sound cards introduce both constant timing delays and time-jitter. These delays arise because sound delivery depends on a series of operations that are performed by the sound card and sound card device drivers once the sound card has received the command to present a sound. First, the sound card will copy the digital sound file into its buffer for digital signal processing (DSP). This may include modification of sound parameters (e.g., intensity, pan, and, for entertainment applications, spectral filtering and any desired special effects such as reverberation). Different sounds can also be mixed together at this stage. All or part of this step may be done in software on the system, or in hardware by the sound card itself. Next, the sound is sent to the D/A converter, where it is converted into an analog waveform. Finally, the analog signals are routed through amplifiers to speakers or headphones. 12

13 SOUND CARD LATENCY DELAYS TYPE MODEL MEAN (ms) VARIANCE MODE Notebook Creative Audigy DirectX Hardware 2ZS DirectX Software Custom primary Custom secondary USB Creative SB Extigy DirectX software Custom primary Custom secondary Internal Creative Audigy DirectX Hardware DirectX Software 0.6 < 0.1 Custom primary 0.7 < 0.1 Custom secondary Internal Voyetra Turtle Beach 2.0 < 0.1 DirectX Hardware DirectX Software Internal Creative E-MU 0.6 < 0.1 DirectX Hardware DirectX Software 0.7 < 0.1 Custom primary 0.7 < 0.1 Custom secondary Table 1. Performance for various sound cards in different playback modes supported by the Presentation SDEC software. Special custom modes are needed to present 24-bit sounds or to independently deliver sounds from more than two speakers. Default software modes are significantly slower than hardware acceleration. Copying, DSP, and D/A conversion can require variable amounts of time that contribute to the latency delays shown in Table 1. High-end soundcards have high-speed hardware buffers and DSP chips that can accelerate each of these operations. Alternatively, the operations can be performed in software using the CPU, with a corresponding increase in the duration of signal processing and corresponding EXP- SYSINTs. Delays can be substantial (up to 40 ms) for certain sound cards. As a result of these delays, the SDEC software will report that the sound has been presented before it actually begins to play. Timing calibration is therefore necessary before sound cards are used. This can be accomplished by sending the sound output with associated event codes to a second computer that will digitally record both signals, or with special sound card latency analysis software and digitized sound outputs. Fortunately, sound card timing delays do not depend on the stimuli being delivered. Hence, hardware calibration does not need to be performed with every experiment, but only when a new sound card is installed. The fact that sound cards are designed primarily for entertainment and computer gaming applications can also complicate their use in scientific experiments. For example, in default modes, certain sound cards may impose slow rise times on auditory outputs, 13

14 regardless of the desires of the experimenter. Some SDEC applications include custom software that can control DSP operations and thus eliminate many of these undesirable sound card features. More complete reports and testing procedures can be found at Response devices. When a subject responds with a USB mouse, keyboard or joystick, circuits within the peripheral device store the response value until it can be polled by the USB bus. The rate of polling varies from 80 Hz to 1000 Hz for different devices. It can sometimes be set over a range of different settings in the control panel depending on the device driver. On older PS/2 devices, information was sent to the motherboard in serial packets at khz. TIMING DELAYS OF SELECTED RESPONSE DEVICES DEVICE MEAN DELAY (ms) VARIANCE (ms) MICE USB Standard USB (Inland, $7) USB 1000 Hz Razer Copperhead ($50) Microsoft Intellimouse (ps/2) ($15) JOYSTICK: Logitech attack USB ($25) KEYBOARD: Keytronic ($25) Table 2. Both mean timing delay and variance vary significantly for different response devices. Peripherals designed for gaming produce faster responses than standard devices. Table 2 shows timing delays for three different mice, a joystick and a keyboard. It can be seen that for USB devices the device delays vary by about 600% from 6.6 ms (for a 1000 Hz gaming mouse) to 34.6 ms for a low-cost USB mouse (polling rate unspecified). This difference is greater than can be accounted for by the differences in USB polling rate. Therefore, a significant component of the latency delay appears to arise in the drivers used to interpret responses from the devices and report them to the OS. Devices designed for gaming (including an inexpensive joystick) have relatively fast drivers. In contrast, standard devices (including the PS/2 mouse) have slower drivers and produce increase delays. The response delay differences suggest that similar hardware should be used on different test systems if the results are to be pooled or compared. Different hardware devices also introduce different amounts of timing variance. For USB devices, different polling rates may contribute significantly to differences in latency variability (e.g., 5 ms for mice polled at ~100 Hz and < 2 ms for mice polled at 1000 Hz). In general, response devices optimized for gaming should be used to reduce mean latency delays and latency variability to acceptably low levels. RTs can also be corrected by the mean latency of the response device. Latency delays can be further reduced (to < 0.2 ms) by wiring a response button to send input to the parallel port using the interrupt line. However, this may often compromise the full response capabilities of the device (e.g., cursor control from the mouse). Alternatively, response boxes with internal timing can be used. Parallel port input and output. Parallel port input can generally be directly accessed by SDEC software and hence is associated with very short latency delays. Some SDEC software includes interrupt drivers for parallel port input. This driver remains resident at all times and is activated very quickly (<0.1 ms) to time-stamp the interrupt events. This provides precise timing and avoids missed events even during SYSINTs. Parallel port output can occur without measurable delay so that sending event codes to the ERP 14

15 recording system is desirable. What should you do if your computer lacks a parallel port? USB parallel port emulators should generally be avoided because of long and variable delays. Parallel port cards (including Express-bus cards in laptops) appear to provide good timing precision (< 1 ms) but may lack the standardized interrupt capability of actual parallel ports. In the absence of a parallel port, digital IO cards with counters should be used in those cases where missed events (e.g., fmri triggers) could critically alter the experimental design. Serial port input and output. The serial port is often used to send trigger codes to the ERP recording system. The serial port sends information serially with a speed that is reflected in the serial port baud (bits/s) rate. For example, to send one byte of information with start and stop bits requires about 1 ms at 9600 baud. Transmission rates up to 128,000 baud are generally available on PC systems. Timing uncertainties in serial port output are usually extremely small (< 0.2 ms). This can be tested by wiring the output of the serial port to its input and examining the delays using SDEC software. Serial port inputs are generally latched, so that events will not be missed. Note however that USB serial port adapters may introduce additional delays and their behavior should be measured. Special hardware. Care should be exercised when connecting special software and hardware devices to your SDEC computer to monitor external inputs. Problems can arise if the device must be polled but only reports event occurrence after a considerable delay. For example, some A/D cards connected through USB interfaces to laptop computers can require more than 20 ms to report their status. As a result, when a timing loop is executed to determine if an event has occurred, the loop will pause for 20 ms while awaiting the report from the device. This will greatly reduce the precision of SDEC timing! Timing uncertainty measures or careful hardware calibration should reveal such problems if they occur. Tactile stimulation. DirectX provides an interface for low-cost force-feedback devices (e.g., joysticks, game controllers, steering wheels, etc.) that can provide vibratory and tactile force outputs. Some SDEC software permits standard or custom waveforms can be sent to these devices. Timing precision has not been reported. 15