Time-course and localization of syntactic and semantic anomaly in sentence processing:

Transcription

1 Time-course and localization of syntactic and semantic anomaly in sentence processing: a within-subjects fmri/meg design ELLEN F. LAU for fulfillment of LING 895 requirement in the University of Maryland Department of Linguistics Project members: Henny Yeung, Ryu Hashimoto, Allen Braun, & Colin Phillips ` ***unfinished draft probably contains errors please don t cite without permission!*** Abstract Determining source location could potentially allow more fine-grained distinctions between functional interpretations of ERP components, but the problem remains difficult. Methodologies which provide more localization information (fmri, intracranial recordings, MEG, patient studies) give results which often seem incommensurate. In this study we combine two of these methodologies, fmri and MEG, in the same participants, using the same paradigm, in an attempt to gain more insight on the source and timecourse of responses to syntactic and semantic anomaly. fmri can provide more accurate information about the location of activity, while MEG can provide more accurate information about the timecourse of activity. We also review the previous literature on the processing of semantic and syntactic anomaly using various methodologies, and we provide a meta-analysis of fmri studies of semantic priming, and semantic and syntactic anomaly. We conclude that the combination of information from such methodologies is informative, but remains subject to considerable limitations of interpretation.

2 I. Introduction The ERP response to any given word usually includes, after the early sensory components, a broad negative peak followed by a broad positive peak, with a fairly broad distribution over the scalp. As a result, any modulation of the amplitude of the ERP response after the first few hundred milliseconds that is not clearly limited to a few electrodes can be described as either an N400 effect (modulation of the broad negative peak) or a P600/late positivity effect (modulation of the subsequent broad positive peak). Many authors have worked to develop definitions of these components that would succeed in capturing all of the manipulations that result in one or the other under a single functional label. However, we might be concerned that a measure which will automatically lump most effects into one of two bins is unlikely to be fine-grained enough to provide a functional carving-up of the space. One solution to this problem is to avoid assigning functional interpretations to ERP components or effects, and simply to make use of the timing of effects to provide information about the processing timecourse. The alternative solution is to try to add an additional measure to provide more finegrained distinctions between ERP effects. One such measure which is appealing for independent reasons is spatial localization. Not only can localizing the source of an ERP effect help us distinguish between effects with different sources, but we are probably interested in the cortical basis of the processes we are looking at anyway. In this study we tackle the localization of three well-known ERP effects in the study of language and in particular, of the violation of linguistic expectations the left anterior negativity (LAN) to syntactic anomaly, the N400 increase for semantic anomaly, and the P600 or late positivity to syntactic anomaly with a particular focus on the N400. The N400 is second perhaps only to the P300 as the most well-studied cognitive component in the ERP literature; certainly hundreds of published papers have examined the effect of various manipulations on the amplitude of this component or used its amplitude as a dependent measure to test processing hypotheses. However despite the boom in functional neuroanatomy over the past few decades, there is still no clear consensus on what brain regions are involved in the generation and/or modulation of the N400 component. The few intracranial studies that have been conducted have provided suggestive evidence of both temporal and occasionally frontal involvement, but were necessarily limited in scope. MEG studies using one method of spatial reconstruction (equivalent current dipole models) find the source of the N400 most often in left posterior temporal cortex, while studies using distributed source models have localized activity to a broad area of cortex spanning temporal and frontal areas over the time window of the N400. The fmri studies done tend to show activity in left frontal regions, but not as often temporal; this activity s relation to the N400 component is difficult to judge because time is integrated over such a long interval in the fmri signal. The results of the few studies that have looked at the N400 response in aphasics with functional lesions have not been able to clearly tie a particular area to a difference in the N400 response. The only clear point of agreement across all these investigations is that the left hemisphere seems to be more often and more strongly involved than the right. For several decades, fruitful research has been carried out in both examining the properties of the N400 and using it as a dependent measure, without knowing much about the location of its generators. Despite this, such knowledge still has the potential to be

3 incredibly useful. Perhaps the most commonly cited reason is the possibility that the N400 wave is actually composed of several functionally distinct components that correspond to different steps in the processing of meaningful stimuli (Halgren et al. 2002, Pylkkanen & Marantz, 2003). If these different components could be localized to generators in distinct regions, more fine-grained experimental manipulations could be carried out. Relatedly, various types of manipulations have been shown to modulate the amplitude of the N400 wave (e.g. repetition priming, semantic priming, contextual constraint, anomaly), and this modulation itself may have different spatial loci.. Knowing whether this is the case would tell us whether we should be searching for an umbrella functional description of N400 amplitude modulation as many authors have attempted, or whether we are actually looking at separate, temporally-overlapping phenomena. In the past decades, fmri studies have had some measured success in fairly reliably associating given parts of cortex with certain cognitive processes. However, there has been less success in using fmri to associate certain parts of cortex with the generation of certain electrophysiological components. Normally when we are interested in localizing function in the brain we go to fmri, which has excellent spatial resolution, and simply look for the regions that show more blood flow when that function is called upon. As with all measures, we know that fmri may not be sensitive enough to show everything not all regions that are involved but we use it because what it does show is informative: If our design is not faulty, and we see a region light up, we know that this region is involved, and we can incorporate this into our theory of the neuroanatomy of this function. The key factor that makes localization difficult in the N400 case is that we are trying not to localize a region or subset of regions that are sensitive to semantic anomaly, but rather to localize the region that generates an independently known ERP response component. If, for example, two regions show differential fmri activity to the manipulation, we would have no way of knowing which corresponds to the N400 effect, or if either corresponds. There are many quite plausible reasons that this region would not show a differential fmri effect: the difference in blood flow corresponding to the N400 may be quite small relative to typical fmri effects, or it may be anatomically variable over trials over the space of a few centimeters such that the effect is spatially diluted, or the region in question may be susceptible to more noise in fmri due to its location. Magnetoencephalography (MEG) has been lauded as just the solution to this kind of problem: it has the temporal resolution of ERP but because the signal is less distorted by intervening material, the source of the signal can be better approximated. Several studies have used this technique to examine the N400 response to semantic anomaly (Simos, Basile, & Papanicolaou, 1997; Helenius, Salmelin, Service & Connolly, 1998; Halgren, Dhond, Christensen, Van Petten, Marinkovic, Lewine & Dale, 2002; Maess, Herrmann, Hahne, Nakamura, & Friederici, 2006; Pylkkänen, Llinás, & McElree, submitted). However, different source localization techniques have given different results: equivalent current dipole analyses have tended to show superior temporal sources ranging from anterior to posterior across subjects, while distributed source modeling techniques have shown additional frontal and temporal activity. There are several potential reasons for the variability that has been found among different localization techniques but one that we don t have a very good handle on yet is

4 to what extent it is caused by individual differences. In this study we tested the same subjects in both MEG and fmri using the same materials and presentation parameters. If the fmri activity seen does correspond to the MEG, we should be able to see correlations within individuals in the location of the fmri activity and the source of the MEG activity. In particular, although the accuracy of different MEG source localization techniques is a topic of debate, one factor which remains fairly constant is that in contrast to ERP, activity which appears in channels over a given hemisphere in MEG likely has its source somewhere in that hemisphere. Thus, one coarse but fairly robust way of determining the degree of correlation different aspects of the fmri and MEG across manipulations and individuals will be to look at which aspects of the signal in each technique appear bilaterally and which are more lateralized. II. Background In this paper, we examine extremely well-studied ERP components (the N400, and, to a lesser extent, the LAN and P600) with a less commonly used combination of measures (both hemodynamic and electrophysiological). In the background section, we focus on the N400 as the best-studied case. Here we provide the reader with an overview of the findings that compose our current understanding of the N400 component from ERP, followed by an overview of current perspectives on the neural substrate for semantic processing from the imaging literature, and only then with a summary of previous attempts to pinpoint the source of the N400 effect. Thus, the reader who is already somewhat familiar with the electrophysiological and imaging literature on semantic processing may wish to skip the first two sections. We also describe a metanalysis of N400 priming effects in fmri, which is not crucial for an understanding of the source of the N400 semantic anomaly effect and can thus be skipped by readers more interested in the sentential effects. Due to constraints of time and space, we provide a much briefer overview of the work on syntactic processing, focusing specifically on the response to syntactic violation. A. The N400: Electrophysiology The list of variables that have been shown to influence N400 amplitude indicates that it shares one attribute of the logogen-type units central to many models of word recognition: a somewhat indiscriminate collection of evidence for the presence of a particular word. -Van Petten, Kutas, Kluender, Mitchiner, & McIsaac, 1991 The large branch of modern-day ERP research devoted to the N400 response was initiated by Kutas and Hillyard (1980), who reported that sentences with a semantically inappropriate ending gave rise to a negative peak 400 ms after the onset of the anomalous final word. At first this peak was thus thought to be an index of violation or interruption of normal processes, because as the authors themselves pointed out, N400 is not a general response to all linguistic or meaningful stimuli because judgments about such stimuli have been specifically associated with the P300 wave. However, several years

5 later Kutas and Hillyard (1984) showed that even within non-anomalous sentences, an N400 response was visible to all sentence endings except those which were the most highly predictable based on the context, and that in fact the amplitude of the N400 varied inversely with the strength of expectation for that ending. Later authors showed that isolated words also elicited a negative peak in the same time window that was modulated by semantic priming (Bentin, McCarthy & Wood, 1985; Rugg, 1985; Holcomb, 1988), and that faces and pictures also elicited a negative peak with a slightly different distribution, but similar timing and repetition priming properties (Barrett, Rugg, & Perett, 1988; Barrett & Rugg, 1990; Ganis, Kutas, & Sereno, 1996). Thus, although outside of the field the N400 is still often thought of as the semantic violation component, within the field today most would agree with the characterization of Kutas and Federmeier in a recent review that the processing of almost any type of meaningful, or potentially meaningful, stimulus seems to be associated with negativity between about 250 and 500 ms post-stimulus-onset (Kutas & Federmeier, 2000). 1 The question remains, however, whether modulations of this negativity index the same process in all cases. In this paper, we will largely focus on localizing the areas which give rise to these modulations, as opposed to those that give rise to the standard N400 response to any meaningful stimuli. To mark this point we will attempt to always use the term N400 effect rather than N400, which would most naturally be taken to refer to the component itself. Lexical properties. As mentioned above, phonotactic regularity/probability affects N400 amplitude for nonwords: phonotactically regular pseudowords show an N400 peak that is comparable in amplitude to the N400 for unprimed real words, while phonotactically illegal nonwords show little or no N400 (Bentin, McCarthy, & Wood; Holcomb & Neville, 1990). High frequency words have been shown to elicit smaller N400s than lower frequency words (Van Petten & Kutas, 1990; Allen, Badecker, & Osterhout, 2003). However, the amplitude difference disappears when other factors that reduce N400 amplitude are in place, e.g. repetition or sentence position. There is some evidence that this is not a floor effect, as Van Petten (1993) showed that the frequency effect disappeared after the first three words of the sentence, while the amplitude of the N400 continued to decrease as a function of sentence position throughout the rest of the sentence. In the same study, Van Petten showed in contrast that in well-structured but semantically random sentences (e.g. all content words chosen randomly), an N400 frequency effect was visible throughout the sentence. This suggests that frequency and contextually-based semantic expectation may have the same effect on the N400. Although partially confounded with other factors like length and frequency, there is some evidence that the property of being a function word as opposed to a content word is associated with differences in N400 amplitude. Some authors have argued that the N400 for function words is actually absent (Neville, Mills, & Lawson, 1992), while others claim that the amplitude of the N400 is simply severely reduced (King & Kutas, 1995). Repetition effects. Both repetition priming and semantic priming in isolated word or word pair presentation reduce the amplitude of the N400, although the reduction seems to 1 Note that Kutas and Hillyard s (1980) original point about the P300 is well-taken; the N400 is typically followed by a positivity that many researchers believe is a member of the P300 family.

6 be greater for repetition priming (Rugg, 1985). Illegal nonwords do not show a repetition effect on the N400 (Rugg & Nagy, 1987), but pseudowords do (Doyle, Rugg, & Wells, 1996). The reduced N400 is often followed by a fairly broadly distributed increased positivity in the repeated condition which is thought to signal recognition and is sometimes referred to as the Late Positive Component (LPC) 2. It is interesting to note for language ERP researchers that while this component likely has a different functional locus, it shares basically the same timecourse, polarity, and distribution as the P600. The LPC has been shown to be larger for low-frequency than high-frequency words (first presentation) in texts (Van Petten, Kutas, Kluender, Mitchiner, & McIsaac, 1991), and the repetition effect on LPC amplitude seems to be greater in low-frequency words (Rugg, 1990; Van Petten et al. 1991); this may be related to the fact that recall is generally better for low-frequency words, and may reflect some kind of ceiling effect on activation. On the other hand, the repetition effect on the LPC seems to be more sensitive to the lag between repetitions than the N400 effect (Van Petten et al. 1991). Within sentence tasks the repetition effect on the N400 is less reliable: In one study, a reduction in amplitude for repeated words across (congruous) sentences was seen only when the contexts they occurred in were identical (Besson & Kutas, 1993) although Van Petten et al. (1991) did observe repetition effects for words in different sentences when the sentences formed connected texts. Besson, Kutas, and Van Petten (1992) showed that the repetition effect on the N400 interacted with other factors, such that repetition reduced N400 amplitude much more for incongruous than congruous endings, open-class words than closed-class words, and low-frequency words than high-frequency words However, there was some tendency for the repetition effect to come on earlier than the congruity effect. Besson and colleagues also found an increased LPC for repetition that was largely seen only in the incongruous condition, which they suggested could be due to less time-locked recognition in the congruous case where the ending could be predicted earlier from the context. Besson and Kutas (1993) also examined the effect of repeating context per se on N400 amplitude in crossing context (same/different) with final word (same/different) in low cloze probability congruous sentences. They found that not only was there no advantage to repeating a final word with a different context, but the reverse, that repeating a context with a different final word did lead to an N400 reduction on the final word even though this final word had not been seen before. Interestingly, this reduction came on only in the second half of the N400 window, such that it patterned with unrepeated sentences in the ms window and with exactly repeated sentences in the ms window. Analyses across all conditions confirmed that an N400 amplitude reduction seemed to be associated with repetition of context regardless of whether the target word itself had been seen before. This raises the possibility that while the early part of the N400 may be sensitive to expectancy, the later part of the N400 may be sensitive to the cost of semantic integration, which is faster if the context has been processed before. Semantic priming. The reduced N400 for semantic priming has been found for both short and long SOAs (e.g. Boddy, 1986; Rugg, 1985; Anderson & Holcomb, 1995), different proportions of primed stimuli (Holcomb, 1988), in different modalities (Holcomb &

7 Anderson, 1993), and when the task decision is delayed until well after stimulus presentation (Kutas & Hillyard, 1989); thus, the N400 effect does not appear to be limited to only late/strategic aspects of priming in the sense of Neely (1977). Sentence effects. One of the most important things to remember about the classic N400 sentence anomaly effect first demonstrated by Kutas and Hillyard (1980) is that the amplitude of the N400 for such anomalies has generally been found to be identical to the amplitude of the N400 for unprimed content words in isolation (e.g., Kutas, 1993; Nobre & McCarthy, 1994). From this perspective, it seems most likely that the true effect in semantic anomaly manipulations is the reduction of N400 amplitude in the nonanomalous, congruous context case. This view is supported by studies showing that the amplitude of the N400 to congruous sentence completions varies as a function of the predictability of the completion given the context, such that more predictable endings have smaller amplitudes (e.g., Kutas & Hillyard, 1984; Federmeier, Wlotko, de Ochoa- Dewald & Kutas, 2007) and that as contextual information accrues as a sentence is processed incrementally, so the N400 for words decreases linearly as a function of sentence position (e.g. Van Petten & Kutas, 1990, 1991; Van Petten, 1993). Discourse effects. Many studies have examined the effect of manipulations of larger discourses on the N400, with the aim of showing that N400 amplitude reflects more than local information (e.g. van Berkum, Hagoort, & Brown, 1999; St George, Mannes, & Hoffman, 1994; van Berkum, Zwitserlood, Hagoort, & Brown, 2003; Camblin, Gordon, & Swaab, 2007). A particularly interesting example comes from St. George and colleagues (1994), who examined the classic Bransford and Johnson effect in which providing an explanatory title for otherwise semantically vague paragraphs vastly improves comprehension and recall. They showed that words in titled paragraphs showed reduced N400 amplitudes relative to words in untitled paragraphs. Since the words in the paragraphs are semantically vague, it seems unlikely that the N400 reduction could result from any kind of long-distance lexical-associative priming from the title, at least not in the traditional sense of priming; rather, it seems more likely that the N400 reduction was due to increased predictability or facilitated integration due to the organization provided by the title. Other properties. Although N400 peak latency is not often affected by experimental manipulation, a few studies have shown such effects. Kutas (1987) showed that the sentence-level N400 seemed to be delayed when the interval between words was very short. Holcomb (1993) showed in two experiments that the N400 peak to isolated words was latency-shifted by ms when the word was visually degraded, presumably because some earlier sensory recognition process took longer. It is interesting to note that N400 amplitude does not always follow behavioral effects of lexical processing. For example, Holcomb (1993) showed that the priming effect on lexical decision was stronger when visual stimuli was partially degraded, but the N400 amplitude priming effect was not larger for degraded stimuli, although as mentioned above, there were latency differences. Using repetition priming, Bentin and McCarthy (1994) showed that although RTs benefit from immediate repetition in both deep and shallow processing tasks (e.g. lexical decision vs letter identification), only the

8 deep tasks tended to show ERP repetition effects in the N400 window. Similarly, Kounios and Holcomb (1992) showed that RT in a sentence verification task did not correlate with N400 amplitude. One notable manipulation that affects N400 onset latency as well as sometimes peak latency is modality. The onset of differences in waveforms between anomalous and nonanomalous conditions occurs earlier with auditory presentation than visual presentation (Holcomb & Neville, 1990). When spoken sentences are recorded in one piece, such that coarticulatory cues are included, divergences can occur as early as 60 ms (Holcomb & Neville, 1991); when final words are recorded in isolation and spliced in, divergences occur around 200 ms, still a bit earlier than for visual presentation (McCallum, Farmer, & Pocock, 1984). This is probably because auditory information reaches the cortex faster than visual information, combined with the fact that the isolation point of words the point at which they can be distinguished from all other words in the language often occurs before the end of the word, meaning that in the auditory case, only part of the signal need be analyzed to do risk-free semantic processing. However, Van Petten and colleagues (Van Petten, Coulson, Rubin, Plante, & Parks, 1999) showed that the divergence between anomalous and expected endings begins before the isolation point of words (as long as the anomalous and expected endings do not share an onset), suggesting that the processes that give rise to the divergence start before the word can be unambiguously identified. The N400 effect due to priming or semantic anomaly is often followed by a late increased positivity between ms (in the same condition which shows an increased negativity in the ms window). This late positivity has received little attention until recently, and the conditions which give rise to it are still poorly understood (see Van Petten & Luka, 2006, for some discussion). We will return to this late aspect of the response later in the text. Topographic distribution. Across different modalities and different stimulus types, the scalp distribution of the N400 effect has been observed to vary. The N400 effect for both isolated words and sentences tends to be larger in posterior than anterior electrodes, although it has sometimes been argued that semantic priming is more likely to show up in anterior electrodes as well (Bentin, 1987; Bentin, McCarthy & Wood, 1985). The N400 elicited by pictures and faces tends to have a more anterior focus (Ganis, Kutas & Sereno, 1996; Holcomb & McPherson, 1994). It has been suggested, however, that decision tasks or other manipulations that lead to early P300-type positive components may be one cause behind some of the more anterior N400s, as such posterior positivities could cancel out the N400 in posterior electrodes (Kutas, Van Petten, & Besson, 1988). Nobre and McCarthy (1994) examined both the semantic priming and the sentence congruity N400 effects using a 50-channel recording array and argued that two subcomponents of the classic N400 effect were spatiotemporally differentiable an earlier left frontotemporal response peaking around 330 ms and a later midline centroparietal response peaking around 400 ms and that the priming manipulation showed a polarity reversal in the earlier subcomponent (more negative for primed items) that was not present in the

9 sentence N400 effect. To my knowledge this polarity reversal has not been replicated, although such a dense recording is unusual in N400 studies 3. Kutas, Van Petten, and Besson (1988) give a thorough discussion of hemispheric asymmetries in scalp distribution for the sentential N400 effect in visual presentation. Based on data from seven experiments, they show a small but highly replicable tendency for the effect to be larger at right than left hemisphere electrodes. The strength of this asymmetry consistently follows the manipulations that elicit the N400 (stronger differences between ERPs from the two hemispheres in the ms window for anomalous than completely predictable endings, stronger hemispheric differences for content words than function words) 4. Semantic priming tasks in the visual modality also sometimes show this rightward asymmetry (Holcomb, 1988; Holcomb & Neville, 1990; Holcomb, 1993) although not always (Holcomb & Anderson, 1993; Anderson & Holcomb, 1995, Kutas & Hillyard, 1989). The rightward bias has usually been interpreted as a case of paradoxical lateralization in which an effect with a lefthemisphere source appears in right-hemisphere electrodes due to fissural morphology and conductance properties (see King, Ganis, & Kutas, 1998, for another perspective). On the other hand, this asymmetry is reported not to be present for the N400 sentence effect when auditory presentation is used (see Van Petten & Luka, 2006, for a summary), which would be unexpected if the asymmetry above was due to the well-known leftward bias in modality-independent language processing. Functional interpretation of the N400 effect. There is some debate about what the N400 effect represents. One view argues that most such N400 effects are actually due to lexical priming of the continuation word by earlier words in the sentence (Diogo Almeida, manuscript; see Ratcliff, 1987, Bradley & Forster, 1987 for relevant discussion with respect to behavioral effects). Another view holds that the amplitude of the N400 reflects the ease with which the current word can be semantically integrated with the previous context (e.g. van Berkum et al., 1996), the idea being that it is more work to process an implausible continuation in a way that fits in with one s prior knowledge of likelihoods of events in the world a reasonable semantic interpretation must be in some sense coerced out of a string that seems unlikely on the first pass. A third view of the N400 holds that its amplitude indexes the extent to which the current word or semantic features of the current word were expected or predicted (Federmeier & Kutas, 1999; DeLong, Urbach & Kutas, 2005; Federmeier et al., 2007) satisfied expectations translating into a reduced N400 relative to baseline. Finally, another plausible view, that N400 amplitude indexes the degree of mismatch of prior expectations, has been less widely held because of findings that a reduction in the strength of the expectation set up by the context does not reduce the amplitude of the N400 in response to a congruent but unexpected verb (Kutas & Hillyard, 1984; Federmeier et al., 2007). We personally favor the third view and we believe that the discourse findings discussed earlier make the first view unlikely, 3 It is also possible that, given that the N400 priming effect is more likely to show up in anterior electrodes, this polarity reversal is often largely cancelled out, resulting in the slightly rightward-focused N400 that is also sometimes reported. 4 Interestingly, right-handed participants with left-handed family members seemed not to show such a hemispheric asymmetry.

10 but in this paper we will remain largely agnostic about the precise interpretation of the sentence-level N400 effect. When considering such candidate functional interpretations of the N400 effect, we should keep in mind that the question of what, if any, one underlying factor N400 amplitude correlates with is a separate question from what neural response the N400 effect is driven by. It may well be that the N400 effect itself is the neural pattern corresponding to a clean-up process like updating of priors that would correlate with lexical/conceptual factors but would not actually represent lexical or conceptual access by the brain. Such a position has been argued for by Sereno and colleagues (Sereno, Rayner, & Posner, 1998), who point out that eye-movements normally take place every 300 ms and yet seem to be affected by contextual plausibility (see also Hauk & Pulvermüller, 2004, for a similar view). Localization of the N400 in ERP. Localization from ERP data has occasionally been attempted for the N400 effect (Curran, Tucker, Kutas, & Posner, 1993; Johnson & Hamm, 2000; Frishkoff et al., 2004). Based on the polarity of potentials at different locations in a high-density 128-channel recording system, Johnson and Hamm argued for bilateral anterior medial sources, while Frishkoff and colleagues used an ERP source analysis technique to argue that left prefrontal and anterior cingulate areas were sources of both early and later parts of the N400 effect, and that right prefrontal and temporal cortex were additional contributors in the later part of the window. However, most ERP researchers agree that brain localization based on ERP data is still too difficult for the data to be taken as conclusive (see, for example, discussion in Luck 2005). In the next section we examine more promising techniques for addressing localization. As alluded to above, the most common stimuli used to probe the N400 have been words, and the most common manipulations showing an N400 effect have been sentence comprehension tasks in which degree of contextual fit is manipulated and single-word-presentation tasks in which the degree of priming between subsequent words is manipulated (the latter can be further divided into repetition priming and semantic priming). Because these are the most widely attested N400 manipulations, they are also the ones that have tended to be used in attempts to localize the effect. Although it is often assumed that the N400 effect in these two manipulations has the same functional locus, the manipulations themselves are fairly different, and as we will see, the fmri activation pattern they engender is somewhat different as well. Thus so as not to predetermine the issue, whenever possible we examine data from the two paradigms separately. B. Brain regions implicated in semantic processing fmri studies, intracranial studies, and aphasia studies have implicated three main regions in semantic processing: left inferior frontal cortex, left anterior temporal cortex, and left middle-posterior superior temporal cortex. Below we review some evidence of the general involvement of these areas in semantic processes, before examining the results of N400-like paradigms in particular. 1. Contribution of left posterior temporal areas to semantic processing

11 Because Wernicke s area traditionally thought of as posterior superior temporal cortex / angular gyrus was thought to be involved in the sound meaning transformation, it has sometimes been associated with semantic processing, in opposition to Broca s area, the only other traditionally recognized language center, which was supposed to be responsible for syntax. Of course, for a long time evidence from aphasia has argued that the role of left posterior STG is limited to this transformation, and that it is not involved in pure semantic storage or processing itself, as STG lesions usually cause defects in speech comprehension specifically and not semantic comprehension more generally, for example in written text ( pure word deafness - Barrett, 1910; Henschen, 1918). Also, imaging studies have more recently shown that posterior STG and STS is activated by speech regardless of the semantic content of the stimuli (Wise, Scott, Blank, Mummery, Murphy, & Warburton, 2001). However, recent work has begun to implicate the nearby angular gyrus as important for processing semantic information. The angular gyrus shows more activity for words than nonwords, seemingly driven both by increased activation for words and deactivation for nonwords relative to rest (Humphries, Binder, Medler, & Liebenthal, 2007; Binder et al., 2003, Binder, Medler, Desai, Conant, & Liebenthal, 2005; Ischebeck et al., 2004; Rissman, Eliassen, & Blumstein, 2003; Mechelli, Gorno-Tempini, & Price, 2003). A study by Humphries and colleagues (Humphries et al. 2006, 2007) showed that activity in the angular gyrus was greater for semantically congruent sentences than when the same semantically related words were scrambled into a word list or than when the same sentence frames were maintained but the content words replaced by unrelated ones. Evidence from several directions has recently been converging on a role for STS/MTG and ITG in semantic storage (e.g., Dronkers, Wilkins, Van Valin, Redfern, & Jaeger, 2004; Humphries et al., 2007; Indefrey & Levelt, 2004; Damasio, Grabowski, Tranel, Hichwa, & Damasio, 1996; Vandenberghe, Price, Wise, Josephs, & Frackowiak, 1996; Fiebach, Friederici, Muller, & von Cramon, 2002; Binder et al., 1997). Dronkers and colleagues showed in a large-scale lesion analysis study that while lesions in other areas were correlated with difficulty in sentences of varying complexity, MTG was the only region in which lesions led to significantly lower performance on the simplest constructions like The girl is sitting. This led them to argue that the left MTG is associated with processing of basic lexical semantics. Several fmri studies employing semantic tasks like categorization or semantic property judgments have shown activity in this region (Pugh et al., 1996; Cappa, Perani, Schnur, Tettamanti, & Fazio, 1998), and some fmri studies have also shown a word > pseudoword effect in left MTG (see review in Fiebach et al., 2002). Indefrey and Levelt review the fmri literature on language production and implicate the left MTG region in conceptually-driven lexical retrieval. Hickok and Poeppel (2004) review similar lines of evidence for involvement of left inferior temporal areas in lexical processing; they suggest that left posterior IT may be the locus of the sound-meaning interface for words. 2. Contribution of left anterior temporal areas to semantic processing Although the classic model focused attention on the role of left inferior frontal and posterior temporal areas in language processing, converging evidence compellingly points to the idea that left anterior temporal cortical structures (LATC) play an important part in sentence processing. Imaging studies which contrast sentence processing with

12 low-level baselines like consonant strings and reversed speech often find activation in anterior STS/MTG (anterior BA 22) and the temporal pole (BA 38), bilaterally but more prominent in the left than the right hemisphere (Bavalier et al., 1997; Scott, Blank, Rosen, & Wise, 2000; Crinion, Lambon-Ralph, Warburton, Howard & Wise, 2003; Noppeney & Price, 2004). A similar pattern of activation in left anterior STS and/or left TP is seen when reading or listening to sentences is contrasted with reading or listening to word lists (Mazoyer et al., 1993; Stowe et al., 1998; Friederici, Meyer, & von Cramon, 2000; Vandenberghe, Nobre, & Price, 2002; Humphries, Love, Swinney, & Hickok, 2005; Humphries, Binder, Medler, & Liebenthal, 2006). Further, although an early study of left TP lobectomy patients suggested that comprehension was not affected (Saykin et al, 1995), a more comprehensive recent study correlating lesion location with sentencepicture-matching deficits in 64 left hemisphere patients showed that left anterior BA 22 damage was associated with significant impairment on most sentence types more complex than simple declaratives (Dronkers et al., 2004). Other studies have ruled out the possibility that these effects are wholly due to more general aspects of processing meaningful sequential auditory events (Humphries, Willard, Buchsbaum & Hickok, 2001) or prosodically contoured stimuli (Humphries et al although a smaller portion of left ATC was sensitive to both prosody and sentential structure). Combined, these studies make a strong case for the involvement of left anterior temporal cortex in sentence processing; what authors in this literature are currently debating is whether the (presumably combinatorial) operations driving the anterior temporal activation are of more of a semantic or syntactic nature. Consistent with the syntactic view, several of the fmri studies also found a similar pattern of activity in left ATC for pseudowords within sentence frames (Jabberwocky) relative to pseudowords in lists, demonstrating that lexicosemantic content is not required to activate ATC (Friederici et al, 2000; Humphries et al, 2006) 5. Noppeney and Price (2004) also showed that left ATC showed a reduction in activity that seemed to correspond to syntactic priming, when blocks of sentences with similar structures were read. Some of these authors thus argue that left ATC may be involved in an early stage of syntactic parsing and phrase structure analysis (e.g. Friederici et al, 2000; Humphries et al, 2006); in a slight variation, Dronkers and colleagues suggested morphosyntax to be the relevant level (Dronkers et al, 1994), although in recent papers they are less tied to this view. However, many of the aphasia results could be equally well explained if it is semantic compositional processes like thematic role assignment that the left ATC performs, as suggested by other authors (Vandenberghe et al., 2002, Noppeney & Price, 2004). In support of this alternative view, an early, influential PET study on a semantic decision task found an area of left TP whose activity was correlated with the amount of semantic relatedness of the items presented in a given block (Mummery, Shallice & Price, 1999). A more recent fmri study also found activation in left TP for a semantic task (synonym judgment) relative to phonological and orthographic tasks (Gitelman, Nobre, Sonty, Parrish, & Mesulam, 2005). Intracranial recordings from the anterior temporal lobe have also been argued to show semantic effects; these will be discussed more fully in a later section. The partial answer to this sub-debate may be simply that further functional subdivisions exist within the left TP/anterior STS area. The two most thorough imaging 5 Although of course, syntactic structures themselves might also be thought to convey meaning.

13 studies on this issue both found areas of the anterior temporal lobe that differentially responded to normal and semantically random ( word salad ) sentences, areas that differentially responded to sentences and word lists, and areas that showed an interaction between these factors (Vandenberghe et al, 2002; Humphries et al, 2006). We also know that a part of left ATC has been associated with a seemingly completely unrelated function, naming of proper nouns (e.g. people and buildings; Damasio et al., 1996; Gorno-Tempini & Price, 2001). Thus, it may be that the left anterior temporal area collects many functions only loosely related by virtue of requiring some higher-order convergence of lower-level information (e.g. Damasio & Damasio, 1994). Taken together, however, the results of the two above-mentioned studies suggest that at least some part of the left anterior temporal cortex is associated with semantic processing at some level. 3. Contribution of frontal areas to semantic processing Although the contribution of left inferior frontal cortex to language processing was traditionally thought to rest in language production, and more recently in syntactic comprehension, in fact in the imaging studies of the past 15 years, areas of left inferior frontal cortex have been reliably associated with semantic processing (see Bookheimer, 2002 for a review). This literature is particularly interesting to consider with respect to the goal of subdividing the inferior frontal cortex into functional subparts, as perhaps the most reliable functional subdivision in this area is the association of anterior ventral prefrontal cortex (BA 47) with some aspect of semantic processing. Based on cytoarchitecture, three main areas of the inferior frontal gyrus (IFG) can be described: BA 47, or pars orbitalis, the most anterior and inferior/ventral; BA 45, or pars triangularis which is more superior and sometimes is considered to extend into middle frontal gyrus, and BA 44, or pars opercularis, which is more posterior (see figures below). From Demonet et al. 2006

14 From Badre et al All three areas have been shown to be activated differentially in at least some semantic tasks. For example, in an early, elegant fmri block-design study, Demb and colleagues (1995) found that not only were BA 45 and 47 activated in a semantic decision task (abstract vs concrete nouns) relative to a difficulty-matched perceptual task, but that the same areas showed repetition-priming deactivation when words were repeated for semantic decision, but not when the second repetition demanded a perceptual decision. The same repetition-priming effect in BA 44, 45, 47 was replicated by Buckner et al. (2000) using both word-stem and verb-generation tasks, in both auditory and visual modalities. The repetition priming effect also held for pictures (Wagner, Desmond, Demb, Glover, & Gabrieli, 1997). However, beginning with Petersen et al. s (1988) PET study, numerous studies have implicated the anterior ventral IFG (BA 47 and sometimes 45) in particular in semantic processing (Binder et al., 2003; McDermott, Petersen, Watson, & Ojemann, 2003; Mechelli et al., 2007; Poldrack et al., 1999; Roskies, Fiez, Balota, Raichle, & Petersen, 2001; Humphries et al., 2007) using tasks such as animacy judgment, semantic categorization, comparison between semantically normal and semantically random sentences, etc. Vandenberghe and colleagues showed in a semantic matching task that not only words but also pictures elicit increased activation in this area (Vandenberghe et al., 1996). Perhaps more impressive than the fact that this area is consistently activated in semantic tasks is that several studies doubly dissociate activation in this area from activation in other parts of IFG in the same subjects. Looking at LIPC with a block design, Poldrack and colleagues (1999) found activation in BA 44, 45 and 47 for a semantic decision task (abstract/concrete), but only in left BA 45 (and right BA 44) for a phonological task (syllable counting). Based on this result and a meta-analysis of previous literature, they argued that anterior/inferior LIPC is specialized for semantic processing. In a similar vein, Dapretto and Bookheimer (1999) used a block fmri task on sentence processing to dissociate syntactic and semantic areas: using the same materials and the same task (judging whether two sentences match in meaning) across conditions, they manipulated whether syntactic transformation or lexical semantics was

15 required to judge a given item pair. They showed BA 47 activation for the semantic condition and BA 44 activation for the syntactic condition when the two were contrasted (both activated BA 45). Wagner and colleagues (Wagner, Koustaal, Maril, Schacter, & Buckner, 2000) replicated Demb et al. s (1995) task (perceptual vs semantic) x repetition manipulation with a bigger field of view and found repetition priming in anterior IFG (BA 45/47) only for the semantic-semantic condition, while posterior IFG (BA 44) showed repetition priming for the perceptual-semantic condition as well (see also Gabrieli, Poldrack, & Desmond, 1998). Similarly, Humphries and colleagues (2007) found that while both anterior and posterior IFG showed increased activity for words in lists and sentences, posterior IFG showed some activity for pseudowords above baseline but anterior-inferior IFG showed no significant activity for pseudowords. Finally, Gough and colleagues conducted a transcranial magnetic stimulation study in which participants performed a semantic task or a phonological task and stimulation was applied to either anterior or posterior LIFG; they found a double dissociation such that anterior LIFG stimulation selectively increased RTs for the semantic task and posterior LIFG stimulation for the phonological task (Gough, Nobre, & Devlin, 2005). It is also interesting to note that anterior LIFG was the only left frontal area to be activated in a more ecological, passive listening experiment which compared narratives and reversed speech (Crinion et al., 2003), suggesting its contribution to language processing is perhaps less strategic than the other frontal areas. Wagner and colleagues have argued that BA 47 underlies controlled semantic retrieval in particular, while at least one of the functions of BA 44/45 is selection among competing alternatives in working memory (Thompson-Schill, D Esposito, Aguirre, & Farah, 1997; Thompson-Schill, D Esposito, & Kan, 1999). In Wagner, Paré-Blagoev, Clark, and Poldrack (2001), the authors show that while BA 44/45 was activated above baseline (fixation) for all four cue-target semantic matching conditions, BA 47 was only activated when the cue-target semantic association was weak. Wagner et al. argue that this is because in the strong association condition, the target was automatically preactivated, while in the weak association condition, controlled top-down retrieval was necessary to activate long-term memory representations, invoking BA 47. Badre and colleagues (Badre, Poldrack, Paré-Blagoev, Insler, & Wagner, 2005) show that in the same type of similarity-matching task BA 44/45 was more activated in conditions that were designed to increase selection difficulty in different ways (increased number of possible targets, cue-associated foils included among the possible targets, matching similarity on a specific dimension rather than global similarity, association of target and cue), while BA 47 was sensitive to the strength of the cue-target association only. From the work reviewed so far, then, the general picture is that anterior ventral LIFG (BA 47) reflects some form of controlled semantic retrieval or semantic working memory, while more posterior and dorsal LIFG (BA 44 and 45) reflect working memory and selection processes more generally (see Vigneau et al for a more detailed, albeit more tenuous, ontology). Thus, we might expect to see different areas of LIFG activated for different reasons in semantic processing: BA 47 should be activated when controlled semantic retrieval is required for a task, while BA 44 and 45 may be activated when a semantic task requires any of the other domain-general processes that have been implicated there, such as selection and working memory. This seems to be true: for example, for sentences containing semantically ambiguous words relative to those

16 without, Rodd and colleagues find significant activation in dorsal LIFG/LIFS, BA 45/9, with or without an explicit task (Rodd, Davis & Johnsrude, 2005). Even though this is a semantic task, the idea is that it activates more domain-general BA 45 rather than more semantic-specific 47 because the ambiguous words presumably create a greater selection load. Similarly, Fiebach and colleagues (2002) find significant activation in left BA 44/45 for low-frequency words relative to high-frequency words, where the lowfrequency words perhaps require more active retrieval than high-frequency ones. In ending this summary of frontal involvement in semantic processing, however, we should sound one cautionary methodological note. Many studies which are not specifically interested in the subdivisions between different areas of LIFG may report activity locations in a way that does not reveal the full extent of the activity for example, the location of a cluster of activity may be reported in BA 45, without noting that it also extended to BA 44 and BA 47, or that the activity was more intense in one of those areas than in the center of activity that is reported. Also, an unavoidable problem in fmri analysis that is especially troubling in attempting to segment the LIFG is individual anatomical variability; when all individual activity maps are transformed to the standard brain and averaged together, the focus of activity may well appear to be shifted from the true location in the individuals. Thus, conclusions about activity found in one LIFG region over another should be made tentatively. C. Localizing the N400: different methodologies and their results 1. Aphasia Studies of aphasia can potentially inform us about the generators of the N400 in one of two ways: indirectly, by showing that brain damage to a given region alters a behavioral phenomenon that correlates with N400 amplitude (e.g., priming), or more directly by showing that brain damage to a given region alters the N400 effect recorded in patients. While data from aphasia is informative, we must be sure to keep in mind the traditional caveats of interpreting lesion data (post-lesion plasticity and flexibility, incomplete damage, connectivity). In an excellent discussion of these issues, Price (2000) gives the example of a group of patients with semantic dementia, a disease known to damage the anterior temporal lobes and cause semantic deficits, who only showed mild behavioral impairments. When these patients were scanned, they were shown to actually have more activity in the anterior temporal region than normal controls, suggesting that their damage was not complete and they were able to compensate for it. Keeping these possibilities in mind is particularly important when assessing the shakier reverse inference that is often made, that if brain damage to a given region does not affect behavior or the size of the N400 effect, this region is necessarily not an N400 generator Behavioral: Priming. Several studies have found normal levels of behavioral semantic priming (both associate and category) in aphasics with frontal lobe damage (e.g. Gershberg, 1997, Blumstein, Milberg & Shrier, 1982; Milberg, Blumstein, & Dworetzky 1988; Hagoort, 1993; Ostrin & Tyler, 1993). Interestingly, the few cases in which Broca s patients showed somewhat abnormal or lack of priming were those which used longer SOAs (Milberg & Blumstein, 1981; Milberg, Blumstein, & Dworetzky, 1987),

17 which might be seen as convergent with the fmri data showing frontal activity for priming only for long SOAs. Behavioral: Sentential integration. Friederici (1983) showed that in a word-targetdetection task within the second sentence of two-sentence discourses, both patients who classified Broca s and patients who classified Wernicke s (lesion location described as left anterior cortex for Broca s and left posterior cortex for Wernicke s) showed a normal facilitation in RTs to open-class target words when the first sentence was contextually congruent vs neutral to the second sentence. ERP: Priming. The most comprehensive examination of the N400 semantic priming effect is probably the study by Hagoort, Brown, and Swaab (1996), who examined both associative and semantic category priming in 13 Broca s and 7 Wernicke s aphasics. From the anatomical data shown, the lesions of the Broca s patients tended to include both left frontal and left temporal lesions, while the Wernicke s patients lesions were limited to left temporal areas. They found that both patient groups showed N400 effects for both prime types. There was some hint that both the N400 component itself and the N400 priming effect might have been slightly smaller and differently distributed in the Wernicke s group, but the small size of the patient group precluded any strong conclusions. Similarly, Kojima and Kaga (2003) also found an N400 priming effect in a group of 10 left-hemisphere aphasics of mixed classification. In a very interestingsounding study described in Kotz and Friederici (2003), Kotz and colleagues examined priming in patients with left and right anterior temporal lesions and claimed to find some reduction and delay in the N400 effect; however, the study was only published in abstract form (Kotz, Friederici, & von Cramon, 1999). ERP: Sentence anomaly. Swaab and colleagues also conducted an ERP study of N400 effects to sentence anomaly in left-hemisphere aphasics (Swaab, Brown, & Hagoort, 1997); however, since they analyze the results in terms of their performance on comprehension batteries which in their subject pool appeared to correlate only with lesion size and not location no conclusions about localization can be drawn from their study. Friederici, von Cramon, & Kotz (1999) examined the sentence anomaly effect in 3 patients with left frontal cortical lesions and 4 patients with basal ganglia damage and found N400 effects in both groups, although slightly reduced in the cortical group 6. This result is hard to assess, however, as visual examination of the subject data suggests that just 1 out of the 3 cortical patients and 2 out of the 4 subcortical patients showed a measurable N400 effect, which may or may not reflect normal amounts of inter-subject variability. Friederici, Hahne and von Cramon (1998) also claimed to find an N400 sentence-level effect in a single left frontal patient, and not for a single left temporal patient, but visual inspection of the frontal waveform for the frontal patient shows a phase-shift between the anomalous and non-anomalous conditions (later peak for the anomalous condition) rather than an amplitude difference, and it is hard to draw strong conclusions from two single cases. 6 The authors argued that the amplitude reduction was due to latency jitter and not to an underlying reduction in amplitude.

18 In summary, the existing aphasia data does not provide very clear evidence either way on the contribution of different brain areas to the sentence-level N400, as the studies reviewed either showed the lesioned populations to have normal N400s, did not report data according to lesion location, or had such a low n and high inter-subject variability that they would need much more supporting evidence to be taken seriously. 2. Intracranial recordings Probably the most direct way of determining an anatomical region s involvement in contributing to the N400 potential measured in ERP is data from intracranial recordings on pre-operative epileptic surgery candidates. For such patients, recordings are collected from surgically inserted electrodes over several days in order to try to locate the focus of the seizures; in the meantime, experimental manipulations can be conducted and eventrelated potentials can be gathered from the recordings. Although the recordings are obviously limited just to the areas which are plausible sources of the seizures of an individual patient, across a large number of patients, data can be gathered from a number of areas. One of the most frequently sampled areas, for clinical reasons, is the medial anterior temporal lobe; the hippocampus and the lateral temporal lobe are also often probed. Over the years, a number of intracranial recording studies have tested N400 manipulations on such patients in order to try to localize the component, and many of them have found such effects in the anterior medial temporal lobe; this response is sometimes termed the AMTL-N400. Smith, Stapleton, and Halgren (1986) showed in evoked potentials recorded from medial temporal cortex a component peaking around 460 ms in response to visually presented words which was reduced in amplitude to repeated words and greater in amplitude in the left hemisphere. Nobre, Allison, and McCarthy (1994) found a posterior area of the fusiform gyrus which showed an early negative response (N200) specific to letter-strings and not other visual stimuli, and an anterior area of fusiform gyrus that showed a later positive response (P400) greater in amplitude to anomalous sentence endings and unprimed words. In a 1995 study of 10 patients using mainly AMTL depth electrodes, Nobre and McCarthy showed that electrodes in AMTL near the amygdala showed a negative deflection around ~400 ms for visual word stimuli in a semantic-category detection task. This response showed greatest amplitude for content words, less for phonotactically legal pseudowords, less still for function words, and least for phonotactically illegal letter strings. Like the scalprecorded N400, this response also showed a small but significant reduction for the second word of semantically related word pairs using the same detection task, although this effect was less localized in time and was not compared against non-related pairs. For some individuals in which surface recordings were made from medial and inferior ATL, these electrodes showed a similar response, again greater for content than function words. Halgren and colleagues (1994a) also found a peak of similar latency to words and faces in AMTL. Although their sample included fewer participants (~10-20 participants per recording location), they sampled more locations, and showed a similar peak at selected electrodes in the hippocampus, amygdala, lingual gyrus, fusiform gyrus, posterior medial STG and MTG, supramarginal gyrus, and posterior cingulate gyrus. In a corresponding paper (Halgren, Baudena, Heit, Clarke, Marinkovic, & Chauvel,1994b) they also showed

19 a similar ~400 ms peak to words and faces from electrodes in prefrontal and orbitofrontal cortex. Elger and colleagues (1997) examined repetition priming effects for both words and pictures in recordings from both medial (AMTL) and lateral temporal (LTL) sites in 26 subjects and saw reliable responses for both types of stimuli in both regions, also modulated by repetition. Repetition effects in AMTL were significant only in the nonepileptic lobe, providing stronger evidence that this area is a generator for the N400 repetition effect. They were also able to show that N400 amplitudes for words were actually correlated with performance on verbal recall tasks for left hemisphere electrodes, while N400 amplitudes for pictures were correlated with performance on picture recognition tasks in the right hemisphere. Grand averages and selected individual waveforms suggested that the left LTL responded much more strongly to words than pictures, while left AMTL responded to both with nearly equal amplitudes. Interestingly, the left LTL sites showing stronger responses to words than pictures were most likely to be placed over posterior MTG compared to the more anterior LTL sites. McCarthy, Nobre, Bentin and Spencer (1995) examined the N400 response to semantic anomaly in 64 patients recording largely from AMTL. They found a strong deflection to anomalous endings bilaterally in AMTL near the amygdala, with onset around ~300 ms, and peaking between ms. More superiorly located AMTL electrodes did not show the effect. More unexpectedly, an effect of anomaly was also seen in electrodes located in the hippocampus, which was later ( ) for most subjects, although some showed a biphasic effect with an early peak that mirrored N400 latency. In their corresponding word-priming study, McCarthy and colleagues found that recordings from hippocampus showed a similar, task-related response for targets of the detection task, which may be related to the P300 (Nobre & McCarthy, 1995). Thus, even though in the semantic anomaly manipulation the task was just passive reading, it seems possible that the hippocampal response is also somehow related. At the same time, this late effect might be related to the post-n400 positivity that is sometimes seen in scalp recordings; and in fact, such a late positivity, perhaps stemming from the hippocampal area, was seen in some of the AMTL recordings here. The results of the intracranial recording studies cannot be seen as the last word on the source(s) of the N400 response observed in the scalp recordings for the reasons stated above: regions are not exhaustively sampled, and the extent of the effect of the pathology of these patients on localization is unknown. On the flip side, however, these studies are extremely important in that they do fairly definitively show that the regions which do show such an effect are quite likely generators (the only caveat to this is the possibility that due to accidents of orientation etc. they match but don t contribute to the scalp signal). Thus, the studies reviewed converge in suggesting that the generators of the N400 response to words include at least both anterior medial temporal structures and posterior middle temporal structures, and that the generators of the N400 response to semantic anomaly within sentences includes anterior medial temporal structures. 3. fmri A sizeable literature has examined the fmri response to the kinds of manipulations that affect N400 amplitude, most importantly semantic priming and contextual semantic fit. From study to study slightly different design parameters are used, and from study to

20 study fairly different results are sometimes reported. In order to give the reader a better idea of which fmri findings are most consistent across studies, we report below two meta-analyses, for the priming studies and for the context studies. Tables 1 and 2 provide a good overview of the findings, while the text provides more details for the interested reader. To keep the meta-analyses more manageable, we focus on the left hemisphere activations only, and report only those cortical areas that are of interest due to the previous literature and/or were found to show an effect by more than one or two studies. N400: priming. In Table 1 we present a meta-analysis of 15 fmri contrasts between processing semantically related and unrelated word pairs, from 11 imaging studies. These contrasts differ on several important parameters that have been shown to affect behavioral priming, among them modality of presentation, task, SOA (stimulus-onset asynchrony), proportion of related pairs, and the nature of the prime-target relationship. Many researchers in the behavioral literature have differentiated two kinds of priming, automatic (e.g. spreading activation through connected semantic nodes ) and strategic (e.g. generating expectancies for faster processing, or using semantic match as a heuristic for lexicality decisions), and have argued that the former is dominant at short SOAs (~250 ms) while the latter is dominant at longer SOAs (~1000 ms) (e.g. Neely, Keefe & Ross, 1989). The prime and target can also be related in various ways (purely associative bread-butter; shared category membership turtle-lion; similarity cord-string), which has also been argued by researchers to affect the priming process (e.g. Moss, Ostrin, Tyler, & Marslen-Wilson, 1995). Another issue to keep in mind is that two kinds of effects have been found in previous, non-linguistic fmri studies of priming or adaptation: suppressive effects and enhancing effects. The reason for suppressive effects of priming, in which primed stimuli show less activation in some areas than others, is intuitive: less effort is needed to process stimuli a second time, where effort can be cashed out in different ways. The reason for enhancement effects is less obvious, but in some situations, for example, increased activation for primed stimuli may represent episodic memory processes (see Henson 2003 for a comprehensive discussion of these issues). The bottom line is that, in contrast to most fmri manipulations, both increased and decreased activation could reasonably be expected in the condition of interest, and both are seen in the meta-analysis presented. However, a serious complication toward doing a full comparison of suppression and enhancement effects across studies is that many studies do not report the related > unrelated contrast at all, so that there is no way of knowing whether they might have found areas of priming enhancement in addition to the areas of suppression that they report. 14 of the 15 priming contrasts examined showed effects of priming somewhere in left temporal cortex, although the location varied (note that the one contrast which did not show a left temporal activation for priming did show an activation in right STG). 9 of the 15 contrasts showed priming effects in left MTG, and 4 of the 15 contrasts showed priming effects in left posterior-inferior temporal areas, consistent with the growing consensus discussed above that these areas are key to storage of lexical meanings. 3 of the 15 contrasts from two studies showed priming effects in left anterior temporal areas, although the activation that Kotz et al. (2003) report is more lateral and the two contrasts Rossell et al. (2003) report are more medial. 4 of the 15 contrasts showed effects on the

21 left posterior-superior temporal edge in the vicinity of the supramarginal and angular gyri. Most of these contrasts showed reduced activity for primed conditions, which could be interpreted as reflecting less effort needed to retrieve such pre-activated lexical items. 9 of the 15 priming contrasts showed effects of priming somewhere in left frontal cortex. Frontal activation seemed to be associated with SOA, as none of the 5 contrasts with a short SOA found frontal effects, and 9 of the 10 contrasts with a long SOA did. Furthermore, Gold and colleagues (2006) tested this contrast with both SOAs on the same subjects, and found significant frontal effects only in the long SOA condition. The one long SOA contrast which did not find frontal effects was Rossell et al. (2003), which used a different prime-target relation than most of the studies (category relationship) and which also did not find the differential activation in left MTG that most of the other studies did. There seemed to be a slight tendency for contrasts using visual presentation to find ventral LIFG effects (~BA 47/45) and contrasts using auditory presentation to find dorsal effects (~BA45/44/46, BA6/9), although there weren t enough auditory studies to draw any strong conclusions about this. Across both the temporal and the frontal priming effects some contrasts found enhancement for priming and some found suppression, and it is unclear what determined this alternation. As visual inspection of Table 1 illustrates, suppressive effects (less activation for primed words) were by far the majority, while just two studies contributed most of the enhancement effects (Raposo et al and Copland et al. 2007). Raposo et al. (2006) found only enhancement effects across various cortical areas for priming. However, their task (abstract/concrete decision), their presentation (all words were targets) and the nature of the prime-target relationship (similar and unassociated) were different than all the other studies, so perhaps one or both of these factors was behind the opposite direction of their effects. More problematic is the study of Copland and colleagues (2007), in which the authors found enhancement for related relative to unrelated associate pairs in BA 44 using the standard lexical decision task. A possible explanation is in their use of homonyms as primes in both the related and unrelated conditions; perhaps this gave rise to some competition-type effect in the related condition that did not hold in the unrelated condition, and which did not have time to come into play in their earlier (2003) experiment which used the same materials with a shorter SOA. Across these priming studies then, the most common finding with respect to frontal areas was significantly less activation in LIFG (BA 44/45/47) for related pairs relative to unrelated pairs, when the SOA between prime and target was greater than 250 ms. How should we interpret this relative suppression? In keeping with the proposal of Wagner and colleagues that BA 47/45 underlies controlled semantic retrieval and BA 45/44 selection/competition, we could think that in a long SOA task these areas mediate some kind of strategic processing triggered by the presentation of the prime, such that when the expected or matching target is encountered, they can stop working, while when an unexpected or nonmatching target is encountered, they must work harder for longer to complete retrieval and selection of the target, potentially resulting in an increase in signal in these areas. In fact, Gold et al. (2006) show that BA 44/45 and BA 47 dissociate when a neutral prime condition is added ( BLANK ); in BA 44/45 the unrelated condition shows the most activation, with the neutral condition patterning together with the related

22 condition, while in BA 47 the neutral condition patterns with the unrelated condition in showing significant activation above baseline (see Figure x). Gold and colleagues use these findings to argue for a view like that of Wagner and colleagues (e.g. Badre et al. 2005), in which BA 47 is sensitive to retrieval difficulty and BA 44/45 to competition. Although these generalizations are suggestive, some questions still remain with respect to interpreting the priming findings in fmri. Even if we take as given the idea that BA 47 underlies controlled semantic retrieval, it is still not clear whether the use of BA 47 to process words should be taken to be the default case (e.g. Crinion et al. s finding of BA 47 in passive listening, Noppeney and Price s 2004 finding of BA 47/45 in passive reading) or the marked case (the view of Wagner and colleagues, e.g. Badre et al. 2005), and this confusion easily transfers to the interpretation of relative differences in signal across conditions with different temporal properties. Clearly, more work will be needed to sort these issues out; however, the meta-analysis together with Gold and colleagues comprehensive study suggest that SOA is the main determinant of whether frontal areas show sensitivity to priming in isolated word tasks in fmri. Since the N400 effect for priming of isolated words has been shown to be robust regardless of SOA, it is thus unlikely that the N400 effect can be tied to these frontal activations found in priming tasks in fmri (although this certainly does not rule out the possibility that the N400 effect has a frontal generator which does not show up as differential activity in fmri recordings). Imaging findings: phrasal context. In Table 2 we present a meta-analysis of 22 fmri contrasts between processing semantically implausible/anomalous/unexpected continuations and normal continuations, from 17 imaging studies (this represents an enlarged version of the meta-analysis presented by Van Petten & Luka (2006); several unpublished manuscripts are included). Factors included in the table are modality, material type (sentence or phrase), presentation design (block, oddball, slow or rapid event-related), contextual manipulation, task, and whether the anomaly was phrase/sentence-final. First looking for activations in LIFG taken broadly, we find that 17 of the 22 contrasts found an activation in LIFG for the semantic-normal contrast. Examining the five that did not find such an activation, three of those five (Kuperberg et al. 2000; Ni et al. 2000) used some form of a block design (where the block included a mix of normal and semantically anomalous sentences), which might affect either processing demands or measurement of the effect,, and two (Ni et al., 2000; Dien et al., under review) found an activation in nearby left middle frontal gyrus (BA 9/46). Importantly, there was no

23 obvious relationship between frontal activation and task frontal activations were found with both explicit judgment tasks and with unrelated tasks or no tasks which argues against the idea that the frontal contribution to semantic processing is purely strategic or task-based (e.g. Hasson, Nusbaum & Small, 2006). Although 11 of the 22 contrasts found activation in some left temporal area for the semantic-normal contrast, the location of this activity seemed more variable than in the frontal case. 7 contrasts found activity in left superior temporal gyrus or sulcus, although 2 of these contrasts were nonsignificant. One study reported left middle temporal gyrus activity (Stringaris et al. 2007), one study reported left anterior temporal activity (Kiehl et al., 2002), and five contrasts showed activity somewhere near left postero-inferotemporal cortex. There was no obvious predictor of temporal involvement among the factors of modality, task, presentation design, or position of anomaly (although it did happen to be the case that the contrasts showing temporal activity all had the anomaly in trial-final position). The fact that over half of the contrasts examined showed temporal activity for semantic anomaly does provide evidence for a temporal component to the fmri response, but its lesser reliability and its increased spatial variability makes it hard to draw any further conclusions about this response s source. Only 11 of the 17 contrasts that found frontal activations found signs of activation in BA 47, the area that has been previously linked to controlled semantic retrieval / semantic working memory. 7 Again, task did not seem to play a clear role in the presence or absence of this activation, nor modality or context length. We may note here that in several studies that performed two contrasts with minimal differences, the authors found BA 47 activity in one contrast but not the other (Braze et al. submitted, visual vs auditory; Rüschemeyer et al and Friederici et al. 2003, visual vs auditory; Rüschemeyer et al. 2005, German speakers vs Russian). Perhaps the BA 47 contribution, though real, is weak such that it is sometimes missed (more often in auditory, for some reason?), or perhaps the close spatial proximity of the three subdivisions of LIFG cause their activations to sometimes blur together due to inhomogeneities in cortical location across subjects. 12 of the 17 contrasts with frontal activations found activation in BA 45 (11), BA 44 (1), or both (5), 5 of these also found activation in BA 47, and again, there was no factor clearly responsible for the difference in results no obvious difference in selection or working memory demands that might lead to more activity in one of these areas than another. On the other hand, perhaps there remains some undiscovered regularity among these experiments that would explain the difference. In sum, the processing of words that are semantically inconsistent with the preceding context seems to fairly reliably activate left inferior frontal cortex, but so far no one particular area of LIFC has been implicated, as seemingly similar studies activate different areas. One interesting but preliminary finding was that of Baumgaertner et al. (2002) in which the data suggested a dissociation between plausibility and expectation: stronger LIFG activation for implausible continuations (relative to normal) but stronger STG activation for unexpected continuations. Given the dissociations between BA 47 and BA 45/44 demonstrated in the semantic retrieval vs selection debate discussed above, 7 Two other contrasts possibly activated this area: the plausible-unexpected activation of Baumgaertner et al (2002) was not significant although it also did not significantly contrast with the implausible condition which did show a significant frontal activation, and it was not clear whether the activation in the auditory condition of Braze et al (submitted) was in BA 47 or 45

24 it would be nice to see more studies like Baumgaertner et al. s that manipulate these types of demands in the sentence domain e.g. contextual constraint or expectation to see if they can be spatially dissociated within a single group of subjects. Summary The fmri studies reviewed above find that two manipulations which result in a reduced N400 (priming in isolated word presentation and contextual semantic fit in sentence presentation) also fairly reliably result in reduced left inferior frontal activation under certain conditions (i.e. long SOA for priming). It seems somewhat unlikely that the frontal effect is purely strategic, because the fmri sentence studies found frontal activation across various tasks and in a few cases, without any overt task; however, strategic factors to the extent that they are reflected by SOA do seem to be relevant for the priming cases at least. As reviewed earlier, N400 effects for priming have been found at short SOAs, although they have sometimes been found to be smaller and more short-lived (Rossell et al., 2003) or less frontally distributed in the later part of the N400 window (Anderson & Holcomb, 1995) than at long SOAs. Thus, while it remains a possibility that the left IFG activity seen in fmri for N400 manipulations may be a component of the N400 effect, the fact that this activity is not seen in a condition which does show an N400 priming effect (short SOA) provides a strong argument that this frontal activity is not the sole basis of the N400 effect. The priming contrasts fairly consistently showed an effect in left temporal cortex, most reliably in the middle/inferior temporal area, but also occasionally in superior or anterior temporal areas. Only about half of the context contrasts showed a left temporal effect, and these were most often in superior mid-posterior temporal cortex. Given previous demonstrations from the ERP literature that the N400 priming and contextual effects have similar latency and scalp distribution (Kutas, 1993; van Petten, 1993), we might be surprised that the pattern of fmri activity would be different. On the one hand, it may be that the priming and contextual N400 effect may have slightly different loci in temporal cortex, but that the difference is small enough that it doesn t affect the ERP scalp distribution. On the other hand, it may be that this temporal cortex activity seen in fmri in particular the middle/inferior temporal effect seen so reliably in the priming studies is actually not part of the activity reflected in the N400 response. 4. MEG Following an early brief report of some measured success in recording evoked magnetic fields to N400-type semantic anomaly (Arthur, Schmidt, Kutas, George, & Flynn, 1989), Simos and colleagues were the first to show strong evidence for an N400-sentence response in MEG (1997). They recorded both ERP and MEG simultaneously, and thus were able to show that both measures demonstrated a similar response-amplitude difference in the same participants, resembling a classic N400. Using an equivalent current dipole (ECD) model and MRIs for a subset of subjects, they localized this response to left posterior temporal cortex in 5 subjects and left medial temporal cortex in 2. However, the number of observations per condition and per scalp location was limited by their use of a free-standing 7-channel magnetometer. Helenius and colleagues (1998) were the first to measure the N400-sentence response using a modern, 122-channel helmet magnetometer. They presented sentences

25 visually word-by-word where the sentence-final word was chosen according to four conditions: most probable (congruous) completion, congruous but improbable completion, incongruous completion, and incongruous completion which shared the first syllable with the most probable completion (e.g. The gambler had a streak of bad luggage). Participants had no explicit task except to read the sentences for meaning. In most of their 10 subjects they found left-hemisphere channels which showed a strong difference in field strength between the congruous and the incongruous conditions, as well as a smaller and later effect in the right hemisphere. As determined by ECD modeling over ~12-20 strong channels, fit to individual MRIs (obtained for a subset of the subjects), 8 of the 10 subjects showed a left temporal source (4 in STG and 3 in MTG), which were fairly distributed from anterior to posterior temporal cortex. Left frontal sources in the left hemisphere were also identified in 2 subjects, and several subjects showed additional sources that were more temporoparietal. 5 of the subjects also showed a posterior STG source in the right hemisphere, but for the 4/5 who showed the left hemisphere source as well, the right hemisphere peak was delayed by ~25 ms. Corroborating this data, Helenius and colleagues later found a similar, left mid-stg source in a study using the same manipulation for 7 of 8 adult dyslexics who showed an N400 ERP effect (Helenius, Salmelin, Service & Connolly, 1999; see also Yeung, Hashimoto, Phillips & Sakai, 2004 for similar results). The source strength for the left superior temporal sources in seven normal subjects showed a pattern in which the congruent, probable ending was first to diverge from the other three, while the rare congruent condition tracked the incongruent conditions until its peak around 350 ms. The two incongruent conditions peaked around ~420 ms, but continued to show significant strength until around ~600 ms. In general, very little response from the chosen ECD was visible for the probable, congruent endings, consistent with the ERP work showing almost no N400 peak for very strongly predicted, word-final endings. Interestingly, Helenius et al. were also able to show a significant positive correlation between reaction time on a separate lexical decision task and the latency of the N400 peak, across subjects. Two MEG studies have examined the source(s) of the N400-sentence response using a distributed source model based on the cortical surface instead of or in addition to the simpler ECD model using a spherical volume conductor to model the head (Halgren et al., 2002; Maess et al., 2006). Although these distributed models are still fairly new and must be interpreted with care, the data they provide is important because they make it easier to model the activity of multiple sources over time. As discussed elsewhere in this paper, there is some reason to think that the N400-sentence response may have several spatial generators and that there may be more than one temporal phase to the response, and if so, it would be hard to see this using a typical ECD model. Halgren et al. s (2002)

26 paper is particularly interesting because, beyond the classic comparison of congruent vs anomalous sentence endings, they take the opportunity to look at several other factors that have been known to affect the N400 in order to compare their spatiotemporal profile with the anomaly response, namely word frequency, sentence position, and word class (content vs function). They also carry out both the classic ECD analysis as well as a minimum-norms distributed source model, making it easier to measure their results against those of Helenius et al. (1998). In the ECD analysis, Halgren et al. (2002) modeled ECDs to the difference field composed of the incongruent congruent subtraction, at the timepoint where the response seemed to peak (440 ms +/- 37), using ~40 channels per subject. For all subjects this dipole localized near the left STS. As in Helenius et al. s (1998) study, these dipoles ranged from more anterior STG to more posterior STG across subjects. When these dipoles were then applied to the incongruent and congruent data separately for each individual, its strength was strongly modulated by anomaly, with the N400 response peaking at 432 +/- 29 ms in the incongruous condition. The same dipole showed a significantly stronger response for early content words than late content words (matched for frequency and length) and for content versus function words (matched for position) hardly any response was seen for function words. Although there was some tendency for lower frequency words to show a stronger response, this effect was weak and nonsignificant. For the distributed analysis, activity was modeled using a noise-normalized minimum-norm type solution anatomically constrained using subjects MRI images (Dale et al. 2000); solutions were calculated for every 5 ms of activity, for each individual. The distributed analysis over the incongruous-congruous contrast showed activation spanning left anterior temporal and posteroventral prefrontal cortices, beginning around ~250 ms and peaking around ~400 ms. The timecourse (on inflated cortex) averaged across subjects can be seen below. These images show differential activity beginning around ~250 ms post-word-onset near left planum temporale, then spreading to more inferior and more anterior temporal cortex, and subsequently spreading to left frontal cortex in what is described as a inferior-superior order: orbital LIFG (~BA 47/45) to Broca s (~BA 45/44) to middle and superior frontal gyri. They report right hemisphere activity only beginning around ~370 ms, with activity largely limited to orbital prefrontal cortex. Halgren et al. used a post-experiment memory task rather than an online task, mitigating suggestions that some of the event-related activity seen could be task-related. Using the same distributed source analysis, Halgren et al. show the sentence-position manipulation to show differential activation in many of the same areas (left frontotemporal), with somewhat less frontal activity (see below). Word-frequency