Social Semiotics Introduction Historical overview

Transcription

1 This is a pre-print of Bezemer, J. & C. Jewitt (2009). Social Semiotics. In: Handbook of Pragmatics: 2009 Installment. Jan-Ola Östman, Jef Verschueren and Eline Versluys (eds). Amsterdam: John Benjamins Social Semiotics Introduction Social semiotics is concerned with meaning makers and meaning making. It studies the media of dissemination and the modes of communication that people use and develop to represent their understanding of the world and to shape power relations with others. It draws on qualitative, fine-grained analysis of records of meaning making, such as artifacts, texts, and transcripts, to examine the production and dissemination of discourse across the variety of social and cultural contexts within which meaning is made. Different versions of social semiotics have emerged since the publication of Michael Halliday s Language as Social Semiotic in The account we offer in this paper is focused on the version proposed by Gunther Kress, Robert Hodge, Theo van Leeuwen, and others. Following a historical overview we discuss its connections with Pragmatics and other approaches; key concepts; analytical focus; and fields of application. Historical overview In Language as Social Semiotic (1978) Michael Halliday proposes that the semiotic resources of language are shaped by how people use them to make meaning-the social functions they are put to. He holds that every sign serves three functions simultaneously: they express something about the world ( ideational metafunction ), position people in relation to each other (interpersonal metafunction) and form connections with other signs to produce coherent text ( textual metafunction ). His ideas were taken up by Robert Hodge and Gunther Kress, who developed a critical account of language and society derived from Marx in Language as Ideology (Kress & Hodge, 1979). In Social Semiotics (1988) they adopted a similar stance to explore any set of semiotic resources that people use in everyday life, the resources of language as much as the resources of image, and of other modes. Hodge & Kress point to a number of distinctive features of a social account of semiotics. First, they problematize essentialist notions of meaning: Traditional semiotics likes to assume that the relevant meanings are frozen and fixed in the text itself, to be extracted and decoded by the analyst by reference to a coding system that is impersonal and neutral, and universal for users of the code. Social semiotics cannot assume that texts produce exactly the meanings and effects that their authors hope for: it is precisely the struggles and their uncertain outcomes that must be studied at the level of social action, and their effects in the production of meaning. (Hodge & Kress, 1988:12) Like Pragmatics, Social Semiotics argued against some of the working hypotheses of traditional linguistics and semiotics, and in favour of a situated perspective on communication. Social semiotics aims to account for context, not based on a naive text-context dichotomy, but rather based on the assumption that context has to be theorized and understood as another set of texts. (Hodge & Kress, 1988:8) Crucially, that involves breaking with the assumption that verbal language is always dominant and autonomous. As Hodge (2009) puts it: Social meanings cannot be tracked only 1

2 in one code, even in verbal language as the dominant one. The supposed dominance and autonomy of the verbal code is indeed an ideological assumption whose takenfor-granted truth needs to be questioned by social semiotics. Hodge & Kress discuss examples from a range of social contexts involving a range of different modes, but writing and image in print media (e.g., magazines, billboards) are discussed more than speech, gesture, gaze and other modes operating in the social encounters with which Pragmatics is concerned. In the late 1980 s, when Gunther Kress and Theo van Leeuwen began to develop a social semiotic approach to the visual, the focus was still on print media, culminating in their book Reading Images. The Grammar of Visual Design (1996). In Reading Images they propose a framework for the analysis of image, which draws on the broad semiotic aspects of Halliday s social semiotic theory and made use of the system networks as a heuristic framework for theorizing meaning as choice. At the same time, Michael O Toole applied Halliday s systemic functional grammar and the tools it offered to examine the visual in his book The Language of Displayed Art (1994). Both accounts of image as a set of meaning making resources were also informed by the insights of film studies, iconography and art history. The issue of mode and multiple modes became, perhaps inevitably, foregrounded in the 1990s. With the focus now on the co-operation of modes rather than the study of modes in isolation the term multimodality became a key term. It provided an alternative to terms such as verbal and non-verbal, which position the verbal as the unmarked, dominant resource for making meaning, and the non-verbal as the marked, auxiliary resource. In Multimodal Discourse. The Modes and Media of Communication (2001), Kress & van Leeuwen describe the multimodal perspective as follows. We aim to explore the common principles behind multimodal communication. We move away from the idea that the different modes in multimodal texts have strictly bounded and framed specialist tasks [...]. Instead we move towards a view of multimodality in which common semiotic principles operate in and across different modes. (Kress & van Leeuwen 2001:2). The framing of the book as being about discourse rather than semiotics marks the connection between social semiotics and discourse (in a Foucauldian sense as a social category). Until then, (Critical) Discourse Analysis, while sharing an interest in reconstructing ideologies (cf. Fairclough, 2003), had not attended to modes other than speech and writing, or to their material, cultural and social affordances; that was the realm of social semiotics (see Analytical Focus). As discourse and critical became more speakable (Hodge 2009) publishers preferred book titles featuring those terms and they became important carriers of social semiotics. Connections with Discourse Analysis and other approaches are also emphasized in the Routledge quarterly Social Semiotics (founded in 1990), which presents itself as a journal for discourse and critique looking for high quality, politically engaged papers that use textual analysis, discourse analysis, political economy, ethnography or combinations of these and/or other methods, to say something concrete about the nature of life in our societies. Connections with other approaches Like Pragmatics, Social Semiotics is concerned with meaning in context. Whether they are seen as disciplines or perspectives they both adopt a functional perspective on meaning making. They are both reactions to traditions which largely 2

3 ignored the social and cultural situatedness and power implications of meaning making. Some other approaches have played a major role in the development of both Pragmatics and Social Semiotics, such as Sociolinguistics, Linguistic Anthropology, Hallidayan Linguistics and Critical Discourse Analysis. Other disciplines have impacted on one more than the other. Social semiotics for instance also draws on Film Theory and Iconography; Pragmatics also draws on Pyscholinguistics. A key difference between the two disciplines is their approach to multimodality. To varying degrees, Pragmatics has been and is increasingly concerned with modes other than speech, such as gesture or gaze, especially in work drawing on conversation analysis, interactional sociology, interactional sociolinguistics, linguistic anthropology, micro-ethnography and linguistic ethnography. Some of this work is based on the working hypothesis that speech or writing is always dominant, carrying the essence of meanings, and that other, simultaneously operating modes can merely expand, exemplify or modify these meanings. This is reflected by fine grained, moment-to-moment analysis of, e.g., lexis, intonation, rhythm and tone, hesitations and restarts, alongside more occasional discussion of, for instance, hand movements or shifts in direction of gaze in talk. The methodological privileging of particular linguistic resources is also reflected in notions like non-verbal, paralinguistic or context. Gumperz (1999), for instance, defines a contextualisation cue as any verbal sign which when processed in cooccurrence with symbolic grammatical and lexical signs serves to construct the contextual ground for situated interpretation (p. 461), thus treating lexis and grammar as text, other verbal signs, such as intonation, rhythm and tone as paralinguistic or context, and any other non-verbal sign is either treated as context or placed beyond the scope of the analysis. Other studies foreground the significance of particular modes (e.g. pointing in linguistic anthropology), and transcribe them in conjunction with speech or writing. Yet other studies attend to a wide range of different modes and their mutually modifying effect, emphasizing their different potentials and constraints and essentially moving towards a semiotic perspective on representation and communication. The multimodal scope of social semiotics means that it can attend to forms of meaning making which remain largely unattended in Pragmatics, notably in contexts where people are not physically co-present and where speech is not involved, such as a child drawing. Where the empirical domains of Pragmatics and Social Semiotics overlap, the two provide distinctive analyses, using concepts derived from their different perspective on meaning making and multimodality. For Social Semiotics, these concepts include the sign, semiotic resource, mode, affordance and orchestration. These concepts are discussed below. Key Concepts This section gives an overview of key concepts in social semiotics. Using illustrative examples where appropriate, we discuss the notion of the sign, semiotic resource, mode, affordance and multimodal orchestration. Sign The notion of the sign is borrowed from traditional semiotics. Signs are elements in which the signified ( meaning ) and signifier ( form ) have been brought together. Social semiotics holds that the process of sign-making is subject to the interest of sign-makers, their availability of semiotic resources and the aptness of those 3

4 resources to the meanings which they wish to realize. That is to say, the relation between form and meaning is not arbitrary but motivated (Kress, 1993). In (mediated) communication, signs are made and remade, and from a social semiotic perspective signs are always newly made in a specific environment and according to the interests of the sign makers interests. In other words, Signs are made not used by a sign-maker who brings meaning into an apt conjunction with a form, a selection/choice shaped by the signmaker s interest, interests that are shaped by the environment and circumstances of use (Kress, forthcoming). Semiotic resource As social semiotics evolved there was a growing emphasis on how the context of communication and the sign maker shaped signs and meaning. It moved towards a more flexible notion of grammar, with a focus on people s situated choice of resources rather than emphasizing the system of available resources. Semiotic resource is central to multimodality although it features slightly differently within particular approaches. Kress and van Leeuwen suggest that a semiotic resource can be thought of as the connection between representational resources and what people do with them. Van Leeuwen describes semiotic resource as follows: Semiotic resources are the actions, materials and artifacts we use for communicative purposes, whether produced physiologically for example, with our vocal apparatus, the muscles we use to make facial expressions and gestures or technologically for example, with pen and ink, or computer hardware and software together with the ways in which these resources can be organized. Semiotic resources have a meaning potential, based on their past uses, and a set of affordances based on their possible uses, and these will be actualized in concrete social contexts where their use is subject to some form of semiotic regime (van Leeuwen, 2005:285). The emphasis on rules within social semiotics is on rules as socially made and changeable through social interaction (van Leeuwen, 2005). Van Leeuwen also notes that the notion of resource has begun to replace the notion of sign. The concept of semiotic resource offers a different starting point for thinking about semiotic systems and the role of the sign maker in the process of making meaning. In this perspective signs are a product of a social process of sign making. A person (sign maker) chooses a semiotic resource from an available system of resources. They bring together a semiotic resource (a signifier) with the meaning (the signified) that they want to express. In other words people express meanings through their selection from the semiotic resources that are available to them in a particular moment: meaning is choice from a system. But this choice is always socially located and regulated, both with respect to what resources are made available to whom, and the discourses that regulate and shape how modes are used by people. There are various kinds of normative discourses for how we use semiotic resources sometimes more, sometimes less binding, and of different kinds, but nonetheless they do provide rules for their use. Discourses of gender, social class, race, generation, institutional norms and other articulations of power shape and regulate people s use of semiotic resources. These are not codes in the sense that they cannot be changed and that they are simply there but they are social rules (van Leeuwen, 2005). Mode 4

5 A mode is often defined as a set of socially and culturally shaped resources for making meaning. A number of different criteria can be considered to establish whether a set of resources counts as a mode or not. The most common test is, following Halliday, the meta-function test. That is, if it can be shown that the set can serve all three meta-functions, it is regarded a mode. For instance, the resources of colour can be used to represent what the world is like (cf. the gardener s green dress), to establish social relations (cf. the doctor s white dress) and to create coherence (cf. the background colour against which different images are set in a magazine). However this test only works if a particular community of users of the mode have been identified. Typeface, for instance, may not qualify as a mode among archeologists; that is, they may not have access to all the resources of font and therefore be unable to use font to serve all three metafunctions. Typesetters, on the other hand, or their contemporary counterparts, graphic designers, are likely to have a sense of the potentials of type. Indeed, their professional identity rests on that sense. Becoming a member of a community, social group, or school means becoming an apt user of the modes used within that community. The identification of a community of users suggests that in order for a form of meaning to be treated as a mode there needs to be a shared understanding of those forms of meaning making. That leads to another test, the community test: the words, or gestures, produced by an individual may be meaningful to the individual, but they are not necessarily shared by a particular, socially and culturally situated group. Inasmuch as they are, they can be treated as belonging to a particular mode ; inasmuch as they are not, they can be treated as idiosyncratic yet no less meaningful and therefore of no less interest to the social semiotician- forms of meaning making. In other words, in order for a set of resources to count as a mode its resources must have come to display regularities. These regularities are the outcome of the historical, social and cultural use of material forms (sounds, parts of the body, canvas, et cetera). Material, social and cultural affordance The term affordances is contested and continuously debated. It has particular emphasis and currency in social semiotic approaches to multimodality (see Jewitt, 2009). Modal affordance is used to refer to what it is possible to express and represent easily with a mode (Kress 1993). He positions affordance as a concept connected to both the material and the cultural, and social historical use of a mode. In other words, the affordance of a mode is shaped by what it has been repeatedly used to mean and do its provenance ), and the social conventions that inform its use in context. Each mode (as it is realized in a particular social context) possesses a specific logic and provides different communicational and representational potentials. The logic of sequence in time is unavoidable for speech: one sound is uttered after another, one word after another, one syntactic and textual element after another. This sequence becomes an affordance or meaning potential: it produces the possibilities for putting things first or last, or somewhere else in a sequence. The mode of speech is therefore strongly governed by the logic of time. In contrast, (still) images are more strongly governed by the logic of space and simultaneity. Like all governing principles they do not hold in all contexts and are realized through the complex interaction of the social as material and vice versa in this sense the material constitutes the social and vice versa. Multimodal orchestration 5

6 The meaning of any message is distributed across different modes and not necessarily evenly. The different aspects of meaning are carried in different ways by each of the modes in the ensemble. Any one mode in that ensemble is carrying a part of the message only: each mode is therefore partial in relation to the whole of the meaning and speech and writing are no exception. Multimodal research attends to the interplay between modes to look at the specific work of each mode and how each mode interacts with and contributes to the others in the multimodal ensemble. At times the meaning realized by two modes can be aligned, at other times they may be complementary and at other times each mode may be used to refer to distinct aspects of meaning and be contradictory, or in tension. As Lemke has stated (2002: 303): No [written] text is an image. No image or visual representation means in all and only the same ways that some text can mean. It is this essential incommensurability that enables genuine new meanings to be made from the combinations of modalities. The relationships between modes as they are orchestrated in interactions (and texts) may realize tensions between the aspects of meaning in a text. This kind of tension can itself be meaningful and a means for encouraging reflection and critique. The structure of a text and hyperlinks realize connections and disconnections between screens. These contribute to the expansion of meaning relations between elements. The question of what to attend to, what to make meaningful is a significant aspect of the work of making meaning. In other words, the task of what to attend to and to select as salient to the task at hand is amplified by a multimodal focus. Analytical Focus Social semiotics aims to understand how people use modes available to them in particular social situations. That involves the collection of records of meaning making -texts, artifacts, video recordings- which capture the semiotic work involved in those situations. The analysis proceeds by attending to the following aspects. Meaning makers and modes The meaning makers and the modes and the resources of those modes that they have used need to be identified. In print media, they are likely to include resources of image (e.g., pictorial detail, depth), writing (e.g., lexis, clause structure), typography (e.g. type, letter fit), and layout (e.g., spread, grid). In embodied interaction, they are likely to include speech (e.g. intonation, tone), gesture (e.g. direction, span), gaze (direction, fixation), and posture. These resources are not necessarily equally available to all sign makers involved. Speech, for instance is often available to only one sign maker at the time; but gaze is not. In other contexts the use of modes is distributed along professional lines: the sign makers involved in the production of print media, for instance, have specialized in writing ( authors ), image ( illustrators, picture editors ) and layout ( graphic designers ). A growing body of literature is available detailing the use of these resources in a variety of social contexts. Crucially, social semiotics aims to understand how sign makers exploit the potentials of these resources to articulate the meanings they wish to express. Chains of semiosis In representing the world, translations are constantly made of meanings made in one mode or ensemble of modes to meanings made in another mode or ensembles of modes. Such translations are inevitable because on the one hand social 6

7 environments are changed in recontextualization and on the other hand the available modes and media and their affordances are constrained. Transduction inevitably brings profound changes in the move from one mode to the other. Given the difference in material and histories of social and cultural work, there can never be a perfect translation from one mode to another. Assume two characters were described in writing. The author might have given a written description like, Sitting in the late autumn sunshine, Sam and Bill share a bench in the park. An illustrator or designer might have been asked to draw across, to transduct the written description into the mode of image. Now the illustrator has to ask how close to each other were they sitting? was Bill to the left or to the right of Sam? The translator/transductor has to become precise, whether she or he wishes to do so or not. While transduction describes changes involving a change in mode, transformation describes changes in arrangement within one mode. Transformations are operations on structures within the one mode, in which entities remain the same while structures change. In a transformation, say within the mode of writing, words remain, syntactic/grammatical categories remain those of the mode, as do textual arrangements. What changes is their arrangement. In transduction, the change from one mode to another, brings with it a change of entities. There are no words in image, there are depictions; semiotic/semantic relations which in speech or writing are expressed in clauses and as verbs are realized through vectors or lines. Other semiotic relations between lexical-syntactic elements prepositions for instance (on, over, by, etc) are realized by spatial means in images; and so on. Transformation and transduction are ubiquitous semiotic processes. It is the task of the analyst to identify them and ask what was gained and what was lost as signs are remade to form chains of semiosis. Inter-semiotic relations and multimodal orchestration The relationships across and between modes in multimodal texts and interaction are a central area of interest for multimodal research. Where the modes of image, writing and layout are involved this modal co-operation is often captured by the notion of design and designing. Where the modes of speech, gesture and gaze are involved this is often captured by the notion of ensemble and orchestrating. The task of the analyst may be seen to be to reconstruct what the distinctive contributions are of the various modes to a modal configuration. Linguistic routes through social semiotics and multimodality, modal relations have been investigated by applying inventories from Hallidayan linguistics. That leads to modal relations being defined in terms of their iterative relation (such as synonymy, antonymy, meronymy), logico-semantics (e.g. one clause expanding another) or cohesion (e.g. reference). There are some difficulties attached to this approach. First, it assumes that the relations which can be drawn between signs made in one mode, namely speech, can also be found in other modes; in other words that the types of relations between signs is stable across modes. Second, it assumes that the levels at which the semiotic features of modes operate in the different modes are the same. Fields of application Despite the close association between social semiotics and multimodality strictly speaking these are two distinct and differently orientated terms - social semiotics is a theory of meaning, and multimodality is a field of application, or a perspective on resources for making meaning. Multimodality opened the door to forms of collaboration with a range of disciplines also concerned with looking at 7

8 communication and interaction beyond language, including pragmatics, sociology, anthropology and psychology. In the first decade of the 21 st century, this joint enterprise was focused on technology mediated interaction, pedagogy, literacy, design, and increasingly, fields such as literary studies, language teaching, and identity studies. The Routledge Handbook of Multimodal Analysis (Jewitt, 2009) provides a detailed account of the history of multimodality as a field of research. Multimodal research has been conducted on the technologization of practices and communication and interaction more generally. Much of this work explores and theorizes the nature of image and writing relations in narratives, relationships between book and computer based versions of texts, and the role of on-line communities of various kinds in the critique, as well as the interpretation and generation of new forms of multimodal and digital narratives and literacy practices. This work often describes new forms of literacy in an attempt to remap the territory of communication in a contemporary context and the kinds of practices that help move across it. The visual character of writing comes to the fore on screen, for instance, to function as objects of literacy in fundamentally different ways than it does on the page and this has been a topic of multimodal research. From early 2000, after the New London Group (1996) made the call to understand knowledge and pedagogy as multimodal, there has been an explosion of interest in multimodality within research and this perspective has been actively taken up by educational researchers leading to substantial work that looks at multimodal meaning making across a wide range of sites. Substantial multimodal research has been undertaken in pre-school and early years contexts, with a focus on multimodal meaning making practices. Science education has proven to be a productive site for multimodal investigations into the construction of knowledge across a range of resources, as has mathematics education, Music education and school English and Media education. Multimodal studies in Higher Education and in Professional settings have also examined pedagogic strategies and learning. Taken as a whole, these multimodal studies show that significant pedagogic work is realized through a range of modes. The need to rethink what it means to learn and to have capacities to engage fully in contemporary communication be is a thread that runs through much multimodal research. How identities are articulated through multimodal means is an area that has attracted some attention within multimodal research. Much of the work on what might broadly be called literacy practices from a multimodal perspective is concerned with the production of identities. References Fairclough, N. (2003). Analysing Discourse. Textual analysis for social research. London: Routledge. Gumperz, J.J. (1999), On interactional sociolinguistic method, in S. Saranghi and C. Roberts (eds), Talk, work and institutional order. Discourse in medical, mediation and management settings. Berlin, New York: Mouton de Gruyter, pp Halliday, M. (1978), Language as Social Semiotic. London: Edward Arnold. Hodge, R. and Kress, G. (1988), Social Semiotics. Cambridge: Polity Press. Hodge, R. (2009). Social Semiotics. In: P. Bouissac (ed.), Semiotics Encyclopedia Online. Retrieved 15 February Jewitt, C. (ed) (2009). The Routledge Handbook for Multimodal Analysis. London: Routledge. 8

9 Kress, G. (1993). Against Arbitrairiness: the social production of the sign as a foundational issue in critical discourse analysis. Discourse and Society 4, 2, Kress, G. & R. Hodge (1979), Language as Ideology. London: Routledge. Kress, G and van Leeuwen, T (1996), Reading Images: The Grammar Of Visual Design. London: Routledge. Kress, G. and van Leeuwen, T. (2001), Multimodal Discourse. The Modes and Media of Contemporary Communication. London: Edward Arnold. Kress, G. (forthcoming). Multimodality. London: Routledge. Lemke, J. (2002). Travels in Hypermodality. Visual Communication 1, 3, New London Group (1996). A pedagogy of multiliteracies. Designing social futures. Harvard Educational Review 66, O Toole, M. (1994). The Language of Displayed Art. London: Leicester University Press. van Leeuwen, T. (2005), Introducing Social Semiotics. London: Routledge. 9