Metonymy and Metaphor in Cross-media Semantic Interplay The COSMOROE Framework & Annotated Corpus Katerina Pastra Institute for Language & Speech Processing ATHENA Research Center Athens, Greece kpastra@ilsp.gr
COSMOROE Framework COSMOROE (CrOSs-Media interaction relations) A framework for modelling multimedia dialectics (semantic interplay between images, language and human actions) Defines a number of semantic relations between different modalities, for formulating multimedia messages Draws analogies to semantic phenomena in language analysis and actually extends these phenomena to multimedia discourse In the process, it takes into consideration the idiosyncratic characteristics of the different media and modalities and does not make use of any contribution criterion (i.e. which modalities contributes more or less to the message) Intended to have: Descriptive Power: general enough to describe interaction between different media-pairs (e.g. image-language, gesture-language) Usability: suitable for computational modeling
COSMOROE Relations Equivalence: the information expressed by the different media is semantically equivalent (paradigmatic relations between modalities) Complementarity: the information expressed in one medium is (an essential or not) complement of the information expressed in another medium (syntagmaticrelations between modalities) Independence: each medium carries an independent message, the combination of which creates the multimedia message
Metonymy The two referents (source and target) come from the same domain,have same array of associations, there is no transfer of qualities from one to another the two modalities refer to different entities but the user intends the two modalities tobe considered semantically equivalent The city, of course, is Athens, and it is here that I will begin my exploration of modern Greece.
Metonymic Patterns - action for purpose (stand visit) - artifact for art (architecture building) - artifact for manufacturer (jewelry jewel maker) - aspect for concept (poverty houses) - cause for action (gratitude kiss) - cause for effect (slippery fall) - characteristic action for manufacturer (cutting butcher) - container for content (bottle milk) - defining property for thing defined by the property (round ball) - entity for purpose/use (camel transportation) - institution for people involved (education student) - material for thing made of the material (gold golden ring) - object/instrument for user (knife butcher) - object/instrument/substance for action employed for (knife cutting) part for whole (wing airplane) - people for place (inhabitant island) - perception event for its trigger or biological event for its trigger state (listen music, snore sleep) - place for organisation/institution (church religion) - place for event (church wedding ceremony) - pre-state of thing for thing (bread mix bread) - producer for product (cow milk) - product sold for retailer (newspaper -kiosk) - pure (Acropolis Athens) - step of a process for process (open luggage unpack) - target audience for performer (children story teller)
Metaphor The two modalities (source and target) refer to different entities of different domains; the user intends the two modalities to be considered semantically equivalent there is a transfer of qualities 1 2 3 4 It s very serene
A more challenging example «απολαμβάνουμε μελωδίες από επαγγελματίεςμουσικούς...» we enjoy melodies by professional musicians Textual metonymy: logical metonymy / perception event for its trigger listening Multimedia Metonymy: Perception event for its trigger Body Movement: Play music
COSMOROE resources 10 hour corpus of TV travel programmes: 5 hours in EL, 5 hours in EN Annotated with COSMOROE relations, in a multifaceted way (transcribed speech, shot segmentation, object and movement detection and identification, gesture identification, labelling of all visuals, OCR and subtitle transcription, acoustic events transcription, resolution of textual semantic phenomena (e.g. metonymies, metaphors, antonomasia), resolution of visual semantic phenomena (e.g. metonymies), multi-step inferences. Preliminary inter-annotator agreement: 0.88 Kappa Validation ongoing Full inter-annotator agreement study on relations pending, while inter-annotator agreement study on visual labelling in progress. Annotation support tools: automatic consistency checking, automatic extraction of conceptual triplets COSMOROE search engine: for novice and advanced users
Documentation Pastra K. (2008), "COSMOROE: A Cross-Media Relations Framework for Modelling Multimedia Dialectics", Multimedia Systems, vol. 14 (5), pp. 299-323, Springer Verlag Pastra K., BaltaEirini(2009), "A text-based search interface for Multimedia Dialectics", in proceedings of EACL 2009 (demo papers). Pastra K. (2009), COSMOROE Annotation Guide, internal technical report, v.3.0 Pastra K, BaltaEirini(2009), COSMOROE Search Engine Help File, internal technical report, v. 1.0
Acknowledgments The COSMOROE Search Engine has been developed in collaboration with: EiriniBalta(ebalta@ilsp.gr) The COSMOROE annotation support tools have been developed in collaboration with: Panagiotis Dimitrakis(p_dim@ilspl.gr) Special thanks to the COSMOROE Annotation Team: Dr ArgyroVatakis ElinaDesypri Maria Koutsompogera Maria Lada EviMpandavanou This research is being supported by the POETICON project (FP7-215843) We also acknowledge the Latsis Foundation Research Grant for applying this work to the Analysis s of Greek Caricatures More details available from August 2009 @ http://www.cosmoroe.eu