Large scale Visual Sentiment Ontology and Detectors Using Adjective Noun Pairs

Large scale Visual Sentiment Ontology and Detectors Using Adjective Noun Pairs Damian Borth 1,2, Rongrong Ji 1, Tao Chen 1, Thomas Breuel 2, Shih-Fu Chang 1 1 Columbia University, New York, USA 2 University of Kaiserslautern & DFKI, Germany

Sentiment Analysis

Sentiment Analysis Application Scenarios Intelligence Political Opinion Brand Monitoring Investment

Sentiment Analysis What is the sentiment of the following tweets?

Sentiment Analysis What is the sentiment of the following tweets? Golden Tweets 2012, Twitter Inc.

Sentiment Analysis What is the sentiment of the following tweets? analysis of visual content is needed Golden Tweets 2012, Twitter Inc.

Related Work Visual Learning of Semantic Concepts [TRECVID] [PASCAL VOC] [LSCOM] [ImageNet] Sentiment Analysis from text [Esuli06] [Thelwall10] Analysis of Affect / Emotion / Aesthetics from Low level Features [Machajdik10] [Jia12] [Yanulevskaya08] [[Yanulevskaya12] [Datta06] [Joshi11] Multimedia Grand Challenge (HP Challenge) [Li12] [Lux12] [Wang12] Public Datasets International Affective Picture System (IAPS) The Geneva Affective Picture Database (GAPED)

Visual Sentiment Analysis Idea predict sentiment by understanding visual content instead of just utilizing low level features a reasonable detection performance

Visual Sentiment Analysis Idea predict sentiment by understanding visual content instead of just utilizing low level features introduce Visual Sentiment Ontology & SentiBank 1. concepts are linked to emotion 2. reflect strong sentiment 3. are frequent on Flickr & Youtube 4. reasonable detection performance

VSO & SentiBank Construction Which sentimental concepts? - data mining to discover visual sentiments in social media predict sentiment by understanding visual content instead of just utilizing color or brightness a reasonable detection performance

Seed Vocabulary: Plutchik s Wheel of Emotion

Data driven Discovery 24 emotions 166,342 videos 3,079,526 tags 38,935 individual tags [elements in dictionary] 24 emotions 150,034 images 3,138,795 tags 17,298 individual tags [elements in dictionary] Adjective (260): needed for expressing emotions frequent positive Adj: beautiful, amazing, cute frequent negative Adj: sad, angry, dark Nouns (520): feasible for computer vision people, places, animals, food, objects, weather

Introduce Adjective Noun Pairs (ANP) beautiful clouds dark clouds

Introduce Adjective Noun Pairs (ANP) beautiful clouds dark clouds cute dog dangerous dog Bi-Concepts [Li12] / Concept Pairs [Over12]

ANP Selection Steps: remove named entities like hot dog via Wikipedia Choose strong sentiment ANP concepts by tools Senti WordNet, SentiStrength and popularity on Flickr retrieved a dataset of 500,000 images for ~3000 ANP

VSO & SentiBank Construction Which ANPs lead to robust detectors? - what is the set of SentiBank detectors predict sentiment by understanding visual content instead of just utilizing color or brightness a reasonable detection performance

SentiBank Detector Training LibSVM 5 fold cross validation Global Features (TODO group) RGB Color Histogram (3x256 dim.) GIST descriptor (512 dim.) Local Binary Pattern (59 dim.) Local Features SIFT Bag of Words (1,000 codewords, 2 layer spatial pyramid, max pooling) Special Features Attribute (2,000 dim.) Object Detection (people, face, car, ) Aesthetics (color harmony, white balance, )

SentiBank Detector Validation Performance vs. Feature Attribute is the strongest feature Aesthetic features / object detection help Improve accuracy between 9% 30%

SentiBank Detector Validation Detector AP@20 vs. Frequency Large number of detectors with reasonable performance Performance not correlated with frequency depends on content variation, abstractness, SentiBank Detectors 1,200 detectors with F Score > 0.6

SentiBank Detector Validation Detection Results detectability of ANP depends on adjective combination

here comes Tao s short video

VSO & SentiBank Construction How does it perform? - experiments on photo tweets predict sentiment by understanding visual content instead of just utilizing color or brightness a reasonable detection performance

Photo Tweet Sentiment Dataset Multi-user sentiment AMT labeling over 2000 photo tweets different topics: #abortion, #championsleague, #police, 31

Photo Tweet Sentiment Dataset Multi-user sentiment AMT labeling over 2000 photo tweets different topics: #abortion, #championsleague, #police, Amazon Mechanic Turk Sentiment/Emotion Label: (image-based labeling) worker 1: Positive, trust:acceptance worker 2: Neutral, interest:unlabeled,sad:pensiveness worker 3: Positive, interest:interest @nickespo89 (text-based labeling) worker 1: Positive, joy:serenity,trust:acceptance worker 2: Positive, anger:neutral,interest:interest,joy:trust:acceptan worker 3: Negative, sad:sadness True stuff. I have mad respect for all the ladies that DO NOT give in to abortion. 33

Photo Tweet Sentiment Dataset Multi-user sentiment AMT labeling over 2000 photo tweets different topics: #abortion, #championsleague, #police, Amazon Mechanic Turk Sentiment/Emotion Label: (image-based labeling) worker 1: Positive, trust:acceptance worker 2: Neutral, interest:unlabeled,sad:pensiveness worker 3: Positive, interest:interest @nickespo89 True stuff. I have mad respect for all the ladies that DO NOT give in to abortion. (text-based labeling) worker 1: Positive, joy:serenity,trust:acceptance worker 2: Positive, anger:neutral,interest:interest,joy:trust:acceptan worker 3: Negative, sad:sadness (text-image-based labeling) worker 1: Positive, joy:serenity,sad:neutral worker 2: Positive, interest:interest,joy:joy,sad:surprise:distraction worker 3: Positive, joy:serenity,surprise:neutral,trust:trust 34

Photo Tweet Sentiment Dataset AMT Results over all topics Text more controversial than image in invoking responses Response inconsistency varies across topics

Photo Tweet Sentiment Prediction True stuff. I have mad respect for all the ladies that DO NOT give in to abortion. #groundzero #hurricanesandy #newjersey

Photo Tweet Sentiment Prediction Photo Tweets (603 tweets) True stuff. I have mad respect for all the ladies that DO NOT give in to abortion. #groundzero #hurricanesandy #newjersey

Photo Tweet Sentiment Prediction Photo Tweets (603 tweets) True stuff. I have mad respect for all the ladies that DO NOT give in to abortion. Accuracy Linear SVM Logistic Reg. Low level 0.55 0.57 Features SentiBank 0.67 0.70 #groundzero #hurricanesandy #newjersey

Conclusion Effort to build Visual Sentiment Ontology Psychology and Web folksonomy Unique adjective noun pair concepts Results in large scale detectors Ontology 3000 concepts, SentiBank 1200 detectors Datasets (0.5 million images) and tools available Applications Multi modal sentiment monitoring Intuitive visualization tools

References [Machajdik10] J. Machajdik and A. Hanbury, Affective Image Classification using Features inspired by Psychology and Art Theory, ACM MM, 2010 [Jia12] J. Jia, S. Wu, X. Wang, P. Hu, L. Cai, and J. Tang. Can we understand van Gogh s Mood?: Learning to infer Affects from Images in Social Networks, ACM MM, 2012. [Yanulevskaya12] V. Yanulevskaya, J. Uijlings, E. Bruni, A. Sartori, E. Zamboni, F. Bacci, D. Melcher, and N. Sebe, In the Eye of the Beholder: Employing Statistical Analysis and Eye Tracking for analyzing Abstract Paintings, ACM MM 2012 [Yanulevskaya08] V. Yanulevskaya, J. van Gemert, K. Roth, A. Herbold, N. Sebe, and J.M. Geusebroek, Emotional Valence Categorization using Holistic Image Features, ICIP, 2008 [Datta06] R. Datta, D. Joshi, J. Li, and J.Z. Wang, Studying Aesthetics in Photographic Images using a Computational Approach, ECCV 2006 [Joshi11] D. Joshi, R. Datta, E. Fedorovskaya, Q-T Luong, J.Z. Wang, J. Li, and J. Luo, Aesthetics and Emotions in Images, Int Journal on Signal Processing 28(5), 2011 [Li12] B. Li et al., Scaring or Pleasing: Exploit Emotional Impact of an Image, ACM MM, 2012 [Lux12] M. Lux et al. Classification of Photos Based on Good Feelings, ACM MM, 2012 [Wang12] X. Wang et al. Understanding the Emotional Impact of Images, ACM MM, 2012 [Esuli06] A. Esuli and F. Sebastiani. SentiWordnet: A publicly available Lexical Resource for Opinion Mining. LREC, 2006. [Thelwall10] M. Thelwall et al. Sentiment Strength Detection in Short Informal Text. J. of the American Soc. for Information Science and Tech., 61(12):2544-2558, 2010.

References [Li12] X. Li, C. Snoek, M. Worring, and A. Smeulders. Harvesting Social Images for Bi-Concept Search. IEEE Transactions on Multimedia, 14(4):1091-1104, 2012. [Over12] Over et al. Trecvid 2012 An Overview of the Goals, Tasks, Data, Evaluation Mechanisms and Metrics. TRECVID Workshop, 2012.

Questions? Thanks for your attention http://visual sentiment ontology.appspot.com we invite you to our demo on Friday, 11:15

Sentiment Words

Visual Sentiment Ontology

SentiBank Detector Validation Detection Results

Photo Tweet Sentiment Prediction