CS 2770: Computer Vision. Introduction. Prof. Adriana Kovashka University of Pittsburgh January 5, 2017

Transcription

1 CS 2770: Computer Vision Introduction Prof. Adriana Kovashka University of Pittsburgh January 5, 2017

2 About the Instructor Born 1985 in Sofia, Bulgaria Got BA in 2008 at Pomona College, CA (Computer Science & Media Studies) Got PhD in 2014 at University of Texas at Austin (Computer Vision)

3 Course Info Course website: Instructor: Adriana Kovashka Use "CS2770" at the beginning of your Subject Office: Sennott Square 5325 Office hours: Tue/Thu, 3:30pm - 5:30pm

4 TA Keren Ye Office: Sennott Square 5501 Office hours: TBD Do the Doodle by the end of Friday:

5 Textbooks Computer Vision: Algorithms and Applications by Richard Szeliski Visual Object Recognition by Kristen Grauman and Bastian Leibe More resources available on course webpage Your notes from class are your best study material, slides are not complete with notes

6 Course Goals To learn about the basic computer vision tasks and approaches To get experience with some computer vision techniques To learn/apply basic machine learning (a key component of modern computer vision) To think critically about vision approaches, and to see connections between works and potential for improvement

7 Policies and Schedule

8 Should I take this class? It will be a lot of work! But you will learn a lot Some parts will be hard and require that you pay close attention! But I will have periodic ungraded pop quizzes to see how you re doing I will also pick on students randomly to answer questions Use instructor s and TA s office hours!!!

9 Questions?

10 Plan for Today Introductions What is computer vision? Why do we care? What are the challenges? What is the current research like? Overview of topics (if time)

11 Introductions What is your name? What one thing outside of school are you passionate about? Do you have any prior experience with computer vision? What do you hope to get out of this class? Every time you speak, please remind me your name

12 Computer Vision

13 What is computer vision? Done? "We see with our brains, not with our eyes (Oliver Sacks and others) Kristen Grauman (adapted)

14 What is computer vision? Automatic understanding of images and video Computing properties of the 3D world from visual data (measurement) Algorithms and representations to allow a machine to recognize objects, people, scenes, and activities (perception and interpretation) Algorithms to mine, search, and interact with visual data (search and organization) Kristen Grauman

15 Vision for measurement Real-time stereo Structure from motion Multi-view stereo for community photo collections NASA Mars Rover Pollefeys et al. Goesele et al. Kristen Grauman Slide credit: L. Lazebnik

16 Vision for perception, interpretation The Wicked Twister ride Lake Erie sky water Ferris wheel amusement park Cedar Point tree ride 12 E Objects Activities Scenes Locations Text / writing Faces Gestures Motions Emotions ride tree people waiting in line people sitting on ride Kristen Grauman deck tree bench tree carousel umbrellas pedestrians maxair

17 Visual search, organization Query Image or video archives Relevant content Kristen Grauman

18 Related disciplines Graphics Image processing Artificial intelligence Computer vision Algorithms Machine learning Cognitive science Kristen Grauman

19 Vision and graphics Images Vision Model Graphics Inverse problems: analysis and synthesis. Kristen Grauman

20 Why vision? Images and video are everywhere! 144k hours uploaded to YouTube daily 4.5 mil photos uploaded to Flickr daily 10 bil images indexed by Google Personal photo albums Movies, news, sports Surveillance and security Adapted from Lana Lazebnik Medical and scientific images

21 Why vision? As image sources multiply, so do applications Relieve humans of boring, easy tasks Human-computer interaction Perception for robotics / autonomous agents Organize and give access to visual content Description of image content for the visually impaired Fun applications (e.g. transfer art styles to my photos) Adapted from Kristen Grauman

22 Faces and digital cameras Camera waits for everyone to smile to take a photo [Canon] Setting camera focus via face detection Kristen Grauman

23 Devi Parikh Face recognition

24 Linking to info with a mobile device Situated search Yeh et al., MIT kooaba MSR Lincoln Kristen Grauman

25 Exploring photo collections Snavely et al. Kristen Grauman

26 Special visual effects The Matrix What Dreams May Come Mocap for Pirates of the Carribean, Industrial Light and Magic Source: S. Seitz Kristen Grauman

27 Yong Jae Lee Interactive systems

28 Video-based interfaces YouTube Link Human joystick NewsBreaker Live Assistive technology systems Camera Mouse Boston College Kristen Grauman

29 Vision for medical & neuroimages fmri data Golland et al. Image guided surgery MIT AI Vision Group Kristen Grauman

30 Safety & security Navigation, driver safety Monitoring pool (Poseidon) Kristen Grauman Pedestrian detection MERL, Viola et al. Surveillance

31 Healthy eating FarmBot.io YouTube Link Im2calories by Myers et al., ICCV 2015 figure source

32 Self-training for sports? Pirsiavash et al., Assessing the Quality of Actions, ECCV 2014

33 Image generation Reed et al., ICML 2016 Radford et al., ICLR 2016

34 YouTube link Seeing AI

35 Obstacles? Kristen Grauman Read more about the history: Szeliski Sec. 1.2

36 What the computer gets Why is this problematic? Adapted from Kristen Grauman and Lana Lazebnik

37 Why is vision difficult? Ill-posed problem: real world much more complex than what we can measure in images 3D 2D Impossible to literally invert image formation process with limited information Need information outside of this particular image to generalize what image portrays (e.g. to resolve occlusion) Adapted from Kristen Grauman

38 Challenges: many nuisance parameters Illumination Object pose Clutter Occlusions Intra-class appearance Viewpoint Think again about the pixels Kristen Grauman

39 Challenges: intra-class variation CMOA Pittsburgh slide credit: Fei-Fei, Fergus & Torralba

40 Challenges: importance of context slide credit: Fei-Fei, Fergus & Torralba

41 Challenges: Complexity Thousands to millions of pixels in an image 3,000-30,000 human recognizable object categories 30+ degrees of freedom in the pose of articulated objects (humans) Billions of images indexed by Google Image Search billion smart camera phones sold in 2015 About half of the cerebral cortex in primates is devoted to processing visual information [Felleman and van Essen 1991] Kristen Grauman

42 Challenges: Limited supervision Less More Kristen Grauman

43 Challenges: Vision requires reasoning Antol et al., VQA: Visual Question Answering, ICCV 2015

44 Ok, clearly the vision problem is deep and challenging time to give up? Active research area with exciting progress! How datasets changed: Kristen Grauman

45 Datasets today ImageNet: 22k categories, 14mil images Microsoft COCO: 80 categories, 300k images PASCAL: 20 categories, 12k images SUN: 5k categories, 130k images

46 Some Visual Recognition Problems

47 Recognition: What is this?

48 Recognition: What objects do you see? building street balcony truck carriage horse table person person car

49 Detection: Where are the cars?

50 Activity: What is this person doing?

51 Scene: Is this an indoor scene?

52 Instance: Which city? Which building?

53 Visual question answering: What are all these people participating in?

54 The Latest at CVPR 2016 * CVPR = IEEE Conference on Computer Vision and Pattern Recognition

55 Our ability to detect objects has gone from 34 map in 2008 to 73 map at 7 FPS (frames per second) or 63 map at 45 FPS in 2016

56 Redmon et al., CVPR 2016 You Only Look Once: Unified, Real-Time Object Detection

57 Force from Motion: Decoding Physical Sensation from a First Person Video Park et al., CVPR 2016

58 MovieQA: Understanding Stories in Movies through Question-Answering Tapaswi et al., CVPR 2016

59 Owens et al., CVPR 2016 Visually Indicated Sounds

60 Anticipating Visual Representations from Unlabeled Video Vondrick et al., CVPR 2016

61 Gatys et al., CVPR 2016 Image Style Transfer Using Convolutional Neural Networks

62 DeepArt.io try it for yourself! (Image Style Transfer Using Convolutional Neural Networks) Images: Styles:

63 DeepArt.io try it for yourself! (Image Style Transfer Using Convolutional Neural Networks) Results:

64 Thomas and Kovashka, CVPR 2016 Seeing Behind the Camera: Identifying the Authorship of a Photograph

65 Is computer vision solved? Given an image, we can guess with 81% accuracy what object categories are shown (ResNet) but we only answer why questions about images with 14% accuracy!

66 Why does it seem that it s solved? Deep learning makes excellent use of massive data (labeled for the task of interest?) But it s hard to understand how it does so It doesn t work well when massive data is not available and your task is different than tasks for which data is available Sometimes the manner in which deep methods work is not intellectually appealing, but our smarter / more complex methods perform worse

67 Overview of Topics

68 Overview of topics Lower-level vision Analyzing textures, edges and gradients in images, without concern for the semantics (e.g. objects) of the image Higher-level vision Making predictions about the semantics or higherlevel functions of content in images (e.g. objects, attributes, styles, motion, etc.) Involves machine learning; we ll cover some basics of this then go back to low-level tasks

69 Features and filters Transforming and describing images; textures, colors, edges Kristen Grauman

70 Features and filters Detecting distinctive + repeatable features Describing images with local statistics

71 Indexing and search Matching features and regions across images Kristen Grauman

72 How does light in 3d world project to form 2d images? Image formation Kristen Grauman

73 Multiple views Multi-view geometry, matching, invariant features, stereo vision Lowe Hartley and Zisserman Fei-Fei Li Kristen Grauman

74 Grouping and fitting Clustering, segmentation, fitting; what parts belong together? Kristen Grauman [fig from Shi et al]

75 Visual recognition Recognizing objects and categories, learning techniques Kristen Grauman

76 Object detection Detecting novel instances of objects Classifying regions as one of several categories

77 Attribute-based description Describing the high-level properties of objects Allows recognition of unseen objects

78 Convolutional neural networks State-of-the-art on many recognition tasks Image Prediction Krizhevsky et al. Yosinski et al., ICML DL workshop 2015

79 Recurrent neural networks Sequence processing, e.g. question answering Wu et al., CVPR 2016

80 Motion and tracking Tracking objects, video analysis Tomas Izo Kristen Grauman

81 Pose and actions Automatically annotating human pose (joints) Recognizing actions in first-person video

82 Your Homework Fill out Doodle Read entire course website Do first reading

83 Next Time Linear algebra review Matlab tutorial