CS415 Human Computer Interaction Lecture 2 Human Side of HCI Part-1 September 4, 2018 Sam Siewert
Things to Remember Assignment #1, Due on Friday CLI Cmd / Rsp Interaction, Command Sequencing, Scripting Chapter 1 in HCI The Human Visual, Auditory, Haptic, Motion Capture (Big 4 for Interaction) Short and Long Term Memory (CLI vs. WIMP) Reasoning, Problem Solving, Skills, Errors Chapter 2 in HCI The Computer Text Input, Pointers, 3D Output Displays touch and stylus VR and Interaction Devices Paper I/O, Memory / Storage, Capacity, Access Processing, Network Lag (client / server delay interactive games), Latency (physical stimulation / response delay, bound by speed of light) Sam Siewert 2
Consider What causes humans to miss what should be obvious in visual scenes? [page 21 of text] What is the value of color vision to humans at 1/20 th resolution of grayscale? (6 million cones, 120 million rods) Sam Siewert https://dog-vision.com/ 3
Importance of Color? Often not so important, but sometimes critical [in NAZ] Red on black, you re OK Jack Red on yellow, kill a fellow [1] Seeing Forbidden Colors, Scientific American, pp. 72-77, Feb 2010. [2] How the Color Red Influences Our Behavior, Scientific American MIND, Nov 2014. Sam Siewert 4
Human Senses and Factors Human Factors in Aviation (Starting with Senses) Classic 5 Senses 1. Visual (Color Tri-stimulus, 2D, 3D), 2. Audio (Tones, Annunciators, Multi-channel Stereo and Surround Sound Speech Recognition), 3. Tactile (Haptics, Ergonomics), 4. Taste (Ignored), 5. Smell (Ignored) Important, but Less Commonly Known: 1. Proprioception Eye to Hand Coordination (VR & AR) 2. Vestibular Balance and Spatial Orientation (VR & AR) 3. Chronoception Time (Interactive and Real-Time Systems) 4. Thermoception (Feeling heat - different from Thermal Imaging) 5. Nociception Pain 6. Interoception Health and Status of Body Organs Sam Siewert 5
Human Aspects of HCI James Cutting & Peter Vishton Perceiving Layout Cortex=10 Billion Neurons (High fan-out) > 1 Trillion Synapses Total=100 Billion Neurons 1. Neuron > Transistor 2. Better Programming? ROM? 3. More Richly Interconnected 4. Storage + Processing Red Epic 645 63 Mega-Pixel Approximately 100+ Mega-Pixel (Rod/Cone Count) 6-7M Cones Neuroscience. 2nd edition. Purves D, Augustine GJ, Fitzpatrick D, et al., editors. Sunderland (MA): Sinauer Associates; 2001. http://www.ncbi.nlm.nih.gov/books/nbk10848/ CPU Local Bus 5 To 7 billion transistors CPU Memory Controller I/O Bus (x16 5Gbps = 8GB/sec) Camera Link Interface Card Sam Siewert 6
Perception Driven by Expectation (Visual Cues + Mental Model) Our Model of the World is as Important as Our View Aviation Visual Perception - Flight into Terrain Research, Misperception and Mishaps Specular Reflection M.C. Escher Master of Use of Misperception In Art Wikipedia - Perception is not only the passive receipt of these signals, but it's also shaped by the recipient's learning, memory, expectation, and attention. [4][5] Gibson, James J. "The perception of the visual world." (1950). [Described optical flow] Sam Siewert 7
Optical Illusions the Ponzo illusion the Muller Lyer illusion
Visual Perception is Complex 8 or More Visual Cues for Human Depth Perception Relative Importance of Cues (Context Driven) Binocular disparity Relative size Motion perspective Height in visual field Occlusion Accommodation Aerial perspective (Contrast decreases, Rayleigh Scattering) Texture gradient Sam Siewert 9
Movement (cont) Fitts' Law describes the time taken to hit a screen target: Mt = a + b log 2 (D/S + 1) where: a and b are empirically determined constants Mt is movement time D is Distance S is Size of target targets as large as possible distances as small as possible
Visual Cues Depend Upon Space (Scene) of Interest James Cutting & Peter Vishton Perceiving Layout Sam Siewert 11
Visual Computing Graphics Forward Rendering From Scene Description to Image RGB Resolution and Bits per Color Channel Computer Vision Inverse Rendering From Image to Scene Description Light Field, Not Limited to Visible Photometry Human Visual Cues Characterized but Far From Fully Understood Virtual Reality, Augmented Reality (Challenges) Sam Siewert 12
Radiometry vs. Photometry Radiometry the study of light from viewpoint of Physics Energy (joules), Flux (photons/cm 2 ), Power (watts/m) Full Electromagnetic Spectrum including Visible from Far Infrared to Gammarays Photometry study of light from perspective of useful, often visible light Sam Siewert 13
Photometry What Does Eye See? Color Models RGB Cube HSV - Hue/Saturation/Value Hue Similarity to R, G, Y, B Saturation Intensity of Specific Color Value Low=Black, High=Color Luminance (Candela/Square-Meter) Light Passing Through Area Forming a Solid Angle in A Direction Candela (Photonic Power )= Watts/Steradian More Precise than Brightness Chrominance ( CrCb or UV in YCrCb or YUV) U=Blue Luminance (Y) V=Red - Luminance (Y) Wavelength Spectrum - ROYGBIV RGB Cube http://en.wikipedia.org/wiki/file:rgb_cube_show_lowgamma_cutout_b.png HSV Cylinder/Cone saturation value hue Sam Siewert 14
PIXEL: srgb Basic Color Model Basic Encoding of Picture Element (Pixel) in Grayscale or Color RGB, 24-bit, 8 bits [0-255] for each color band (x, y, z) sampled Each Pixel is a 3-D Vector in RGB Space, Opponent Colors Based On Additive Color Model Compared to Printer s Subtractive Model CYMK Emission of Light in Color Bands (CRT, LCD, etc.) Colorimetry science of color measurement and response a human response to the visible spectrum based on physiology Magenta Blue White Cyan Black Green Bayer 2x2 Pixel Example Red Yellow Sam Siewert 15
Conversion from RGB to HSV Two Simple Color Models saturation value hue Sam Siewert https://www.rapidtables.com/convert/color/rgb-to-hsv.html 16
Machine Vision Color in Brief Physics of Wavelength, Intensity of Electromagnetic Radiation Luminance (Y), Chrominance U=B-Y (blue-luma), V=R-Y (red-luma) Additive Primary Hues (R, G, B) Secondary Hues (Combined R+G, R+B, G+B) CMY Complete subset of Colors is Gamut Chromaticity is independent of the luminance Additive model Measured in Visible for Classic Photometry (UV and Infrared) Visible Photometer (detectors sensitive to intensity in λ bands) Luminous flux compared to standard source (lumen) visible to eye Number of photons or flux in photons/cm2 Spectrograph (frequency separation from incident wave) Radiometry or Radiant flux total power in watts Subtractive model Physiology of What We Can See Tristimulus L (560-580nm) S (420-440nm) M (530-540nm) of Photoreceptors What We See in Various Lighting Conditions Different and Perhaps Not Identical to the Photometer Perhaps Not Identical to Display in Lighting Conditions Tri-Stimulus (Human) Encoding from Measurements to Digital Samples Per Pixel, by color band, in an array Sam Siewert 17
Display Color Transformation Adapting Rec 709 srgb Color Model to Different Lighting Conditions Cinematic, Ambient Fluorescent or Tungsten, Full-spectrum sunlight Based on Primaries, Tri-stimulus, and Lighting, We See a different Gamut Rec 709 Color Space And D65 White Point Plot of Colors Visible to Eye CRT Gamut and CIE Chromaticity Diagram Chromaticity Diagram L (560-580nm) S (420-440nm) M (530-540nm) http://en.wikipedia.org/wiki/file:cie-1931_diagram_in_lab_space.svg Sam Siewert 18
Color Adaptation to Ambient Lighting E.g. Work with eecolor for RT Color Transform at 1080p Gamut Boost for Lighting Conditions and Display Characteristics Sam Siewert http://www.eecolor.com/ 19
What about Grayscales? The Eye Does Not See gray levels lineraly Gray levels Designed for the Eye E.g. Gamma Correction Tri-stimulus of Eye Follows gamma power function Gamma FAQ Color FAQ Panchromatic http://en.wikipedia.org/wiki/file:gammacorrection_demo.jpg Sam Siewert 20
YUV ITU-R BT.601 Component Video Standard Y is Luma UV is Color information designed so that BW TV (CRT) will still display grayscale image Used by NTSC, PAL, SECAM (Standard Definition TV) RGB to YUV Conversion Y = (0.299 * R) + (0.587 * G) + (0.114 * B) U = -(0.147 * R) - (0.289 * G) + (0.436 * B) V = (0.615 * R) - (0.515 * G) - (0.100 * B) YUV to RGB Conversion R = Y + 1.140V G = Y (0.394 * U) (0.581 * V) B = Y + (2.032 * U) Sam Siewert 21
POINT: R, G, or B band only vs. Linear Combination (Human Balance) R G B Y = R + G=R + B=R Y = R=G + G + B=G Y = R=B + G=B + B Balanced Y = 0.3R + 0.59G + 0.11B Sam Siewert 22
OpenCV Single Channel - Balanced 3 Shades of Gray, from Color Channels split out of BGR Code for Windows is here Sam Siewert 23
OpenCV Single Channel - Unbalanced 3 Hues - B+zero(GR), G+zero(BR), R+zero(BG) Sam Siewert 24
POINT: YCrCb RGB An Alternative to RGB is YUV, Where Y is Luminance and CrCb is Chrominance The following 2 sets of formulae are taken from information from Keith Jack's excellent book "Video Demystified" (ISBN 1-878707-09-4). RGB to YCrCb Conversion (For Computers with RGB [0-255]) Y = (0.257 * R) + (0.504 * G) + (0.098 * B) + 16 Cr = (0.439 * R) - (0.368 * G) - (0.071 * B) + 128 Cb = -(0.148 * R) - (0.291 * G) + (0.439 * B) + 128 YCrCb to RGB Conversion R = 1.164(Y - 16) + 1.596(Cr - 128) G = 1.164(Y - 16) - 0.813(Cr - 128) - 0.392(Cb - 128) B = 1.164(Y - 16) + 2.017(Cb - 128) In both these cases, you have to clamp the output values to keep them in the [0-255] range. Sam Siewert 25
POINT: YCrCb 4:4:4 24-bit Format For every Y sample in a scan-line, there is also one CrCb sample Each Y (Y7:Y0), Cr (Cr7:Cr0), & Cb (Cb7:Cb0) Sample is 8 bits No compression between RGB and YCrCb 4:4:4 0 319 76,480 76,799 = Y, Cr, and Cb sample = Y sample only Sam Siewert 26
POINT: YCrCb 4:2:2 16-bit Format For every 2 Y samples in a scan-line, there is one CrCb sample 0 319 48 bit to 32 bit 76,480 76,799 = Y, Cr, and Cb sample = Y sample only Pixel-0 = Y7:Y0 0, Cb7:Cb0 0 ; Pixel-1 = Y7:Y0 1, Cr7:Cr0 0 Pixel-2 = Y7:Y0 2, Cb7:Cb0 1 ; Pixel-3 = Y7:Y0 3, Cr7:Cr0 1 Pixel-4 = Y7:Y0 4, Cb7:Cb0 2 ; Pixel-5 = Y7:Y0 5, Cr7:Cr0 2 Sam Siewert 27
FRAME: XY Pixel Maps FRAME Resolution Computer Graphics Resolutions VGA = 640x480 SVGA=800x600 NTSC = Standard Defintion, 720x480 High Definition (Progressive full frame or Interlaced) 720i/p = 1280x720 1080i/p = 1920x1080 FRAME Aspect Ratios X to Y Ratio NTSC = 3:2, 720x480 (240) HD 720 = 16:9, 1280x720 (80) HD 1080 = 16:9, 1920x1080 (120) 2K = 17:9, 2048x1080 (120) http://en.wikipedia.org/wiki/display_resolution FRAME Rates 60i, NTSC = 59.94 odd/even, or 29.97 24p, Cinema = 24 fps (motion blur) 60p, HDTV Progressive Modes Sam Siewert 28
Display and Camera Resolutions Red Epic 645 9K: 9334x7000, Red Epic 617 28K: 28000x9334 Sam Siewert 29
Perspective Studies by Artists and Physics of Light Silk Screen Painted/Traced Edge Illumination Observer Viewpoint Rays http://en.wikipedia.org/wiki/file:perspectiva-2.svg Sam Siewert 30
Perspective Concept Object Render Plane Intersections viewpoint http://en.wikipedia.org/wiki/file:perspectiva-1.svg Sam Siewert 31
Parallel Projection Cartography Orthographic Mercator (Cylindrical) Orthographic http://en.wikipedia.org/wiki/file:axonometric_projection.svg http://en.wikipedia.org/wiki/file:cylindrical_projection_basics2.svg Sam Siewert 32
Vanishing Point Linear Perspective (1 to 3 Vanishing Points) Curvilinear (4 or 5 Vanishing Points) http://en.wikipedia.org/wiki/file:frankfurt_airport_tunnel.jpg Single Vanishing Point Double Vanishing Point http://en.wikipedia.org/wiki/file:zentralperspektive.png http://en.wikipedia.org/wiki/file:twopointperspective.png Sam Siewert 33
Perspective Projection Projection Method Used in Ray Tracing and Rendering Define Extents of Render Plane l=left, r=right, t=top, b=bottom Viewpoint e is shown on Render Plane Move e out of Plane to Location of Camera/Viewer Fundamentals of Computer Graphics, 3 rd Edition - page 75 Sam Siewert 34
Short-term memory (STM) Scratch-pad for temporary recall rapid access ~ 70ms rapid decay ~ 200ms limited capacity - 7± 2 chunks
Long-term memory (LTM) Repository for all our knowledge slow access ~ 1/10 second slow decay, if any huge or unlimited capacity Two types episodic serial memory of events semantic structured memory of facts,concepts, skills semantic LTM derived from episodic LTM