1
2	Roadmap Mystery of vision and historical theories Ecological basis High-level perception concepts Figure-Ground Size constancy Depth and object solidity Lower-level: rules for image “construction” Visual Intelligence [Hoffman 1998] book has 35 rules that the perceptual system uses to decode visual stimulus and create a useful description of the world These underly the higher-level perceptual effects Perception research in the Visual Methodologies [Rose 2001] framework
3	Vision Science “…scientists in neurology, neurophysiology, the cognitive psychology of vision, artificial vision, machine-assisted imaging, image analysis, and the comparative study of animal vision generate more papers and monograph per year than all of visual studies [i.e., visual theory, as in last lecture] combined (by far…I would estimate the ratio is fifty to one. Statistically, science is where vision is studied, not the humanities.” [Elkins 2003] p87. (Visual Studies: A Skeptical Introduction)
4	Brown U. Vision Research (not comprehensive) Brown Brain Science Program (BSP) Cog sci, vision, neuro, psych, engineering, and more Work in neurological systems of vision; physiology of vision; pattern theory; visual attention; depth cues, neural, behavioral and computational models of faces and facial expression; computer vision; perceptual learning; semantic-perceptual interaction; visual navigation; and more. Human vision (examples, not comprehensive) Michael Tarr (Cog Sci) Bill Warren (Cog Sci) Billy Wooten (Psychology) Computer vision (examples, not comprehensive) Michael Black (CS) Gabriel Taubin (Engineering)
5	History Not at all obvious how vision works New type of retinal cell just discovered 3 years ago! Research at Brown (BDH article) Plato, Aristotle (and others) eyes send out light to “see” objects Epicurus: “ ‘thin, hollow films’ of atoms—eidola—which retain the shape of the object shedding them, continuously form around macro-objects, travel swiftly through the air, and enter the eye, stimulating visual sensations.” http://bulldog2.redlands.edu/fac/jeremy_anderson/research/Epicurus.pdf Kepler, 1604: Eye is a lens that focuses light on the retina But how do we make meaning out of light patterns?
6	Hierarchy of Processing How do we get from photons to object recognition and categorization to conscious understanding to meaning and emotion? Many stages of processing, with feed-forward but also feedback between stages
7	Gestalt Theory Developed in Germany in early 1900s. Response to reductionist methods of science “There are wholes, the behavior of which is not determined by that of their individual elements, but where the part-processes are themselves determined by the intrinsic nature of the whole. It is the hope of Gestalt theory to determine the nature of such wholes.” Max Wertheimer, 1924 http://gestalttheory.net/archive/wert1.html#fn1 No one ingredient (or all of them randomly thrown together) make a cake—need recipe Detail elision Hierarchy of abstraction Use in psychology, art, other fields, as well as vision science and perception
8	Anatomy of Visual System
9	The Retina “Mammalian lenses—look like onions. Hard to imagine more crappy optics.” Michael Tarr Retina has 100+ million photoreceptors Photons changed to electro-chemical signals, go through additional layers of cells + processing Only ~1million nerves leave the retina: 100/1 data compression “The computing power of your retina dwarfs the most advanced super computers.” [Hoffman 1998] p66
10	Retinal Array You somehow construct lines, curves, etc from patterns transmitted by this neuron array Visual cortex has cells sensitive to: Line orientation: horizontal, vertical, and diagonal, Changes in brightness (edge detection) Length Motion Even direction of motion and more… Movie of retinal excitation (guinea pig) in response to moving bars http://www.snl-e.salk.edu/technology/ Vision is “noisy”—massive statistical processing apparently takes place
11	Processing in Visual Cortex Light stimulates retinal cells, impulses are transmitted, info arrives in visual cortex creating a retinal map. Retina 2D and curved. Visual cortex convoluted. How does brain extract meaningful information?
12	Components of Vision Processing Revealed by Brain Injuries Man who, after brain damage from carbon monoxide poisoning, could not see/make sense of objects, despite no problem with visual acuity or seeing motion, etc. [Hoffman 1998] p47 Woman with dorsal simultanagnosia: can see parts but not assemble large group of them into a scene or even simple parts into one object. Saw pitcher, handle—said maybe suitcase? [Hoffman 1998] p79 Woman who could see “fine” but could not see motion: liquid pouring looked frozen, cars appeared suddenly (this state can be induced by magnetically (temporarily) impairing certain area of visual cortex). [Hoffman 1998] p139
13	Computer Vision Despite power of today’s computers, they can’t process visual information nearly as well as we can Complexity of our visual system explains why it's so difficult to get computer visual systems to "recognize" anything even remotely as well answer do... Advances in computer vision and vision science influence each other E.g., exciting new work in artificial retinas
14	Roadmap Mystery of vision and historical theories Ecological basis High-level perception concepts Figure-Ground Size constancy Depth and object solidity Lower-level: rules for image “construction” Visual Intelligence [Hoffman 1998] book has 35 rules that the perceptual system uses to decode visual stimulus and create a useful description of the world These underly the higher-level perceptual effects Perception research in the Visual Methodologies [Rose 2001] framework
15	Ecological Development of Vision 1/3 Visual system solves problems with no unique solution but gets “right” answer vast majority of time
16	Ecological Development of Vision 2/3 James J. Gibson: vision evolved to help us stay alive in ancestral world—find food, avoid falling off cliffs, being eaten by lions, etc. Lab experiments leaving out crucial context for vision May understand physiology without grasping “vision” World not simplified shapes or isolated dots—not surprising trouble decoding them AI researcher David Marr asked “what is vision for?”: “…a process that produces from images of the external world a description that is useful to the viewer and not cluttered with irrelevant information.” [Pinker 1999] p213. [emphasis mine]
17	Eye-Brain System Pragmatic Hermann von Helmholtz (1821-94) Wore prisms shifting world 11-degrees 1896 George Stratton-glasses that inverted retinal image 8 days Another experimenter (Kohler, 1962) did same for longer periods. Able to ride bicycle, ski, and more. [Palmer 1999]
18	Ecological Development of Vision 3/3 “…objects don’t go out of their way to line up in confusing arrangements.” [Pinker 1999] p212 Cohesion makes smooth contours Motion, tension, gravity cause straight lines, right angles Near-parallel lines in image or from one point of view usually relate to parallel lines real world Organisms that move evolve to be symmetric Objects look different under different lighting, but important to be able to identify them nevertheless
19	Eye-Brain System Powerful—e.g., Parallel Preconscious Processing We preconsciously (aka preattentively) process whole “fields” of certain types of visual information all at once, i.e., in parallel Don’t have to search sequentially through each visual object to see “pop outs”
20	Parallel Preconscious Processing—Another Example This example uses assumption that light comes above (more on that later) Choosing visual representations carefully can help make scientific visualization work better
21	Physiological Basis for Parallel Processing Layers of retinal maps in visual cortex with sensitivities to features such as lines, color, etc.
22	Preconscious Processing in Scientific Visualization Top: red = salmon circles (vs. squares) = hot temperature. Bottom: red = hot temp squares = salmon (circles = no salmon) “Results from our work provide a number of guidelines for the use of hue and form in real-time visualization. Hue can be used to perform rapid and accurate boundary and target detection. Form can be used to perform boundary detection, but it cannot be used as readily to perform target detection if a secondary data dimension is encoded with hue. If a user wants to perform real-time multidimensional visualization, hue should be used to encode the primary data dimension being investigated. Secondary data dimensions can be encoded with form. This will not interfere with boundary and target detection tasks performed using hue.”
23	OK, But We Already Know How to See… Why study vision system (or teach visual thinking) when we (mostly) all see fine? …And we learned without any apparent effort Parents didn’t coach us or correct us (as they probably did with language) No courses in “seeing” as there are in reading, writing and even speaking Music, foreign languages after age of 12 usually require instruction (and some of us never “get it”)
24	Why is Studying Vision Important? Interesting for its own sake Exciting field Become more conscious of vision process Better interpret visual communications Create better visual communications, from art to graphic design to UI design Help develop computer vision Better understand the human experience and thought process
25	An Aside  Importance of Optical Illusions Are exactly situations that don’t occur in nature Illusions used to reveal processing rules of eye-brain perceptual system May seem silly, but can reveal profound things Perceptual system makes a best-guess, but sometimes, especially in carefully chosen situations, the system is wrong
26
27	In-Class Exercise—Pop Out Link to explanations and worksheet Study your brain Experiment with parallel processing pop-out to help design useful visualizations Tips (parameters to change) Orientation Line length Value (how dark/light) Curvature Shape Size Enclosure Juncture (whether lines fully connect) Parallelism Number
28	Roadmap Mystery of vision and historical theories Ecological basis High-level perception concepts Figure-ground Invariance Frame of reference Depth cues Lower-level: rules for image “construction” Visual Intelligence [Hoffman 1998] book has 35 rules that the perceptual system uses to decode visual stimulus and create a useful description of the world These underly the higher-level perceptual effects Perception research in the Methodologies framework
29	Figure and Ground Discovered in 1921 [Palmer 1999] p.280 We tend to divide a scene into figures and background This happens pre-consciously Usually easy to tell which is which (some ways of telling next) Usually only one side of a contour belongs to a “thing” or figure. When not, brain is confused Familiar shapes appear to be figures faster than unfamiliar ones…
30
31	Impossible Trident (aka Three-Pronged Blivet)
32	We Look Mostly at “Figures”… Tracking eye motion. Difficult to jump to a void—we focus on the “figure”.
33	What Makes a Figure? Ecologically Sensible and Tested Surroundedness Size (smaller) Vertically or horizontally orientated objects more likely to be figures Contrast (including of color and texture) Symmetry Convexity (bulging out vs. caving in) Parallelism (contours parallel) Familiarity NB: motion often dominates in real life and moving images
34	Invariance “The mechanism that recovers stable and rigid objects from a myriad of continuously changing retinal stimulations is called perceptual constancy.” [Massironi 2001] The Psychology of Graphic Images: Seeing, Drawing, Communicating. P27. Prof Fulvio to discuss 3D “optic flow” theory behind this, and other research hypotheses (April 12)
35	In-Class Exercise—Disassembling a Face “from [Massironi 2001] pp. 43, 44. Link to explanation and worksheet
36	Frames of Reference Largest surrounding “frame” will be reference for all Evolutionary: we are sensitive to up-down orientation of our frame (e.g., tilted room), but not so much to right-left
37	Depth Cues 1: Overlap, Relative Height, Atmospheric We’ll be looking at depth cues again in more detail in the 3D Graphics lecture Interposition (overlapping) Relative height Aerial or atmospheric attenuation Many shown live at http://psych.hanover.edu/Krantz/sen_tut.html
38	Depth Cues 2--Shadows Shadows help induce a ground plane
39	Depth Cues 4: Shape from Shading Gradient on surface implies changing surface orientation (vs. changing surface color) Use “light comes above rule” to further determine shape of surface
40	Depth Cues 5: Projection more on projection (and binocular vision) later Parallel lines converge Relative size/Size constancy Texture gradient Interactive animation
41	In-Class Exercise—Depth Cues Link to explanation and worksheet Tips: Overlap Relative height Atmosphere Shadows Shape from shading Projection (includes perspective)
42	Example…