Gestural Interaction

Gestural Interaction Enhancing Human Judgment when exploring large information spaces Ray Garcia, MIT Media Lab, March 23rd, 2009 rgarcia at media dot mit dot edu

Scene – Augmenting Intelligence

LARGE SCALE RICH INFORMATION SPACE MEDIATIONAL MEANS

Layered Expression

SELECTION NAVIGATION GESTURAL SHAPING INTERFAC

E MATERIAL OF MANIPULATION High Fidelity

Grok

INTUITION

UNCOVERING PATTERNS FOR DECISION Visual MAKING Configuration

Geek

ANALYSIS

FINDING THE STORY IN THE GAPS WHOLE BEING INTELLIGEN CE

Human Communications • Layered channels of expression – Voice with tonal variation – Facial qualifiers – Body positions – Spatial distance to objects – Hand movements – Writing, drawing

Hidden honest signals • Thoughts inscribed in media – Media: stone carving to pixels – Language expression • • • • •

Emotions Symbols Visual representations Musical notations Artistic rendering

Technology as Mediational Means • • • •

Remote versus Local communication Asynchronous versus synchronous Human to Computer to Human Technology provides the object, symbols, material to manipulate • Need to learn the affordance of the interface

Improving Fidelity of HCI • Signals – Add gestural interface to complement symbol and voice commands. – Initially only hand movement – Future include facial expression

• Response – Visual screen that fill peripheral vision – Spatial Audio – Expressive 3D object in motion

Material of Manipulation • Light and sound produce meaningful representations • Appeal to human cognition and perceptive input • Build comprehension and significance of information space • Meta-cognition may be enhanced by mix mode of interaction

Whole being Intelligence • Intelligence is in the nervous system as well as the brain • Human judgment under uncertainty uses whole being intelligence • Much information is ambiguous and vague • Human comprehension is based on impressions of patterns • Interpretation of representations needs advances

Augmenting Human Intelligence • HCI as a moderating social engagement • Two negotiating parties • Fusing interpretations of representations • Synergistic symbiotic interaction through Gestures

Scenario • Saul manipulates representation using gestural interface • Ava manipulated the system configuration • Saul uses intuition to find patterns while Ava provides analysis to extract the material and representations • The material is large scale rich information space needing discovery and analysis to uncover patterns for decision making

Finding the Story • The scenario fosters story telling • Pattern recognition has limitation • People fill the gaps by creating a hidden narrative using interpretive explanation • In business the story is left to intuition • The intuition dominants decision making in real-world situations

Dance of Gestural Interaction • Discovery of the aesthetic of representation and expression that fosters better augmented human judgment • Archeology of knowledge as told by the story of a social experience manipulating the material of rich information spaces

Research is needed • Whole being intelligence • Cognitive interfaces • Improve Human to Computer to Human social communications • Adopt Gestural Interfaces to enhance expression • Goal should be to improve human judgment under uncertainty when information space is large