The INTERMODAL project (1999 - present)

Acronym: INTERMODAL
Name: Intermodal Perception and Movement Control
Type: Collaborative project
Funding: European Network of Excellence ENACTIVE, Institut Universitaire de France
UM1 key researchers: Benoît Bardy – Bruno Mantel (PhD student)
Collaborator: Tom Stoffregen (University of Minnesota, USA), Federico Avanzini (University of Padova, Italy)

INTERMODAL is a theoretical and experimental project questioning the assumption that perception is divided into separate domains of vision, hearing, touch, taste, and smell.

Theory:

In a target article published in Behavioral and Brain Sciences (Stoffregen & Bardy, 2001, 2004), we have reviewed implications of this assumption for theories of perception and for our understanding of ambient energy arrays (e.g., the optic and acoustic arrays) that are available to perceptual systems. We have analyzed three hypotheses about relations between ambient arrays and physical reality: (1) that there is an ambiguous relation between ambient energy arrays and physical reality, (2) that there is a unique relation between individual energy arrays and physical reality, and (3) that there is a redundant but unambiguous relation, within or across arrays, between energy arrays and physical reality.

This analysis has been followed by a review of the physics of motion, focusing on the existence and status of referents for physical motion. Our review indicated that it is not possible, in principle, for there to be a unique relation between physical motion and the structure of individual energy arrays. We have argued that physical motion relative to different referents is specified only in the global array, which consists of higher-order relations across different forms of energy (see Figure). The existence of specificity in the global array is consistent with the idea of direct perception, and so poses a challenge to traditional, inference-based theories of perception and cognition. However, it also presents a challenge to much of the ecological approach to perception and action, which has accepted the assumption of separate senses. 

Experiments:

The theory predicts that human perception and movement control are inherently intermodal, i.e., exist in relations between modalities. The theory is currently tested in the context of the perception of object reachability.

Testing the global array theory involves two distinct steps. The first step deals with the specificity relation that the global array bears to physical reality and that has to be investigated analytically.

The second step is to test whether or not the global array is perceived and used by humans, which is better investigated empirically.

In the experiments currently performed in Montpellier and Minneapolis (see Mantel, Bardy, & Stoffregen, 2005 for preliminary data), we are evaluating the existence of the global array in the context of layout perception, which is often considered a property of the visual system. We formalized an intermodal invariant specifying the egocentric distance between a moving observer and a static object:

where Sx is the linear position of the head along the frontal axis, Vx and Vy are its linear speed (with Vy along the antero-posterior axis). These constitute the inertial terms. On the other hand,   and   are respectively the angular position and velocity of the target in optic terms. An identical equation describes the movement in the antero-posterior plane (with Vz instead of Vy).

In the experiments currently conducted, participants are asked to judge (yes/no) whether, with a simple movement of their preferred arm, they could reach for a simulated object (a target on a screen) located at different distances.

Subjects are sited in a dark room, with one eye patched. Their head position and orientation are captured by means of an electromagnetic sensor (Ascension Technology's Flock of Bird). Displays of the target are driven in real time by the motion of the observer’s head, allowing the target to be virtually located at any distance between the subject and the screen (or even behind the screen).

This phenomenon is illustrated in the figure below. Three main conditions are tested: Vision-Movement (VM - all terms of Equation 1 are present), Vision only (VO - only optical terms), Movement only (MO - only inertial terms). Egocentric distance is specified correctly only in the VM condition, and we expected participants to detect it. In each condition several target distances are used, in terms of percentage of actual reaching distance. Main dependent variables are yes/no responses (approximated with a logistic function), response time and head movements (described by SD and amplitude).

Simulating a virtual target for a monocular observer. The figure illustrate three examples of where the virtual target can be simulated, when the displayed target is moved as a function of the observer movements.Figure 1: Simulating a virtual target for a monocular observer. The figure illustrate three examples of where the virtual target can be simulated, when the displayed target is moved as a function of the observer movements. 

Preliminary data obtained in these experiments indicate that the reachability of an object is specified across optical and inertial energy structure (Equation 1), that is, in the global array. Human participants are sensitive to Equation 1 and, on this basis, detect accurately the egocentric distance.

Key references (downloadable version in page Vitae):

1. Mantel, B. Bardy, B.G., & Stoffregen, T.A. (2005). Intermodal specification of egocentric distance in a target reaching task. In H. Heft & K. L Marsh (Eds.), Studies in perception and action VIII (pp. 173-176). Hillsdale, NJ: Erlbaum.
2. Stoffregen, T. A., Bardy, B. G., Mantel, B. (2006). Affordances in the design of Enactive systems. Journal of Virtual Reality Research, 10, 4-10.
3. Mantel, B. Bardy, B. G., & Stoffregen, T.A. (2007). Critical boundaries and median values in affordance perception. In S. Cummins-Sebree, M. Riley, & K. Shockley (Eds.), Studies in perception and action IX (pp. 222-224). Hillsdale, NJ: Erlbaum.
4. Mion, L., Avanzini, F., Mantel, B., Bardy, B. G. & Stoffregen, T. A. (2007). Real-time auditory-visual distance rendering for a virtual reaching task. In Proceedings of the ACM Conference on Virtual Reality Software and Technology (pp. 179-182). New York, NY: ACM Press.
5. Mantel, B., Bardy, B. G. Stoffregen, T. A. (2010). Locomotor assessment of whether an object is reachable. Ecological Psychology, 22, 192-211.