Creative opportunities: Playing with expectation

Creative opportunities: Playing with expectation 8 min

As we build up multisensory experience throughout our lifetimes, we develop expectations about the types of sensory input that are likely to appear together. For example, if you see that you are approaching a rose bush, you may also start to expect the smell of roses. In the following sections, we consider some creative opportunities that these multisensory expectations might provide. Building on the example, if it turns out that the roses actually smell of peppermint then this is likely to surprise and perhaps delight an audience (whereas if the roses smell of something unpleasant then that could be shocking and unnerving).

Sensory substitution 8 min

Left Figure: 3D-printed tactile map generated with Touch Mapper, showing raised braille and surface features of Egham, UK. Right Figure: the corresponding OpenStreetMap base map of the approximate same area of Egham as shown in left. Map data © OpenStreetMap contributors, available under the Open Database License (ODbL).

It is increasingly possible to translate information from one sense into another, a process known as sensory substitution(e.g., sound to vibration via haptic vests; vision to audio* sonification*). This may happen in real-time (e.g., haptic navigation vests) or through pre-prepared representations such as tactile diagrams. In the figure above, a tactile map portrays visual information represented in tactile form; this can improve accessibility by providing alternative ways to represent information for people with diverse sensory needs (e.g., providing real-time subtitles for people with hearing loss). It also opens exciting creative opportunities, for example, through translating a performer's body movements into sound and light, such as in Steady State.

Mid-performance photograph of Steady State. © R. Murphy O'Sullivan, 2024.

Case Study: Steady State Created by performer-technologist and research lead Zubin Kanga (Royal Holloway, University of London), composer Alexander Schubert (Musikhochschule Hamburg), brain sensor and neuro-diagnostics company* ANT Neuro*(Netherlands/UK/Germany), brain-computer interaction engineer Serafeim Perdikis (University of Essex) and a wider team of artists, Steady State(2024) recasts the recital format as a retro-sci-fi laboratory in which the performer is both subject and instrument.Steady State was created as part of Cyborg Soloists, Zubin Kanga's* UKRI Future Leaders Fellowship* research project. 6 min

Wearing an ANT Neuro Electroencephalography (EEG) cap, hand-controlled motion sensors and multiple muscle sensors, the artist Zubin Kanga streams brainwave rhythms into a custom Max system, while finger movements and torso gestures are routed to TouchDesigner for real-time processing. This system simultaneously moulds and morphs the immersive electronics sounds and sculpts dreamlike holographic projections. Using Steady State Visually Evoked Potential signals (a standard brain-computer interface technique), Kanga controls audiovisual patterns by focusing on different strobing objects, facilitating the first ever use of brain sensors to consciously control sound and visuals on stage. Finger articulations trigger a virtual piano via motion sensors, while Electromyography (EMG) sensors modulate the bass electronics. In the work's climax, the performer's brain and the audiovisual system enter a feedback loop, accelerating to a hallucinatory and transcendental conclusion.

Case Study: Watch-based haptic navigation 5 min

Apple Watch uses distinct haptic patterns paired with brief tones to signal left versus right, turns and arrival, letting people navigate without inspecting a map. This feature can now work when there is no internet connection. Similar navigation is available with Google Maps on mobile and Wear OS (see figure below).

Illustrative haptic navigation patterns across smartwatch platforms. © A. Woods, 2026.

Crossmodal illusions 4 min

The senses can interact in ways that create a range of illusions, where information from one sense alters how we perceive another. Some of these could have interesting creative applications. For example:

• Anti-gravity illusion: By constructing an environment with a sloped floor but building the walls and props perpendicular to that slope, the visual cues convince people that they are standing upright even when they are actually standing at an angle. Other people in the room can then appear to lean at impossible angles, upside down, or pour liquids uphill.

The Reversed Room at Illuseum Berlin. A rotated set makes visitors appear to stand on the ceiling, creating an upside-down gravity effect. Photo: © A. Woods, 2023.

• Rubber Hand illusion: Visual and tactile information can be integrated to alter a person's sense of body ownership. In a typical setup, the person's real hand is hidden from view, while a visible fake, or rubber hand is placed in front of them. When the (hidden) real hand and (visible) fake hand are stroked in the same way at the same time, 80% of people report feeling that the rubber hand is becoming part of their own body³³. Similar principles extend to whole-body illusions in VR, where synchronous visual and tactile stimulation can induce the compelling experience of inhabiting a virtual avatar³⁴.

• Parchment-skin illusion: This illustrates how auditory information can influence tactile experience. People rub their hands together in front of a microphone which plays back the resulting sounds over headphones with minimal delay. When the high frequency components of the audio are boosted, people typically perceive their skin as feeling drier or rougher, like parchment³⁵.

• Size-weight illusion: The size of an object can affect how heavy it feels: smaller objects often seem heavier than larger ones of the same weight. This is because we expect the larger object to weigh more than the smaller one, so when they weigh the same, the contrast with our expectation makes the smaller one feel relatively heavier. VR researchers can amplify this³⁶ by making a virtual object shrink or swell just before the user picks it up.

Crossmodal correspondences 2 min

Our brains tend to link sensory features across the different senses, creating correspondences³⁷ that shape how we perceive the world. For instance, people from across many cultures have been shown to pair the sound kiki with a spiky, more angular shape and the sound bouba with a more rounded shape, demonstrating correspondences between certain speech sounds and particular visual patterns³⁸ (see Figure 3).

The Bouba-Kiki effect: observers overwhelmingly match smooth shapes with soft-sounding pseudowords such as bouba or maluma, and spiky shapes with sharp-sounding pseudowords such as* kiki and takete*. A. Dunn/Wikimedia Commons, CC BY-SA 3.0 or GFDL 1.2+; monochrome adaptation by Bendž.

There are too many of these correspondences to attempt to list them all in full, and new ones are frequently being identified, but some additional key examples are:

• Bright visuals are often associated with louder sounds and dimmer visuals with quieter sounds. Brighter visuals are also typically paired with higher-pitched tones³⁹.

• Visual object size is also linked to pitch, with smaller objects associated with higher pitches.

• Rougher tactile textures are associated with harsher, higher energy sounds (e.g., white noise or scratchy static)⁴⁰.

• Heavier objects are expected to sound lower in pitch.

Many of these links are influenced by culture, language and personal experience³⁷. For example, in many Western societies, people report that cinnamon smells sweet, presumably because it is frequently paired with sugar in desserts⁴¹. In contrast, in cultures where cinnamon is more commonly used in savoury dishes, it may not be perceived as sweet at all. Learned correspondences like these, shape how we interpret sensory information, meaning that different audience members are likely to have different patterns of correspondences.

These observations highlight the potential of experimenting with sensory pairings: consistent, intuitive pairings can feel seamless, whereas deliberate mismatches can create novelty or tension by disrupting expectations⁴². Well-established correspondences can also be recruited to imply certain aspects of an environment that aren't simulated, such as reverb being used to evoke a sense of space in an auditory-only experience⁴³.