산 • MOUNTAIN

Synesthesia is a neurological condition in which stimulation of one sensory or cognitive pathway like hearing, leads to automatic, involuntary experiences in a second sensory or cognitive pathway such as vision. Simply put, when one sense is activated, another unrelated sense is activated at the same time. This may, for instance, take the form of hearing music and simultaneously sensing the sound as swirls or patterns of color.

This is what Raina Liao a brilliant korean-canadian percussionists experience each time she plays the marimba. Together, with 8000 km between us, we worked on a live performance that could help her share with others what she sees by playing Clark Hubbard’s “Mountain” a very challenging 6 and a half minute piece about the sense of ‘wonder’ one experiences when out in nature.

Using the composition theme as a starting point for the visuals, I developed a digital system in TouchDesigner that could reproduce what Raina’s brain does, turning musical notes into spots of colors in the space. True to her description each note (C, D, E, etc) matches an specific color (Red, Orange, Yellow, etc), the pitch is expressed by the lightness or darkness of the color and the position of the sound source marches the position of the color spot.

The protagonist of the visual is Raina’s performance, expressed by the color spots generated by the marimba score. Secondary elements react to the electronic track of the composition and represent more closely the theme by generating a changing landscape that spans across the 5 screens surrounding the performer.

Watch the full performance here.