Creating empathy through sound: A case study of Notes on Blindness: Into Darkness

The article is a case study of Notes on Blindness: Into Darkness, a VR experience built from the audio-cassette recordings of Prof. John Martin Hull, in which he documented his experiences as a blind man.

The article investigates the technology of spatial audio and its application in the experience to understand how the technology creates a sense of presence and can tell a story with little visual detailing.

UNDERSTANDING SPATIAL AUDIO

Spatial audio is an immersive sound format used to create a 3-dimensional sound experience when listening to it. This format enhances music, movies as well as VR simulations.

Spatial audio presents the listener with a realistic experience of sound. It provides the experience of sound reaching the audience from a particular point in 3-dimensional space. The technology gives the audience a feeling of presence in a virtual environment. Unlike surround sound format which uses front, back and side speakers to create a 360° audio experience in 2-dimensional space, spatial audio adds the element of height to the sound field.

Binaural 3D sound rendering creates spatial audio for immersive experiences. In Digital Illusion: Entertaining the future with high technology by Clark Dodsworth Jr., authors Scott Foster and Toni Schneider mention that the goal of an accurate binaural 3D sound rendering system is to present the participant with an audio experience that mirrors the way sound is experienced in real life, creating the illusion of being present in a virtual space (Dodsworth Jr., 1998).

In addition to this, the chapter You Are Hear: Positional 3D Audio also gave insight into how sound is perceived in the physical world as well as an introduction to phenomena such as sound localization i.e, the ability of the listener to identify the origin or location of a detected sound in direction and distance as well as primary spatial localization cues such as Interaural Intensity Difference (IID) and Interaural Time Difference (ITD) that help the brain narrow the position of the sound source.

Research into the workings of spatial audio gave an understanding of how the technology is used to create realistic auditory experiences in VR. While visual fidelity plays an important role in the suspension of disbelief in VR, the realism of sound adds another layer of believability to the experience.

CREATING EMPATHY THROUGH IMMERSIVE SOUND IN VR

Notes on Blindness: Into Darkness is an immersive VR experience based on Prof. John M. Hull's sensory and psychological experiences with blindness. The experience was released alongside a feature film.

Through the use of spatial audio, the experience makes the audience empathize with Hull, taking them on a journey inside a blind man's psyche, revisiting his thoughts, fears and joys of life after losing his sight.

THE JOURNEY

The experience splits into six segments, each corresponding to a particular recording and memory.

HOW DOES IT FEEL TO BE BLIND?

The first recording consists of Hull describing a day in the park by the lake. As he describes the environment, sounds of various elements of his surroundings begin to play in sync with the corresponding visuals. Through the narration, a soundscape is built comprising the sounds of children playing, birds flying overhead, cars moving behind him, ducks swimming in the lake and so on.

This segment served as an introduction to the application of spatial audio in a VR experience. Sounds of people, birds, trees and inanimate objects such as cars and merry-go-rounds come together to build a soundscape that defines the environment the audience is in. Aside from demonstrating the impact of spatial audio, it gives insight into what life would be like for the blind.

The above image is a screenshot taken during the first segment of the experience. It shows visualizations of people walking past the participant in a park.

FEELING THE WIND

In the second segment of the experience, the participant finds themselves in a scenario that resembles a rainy day. Hull reflects on the difference in the idea of a nice day between an individual with sight and one without. For a person with sight, a 'nice' day is sunny and dry while, for Hull, his concept of a 'nice' day has rain and thunder with the wind taking the place of the sun.

Hull explains how the effect of wind gives life to a blind person's surroundings. The participant is asked to hold the trigger button of the controller, causing the wind to blow in a particular direction, revealing various parts of the environment such as trees and the swing. He also describes thunder as a phenomenon that defines the dimensions of the outdoor space.

This segment sheds light on how the blind use natural phenomena to perceive their surroundings. For the blind, the rainy weather is a medium to 'see' and experience the world around them.

The above image shows the porch environment where the participant is positioned.

ON PANIC

In the third segment, the audience is asked to walk from the porch into the yard as Hull gives an account of his fears and anxiety as a blind man. He compares walking through this desolate environment to walking through nothingness, with each moment amplifying a sense of loneliness and the feeling of being enclosed.

This chapter demonstrates how silence creates a feeling of helplessness for the blind. In the absence of sight, sound plays a crucial role in visualizing the world around us. In a desolate environment, with little to no life, the audience is given a first-hand account of the fear, uncertainty and restlessness a blind person would feel without sound.

The environment in 'On Panic' features a far lower density of visuals and sound than the previous two segments.

COGNITION IS BEAUTIFUL

In the fourth segment of the experience, Hull speaks about the beauty of rain and how it gives shape to the objects of his surroundings. Following a fade, the audience is brought inside and prompted to gaze at various objects in the room. As the participant looks at each object, raindrops begin to fall into the room against various objects, defining their shape and the space around the audience. The screenshot below shows the participant inside the Hull's home.

The events of this segment illustrated how rain reveals the presence of objects in a blind person's surroundings while giving identity to these objects depending on the sound produced when a raindrop collides with the surface of an object.

THE CHOIR

In the fifth segment of the experience, Hull reflects on his experience of listening to a performance by the choir at a party. The audience experiences a soundscape comprised of the mumblings of people, piano music and choir harmonies. Hull reflects on how the description of the visuals of his world would not have amplified his experience of the performance. Hull's time and familiarity with blindness came with the acceptance and realization of the irrelevance of what he had lost.

Through the performance, the visuals defining the space fade while the form of the musician and the singers lose their definition. The diminishment in clarity of the visuals as the performance progresses could be interpreted as Hull's immersion into the performance, with the sound of the music and the harmonies sung by the choir being the sole focus of his experience.

EPILOGUE

In the final segment of the experience, the audience finds themselves back in a park with Hull sharing his thoughts on revisiting the recordings he made of himself more than thirty years ago.

In this segment, the visual style is brighter and features a broader range of hues. The park appears to be similar to the one in the first segment. One could interpret this as the narrator (and the audience) coming full circle in their understanding of blindness. Apart from the broader range of colours used, the people are rendered as blurred, vague shapes. This could be interpreted as the narrator's acceptance of being blind and 'seeing' the world beyond appearances.

COMMENTARY ON THE VISUAL STYLE

This experience effectively demonstrated the use of spatial audio in creating a sense of presence in VR. Unlike most experiences that rely on the need for flashy visuals and animations to define a space, the producers created an experience that uses sound to give identity to a space.

While the visuals were minimalistic, they complemented the sound, painting the audience a picture of a blind person's thoughts and perception of the world around them. This could be seen in the difference in the style in which people were rendered between the first four segments and the epilogue. The people in the first four segments of the experience are rendered as crisp and clear silhouettes which could be interpreted as the narrator's need to paint of picture of what they would have looked like, had he still been able to see. In the epilogue, the narrator has accepted the loss of his sight, no longer requiring it to 'see' and experience the world around him. Hence, the act of drawing approximations of people's appearances as well as the world around him has become irrelevant.

While the experience represented how a blind man would 'see' the world around them, the lack of cohesion with the style of rendering felt unsettling, particularly from the point of view of a visual artist and designer. This lack of uniformity in the style of rendering can be seen when comparing the first segment, How does it feel to be blind? and the second segment, Feeling the wind. In the first segment, the merry-go-round and the car, which are both inanimate objects are rendered as translucent, holographic forms, but in the second segment, the wind chimes and the swing are represented as forms made up of particles. Despite there being a visual language, it is crude and not presented well enough.

COMMENTARY ON THE PACING

Rather than using action to create engagement among the audience, the experience uses empathy and emotion. Compared to most VR simulations, the experience progresses at a much slower pace, giving the audience sufficient time to absorb their surroundings. On one hand, this gives the audience to fully experience the simulation without getting distracted by the tasks within it. On the other hand, the lack of interaction and slow pacing can, at times, feel frustrating.

FINAL THOUGHTS

The experience demonstrated how VR experiences could create a sense of presence and engagement with little interaction, creating a scenario where the participant is mostly a spectator. A simulation of such nature allows the participant to pay closer attention to the story and the experience, without being distracted by the task of interacting with it.

REFERENCES

• Foster, S. & Schneider, T. (1998). You Are Hear: Positional 3D Audio. In Dodsworth Jr., C. (Ed.). Digital Illusion: Entering the future with high technology. Addison Wesley: Boston.

• Notes on Blindness: Into Darkness (2016). Oculus Quest [Game]. Ex Nihilo, ARTE France & Archer’s Mark: France.