Rachel Van Besouw & The Interactive Music Awareness Programme
Cochlear implants help the profoundly deaf experience sound. But can users overcome the limitations of the technology to understand music?
Pete Thomas
Cochlear implants can sometimes help people overcome profound hearing loss, but experiences such as listening to music can become very confusing.Cochlear implants can sometimes help people overcome profound hearing loss, but experiences such as listening to music can become very confusing.
Cochlear implants have been a considerable success story in helping profoundly deaf people to hear speech. However, it has always been assumed that they would be of little use for the appreciation, performance, composition or production of music. At the world-renowned Institute of Sound and Vibration Research at the University of Southampton, scientists are working to overcome this perception. Dr Rachel van Besouw has teamed up with Professor David Nicholls and Dr Ben Oliver from the Department of Music to create the Interactive Music Awareness Programme (IMAP). With the help of AHRC funding, this is part of a research project called Compositions for Cochlear Implantees, which is proving that, with the right training, breakthroughs can be made in the area of music appreciation for cochlear implant users.
What Cochlear Implants Can Do"The main limitation of a cochlear implant,” says Rachel van Besouw, "is that with up to 22 electrodes we have to try to replicate the job of upwards of 15,000 hair cells connected to about 30,000 auditory nerve fibres in the cochlea, which help us get very fine frequency resolution.”
This makes it sound as though a cochlear implant is attempting to replace a 15,000-band graphic equaliser with a 22-band equaliser, but it's a lot more complicated than that. We can begin to get a very rough idea of what a cochlear implant sounds like to the user by using a visual analogy. We can compare a well-focused, clear image with a blurred, out-of-focus image, analogous to somebody with typical old-age-related hearing loss and a clearer, but pixellated image which corresponds to how somebody with cochlear implants hears sound. The experience of hearing music through a cochlear implant is different from the limitations imposed by, for example, age-related hearing loss. To make a visual analogy, the latter might be considered similar to fuzzy or blurred vision, while cochlear implants present a 'pixelated' or low-resolution image.The experience of hearing music through a cochlear implant is different from the limitations imposed by, for example, age-related hearing loss. To make a visual analogy, the latter might be considered similar to fuzzy or blurred vision, while cochlear implants present a 'pixelated' or low-resolution image.
"The cochlea is 'tonotopically organised'; that is, the auditory nerve fibres towards the tip (the 'apex') of the cochlea spiral respond best to low frequencies, and those near the start (the 'base') of the spiral respond best to the high frequencies. In a healthy cochlea, pitch discrimination is best in the low-frequency region, around 500Hz, and in that region we can discriminate differences in frequency as small as 1Hz. The cochlear implant makes use of this tonotopic organisation by sending low-frequency information to the most deeply inserted electrodes and high-frequency information to the more shallowly inserted electrodes. However, unlike a graphic equaliser, a cochlear implant does not convey what we call the 'temporal fine structure' in each band, which we rely on for perceiving pitch accurately; it only provides the amplitude envelope information.
"The input frequency range of an implant is also typically between 100Hz or so to 8kHz, depending on the make, and so an implant user won't hear anything above or below. Part of the problem is because the bass area is deep inside the cochlea, and we can only physically insert the electrodes about two thirds of the way in. When an implant patient has what we call their first 'tuning session', about a month after their operation, they often complain that everything sounds shifted, high, electronic, tinny or like a Dalek.”
Cochlear implants also present a vastly smaller dynamic range than a typical human ear. "The dynamic range between sounds that the normal ear can only just perceive and sounds that are too painful is about 120dB. The input dynamic range of a cochlear implant is between about 40 and 80 dB, and levels below a certain threshold get cut off. But the electrical dynamic range of a cochlear implant is even smaller, and so sounds are further compressed to a range of about 10 or 20 dB.
"With a cochlear implant, things get complicated, because if you change, for example, the loudness of a sound, this can also change the number of electrodes responding and the pattern of stimulation, which can sound like a pitch change. So loudness can affect pitch, pitch changes can also affect the perceived timbre of a sound, and changes in timbre can affect pitch and loudness... these three things affect each other.
"Implant users rely a lot on what we call place pitch cues. As a musical note changes pitch, the stimulation pattern across the electrodes changes, but not always in a way that you think would be consistent with such a change in pitch. Sometimes implant users experience 'pitch reversals' where say, an increase in pitch is perceived as a decrease.
"The amplitude modulation of the electrical pulses, particularly for the apical (low-frequency) electrodes can also give what we call rate pitch cues. It's a weak cue, but it appears that some implant users can make use of it to discriminate differences in pitch that are much smaller than one might expect, given the few electrodes and their spacing in the cochlea.”
Rachel van Besouw of the University of Southampton's Interactive Music Awareness Programme.Rachel van Besouw of the University of Southampton's Interactive Music Awareness Programme."Some people have described listening to music through an implant as like listening to a piano being played with a boxing glove on: the rhythmic information is good, but the frequency resolution is poor. Sound separation is also very difficult. In a normal hearing situation, we use harmonics to group sounds, but an implant user does not get fine enough frequency resolution with the electrodes, so picking individual instruments out of a mix is a real challenge.
"So our big limitations are frequency resolution and dynamic range. In addition to the limited number of electrodes to do the job of stimulating those thousands of auditory nerve fibres (the 'electro-neural' bottleneck), there may well be 'dead regions' in the cochlea which won't respond to the stimulation, so some of those electrodes may not be doing anything useful. We would love to have more electrodes if we could control the pattern of current spread in the cochlea, but at the moment we can't control that, so having more would not necessarily help, unless we can come up with another way of stimulating the auditory nerve fibres.”
Learning To Listen
This is where the Interactive Music Awareness Programme (IMAP) comes in. Its underlying principle is that with the right guidance, cochlear implant users can develop an understanding and appreciation of music. "Many implant users, especially those with a memory of what music sounded like prior to losing their hearing, are often disappointed when they first hear music, and this can result in them actively avoiding music,” explains Rachel van Besouw. "Compared to speech, they also have fewer opportunities to develop their music perception abilities. It might sound strange, but many implant users find that they have to re-learn what a piano or a violin sounds like.
"We are developing a rehabilitation programme which is about training. We have found that with training it is possible to improve a cochlear implant user's ability to recognise instruments. Because cochlear implant users receive little, if any, harmonic information, they rely heavily on an instrument's attack/decay/sustain/release (ADSR) envelope for recognition. Training may help them to use these more subtle cues. However, cochlear implant users still find it especially difficult to distinguish instruments of the same family.
"The Interactive Music Awareness Programme is a training programme that we have developed together with cochlear implant users. We have taken this approach because we believe that in order to meet the needs and desires of users, it is essential to actively involve them in the development process. From consultations and workshops, we found that cochlear implant users wanted a resource that would not only enable them to develop their music-perception abilities, but also help them re-engage with music. They wanted help to discover new kinds of music that they might be able to appreciate, giving them the motivation to continue to practise listening to music.”
Implanting AwarenessPhoto: Image Cochlear EuropeAs part of the process of engaging implant users in music, the team have developed a way to give listeners more control over what they're hearing. "One of the tools we've come up with in the IMAP is called the N-Machine, named after Professor David Nicholls in the Department of Music, which enables the user to come up with a mix of a piece of music that suits them. This involves getting hold of individual stems from artists. The importance of this tool is that an implant user can work out what they can hear clearly and what they can't. They can start to bring other things into the mix when they feel more confident and start to discriminate between sounds. We've also done workshops in which they do a mix of live musicians, and typically we find they start with a mix of just one or two instruments. It helps them to make sense of a song that they may have been familiar with before losing their hearing. They are mixing to suit their implant. One of the pieces that we have stems for is Ravel's Bolero, and that's gone down very well. It's very useful as an orchestral track because it's highly rhythmic and repeats a lot, with different instruments playing the same refrain. The repetition helps them to 'put the puzzle together'.
"So far, what we have used comes from generous upcoming artists such as Robin Grey, Blue Swerver and Madelaine Hart, as well as music from the Research Assistant composer on the project, Dr Ben Oliver. When we extend this, of course it would be wonderful to get stems from some more established artists. Philip Selway from Radiohead is one artist who has kindly donated some stems for the research phase of the project, and we're hoping to get more established artists on board. We'd be over the moon to get more stems from artists that we can use, not only for implant users but hearing-impaired listeners in general. One of the important aspects of the training programme is about using 'real world' material to help implant users explore and engage with music again, and helping them know what's out there. One of the frequent comments we get is, 'I want to get into music again, I just don't know where to look and what's best for my situation.' They might have been listening to quite complex orchestral music which may no longer be suitable, and they might not have had the courage or confidence to try another genre.
"Another tool that we have in the IMAP is called the 'Environmental Rhythm Machine', which uses everyday sounds around the house: teaspoons, microwave beeps, car door slams and so on. Users can create a simple drum-machine type track with samples of everyday sounds, and then build these up and layer them. It's a gradual way of helping cochlear implant users re-engage with music through the creative use of sound effects. What's nice about that is that they don't need any prior knowledge of music and don't have to worry about not knowing the difference between a cello and a violin, which can be intimidating. We also have another piece of software in the IMAP that allows the user to listen to a melody and change the instrument. They can then find out which instrument is best for hearing pitch changes. Some of the software tools in the IMAP allow the user to pitch-shift stems, so if there is a bass guitar playing a riff that they can't hear, they could try shifting it into a different range to see if this helps.”
SOS readers will, of course, be very familiar with what we've just discussed, ie. basic mixing, but for many people, this is a brand new way to experience music. It reflects the fact that the experiences of cochlear implant users seem to vary a lot, to the extent that there is no 'one size fits all' mix that will suit everyone: each person may need something different. "Yes, it's highly individual,” agrees Rachel van Besouw. "And the situation where users can mix and manipulate the instruments encourages active listening. Just as you don't get fit by watching sport, it is less likely that you will improve your music perception abilities with incidental, passive exposure to music.”
The Rate Of Progress
Results so far have been encouraging, but it's early days. I asked Rachel to speculate about what might be possible in the future. Will we ever see the day when a profoundly deaf person with cochlear implants becomes a sound engineer or producer?
"At the moment, the limitations of current devices makes it seem unlikely, but then 30 years ago many believed that cochlear implants would merely aid lip reading — and now, many implant users are able to use the telephone and perceive speech without any visual cues at all. Currently, we have implant users at the South of England Cochlear Implant Centre who play musical instruments and are doing very well. It certainly seems easier for young implant users who haven't experienced normal hearing before to accept and engage in music. Many have music lessons at school and learn to play instruments along with their peers. Cochlear implant users are still are exceeding our expectations, and I am optimistic about future developments in the technology. I think when the breakthrough occurs it will be at that electro-neural interface. It might be a different kind of electrode, different placement of the array or a form of stimulation other than current. Or it could be in the form of drugs that encourage auditory hair cell and nerve fibre regeneration, but I think we are a long way off being able to generate hair cells where we want them and get them connected to the nerve fibres that we want them to be connected to in humans.”
How about the many musicians and engineers who suffer gradual hearing loss as a result of exposure to loud music? As Rachel explains, current cochlear implant technology is not really designed to address this sort of problem. "The first step would be to look for opportunities to reduce the level and duration of their exposure to noise and to protect their hearing to prevent further hearing loss and tinnitus. If they are concerned about their hearing, they should see their GP for a referral to an audiologist who can assess their hearing to determine the type and degree of hearing loss, recommend interventions that may help and provide advice. One option might be a hearing aid. Cochlear implants are only recommended for people with severe-to-profound bilateral hearing loss, who get little benefit from hearing aids.
"Unfortunately, there is still a stigma attached to hearing aids and many hearing-aid users still experience prejudice. Whilst a hearing aid makes a person's hearing impairment visible, it does not tell you about the magnitude of that person's impairment and whether or not their impairment would affect their role as a producer or engineer. Would you rather have a sound engineer who has addressed their hearing loss with the latest hearing aid technology, or a sound engineer who has not addressed their hearing loss?”
The Interactive Music Awareness Programme has a web site at www.southampton.ac.uk/mfg/current_projects/trial.html .
What Is A Cochlear Implant?
Once implanted, each of the electrodes within a cochlear implant needs to be 'tuned'. The result of this process is a 'map'; it is often helpful for users to have different maps for different listening situations.Once implanted, each of the electrodes within a cochlear implant needs to be 'tuned'. The result of this process is a 'map'; it is often helpful for users to have different maps for different listening situations.
In short, a cochlear implant is a surgically implanted electronic 'hearing' device, which uses an electrode array to stimulate nerve fibres in the cochlea to provide the auditory signals which would normally be transmitted by the hair cells.
1. Sound is captured by a microphone.
2. A sound processor converts the signal from analogue to digital information.
3. Digital signals are sent to the implant from the headpiece.
4. The implant converts the digitally coded sound into electrical impulses, and sends them along the electrode array, which is positioned in the cochlea (inner ear).
5. The implant's electrodes stimulate the cochlea's hearing nerve, which then sends the impulses to the brain where they are interpreted as sound.
Cochlear implants are fine-tuned for the individual recipient, as Rachel van Besouw explains: "The clinician will tune each of the electrodes, setting the lower threshold and maximum or 'comfort' level for each one. This process is called creating a 'map'. You can also change the frequency-to-electrode allocation. For example in this case (image below) the filter centre frequency of the most deeply inserted electrode is 333Hz and the filter centre frequency of the highest frequency electrode 6665Hz.
"A cochlear implant user will undergo many tuning sessions, more frequently to begin with, and it's possible for them to have different maps that they can switch between, like presets, using a remote control. The maps might have a different dynamic range, microphone directional characteristic or noise-reduction algorithm for listening in certain environments.”
One limitation on current technology is that low frequencies cannot easily be conveyed, partly because the electrode array cannot be inserted right up to the apical end of the cochlea, where the nerve fibres respond best to bass frequencies. "Bass frequencies will be limited by the input bandwidth of the implant, by the insertion depth of the electrode array in the cochlea and by the ability of the implant user to make use of the weak rate pitch cues from the amplitude envelope of the stimulation on the apical electrodes. If an implant user has some residual hearing in their other ear, they may be able to perceive (and benefit from) some additional low-frequency information.”
No comments:
Post a Comment