Music Technology & Special Needs: Part 1

Pioneering work in assistive technology is transforming the lives of people who are unable to play conventional musical instruments.

Doug Briggs is a freelance specialist in assistive and adaptive technology for music-making.

Although many people feel nostalgic for the days of tape, the rise of digital technology has made it easy for us to record faster, more cheaply and more in tune. It's also given most of us new ways of disseminating our work. But for some, the digital revolution hasn't just made music-making easier or faster: it's made it possible.

We all know that in popular music image is all important. Record labels love 'beautiful people'. It is interesting to note that in the history of popular music, until performers such as Robert Wyatt and Ian Dury emerged in the '70s, the number of disabled performers was approximately zero — with one notable exception. And despite the fact that the performer in question was probably the most famous singer of the '30s, she is now very much forgotten, in spite of the fact that she was a bigger hit than Bing Crosby, taught Glenn Miller to arrange, and prompted Ella Fitzgerald to say "I tried to sing like her all the time, because everything she did made sense musically.”

So what happened to Connee Boswell? As a radio star, the public didn't need to know she had polio and was confined to a wheelchair. But as visual media such as film and then television became more important, so her career went into decline. The advent of sound in movies meant that Bing could be promoted as a film star. He is still a household name to this day, but nobody wanted a star of the silver screen in a wheelchair.

What Is Assistive Technology?

Dr. Ian Gibson leads the Adaptive Music Technology Research Group (AMTRG) at Huddersfield University.

For a few years I have been involved in fundraising to support special-needs musicians, and recently decided I should find out a bit more about it. Assistive Music Technology (AMT) is a term coined by freelancer Doug Briggs, one of the pioneering practitioners in making music accessible to those with special needs. There is a common misconception that AMT is purely an extension of music therapy, and I discovered that this is far from the truth. Historically misconceptions also abound about certain disabilities; for instance, percussionist Dame Evelyn Glennie contends that deafness is largely misunderstood by the public. She claims to have taught herself to hear with parts of her body other than her ears, and often performs barefoot to feel the music through her feet.

The world of assistive technology covers two distinct but very much overlapping areas. On one hand you have what may start purely as music therapy, but can awaken a real talent within somebody who otherwise may never have had that opportunity. Alternatively we could be dealing with an established musician, who due to an injury, is suddenly projected into the category of 'disabled', like trumpeter Clarence Adoo of the Jazz Warriors. Some of the technology is thus designed for beginners and for therapeutic needs, but some is for the creative professional, or both.

My research for this article involved a tour of various parts of the UK to talk to some experts in the field, starting off with a train journey up North to speak to experts in the field Doug Briggs, Dr. Ian Gibson and Mark Hildred at the Adaptive Music Technology Research Group (AMTRG) at Huddersfield University. Ian is the group leader, Mark runs Apollo Creative — a company researching, developing and producing AMT — and Doug is a freelance practitioner on the frontline of education and performance, who initiated the idea of the research group. He has adapted many pieces of existing technology for special needs education, and where tight budgets are concerned, is not afraid to get his hands dirty with a roll of gaffer tape and a soldering iron.

Adapt & Survive

Mark Hildred runs Apollo Creative, a company specialising in adaptive technology for music.

It's significant that the name of the research group used the expression Adaptive rather than Assistive Music Technology. "It looks at adapting technology in novel ways,” says Ian Gibson. "It could come from a disabled person battling against having a limited ability in a certain area. We can make it easier for those people to make music, a wonderful way of communication. The other side is that we are all in a digital world of music-making where the possibilities are infinite, and therefore it becomes harder and harder to control the way we make music. It works for disabled people to be able to make music, and for non-disabled people to do it more easily. If you have technology from the industry that makes it easier for a non-disabled person, you should be able to adapt that for people with special needs. It doesn't always work to design technology solely for disabled people.”

"It's an interesting side-effect that if you do design something for a specific disability, often you do end up with a musical instrument suitable for a whole range of people,” adds Mark Hildred. "It might introduce any very young children to music making, or you might design something for mainstream music production as an iPad app, for example, which you discover all of a sudden might be playable by somebody in a wheelchair using their nose.”

"We look at different approaches to making music, and then we see where they can be applied in a number of different areas,” continues Ian. "For instance, we have a graduate student here who has developed an iPad app with a very simple and intuitive interface for controlling surround sound by drawing a path, then manipulating it by the now familiar iPad two-finger squeeze or three-finger rotation. So as well as just being easier for anyone, a person with the use of only three fingers can control the mix.”

"It's interesting to see that the iPad is starting to find an increasingly important role within mainstream and assistive music technology,” says Doug. "Several applications exist which turn the iPad into a touch-sensitive synthesizer as well as a movement-to-audio converter, both of which can be used by some disabled musicians to make music.”

Three Stages

Doug Briggs explains that "In AMT there are three areas we need to think about: the input stage, the processing stage and the output stage.”

Is this analogous to a typical modern digital recording situation where input corresponds to playing an instrument, processing to sequencing and/or processing, and the output being listening to your mix? "Yes, but there are some big differences. For example, you and I know that when you press a certain key on the keyboard you will hear that note, with whatever processing, coming from the monitors. One of the biggest hurdles to overcome is to communicate this idea of cause and effect. It can take quite a while for somebody with learning difficulties to associate the pressing of a button with a sound coming out of a speaker some distance away. Because of these special needs, there may be quite a lot of intervention and help at the input and processing stages, so it's different to a typical sequencing scenario.”

In this first article we'll be focusing mainly on the input stage, and the means by which a performer can generate MIDI data. Mainstream music-making generally employs purpose-designed switches such as keyboards or drumpads, which are built to output MIDI. For AMTRG's work, it's often necessary to make use of the plethora of generic AAC (Alternative and Augmentative Communication) switches available, and in these cases, a separate MIDI conversion stage is required. Electronics enthusiasts can make their own interfaces using an Arduino kit from Maplins.

"They are switches, but not integrated MIDI switches in the way you might think of a keyboard or drumpad,” says Doug Briggs. "We can use those, but we also need to use many other types of switches to deal with the different abilities. For example, one person may only be able to move their little finger, another their elbow. Somebody may not be able to move any limbs, so we might have a switch activated by breath, or by sensing a sound they make.”

Two typical AAC switches that can be adapted for use in assistive music technology projects.Doug gives some examples of the kinds of AAC switches which can be adapted to musical purposes. "First we have momentary switches, a simple on/off trigger. You hold it down for on and release for off, or else you press it once and it latches, then again to turn it off. It could be a big round switch, which we call a jelly bean switch, or an eye-blink switch, for example.”

"There are a huge number of these switches to suit a whole range of disabilities because simple on/off switches are used for computer controls, wheelchair control,” continues Mark Hildred. "There are switches you blow, switches you suck like a straw.”

"The possibilities are almost endless,” says Doug. "There are many different types of switches from the world of AAC available to us, and luckily there is a standard output, the ubiquitous 3.5mm mono jack. We can use these to connect a switch to a MIDI interface, which can then address the software in the processing stage. It makes sense to use these off-the-shelf AAC switches and integrate them, rather than try to develop thousands of dedicated AMT switches.”

In Proportion

As well as momentary or on/off switches, music-making also requires variable-voltage switches that can output a range of values. "These are what we call proportional or variable sensors,” says Mark Hildred. "We have specific ones like the Soundbeam [the widely used system designed by EMS's Robin Wood and composer Edward Williams], which is an ultrasonic sensor that can detect how far away something is. And of course, that was originally developed for dance, so it's an example of something that has been adapted for AMT.”

"Another sensor we used was originally designed for sorting potatoes, but discovered and adapted for music at the University of York,” adds Doug.

"You see something and think 'I can use that!'” continues Mark. "Some sensors and switches would be much too expensive for us to develop, but when they get used by the mega corporations in the games industry, we can get something really low-cost and use or adapt it. Xbox wireless controllers are a prime example.”

To what extent is it necessary to adapt triggers as opposed to buying them off the shelf? "It's an interesting area, because with music, there's a huge amount of data to be controlled,” says Ian. "We are lucky because in recent years we've seen companies like Nintendo tackle a similar problem. We are looking for controllers which allow broad gestures to control a whole number of different parameters in one go, as opposed to just one parameter such as pitch, timbre or volume individually. If you have one controller for each parameter, you would soon run out of fingers or things available to use for these controllers. For some applications, a greater deal of customisation is required, that's where companies such as Mark's [Apollo Creative] come into play.”

"Our system doesn't use the Wii controller, though it is possible to download software from the net to convert that data to MIDI,” continues Mark. "From my point of view the data it can grab would be very useful, but the question is whether the people who might benefit from a disability point of view could actually hold and use a Wii controller — it can take a lot of physical effort. However, the interesting thing is because the MEMS chips that detect the rate of acceleration and position are used by companies such as Nintendo and Apple in their products, all of a sudden the cost of them has dropped dramatically because they are now being made in their millions, which makes it viable for a small company like us to put them into a specialist product. We have some that are like an easier-to-use Wii controller, which is either handheld or strapped to a wrist, elbow, leg — whatever. Every individual you meet has a different need.”

"People are actually getting more interested in this area, because of these new ways to trigger sounds,” explains Ian. "In the past, people were just used to key presses triggering sounds. Things like the Wii have made people gain interest in controlling sounds through other devices. Also people can now use images — there is all sorts of software now to map images from webcams to musical parameters. I think people can often actually get more inspired to create music with devices other than simple keyboards.”

Lateral Thinking

As well as being adapted to generate MIDI data, the switches themselves often need to be customised, as Doug explains. "Everyone is different, so it makes sense to quickly knock something up out of household materials rather than go out searching for expensive off-the-shelf products. For example, for one musician I gaffa-taped a piece of sponge to a proximity sensor.Adaptive technology is often a matter of finding simple and practical solutions to problems. Josie O'Malley found it easier to use this proximity sensor, and to associate her actions as causes with the musical results as effects, when Doug Briggs taped a piece of sponge to it to provide tactile feedback. The problem for children with learning disabilities is that technology can sometimes get in the way of cause and effect; they don't get the tactile feedback. Josie has cerebral palsy, erratic motor control, and I just figured she needed some resistance to press against, so she can actually feel something is there, rather than just her hand moving through air. It works really well with her. It's also a problem with Soundbeam, which is an invisible beam in the air that effectively converts movement along the beam to MIDI. To help children understand the beam is actually there, I sometimes just tie a piece of string along the beam for them to follow.Mary Jarvis performs using a Soundbeam system. Again, a simple low-tech addition — a piece of string following the line of the invisible beam — helped make the instrument easier to use. I've also started using Arduino boards to make very cost-effective switch-to-MIDI interfaces.”

"As musicians, we interact with our instruments in a physical way: we get vibration and feedback from them,” says Mark. "People are looking at things like video cameras and Microsoft Kinect as ways to interface, but the problem with that is you're waving around in mid-air. It's difficult to work out how to get that muscle memory going and get the physical feedback and learn the repeatability. And if that equipment is set up slightly differently next week — say the video camera is half an inch further one way or the other — all of a sudden you make that same motion in the air and it doesn't respond to you. That's why, in spite of these amazing advances, a lot of the technology I'm involved with uses physical objects people can interact with.”

There's also the question of what assistive technology should look like. "Again it's an interesting area,” says Mark. "A lot of the companies which produce the specialist equipment are quite small, and the market is not massive in comparison with that for standard instruments. You have to produce an item which might be used by a young child who's happy with something that's bright red and yellow. On the other hand you might also be dealing with a professional musician who's been involved in a traumatic accident and been paralysed. They certainly don't want something that looks like it belongs in a nursery. So we find a balance between those two extremes and go for generic shapes, sizes and colours which work for everybody. But if you want to put it in a bright yellow enclosure, you can do.”

Further Hurdles

Despite all their achievements so far, the group are aware that many further challenges remain. "The first big challenge for me is to be able to use proximity and movement sensors to play rhythmically,” says Doug. "Young people love rhythm, obviously, there's a groove and it pulls them in. Even for a non-disabled person it's extremely difficult to play a Soundbeam rhythmically. I can address that problem to a certain extent using Sound Sculpture, but it's still a big challenge to go further with it.

"Second, as we mentioned before, this idea of establishing and reinforcing cause and effect is an enormous challenge with the use of technology, which can often get in the way depending on how it's used. It's all very well having a piece of kit that is easy and sounds great, but if they don't feel what's happening is caused by them intentionally, there is no learning. Cause and effect is the first thing we aim for; we then establish intentionality, and learning flows from that.”

"A simple problem is, you've got a switch and you press it, but the sound comes out of the speakers over there by the computer,” explains Mark. "It seems obvious to us, but only because we've already learnt that.”

"Somebody with learning difficulties might not make that connection for a long time,” continues Doug Briggs. "This is why we sometimes prefer the idea of lots of different hardware boxes and modules. Even if everything can now be done in one computer, it's easier for a teaching situation if what's going on is more obvious. This sensor talks to that box, which then goes to that computer, and the sound comes out of this monitor.”

"If you have, say, six people all making music, and all the sound coming out of one pair of speakers, it's hard for them to differentiate their own sound,” says Mark Hildred. "It's also difficult for the audience to know just what is going on, the magnitude of the achievement. We've added lighting, video and images into our software, so not only might there be a video, but the lighting can change to give a visual cue.”

"It's a performance language,” explains Ian Gibson. "The audience have to learn about how you are performing. They get to understand that a certain gesture is particularly skilled. It's all about educating the audience as well as supplementing the performance.”

Next month, we will be looking at a few more switches before moving on to the music processing and performance stage. We'll look at the most oft-used software and hardware products, and explore some of the ways in which these have been put to use by artists. 

Go To Part 2