Paul White is given an exclusive preview of a new technology that will revolutionise music making — and this time, it really will!
Have you ever wondered exactly how the great classical composers manipulated the emotions of their listeners so precisely, or how a virtuoso player can make the hair stand up on the back of your neck? And have you ever wondered if there is any way to inject this quality into synthesized music? It's tempting to dismiss the question by observing that some people just 'have it' while others don't, and that music, being an art form, can never be precisely analysed, but that explanation obviously didn't satisfy Olaf Lopir, head of the Music Psychology Research department at Norway's Trøndheim University.
Like many scientists, Olaf maintains that all art forms have an underlying logic that does stand up to traditional scientific analysis — but rather than trying to prove this point by looking for some quality in the music, he reverse‑engineered the problem, by looking first at the human brain. When you think about it, this makes perfect sense, as both the creation and appreciation of music happens in the brain.
Over the past four years, Olaf's research team has been working with over 200 volunteers to try to isolate not only that part of the brain that responds to music, but also to isolate the key components in music that create the maximum stimuli. Working closely with the university's medical department, and using a Computerised Axial Tomography (or CAT) scanner, coupled to a computer system capable of assigning colours to differing degrees of brain activity, three‑dimensional images of the working brain of the listener were produced for a variety of musical and non‑musical auditory experiences.
Patients undergoing CAT scans are normally placed on a table inside a large, movable coil, and as this isn't the ideal environment in which to appreciate music, it was decided to use high‑quality headphones rather than conventional stereo speakers. Soft lighting was introduced to create a neutral, non‑threatening atmosphere, and any volunteers prone to claustrophobia were eliminated from the trial.
It will come as no surprise to those who have an interest in such matters that the majority of the brain's response to music takes place in the right‑hand side of the brain — but that isn't the only part of the brain that reacts. Apparently, a lesser degree of activity also takes place in the visual cortex, suggesting that music is subconsciously translated into patterns or images.
Mood Modelling
Of course, it's one thing to look at a general reaction in the brain, but quite another to isolate the musical element responsible for a particular emotion. This problem was tackled by getting the panel of listeners to describe their emotional state while test pieces of music were being played. It turned out that the CAT scan images for any individual show significantly different patterns corresponding to the emotions eventually categorised as: Sad, Uplifting, Anticipation, and Gratification.
The next step was to take all the musical passages that produced the same broad emotional reaction, then cross‑correlate them to identify what elements they had in common. The various musical passages were checked for spectral content, timing deviation from a quantisation grid based on the mean tempo, and, in the case of non‑chromatic instruments, pitch fluctuation (both degree and envelope). Not surprisingly, the timing of the notes played has a profound effect on the depth of the emotion created, but not, as it turned out, on the type of emotion. Similarly, pitch deviation (such as vibrato) enhances the sense of emotion, but doesn't usually determine the type of emotion. The only exception to this was found in tests with single sustained notes, during which it was found that pitch modulation could influence the way in which the note was perceived.
The results of the spectral analysis were more complex. It seems the musical scale on which any tune is built creates a kind of emotional expectation matrix, which can take the listener in different emotional directions depending on the way in which musical phrases are resolved. What wasn't expected, though, was the result of the 'deep cross‑correlation' test, which was designed to strip away everything except the common tonal elements of the test pieces, in the hope of identifying the 'essential something' that makes music have the emotional effect it does.
This part of the experiment was more successful than anyone could have imagined; once the superficial musical elements had been mathematically filtered out, the resulting audio file turned out to be remarkably consistent for any piece of music known to create a specific emotion. Heard on its own, the audio is meaningless, and sounds not unlike random noise passed through a vocoder, but when the signal is mathematically subtracted from the original music file, the emotional response of the panel of listeners was found to be considerably diminished, even though there was no obvious difference in the overall subjective sound. In fact, the 'emotokinetic' signal, as it has come to be known, is always at least 10dB below the level of the music from which it is extracted, and thus is never clearly audible in isolation. This finding led to the next obvious experiment — adding a looped version of the signal to bland, synthesized music.
Virtual Artificiality
This is where things start to get really interesting; a simple, sequenced string passage was played, using a General MIDI module as a sound source, with no vibrato, and strict quantisation. Heard on its own, the general response was pretty neutral, but when it was played for a second time with the subliminal 'Sad' emotokinetic signal added, the result was dramatic, with some of the panel members actually dabbing their eyes with tissues!
In addition to the unique emotokinetic signals for the different mood types, the pitch bend characteristics were similarly correlated, and it was found that a specific contour of pitch bend, plus a slightly asymmetrical, delayed vibrato, further enhanced the mood being projected. This contour was extracted, recreated as a MIDI sequencer controller track, and then applied to the lead string part. Again, the panel of listeners noted an increase in emotion, and claimed to feel as moved by the simple MIDI file as they did by a real virtuoso string performance of the same piece of music.
The Spin‑Offs
As you might imagine, once the major synth manufacturers got wind of the project, it didn't take them long to work out a way to include this technology in their instruments, and already three key manufacturers have signed up to license Lopir's emotokinetic mood signature files and pitch modulation profiles to use in their next generation of synths. The new technology, licensed under the name of TWANGO (Trøndheim Wave Articulation Nuance Generator/Operator), will enable the user to add subliminal, emotional overtones to their compositions with full vector mapping. Using a simple joystick interface, the user will be able to automatically and dynamically move from, say, a sense of expectation to fulfilment as a chord resolves, or to increase the depth of sadness at the end of a sustained note.
The latest news from the university is that further emotions and feelings are being investigated, and that already, an emotokinetic waveform corresponding to paranoia has been extracted. Perhaps one of the most useful side effects of the paranoia emotokinetic wave is that in most listeners, it creates the illusion that what they are hearing is directly behind them, regardless of the actual sound source. It's obvious that this system is going to become standard issue for all but the cheapest synths, so if you find yourself inexplicably moved to tears by a GM module rendition of 'The Birdy Song' or 'Agadoo', you'll know that you've been TWANGO'd.