Gesture Control For Electronic Music

Imogen Heap uses Glover extensively in her performances.Photo: Fiona Garden

Imogen Heap and Chagall are among the artists pioneering new ways of interacting with synths and effects, courtesy of the Glover software.

Any acoustic musician knows that there is a deep interaction between how you engage with your instrument and how it sounds. Body, posture and movement are all integral to what and how you play. Despite all the amazing inventions and developments in music technology, machines, pads, mouse and faders can still get in the way of this interaction, and performing electronic musicians often look for ways to humanise their performances. Grammy Award‑winning recording artist Imogen Heap and her team of innovators think they have a solution in the shape of Glover.

“Glover is the magic glue enabling and empowering artists to gesturally interface with off‑the‑shelf interfaces such as Leap Motion, MicroBit, your phone [with Gliss] and our MiMu gloves,” explains Imogen. “There’s nothing off the shelf that would do this with the flexibility and detail I needed, suited to musicians. I wanted to be able to reach into the software I love, not have to turn a pan pot to pan a sound, and not look like I’m checking my emails.”

Tired of having cables, screens and controllers get in the way of making and sharing her music, Imogen sees Glover as enabling musicians to focus more deeply on exploring their creativity. It integrates with a DAW‑based setup, adding a new dimension to existing kit, and is also OSC (Open Sound Control) addressable, allowing it to control and manipulate other performance elements such as visuals and lighting. “At the flick of a wrist you can bring in an orchestra, or set the stage lights to follow you around on the stage,” enthuses Imogen.

Surely You Gest?

Glover is part of a larger mission that has been looking at new ways of making music with computers. MiMU’s managing director and lead software developer, Adam Stark, got on board about 10 years ago to help develop what was a one‑off addition to the commercially available MiMu gloves — a wearable musical instrument providing a complex gestural interface. With the MiMu gloves and Glover software, you can use any posture, finger and hand gesture to control music wirelessly.

The Glover software was originally designed to work with the MiMu Glove wearable controller.Photo: Nathan Gallagher

“Whatever I do with the gloves and Glover, I have a start‑up scene in Ableton,” explains Imogen. “From that preset, I put my hands up facing North or 0 degrees, pointing upwards, and that takes me into my first scene, which clears any loops in that session. So it mutes any effects, the clock is set to stop and then I am ready to go into my first scene. What I do is set North with a double rock sign.”

This may sound complicated on paper, but the point is that in practice it feels completely intuitive and natural to Imogen, allowing her to carry out a number of commands with just one or two movements. She has different scenes configured to map movement to a software instrument or parameter. Once her ‘North’ is set up, all other movements and gestures are relative. So, when she is panning sound, she does so with reference to the centre of the stage. If she’s at her piano, she will want everything panned and mapped relative to where she is sitting. She can then go into a preset that means she can create a live loop. “This is fun! I can always access my effects. My delays and reverb are in my right hand. On my left, I close my fist to create my first loop. I start in a free tempo so the first loop creates the tempo. I open the left had to start the playback of that loop. Open and close my fist, and I can add overdubs. If I want to delete that, I drop my hand down, past the 90‑degree point.”

Imogen Heap: It’s not a thought, it’s fluid. It’s an extension of my singing. It’s a quick motion that feels very natural.

Imogen also controls effects, using a ‘secret’ finger‑programmed posture which involves closing and opening her index finger. “When I put my hand up directly in that scene, that controls my delay. And when I push my hand to the right, I have access to a different, shorter reverb. It’s not a thought, it’s fluid. It’s an extension of my singing. It’s a quick motion that feels very natural.” Another trick she does is to close her fist to send to a really long reverse reverb in Reaktor, mainly routing her piano and her voice and playing it back in contrary motion. For as long as she keeps her fist closed, the reverb reverses the signal.

The Gloves Are Off

Chagall is another keen Glover (a nickname for MiMu glove musician community members) and the UX designer for Glover and Gliss. “I have been using the gloves and Glover in my live music performances since 2015 — with Ableton Loop, South by Southwest and TEDx as some of the highlights — so a lot of ideas for the software actually emerged on the road. I use Glover to control my sounds and effects, as well as real‑time‑generated visuals and lights. For me the result of using my body movement to control everything has been life‑changing.”

Chagall is a pioneering user of gestural control who has also been involved in the development of Glover.Photo: Eduardo Fitch

When Chagall came on board as a designer in 2014, she felt Glover could be more user‑friendly. “We used to joke that the team wasn’t allowed on the same plane, because if it crashed no one would know how Glover worked any more. I realised that if we wanted to share Glover with lots of musicians it needed a redesign.”

As Adam explains, this process has seen Glover evolving into a more general application that works with many controllers. “Glover started as the software for the MiMU gloves, beginning with Tom Mitchell’s early experiments with Imogen Heap. It began as a way for a musician to connect the different sensors in their gloves to sound controls, such as MIDI. It also contains many different algorithms for gesture detection, and machine‑learning algorithms to recognise hand postures and the ability to connect all these things together to control music.

“About five years ago the penny dropped for me, and I realised that what we had developed with Glover was really a general tool for making music with gesture. The gloves were just one possible controller in that world. The wider opportunity was to open up the world of music to gesture control by allowing musicians to use many different sensor‑enabled devices. Over the following years, we began to add support in Glover for new devices such as Ultraleap’s Leap Motion controller, the micro:bit developed by the BBC and Thalmic Labs Myo. We then began work on our own smartphone app, Gliss, to allow people to control music with their phones too. Around 2017, we opened up Glover to a beta tester programme and had an amazing group of 100 people testing Glover and giving us feedback. This group, plus all the MiMU glove users, integrated Glover into real performances, for real projects, and the software was fully road‑tested by professional musicians.”

All You Need Is Glover

Glover is by no means the first tool to implement gesture control in music. It stands on the shoulders of more bespoke and individual contributions by academic researchers — check out Rebecca Fiebrink’s work — and there have been apps supporting limited musical mapping capability for specific gestural devices, such as GECO, OSCulator or MyoMapper and STEIM’s junXion. Adam Stark believes “what sets [Glover] apart is that it is the first software to be a hub for different gesture‑control devices to access the same user‑friendly interface for composing music in an expressive way.”

Glover can run on Mac or Windows, and sits between music software such as Ableton Live and a compatible controller. This could be an iPhone (an Android version is in development), but it could also be Leap Motion, micro:bit, the MiMU gloves or anything that can send OSC (Open Sound Control, a protocol for networking) data. It can be used to add effects or to play software instruments directly, while manipulating a number of parameters. Other musicians have combined Glover with real instruments such as a guitar or piano to add effects to those instruments in an expressive way.

The Glover team hope their work is opening up a new phase of computer music‑making and performance. Adams talks about how Glover “comes to life when it crashes into the ideas of artists. The most exciting thing is just watching the way that different people use it to do things I’d never even thought of.” Chagall dreams “of some sort of way to not have to wear or hold any controller, and Glover just knowing what you are doing”. Meanwhile, Imogen, the instigator of this new paradigm of working with your computer to make and manipulate music, is looking forward to a world where you can more fully “develop your own space for your musical idiosyncrasies, create presets to explore a world of sound to write fluidly in, taking you away from your usual patterns that you can fall into, building in happy accidents at all the crossover points possible”. Check it out. Maybe the future is gestural.