Sound Design For Visual Media

You have the skills and the gear, but how should you tackle your first sound-for-picture projects?

Anyone who has experience of producing music with computers should already have the basic technical skills and equipment required to design sound for moving pictures, whether that be for film, TV, animation, computer games or anything else. Rather fewer, though, have a realistic idea of how best to apply those skills in what's a very different industry, usually involving a bigger and more varied creative team. With that in mind, I'm going to explain how you might best approach your first projects, in terms both of planning and organisation — including how you can figure out precisely what's required of you — and of how to make more effective use of your skills when actually fashioning soundscapes in your DAW.

Refine The Brief

So, you've been given the opportunity to do some sound design on a project. Where do you start? For me, the first task is to find out as much about the project as possible by figuring out — and perhaps also contributing to — your client's creative vision. Whether the project is big or small, it's important that the sound designer and their client have a common understanding of the creative direction the sound design should take. Usually this will involve a written brief of some sort. A vague written brief is not uncommon, but don't assume that this means you have complete artistic freedom. Sometimes the client will give you a level of autonomy that results in you having to try out lots of ideas until you find something that works, other times they'll be very specific about what they want and give you clear direction. You really must gain an understanding of what the client actually wants — it's up to you to take the brief, ask questions and gather references to refine it down to something that enables you to do your job.

A good approach is to start with the basic details of the project — identify the genre, the period in time and the style of the project — and then ask yourself some questions. What are the creative influences? What is the visual or graphical style you're attempting to portray? What kind of atmosphere are the producers looking to create? Having a grasp on the answers to these questions will give you a great starting point before work commences on creating the sound assets you'll use in the project.

In The Beginning

In my career in game audio, some of the most memorable moments for me have been at the beginning of projects, where all the creative heads meet to discuss the style and creative direction of the game. There may be early concept art, or reference images from other movies or games to look at and discuss, or clips from movies and other influential videos. Eventually, a general style guide is built up that provides just enough information for the different creative departments to begin their tasks.

A planned pre-production phase follows, during which these initial creative ideas are assembled. My goal during this period is to try and answer the following questions:

• What role does sound play in the piece overall?
• How should the sound-effect design convey the world?
• What role does music have in the production, and what are appropriate styles and genres?
• What role will dialogue have in the production? How many characters are there? Is the piece light or heavy on dialogue? Are the voices effected in any way?It might help if I run through a typical scenario. You've been asked to produce sound and music for a short animation (two or three minutes long), and the visuals are still being animated. Obviously you can't start sync'ing sounds to the piece, as it's not finished.

  Get hold of any relevant material you can to help refine your brief during the early stages of your project — even crude storyboards will help.Photo: Creative Commons 30So you start by acquiring a storyboard of key events (even crude drawings will be useful) and any stills or visual inspiration, such as clips of other movies the animators are using as their own reference. You can probably also get hold of early animatics of the movie (animations that are not properly rendered, or completed visually, but are sequenced chronologically as per the storyboard).

Armed with these, discuss with the producers their creative goals, and have them explain the style, mood and themes of the piece. What do they see as the message, assuming there is one, that underpins the sequence of events? Is it intended to be comical and light-hearted, or dark and foreboding? And so on.

Now you can start to build yourself a style guide. Creating a style guide is a process where you try out different ideas to determine which ones work the most effectively. This might entail creating a number of small movie sequences that feature different sound styles and music genres — outside of radio, these are usually, but not always, set to picture.

The music will probably benefit from its own separate style guide — as with the other sounds, you assemble selections of appropriate tracks and try them out against the visuals. These can be used as a reference for pace and tone for the composer, if and when they come on board to create original music later on in the process (I'm assuming here that the sound designer is not also the composer, although they may be invited to do both roles on low-budget productions).

When you listen to sound design you admire in movies, games or other mediums, try to imagine just how many revisions that work went through before it was finally released!

How Much Detail?

Now you know the brief, you've tried out some ideas and have determined a style, you're almost ready to begin work on the actual sound design. First, there's just one more creative decision to make: how much detail needs to be added? Or to put it another way, how many sounds should you put in to reinforce the visuals, and how much sound is too much? How much sonic detail is required in each project is something you need to establish early on. At one end of the spectrum, an animation such as Pingu works best with a sparse soundscape — too much sonic detail would detract from the overall effect. But try applying such a minimalist approach to a project that's aiming for gritty realism, and the result be completely inappropriate!Novice sound designers will often underestimate how many sounds are required in a scene and, consequently, produce lightweight, unconvincing soundscapes. They let the music do all the work required to carry the piece. This is not a music video: if you fail to inject sufficient sonic detail, the finished piece will seem sparse — in fact, it will seem unfinished.

At the other extreme, I've seen professional sound designers add far too much detail, such that scenes sound too busy. Just as when you're arranging or mixing music, if there are too many sounds vying for attention, it's really difficult to bring the right things to the fore; similarly, there's much more potential for frequency clashes and masking. The result is usually an undesirable cacophony!

So how do you strike the right balance? Largely, the visual style of the piece will determine how much detail is required. A simple animation, for instance, may be suited to a sparse soundtrack that's largely driven by music, especially if the visual style is clean and minimalistic. On the other hand, an immersive sci-fi film may require multiple layers of sound effects and music cues, all of which are necessary to convey the gritty reality of a future world. A general rule is that if something moves on screen, some sort of sound effect is usually added. The exception is that you shouldn't add any sound that you know won't be heard when music is added.

When Less Is More

If there are a lot of characters on screen or a number of similar events (a regiment of soldiers marching, for instance), it's not necessary to laboriously spot sounds for each and every event. That would be a long, painstaking and pointless job. Instead you can create sound effects that give a sense of the action, with just enough sound to convey what is going on. In fact, with the marching soldiers, you might be surprised by how effectively the on-screen action is reinforced when you add just three or four pairs of footstep sounds. As members of the audience, our brains may be able spot bad sync'ing, but they're not cynical — they're actively expecting to sync sight and sound, and this example works because there's just enough information for our brains to make the connection that there are a multitude of actors all performing the same action.

Similarly, in a battle scene we may add several weapon-fire sounds, but it's usually best to use just enough to convey the story, with different details coming to the fore to suggest that more is going on. Any more, and once again we run the risk of creating a distracting cacophony. This is an example of how sound in films must differ from real-world sound: in a real battle, the intensity of sound would disorientate the combatants, but in a movie depiction of a battle the sound designer must consider the space required for dialogue and music.

Painting In Layers

Different sound designers have different workflow philosophies, but assuming you've gathered the materials you need to start work, including video and other required media, you could do worse than follow the general approach that I use, which is detailed below. First, though, before you place any sounds on the timeline, configure your DAW project to minimise hassle later on. Create tracks in your DAW that relate to each sound-effect group. Typically, you might assign three or four stereo tracks for atmospheres, five or six mono tracks for Foley, and five or six more mono and stereo tracks for spot effects. Colour-coding these, if your DAW allows, can be a great help when it comes to mixing. The mixing stage becomes much more manageable if you bus these tracks together to subgroups for atmospheres, Foley and spot effects, and there's no harm in doing that now.

When it comes to Foley, footsteps are a good starting point, and it's crucial that the sound is believable. Although that means taking into account the shoe type, size of person, and the terrain on which they tread, it does not necessarily mean placing a sound for each individual footstep when there are several characters walking.With a project ready to roll, I find that the best approach is to build your sounds up in layers, rather like you might expect to approach an oil painting. Pretty much all sounds, other than music and dialogue, contributes to one of three layers: atmospheres (or 'atmos'), Foley and spot effects (SFX). The atmospheres create a sense of place, time and location, and are usually subtle signals that you hear pretty much subconsciously. The Foley is intended to add a sense of realism and believability to characters and their immediate surroundings. It includes things like footsteps and clothing noise. And spot-effects are exactly that — crowd-pleasing special effects that take centre stage. Let's take each layer in turn.

Atmospheres

I start every project by creating the atmos tracks. Atmosphere sound is crucial if we wish to immerse the viewer or listener into the world that the project aims to portray. Many films or games that are said to have a great 'atmosphere' owe a lot to this aspect of their sound design. Good atmosphere sound design should not be overly prominent. It is there to suggest the environment, rather than bombard the viewer with it.

Instead of working through in a piecemeal way, I usually start by sourcing the various sounds I think I'll want to use. You might find suitable recordings in commercial sound effects libraries, but remember that you can always record your own, or even get creative with synths. Atmos tracks should normally be recorded in stereo (at least), as mono atmospheres lack, well, atmosphere! Although in recent years surround-sound mics have become more popular for atmos tracks in some media — these can record the atmosphere of an environment in six channels or more, which is very handy for immersive surround mixes — a good stereo mic does the job perfectly well for most projects. The resulting stereo atmos track will provide some of the width in your final mix that your mono sounds, like dialogue and Foley, sit within.

With my atmos sounds chosen, I'll generally move through the piece and take note of any location changes, any time of day changes and any interior/exterior changes — I suggest using markers in your DAW for this. I'll then edit the material to make a consistent atmosphere track, the aim being to create a consistent-sounding atmos track that's neither too repetitive nor too busy. Remember, there are several layers of sound that need to play on top of the atmos. You're not looking for this sound to stand out: you're simply looking for a bed of sound that sets the tone in the audience's subconscious. Your edit will probably involve cutting out sections and crossfading them together to create the whole piece. You might find that some compression is required: you want to keep the atmospheres in their place, and if the track is too dynamic that could cause headaches later on when mixing.

You really do need to be organised with your sound-design DAW projects — start by setting up a template that reflects the working-in-layers approach and includes a marker track. Colour-coding can be a huge help in navigating your project as it grows.Depending on the nature of the atmos you're creating, you may want to punctuate the atmos track with the occasional 'single-shot' sound to add variety. For example, a horror theme may have a scary-sounding ominous undertone that's punctuated by occasional screams or wolf howls; an office ambience might be improved with the subtle placement of a printer whirring into life and so on. These single-shot sounds are invariably most effective when they're mixed against the main track and panned slightly to either side — this helps to complete the illusion that there's a larger world outside the one which can be seen on the screen.

As with everything in sound design, the atmospheres need to be mixed carefully if you're to avoid cacophony. In films, the atmosphere tracks will usually be mixed higher in level during the establishing shots at the beginning of a scene. Once the scene is established they'll fall back in order to give prominence to dialogue or other sounds. Scenes that are — from an audio perspective — sound-effect driven rather than music-driven, may also feature atmospheres in a more prominent position in the mix, since there's no danger of clashing with a music cue.

But just what sounds should you use? The answer to that, I'm afraid, is that you must draw on your experience and imagination. Watch films and animations, play games, listen to radio plays, and when you think something's good, pay particular attention to the sounds — try to separate out those layers, and hone in on the atmos sounds. In a sci-fi film set in space, for instance, you'll often hear common themes such as a deep subsonic rumble, indicating the hugeness of a planet or the vastness of the cosmos. Inside a building you might have something like the quiet rush of an air conditioning unit. Ask yourself what you might hear in a given location, and if the answer is nothing then, unless it's in the vacuum of space, think again. Can you hear heart beats? Or breathing? Consider making these things more prominent if it serves the creative aims of the project.

Foley

Foley (always with a capital F — it's named after a pioneer in this field, Jack Donovan Foley) is the term used to describe the seemingly insignificant sounds that give a sense of life and realism to characters and their immediate surroundings. This includes obvious sounds such as footsteps, but also clothing, and the interaction sounds between characters and objects, such as opening and closing doors. In fact, anything that a character does that could result in a sound can be considered the job of Foley.

The completed Foley track will tell us a great detail about each character. Their size, for example, the speed they move, and over what terrain (are the footsteps on snow, rock, gravel, sand or grass?), the clothes they wear and items on their person, such as jewellery, bags or weapons.

Foley can be very convincing, but getting it wrong can ruin a project. Good Foley means paying minute attention to detail, but it almost always sounds subtle: everything you hear when you watch the character should sound natural and believable, so that the audience remains immersed in the world of the film/game/play or whatever you're working on. Bad Foley, by contrast, breaks the spell. Footsteps may be mixed too loud or be of the wrong type, overzealous cloth or movement noise might prove distracting, and any sounds that are played too prominently in the mix can ruin the illusion.

Much has been written about the use of Foley, and about the Foley artists who specialise in creating it. You'll also find quite detailed documentaries on the subject. There's little point me regurgitating all that here — just remember that you can be as creative as you like with the source of your sounds (the Wikipedia page about Foley lists some possibilities to get you thinking along the right lines). As long as it's believable when it's played back alongside the picture, you've got it right. What I will do, though, is explain how to work effectively with the sounds you've either recorded yourself, or sourced from a sound-effects library.

With the atmos tracks in place, I'll usually start my Foley work with footsteps, because they immediately give the character a sense of weight in the world, helping them to sound more attached to the surface they're walking on. This is particularly important in animations or video games, where the character has essentially been animated outside of the world and added to it afterwards.

Consider the direction a character has come from before they entered the scene; let's hear them before we see them. Consider their weight, their shoes (if worn) and the surface they're walking on when you're choosing sounds to use. You may need to cut a recording you've made or sourced into individual footsteps so that you can scrub through the visuals and apply one to each visual foot fall. Often, you can't always see the actor's or animated character's feet — but they'll still be making sounds, so guess the position by what you can see on screen. Make sure it sounds natural, though, with a rhythm that makes sense against the visuals. For added realism, don't forget to add the details such as scuffs, changes of direction, and the character shifting weight from one foot to the other. Pan and mix according to the character's position in relation to the camera — but please be subtle with that pan pot!

As I touched on earlier, when working with a group of characters (three people or more) sometimes a more general footstep sound, comprising several sounds in one clip, will effectively convey the sound of a group moving. It's not always necessary to laboriously sync up the foot falls of every character if a general sound will suffice.

The next layer of realism usually comes from a character's clothes. The addition of clothing sounds is a must if you're aiming for a realistic feel — but good Foley must never be the audience's main focus! The occasional quiet squeak of a leather jacket, for example, can be enough to make an audience believe they're hearing what they see.Foley cloth sounds, which represent the movement of the character's clothes as they move around, add greatly to the realism of any sound design piece. Subtlety is often key here though, just a suggestion of leather movement is enough to convey a character wearing a leather jacket, for example. When combined with the footsteps, creaking of chairs or sofas and other such subtle sounds, you'll find that the character really starts to come alive in the scene.

Effective mixing is crucial to getting the Foley sounds to sit naturally. It should underpin the movement of characters but never distract or stand out too prominently against other sounds. In essence, if the Foley you have created stands out and sounds like, well, Foley, then it is probably too loud or fails to blend into the world effectively! It may be necessary to add a touch of reverb to your whole Foley mix; this will help it all to gel together, which can be very useful.

Use highly detailed sounds that are edited well, with clean beginnings and ends — make sure there are no pops or clicks, because if a sound seems 'edited' it won't work effectively and the illusion will be broken. Foley engineers often record sounds such as footsteps with a condenser mic, placed at close proximity to the Foley artist. This ensures a detailed sound is captured. Good-quality sound-library Foley will sometimes feature different distance and angle perspectives to choose from as well.

Make sure you don't miss any sounds out, as their absence will stand out as much as something that's mixed too high in level. For example, you may have worked laboriously and produced some marvellous Foley on your principle character, but don't forget about the other bit-part characters that appear in the piece.

Spot Effects

Spot effects may get all the attention, but that's because they're crowd pleasers, and are thus given the spotlight. Explosions, crashes and special sound-effects — for example the sounds of spacecraft, futuristic weapons or fantasy creatures — all fall into this category. The techniques used to create such effects don't really differ that much from those used for Foley — you can start with recorded audio, record your own, or think about using synths, or combine one or more of those techniques to create a more complex sound, comprising several layers. I explored these sorts of techniques in a couple of articles last year, which you can find at http://sosm.ag/designersounds-1 and http://sosm.ag/designersounds-2 so if you're in search of more guidance, do check those out.

Because they're placed right up front, these sounds usually demand most of your time and they play an important role in telling the story. In fact, an argument could be made for working on the spot effects first, especially if you are short of time, but I usually find that my best work is done when I have the fundamental layers of atmos and Foley in place already: the sonic backdrop just makes it easier to create better spot effects.

Breaking With Convention

In the first of the articles I wrote last year, I explained how contrast and silence can be used to great effect in creative sound design. You need to keep such devices in mind when you're working your way through the three sonic layers I've just described, so let's explore some more, starting with the use of perspective. The usual convention when creating sound design for picture is to create sound that represents the imagery that's directly visible on screen — we see a door close abruptly and we hear the accompanying slam sound from the perspective of the camera. Sometimes, though, the most interesting sound design defies this convention. Study Disney/Pixar's 2007 animation Ratatouille for some great examples of using sound design to emphasise character perspective.In the 2007 Disney/Pixar animated movie Ratatouille, the main character, a rat called Remy, is often placed in situations where the sound designers use his diminutive size as a focus. For example, in one kitchen scene Remy is scurrying across the floor, and each of the obstacles he meets features sound design that enhances its scale, making everything seem larger and more ominous: we hear exaggerated interpretations of meal trolleys passing by, with wheels that squeak and grind menacingly. The detail in the sound subtly draws the audience closer to the action. In another example, the burners of a gas oven fire into life as Remy runs underneath. To Remy, these burners sound like an inferno, so the design exaggerates the fiery sound to great effect. This focus on sound from the perspective of Remy himself helps to characterise him further as a small character in a large and sometimes dangerous world, which really enhances the story.

We can contrast the visuals with the sound in even more dramatic ways. The establishing scene to the movie Michael Clayton (2007) breaks sound-design conventions to great effect. In Michael Clayton, the opening scene features an audio monologue delivered by an unseen character, set to shots of a law firm at night, and to music that slowly escalates in intensity. Night-shift cleaners vacuum the floors and a trolley is pushed through empty corridors before entering a busy boardroom, where a hive of activity is revealed. Next we hear a phone conversation, both voices are audible but we don't immediately see who is taking the call. The camera slowly pans forward until, eventually, it focuses on a character. From there, it becomes apparent who he is and what the call entails. This is a great example of sound design breaking convention. Voices are heard in unrealistic ways and perspectives are skewed, but this great piece of sound design contributes directly to the dramatic and engaging opening of this movie.

Another creative option is to play around with the audience expectation of where sound will change in relation to the picture. Cutting sound in time with visual edits seems logical, but toying with this convention — introducing sounds early or letting sounds play over a visual edit for a few seconds — can be extremely effective. You often see this when a character opens a door on a vehicle or building, we hear the handle turn and the door swing open before we see it, so that when we cut to that scene the door is already open, and perhaps dialogue commences. This type of editing gives the sound a moment to be introduced before it ends, so that the audience can register it. The same trick can be used with atmosphere crossfades, introducing a new feel a couple of seconds before the new location is seen. Your most creative work will come from playing with ideas like this to keep the audience engaged.