A Practical Guide To Working With Pictures: Part 1

The simple audio‑for‑picture setup of SOS reader Martin Bushell, as featured in Readerzone December 1999. For many applications, Martin's approach of simply playing the video and sequencer manually is perfectly adequate, and there is no need to sync one to the other.

With the growth of digital television and affordable solutions for recording audio to picture, many musicians are keen to get into writing and recording for visual media. In this introduction to the series, Hugh Robjohns investigates basic working methods and suggests some system requirements. This is the first article in a three‑part series.

If you write, record or mix music, it's quite likely that at some point you will want to do it in conjunction with video, computer animation or other moving pictures — and this means addressing the thorny topics of synchronisation and timecode. This article is intended to provide an overview of some of the requirements and practicalities involved in working to pictures, but over the coming months we will also examine some case studies with typical working examples of some of these approaches to see just what is possible, what equipment is required, and how it can be set up to provide the best results.

The Basics

As this example from Nathan McCree and Matt Kemp's soundtrack for Tomb Raider III shows, synchronising audio clips to specific timecode hits in the picture is made easier with software like Sonic Foundry's Sound Forge PC audio editor.

There are many ways of applying sound to pictures (I will concentrate on music here, rather than consider the complexities of dialogue and effects), and the best approach depends on what you want to achieve and how your music will be used. It is perfectly possible to get excellent results with some extremely simple setups. The most sophisticated sound‑for‑picture installations, which are required for frame‑accurate timing, are necessarily complex, but now cost a fraction of what they did a decade ago. There is also a range of intermediate stages between these two extremes, with numerous equipment options and plenty of upgrade paths to follow as your requirements and abilities develop.

The first thing to consider is just how the music is required to relate to the pictures: how tight and accurate does the synchronisation have to be? In many applications, particularly for background music, the requirement is only that the piece should run to an approximate duration — say 29 seconds. There are no specific 'sync points' to hit with a musical effect other than the start and end.

Of course, it helps enormously if you can see the pictures for which your music is intended — partly to get the right feel and mood as you are writing the piece, but also to check the suitability and accuracy of the completed work. This is not essential, and many writers prefer to write 'blind', but it is not usually very difficult to arrange to have a video player and TV in a corner of the studio to view alongside your music system. The fact that the music does not have a precise sync point with the pictures means that hitting the play button on your recorder or sequencer when you see the appropriate picture frame is often good enough to check the results.

Another scenario where this kind of arrangement is perfectly adequate is, strangely, where the requirement is for extremely accurate synchronisation between pictures and sound. Where a strongly rhythmic musical piece is used it is usually necessary to have the picture cuts timed precisely to the beat. Trying to write music to fit ready‑edited pictures would be a complete nightmare, so in this application, as before, it is only necessary to write music to a specified length and tempo. The picture editor will then cut the pictures to fit the music as necessary after the music has been copied into the video editing system.

In both of these scenarios, all that is needed in addition to your existing audio setup is a suitable video player — a domestic VHS machine will suffice quite adequately — and a television. It would be wise to be careful where you put the TV as the magnetic field from the screen can cause interference on electric guitars and some microphones, and the acoustic noise from the line scanning at 15.625kHz is often picked up by any live mics in the room. It is not uncommon for other electrical interference from the TV to find its way into mixers, amplifiers and signal processors too, so listen carefully when you switch it all on. Try using different mains sockets from the audio equipment, and position it all as far away as possible.

An alternative solution, which is becoming a lot more practical now that hard drives are so large and relatively cheap, is to copy the original video into a computer using a video grabber card and something like the QuickTime system. The picture quality may be relatively poor, but it has the advantage of instant cueing, no need to clutter up the studio with extra equipment and the ability to control the video and audio from the same computer screen — often with reasonably accurate synchronisation too.

Ideally, the original video tape of the pictures you are working with should be supplied with the appropriate timecode visible as a time display on the screen (usually referred to as burnt‑in timecode, or timecode‑in‑vision). This will help in the accurate identification of scenes and for repeatable cueing of the tape. One of the audio tracks of the video will probably have the timecode as an audio signal and the other a rough (mono) mix of whatever live‑action sound may be associated with the pictures, perhaps consisting of dialogue and some effects. It could be useful to arrange for the audio premix track to be fed through your studio monitoring, but beware of getting any crosstalk from this into your music recording! Also watch out for timecode breaking through — the powerful square‑wave signals that make up timecode are pitched in the ear's most sensitive region and have a habit of getting everywhere!

With this kind of equipment setup, all you have to do to check the accuracy and suitability of your music is simply to start it replaying when you see the appropriate timecode number on the screen. The steadily incrementing frame counts make it easy to anticipate the precise start point, and with a little practice you should find you can fire off the audio on the exact second. When you are completely happy with your composition and mix, all you have to do is copy it to a suitable delivery medium (DAT, CD‑R, ADAT, DTRS, or whatever — see 'Formatted For Dispatch' box, on page 134) and write a cue sheet stating where you think it should start and how long it runs. The editor or dubbing mixer will subsequently lay it against the pictures and you'll see your name in the credits, hopefully!

Making The Sync

MOTU's Digital Timepiece is a sophisticated synchroniser, capable not only of converting video timecode into MIDI timecode for sync'ing a sequencer, but also word clock for synchronising digital audio recorders.

The next stage up is where the sound and pictures have to remain precisely locked in synchronism because there are specific cue points which the music has to hit, and this is where things start to get a little more complex. The normal way to work is to make the video machine the 'master' and the audio equipment the 'slave'. This is principally because timecode — the synchronisation medium — counts video frames and so has to come from the video machine. However, this is not the only way to work, and there are professional video machines which can be slaved to timecode sourced from the sequencer or master recorder, which is often a neater and more flexible way of working. Equally, loading pictures into the computer and controlling them from the sequencer is another good solution — more on that later.

Back to the basic approach first, though, and if you have a VHS tape with timecode recorded on one of its audio tracks this could be used as the timecode master source, with your MIDI sequencing package configured to chase‑lock to the timecode — in other words to locate to the required timecode point and then run in synchronism with it. Alternatively, a timecode‑equipped audio recorder could chase the video timecode in the same way.

In days of old, separate timecode synchronisers were required to control audio recorders via their remote control ports, but these days manufacturers tend to build optimised chase synchronisers into their flagship machines, which makes life a great deal easier. Timecode DAT machines, DTRS recorders, MO recorders and MIDI synchronisers are all available with the ability to chase timecode directly.

However, the main drawback with this simple chase‑to‑video approach is that any wow and flutter from the video machine (and VHS recorders generally do not have the most stable transports in the universe) will result in speed and timing variations in the chase‑synchronised equipment locked to the timecode it is putting out. And that could mean an unstable and unsatisfactory recording!

Re‑chasing

Emagic Unitor 8 synchroniser.

There are several ways around this problem, depending on the flexibility of the equipment used. One way would be to take the timecode from the video and re‑clock it using a stable timecode generator. However, these are expensive boxes to have to add, and require a stable video reference feed too, so it would probably be more cost‑effective to purchase a second‑hand professional video player (like a discarded Betacam machine). A lot of timecode‑equipped audio machines are provided with 're‑chase' options where the timecode is used as a position guide rather than as a speed controller. This means that any wow and flutter in the timecode from the video player will not affect the speed of the audio, but that overall synchronisation will be maintained, though potentially with less accuracy. Typical re‑chase programs allow timecode errors of up to a second before they do something about it! In practice, this kind of setup works very well.

A typical arrangement might be to feed timecode from a video machine to a recorder such as the Tascam DA78 or Yamaha D24, set to re‑chase. These machines would then provide either re‑clocked timecode or an MTC signal to synchronise a MIDI sequencer. When working without pictures, the sequencer could be used to control the recorder transport via MIDI Machine Control, but when synchronised to the pictures it will be necessary to 'drive' the system from the video machine. This is often rather cumbersome and slow, but is normally effective and reliable.

As already mentioned, an alternative approach might be to transfer the source video and timecode to the video format within the computer, where it may be controlled from the sequencer with the advantages of instant access to any scene and often much faster sync lockup times. The down side is reduced picture quality (making it hard to read timecode numbers or to see specific action cues) and more clutter on the computer screen!

In next month's article we will take a closer look at the practicalities of setting up a basic audio‑for‑picture system and see what kind of results can be achieved.

Formatted For Dispatch

Once all the issues of synchronisation have been addressed and you are ready to record your music to match the pictures, you have to decide what format to record on to. This is something which should be agreed with the producers of the video, well in advance, and may require discussions with the picture editor and/or dubbing mixer.

Simple non‑sync work, such as background music, or strongly rhythmic music which will be laid before the pictures are cut to it may be supplied as a completed stereo mix on a non‑timecode medium such as CD‑R, DAT, or even a Minidisc. Supplying your greatest masterpiece on a reused cassette may not be the best way to enhance your credibility! Personally, I would avoid data‑reduced formats such as Minidisc if possible, simply because of the unpredictable interaction with subsequent signal processing.

However, for more complex work, it is far better to supply your material on an eight‑track format such as ADAT or, better still, DTRS (the Tascam DA88 format). Any decent dubbing house and post‑production facility will be able to work with DTRS tapes, though there are numerous other formats which may be suitable too, such as MO and hard‑disk formats from Akai, Digidesign, SADiE, Yamaha, Genex and others. One advantage of supplying your work on multitrack is that you can provide a discrete surround sound mix across six channels, along with a straight stereo mix on the last two. Alternatively, you could provide submixes of the various sections of your music (percussion, strings, lead or whatever) so that the final music balance can be performed during the dubbing process, allowing the music to be blended with the live‑action dialogue and sound effects — dubbing mixers often appreciate the flexibility of this approach.

The DTRS format has become something of a de facto standard in the video post‑production world and would be my preferred mastering format for this kind of work, especially now that 24‑bit resolution is available. Of course, timecode is an inherent part of the DTRS format and with the latest generation of machine (the DA78, shown above; see review on page 122), comprehensive timecode and MIDI synchronisation facilities are built‑in, allowing easy interfacing with both a video source and the MIDI sequencing system without breaking the bank.