Celemony Melodyne

Melodyne's main Arrange page, with three audio tracks in it. The top one is un‑Detected, which is why it's not showing separate pitches like the others. The shadowy blue stripes in the background provide a quarter‑note timing 'grid'. You can also see the mixer interface, which has no track labelling, making it hard to know what you're muting, soloing and panning. However, labelling is on the way for the next version.

Few recent software products have stirred as much interest as Melodyne — a program offering unique pitch, time and formant‑processing features, which was created not by one of the major music developers but by a maverick Bavarian musician and his software‑designer friend.

The small team behind Melodyne first came to the attention of the hi‑tech music world early this year, when they attended the Los Angeles NAMM show to publicise their high‑quality audio processing and manipulation software, which is based on some unusual and rather secret technology. Less than a year on, Melodyne has reached release level.

Capabilities

The melody after Detection. Notation is overlaid on the waveform, which now shows different pitches. You can choose to view the notation under the waveform, if preferred.

Melodyne is currently a stand‑alone application for Mac (a PC version is coming), and its purpose in life is to analyse audio minutely and then process it such that almost everything about it can be altered at will. It can shift the pitch of monophonic audio without changing duration, change duration or tempo with or without changing pitch, and manipulate formants to alter voice or instrument character or to improve the character of shifted audio. It allows you to create harmonies from one vocal, or indeed construct whole new melodies, as well as making vocals and instruments do things they couldn't normally do. It can force audio of different tempos to run at one tempo, quantise audio timing, correct pitching errors across an entire track or one note at a time, superimpose pitch information from one melody onto another without affecting lyrics, alter amplitude, edit transitions between notes, and even alter vibrato. While it usually requires monophonic audio, it works on drum and percussion loops and will also process strummed guitar. Sounds impressive? It does to us...

Software & Hardware Setup

The Detect Melody dialogue, showing some of the specialised Detection routines for processing different types of material.

Installing Melodyne — which is compatible with any Mac OS above 9.0.4, including Mac OS X — is simple. Apple's CarbonLib 1.3.1 extension is required, but if you don't have it, it's a quick download from Apple's web site. Also, Melodyne wouldn't run for us until we enabled the Apple Enet extension.

MIDI isn't implemented in this release, so there's no OMS to set up, and Melodyne doesn't even have an ASIO driver folder: all you need do is tell the software the location of any ASIO driver in your system and it can then employ ASIO‑compatible hardware's inputs or outputs. Sound Manager is supported, too, for use with the Mac's built‑in hardware. The program CD contains a PDF manual — happily, in addition to a printed manual — and two folders of tutorial and demo files.

Recording & Importing Audio

The Edit Definition window showing the same melody. The Detected notation runs along the top of the screen. Overlaid on the waveforms here are representations of vibrato (the wobbly line) and transitions (seen most clearly between the first and second notes). The pale grey vertical lines appear when the Note Separation tool is chosen and show where Melodyne has divided the waveform into individual notes. You can edit these separations.

Booting up Melodyne takes a while compared to other music applications, and we noticed a similar sluggishness elsewhere in the program, including when saving files. The first thing you see is the Arrange Window, which is roughly equivalent to the Arrange page of a mainstream sequencer. You can record audio directly into here, or import it. Celemony quote a typical maximum track count of 20 to 30, depending on your Mac. Our G4 managed 20, though on‑screen controls began responding rather slowly at this point. We're told that Melodyne isn't currently optimised for song‑length tracks, but we managed to work with long sections of music.

If audio is to be recorded (at 16‑ or 24‑bit, with sampling rate depending on your hardware), Melodyne supplies a click, but since it is always at a fixed, rather loud, level it can be annoying. The best plan may be to import a suitable drum part and play along with that. However, anyone who hates playing to a click will be happy to hear that Melodyne allows a tempo to be assigned to freely recorded audio later. This sort of flexibility is at the heart of the program.

Actually recording with Melodyne is inconvenient, at least with this first release, as there's no play‑through, so you can't monitor what's going to disk. Our solution involved patching the headphone output of our Digi 001 hardware into an analogue mixer alongside the audio to be recorded, so that the latter could be monitored during overdubbing. A direct out from the audio's mixer channel routed it to a 001 input, extending the signal path unnecessarily. A similar dodge was required with Sound Manager. However, recording was always successful.

Importing audio is rather easier. Whatever the type of audio, not only should it be monophonic, it should also be as dry as possible. Even small amounts of reverb can render audio unusable for certain processes, since reverb essentially provides an aural simulation of polyphony.

Detecting Melody

The Editor window for the same melody. This is where you change audio pitch and duration, and edit vibrato, transitions and amplitude. The blue rectangles behind the notes show the nearest semitone for pitch correction. There's a small display at the top of the screen that can tell you how far out the notes are.

Melodyne does some preliminary analysis on every new 'melody' (Celemony's term for the contents of a track — which could equally be a speech or percussion recording), and has a stab, which is quite often 100 percent accurate, at determining its tempo. The next step prepares the audio for further manipulation. Choosing Detect Melody from the Define Melody pop‑up launches a window offering a number of parameters. Usually, you can leave these alone, as Melodyne functions quite well with the default settings. Exceptions to the rule are speech and percussion: it's best to select their specific Detection options. There are settings for many instrument and vocal types, which define lowest and highest notes detected, although note‑range choices can be overridden. (There's a good practical reason for limiting note ranges: if what's being detected suffers from spill, limiting note range should mean that unwanted audio won't confuse the process.) A tempo‑detection parameter offers two choices. Present Tempo uses the overall Arrangement's tempo, while Own Tempo derives tempo from the audio being detected. We preferred to use Own Tempo and then match that to the Arrangement tempo, though either option can produce satisfactory results, since there are post‑detection tools that allow further manipulation and matching of tempos.

The manual advises that you tell Melodyne what tonality you'll be working in, though we did pretty well without this step. You can choose from the 12 equal‑tempered keys, or from a range of non‑equal‑tempered tone schemes (Pythagorean, mean‑tone and pure ratios), or create a custom tuning. The tonality chosen can affect what notes are detected, and also how far the software moves notes when correcting pitches. The fact that the overall pitch reference can be offset, in cents or Hertz values, allows you to account for sound sources not at concert pitch.

Detection should take approximately as long as the audio itself, depending on computer power and, to some extent, the complexity of the audio. Once the audio is processed, it's possible to view the notes detected in the melody superimposed in staff notation over the waveform. This will be your first indication of whether Detection has been successful. Prepare to be impressed: for the most part, it will have been.

If speech or percussion has been Detected, the note overlay represents the rhythm of the audio only. And on the subject of rhythmic material, one consequence of Melodyne's ability to work out the tempos of drum parts, and then allow them to be manipulated, is that it's rather easy to match loops with wildly varying tempos. One tempo is always the main reference for the Arrangement, but there are tools for instantly matching foreign tempos to that reference — and then you can change the tempo of the whole Arrangement. There are methods for tempo‑matching in other software, not to mention certain samplers and Roland's VP9000, but it happens here with much less in the way of artifacts, at least to our ears.

Editing The Melody Definition

Here we've dragged out the wobbly vibrato line to remove vibrato from this note altogether.

Perhaps it would be too much to expect, but the results of the Detect Melody routine are seldom perfect. Its 80‑90 percent accuracy is amazing enough — the melody of what you've sung or played into Melodyne will be clearly identifiable in the note display — but almost because the process works so well you'll immediately expect every nuance and grace note to be represented. If they're not, it's time to get your hands dirty in the Edit Definition window (shown below). This window organises Detected audio on a grid, where you can see individual note relationships against a vertical piano keyboard, with the score‑based transcription running across the top. Here, with special tools, you can separate out notes that Melodyne hasn't caught, insert or correct bar lines (which also automatically corrects the audio file's Melodyne 'tempo map', useful for freely recorded audio), and fix mis‑detected notes. Besides missed notes, common problems include notes detected correctly but placed in the wrong octave, and coughs and breaths detected as notes.

During editing, it becomes apparent that Melodyne may actually have detected notes that it appears to have missed. In the Edit Definition window, a 'phrasing' graph is superimposed on the note waveforms. Phrasing in this case indicates vibrato, swoops and falls, and also pitch deviations that Melodyne doesn't consider significant enough to count as separate notes. However, it's these deviations that you should look for first when editing a melody: using the Separation tool on or near such a deviation nearly always separates out a note that was missed during detection. It's as if Melodyne wasn't sure, but as soon as you nudge it by clicking on an obvious note, it knows what you mean.

Depending on your perspective, you could find the Definition editing process either annoying and fiddly or quite satisfying, but it is essential, because the accuracy of the Melody Definition has repercussions for what you will be asking Melodyne to do. If you haven't separated all the notes from each other, for example, they won't behave properly if tempo is altered later. The same goes for the placing of bar lines in a performance without a steady tempo.

Celemony know that the detection process doesn't always detect every note perfectly. We suggested that they consider a 'sensitivity' option in the Melody Detection window, to get Melodyne to work harder and pick up fractional pitch/rhythm values more reliably. Celemony's Carsten Gehle responded: "It is a general problem that a 'sensitivity control' would not be a linear parameter here, like 'less or more', but a whole set of parameters. We found that it is faster for the user to correct some separations in the Edit Definition window than to specify a set of detection parameters and still never be sure of getting 100 percent correct note separations. It's often very obvious where note separations are, but only if you know the music and the context. Generally, we would like the user to consider it a necessary step to edit the Melody Definition after detection."

Closing the Edit Melody window causes a Save Melody Definition? box to appear; clicking Save returns you to the Arrange window and assigns your tweaks to the melody — or so you'd think. What actually happens is that no changes appear to have been made, and the melody still appears as originally Detected. This is because you need to do a couple more things to make these changes update, and you're not told of them if you work through the manual tutorial. First, you have to choose Reset to Definition in the Define Melody pop‑up. Then you select Tempo from Melody in the Edit Score Time tool's Edit Tempo pop‑up if the new tempo is to be used for the whole Arrangement. Alternatively, you use Adapt Time if you want the Arrangement's tempo to override that of the Detected audio. This process speeds up or slows down the audio so it fits its correct number of bars at the new tempo.

It's easy to get confused by the Edit Definition window, especially since the manual fails to explain the software properly. This, in conjuction with a certain lack of streamlining in the interface, makes Melodyne seem more daunting than it really is. It's worth remembering that, whatever you do to the audio in the Arrange or Editor windows, in the Edit Definition window you'll always see the original Detected (and manually corrected) melody. The hierarchy seems to be: audio file, which is always untouched; Melody Definition, which determines how pitch, expression and timing information are interpreted; and a third level in which the melody is manipulated in the Arrange or Editor windows.

The Editor Window

We've pitch‑corrected these notes and they now appear centred in the blue rectangles.

If the Edit Definition window is for corrective processing, creative activity starts in the Editor window (above), which is something like the individual track editor in a MIDI + Audio sequencer. Pitch information, rhythm, formant response, expression, articulation and velocity are available for manipulation here. You're able to specify extreme values well beyond the bounds of sensible usability — pitch and formant shifts of several thousand cents, for example. It's even possible to drag the tempo right down to 0bpm, at which point the Local Sound Synthesis at the heart of Melodyne resynthesizes indefinitely a single sonic point in time. The result sounds like an infinite version of the note(s) being played at that point, with no artifacts.

'Expression' includes such aspects of a performance as note transition, vibrato and swoops/falls. It's an eerie experience to tweak vibrato as simply as if it were an LFO control: you can literally remove vibrato from a vocal, or crank it up to serious pitch‑modulation levels. Problems caused by vibrato changing speed when audio is stretched or time‑compressed can be mitigated in Melodyne by editing vibrato depth (you can't change speed) until the result sounds natural — a feature unique to the program, as far as we know. Even Roland's VP9000 advanced vocal processor can't deal with vibrato, while TC's VoicePrism Plus (reviewed in this issue) can add it, but can't remove existing vibrato. Melodyne's ability to tweak a swooped transition between two sung notes is also shared only by the VoicePrism Plus. We weren't able to compare the two, but Melodyne's implementation is very powerful, creating convincing new transitions between original notes in a melody and ones that you've moved to create a new tune.

One obvious question to ask about a program like Melodyne is whether it can do automatic pitch correction, like Antares' Auto‑Tune plug‑in and hardware processor. The answer is yes, via a simple three‑step process: highlight the note or notes you want to correct, choose Note Snap in the Quantise pop‑up, then click on any note in the highlighted selection. Highlighted notes snap to the nearest semitone.

Even timing quantisation of audio is possible, with resolution of between a bar and a 32nd note (there are no triplet options yet). You highlight the notes to be quantised with the Move Notes tool active, select the quantise value and click on any note in the highlighted group, whereupon all events move to the nearest quantise position. It works very well. Unfortunately, the background display's quarter‑note resolution can't be changed.

Formant offsets, applied at note or melody level, let you make extreme pitch‑shifts sound more natural. For example, if you shift a vocal an octave down and it doesn't sound completely natural, you can choose the formant manipulation tool and drag intuitive on‑screen bars to alter formant response. Surprisingly little tweaking produces the best results. Such a shifted vocal won't always stand in isolation, but when creating harmonies, for example, it's ideal.

The Arrange Window

The same notes with their formant bars showing. Drag them up or down to change formant position for the note.

The Arrange Window is the place to create multitrack Arrangements and do limited cut, copy and paste editing. Here you can build up a piece from loops, snippets or sections of takes, and re‑use existing audio to create new material for harmonies, counter‑melodies and so on. One minor irritation is that you can't highlight audio across several tracks.

You can also access a Temporary Play Offsets window in the Arrange page, to quickly create pitch and formant offsets and tempo changes for the whole Arrangement. If you produce a desirable result, it's possible to 'Fixate' (Melodyne's term) the settings to the Arrangement.

One unusual feature in the Edit menu is Paste Notes, which magically applies pitch data extracted from one melody onto another. This seems to work best when source and target have roughly the same rhythm or number of syllables, and could be used for copying harmonies created in Melodyne from one section of a song to another, where the tune is the same but the lyrics are different. We successfully pasted a tune onto a recording of a held synth note, after first 'separating' the synth note into 16th notes in the Edit Definition window. (This process had no audible effect on the note's playback.) Pasting the pitches of another melody onto the note then caused it to play back with those pitches — very clever stuff. A remarkable example of what can be achieved with this function is provided in the audio demos: the melodies of a six‑part piece of Bavarian wind music used during the manual tutorial have been imposed on a sloppily recorded vocal version of the same piece.

Balancing tracks in an Arrangement is done with the basic mixer that's also used during recording. It's a flexible enough device, and grows with each new track that you add. Unfortunately, it'll grow right off your screen if you let it, and there are no scroll bars to help — you need to keep dragging it back and forth. Graphically, the mixer is very plain and is one of the more unfinished aspects of Melodyne. There's no track numbering, little labelling, controls are hard to move with the mouse, and the metering is inadequate. VST plug‑in support isn't available in this version of Melodyne, so the only effect you can apply is a simple, uneditable reverb.

In Use

It took us several days to even begin to feel familiar with the way Melodyne's user interface works. Its audio‑processing powers are extremely impressive (many of them are unique), and once you grasp the basics you can achieve great results without knowing all there is to know about the software, but it's not the easiest program to learn.

A major source of annoyance is the fact that there's a slightly different set of tools for each window, and each tool has its own buttons and pop‑ups, which change when a new tool is selected. So if you're looking for a function you know you've come across somewhere, it's a case of searching through windows, clicking on all their tools and trying to spot the option you're after. Why not leave all the options for a particular window on display, but greyed out if the right tool is not currently selected? Your impulse might be to look up the elusive function in the manual, but you'll be hindered by a lack of consistency in nomenclature: tools have one name in the program, and one, two or three others (some of which are long‑winded and confusing) in the manual. And there's no index.

Other things that bothered us were the difficulty of recording audio conveniently, and the lack of undo. This first release of Melodyne also misses several key features that were promised, namely an ASIO pipeline to compatible applications, and VST plug‑in support. Most users will probably prefer to record in a dedicated recording application and import audio into Melodyne. Celemony don't see the program trying to rival audio‑recording applications anyway. Audio import and export, in WAV, AIFF, SDII or SND formats, is easy, though we once or twice found unexplained gaps inserted at the start of audio exported from Melodyne. Also, the occasional exported file turned out to be stereo white noise. Even more oddly, the same files re‑imported just fine into Melodyne.

Missing features to be introduced by the imminent, and free, v1.1 upgrade include ASIO monitoring, an improved click, improved (faster) handling of longer recordings, and multiple undo — all good news. Perhaps Celemony can also look at one or two odd, fleeting bugs, whereby tools or program elements don't appear when they should, or appear in the wrong form. It can also be disconcerting that the zoom levels and song position you've set are reset whenever you leave and then return to the Editor Window.

To put our comments here in context, Melodyne is still an amazing achievement from a small, new company, and if they listen to what users tell them about their experience of the software it will rapidly become more streamlined. In the meantime, no‑one should be deterred from trying it just because it requires a bit of head‑scratching!

Conclusion

What you can do with this program is amazing. Extreme changes to essential elements of all sorts of audio can be done smoothly and instantly. As always, you should try it for yourself before buying, but Melodyne seems to us to offer unprecedented levels of transparency, with minimal processing artifacts. Loops and snips of audio work together immediately, sopranos become tenors and vice versa (with some work on formants), tunes can be convincingly created from practically any source material. At the end of the review period, just for interest, we randomly threw together 13 tracks of disparate bits of Detected audio we'd been working with — a lead vocal and some 'created' harmonies, strummed guitar, a bass part (modified with Melodyne's help to play a different line), some sampled speech, four thoroughly unrelated drum loops, and a simple piano line — used the Adapt Time function so that they ran at the same speed, and it sounded fantastic. Weird, but fantastic. And armed with our edited Melody Definitions, we could then do pretty much anything with those audio elements.

Melodyne costs more than Auto‑Tune, but then it does vastly more than Auto‑Tune (which itself is a great tool for its purposes). However, it costs a lot less than either Roland's VP9000 or TC's VoicePrism Plus, which come closest to sharing its unique functionality. Even then, neither unit does everything Melodyne does (although the VoicePrism Plus's modelling of vocal characteristics such as rasp and growl is exclusive to it). This really is unique software, with numerous applications in composition, recording, remixing, even broadcast and post‑production (its speech handling has to be heard to be believed). It's not cheap, and its user interface needs polishing, but it's a truly revolutionary creative musical tool, and we don't get so many of those these days.