Electronic Sound & Pictures: Producing Interactive Multimedia

Though many of us still don't fully understand it, interactive media is the way of the future for music and associated images. Nigel Humberstone visited ESP, a company with a long history in what is still a relatively young industry, to find out what it's all about...

For a design and production company, the Cambridge‑based Electronic Sound and Pictures (ESP) have received an awful lot of media interest, with articles appearing in magazines like The Face and Vox. This level of attention is largely due to the pioneering work they're undertaking in the arena of interactive music products, namely CD‑i. Currently The Orb, Stereo MCs, Depeche Mode, Erasure and the Shamen all have ESP‑designed interactive products set for release later this year, with numerous other 'hot' projects in development.

Described as "a collection of young artists, designers, musicians, programmers and misfits", ESP are at the forefront of the new technology, led by their forthright managing director Graham Browne‑Martin. As a spokesperson for CD‑i, Browne‑Martin has been called upon to give various lectures, demonstrations and talks on the subject with a recent keynote address at MIDEM.

Interactivity

Interactive music and the ability to manipulate the means of entertainment is still in its infancy, and the term CD‑i is essentially a generic one used to describe more than one particular format.

"There is an issue here about interactivity," extols Browne‑Martin. "Should a machine do everything you tell it to do, where the machine is acting as a slave? That level of interactivity is fairly boring — it's like me having a conversation with you and you agreeing with everything I say, because interaction occurs when there's a debate and discussion.

"It's funny because although it's (CD‑i) a young industry, we all meet at conferences and discuss the 'rules'. There's always talk that it should be intuitive and predictable, which is fine if you're designing a word processor, but when you're designing a piece of entertainment it's stupid to have rules. All good pieces of music and film have been when someone has stepped forward and broken the rules."

A vivid example of this is ESP's interactive work with controversial US industrial band Nine Inch Nails. Browne‑Martin obviously relishes the experience of not only playing with the CD‑i but also demonstrating it: "The Nine Inch Nails CD‑i is definitely not user friendly," shouts Browne‑Martin as he grapples with the CD‑i remote in a vain attempt to maintain control of the random images and 'play loud' soundtrack featured on this particular demo. "It's user‑hostile, but that's the point — the mixture of anger, hostility and sexuality all relate to the character of Trent Reznor and his band."

Using a process described as 'Technological Terrorism', the NIN CD‑i is so unpredictable that in addition to randomly denying you interactive control, it may also suddenly crash the system and leave a virus just for good measure.

New Format

With CD‑i still in its infancy, I asked whether ESP found themselves having to prove the new format to potential clients, or whether clients instead came to them.

"It's a mixture of both — in the early days when we started the company (1987), we approached record companies but didn't get very far, mainly because at the time there was nothing to play the discs on, which is self‑explanatory, really! But during the last couple of years we've really switched the heat on with the music industry, within which there's been a general change of perspective. They are now begining to understand that they've got to do something, in terms of not only recapturing a lost generation, but also in restructuring and re‑inventing themselves so that they can continue to sell records into the next century.

"So we're doing a mixture of educating the different parts of the industry and creating products. But we're only a small company of 25 people, although we're expanding gradually. The situation with CD‑i is very reminiscent of the early promo video days, and of course when that started, nobody knew it would become big business. We could become a next generation record company or we could just remain a design and production unit. I guess I have interests in both, really, and there's certainly a vibe going on about the interactive music side — however, there's also a lot of hype, which is a bit worrying because there's still a distinct lack of product available."

Browne‑Martin is referring to both software titles and units to play the discs on. Recent figures from Philips show that they have sold somewhere in the region of 300,000 CD‑i players worldwide. On top of that, there are something like 1,000,000 MPC players (personal PCs fitted with a CD‑ROM and soundcard), around 500,000 Sega CD Drives and about 50,000 3DO players.

"We don't look at it like one format; we don't support just one format and the market isn't like the Beta/VHS thing — it's more like with video games, where you've always had multiple formats. Worldwide, we can look at two million installed systems, which makes it a viable publishing market. But it's still very much early days."

The Early Days

Browne‑Martin's relationship with leading edge technology has been long term, but when exactly was it that he first saw the potential of CD‑i?

"Well, very early on. I started off as a musician, then got involved in the computer industry at an early stage, never really thinking of myself as a computer person, just seeing it as a way to earn money to buy all these instruments. I began writing games software in the early '80s — part of the 'first wave', really — and started working for a computer company, where I got into messing around with interactive media — plugging in video disc players (laser discs) and making the computers control them. Then in about 1986 CD‑ROM happened, and whereas most companies only saw it as an information thing, I saw that you could do much more with it and that it was cheap to press. I had access to a lot of technology at the time, and was involved with a lot of 'under the bench' work in the evenings — messing around." Subsequently Browne‑Martin was also lucky enough to visit Japan, Holland and the USA in order to see new developments, at a time when, as he puts it "CD‑i was just a twinkle in Philips' eye."

The eventual launch of CD‑i in 1991/92 was late into the market; Browne‑Martin had seen it originally in Philips' laboratories way back in 1986 and had thought then that the best way of selling the concept was as a music machine. For him it was the ideal opportunity to combine all his varied interests in computers, music, television and video games.

"I had set up this company (ESP) in 1987 to design and produce CD‑i in anticipation of its release in 1988. When that was delayed, we did consultancy for computer companies and others like Sony, Apple, Philips and Goldstar, building up our knowledge. Philips initially thought of CD‑i as an information appliance, but we saw it in other ways. I think people are more inclined to invest in new technology if they're going to be entertained by it. And I suppose we've been abusing the technology, making it do things it wasn't supposed to do and taking it into different areas."

One particular breakthrough is the ESP patent on continuous and uninterrupted playback — a feature that most users would take for granted, but one that CD‑i was not originally capable of providing. So how did ESP achieve this?

"Our first version of interactive software kept stopping the music when you went for information, which was pretty annoying, because the interactive experience really needs to be a seamless one. So we set about a new way of doing it."

Browne‑Martin is understandably cagey about the commercial advantage that continuous playback presents, and therefore not very forthcoming with technical explanations. The change was made in software because ESP did not want to get involved with hardware modifications when they don't actually endorse any particular hardware platform. Not easily deterred, I pushed for more clarification.

"It's magic — I can't spell it out!" smirks Browne‑Martin. "It's a trick that we've achieved through 'interleaving' information. There's also some extremely fast executing and type code. The raw clues are that CD‑i has an operating system called RTOS (Real Time Operating System), which is based around RS9. Now inside a CD‑i player is a 68000‑based computer, the equivalent of a Macintosh Plus, and traditionally software engineers are advised and encouraged by Philips to write all their code through that operating system. But because our engineers are real 'wire heads', they go straight to the metal and program straight to the machine — which is what you would always do in the video game industry. So you have to write in machine code and that was a technique that wasn't really used in interactive CD production. A lot of our engineers come from the video game industry and that's why this works, because we've approached it from a different angle — laterally, rather than doing what we're told to do. I suppose it's just a natural rebelliousness!

"What we've done with ESP is to combine a mixture of different people; artists, graphic designers, TV producers, software and games designers — and of course these 'kids' are animals in the sense that they like to 'hack' into machines."

Data Compression

Bearing in mind the current maximum storage capability of audio CDs, what room is there for interactive information and data?

"It's difficult to say — there are trade‑offs. The figure of 74 minutes of audio is achieved using an existing coding technology which is becoming rapidly outdated. There are other encoding techniques which basically use data compression. One of the good things that came out of DCC [Digital Compact Cassette] was PASC, an encoding format which is part of what's called MPEG. It is possible to compress audio and get up to 96 hours on a single CD, but those 96 hours of MPEG‑encoded audio wouldn't play back on a standard CD player, and there's around 70 million CD players out there!

"Now if we took standard CDDA (CD digital audio), which is where it's at today, the average album length is around 50 minutes which still leaves us with about 30 minutes of extra space — around 200Mb — and we can cram enormous amounts of stuff in there."

Interactive CDs also incorporate data‑saving techniques, like reducing any full‑motion video to 'partial screen' areas. Unlike analogue video, if you use a quarter of the screen, you only use a quarter of the data space. So in theory you could put over four hours of quarter‑screen FMV [Full Motion Video] on a CD. There's also the option of releasing two‑ or even three‑disc products. Browne‑Martin announces that they are currently working with a "major Irish rock band" (who wish to remain nameless), which may well be a three‑disc release because it is "a very video orientated project."

Bearing in mind the emergence of new encoding systems, does Browne‑Martin envisage a new generation of playback machines?

"Even though there are around 70 million CD players out there, the penetration into homes is quite low at around 30%, unlike VCRs at 70%. But there will be a gradual increase, especially as CD will become a de‑facto standard for video game machines, principally because of CD's cheap replication costs."

So the production techniques of CD and CD‑i are identical?

"Yes, and the advantages are enormous, especially for games publishers. But the point is that this will generate more CD player sales, which will increase the penetration, so prices of games will fall."

Browne‑Martin predicts that although mere audio CD players will continue to sell well into the next century, the emergence of affordable multi‑application CD players (those that will accommodate video CDs, photo CDs, and interactive and audio CDs) will entice the first ‑time buyer and upgrader to purchase an entertainment appliance rather than just an audio playback device.

"Philips are betting the farm on CD‑i (following the relative failure of DCC). It's either that or they go back to the lightbulb!" Marketing is going to become aggressive, but Philips have already made a major coup in securing deals with Paramount and MGM for movie video release.

"The whole idea of owning an 'audio‑only' product is going to become archaic. So we're moving into a new era for music in general. I'm convinced that what we're seeing is not revolutionary; it's an evolution, and by the end of the century you're not going to be able to buy music without an audio‑visual interface."

Production Equipment

At the time of my visit, ESP were utilising ProTools on the Macintosh and an SSL Screensound digital workstation, although this has since been replaced with a Digital Audio Research Sigma. The reason for the company's involvement with SSL is partly due to the fact that Carlton Communications (who own SSL) have a 10% share in ESP, and they therefore have access to the inventors of the technology. I wondered if ESP really needed to use such 'high end' audio equipment?

"Well, yes and no. We've spent £1m here on production equipment. We use a variety of equipment because, being a multimedia production studio, we have to be able to work in multiple medias. You can do a lot with ProTools, but it's like the difference in quality between a Quantel Paintbox and Macintosh Photoshop. You can do everything on a ProTools system that you can do with Screensound or a DAR — the difference is that Screensound does it faster. It also means we can do rapid conversions from 44.1 CDDA straight to PCM audio (as used on CD‑i) or PASC audio."

Browne‑Martin likes to keep everything in the digital domain, especially when audio data compression is being used.

"If, when you post‑produce your audio, you get analogue artifacts like tape hiss, pops and clicks, then all those things will reduce the quality of the compression and, as a result, playback may not be as good. We aim to keep everything pure so that the audio is optimised for compression."

Powerful Macintosh computers are utilised heavily within ESP's production studios (see equipment list). The decision to use them was taken because of their ease of use, but Browne‑Martin points out that the company is not 'married' to Macintosh and a major overhaul of equipment is likely in the near future.

"The big question at the moment is whether we'll go with Silicon Graphic machines [high‑end computer graphics systems like the ones used for Terminator 2 and 50 times faster than Macs]. The quality of the software is important, but we're always looking for faster machines. We might well keep the Macs on board but get a couple of SG (Silicon Graphic) machines for 3D animation. We've got high‑quality staff, and you get more out of them when they've got faster equipment.

"The CD mastering we have here [Yamaha] is actually a hybrid system — we're the first company in the world to use recordable CD in the CD‑ROM production process. We got involved with CDR as soon as it was introduced for audio applications because previously we'd had to get a real master cut each time we wanted to test a CD‑i. Sometimes we'd have to test it 10 times, and at £2000 a throw it was a lot of money! We're actually getting a couple of low‑cost CDR devices to supplement the Yamaha system and sit around the studio so that people don't have to queue up."

ESP have their own software (using a 486 PC) running the Yamaha CDR system, which allows them to mix and match all the different media assets. Everything is networked so that they just shoot data down a line to the mastering studio, where a disc is cut. Then, once a finished version is achieved, it is shipped out to a standard CD duplication company.