VariAudio: Extracting MIDI From Audio Files

Steinberg Cubase Tips & Techniques

Technique : Cubase Notes



John Walden



VariAudio's Extract MIDI function provides three different ways of dealing with pitch-bend data.VariAudio's Extract MIDI function provides three different ways of dealing with pitch-bend data.



Back in SOS August 2009, I took a look at pitch‑correction using Cubase 5's new VariAudio tool, which can be found in the sample editor. While pitch and timing changes are obviously VariAudio's raison d'être, the tool has a further trick up its sleeve: MIDI extraction. In other words, using VariAudio, you can take a monophonic audio performance and extract MIDI timing and pitch information from it, leaving you with a MIDI part in place of your audio file. This type of audio-to-MIDI conversion has been around for quite some time (Logic, for example, has offered it for many years now, and it's also possible in Celemony's Melodyne), but this is the first time the functionality has been built into Cubase.



So why would you want to extract MIDI from your audio? Although none of the current audio-to-MIDI conversion tools are without their particular quirks, they have a number of uses, which include extracting MIDI from a bass-guitar performance, enabling you to double the part with a bass synth (if the bottom end of your mix needs a little beefing up, for example). It can also be a useful way for a guitarist to create melody lines for synths, without using a keyboard or entering the murky waters of MIDI guitar controllers. In addition, extracted MIDI data can be used with Cubase's scoring features to provide notation of parts for other musicians. Surprisingly, for such a potentially useful feature, the MIDI extraction process is given pretty short shrift in Cubase's Operation Manual.

Extraction Basics



The same eight-note audio phrase extracted using each of the three pitch‑bend data modes.The same eight-note audio phrase extracted using each of the three pitch‑bend data modes.



The basic principles of the extraction process are straightforward. Having identified the pitch of individual notes in an audio performance, VariAudio can then export that data to a suitable MIDI part via the Function / Extract MIDI command. If you've used VariAudio's Pitch & Warp or Segment tools, the changes you've made to the segments will be reflected in the timing and pitch of the MIDI notes created.



There are three options, in terms of pitch‑bend data, when outputting the extracted MIDI information. First, the data can be excluded, so you just get the individual note, and this automatically pitch‑quantises MIDI notes to the nearest semitone. As it ignores pitch-bend data, this mode produces a fairly simple output, and is useful if you want to play it back via an instrument that doesn't naturally support pitch-bend (such as a piano), for producing notation (where simpler is generally better for generating a clean-looking score), or if you just need the basics of the part extracted for further MIDI editing. The second option, Notes and Static Pitch-bend Data, simply adds a single pitch-bend event to each of the segment pitches to move it to the nearest semitone — which is effectively a kind of pitch‑quantise by using pitch-bend. I'm not convinced of the value of this mode, so let me know if you find a practical use for it!



The final mode, Notes and Continuous Pitch-bend Data, does exactly what you'd expect, and uses pitch-bend data to match as closely as possible the pitch curve that you see displayed by VariAudio in the Sample Editor. This is great for getting the maximum amount of expression from your audio performance, but it can generate large pitch‑bends, where legato notes slide from one pitch to another. Depending on the pitch‑bend range set in the output dialogue box, a warning indicating the semitone range of the largest pitch-bend in the extracted part might be displayed. You should change the setting in the dialogue to match this, and the setting is then embedded as controller data at the beginning of the MIDI event (you can, of course, view this using the MIDI List Editor). For many MIDI instruments, this will automatically change the pitch-bend range, but if things don't sound right you may also need to manually adjust your sampler patch settings. This is easier to do with some MIDI sound sources than others (for example, you can't manually change this setting for HALion One instruments: you're stuck with whatever default is built into the patch), and even then it can require experimentation. If you get desperate, it's also possible to scale pitch‑bend data using the Logical Editor, but that's outside the scope of this article.

GIGO



VariAudio's pitch detection applied twice to the same guitar phrase. The upper pane is the clean DI'd part, while the lower one is based on the performance recorded through an amp with various effects. Note how the clean performance produces the simpler pitch and note data, which in turn is easier for the MIDI extraction process to work with.VariAudio's pitch detection applied twice to the same guitar phrase. The upper pane is the clean DI'd part, while the lower one is based on the performance recorded through an amp with various effects. Note how the clean performance produces the simpler pitch and note data, which in turn is easier for the MIDI extraction process to work with.



Good though the VariAudio MIDI extraction process can be, some audio sources seem to provide better input signals than others, and it's worth avoiding the frustration of 'Garbage In, Garbage Out'. Performances featuring single notes (no chords), and recorded as mono rather than stereo, are essential. It also helps if the audio is recorded at a reasonable level, isn't smothered with effects and has notes with a clearly defined attack. For example, a dry, DI'd guitar or bass (without amp simulation or effects) can work well and, in general, instruments are easier to work with than vocals, which tend to produce more complex pitch variations. The screenshot above shows a simple example comparing a clean DI'd guitar with the same part recorded via an amp with some effects applied. VariAudio has clearly found it easier to determine pitch data from the former.



It's also worth editing the VariAudio segments prior to extracting the MIDI data. The most important step is to get the start of each new segment aligned with the attacks of the notes in the audio waveform. This ensures that the timing of the MIDI notes generated will closely match that of the audio performance. Segments can be split or deleted where the VariAudio pitch detection algorithm has either created a single segment that spans more than one note, or where two segments have been created for a single note. This ensures that the number of MIDI notes matches the number of notes in the audio part. For greater accuracy, you can zoom in on the audio display as you use the Segments tool.

A Dose Of The Bends



Here, the same audio phrase shown in the previous screenshot has had some minor edits to its segmentation, as described in the main text. VariAudio's Pitch Quantize and Straighten Pitch options have also been applied, prior to using the Extract MIDI function.Here, the same audio phrase shown in the previous screenshot has had some minor edits to its segmentation, as described in the main text. VariAudio's Pitch Quantize and Straighten Pitch options have also been applied, prior to using the Extract MIDI function.



Once you've edited the position and number of segments, you can use the Pitch Quantize and Straighten Pitch sliders to clean up the performance. The usual qualifiers apply here. For example, make sure that when quantising the pitch, wayward notes don't get dragged to the wrong semitone. This is easily corrected by dragging and dropping the segment back where you want it, so you can be quite aggressive with the Pitch Quantize setting.



If you intend to extract the pitch-bend data as well, Straighten Pitch is useful for cleaning up more extreme wobbles. This is perhaps more useful on vocals than instruments, but in either case it needs to be done subtly or you might change (or even lose) the expression of the performance. If you plan to use the MIDI data for a piano or to create notation, extracting without pitch‑bend data is a better option anyway, but if you want, for example, to turn a vocal or guitar performance into a MIDI‑based violin or saxophone performance, retaining the bends of pitch — and experimenting with your settings to get the pitch-bend performance of the MIDI instrument sounding believable — is an important part of the process.

Wot, No Drums?



While VariAudio's MIDI extraction does require a little patience, with practice it can produce very good results. I've posted a couple of audio examples on the SOS web site (/sos/nov09/articles/cubaseaudio.htm). As you'd expect, the quality of what can be achieved depends to a large extent on the playback instrument. It's easy to make the translation to piano‑style instruments, as there are no pitch-bend issues. With parts containing pitch‑bend data, synth‑based sounds tend to be more convincing than virtual versions of 'real' instruments; not surprising when you think how hard it can be to program realistic solo parts for many string or wind instruments!



There are also a couple of things that this audio‑to‑MIDI function can't do. First, the amplitude of the audio is not converted into MIDI velocity data, and all the MIDI notes created are given a default velocity of 100, so volume expression has to be added via subsequent editing. Steinberg might add an amplitude‑to‑velocity element to the algorithm sometime, but for the moment you'll have to look at third‑party plug‑ins to do that.



Second, VariAudio's note- and pitch- detection algorithm is not really designed to work with drum or percussion instruments, so if you were hoping to use it to convert audio drum performances into MIDI data (for example, so you could replace or layer individual drum sounds), you might find yourself frustrated by the results. Again, while this would be nice to see as a future feature, for now I'd stick with third‑party plug‑ins such as KTDrum Trigger (see the SOS May 2007 Cubase workshop for details). A MIDI output on the Gate plug‑in would also be useful in this application. Alternatively, if you're just working with short(ish) loops, you could experiment with some slicing in the Sample Editor and Groove Agent One's drag-and‑drop capabilities — which is perhaps a fitting subject for a future column.    

Published in SOS November 2009