# Phase Vocoder Tutorial

Roger B. Dannenberg

This page describes how to use the phase vocoder in Nyquist. The phasevocoder is used for time stretching a signal without changing the pitch, and it can be combined with resampling to change both time and pitch independently.

The phase vocoder function is `phasevocoder`, which is called as follows:

> `phasevocoder(input, map, fftsize, hopsize, mode)
> `

where `input` is the input sound, `map` is a map from output time to input time, `fftsize` is the number of samples to use in the analysis, `hopsize` is the step size for use in synthesis, and `mode` controls the synthesis algorithm. Only the first two parameters are required. These parameters will be explained further below.

## How It Works

A phase vocoder works by analyzing the input signal using the FFT, which decomposes the signal into frequency components. The basic assumption is that the input signal consists of sinusoids (no impulses) and that the FFT can resolve these sinusoids into separate components.

The phase vocoder is similar to granular synthesis in that the input is broken into short grains of sound which are reassembled to create an output sound. In the phase vocoder, the output signal is always constructed from equally spaced grains. To achieve time stretching, the spacing of the grains extracted from the input is variable. If grains are taken with higher amounts of overlap, the resulting output is longer so the input is apparently stretched. If grains have less overlap in the input sound, the input will sound speeded up.

The distinctive aspect of the phase vocoder is that when grains are assembled into the output, the phase of each frequency component is adjusted to avoid phase cancellation. Normally, if the spacing of grains is changed from input to output, different frequencies will be shifted by different amounts of phase. Some will add constructively and some will add destructively. Thus, some frequencies will be emphasized over others, often creating a buzz at grain rate. The phase vocoder attempts to eliminate this artifact by adjusting all phases to be coherent.

## The Parameters

`input` is the input sound. It can be any mono sound.
`map` is also a sound type, but it is a control function that maps output to input. For example, if we want to stretch a sound by a factor of two with a result that has a duration of 6 seconds, we should make a control function that is 6 seconds long (the output duration is determined by the duration of `map`). This `map` function should increase from 0 to 3 (because if we stretch by a factor of two, output time 6 will correspond to input time 6). Thus, the `map` parameter could be `pwl(6, 3, 6)` or `ramp(6) * 3`.
`fftsize` is the size of the analysis and synthesis grains. This number must be a power of 2. The default is 2048. Smaller values give less smearing, especially at onsets or percussive sounds that do not meet the sinusoidal content assumption. Larger values give more accurate analysis, especially with dense spectra that do not meet the "FFT can resolve these sinusoids into separate components" assumption.
`hopsize` defaults to `fftsize / 8`. The hopsize should be chosen to be at most `fftsize / 4`, should be a power of 2, and should be less than `fftsize * stretchfactor / 3`. E.g. if the stretch factor is 0.25 (output sounds 4 times as fast as the input), `hopsize` should be decreased to `fftsize / 16`.`
` `mode` is an integer that controls some algorithmic details. The default is `0` which gives a standard phase vocoder. The value 1 uses an alternative computation for phase adjustments that preserves the phase relationship between spectral peaks and nearby bins. The value 2 produces a "robot voice" effect where phase is fixed in each grain, creating a vocoder-like effect.

## Example

Read the audio from `demos/audio/happy.wav` and stretch the first 4 seconds by a factor of 4 so that the output duration is 16 seconds:

    function test9()
      begin
        with inp = s-read("../audio/happy.wav")
        play phasevocoder(inp, pwlv(0, 16, 4))
      end

## Time and Pitch Manipulation

The `pv-time-pitch` function can be used to simultaneously stretch and pitch shift an input signal. It shifts pitch by resampling after phase vocoding. E.g. by stretching by a factor of two and then resampling by a factor of two, the input can be raised one octave in pitch without changing the duration.

The parameters are:

> `pv-time-pitch(input, stretchfn, pitchfn, dur,
> fftsize, hopsize, mode)
> `

where `input` is the input sound to be altered, `stretchfn` controls stretching, `pitchfn` controls pitch shifting, `dur` is the approximate duration, and the remaining parameters are optional parameters passed to the `phasevocoder` function.

`stretchfn` and `pitchfn` work differently from the `map` parameter passed to `phasevocoder`. `stretchfn` gives the amount by which the input should be stretched at each point in time. Thus, if you want the input at time 3.1 to be stretched by a factor of 1.5, the value of `stretchfn` at time 3.1 should be 1.5. Similarly, `pitchfn` is the pitch shift amount (a frequency multiplier) to use at each point of time in the input. If the pitch should be unaltered, the function value should be 1. To raise the pitch, use values greater than 1 (e.g. 2 means go up by one octave).

### Example

In this example, we will alter a familiar tune, Happy Birthday, to another familiar tune, Twinkle Twinkle, using `pv-time-pitch`. To begin, I downloaded a rendition of Happy Birthday from YouTube. (Credit: Artist is unknown, but the URL is `https://www.youtube.com/watch?v=vRJQ3QxLjG0`). Next, I used Audacity to extract a segment of the music and save it to `demos/audio/happy.wav`. I also labeled each syllable with a label track in Audacity and exported the labels to `demos/audio/labels.txt`.

Let's work on `stretchfn` first. We want to make the rhythm mostly quarter notes. For simplicity, let's make the tempo 120, so each quarter note is 0.5s. The stretch factor for a note of length x to become length 0.5 will be 0.5/x. Thus, we need to construct `stretchfn` to have a constant stretch factor for each note, where the stretch factor is 0.5/(t2 - t1) where t2 is the time of the next note and t1 is the time of the note to be stretched, i.e. (t2 - t1) is the duration.

We'll get the note times from the labels file:

    0.000000	0.000000	Hap
    0.166198	0.166198	py
    0.369078	0.369078	birth
    1.085618	1.085618	day
    1.733822	1.733822	to
    2.209605	2.209605	you
    3.578676	3.578676	hap
    3.857731	3.857731	py
    4.055933	4.055933	birth
    4.776379	4.776379	day
    5.425378	5.425378	to
    5.905255	5.905255	you
    7.271426	7.271426	hap
    7.550667	7.550667	py
    7.746811	7.746811	birth
    8.494934	8.494934	day

I assigned each time to a short local variable to make the calculations simpler (although I probably could have created an array and automated things a bit more):

        with h1 = 0.000000,
             p1 = 0.166198,
             b1 = 0.369078,
             d1 = 1.085618,
             t1 = 1.733822,
             y1 = 2.209605,
             h2 = 3.578676,
             p2 = 3.857731,
             b2 = 4.055933,
             d2 = 4.776379,
             t2 = 5.425378,
             y2 = 5.905255,
             h3 = 7.271426,
             p3 = 7.550667,
             b3 = 7.746811,
             d3 = 8.494934

Next, I calculate stretch factors for each note, being careful to make notes 7 and 14 half notes (using 1.0 instead of 0.5 as the target duration):

             u1 = 0.5 / (p1 - h1),
             u2 = 0.5 / (b1 - p1),
             u3 = 0.5 / (d1 - b1),
             u4 = 0.5 / (t1 - d1),
             u5 = 0.5 / (y1 - t1),
             u6 = 0.5 / (h2 - y1),
             u7 = 1.0 / (p2 - h2),
             u8 = 0.5 / (b2 - p2),
             u9 = 0.5 / (d2 - b2),
             u10 = 0.5 / (t2 - d2),
             u11 = 0.5 / (y2 - t2),
             u12 = 0.5 / (h3 - y2),
             u13 = 0.5 / (p3 - h3),
             u14 = 1.0 / (b3 - p3)

Next, I create the `stretchfn` using `pwlv`. This is a little awkward because to obtain steps in the function, each time point must have a time and value for the previous note, then the same time but a new value for the next note, creating a step in the function:

        set *stretch* = pwlv(    u1, p1, u1, p1, u2, b1, u2,
                             b1, u3, d1, u3, d1, u4, t1, u4,
                             t1, u5, y1, u5, y1, u6, h2, u6,
                             h2, u7, p2, u7, p2, u8, b2, u8,
                             b2, u9, d2, u9, d2, u10, t2, u10,
                             t2, u11, y2, u11, y2, u12, h3, u12,
                             h3, u13, p3, u13, p3, u14, b3, u14)

The function is assigned to a global variable. This is not generally recommended, but it will make it easy for us to plot and inspect the function when debugging.
Next, we construct the `pitchfn` based on the difference in pitch between the two melodies. Let's write out the melodies side-by-side using step numbers (only relative pitch matters here):

        67 Hap   60 Twin
        67 py    60 kle
        69 Birth 67 Twin
        67 Day   67 kle
        72 To    69 Lit
        71 You   69 tle
        67 Hap   67 Star
        67 py    65 How
        69 Birth 65 I
        67 day   64 Won
        74 To    64 der
        72 You   62 Who
        67 Hap   62 You
        67 py    60 Are

We'll need another function similar to `*stretch*` only using the pitch differences to create a transposition amount. Let's make `v1` through `v14` and substitute for `u1` through `u14` to make another `pwl` function.

Here are the vn variables, based on the numbers above (pairs are reversed, e.g. to go from the 67 “Hap” to 60 “Twin,” we call `to-ratio(67, 60)`), the function `to-ratio`, and finally, the pitch control function:

             v1 = to-ratio(60, 67),
             v2 = to-ratio(60, 67),
             v3 = to-ratio(67, 69),
             v4 = to-ratio(67, 67),
             v5 = to-ratio(69, 72),
             v6 = to-ratio(69, 71),
             v7 = to-ratio(67, 67),
             v8 = to-ratio(65, 67),
             v9 = to-ratio(65, 69),
             v10 = to-ratio(64, 67),
             v11 = to-ratio(64, 74),
             v12 = to-ratio(62, 72),
             v13 = to-ratio(62, 67),
             v14 = to-ratio(60, 67)

    function to-ratio(a, b) return step-to-hz(a) / step-to-hz(b)

        set *pitch* = pwlv(    v1, p1, v1, p1, v2, b1, v2,
                           b1, v3, d1, v3, d1, v4, t1, v4,
                           t1, v5, y1, v5, y1, v6, h2, v6,
                           h2, v7, p2, v7, p2, v8, b2, v8,
                           b2, v9, d2, v9, d2, v10, t2, v10,
                           t2, v11, y2, v11, y2, v12, h3, v12,
                           h3, v13, p3, v13, p3, v14, b3, v14)

Here is the entire program. (You might need to change the path to `happy.wav` depending on where you run the program. The program is in `demos/src/phasevocoder.sal` and it should run from there.)

    ;; test 9 -- happy birthday to twinkle twinkle
    function test9()
      begin
        with h1 = 0.000000,
             p1 = 0.166198,
             b1 = 0.369078,
             d1 = 1.085618,
             t1 = 1.733822,
             y1 = 2.209605,
             h2 = 3.578676,
             p2 = 3.857731,
             b2 = 4.055933,
             d2 = 4.776379,
             t2 = 5.425378,
             y2 = 5.905255,
             h3 = 7.271426,
             p3 = 7.550667,
             b3 = 7.746811,
             d3 = 8.494934,
             u1 = 0.5 / (p1 - h1),
             u2 = 0.5 / (b1 - p1),
             u3 = 0.5 / (d1 - b1),
             u4 = 0.5 / (t1 - d1),
             u5 = 0.5 / (y1 - t1),
             u6 = 0.5 / (h2 - y1),
             u7 = 1.0 / (p2 - h2),
             u8 = 0.5 / (b2 - p2),
             u9 = 0.5 / (d2 - b2),
             u10 = 0.5 / (t2 - d2),
             u11 = 0.5 / (y2 - t2),
             u12 = 0.5 / (h3 - y2),
             u13 = 0.5 / (p3 - h3),
             u14 = 1.0 / (b3 - p3),
             v1 = to-ratio(60, 67),
             v2 = to-ratio(60, 67),
             v3 = to-ratio(67, 69),
             v4 = to-ratio(67, 67),
             v5 = to-ratio(69, 72),
             v6 = to-ratio(69, 71),
             v7 = to-ratio(67, 67),
             v8 = to-ratio(65, 67),
             v9 = to-ratio(65, 69),
             v10 = to-ratio(64, 67),
             v11 = to-ratio(64, 74),
             v12 = to-ratio(62, 72),
             v13 = to-ratio(62, 67),
             v14 = to-ratio(60, 67)

        set *in* = s-read("/Users/rbd/nyquist/demos/audio/happy.wav")
        set *stretch* = pwlv(    u1, p1, u1, p1, u2, b1, u2,
                             b1, u3, d1, u3, d1, u4, t1, u4,
                             t1, u5, y1, u5, y1, u6, h2, u6,
                             h2, u7, p2, u7, p2, u8, b2, u8,
                             b2, u9, d2, u9, d2, u10, t2, u10,
                             t2, u11, y2, u11, y2, u12, h3, u12,
                             h3, u13, p3, u13, p3, u14, b3, u14)
        set *pitch* = pwlv(    v1, p1, v1, p1, v2, b1, v2,
                           b1, v3, d1, v3, d1, v4, t1, v4,
                           t1, v5, y1, v5, y1, v6, h2, v6,
                           h2, v7, p2, v7, p2, v8, b2, v8,
                           b2, v9, d2, v9, d2, v10, t2, v10,
                           t2, v11, y2, v11, y2, v12, h3, v12,
                           h3, v13, p3, v13, p3, v14, b3, v14)
        play pv-time-pitch(*in*, *stretch*, *pitch*, 10)
      end

    function to-ratio(a, b) return step-to-hz(a) / step-to-hz(b)