Deep Audio Redundancy (DRED) Extension for the Opus Codec

2.1.1. Pre-emphasis and Windowing

The input audio signal is first pre-emphasized using a filter A(z) = 1 - 0.85 * z^-1.¶

The pre-emphasized signal is then windowed over a 20 ms window (320 samples) with a 10 ms overlap, using a Vorbis window (see [Vorbis-Spec]).¶

2.1.2. Fast Fourier Transform and Band Energy

A 320-point Fast Fourier Transform (FFT) is computed on the windowed signal. The power spectrum is then integrated into 18 triangular-shaped bands matching the CELT bands. The band boundaries are defined by the start, center, and end frequencies in Table 1 (converted to FFT bins assuming a 50 Hz bin spacing).¶

The band energies are normalized using compensation factors C_i to ensure that a flat spectrum yields flat band energies. These factors are inversely proportional to the sum of the filter weights for each band, and are defined as: C = [0.8, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.666667, 0.5, 0.5, 0.5, 0.333333, 0.25, 0.25, 0.2, 0.166667, 0.173913].¶

2.1.3. Log Spectral Smoothing and BFCC

To model spectral masking, a log-spectral smoothing is applied to the log band energies. The log energy for each band is limited so that it does not drop by more than 80 dB relative to the maximum of the preceding bands, nor by more than 25 dB per band relative to any preceding band.¶

The 18 cepstral features are computed by applying the orthonormal Discrete Cosine Transform (DCT-II) to the smoothed log energies, with a normalization bias of -4 applied to the first coefficient (DC).¶

2.1.4. Pitch and Voicing Features

Feature 18 represents the pitch period, and feature 19 represents the voicing information.¶

The pitch estimator is not normative. The reference implementation uses a neural pitch estimator described in [subramani2024].¶

The voicing feature is computed from the normalized correlation of the low-pass filtered Linear Predictive Coding (LPC) residual. The LPC analysis filter is derived from the reconstructed cepstral features using standard Levinson-Durbin recursion. The residual is low-pass filtered using a second-order elliptic filter with a cutoff frequency of 1200 Hz (e.g., designed as `ellip(2, 2, 20, 1200/8000)` in MATLAB).¶

Let r_lp[n] be the low-pass filtered residual, and P be the pitch period in samples. The normalized correlation rho is computed as: rho = sum(r_lp[n] * r_lp[n-P]) / sqrt((1 + sum(r_lp[n]^2)) * (1 + sum(r_lp[n-P]^2))) for n from 0 to 159. The voicing feature is then obtained by applying a soft threshold: c[19] = ln(1 + exp(5 * rho)) / ln(1 + exp(5)) - 0.5.¶

2.2.1. Encoder architecture

Every 20 ms, the encoder takes in a pair of 20-dimensional acoustic feature vectors as input and produces one initial state (IS) and one latent vector. Each latent vector encodes 40 ms (their information overlaps), so only half the latent vectors need to be transmitted. Although an encoder is provided for reference, the encoder architecture is not normative. Each redundancy packet contains the latest initial state, along with latent vectors ordered from the latest (the one aligned with the initial state) to the earliest one the encoder includes. Each component of the IS and latent vectors are quantized and then entropy-coded following a Laplace distribution. The same procedure is used for both the latent vectors and the initial state (we will describe the process for a latent variable). The quantized index X is obtained by scaling the i'th latent variable z_i by a scaling factor s_{i,q} that depends on both i and on the quantizer q. We then apply a "dead-zone" function zeta(z) = z - (d/256)*tanh(z / (d/256 + epsilon)), where d also depends on i and q, and epsilon=0.1. The result is then rounded to the nearest integer: X = round(zeta(s_{i,q}*z_i/256)). The Laplace distribution used for entropy coding is parameterized with a probability that the value is zero (p0), as well as a decay factor r (0 < r < 1). Both p0 and r depend on i and q. The probability p(X) for a coefficient is given by:¶

                          /
                          | p0                          ,   if X = 0
                          |
                   P(X) = <                       |X|-1
                          | (1 - p0) * (1 - r) * r      ,   if X != 0
                          | ---------------------------
                          \              2

2.2.2. Decoder architecture

Unlike the encoder, the decoder is normative. The decoder uses the same Laplace distribution above to decode the symbols and then scales them back by 256/max(1, s_{i,q}). The initial state is used as input to initialize the decoder's gated recurrent units (GRUs). The latent vectors are used one at a time as input the DNN decoder, which produces 4 vectors of 20 acoustic features for each input latent vector.¶

The decoder is mostly structured as a DenseNet network, with 5 sets of alternating GRU and convolutional layers. Let gru1..gru5 denote the 5 GRUs, conv1..conv5 denote the 5 1-D causal convolutional layers with kernel size 2, hidden_init/gru_init/dense1/output/cdense* denote fully-connected layers, glu1..glu5 denote gated linear units (GLUs), and cat() denote tensor concatenation. All GRU layers have 64 outputs (number of neurons), all convolutional layers and cdense* layers have 32 outputs, and dense1 has 96 outputs. Despite using a functional notation, both the GRU and convolutional layers have an internal state when used one latent vector at a time. The fully-connected layers all have different sizes. Fully-connected (dense) layers apply an affine transformation W*y+b followed by a tanh activation (except for the final "output" layer, which uses a linear activation). Gated Linear Units (GLUs) take an input vector y and apply an affine transformation W*y+b followed by a sigmoid activation and an element-wise multiplication (*):¶

                   L(y) = sigmoid(W*y + b) * y

1D causal convolutional layers with kernel size 2 take an input vector y[k] at chunk k >= 0, concatenate the previous chunk's input y[k-1] (where y[-1] is a zero vector for k=0), and apply an affine transformation and tanh activation:¶

                   conv(y[k]) = tanh(W * [y[k-1], y[k]] + b)

Gated Recurrent Units (GRUs) take an input vector x and previous state h_prev of dimension N=64. Let [z_in, r_in, h_in] = W_in*x + b_in be the 3N-dimensional input projection, and [z_rec, r_rec, h_rec] = W_rec*h_prev be the 3N-dimensional recurrent projection. The update gate z, reset gate r, candidate state h_cand, and new state h_new are computed as:¶

                   z = sigmoid(z_in + z_rec)
                   r = sigmoid(r_in + r_rec)
                   h_cand = tanh(h_in + r * h_rec)
                   h_new = z * h_prev + (1 - z) * h_cand

The decoder starts with the 50-dimensional initial state vector IS. The IS is used to compute the GRU initialization vector V using both hidden_init and gru_init:¶

                   V = gru_init(hidden_init(IS))

where hidden_init has 50 inputs and 128 outputs, and gru_init has 128 inputs and 320 (5*64) outputs. The components of V are split (sequentially) into the V1..V5 initialization vectors (original state before the decoding process) for GRUs gru1..gru5. Let Z be a 26-dimensional vector constructed from the decoded 25-dimensional latent vector for a particular 40-ms chunk, to which we append the value of q/8-1, where q is the quantizer index used for the current chunk (as defined in Section 3). From there, the DenseNet structure can be expressed as:¶

                   t1 = dense1(Z)
                   t2 = cat(t1, glu1(gru1(t1)))
                   t3 = cat(t2, conv1(cdense1(t2)))
                   t4 = cat(t3, glu2(gru2(t3)))
                   t5 = cat(t4, conv2(cdense2(t4)))
                   t6 = cat(t5, glu3(gru3(t5)))
                   t7 = cat(t6, conv3(cdense3(t6)))
                   t8 = cat(t7, glu4(gru4(t7)))
                   t9 = cat(t8, conv4(cdense4(t8)))
                   t10 = cat(t9, glu5(gru5(t9)))
                   t11 = cat(t10, conv5(cdense5(t10)))
                   x = output(t11)

where t1..tN are temporary vectors and "output" is the only layer to have a linear output activation, with 80 output neurons (4*20). The dimensionality of t1..t11 (and corresponding GRU/convolutional input size) can be inferred from the concatenation operations. The output vector x is split (sequentially) into 4 feature vectors of 20 dimensions each that can be sent to the vocoder if packets are lost. Since the decoder operates in reverse chronological order, the output vectors must be reversed before they are sent to the vocoder.¶

2.2.3. Statistical data

We define 16 different quantization settings, ranging from q=0 (higher bitrate) to q=15 (lower bitrate). For each quantizer and for each latent variable or initial state coefficient, we have a normative scale (s), decay (r), and p0 value. Note that the dead-zone parameters d are not normative.¶

2.2.4. Vocoder

A vocoder is needed to turn the acoustic features into actual speech to fill in the audio for any missing packets. Although the decoder is not normative, certain properties are needed for DRED to function adequately. First, the vocoder SHOULD be able to start synthesizing speech by continuing an existing waveform, reducing the artifacts caused at the beginning of a lost packet. If such property cannot be achieved, then the implementation SHOULD at least make an attempt to synchronize the phase of the synthesized speech with the last received speech, and attempt some form of blending, e.g. by splicing the signals in the LPC residual domain.¶

A second important property of the vocoder is to not rely on more than one feature vector of look-ahead. To synthesize speech between time t-10ms and t, the vocoder SHOULD NOT rely on acoustic features centered beyond t+5ms (i.e. covering t-5ms to t+15ms). The vocoder MAY use more look-ahead when it is available, but there are cases (e.g. last lost packet) where the amount of acoustic feature vectors will be limited. For frames sizes less than 20 ms, the decoder SHOULD be prepared to deal with having less than one feature vector of look-ahead.¶