The Problem of Resonances in the Transition Bands

Consider the following design from [1].  It is a stopband filter with two
transition bands.  The following shell script shows the design values.

DFiltFIR -n 75 Ex2-1.cof << EoF
0           1 0.04588809               # Passband #1
0.143759973 1 0.04588809

0.16533943  0 1                        # Stopband
0.37032451  0 1

0.41679744  1 0.03853275               # Passband #2
0.5         1 0.03853275
EoF
Linear Phase FIR Filter
           Freq.      Value    Weight           Limits      Deviation  Dev dB
Band  1:  0.0000          1     0.046      ----       ----      0.072    0.60
          0.1438          1     0.046      ----       ----      0.072    0.60
Band  2:  0.1653          0         1      ----       ----     0.0033  -49.62
          0.3703          0         1      ----       ----     0.0033  -49.62
Band  3:  0.4168          1     0.039      ----       ----      0.086    0.71
          0.5000          1     0.039      ----       ----      0.086    0.71

This filter exhibits a non-monotonic transition in the transition from the
stopband to the second passband.  The transition band response rises to a
maximum of about 2.6 near frequency 0.4, before settling down to near unity
at the passband edge (0.417).  The band edge extremum in the passband is a
downward one (as would normally occur at the passband edge), but the slope is
zero, and the response rises as we go down in frequency into the transition
band.  This kind of "resonance" does not occur in simple low-pass or high-pass
filters.  It occurs only in some multiband designs.

Consider using constraints to force the transition band to stay below unity.
First let us specify the values in a new band which overlays the previous
transition band.  The desired value will rise from 0 to 1 in this new band,
but the weight will be extremely small.  First let us verify that just
specifying this new band does not substantially change the design.

DFiltFIR -n 75 Ex2-2.cof << EoF
0           1 0.04588809
0.143759973 1 0.04588809

0.16533943  0 1
0.37032451  0 1

0.370326    0 1e-6
0.416796    1 1e-6

0.41679744  1 0.03853275
0.5         1 0.03853275
EoF
Linear Phase FIR Filter
           Freq.      Value    Weight           Limits      Deviation  Dev dB
Band  1:  0.0000          1     0.046      ----       ----      0.072    0.60
          0.1438          1     0.046      ----       ----      0.072    0.60
Band  2:  0.1653          0         1      ----       ----     0.0033  -49.63
          0.3703          0         1      ----       ----     0.0033  -49.63
Band  3:  0.3703          0     1e-06      ----       ----    3.3e+03   70.37
          0.4168          1     1e-06      ----       ----    3.3e+03   70.37
Band  4:  0.4168          1     0.039      ----       ----      0.086    0.71
          0.5000          1     0.039      ----       ----      0.086    0.71

This filter is substantially the same as without the extra band.  Now let us
add constraints to the transition band.  This band is now constrained to have
values betweeen 0 and 1.

DFiltFIR -n 75 Ex2-3.cof << EoF
0           1 0.04588809
0.143759973 1 0.04588809

0.16533943  0 1
0.37032451  0 1

0.370326    0 1e-6        0 1
0.416796    1 1e-6        0 1

0.41679744  1 0.03853275
0.5         1 0.03853275
EoF
Linear Phase FIR Filter
           Freq.      Value    Weight           Limits      Deviation  Dev dB
Band  1:  0.0000          1     0.046      ----       ----      0.074    0.62
          0.1438          1     0.046      ----       ----      0.074    0.62
Band  2:  0.1653          0         1      ----       ----     0.0034  -49.43
          0.3703          0         1      ----       ----     0.0034  -49.43
Band  3:  0.3703          0     1e-06          0          1         1    0.00
          0.4168          1     1e-06          0          1         1    6.02
Band  4:  0.4168          1     0.039      ----       ----      0.088    0.73
          0.5000          1     0.039      ----       ----      0.088    0.73

The new filter does not go above unity in the "transition" band.  Note however,
the slight increase in passband and stopband ripple.  This is the price paid
for the constraint.  A plot of the frequency response reveals that the
resonance is still present at f=0.4, but the peak value is now unity.  This
effectively means that the passband really extends to some frequency below
0.4.

Noting that the passband effectively goes below the actual band edge, suggests
that we can move the passband edge so that the resonance becomes a ripple in
the passband instead of the transition band.  In a final filter design, we
redefine the upper passband edge to 0.398, while leaving all other frequencies
the same as in the original design (no constraints, no extra band).

DFiltFIR -n 75 Ex2-4.cof << EoF
0           1 0.04588809
0.143759973 1 0.04588809

0.16533943  0 1
0.37032451  0 1

0.398       1 0.03853275
0.5         1 0.03853275
EoF
Linear Phase FIR Filter
           Freq.      Value    Weight           Limits      Deviation  Dev dB
Band  1:  0.0000          1     0.046      ----       ----      0.073    0.62
          0.1438          1     0.046      ----       ----      0.073    0.62
Band  2:  0.1653          0         1      ----       ----     0.0034  -49.45
          0.3703          0         1      ----       ----     0.0034  -49.45
Band  3:  0.3980          1     0.039      ----       ----      0.087    0.73
          0.5000          1     0.039      ----       ----      0.087    0.73

A ripple still occurs at around f=0.4, but this is now considered part of
the passband.  This "overshoot" is the same as the ripple amplitude in the
rest of the passband.

As a final note, a number of other options could have been considered.  The
order of the filter could have been reduced.  In this case reducing the order
to 73, stops an overshoot in the transition band, though there is now an
extra zero at around f=0.4.  A true monotonic transition can be forced by
carefully choosing the constraint values - in particular, the constraint values
should rise from a low value (0.5 works) to a value equal to one minus the
deviation in the passband at the passband edge (say 0.913).

[1] G. Cortelazzo, M. R. Lightner and W. K. Jenkins,
    "An Alternate Technique for Min-Max Design of Multiband Finite Impulse
    Response Digital Filters",
    Circuits, Systems and Signal Processing, pp. 285-309, Vol. 2, No. 3, 1983.

$Id: Example2,v 1.1 1994/06/03 FilterDesign-V1R7a $
