This is gsl-ref.info, produced by makeinfo version 5.1 from
gsl-ref.texi.

Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016
The GSL Team.

   Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with the
Invariant Sections being "GNU General Public License" and "Free Software
Needs Free Documentation", the Front-Cover text being "A GNU Manual",
and with the Back-Cover Text being (a) (see below).  A copy of the
license is included in the section entitled "GNU Free Documentation
License".

   (a) The Back-Cover Text is: "You have the freedom to copy and modify
this GNU Manual."
INFO-DIR-SECTION Software libraries
START-INFO-DIR-ENTRY
* gsl-ref: (gsl-ref).                   GNU Scientific Library - Reference
END-INFO-DIR-ENTRY


File: gsl-ref.info,  Node: Series Acceleration References,  Prev: Example of accelerating a series,  Up: Series Acceleration

31.4 References and Further Reading
===================================

The algorithms used by these functions are described in the following
papers,

     T. Fessler, W.F. Ford, D.A. Smith, HURRY: An acceleration algorithm
     for scalar sequences and series 'ACM Transactions on Mathematical
     Software', 9(3):346-354, 1983.  and Algorithm 602 9(3):355-357,
     1983.

The theory of the u-transform was presented by Levin,

     D. Levin, Development of Non-Linear Transformations for Improving
     Convergence of Sequences, 'Intern. J. Computer Math.' B3:371-388,
     1973.

A review paper on the Levin Transform is available online,
     Herbert H. H. Homeier, Scalar Levin-Type Sequence Transformations,
     <http://arxiv.org/abs/math/0005209>.


File: gsl-ref.info,  Node: Wavelet Transforms,  Next: Discrete Hankel Transforms,  Prev: Series Acceleration,  Up: Top

32 Wavelet Transforms
*********************

This chapter describes functions for performing Discrete Wavelet
Transforms (DWTs).  The library includes wavelets for real data in both
one and two dimensions.  The wavelet functions are declared in the
header files 'gsl_wavelet.h' and 'gsl_wavelet2d.h'.

* Menu:

* DWT Definitions::             
* DWT Initialization::          
* DWT Transform Functions::     
* DWT Examples::                
* DWT References::              


File: gsl-ref.info,  Node: DWT Definitions,  Next: DWT Initialization,  Up: Wavelet Transforms

32.1 Definitions
================

The continuous wavelet transform and its inverse are defined by the
relations,

     w(s,\tau) = \int f(t) * \psi^*_{s,\tau}(t) dt

and,

     f(t) = \int \int_{-\infty}^\infty w(s, \tau) * \psi_{s,\tau}(t) d\tau ds

where the basis functions \psi_{s,\tau} are obtained by scaling and
translation from a single function, referred to as the "mother wavelet".

   The discrete version of the wavelet transform acts on equally-spaced
samples, with fixed scaling and translation steps (s, \tau).  The
frequency and time axes are sampled "dyadically" on scales of 2^j
through a level parameter j.  The resulting family of functions
{\psi_{j,n}} constitutes an orthonormal basis for square-integrable
signals.

   The discrete wavelet transform is an O(N) algorithm, and is also
referred to as the "fast wavelet transform".


File: gsl-ref.info,  Node: DWT Initialization,  Next: DWT Transform Functions,  Prev: DWT Definitions,  Up: Wavelet Transforms

32.2 Initialization
===================

The 'gsl_wavelet' structure contains the filter coefficients defining
the wavelet and any associated offset parameters.

 -- Function: gsl_wavelet * gsl_wavelet_alloc (const gsl_wavelet_type *
          T, size_t K)
     This function allocates and initializes a wavelet object of type T.
     The parameter K selects the specific member of the wavelet family.
     A null pointer is returned if insufficient memory is available or
     if a unsupported member is selected.

   The following wavelet types are implemented:

 -- Wavelet: gsl_wavelet_daubechies
 -- Wavelet: gsl_wavelet_daubechies_centered
     This is the Daubechies wavelet family of maximum phase with k/2
     vanishing moments.  The implemented wavelets are k=4, 6, ..., 20,
     with K even.

 -- Wavelet: gsl_wavelet_haar
 -- Wavelet: gsl_wavelet_haar_centered
     This is the Haar wavelet.  The only valid choice of k for the Haar
     wavelet is k=2.

 -- Wavelet: gsl_wavelet_bspline
 -- Wavelet: gsl_wavelet_bspline_centered
     This is the biorthogonal B-spline wavelet family of order (i,j).
     The implemented values of k = 100*i + j are 103, 105, 202, 204,
     206, 208, 301, 303, 305 307, 309.

The centered forms of the wavelets align the coefficients of the various
sub-bands on edges.  Thus the resulting visualization of the
coefficients of the wavelet transform in the phase plane is easier to
understand.

 -- Function: const char * gsl_wavelet_name (const gsl_wavelet * W)
     This function returns a pointer to the name of the wavelet family
     for W.

 -- Function: void gsl_wavelet_free (gsl_wavelet * W)
     This function frees the wavelet object W.

   The 'gsl_wavelet_workspace' structure contains scratch space of the
same size as the input data and is used to hold intermediate results
during the transform.

 -- Function: gsl_wavelet_workspace * gsl_wavelet_workspace_alloc
          (size_t N)
     This function allocates a workspace for the discrete wavelet
     transform.  To perform a one-dimensional transform on N elements, a
     workspace of size N must be provided.  For two-dimensional
     transforms of N-by-N matrices it is sufficient to allocate a
     workspace of size N, since the transform operates on individual
     rows and columns.  A null pointer is returned if insufficient
     memory is available.

 -- Function: void gsl_wavelet_workspace_free (gsl_wavelet_workspace *
          WORK)
     This function frees the allocated workspace WORK.


File: gsl-ref.info,  Node: DWT Transform Functions,  Next: DWT Examples,  Prev: DWT Initialization,  Up: Wavelet Transforms

32.3 Transform Functions
========================

This sections describes the actual functions performing the discrete
wavelet transform.  Note that the transforms use periodic boundary
conditions.  If the signal is not periodic in the sample length then
spurious coefficients will appear at the beginning and end of each level
of the transform.

* Menu:

* DWT in one dimension::        
* DWT in two dimension::        


File: gsl-ref.info,  Node: DWT in one dimension,  Next: DWT in two dimension,  Up: DWT Transform Functions

32.3.1 Wavelet transforms in one dimension
------------------------------------------

 -- Function: int gsl_wavelet_transform (const gsl_wavelet * W, double *
          DATA, size_t STRIDE, size_t N, gsl_wavelet_direction DIR,
          gsl_wavelet_workspace * WORK)
 -- Function: int gsl_wavelet_transform_forward (const gsl_wavelet * W,
          double * DATA, size_t STRIDE, size_t N, gsl_wavelet_workspace
          * WORK)
 -- Function: int gsl_wavelet_transform_inverse (const gsl_wavelet * W,
          double * DATA, size_t STRIDE, size_t N, gsl_wavelet_workspace
          * WORK)

     These functions compute in-place forward and inverse discrete
     wavelet transforms of length N with stride STRIDE on the array
     DATA.  The length of the transform N is restricted to powers of
     two.  For the 'transform' version of the function the argument DIR
     can be either 'forward' (+1) or 'backward' (-1).  A workspace WORK
     of length N must be provided.

     For the forward transform, the elements of the original array are
     replaced by the discrete wavelet transform f_i -> w_{j,k} in a
     packed triangular storage layout, where J is the index of the level
     j = 0 ... J-1 and K is the index of the coefficient within each
     level, k = 0 ... (2^j)-1.  The total number of levels is J =
     \log_2(n).  The output data has the following form,

          (s_{-1,0}, d_{0,0}, d_{1,0}, d_{1,1}, d_{2,0}, ...,
            d_{j,k}, ..., d_{J-1,2^{J-1}-1})

     where the first element is the smoothing coefficient s_{-1,0},
     followed by the detail coefficients d_{j,k} for each level j.  The
     backward transform inverts these coefficients to obtain the
     original data.

     These functions return a status of 'GSL_SUCCESS' upon successful
     completion.  'GSL_EINVAL' is returned if N is not an integer power
     of 2 or if insufficient workspace is provided.


File: gsl-ref.info,  Node: DWT in two dimension,  Prev: DWT in one dimension,  Up: DWT Transform Functions

32.3.2 Wavelet transforms in two dimension
------------------------------------------

The library provides functions to perform two-dimensional discrete
wavelet transforms on square matrices.  The matrix dimensions must be an
integer power of two.  There are two possible orderings of the rows and
columns in the two-dimensional wavelet transform, referred to as the
"standard" and "non-standard" forms.

   The "standard" transform performs a complete discrete wavelet
transform on the rows of the matrix, followed by a separate complete
discrete wavelet transform on the columns of the resulting
row-transformed matrix.  This procedure uses the same ordering as a
two-dimensional Fourier transform.

   The "non-standard" transform is performed in interleaved passes on
the rows and columns of the matrix for each level of the transform.  The
first level of the transform is applied to the matrix rows, and then to
the matrix columns.  This procedure is then repeated across the rows and
columns of the data for the subsequent levels of the transform, until
the full discrete wavelet transform is complete.  The non-standard form
of the discrete wavelet transform is typically used in image analysis.

   The functions described in this section are declared in the header
file 'gsl_wavelet2d.h'.

 -- Function: int gsl_wavelet2d_transform (const gsl_wavelet * W, double
          * DATA, size_t TDA, size_t SIZE1, size_t SIZE2,
          gsl_wavelet_direction DIR, gsl_wavelet_workspace * WORK)
 -- Function: int gsl_wavelet2d_transform_forward (const gsl_wavelet *
          W, double * DATA, size_t TDA, size_t SIZE1, size_t SIZE2,
          gsl_wavelet_workspace * WORK)
 -- Function: int gsl_wavelet2d_transform_inverse (const gsl_wavelet *
          W, double * DATA, size_t TDA, size_t SIZE1, size_t SIZE2,
          gsl_wavelet_workspace * WORK)

     These functions compute two-dimensional in-place forward and
     inverse discrete wavelet transforms in standard form on the array
     DATA stored in row-major form with dimensions SIZE1 and SIZE2 and
     physical row length TDA.  The dimensions must be equal (square
     matrix) and are restricted to powers of two.  For the 'transform'
     version of the function the argument DIR can be either 'forward'
     (+1) or 'backward' (-1).  A workspace WORK of the appropriate size
     must be provided.  On exit, the appropriate elements of the array
     DATA are replaced by their two-dimensional wavelet transform.

     The functions return a status of 'GSL_SUCCESS' upon successful
     completion.  'GSL_EINVAL' is returned if SIZE1 and SIZE2 are not
     equal and integer powers of 2, or if insufficient workspace is
     provided.

 -- Function: int gsl_wavelet2d_transform_matrix (const gsl_wavelet * W,
          gsl_matrix * M, gsl_wavelet_direction DIR,
          gsl_wavelet_workspace * WORK)
 -- Function: int gsl_wavelet2d_transform_matrix_forward (const
          gsl_wavelet * W, gsl_matrix * M, gsl_wavelet_workspace * WORK)
 -- Function: int gsl_wavelet2d_transform_matrix_inverse (const
          gsl_wavelet * W, gsl_matrix * M, gsl_wavelet_workspace * WORK)
     These functions compute the two-dimensional in-place wavelet
     transform on a matrix A.

 -- Function: int gsl_wavelet2d_nstransform (const gsl_wavelet * W,
          double * DATA, size_t TDA, size_t SIZE1, size_t SIZE2,
          gsl_wavelet_direction DIR, gsl_wavelet_workspace * WORK)
 -- Function: int gsl_wavelet2d_nstransform_forward (const gsl_wavelet *
          W, double * DATA, size_t TDA, size_t SIZE1, size_t SIZE2,
          gsl_wavelet_workspace * WORK)
 -- Function: int gsl_wavelet2d_nstransform_inverse (const gsl_wavelet *
          W, double * DATA, size_t TDA, size_t SIZE1, size_t SIZE2,
          gsl_wavelet_workspace * WORK)
     These functions compute the two-dimensional wavelet transform in
     non-standard form.

 -- Function: int gsl_wavelet2d_nstransform_matrix (const gsl_wavelet *
          W, gsl_matrix * M, gsl_wavelet_direction DIR,
          gsl_wavelet_workspace * WORK)
 -- Function: int gsl_wavelet2d_nstransform_matrix_forward (const
          gsl_wavelet * W, gsl_matrix * M, gsl_wavelet_workspace * WORK)
 -- Function: int gsl_wavelet2d_nstransform_matrix_inverse (const
          gsl_wavelet * W, gsl_matrix * M, gsl_wavelet_workspace * WORK)
     These functions compute the non-standard form of the
     two-dimensional in-place wavelet transform on a matrix A.


File: gsl-ref.info,  Node: DWT Examples,  Next: DWT References,  Prev: DWT Transform Functions,  Up: Wavelet Transforms

32.4 Examples
=============

The following program demonstrates the use of the one-dimensional
wavelet transform functions.  It computes an approximation to an input
signal (of length 256) using the 20 largest components of the wavelet
transform, while setting the others to zero.

     #include <stdio.h>
     #include <math.h>
     #include <gsl/gsl_sort.h>
     #include <gsl/gsl_wavelet.h>

     int
     main (int argc, char **argv)
     {
       (void)(argc); /* avoid unused parameter warning */
       int i, n = 256, nc = 20;
       double *data = malloc (n * sizeof (double));
       double *abscoeff = malloc (n * sizeof (double));
       size_t *p = malloc (n * sizeof (size_t));

       FILE * f;
       gsl_wavelet *w;
       gsl_wavelet_workspace *work;

       w = gsl_wavelet_alloc (gsl_wavelet_daubechies, 4);
       work = gsl_wavelet_workspace_alloc (n);

       f = fopen (argv[1], "r");
       for (i = 0; i < n; i++)
         {
           fscanf (f, "%lg", &data[i]);
         }
       fclose (f);

       gsl_wavelet_transform_forward (w, data, 1, n, work);

       for (i = 0; i < n; i++)
         {
           abscoeff[i] = fabs (data[i]);
         }

       gsl_sort_index (p, abscoeff, 1, n);

       for (i = 0; (i + nc) < n; i++)
         data[p[i]] = 0;

       gsl_wavelet_transform_inverse (w, data, 1, n, work);

       for (i = 0; i < n; i++)
         {
           printf ("%g\n", data[i]);
         }

       gsl_wavelet_free (w);
       gsl_wavelet_workspace_free (work);

       free (data);
       free (abscoeff);
       free (p);
       return 0;
     }

The output can be used with the GNU plotutils 'graph' program,

     $ ./a.out ecg.dat > dwt.txt
     $ graph -T ps -x 0 256 32 -h 0.3 -a dwt.txt > dwt.ps


File: gsl-ref.info,  Node: DWT References,  Prev: DWT Examples,  Up: Wavelet Transforms

32.5 References and Further Reading
===================================

The mathematical background to wavelet transforms is covered in the
original lectures by Daubechies,

     Ingrid Daubechies.  Ten Lectures on Wavelets.  'CBMS-NSF Regional
     Conference Series in Applied Mathematics' (1992), SIAM, ISBN
     0898712742.

An easy to read introduction to the subject with an emphasis on the
application of the wavelet transform in various branches of science is,

     Paul S. Addison.  'The Illustrated Wavelet Transform Handbook'.
     Institute of Physics Publishing (2002), ISBN 0750306920.

For extensive coverage of signal analysis by wavelets, wavelet packets
and local cosine bases see,

     S. G. Mallat.  'A wavelet tour of signal processing' (Second
     edition).  Academic Press (1999), ISBN 012466606X.

The concept of multiresolution analysis underlying the wavelet transform
is described in,

     S. G. Mallat.  Multiresolution Approximations and Wavelet
     Orthonormal Bases of L^2(R). 'Transactions of the American
     Mathematical Society', 315(1), 1989, 69-87.

     S. G. Mallat.  A Theory for Multiresolution Signal
     Decomposition--The Wavelet Representation.  'IEEE Transactions on
     Pattern Analysis and Machine Intelligence', 11, 1989, 674-693.

The coefficients for the individual wavelet families implemented by the
library can be found in the following papers,

     I. Daubechies.  Orthonormal Bases of Compactly Supported Wavelets.
     'Communications on Pure and Applied Mathematics', 41 (1988)
     909-996.

     A. Cohen, I. Daubechies, and J.-C. Feauveau.  Biorthogonal Bases of
     Compactly Supported Wavelets.  'Communications on Pure and Applied
     Mathematics', 45 (1992) 485-560.

The PhysioNet archive of physiological datasets can be found online at
<http://www.physionet.org/> and is described in the following paper,

     Goldberger et al.  PhysioBank, PhysioToolkit, and PhysioNet:
     Components of a New Research Resource for Complex Physiologic
     Signals.  'Circulation' 101(23):e215-e220 2000.


File: gsl-ref.info,  Node: Discrete Hankel Transforms,  Next: One dimensional Root-Finding,  Prev: Wavelet Transforms,  Up: Top

33 Discrete Hankel Transforms
*****************************

This chapter describes functions for performing Discrete Hankel
Transforms (DHTs).  The functions are declared in the header file
'gsl_dht.h'.

* Menu:

* Discrete Hankel Transform Definition::  
* Discrete Hankel Transform Functions::  
* Discrete Hankel Transform References::  


File: gsl-ref.info,  Node: Discrete Hankel Transform Definition,  Next: Discrete Hankel Transform Functions,  Up: Discrete Hankel Transforms

33.1 Definitions
================

The discrete Hankel transform acts on a vector of sampled data, where
the samples are assumed to have been taken at points related to the
zeros of a Bessel function of fixed order; compare this to the case of
the discrete Fourier transform, where samples are taken at points
related to the zeroes of the sine or cosine function.

   Starting with its definition, the Hankel transform (or Bessel
transform) of order \nu of a function f with \nu > -1/2 is defined as
(see Johnson, 1987 and Lemoine, 1994)
     F_\nu(u) = \int_0^\infty f(t) J_\nu(u t) t dt

If the integral exists, F_\nu is called the Hankel transformation of f.
The reverse transform is given by
     f(t) = \int_0^\infty F_\nu(u) J_\nu(u t) u du ,

where \int_0^\infty f(t) t^{1/2} dt must exist and be absolutely
convergent, and where f(t) satisfies Dirichlet's conditions (of limited
total fluctuations) in the interval [0,\infty].

   Now the discrete Hankel transform works on a discrete function f,
which is sampled on points n=1...M located at positions
t_n=(j_{\nu,n}/j_{\nu,M}) X in real space and at u_n=j_{\nu,n}/X in
reciprocal space.  Here, j_{\nu,m} are the m-th zeros of the Bessel
function J_\nu(x) arranged in ascending order.  Moreover, the discrete
functions are assumed to be band limited, so f(t_n)=0 and F(u_n)=0 for
n>M.  Accordingly, the function f is defined on the interval [0,X].

   Following the work of Johnson, 1987 and Lemoine, 1994, the discrete
Hankel transform is given by
     F_\nu(u_m) = (2 X^2 / j_(\nu,M)^2)
           \sum_{k=1}^{M-1} f(j_(\nu,k) X/j_(\nu,M))
               (J_\nu(j_(\nu,m) j_(\nu,k) / j_(\nu,M)) / J_(\nu+1)(j_(\nu,k))^2).

It is this discrete expression which defines the discrete Hankel
transform calculated by GSL. In GSL, forward and backward transforms are
defined equally and calculate F_\nu(u_m).  Following Johnson, the
backward transform reads
     f(t_k) = (2 / X^2)
           \sum_{m=1}^{M-1} F(j_(\nu,m)/X)
               (J_\nu(j_(\nu,m) j_(\nu,k) / j_(\nu,M)) / J_(\nu+1)(j_(\nu,m))^2).

Obviously, using the forward transform instead of the backward transform
gives an additional factor X^4/j_{\nu,M}^2=t_m^2/u_m^2.

   The kernel in the summation above defines the matrix of the
\nu-Hankel transform of size M-1.  The coefficients of this matrix,
being dependent on \nu and M, must be precomputed and stored; the
'gsl_dht' object encapsulates this data.  The allocation function
'gsl_dht_alloc' returns a 'gsl_dht' object which must be properly
initialized with 'gsl_dht_init' before it can be used to perform
transforms on data sample vectors, for fixed \nu and M, using the
'gsl_dht_apply' function.  The implementation allows to define the
length X of the fundamental interval, for convenience, while discrete
Hankel transforms are often defined on the unit interval instead of
[0,X].

   Notice that by assumption f(t) vanishes at the endpoints of the
interval, consistent with the inversion formula and the sampling formula
given above.  Therefore, this transform corresponds to an orthogonal
expansion in eigenfunctions of the Dirichlet problem for the Bessel
differential equation.


File: gsl-ref.info,  Node: Discrete Hankel Transform Functions,  Next: Discrete Hankel Transform References,  Prev: Discrete Hankel Transform Definition,  Up: Discrete Hankel Transforms

33.2 Functions
==============

 -- Function: gsl_dht * gsl_dht_alloc (size_t SIZE)
     This function allocates a Discrete Hankel transform object of size
     SIZE.

 -- Function: int gsl_dht_init (gsl_dht * T, double NU, double XMAX)
     This function initializes the transform T for the given values of
     NU and XMAX.

 -- Function: gsl_dht * gsl_dht_new (size_t SIZE, double NU, double
          XMAX)
     This function allocates a Discrete Hankel transform object of size
     SIZE and initializes it for the given values of NU and XMAX.

 -- Function: void gsl_dht_free (gsl_dht * T)
     This function frees the transform T.

 -- Function: int gsl_dht_apply (const gsl_dht * T, double * F_IN,
          double * F_OUT)
     This function applies the transform T to the array F_IN whose size
     is equal to the size of the transform.  The result is stored in the
     array F_OUT which must be of the same length.

     Applying this function to its output gives the original data
     multiplied by (1/j_(\nu,M))^2, up to numerical errors.

 -- Function: double gsl_dht_x_sample (const gsl_dht * T, int N)
     This function returns the value of the N-th sample point in the
     unit interval, (j_{\nu,n+1}/j_{\nu,M}) X.  These are the points
     where the function f(t) is assumed to be sampled.

 -- Function: double gsl_dht_k_sample (const gsl_dht * T, int N)
     This function returns the value of the N-th sample point in
     "k-space", j_{\nu,n+1}/X.


File: gsl-ref.info,  Node: Discrete Hankel Transform References,  Prev: Discrete Hankel Transform Functions,  Up: Discrete Hankel Transforms

33.3 References and Further Reading
===================================

The algorithms used by these functions are described in the following
papers,

     H. Fisk Johnson, Comp. Phys. Comm. 43, 181 (1987).

     D. Lemoine, J. Chem. Phys. 101, 3936 (1994).


File: gsl-ref.info,  Node: One dimensional Root-Finding,  Next: One dimensional Minimization,  Prev: Discrete Hankel Transforms,  Up: Top

34 One dimensional Root-Finding
*******************************

This chapter describes routines for finding roots of arbitrary
one-dimensional functions.  The library provides low level components
for a variety of iterative solvers and convergence tests.  These can be
combined by the user to achieve the desired solution, with full access
to the intermediate steps of the iteration.  Each class of methods uses
the same framework, so that you can switch between solvers at runtime
without needing to recompile your program.  Each instance of a solver
keeps track of its own state, allowing the solvers to be used in
multi-threaded programs.

   The header file 'gsl_roots.h' contains prototypes for the root
finding functions and related declarations.

* Menu:

* Root Finding Overview::       
* Root Finding Caveats::        
* Initializing the Solver::     
* Providing the function to solve::  
* Search Bounds and Guesses::   
* Root Finding Iteration::      
* Search Stopping Parameters::  
* Root Bracketing Algorithms::  
* Root Finding Algorithms using Derivatives::  
* Root Finding Examples::       
* Root Finding References and Further Reading::  


File: gsl-ref.info,  Node: Root Finding Overview,  Next: Root Finding Caveats,  Up: One dimensional Root-Finding

34.1 Overview
=============

One-dimensional root finding algorithms can be divided into two classes,
"root bracketing" and "root polishing".  Algorithms which proceed by
bracketing a root are guaranteed to converge.  Bracketing algorithms
begin with a bounded region known to contain a root.  The size of this
bounded region is reduced, iteratively, until it encloses the root to a
desired tolerance.  This provides a rigorous error estimate for the
location of the root.

   The technique of "root polishing" attempts to improve an initial
guess to the root.  These algorithms converge only if started "close
enough" to a root, and sacrifice a rigorous error bound for speed.  By
approximating the behavior of a function in the vicinity of a root they
attempt to find a higher order improvement of an initial guess.  When
the behavior of the function is compatible with the algorithm and a good
initial guess is available a polishing algorithm can provide rapid
convergence.

   In GSL both types of algorithm are available in similar frameworks.
The user provides a high-level driver for the algorithms, and the
library provides the individual functions necessary for each of the
steps.  There are three main phases of the iteration.  The steps are,

   * initialize solver state, S, for algorithm T

   * update S using the iteration T

   * test S for convergence, and repeat iteration if necessary

The state for bracketing solvers is held in a 'gsl_root_fsolver' struct.
The updating procedure uses only function evaluations (not derivatives).
The state for root polishing solvers is held in a 'gsl_root_fdfsolver'
struct.  The updates require both the function and its derivative (hence
the name 'fdf') to be supplied by the user.


File: gsl-ref.info,  Node: Root Finding Caveats,  Next: Initializing the Solver,  Prev: Root Finding Overview,  Up: One dimensional Root-Finding

34.2 Caveats
============

Note that root finding functions can only search for one root at a time.
When there are several roots in the search area, the first root to be
found will be returned; however it is difficult to predict which of the
roots this will be.  _In most cases, no error will be reported if you
try to find a root in an area where there is more than one._

   Care must be taken when a function may have a multiple root (such as
f(x) = (x-x_0)^2 or f(x) = (x-x_0)^3).  It is not possible to use
root-bracketing algorithms on even-multiplicity roots.  For these
algorithms the initial interval must contain a zero-crossing, where the
function is negative at one end of the interval and positive at the
other end.  Roots with even-multiplicity do not cross zero, but only
touch it instantaneously.  Algorithms based on root bracketing will
still work for odd-multiplicity roots (e.g.  cubic, quintic, ...).  Root
polishing algorithms generally work with higher multiplicity roots, but
at a reduced rate of convergence.  In these cases the "Steffenson
algorithm" can be used to accelerate the convergence of multiple roots.

   While it is not absolutely required that f have a root within the
search region, numerical root finding functions should not be used
haphazardly to check for the _existence_ of roots.  There are better
ways to do this.  Because it is easy to create situations where
numerical root finders can fail, it is a bad idea to throw a root finder
at a function you do not know much about.  In general it is best to
examine the function visually by plotting before searching for a root.


File: gsl-ref.info,  Node: Initializing the Solver,  Next: Providing the function to solve,  Prev: Root Finding Caveats,  Up: One dimensional Root-Finding

34.3 Initializing the Solver
============================

 -- Function: gsl_root_fsolver * gsl_root_fsolver_alloc (const
          gsl_root_fsolver_type * T)
     This function returns a pointer to a newly allocated instance of a
     solver of type T.  For example, the following code creates an
     instance of a bisection solver,

          const gsl_root_fsolver_type * T
            = gsl_root_fsolver_bisection;
          gsl_root_fsolver * s
            = gsl_root_fsolver_alloc (T);

     If there is insufficient memory to create the solver then the
     function returns a null pointer and the error handler is invoked
     with an error code of 'GSL_ENOMEM'.

 -- Function: gsl_root_fdfsolver * gsl_root_fdfsolver_alloc (const
          gsl_root_fdfsolver_type * T)
     This function returns a pointer to a newly allocated instance of a
     derivative-based solver of type T.  For example, the following code
     creates an instance of a Newton-Raphson solver,

          const gsl_root_fdfsolver_type * T
            = gsl_root_fdfsolver_newton;
          gsl_root_fdfsolver * s
            = gsl_root_fdfsolver_alloc (T);

     If there is insufficient memory to create the solver then the
     function returns a null pointer and the error handler is invoked
     with an error code of 'GSL_ENOMEM'.

 -- Function: int gsl_root_fsolver_set (gsl_root_fsolver * S,
          gsl_function * F, double X_LOWER, double X_UPPER)
     This function initializes, or reinitializes, an existing solver S
     to use the function F and the initial search interval [X_LOWER,
     X_UPPER].

 -- Function: int gsl_root_fdfsolver_set (gsl_root_fdfsolver * S,
          gsl_function_fdf * FDF, double ROOT)
     This function initializes, or reinitializes, an existing solver S
     to use the function and derivative FDF and the initial guess ROOT.

 -- Function: void gsl_root_fsolver_free (gsl_root_fsolver * S)
 -- Function: void gsl_root_fdfsolver_free (gsl_root_fdfsolver * S)
     These functions free all the memory associated with the solver S.

 -- Function: const char * gsl_root_fsolver_name (const gsl_root_fsolver
          * S)
 -- Function: const char * gsl_root_fdfsolver_name (const
          gsl_root_fdfsolver * S)
     These functions return a pointer to the name of the solver.  For
     example,

          printf ("s is a '%s' solver\n",
                  gsl_root_fsolver_name (s));

     would print something like 's is a 'bisection' solver'.


File: gsl-ref.info,  Node: Providing the function to solve,  Next: Search Bounds and Guesses,  Prev: Initializing the Solver,  Up: One dimensional Root-Finding

34.4 Providing the function to solve
====================================

You must provide a continuous function of one variable for the root
finders to operate on, and, sometimes, its first derivative.  In order
to allow for general parameters the functions are defined by the
following data types:

 -- Data Type: gsl_function
     This data type defines a general function with parameters.

     'double (* function) (double X, void * PARAMS)'
          this function should return the value f(x,params) for argument
          X and parameters PARAMS

     'void * params'
          a pointer to the parameters of the function

   Here is an example for the general quadratic function,

     f(x) = a x^2 + b x + c

with a = 3, b = 2, c = 1.  The following code defines a 'gsl_function'
'F' which you could pass to a root finder as a function pointer:

     struct my_f_params { double a; double b; double c; };

     double
     my_f (double x, void * p) {
        struct my_f_params * params
          = (struct my_f_params *)p;
        double a = (params->a);
        double b = (params->b);
        double c = (params->c);

        return  (a * x + b) * x + c;
     }

     gsl_function F;
     struct my_f_params params = { 3.0, 2.0, 1.0 };

     F.function = &my_f;
     F.params = &params;

The function f(x) can be evaluated using the macro 'GSL_FN_EVAL(&F,x)'
defined in 'gsl_math.h'.

 -- Data Type: gsl_function_fdf
     This data type defines a general function with parameters and its
     first derivative.

     'double (* f) (double X, void * PARAMS)'
          this function should return the value of f(x,params) for
          argument X and parameters PARAMS

     'double (* df) (double X, void * PARAMS)'
          this function should return the value of the derivative of F
          with respect to X, f'(x,params), for argument X and parameters
          PARAMS

     'void (* fdf) (double X, void * PARAMS, double * F, double * DF)'
          this function should set the values of the function F to
          f(x,params) and its derivative DF to f'(x,params) for argument
          X and parameters PARAMS.  This function provides an
          optimization of the separate functions for f(x) and f'(x)--it
          is always faster to compute the function and its derivative at
          the same time.

     'void * params'
          a pointer to the parameters of the function

   Here is an example where f(x) = 2\exp(2x):

     double
     my_f (double x, void * params)
     {
        return exp (2 * x);
     }

     double
     my_df (double x, void * params)
     {
        return 2 * exp (2 * x);
     }

     void
     my_fdf (double x, void * params,
             double * f, double * df)
     {
        double t = exp (2 * x);

        *f = t;
        *df = 2 * t;   /* uses existing value */
     }

     gsl_function_fdf FDF;

     FDF.f = &my_f;
     FDF.df = &my_df;
     FDF.fdf = &my_fdf;
     FDF.params = 0;

The function f(x) can be evaluated using the macro
'GSL_FN_FDF_EVAL_F(&FDF,x)' and the derivative f'(x) can be evaluated
using the macro 'GSL_FN_FDF_EVAL_DF(&FDF,x)'.  Both the function y =
f(x) and its derivative dy = f'(x) can be evaluated at the same time
using the macro 'GSL_FN_FDF_EVAL_F_DF(&FDF,x,y,dy)'.  The macro stores
f(x) in its Y argument and f'(x) in its DY argument--both of these
should be pointers to 'double'.


File: gsl-ref.info,  Node: Search Bounds and Guesses,  Next: Root Finding Iteration,  Prev: Providing the function to solve,  Up: One dimensional Root-Finding

34.5 Search Bounds and Guesses
==============================

You provide either search bounds or an initial guess; this section
explains how search bounds and guesses work and how function arguments
control them.

   A guess is simply an x value which is iterated until it is within the
desired precision of a root.  It takes the form of a 'double'.

   Search bounds are the endpoints of an interval which is iterated
until the length of the interval is smaller than the requested
precision.  The interval is defined by two values, the lower limit and
the upper limit.  Whether the endpoints are intended to be included in
the interval or not depends on the context in which the interval is
used.


File: gsl-ref.info,  Node: Root Finding Iteration,  Next: Search Stopping Parameters,  Prev: Search Bounds and Guesses,  Up: One dimensional Root-Finding

34.6 Iteration
==============

The following functions drive the iteration of each algorithm.  Each
function performs one iteration to update the state of any solver of the
corresponding type.  The same functions work for all solvers so that
different methods can be substituted at runtime without modifications to
the code.

 -- Function: int gsl_root_fsolver_iterate (gsl_root_fsolver * S)
 -- Function: int gsl_root_fdfsolver_iterate (gsl_root_fdfsolver * S)
     These functions perform a single iteration of the solver S.  If the
     iteration encounters an unexpected problem then an error code will
     be returned,

     'GSL_EBADFUNC'
          the iteration encountered a singular point where the function
          or its derivative evaluated to 'Inf' or 'NaN'.

     'GSL_EZERODIV'
          the derivative of the function vanished at the iteration
          point, preventing the algorithm from continuing without a
          division by zero.

   The solver maintains a current best estimate of the root at all
times.  The bracketing solvers also keep track of the current best
interval bounding the root.  This information can be accessed with the
following auxiliary functions,

 -- Function: double gsl_root_fsolver_root (const gsl_root_fsolver * S)
 -- Function: double gsl_root_fdfsolver_root (const gsl_root_fdfsolver *
          S)
     These functions return the current estimate of the root for the
     solver S.

 -- Function: double gsl_root_fsolver_x_lower (const gsl_root_fsolver *
          S)
 -- Function: double gsl_root_fsolver_x_upper (const gsl_root_fsolver *
          S)
     These functions return the current bracketing interval for the
     solver S.


File: gsl-ref.info,  Node: Search Stopping Parameters,  Next: Root Bracketing Algorithms,  Prev: Root Finding Iteration,  Up: One dimensional Root-Finding

34.7 Search Stopping Parameters
===============================

A root finding procedure should stop when one of the following
conditions is true:

   * A root has been found to within the user-specified precision.

   * A user-specified maximum number of iterations has been reached.

   * An error has occurred.

The handling of these conditions is under user control.  The functions
below allow the user to test the precision of the current result in
several standard ways.

 -- Function: int gsl_root_test_interval (double X_LOWER, double
          X_UPPER, double EPSABS, double EPSREL)
     This function tests for the convergence of the interval [X_LOWER,
     X_UPPER] with absolute error EPSABS and relative error EPSREL.  The
     test returns 'GSL_SUCCESS' if the following condition is achieved,

          |a - b| < epsabs + epsrel min(|a|,|b|)

     when the interval x = [a,b] does not include the origin.  If the
     interval includes the origin then \min(|a|,|b|) is replaced by zero
     (which is the minimum value of |x| over the interval).  This
     ensures that the relative error is accurately estimated for roots
     close to the origin.

     This condition on the interval also implies that any estimate of
     the root r in the interval satisfies the same condition with
     respect to the true root r^*,

          |r - r^*| < epsabs + epsrel r^*

     assuming that the true root r^* is contained within the interval.

 -- Function: int gsl_root_test_delta (double X1, double X0, double
          EPSABS, double EPSREL)

     This function tests for the convergence of the sequence ..., X0, X1
     with absolute error EPSABS and relative error EPSREL.  The test
     returns 'GSL_SUCCESS' if the following condition is achieved,

          |x_1 - x_0| < epsabs + epsrel |x_1|

     and returns 'GSL_CONTINUE' otherwise.

 -- Function: int gsl_root_test_residual (double F, double EPSABS)
     This function tests the residual value F against the absolute error
     bound EPSABS.  The test returns 'GSL_SUCCESS' if the following
     condition is achieved,

          |f| < epsabs

     and returns 'GSL_CONTINUE' otherwise.  This criterion is suitable
     for situations where the precise location of the root, x, is
     unimportant provided a value can be found where the residual,
     |f(x)|, is small enough.


File: gsl-ref.info,  Node: Root Bracketing Algorithms,  Next: Root Finding Algorithms using Derivatives,  Prev: Search Stopping Parameters,  Up: One dimensional Root-Finding

34.8 Root Bracketing Algorithms
===============================

The root bracketing algorithms described in this section require an
initial interval which is guaranteed to contain a root--if a and b are
the endpoints of the interval then f(a) must differ in sign from f(b).
This ensures that the function crosses zero at least once in the
interval.  If a valid initial interval is used then these algorithm
cannot fail, provided the function is well-behaved.

   Note that a bracketing algorithm cannot find roots of even degree,
since these do not cross the x-axis.

 -- Solver: gsl_root_fsolver_bisection

     The "bisection algorithm" is the simplest method of bracketing the
     roots of a function.  It is the slowest algorithm provided by the
     library, with linear convergence.

     On each iteration, the interval is bisected and the value of the
     function at the midpoint is calculated.  The sign of this value is
     used to determine which half of the interval does not contain a
     root.  That half is discarded to give a new, smaller interval
     containing the root.  This procedure can be continued indefinitely
     until the interval is sufficiently small.

     At any time the current estimate of the root is taken as the
     midpoint of the interval.

 -- Solver: gsl_root_fsolver_falsepos

     The "false position algorithm" is a method of finding roots based
     on linear interpolation.  Its convergence is linear, but it is
     usually faster than bisection.

     On each iteration a line is drawn between the endpoints (a,f(a))
     and (b,f(b)) and the point where this line crosses the x-axis taken
     as a "midpoint".  The value of the function at this point is
     calculated and its sign is used to determine which side of the
     interval does not contain a root.  That side is discarded to give a
     new, smaller interval containing the root.  This procedure can be
     continued indefinitely until the interval is sufficiently small.

     The best estimate of the root is taken from the linear
     interpolation of the interval on the current iteration.

 -- Solver: gsl_root_fsolver_brent

     The "Brent-Dekker method" (referred to here as "Brent's method")
     combines an interpolation strategy with the bisection algorithm.
     This produces a fast algorithm which is still robust.

     On each iteration Brent's method approximates the function using an
     interpolating curve.  On the first iteration this is a linear
     interpolation of the two endpoints.  For subsequent iterations the
     algorithm uses an inverse quadratic fit to the last three points,
     for higher accuracy.  The intercept of the interpolating curve with
     the x-axis is taken as a guess for the root.  If it lies within the
     bounds of the current interval then the interpolating point is
     accepted, and used to generate a smaller interval.  If the
     interpolating point is not accepted then the algorithm falls back
     to an ordinary bisection step.

     The best estimate of the root is taken from the most recent
     interpolation or bisection.


File: gsl-ref.info,  Node: Root Finding Algorithms using Derivatives,  Next: Root Finding Examples,  Prev: Root Bracketing Algorithms,  Up: One dimensional Root-Finding

34.9 Root Finding Algorithms using Derivatives
==============================================

The root polishing algorithms described in this section require an
initial guess for the location of the root.  There is no absolute
guarantee of convergence--the function must be suitable for this
technique and the initial guess must be sufficiently close to the root
for it to work.  When these conditions are satisfied then convergence is
quadratic.

   These algorithms make use of both the function and its derivative.

 -- Derivative Solver: gsl_root_fdfsolver_newton

     Newton's Method is the standard root-polishing algorithm.  The
     algorithm begins with an initial guess for the location of the
     root.  On each iteration, a line tangent to the function f is drawn
     at that position.  The point where this line crosses the x-axis
     becomes the new guess.  The iteration is defined by the following
     sequence,

          x_{i+1} = x_i - f(x_i)/f'(x_i)

     Newton's method converges quadratically for single roots, and
     linearly for multiple roots.

 -- Derivative Solver: gsl_root_fdfsolver_secant

     The "secant method" is a simplified version of Newton's method
     which does not require the computation of the derivative on every
     step.

     On its first iteration the algorithm begins with Newton's method,
     using the derivative to compute a first step,

          x_1 = x_0 - f(x_0)/f'(x_0)

     Subsequent iterations avoid the evaluation of the derivative by
     replacing it with a numerical estimate, the slope of the line
     through the previous two points,

          x_{i+1} = x_i f(x_i) / f'_{est} where
           f'_{est} = (f(x_i) - f(x_{i-1})/(x_i - x_{i-1})

     When the derivative does not change significantly in the vicinity
     of the root the secant method gives a useful saving.
     Asymptotically the secant method is faster than Newton's method
     whenever the cost of evaluating the derivative is more than 0.44
     times the cost of evaluating the function itself.  As with all
     methods of computing a numerical derivative the estimate can suffer
     from cancellation errors if the separation of the points becomes
     too small.

     On single roots, the method has a convergence of order (1 + \sqrt
     5)/2 (approximately 1.62).  It converges linearly for multiple
     roots.

 -- Derivative Solver: gsl_root_fdfsolver_steffenson

     The "Steffenson Method"(1) provides the fastest convergence of all
     the routines.  It combines the basic Newton algorithm with an
     Aitken "delta-squared" acceleration.  If the Newton iterates are
     x_i then the acceleration procedure generates a new sequence R_i,

          R_i = x_i - (x_{i+1} - x_i)^2 / (x_{i+2} - 2 x_{i+1} + x_{i})

     which converges faster than the original sequence under reasonable
     conditions.  The new sequence requires three terms before it can
     produce its first value so the method returns accelerated values on
     the second and subsequent iterations.  On the first iteration it
     returns the ordinary Newton estimate.  The Newton iterate is also
     returned if the denominator of the acceleration term ever becomes
     zero.

     As with all acceleration procedures this method can become unstable
     if the function is not well-behaved.

   ---------- Footnotes ----------

   (1) J.F. Steffensen (1873-1961).  The spelling used in the name of
the function is slightly incorrect, but has been preserved to avoid
incompatibility.


File: gsl-ref.info,  Node: Root Finding Examples,  Next: Root Finding References and Further Reading,  Prev: Root Finding Algorithms using Derivatives,  Up: One dimensional Root-Finding

34.10 Examples
==============

For any root finding algorithm we need to prepare the function to be
solved.  For this example we will use the general quadratic equation
described earlier.  We first need a header file ('demo_fn.h') to define
the function parameters,

     struct quadratic_params
       {
         double a, b, c;
       };

     double quadratic (double x, void *params);
     double quadratic_deriv (double x, void *params);
     void quadratic_fdf (double x, void *params,
                         double *y, double *dy);

We place the function definitions in a separate file ('demo_fn.c'),

     double
     quadratic (double x, void *params)
     {
       struct quadratic_params *p
         = (struct quadratic_params *) params;

       double a = p->a;
       double b = p->b;
       double c = p->c;

       return (a * x + b) * x + c;
     }

     double
     quadratic_deriv (double x, void *params)
     {
       struct quadratic_params *p
         = (struct quadratic_params *) params;

       double a = p->a;
       double b = p->b;

       return 2.0 * a * x + b;
     }

     void
     quadratic_fdf (double x, void *params,
                    double *y, double *dy)
     {
       struct quadratic_params *p
         = (struct quadratic_params *) params;

       double a = p->a;
       double b = p->b;
       double c = p->c;

       *y = (a * x + b) * x + c;
       *dy = 2.0 * a * x + b;
     }

The first program uses the function solver 'gsl_root_fsolver_brent' for
Brent's method and the general quadratic defined above to solve the
following equation,

     x^2 - 5 = 0

with solution x = \sqrt 5 = 2.236068...

     #include <stdio.h>
     #include <gsl/gsl_errno.h>
     #include <gsl/gsl_math.h>
     #include <gsl/gsl_roots.h>

     #include "demo_fn.h"
     #include "demo_fn.c"

     int
     main (void)
     {
       int status;
       int iter = 0, max_iter = 100;
       const gsl_root_fsolver_type *T;
       gsl_root_fsolver *s;
       double r = 0, r_expected = sqrt (5.0);
       double x_lo = 0.0, x_hi = 5.0;
       gsl_function F;
       struct quadratic_params params = {1.0, 0.0, -5.0};

       F.function = &quadratic;
       F.params = &params;

       T = gsl_root_fsolver_brent;
       s = gsl_root_fsolver_alloc (T);
       gsl_root_fsolver_set (s, &F, x_lo, x_hi);

       printf ("using %s method\n",
               gsl_root_fsolver_name (s));

       printf ("%5s [%9s, %9s] %9s %10s %9s\n",
               "iter", "lower", "upper", "root",
               "err", "err(est)");

       do
         {
           iter++;
           status = gsl_root_fsolver_iterate (s);
           r = gsl_root_fsolver_root (s);
           x_lo = gsl_root_fsolver_x_lower (s);
           x_hi = gsl_root_fsolver_x_upper (s);
           status = gsl_root_test_interval (x_lo, x_hi,
                                            0, 0.001);

           if (status == GSL_SUCCESS)
             printf ("Converged:\n");

           printf ("%5d [%.7f, %.7f] %.7f %+.7f %.7f\n",
                   iter, x_lo, x_hi,
                   r, r - r_expected,
                   x_hi - x_lo);
         }
       while (status == GSL_CONTINUE && iter < max_iter);

       gsl_root_fsolver_free (s);

       return status;
     }

Here are the results of the iterations,

     $ ./a.out
     using brent method
      iter [    lower,     upper]      root        err  err(est)
         1 [1.0000000, 5.0000000] 1.0000000 -1.2360680 4.0000000
         2 [1.0000000, 3.0000000] 3.0000000 +0.7639320 2.0000000
         3 [2.0000000, 3.0000000] 2.0000000 -0.2360680 1.0000000
         4 [2.2000000, 3.0000000] 2.2000000 -0.0360680 0.8000000
         5 [2.2000000, 2.2366300] 2.2366300 +0.0005621 0.0366300
     Converged:
         6 [2.2360634, 2.2366300] 2.2360634 -0.0000046 0.0005666

If the program is modified to use the bisection solver instead of
Brent's method, by changing 'gsl_root_fsolver_brent' to
'gsl_root_fsolver_bisection' the slower convergence of the Bisection
method can be observed,

     $ ./a.out
     using bisection method
      iter [    lower,     upper]      root        err  err(est)
         1 [0.0000000, 2.5000000] 1.2500000 -0.9860680 2.5000000
         2 [1.2500000, 2.5000000] 1.8750000 -0.3610680 1.2500000
         3 [1.8750000, 2.5000000] 2.1875000 -0.0485680 0.6250000
         4 [2.1875000, 2.5000000] 2.3437500 +0.1076820 0.3125000
         5 [2.1875000, 2.3437500] 2.2656250 +0.0295570 0.1562500
         6 [2.1875000, 2.2656250] 2.2265625 -0.0095055 0.0781250
         7 [2.2265625, 2.2656250] 2.2460938 +0.0100258 0.0390625
         8 [2.2265625, 2.2460938] 2.2363281 +0.0002601 0.0195312
         9 [2.2265625, 2.2363281] 2.2314453 -0.0046227 0.0097656
        10 [2.2314453, 2.2363281] 2.2338867 -0.0021813 0.0048828
        11 [2.2338867, 2.2363281] 2.2351074 -0.0009606 0.0024414
     Converged:
        12 [2.2351074, 2.2363281] 2.2357178 -0.0003502 0.0012207

   The next program solves the same function using a derivative solver
instead.

     #include <stdio.h>
     #include <gsl/gsl_errno.h>
     #include <gsl/gsl_math.h>
     #include <gsl/gsl_roots.h>

     #include "demo_fn.h"
     #include "demo_fn.c"

     int
     main (void)
     {
       int status;
       int iter = 0, max_iter = 100;
       const gsl_root_fdfsolver_type *T;
       gsl_root_fdfsolver *s;
       double x0, x = 5.0, r_expected = sqrt (5.0);
       gsl_function_fdf FDF;
       struct quadratic_params params = {1.0, 0.0, -5.0};

       FDF.f = &quadratic;
       FDF.df = &quadratic_deriv;
       FDF.fdf = &quadratic_fdf;
       FDF.params = &params;

       T = gsl_root_fdfsolver_newton;
       s = gsl_root_fdfsolver_alloc (T);
       gsl_root_fdfsolver_set (s, &FDF, x);

       printf ("using %s method\n",
               gsl_root_fdfsolver_name (s));

       printf ("%-5s %10s %10s %10s\n",
               "iter", "root", "err", "err(est)");
       do
         {
           iter++;
           status = gsl_root_fdfsolver_iterate (s);
           x0 = x;
           x = gsl_root_fdfsolver_root (s);
           status = gsl_root_test_delta (x, x0, 0, 1e-3);

           if (status == GSL_SUCCESS)
             printf ("Converged:\n");

           printf ("%5d %10.7f %+10.7f %10.7f\n",
                   iter, x, x - r_expected, x - x0);
         }
       while (status == GSL_CONTINUE && iter < max_iter);

       gsl_root_fdfsolver_free (s);
       return status;
     }

Here are the results for Newton's method,

     $ ./a.out
     using newton method
     iter        root        err   err(est)
         1  3.0000000 +0.7639320 -2.0000000
         2  2.3333333 +0.0972654 -0.6666667
         3  2.2380952 +0.0020273 -0.0952381
     Converged:
         4  2.2360689 +0.0000009 -0.0020263

Note that the error can be estimated more accurately by taking the
difference between the current iterate and next iterate rather than the
previous iterate.  The other derivative solvers can be investigated by
changing 'gsl_root_fdfsolver_newton' to 'gsl_root_fdfsolver_secant' or
'gsl_root_fdfsolver_steffenson'.


File: gsl-ref.info,  Node: Root Finding References and Further Reading,  Prev: Root Finding Examples,  Up: One dimensional Root-Finding

34.11 References and Further Reading
====================================

For information on the Brent-Dekker algorithm see the following two
papers,

     R. P. Brent, "An algorithm with guaranteed convergence for finding
     a zero of a function", 'Computer Journal', 14 (1971) 422-425

     J. C. P. Bus and T. J. Dekker, "Two Efficient Algorithms with
     Guaranteed Convergence for Finding a Zero of a Function", 'ACM
     Transactions of Mathematical Software', Vol. 1 No. 4 (1975) 330-345


File: gsl-ref.info,  Node: One dimensional Minimization,  Next: Multidimensional Root-Finding,  Prev: One dimensional Root-Finding,  Up: Top

35 One dimensional Minimization
*******************************

This chapter describes routines for finding minima of arbitrary
one-dimensional functions.  The library provides low level components
for a variety of iterative minimizers and convergence tests.  These can
be combined by the user to achieve the desired solution, with full
access to the intermediate steps of the algorithms.  Each class of
methods uses the same framework, so that you can switch between
minimizers at runtime without needing to recompile your program.  Each
instance of a minimizer keeps track of its own state, allowing the
minimizers to be used in multi-threaded programs.

   The header file 'gsl_min.h' contains prototypes for the minimization
functions and related declarations.  To use the minimization algorithms
to find the maximum of a function simply invert its sign.

* Menu:

* Minimization Overview::       
* Minimization Caveats::        
* Initializing the Minimizer::  
* Providing the function to minimize::  
* Minimization Iteration::      
* Minimization Stopping Parameters::  
* Minimization Algorithms::     
* Minimization Examples::       
* Minimization References and Further Reading::  


File: gsl-ref.info,  Node: Minimization Overview,  Next: Minimization Caveats,  Up: One dimensional Minimization

35.1 Overview
=============

The minimization algorithms begin with a bounded region known to contain
a minimum.  The region is described by a lower bound a and an upper
bound b, with an estimate of the location of the minimum x.

The value of the function at x must be less than the value of the
function at the ends of the interval,

     f(a) > f(x) < f(b)

This condition guarantees that a minimum is contained somewhere within
the interval.  On each iteration a new point x' is selected using one of
the available algorithms.  If the new point is a better estimate of the
minimum, i.e. where f(x') < f(x), then the current estimate of the
minimum x is updated.  The new point also allows the size of the bounded
interval to be reduced, by choosing the most compact set of points which
satisfies the constraint f(a) > f(x) < f(b).  The interval is reduced
until it encloses the true minimum to a desired tolerance.  This
provides a best estimate of the location of the minimum and a rigorous
error estimate.

   Several bracketing algorithms are available within a single
framework.  The user provides a high-level driver for the algorithm, and
the library provides the individual functions necessary for each of the
steps.  There are three main phases of the iteration.  The steps are,

   * initialize minimizer state, S, for algorithm T

   * update S using the iteration T

   * test S for convergence, and repeat iteration if necessary

The state for the minimizers is held in a 'gsl_min_fminimizer' struct.
The updating procedure uses only function evaluations (not derivatives).


File: gsl-ref.info,  Node: Minimization Caveats,  Next: Initializing the Minimizer,  Prev: Minimization Overview,  Up: One dimensional Minimization

35.2 Caveats
============

Note that minimization functions can only search for one minimum at a
time.  When there are several minima in the search area, the first
minimum to be found will be returned; however it is difficult to predict
which of the minima this will be.  _In most cases, no error will be
reported if you try to find a minimum in an area where there is more
than one._

   With all minimization algorithms it can be difficult to determine the
location of the minimum to full numerical precision.  The behavior of
the function in the region of the minimum x^* can be approximated by a
Taylor expansion,

     y = f(x^*) + (1/2) f''(x^*) (x - x^*)^2

and the second term of this expansion can be lost when added to the
first term at finite precision.  This magnifies the error in locating
x^*, making it proportional to \sqrt \epsilon (where \epsilon is the
relative accuracy of the floating point numbers).  For functions with
higher order minima, such as x^4, the magnification of the error is
correspondingly worse.  The best that can be achieved is to converge to
the limit of numerical accuracy in the function values, rather than the
location of the minimum itself.


File: gsl-ref.info,  Node: Initializing the Minimizer,  Next: Providing the function to minimize,  Prev: Minimization Caveats,  Up: One dimensional Minimization

35.3 Initializing the Minimizer
===============================

 -- Function: gsl_min_fminimizer * gsl_min_fminimizer_alloc (const
          gsl_min_fminimizer_type * T)
     This function returns a pointer to a newly allocated instance of a
     minimizer of type T.  For example, the following code creates an
     instance of a golden section minimizer,

          const gsl_min_fminimizer_type * T
            = gsl_min_fminimizer_goldensection;
          gsl_min_fminimizer * s
            = gsl_min_fminimizer_alloc (T);

     If there is insufficient memory to create the minimizer then the
     function returns a null pointer and the error handler is invoked
     with an error code of 'GSL_ENOMEM'.

 -- Function: int gsl_min_fminimizer_set (gsl_min_fminimizer * S,
          gsl_function * F, double X_MINIMUM, double X_LOWER, double
          X_UPPER)
     This function sets, or resets, an existing minimizer S to use the
     function F and the initial search interval [X_LOWER, X_UPPER], with
     a guess for the location of the minimum X_MINIMUM.

     If the interval given does not contain a minimum, then the function
     returns an error code of 'GSL_EINVAL'.

 -- Function: int gsl_min_fminimizer_set_with_values (gsl_min_fminimizer
          * S, gsl_function * F, double X_MINIMUM, double F_MINIMUM,
          double X_LOWER, double F_LOWER, double X_UPPER, double
          F_UPPER)
     This function is equivalent to 'gsl_min_fminimizer_set' but uses
     the values F_MINIMUM, F_LOWER and F_UPPER instead of computing
     'f(x_minimum)', 'f(x_lower)' and 'f(x_upper)'.

 -- Function: void gsl_min_fminimizer_free (gsl_min_fminimizer * S)
     This function frees all the memory associated with the minimizer S.

 -- Function: const char * gsl_min_fminimizer_name (const
          gsl_min_fminimizer * S)
     This function returns a pointer to the name of the minimizer.  For
     example,

          printf ("s is a '%s' minimizer\n",
                  gsl_min_fminimizer_name (s));

     would print something like 's is a 'brent' minimizer'.


File: gsl-ref.info,  Node: Providing the function to minimize,  Next: Minimization Iteration,  Prev: Initializing the Minimizer,  Up: One dimensional Minimization

35.4 Providing the function to minimize
=======================================

You must provide a continuous function of one variable for the
minimizers to operate on.  In order to allow for general parameters the
functions are defined by a 'gsl_function' data type (*note Providing the
function to solve::).


File: gsl-ref.info,  Node: Minimization Iteration,  Next: Minimization Stopping Parameters,  Prev: Providing the function to minimize,  Up: One dimensional Minimization

35.5 Iteration
==============

The following functions drive the iteration of each algorithm.  Each
function performs one iteration to update the state of any minimizer of
the corresponding type.  The same functions work for all minimizers so
that different methods can be substituted at runtime without
modifications to the code.

 -- Function: int gsl_min_fminimizer_iterate (gsl_min_fminimizer * S)
     This function performs a single iteration of the minimizer S.  If
     the iteration encounters an unexpected problem then an error code
     will be returned,

     'GSL_EBADFUNC'
          the iteration encountered a singular point where the function
          evaluated to 'Inf' or 'NaN'.

     'GSL_FAILURE'
          the algorithm could not improve the current best approximation
          or bounding interval.

   The minimizer maintains a current best estimate of the position of
the minimum at all times, and the current interval bounding the minimum.
This information can be accessed with the following auxiliary functions,

 -- Function: double gsl_min_fminimizer_x_minimum (const
          gsl_min_fminimizer * S)
     This function returns the current estimate of the position of the
     minimum for the minimizer S.

 -- Function: double gsl_min_fminimizer_x_upper (const
          gsl_min_fminimizer * S)
 -- Function: double gsl_min_fminimizer_x_lower (const
          gsl_min_fminimizer * S)
     These functions return the current upper and lower bound of the
     interval for the minimizer S.

 -- Function: double gsl_min_fminimizer_f_minimum (const
          gsl_min_fminimizer * S)
 -- Function: double gsl_min_fminimizer_f_upper (const
          gsl_min_fminimizer * S)
 -- Function: double gsl_min_fminimizer_f_lower (const
          gsl_min_fminimizer * S)
     These functions return the value of the function at the current
     estimate of the minimum and at the upper and lower bounds of the
     interval for the minimizer S.


File: gsl-ref.info,  Node: Minimization Stopping Parameters,  Next: Minimization Algorithms,  Prev: Minimization Iteration,  Up: One dimensional Minimization

35.6 Stopping Parameters
========================

A minimization procedure should stop when one of the following
conditions is true:

   * A minimum has been found to within the user-specified precision.

   * A user-specified maximum number of iterations has been reached.

   * An error has occurred.

The handling of these conditions is under user control.  The function
below allows the user to test the precision of the current result.

 -- Function: int gsl_min_test_interval (double X_LOWER, double X_UPPER,
          double EPSABS, double EPSREL)
     This function tests for the convergence of the interval [X_LOWER,
     X_UPPER] with absolute error EPSABS and relative error EPSREL.  The
     test returns 'GSL_SUCCESS' if the following condition is achieved,

          |a - b| < epsabs + epsrel min(|a|,|b|)

     when the interval x = [a,b] does not include the origin.  If the
     interval includes the origin then \min(|a|,|b|) is replaced by zero
     (which is the minimum value of |x| over the interval).  This
     ensures that the relative error is accurately estimated for minima
     close to the origin.

     This condition on the interval also implies that any estimate of
     the minimum x_m in the interval satisfies the same condition with
     respect to the true minimum x_m^*,

          |x_m - x_m^*| < epsabs + epsrel x_m^*

     assuming that the true minimum x_m^* is contained within the
     interval.


File: gsl-ref.info,  Node: Minimization Algorithms,  Next: Minimization Examples,  Prev: Minimization Stopping Parameters,  Up: One dimensional Minimization

35.7 Minimization Algorithms
============================

The minimization algorithms described in this section require an initial
interval which is guaranteed to contain a minimum--if a and b are the
endpoints of the interval and x is an estimate of the minimum then f(a)
> f(x) < f(b).  This ensures that the function has at least one minimum
somewhere in the interval.  If a valid initial interval is used then
these algorithm cannot fail, provided the function is well-behaved.

 -- Minimizer: gsl_min_fminimizer_goldensection

     The "golden section algorithm" is the simplest method of bracketing
     the minimum of a function.  It is the slowest algorithm provided by
     the library, with linear convergence.

     On each iteration, the algorithm first compares the subintervals
     from the endpoints to the current minimum.  The larger subinterval
     is divided in a golden section (using the famous ratio (3-\sqrt
     5)/2 = 0.3819660...) and the value of the function at this new
     point is calculated.  The new value is used with the constraint
     f(a') > f(x') < f(b') to a select new interval containing the
     minimum, by discarding the least useful point.  This procedure can
     be continued indefinitely until the interval is sufficiently small.
     Choosing the golden section as the bisection ratio can be shown to
     provide the fastest convergence for this type of algorithm.

 -- Minimizer: gsl_min_fminimizer_brent

     The "Brent minimization algorithm" combines a parabolic
     interpolation with the golden section algorithm.  This produces a
     fast algorithm which is still robust.

     The outline of the algorithm can be summarized as follows: on each
     iteration Brent's method approximates the function using an
     interpolating parabola through three existing points.  The minimum
     of the parabola is taken as a guess for the minimum.  If it lies
     within the bounds of the current interval then the interpolating
     point is accepted, and used to generate a smaller interval.  If the
     interpolating point is not accepted then the algorithm falls back
     to an ordinary golden section step.  The full details of Brent's
     method include some additional checks to improve convergence.

 -- Minimizer: gsl_min_fminimizer_quad_golden
     This is a variant of Brent's algorithm which uses the safeguarded
     step-length algorithm of Gill and Murray.


File: gsl-ref.info,  Node: Minimization Examples,  Next: Minimization References and Further Reading,  Prev: Minimization Algorithms,  Up: One dimensional Minimization

35.8 Examples
=============

The following program uses the Brent algorithm to find the minimum of
the function f(x) = \cos(x) + 1, which occurs at x = \pi.  The starting
interval is (0,6), with an initial guess for the minimum of 2.

     #include <stdio.h>
     #include <gsl/gsl_errno.h>
     #include <gsl/gsl_math.h>
     #include <gsl/gsl_min.h>

     double fn1 (double x, void * params)
     {
       (void)(params); /* avoid unused parameter warning */
       return cos(x) + 1.0;
     }

     int
     main (void)
     {
       int status;
       int iter = 0, max_iter = 100;
       const gsl_min_fminimizer_type *T;
       gsl_min_fminimizer *s;
       double m = 2.0, m_expected = M_PI;
       double a = 0.0, b = 6.0;
       gsl_function F;

       F.function = &fn1;
       F.params = 0;

       T = gsl_min_fminimizer_brent;
       s = gsl_min_fminimizer_alloc (T);
       gsl_min_fminimizer_set (s, &F, m, a, b);

       printf ("using %s method\n",
               gsl_min_fminimizer_name (s));

       printf ("%5s [%9s, %9s] %9s %10s %9s\n",
               "iter", "lower", "upper", "min",
               "err", "err(est)");

       printf ("%5d [%.7f, %.7f] %.7f %+.7f %.7f\n",
               iter, a, b,
               m, m - m_expected, b - a);

       do
         {
           iter++;
           status = gsl_min_fminimizer_iterate (s);

           m = gsl_min_fminimizer_x_minimum (s);
           a = gsl_min_fminimizer_x_lower (s);
           b = gsl_min_fminimizer_x_upper (s);

           status
             = gsl_min_test_interval (a, b, 0.001, 0.0);

           if (status == GSL_SUCCESS)
             printf ("Converged:\n");

           printf ("%5d [%.7f, %.7f] "
                   "%.7f %+.7f %.7f\n",
                   iter, a, b,
                   m, m - m_expected, b - a);
         }
       while (status == GSL_CONTINUE && iter < max_iter);

       gsl_min_fminimizer_free (s);

       return status;
     }

Here are the results of the minimization procedure.

     $ ./a.out
     using brent method
      iter [    lower,     upper]       min        err  err(est)
         0 [0.0000000, 6.0000000] 2.0000000 -1.1415927 6.0000000
         1 [2.0000000, 6.0000000] 3.5278640 +0.3862713 4.0000000
         2 [2.0000000, 3.5278640] 3.1748217 +0.0332290 1.5278640
         3 [2.0000000, 3.1748217] 3.1264576 -0.0151351 1.1748217
         4 [3.1264576, 3.1748217] 3.1414743 -0.0001183 0.0483641
         5 [3.1414743, 3.1748217] 3.1415930 +0.0000004 0.0333474
     Converged:
         6 [3.1414743, 3.1415930] 3.1415927 +0.0000000 0.0001187


File: gsl-ref.info,  Node: Minimization References and Further Reading,  Prev: Minimization Examples,  Up: One dimensional Minimization

35.9 References and Further Reading
===================================

Further information on Brent's algorithm is available in the following
book,

     Richard Brent, 'Algorithms for minimization without derivatives',
     Prentice-Hall (1973), republished by Dover in paperback (2002),
     ISBN 0-486-41998-3.


File: gsl-ref.info,  Node: Multidimensional Root-Finding,  Next: Multidimensional Minimization,  Prev: One dimensional Minimization,  Up: Top

36 Multidimensional Root-Finding
********************************

This chapter describes functions for multidimensional root-finding
(solving nonlinear systems with n equations in n unknowns).  The library
provides low level components for a variety of iterative solvers and
convergence tests.  These can be combined by the user to achieve the
desired solution, with full access to the intermediate steps of the
iteration.  Each class of methods uses the same framework, so that you
can switch between solvers at runtime without needing to recompile your
program.  Each instance of a solver keeps track of its own state,
allowing the solvers to be used in multi-threaded programs.  The solvers
are based on the original Fortran library MINPACK.

   The header file 'gsl_multiroots.h' contains prototypes for the
multidimensional root finding functions and related declarations.

* Menu:

* Overview of Multidimensional Root Finding::  
* Initializing the Multidimensional Solver::  
* Providing the multidimensional system of equations to solve::  
* Iteration of the multidimensional solver::  
* Search Stopping Parameters for the multidimensional solver::  
* Algorithms using Derivatives::  
* Algorithms without Derivatives::  
* Example programs for Multidimensional Root finding::  
* References and Further Reading for Multidimensional Root Finding::  


File: gsl-ref.info,  Node: Overview of Multidimensional Root Finding,  Next: Initializing the Multidimensional Solver,  Up: Multidimensional Root-Finding

36.1 Overview
=============

The problem of multidimensional root finding requires the simultaneous
solution of n equations, f_i, in n variables, x_i,

     f_i (x_1, ..., x_n) = 0    for i = 1 ... n.

In general there are no bracketing methods available for n dimensional
systems, and no way of knowing whether any solutions exist.  All
algorithms proceed from an initial guess using a variant of the Newton
iteration,

     x -> x' = x - J^{-1} f(x)

where x, f are vector quantities and J is the Jacobian matrix J_{ij} = d
f_i / d x_j.  Additional strategies can be used to enlarge the region of
convergence.  These include requiring a decrease in the norm |f| on each
step proposed by Newton's method, or taking steepest-descent steps in
the direction of the negative gradient of |f|.

   Several root-finding algorithms are available within a single
framework.  The user provides a high-level driver for the algorithms,
and the library provides the individual functions necessary for each of
the steps.  There are three main phases of the iteration.  The steps
are,

   * initialize solver state, S, for algorithm T

   * update S using the iteration T

   * test S for convergence, and repeat iteration if necessary

The evaluation of the Jacobian matrix can be problematic, either because
programming the derivatives is intractable or because computation of the
n^2 terms of the matrix becomes too expensive.  For these reasons the
algorithms provided by the library are divided into two classes
according to whether the derivatives are available or not.

   The state for solvers with an analytic Jacobian matrix is held in a
'gsl_multiroot_fdfsolver' struct.  The updating procedure requires both
the function and its derivatives to be supplied by the user.

   The state for solvers which do not use an analytic Jacobian matrix is
held in a 'gsl_multiroot_fsolver' struct.  The updating procedure uses
only function evaluations (not derivatives).  The algorithms estimate
the matrix J or J^{-1} by approximate methods.


File: gsl-ref.info,  Node: Initializing the Multidimensional Solver,  Next: Providing the multidimensional system of equations to solve,  Prev: Overview of Multidimensional Root Finding,  Up: Multidimensional Root-Finding

36.2 Initializing the Solver
============================

The following functions initialize a multidimensional solver, either
with or without derivatives.  The solver itself depends only on the
dimension of the problem and the algorithm and can be reused for
different problems.

 -- Function: gsl_multiroot_fsolver * gsl_multiroot_fsolver_alloc (const
          gsl_multiroot_fsolver_type * T, size_t N)
     This function returns a pointer to a newly allocated instance of a
     solver of type T for a system of N dimensions.  For example, the
     following code creates an instance of a hybrid solver, to solve a
     3-dimensional system of equations.

          const gsl_multiroot_fsolver_type * T
              = gsl_multiroot_fsolver_hybrid;
          gsl_multiroot_fsolver * s
              = gsl_multiroot_fsolver_alloc (T, 3);

     If there is insufficient memory to create the solver then the
     function returns a null pointer and the error handler is invoked
     with an error code of 'GSL_ENOMEM'.

 -- Function: gsl_multiroot_fdfsolver * gsl_multiroot_fdfsolver_alloc
          (const gsl_multiroot_fdfsolver_type * T, size_t N)
     This function returns a pointer to a newly allocated instance of a
     derivative solver of type T for a system of N dimensions.  For
     example, the following code creates an instance of a Newton-Raphson
     solver, for a 2-dimensional system of equations.

          const gsl_multiroot_fdfsolver_type * T
              = gsl_multiroot_fdfsolver_newton;
          gsl_multiroot_fdfsolver * s =
              gsl_multiroot_fdfsolver_alloc (T, 2);

     If there is insufficient memory to create the solver then the
     function returns a null pointer and the error handler is invoked
     with an error code of 'GSL_ENOMEM'.

 -- Function: int gsl_multiroot_fsolver_set (gsl_multiroot_fsolver * S,
          gsl_multiroot_function * F, const gsl_vector * X)
 -- Function: int gsl_multiroot_fdfsolver_set (gsl_multiroot_fdfsolver *
          S, gsl_multiroot_function_fdf * FDF, const gsl_vector * X)
     These functions set, or reset, an existing solver S to use the
     function F or function and derivative FDF, and the initial guess X.
     Note that the initial position is copied from X, this argument is
     not modified by subsequent iterations.

 -- Function: void gsl_multiroot_fsolver_free (gsl_multiroot_fsolver *
          S)
 -- Function: void gsl_multiroot_fdfsolver_free (gsl_multiroot_fdfsolver
          * S)
     These functions free all the memory associated with the solver S.

 -- Function: const char * gsl_multiroot_fsolver_name (const
          gsl_multiroot_fsolver * S)
 -- Function: const char * gsl_multiroot_fdfsolver_name (const
          gsl_multiroot_fdfsolver * S)
     These functions return a pointer to the name of the solver.  For
     example,

          printf ("s is a '%s' solver\n",
                  gsl_multiroot_fdfsolver_name (s));

     would print something like 's is a 'newton' solver'.


File: gsl-ref.info,  Node: Providing the multidimensional system of equations to solve,  Next: Iteration of the multidimensional solver,  Prev: Initializing the Multidimensional Solver,  Up: Multidimensional Root-Finding

36.3 Providing the function to solve
====================================

You must provide n functions of n variables for the root finders to
operate on.  In order to allow for general parameters the functions are
defined by the following data types:

 -- Data Type: gsl_multiroot_function
     This data type defines a general system of functions with
     parameters.

     'int (* f) (const gsl_vector * X, void * PARAMS, gsl_vector * F)'
          this function should store the vector result f(x,params) in F
          for argument X and parameters PARAMS, returning an appropriate
          error code if the function cannot be computed.

     'size_t n'
          the dimension of the system, i.e.  the number of components of
          the vectors X and F.

     'void * params'
          a pointer to the parameters of the function.

Here is an example using Powell's test function,

     f_1(x) = A x_0 x_1 - 1,
     f_2(x) = exp(-x_0) + exp(-x_1) - (1 + 1/A)

with A = 10^4.  The following code defines a 'gsl_multiroot_function'
system 'F' which you could pass to a solver:

     struct powell_params { double A; };

     int
     powell (gsl_vector * x, void * p, gsl_vector * f) {
        struct powell_params * params
          = (struct powell_params *)p;
        const double A = (params->A);
        const double x0 = gsl_vector_get(x,0);
        const double x1 = gsl_vector_get(x,1);

        gsl_vector_set (f, 0, A * x0 * x1 - 1);
        gsl_vector_set (f, 1, (exp(-x0) + exp(-x1)
                               - (1.0 + 1.0/A)));
        return GSL_SUCCESS
     }

     gsl_multiroot_function F;
     struct powell_params params = { 10000.0 };

     F.f = &powell;
     F.n = 2;
     F.params = &params;

 -- Data Type: gsl_multiroot_function_fdf
     This data type defines a general system of functions with
     parameters and the corresponding Jacobian matrix of derivatives,

     'int (* f) (const gsl_vector * X, void * PARAMS, gsl_vector * F)'
          this function should store the vector result f(x,params) in F
          for argument X and parameters PARAMS, returning an appropriate
          error code if the function cannot be computed.

     'int (* df) (const gsl_vector * X, void * PARAMS, gsl_matrix * J)'
          this function should store the N-by-N matrix result J_ij = d
          f_i(x,params) / d x_j in J for argument X and parameters
          PARAMS, returning an appropriate error code if the function
          cannot be computed.

     'int (* fdf) (const gsl_vector * X, void * PARAMS, gsl_vector * F, gsl_matrix * J)'
          This function should set the values of the F and J as above,
          for arguments X and parameters PARAMS.  This function provides
          an optimization of the separate functions for f(x) and
          J(x)--it is always faster to compute the function and its
          derivative at the same time.

     'size_t n'
          the dimension of the system, i.e.  the number of components of
          the vectors X and F.

     'void * params'
          a pointer to the parameters of the function.

The example of Powell's test function defined above can be extended to
include analytic derivatives using the following code,

     int
     powell_df (gsl_vector * x, void * p, gsl_matrix * J)
     {
        struct powell_params * params
          = (struct powell_params *)p;
        const double A = (params->A);
        const double x0 = gsl_vector_get(x,0);
        const double x1 = gsl_vector_get(x,1);
        gsl_matrix_set (J, 0, 0, A * x1);
        gsl_matrix_set (J, 0, 1, A * x0);
        gsl_matrix_set (J, 1, 0, -exp(-x0));
        gsl_matrix_set (J, 1, 1, -exp(-x1));
        return GSL_SUCCESS
     }

     int
     powell_fdf (gsl_vector * x, void * p,
                 gsl_matrix * f, gsl_matrix * J) {
        struct powell_params * params
          = (struct powell_params *)p;
        const double A = (params->A);
        const double x0 = gsl_vector_get(x,0);
        const double x1 = gsl_vector_get(x,1);

        const double u0 = exp(-x0);
        const double u1 = exp(-x1);

        gsl_vector_set (f, 0, A * x0 * x1 - 1);
        gsl_vector_set (f, 1, u0 + u1 - (1 + 1/A));

        gsl_matrix_set (J, 0, 0, A * x1);
        gsl_matrix_set (J, 0, 1, A * x0);
        gsl_matrix_set (J, 1, 0, -u0);
        gsl_matrix_set (J, 1, 1, -u1);
        return GSL_SUCCESS
     }

     gsl_multiroot_function_fdf FDF;

     FDF.f = &powell_f;
     FDF.df = &powell_df;
     FDF.fdf = &powell_fdf;
     FDF.n = 2;
     FDF.params = 0;

Note that the function 'powell_fdf' is able to reuse existing terms from
the function when calculating the Jacobian, thus saving time.


File: gsl-ref.info,  Node: Iteration of the multidimensional solver,  Next: Search Stopping Parameters for the multidimensional solver,  Prev: Providing the multidimensional system of equations to solve,  Up: Multidimensional Root-Finding

36.4 Iteration
==============

The following functions drive the iteration of each algorithm.  Each
function performs one iteration to update the state of any solver of the
corresponding type.  The same functions work for all solvers so that
different methods can be substituted at runtime without modifications to
the code.

 -- Function: int gsl_multiroot_fsolver_iterate (gsl_multiroot_fsolver *
          S)
 -- Function: int gsl_multiroot_fdfsolver_iterate
          (gsl_multiroot_fdfsolver * S)
     These functions perform a single iteration of the solver S.  If the
     iteration encounters an unexpected problem then an error code will
     be returned,

     'GSL_EBADFUNC'
          the iteration encountered a singular point where the function
          or its derivative evaluated to 'Inf' or 'NaN'.

     'GSL_ENOPROG'
          the iteration is not making any progress, preventing the
          algorithm from continuing.

   The solver maintains a current best estimate of the root 's->x' and
its function value 's->f' at all times.  This information can be
accessed with the following auxiliary functions,

 -- Function: gsl_vector * gsl_multiroot_fsolver_root (const
          gsl_multiroot_fsolver * S)
 -- Function: gsl_vector * gsl_multiroot_fdfsolver_root (const
          gsl_multiroot_fdfsolver * S)
     These functions return the current estimate of the root for the
     solver S, given by 's->x'.

 -- Function: gsl_vector * gsl_multiroot_fsolver_f (const
          gsl_multiroot_fsolver * S)
 -- Function: gsl_vector * gsl_multiroot_fdfsolver_f (const
          gsl_multiroot_fdfsolver * S)
     These functions return the function value f(x) at the current
     estimate of the root for the solver S, given by 's->f'.

 -- Function: gsl_vector * gsl_multiroot_fsolver_dx (const
          gsl_multiroot_fsolver * S)
 -- Function: gsl_vector * gsl_multiroot_fdfsolver_dx (const
          gsl_multiroot_fdfsolver * S)
     These functions return the last step dx taken by the solver S,
     given by 's->dx'.


File: gsl-ref.info,  Node: Search Stopping Parameters for the multidimensional solver,  Next: Algorithms using Derivatives,  Prev: Iteration of the multidimensional solver,  Up: Multidimensional Root-Finding

36.5 Search Stopping Parameters
===============================

A root finding procedure should stop when one of the following
conditions is true:

   * A multidimensional root has been found to within the user-specified
     precision.

   * A user-specified maximum number of iterations has been reached.

   * An error has occurred.

The handling of these conditions is under user control.  The functions
below allow the user to test the precision of the current result in
several standard ways.

 -- Function: int gsl_multiroot_test_delta (const gsl_vector * DX, const
          gsl_vector * X, double EPSABS, double EPSREL)

     This function tests for the convergence of the sequence by
     comparing the last step DX with the absolute error EPSABS and
     relative error EPSREL to the current position X.  The test returns
     'GSL_SUCCESS' if the following condition is achieved,

          |dx_i| < epsabs + epsrel |x_i|

     for each component of X and returns 'GSL_CONTINUE' otherwise.

 -- Function: int gsl_multiroot_test_residual (const gsl_vector * F,
          double EPSABS)
     This function tests the residual value F against the absolute error
     bound EPSABS.  The test returns 'GSL_SUCCESS' if the following
     condition is achieved,

          \sum_i |f_i| < epsabs

     and returns 'GSL_CONTINUE' otherwise.  This criterion is suitable
     for situations where the precise location of the root, x, is
     unimportant provided a value can be found where the residual is
     small enough.


File: gsl-ref.info,  Node: Algorithms using Derivatives,  Next: Algorithms without Derivatives,  Prev: Search Stopping Parameters for the multidimensional solver,  Up: Multidimensional Root-Finding

36.6 Algorithms using Derivatives
=================================

The root finding algorithms described in this section make use of both
the function and its derivative.  They require an initial guess for the
location of the root, but there is no absolute guarantee of
convergence--the function must be suitable for this technique and the
initial guess must be sufficiently close to the root for it to work.
When the conditions are satisfied then convergence is quadratic.

 -- Derivative Solver: gsl_multiroot_fdfsolver_hybridsj
     This is a modified version of Powell's Hybrid method as implemented
     in the HYBRJ algorithm in MINPACK.  Minpack was written by Jorge J.
     More', Burton S. Garbow and Kenneth E. Hillstrom.  The Hybrid
     algorithm retains the fast convergence of Newton's method but will
     also reduce the residual when Newton's method is unreliable.

     The algorithm uses a generalized trust region to keep each step
     under control.  In order to be accepted a proposed new position x'
     must satisfy the condition |D (x' - x)| < \delta, where D is a
     diagonal scaling matrix and \delta is the size of the trust region.
     The components of D are computed internally, using the column norms
     of the Jacobian to estimate the sensitivity of the residual to each
     component of x.  This improves the behavior of the algorithm for
     badly scaled functions.

     On each iteration the algorithm first determines the standard
     Newton step by solving the system J dx = - f.  If this step falls
     inside the trust region it is used as a trial step in the next
     stage.  If not, the algorithm uses the linear combination of the
     Newton and gradient directions which is predicted to minimize the
     norm of the function while staying inside the trust region,

          dx = - \alpha J^{-1} f(x) - \beta \nabla |f(x)|^2.

     This combination of Newton and gradient directions is referred to
     as a "dogleg step".

     The proposed step is now tested by evaluating the function at the
     resulting point, x'.  If the step reduces the norm of the function
     sufficiently then it is accepted and size of the trust region is
     increased.  If the proposed step fails to improve the solution then
     the size of the trust region is decreased and another trial step is
     computed.

     The speed of the algorithm is increased by computing the changes to
     the Jacobian approximately, using a rank-1 update.  If two
     successive attempts fail to reduce the residual then the full
     Jacobian is recomputed.  The algorithm also monitors the progress
     of the solution and returns an error if several steps fail to make
     any improvement,

     'GSL_ENOPROG'
          the iteration is not making any progress, preventing the
          algorithm from continuing.

     'GSL_ENOPROGJ'
          re-evaluations of the Jacobian indicate that the iteration is
          not making any progress, preventing the algorithm from
          continuing.

 -- Derivative Solver: gsl_multiroot_fdfsolver_hybridj
     This algorithm is an unscaled version of 'hybridsj'.  The steps are
     controlled by a spherical trust region |x' - x| < \delta, instead
     of a generalized region.  This can be useful if the generalized
     region estimated by 'hybridsj' is inappropriate.

 -- Derivative Solver: gsl_multiroot_fdfsolver_newton

     Newton's Method is the standard root-polishing algorithm.  The
     algorithm begins with an initial guess for the location of the
     solution.  On each iteration a linear approximation to the function
     F is used to estimate the step which will zero all the components
     of the residual.  The iteration is defined by the following
     sequence,

          x -> x' = x - J^{-1} f(x)

     where the Jacobian matrix J is computed from the derivative
     functions provided by F.  The step dx is obtained by solving the
     linear system,

          J dx = - f(x)

     using LU decomposition.  If the Jacobian matrix is singular, an
     error code of 'GSL_EDOM' is returned.

 -- Derivative Solver: gsl_multiroot_fdfsolver_gnewton
     This is a modified version of Newton's method which attempts to
     improve global convergence by requiring every step to reduce the
     Euclidean norm of the residual, |f(x)|.  If the Newton step leads
     to an increase in the norm then a reduced step of relative size,

          t = (\sqrt(1 + 6 r) - 1) / (3 r)

     is proposed, with r being the ratio of norms |f(x')|^2/|f(x)|^2.
     This procedure is repeated until a suitable step size is found.


File: gsl-ref.info,  Node: Algorithms without Derivatives,  Next: Example programs for Multidimensional Root finding,  Prev: Algorithms using Derivatives,  Up: Multidimensional Root-Finding

36.7 Algorithms without Derivatives
===================================

The algorithms described in this section do not require any derivative
information to be supplied by the user.  Any derivatives needed are
approximated by finite differences.  Note that if the
finite-differencing step size chosen by these routines is inappropriate,
an explicit user-supplied numerical derivative can always be used with
the algorithms described in the previous section.

 -- Solver: gsl_multiroot_fsolver_hybrids
     This is a version of the Hybrid algorithm which replaces calls to
     the Jacobian function by its finite difference approximation.  The
     finite difference approximation is computed using
     'gsl_multiroots_fdjac' with a relative step size of
     'GSL_SQRT_DBL_EPSILON'.  Note that this step size will not be
     suitable for all problems.

 -- Solver: gsl_multiroot_fsolver_hybrid
     This is a finite difference version of the Hybrid algorithm without
     internal scaling.

 -- Solver: gsl_multiroot_fsolver_dnewton

     The "discrete Newton algorithm" is the simplest method of solving a
     multidimensional system.  It uses the Newton iteration

          x -> x - J^{-1} f(x)

     where the Jacobian matrix J is approximated by taking finite
     differences of the function F.  The approximation scheme used by
     this implementation is,

          J_{ij} = (f_i(x + \delta_j) - f_i(x)) /  \delta_j

     where \delta_j is a step of size \sqrt\epsilon |x_j| with \epsilon
     being the machine precision (\epsilon \approx 2.22 \times 10^-16).
     The order of convergence of Newton's algorithm is quadratic, but
     the finite differences require n^2 function evaluations on each
     iteration.  The algorithm may become unstable if the finite
     differences are not a good approximation to the true derivatives.

 -- Solver: gsl_multiroot_fsolver_broyden

     The "Broyden algorithm" is a version of the discrete Newton
     algorithm which attempts to avoids the expensive update of the
     Jacobian matrix on each iteration.  The changes to the Jacobian are
     also approximated, using a rank-1 update,

          J^{-1} \to J^{-1} - (J^{-1} df - dx) dx^T J^{-1} / dx^T J^{-1} df

     where the vectors dx and df are the changes in x and f.  On the
     first iteration the inverse Jacobian is estimated using finite
     differences, as in the discrete Newton algorithm.

     This approximation gives a fast update but is unreliable if the
     changes are not small, and the estimate of the inverse Jacobian
     becomes worse as time passes.  The algorithm has a tendency to
     become unstable unless it starts close to the root.  The Jacobian
     is refreshed if this instability is detected (consult the source
     for details).

     This algorithm is included only for demonstration purposes, and is
     not recommended for serious use.


File: gsl-ref.info,  Node: Example programs for Multidimensional Root finding,  Next: References and Further Reading for Multidimensional Root Finding,  Prev: Algorithms without Derivatives,  Up: Multidimensional Root-Finding

36.8 Examples
=============

The multidimensional solvers are used in a similar way to the
one-dimensional root finding algorithms.  This first example
demonstrates the 'hybrids' scaled-hybrid algorithm, which does not
require derivatives.  The program solves the Rosenbrock system of
equations,

     f_1 (x, y) = a (1 - x)
     f_2 (x, y) = b (y - x^2)

with a = 1, b = 10.  The solution of this system lies at (x,y) = (1,1)
in a narrow valley.

   The first stage of the program is to define the system of equations,

     #include <stdlib.h>
     #include <stdio.h>
     #include <gsl/gsl_vector.h>
     #include <gsl/gsl_multiroots.h>

     struct rparams
       {
         double a;
         double b;
       };

     int
     rosenbrock_f (const gsl_vector * x, void *params,
                   gsl_vector * f)
     {
       double a = ((struct rparams *) params)->a;
       double b = ((struct rparams *) params)->b;

       const double x0 = gsl_vector_get (x, 0);
       const double x1 = gsl_vector_get (x, 1);

       const double y0 = a * (1 - x0);
       const double y1 = b * (x1 - x0 * x0);

       gsl_vector_set (f, 0, y0);
       gsl_vector_set (f, 1, y1);

       return GSL_SUCCESS;
     }

The main program begins by creating the function object 'f', with the
arguments '(x,y)' and parameters '(a,b)'.  The solver 's' is initialized
to use this function, with the 'hybrids' method.

     int
     main (void)
     {
       const gsl_multiroot_fsolver_type *T;
       gsl_multiroot_fsolver *s;

       int status;
       size_t i, iter = 0;

       const size_t n = 2;
       struct rparams p = {1.0, 10.0};
       gsl_multiroot_function f = {&rosenbrock_f, n, &p};

       double x_init[2] = {-10.0, -5.0};
       gsl_vector *x = gsl_vector_alloc (n);

       gsl_vector_set (x, 0, x_init[0]);
       gsl_vector_set (x, 1, x_init[1]);

       T = gsl_multiroot_fsolver_hybrids;
       s = gsl_multiroot_fsolver_alloc (T, 2);
       gsl_multiroot_fsolver_set (s, &f, x);

       print_state (iter, s);

       do
         {
           iter++;
           status = gsl_multiroot_fsolver_iterate (s);

           print_state (iter, s);

           if (status)   /* check if solver is stuck */
             break;

           status =
             gsl_multiroot_test_residual (s->f, 1e-7);
         }
       while (status == GSL_CONTINUE && iter < 1000);

       printf ("status = %s\n", gsl_strerror (status));

       gsl_multiroot_fsolver_free (s);
       gsl_vector_free (x);
       return 0;
     }

Note that it is important to check the return status of each solver
step, in case the algorithm becomes stuck.  If an error condition is
detected, indicating that the algorithm cannot proceed, then the error
can be reported to the user, a new starting point chosen or a different
algorithm used.

   The intermediate state of the solution is displayed by the following
function.  The solver state contains the vector 's->x' which is the
current position, and the vector 's->f' with corresponding function
values.

     int
     print_state (size_t iter, gsl_multiroot_fsolver * s)
     {
       printf ("iter = %3u x = % .3f % .3f "
               "f(x) = % .3e % .3e\n",
               iter,
               gsl_vector_get (s->x, 0),
               gsl_vector_get (s->x, 1),
               gsl_vector_get (s->f, 0),
               gsl_vector_get (s->f, 1));
     }

Here are the results of running the program.  The algorithm is started
at (-10,-5) far from the solution.  Since the solution is hidden in a
narrow valley the earliest steps follow the gradient of the function
downhill, in an attempt to reduce the large value of the residual.  Once
the root has been approximately located, on iteration 8, the Newton
behavior takes over and convergence is very rapid.

     iter =  0 x = -10.000  -5.000  f(x) = 1.100e+01 -1.050e+03
     iter =  1 x = -10.000  -5.000  f(x) = 1.100e+01 -1.050e+03
     iter =  2 x =  -3.976  24.827  f(x) = 4.976e+00  9.020e+01
     iter =  3 x =  -3.976  24.827  f(x) = 4.976e+00  9.020e+01
     iter =  4 x =  -3.976  24.827  f(x) = 4.976e+00  9.020e+01
     iter =  5 x =  -1.274  -5.680  f(x) = 2.274e+00 -7.302e+01
     iter =  6 x =  -1.274  -5.680  f(x) = 2.274e+00 -7.302e+01
     iter =  7 x =   0.249   0.298  f(x) = 7.511e-01  2.359e+00
     iter =  8 x =   0.249   0.298  f(x) = 7.511e-01  2.359e+00
     iter =  9 x =   1.000   0.878  f(x) = 1.268e-10 -1.218e+00
     iter = 10 x =   1.000   0.989  f(x) = 1.124e-11 -1.080e-01
     iter = 11 x =   1.000   1.000  f(x) = 0.000e+00  0.000e+00
     status = success

Note that the algorithm does not update the location on every iteration.
Some iterations are used to adjust the trust-region parameter, after
trying a step which was found to be divergent, or to recompute the
Jacobian, when poor convergence behavior is detected.

   The next example program adds derivative information, in order to
accelerate the solution.  There are two derivative functions
'rosenbrock_df' and 'rosenbrock_fdf'.  The latter computes both the
function and its derivative simultaneously.  This allows the
optimization of any common terms.  For simplicity we substitute calls to
the separate 'f' and 'df' functions at this point in the code below.

     int
     rosenbrock_df (const gsl_vector * x, void *params,
                    gsl_matrix * J)
     {
       const double a = ((struct rparams *) params)->a;
       const double b = ((struct rparams *) params)->b;

       const double x0 = gsl_vector_get (x, 0);

       const double df00 = -a;
       const double df01 = 0;
       const double df10 = -2 * b  * x0;
       const double df11 = b;

       gsl_matrix_set (J, 0, 0, df00);
       gsl_matrix_set (J, 0, 1, df01);
       gsl_matrix_set (J, 1, 0, df10);
       gsl_matrix_set (J, 1, 1, df11);

       return GSL_SUCCESS;
     }

     int
     rosenbrock_fdf (const gsl_vector * x, void *params,
                     gsl_vector * f, gsl_matrix * J)
     {
       rosenbrock_f (x, params, f);
       rosenbrock_df (x, params, J);

       return GSL_SUCCESS;
     }

The main program now makes calls to the corresponding 'fdfsolver'
versions of the functions,

     int
     main (void)
     {
       const gsl_multiroot_fdfsolver_type *T;
       gsl_multiroot_fdfsolver *s;

       int status;
       size_t i, iter = 0;

       const size_t n = 2;
       struct rparams p = {1.0, 10.0};
       gsl_multiroot_function_fdf f = {&rosenbrock_f,
                                       &rosenbrock_df,
                                       &rosenbrock_fdf,
                                       n, &p};

       double x_init[2] = {-10.0, -5.0};
       gsl_vector *x = gsl_vector_alloc (n);

       gsl_vector_set (x, 0, x_init[0]);
       gsl_vector_set (x, 1, x_init[1]);

       T = gsl_multiroot_fdfsolver_gnewton;
       s = gsl_multiroot_fdfsolver_alloc (T, n);
       gsl_multiroot_fdfsolver_set (s, &f, x);

       print_state (iter, s);

       do
         {
           iter++;

           status = gsl_multiroot_fdfsolver_iterate (s);

           print_state (iter, s);

           if (status)
             break;

           status = gsl_multiroot_test_residual (s->f, 1e-7);
         }
       while (status == GSL_CONTINUE && iter < 1000);

       printf ("status = %s\n", gsl_strerror (status));

       gsl_multiroot_fdfsolver_free (s);
       gsl_vector_free (x);
       return 0;
     }

The addition of derivative information to the 'hybrids' solver does not
make any significant difference to its behavior, since it able to
approximate the Jacobian numerically with sufficient accuracy.  To
illustrate the behavior of a different derivative solver we switch to
'gnewton'.  This is a traditional Newton solver with the constraint that
it scales back its step if the full step would lead "uphill".  Here is
the output for the 'gnewton' algorithm,

     iter = 0 x = -10.000  -5.000 f(x) =  1.100e+01 -1.050e+03
     iter = 1 x =  -4.231 -65.317 f(x) =  5.231e+00 -8.321e+02
     iter = 2 x =   1.000 -26.358 f(x) = -8.882e-16 -2.736e+02
     iter = 3 x =   1.000   1.000 f(x) = -2.220e-16 -4.441e-15
     status = success

The convergence is much more rapid, but takes a wide excursion out to
the point (-4.23,-65.3).  This could cause the algorithm to go astray in
a realistic application.  The hybrid algorithm follows the downhill path
to the solution more reliably.


File: gsl-ref.info,  Node: References and Further Reading for Multidimensional Root Finding,  Prev: Example programs for Multidimensional Root finding,  Up: Multidimensional Root-Finding

36.9 References and Further Reading
===================================

The original version of the Hybrid method is described in the following
articles by Powell,

     M.J.D. Powell, "A Hybrid Method for Nonlinear Equations" (Chap 6, p
     87-114) and "A Fortran Subroutine for Solving systems of Nonlinear
     Algebraic Equations" (Chap 7, p 115-161), in 'Numerical Methods for
     Nonlinear Algebraic Equations', P. Rabinowitz, editor.  Gordon and
     Breach, 1970.

The following papers are also relevant to the algorithms described in
this section,

     J.J. More', M.Y. Cosnard, "Numerical Solution of Nonlinear
     Equations", 'ACM Transactions on Mathematical Software', Vol 5, No
     1, (1979), p 64-85

     C.G. Broyden, "A Class of Methods for Solving Nonlinear
     Simultaneous Equations", 'Mathematics of Computation', Vol 19
     (1965), p 577-593

     J.J. More', B.S. Garbow, K.E. Hillstrom, "Testing Unconstrained
     Optimization Software", ACM Transactions on Mathematical Software,
     Vol 7, No 1 (1981), p 17-41


File: gsl-ref.info,  Node: Multidimensional Minimization,  Next: Least-Squares Fitting,  Prev: Multidimensional Root-Finding,  Up: Top

37 Multidimensional Minimization
********************************

This chapter describes routines for finding minima of arbitrary
multidimensional functions.  The library provides low level components
for a variety of iterative minimizers and convergence tests.  These can
be combined by the user to achieve the desired solution, while providing
full access to the intermediate steps of the algorithms.  Each class of
methods uses the same framework, so that you can switch between
minimizers at runtime without needing to recompile your program.  Each
instance of a minimizer keeps track of its own state, allowing the
minimizers to be used in multi-threaded programs.  The minimization
algorithms can be used to maximize a function by inverting its sign.

   The header file 'gsl_multimin.h' contains prototypes for the
minimization functions and related declarations.

* Menu:

* Multimin Overview::           
* Multimin Caveats::            
* Initializing the Multidimensional Minimizer::  
* Providing a function to minimize::  
* Multimin Iteration::          
* Multimin Stopping Criteria::  
* Multimin Algorithms with Derivatives::  
* Multimin Algorithms without Derivatives::  
* Multimin Examples::           
* Multimin References and Further Reading::  


File: gsl-ref.info,  Node: Multimin Overview,  Next: Multimin Caveats,  Up: Multidimensional Minimization

37.1 Overview
=============

The problem of multidimensional minimization requires finding a point x
such that the scalar function,

     f(x_1, ..., x_n)

takes a value which is lower than at any neighboring point.  For smooth
functions the gradient g = \nabla f vanishes at the minimum.  In general
there are no bracketing methods available for the minimization of
n-dimensional functions.  The algorithms proceed from an initial guess
using a search algorithm which attempts to move in a downhill direction.

   Algorithms making use of the gradient of the function perform a
one-dimensional line minimisation along this direction until the lowest
point is found to a suitable tolerance.  The search direction is then
updated with local information from the function and its derivatives,
and the whole process repeated until the true n-dimensional minimum is
found.

   Algorithms which do not require the gradient of the function use
different strategies.  For example, the Nelder-Mead Simplex algorithm
maintains n+1 trial parameter vectors as the vertices of a n-dimensional
simplex.  On each iteration it tries to improve the worst vertex of the
simplex by geometrical transformations.  The iterations are continued
until the overall size of the simplex has decreased sufficiently.

   Both types of algorithms use a standard framework.  The user provides
a high-level driver for the algorithms, and the library provides the
individual functions necessary for each of the steps.  There are three
main phases of the iteration.  The steps are,

   * initialize minimizer state, S, for algorithm T

   * update S using the iteration T

   * test S for convergence, and repeat iteration if necessary

Each iteration step consists either of an improvement to the
line-minimisation in the current direction or an update to the search
direction itself.  The state for the minimizers is held in a
'gsl_multimin_fdfminimizer' struct or a 'gsl_multimin_fminimizer'
struct.


File: gsl-ref.info,  Node: Multimin Caveats,  Next: Initializing the Multidimensional Minimizer,  Prev: Multimin Overview,  Up: Multidimensional Minimization

37.2 Caveats
============

Note that the minimization algorithms can only search for one local
minimum at a time.  When there are several local minima in the search
area, the first minimum to be found will be returned; however it is
difficult to predict which of the minima this will be.  In most cases,
no error will be reported if you try to find a local minimum in an area
where there is more than one.

   It is also important to note that the minimization algorithms find
local minima; there is no way to determine whether a minimum is a global
minimum of the function in question.


File: gsl-ref.info,  Node: Initializing the Multidimensional Minimizer,  Next: Providing a function to minimize,  Prev: Multimin Caveats,  Up: Multidimensional Minimization

37.3 Initializing the Multidimensional Minimizer
================================================

The following function initializes a multidimensional minimizer.  The
minimizer itself depends only on the dimension of the problem and the
algorithm and can be reused for different problems.

 -- Function: gsl_multimin_fdfminimizer *
          gsl_multimin_fdfminimizer_alloc (const
          gsl_multimin_fdfminimizer_type * T, size_t N)
 -- Function: gsl_multimin_fminimizer * gsl_multimin_fminimizer_alloc
          (const gsl_multimin_fminimizer_type * T, size_t N)
     This function returns a pointer to a newly allocated instance of a
     minimizer of type T for an N-dimension function.  If there is
     insufficient memory to create the minimizer then the function
     returns a null pointer and the error handler is invoked with an
     error code of 'GSL_ENOMEM'.

 -- Function: int gsl_multimin_fdfminimizer_set
          (gsl_multimin_fdfminimizer * S, gsl_multimin_function_fdf *
          FDF, const gsl_vector * X, double STEP_SIZE, double TOL)
 -- Function: int gsl_multimin_fminimizer_set (gsl_multimin_fminimizer *
          S, gsl_multimin_function * F, const gsl_vector * X, const
          gsl_vector * STEP_SIZE)
     The function 'gsl_multimin_fdfminimizer_set' initializes the
     minimizer S to minimize the function FDF starting from the initial
     point X.  The size of the first trial step is given by STEP_SIZE.
     The accuracy of the line minimization is specified by TOL.  The
     precise meaning of this parameter depends on the method used.
     Typically the line minimization is considered successful if the
     gradient of the function g is orthogonal to the current search
     direction p to a relative accuracy of TOL, where dot(p,g) < tol |p|
     |g|.  A TOL value of 0.1 is suitable for most purposes, since line
     minimization only needs to be carried out approximately.  Note that
     setting TOL to zero will force the use of "exact" line-searches,
     which are extremely expensive.

     The function 'gsl_multimin_fminimizer_set' initializes the
     minimizer S to minimize the function F, starting from the initial
     point X.  The size of the initial trial steps is given in vector
     STEP_SIZE.  The precise meaning of this parameter depends on the
     method used.

 -- Function: void gsl_multimin_fdfminimizer_free
          (gsl_multimin_fdfminimizer * S)
 -- Function: void gsl_multimin_fminimizer_free (gsl_multimin_fminimizer
          * S)
     This function frees all the memory associated with the minimizer S.

 -- Function: const char * gsl_multimin_fdfminimizer_name (const
          gsl_multimin_fdfminimizer * S)
 -- Function: const char * gsl_multimin_fminimizer_name (const
          gsl_multimin_fminimizer * S)
     This function returns a pointer to the name of the minimizer.  For
     example,

          printf ("s is a '%s' minimizer\n",
                  gsl_multimin_fdfminimizer_name (s));

     would print something like 's is a 'conjugate_pr' minimizer'.


File: gsl-ref.info,  Node: Providing a function to minimize,  Next: Multimin Iteration,  Prev: Initializing the Multidimensional Minimizer,  Up: Multidimensional Minimization

37.4 Providing a function to minimize
=====================================

You must provide a parametric function of n variables for the minimizers
to operate on.  You may also need to provide a routine which calculates
the gradient of the function and a third routine which calculates both
the function value and the gradient together.  In order to allow for
general parameters the functions are defined by the following data
types:

 -- Data Type: gsl_multimin_function_fdf
     This data type defines a general function of n variables with
     parameters and the corresponding gradient vector of derivatives,

     'double (* f) (const gsl_vector * X, void * PARAMS)'
          this function should return the result f(x,params) for
          argument X and parameters PARAMS.  If the function cannot be
          computed, an error value of 'GSL_NAN' should be returned.

     'void (* df) (const gsl_vector * X, void * PARAMS, gsl_vector * G)'
          this function should store the N-dimensional gradient g_i = d
          f(x,params) / d x_i in the vector G for argument X and
          parameters PARAMS, returning an appropriate error code if the
          function cannot be computed.

     'void (* fdf) (const gsl_vector * X, void * PARAMS, double * f, gsl_vector * G)'
          This function should set the values of the F and G as above,
          for arguments X and parameters PARAMS.  This function provides
          an optimization of the separate functions for f(x) and
          g(x)--it is always faster to compute the function and its
          derivative at the same time.

     'size_t n'
          the dimension of the system, i.e.  the number of components of
          the vectors X.

     'void * params'
          a pointer to the parameters of the function.
 -- Data Type: gsl_multimin_function
     This data type defines a general function of n variables with
     parameters,

     'double (* f) (const gsl_vector * X, void * PARAMS)'
          this function should return the result f(x,params) for
          argument X and parameters PARAMS.  If the function cannot be
          computed, an error value of 'GSL_NAN' should be returned.

     'size_t n'
          the dimension of the system, i.e.  the number of components of
          the vectors X.

     'void * params'
          a pointer to the parameters of the function.

The following example function defines a simple two-dimensional
paraboloid with five parameters,

     /* Paraboloid centered on (p[0],p[1]), with
        scale factors (p[2],p[3]) and minimum p[4] */

     double
     my_f (const gsl_vector *v, void *params)
     {
       double x, y;
       double *p = (double *)params;

       x = gsl_vector_get(v, 0);
       y = gsl_vector_get(v, 1);

       return p[2] * (x - p[0]) * (x - p[0]) +
                p[3] * (y - p[1]) * (y - p[1]) + p[4];
     }

     /* The gradient of f, df = (df/dx, df/dy). */
     void
     my_df (const gsl_vector *v, void *params,
            gsl_vector *df)
     {
       double x, y;
       double *p = (double *)params;

       x = gsl_vector_get(v, 0);
       y = gsl_vector_get(v, 1);

       gsl_vector_set(df, 0, 2.0 * p[2] * (x - p[0]));
       gsl_vector_set(df, 1, 2.0 * p[3] * (y - p[1]));
     }

     /* Compute both f and df together. */
     void
     my_fdf (const gsl_vector *x, void *params,
             double *f, gsl_vector *df)
     {
       *f = my_f(x, params);
       my_df(x, params, df);
     }

The function can be initialized using the following code,

     gsl_multimin_function_fdf my_func;

     /* Paraboloid center at (1,2), scale factors (10, 20),
        minimum value 30 */
     double p[5] = { 1.0, 2.0, 10.0, 20.0, 30.0 };

     my_func.n = 2;  /* number of function components */
     my_func.f = &my_f;
     my_func.df = &my_df;
     my_func.fdf = &my_fdf;
     my_func.params = (void *)p;


File: gsl-ref.info,  Node: Multimin Iteration,  Next: Multimin Stopping Criteria,  Prev: Providing a function to minimize,  Up: Multidimensional Minimization

37.5 Iteration
==============

The following function drives the iteration of each algorithm.  The
function performs one iteration to update the state of the minimizer.
The same function works for all minimizers so that different methods can
be substituted at runtime without modifications to the code.

 -- Function: int gsl_multimin_fdfminimizer_iterate
          (gsl_multimin_fdfminimizer * S)
 -- Function: int gsl_multimin_fminimizer_iterate
          (gsl_multimin_fminimizer * S)
     These functions perform a single iteration of the minimizer S.  If
     the iteration encounters an unexpected problem then an error code
     will be returned.  The error code 'GSL_ENOPROG' signifies that the
     minimizer is unable to improve on its current estimate, either due
     to numerical difficulty or because a genuine local minimum has been
     reached.

The minimizer maintains a current best estimate of the minimum at all
times.  This information can be accessed with the following auxiliary
functions,

 -- Function: gsl_vector * gsl_multimin_fdfminimizer_x (const
          gsl_multimin_fdfminimizer * S)
 -- Function: gsl_vector * gsl_multimin_fminimizer_x (const
          gsl_multimin_fminimizer * S)
 -- Function: double gsl_multimin_fdfminimizer_minimum (const
          gsl_multimin_fdfminimizer * S)
 -- Function: double gsl_multimin_fminimizer_minimum (const
          gsl_multimin_fminimizer * S)
 -- Function: gsl_vector * gsl_multimin_fdfminimizer_gradient (const
          gsl_multimin_fdfminimizer * S)
 -- Function: gsl_vector * gsl_multimin_fdfminimizer_dx (const
          gsl_multimin_fdfminimizer * S)
 -- Function: double gsl_multimin_fminimizer_size (const
          gsl_multimin_fminimizer * S)
     These functions return the current best estimate of the location of
     the minimum, the value of the function at that point, its gradient,
     the last step increment of the estimate, and minimizer specific
     characteristic size for the minimizer S.

 -- Function: int gsl_multimin_fdfminimizer_restart
          (gsl_multimin_fdfminimizer * S)
     This function resets the minimizer S to use the current point as a
     new starting point.


File: gsl-ref.info,  Node: Multimin Stopping Criteria,  Next: Multimin Algorithms with Derivatives,  Prev: Multimin Iteration,  Up: Multidimensional Minimization

37.6 Stopping Criteria
======================

A minimization procedure should stop when one of the following
conditions is true:

   * A minimum has been found to within the user-specified precision.

   * A user-specified maximum number of iterations has been reached.

   * An error has occurred.

The handling of these conditions is under user control.  The functions
below allow the user to test the precision of the current result.

 -- Function: int gsl_multimin_test_gradient (const gsl_vector * G,
          double EPSABS)
     This function tests the norm of the gradient G against the absolute
     tolerance EPSABS.  The gradient of a multidimensional function goes
     to zero at a minimum.  The test returns 'GSL_SUCCESS' if the
     following condition is achieved,

          |g| < epsabs

     and returns 'GSL_CONTINUE' otherwise.  A suitable choice of EPSABS
     can be made from the desired accuracy in the function for small
     variations in x.  The relationship between these quantities is
     given by \delta f = g \delta x.

 -- Function: int gsl_multimin_test_size (const double SIZE, double
          EPSABS)
     This function tests the minimizer specific characteristic size (if
     applicable to the used minimizer) against absolute tolerance
     EPSABS.  The test returns 'GSL_SUCCESS' if the size is smaller than
     tolerance, otherwise 'GSL_CONTINUE' is returned.


File: gsl-ref.info,  Node: Multimin Algorithms with Derivatives,  Next: Multimin Algorithms without Derivatives,  Prev: Multimin Stopping Criteria,  Up: Multidimensional Minimization

37.7 Algorithms with Derivatives
================================

There are several minimization methods available.  The best choice of
algorithm depends on the problem.  The algorithms described in this
section use the value of the function and its gradient at each
evaluation point.

 -- Minimizer: gsl_multimin_fdfminimizer_conjugate_fr
     This is the Fletcher-Reeves conjugate gradient algorithm.  The
     conjugate gradient algorithm proceeds as a succession of line
     minimizations.  The sequence of search directions is used to build
     up an approximation to the curvature of the function in the
     neighborhood of the minimum.

     An initial search direction P is chosen using the gradient, and
     line minimization is carried out in that direction.  The accuracy
     of the line minimization is specified by the parameter TOL.  The
     minimum along this line occurs when the function gradient G and the
     search direction P are orthogonal.  The line minimization
     terminates when dot(p,g) < tol |p| |g|.  The search direction is
     updated using the Fletcher-Reeves formula p' = g' - \beta g where
     \beta=-|g'|^2/|g|^2, and the line minimization is then repeated for
     the new search direction.

 -- Minimizer: gsl_multimin_fdfminimizer_conjugate_pr
     This is the Polak-Ribiere conjugate gradient algorithm.  It is
     similar to the Fletcher-Reeves method, differing only in the choice
     of the coefficient \beta.  Both methods work well when the
     evaluation point is close enough to the minimum of the objective
     function that it is well approximated by a quadratic hypersurface.

 -- Minimizer: gsl_multimin_fdfminimizer_vector_bfgs2
 -- Minimizer: gsl_multimin_fdfminimizer_vector_bfgs
     These methods use the vector Broyden-Fletcher-Goldfarb-Shanno
     (BFGS) algorithm.  This is a quasi-Newton method which builds up an
     approximation to the second derivatives of the function f using the
     difference between successive gradient vectors.  By combining the
     first and second derivatives the algorithm is able to take
     Newton-type steps towards the function minimum, assuming quadratic
     behavior in that region.

     The 'bfgs2' version of this minimizer is the most efficient version
     available, and is a faithful implementation of the line
     minimization scheme described in Fletcher's 'Practical Methods of
     Optimization', Algorithms 2.6.2 and 2.6.4.  It supersedes the
     original 'bfgs' routine and requires substantially fewer function
     and gradient evaluations.  The user-supplied tolerance TOL
     corresponds to the parameter \sigma used by Fletcher.  A value of
     0.1 is recommended for typical use (larger values correspond to
     less accurate line searches).

 -- Minimizer: gsl_multimin_fdfminimizer_steepest_descent
     The steepest descent algorithm follows the downhill gradient of the
     function at each step.  When a downhill step is successful the
     step-size is increased by a factor of two.  If the downhill step
     leads to a higher function value then the algorithm backtracks and
     the step size is decreased using the parameter TOL.  A suitable
     value of TOL for most applications is 0.1.  The steepest descent
     method is inefficient and is included only for demonstration
     purposes.


File: gsl-ref.info,  Node: Multimin Algorithms without Derivatives,  Next: Multimin Examples,  Prev: Multimin Algorithms with Derivatives,  Up: Multidimensional Minimization

37.8 Algorithms without Derivatives
===================================

The algorithms described in this section use only the value of the
function at each evaluation point.

 -- Minimizer: gsl_multimin_fminimizer_nmsimplex2
 -- Minimizer: gsl_multimin_fminimizer_nmsimplex
     These methods use the Simplex algorithm of Nelder and Mead.
     Starting from the initial vector X = p_0, the algorithm constructs
     an additional n vectors p_i using the step size vector s =
     STEP_SIZE as follows:

          p_0 = (x_0, x_1, ... , x_n)
          p_1 = (x_0 + s_0, x_1, ... , x_n)
          p_2 = (x_0, x_1 + s_1, ... , x_n)
          ... = ...
          p_n = (x_0, x_1, ... , x_n + s_n)

     These vectors form the n+1 vertices of a simplex in n dimensions.
     On each iteration the algorithm uses simple geometrical
     transformations to update the vector corresponding to the highest
     function value.  The geometric transformations are reflection,
     reflection followed by expansion, contraction and multiple
     contraction.  Using these transformations the simplex moves through
     the space towards the minimum, where it contracts itself.

     After each iteration, the best vertex is returned.  Note, that due
     to the nature of the algorithm not every step improves the current
     best parameter vector.  Usually several iterations are required.

     The minimizer-specific characteristic size is calculated as the
     average distance from the geometrical center of the simplex to all
     its vertices.  This size can be used as a stopping criteria, as the
     simplex contracts itself near the minimum.  The size is returned by
     the function 'gsl_multimin_fminimizer_size'.

     The 'nmsimplex2' version of this minimiser is a new O(N) operations
     implementation of the earlier O(N^2) operations 'nmsimplex'
     minimiser.  It uses the same underlying algorithm, but the simplex
     updates are computed more efficiently for high-dimensional
     problems.  In addition, the size of simplex is calculated as the
     RMS distance of each vertex from the center rather than the mean
     distance, allowing a linear update of this quantity on each step.
     The memory usage is O(N^2) for both algorithms.

 -- Minimizer: gsl_multimin_fminimizer_nmsimplex2rand
     This method is a variant of 'nmsimplex2' which initialises the
     simplex around the starting point X using a randomly-oriented set
     of basis vectors instead of the fixed coordinate axes.  The final
     dimensions of the simplex are scaled along the coordinate axes by
     the vector STEP_SIZE.  The randomisation uses a simple
     deterministic generator so that repeated calls to
     'gsl_multimin_fminimizer_set' for a given solver object will vary
     the orientation in a well-defined way.


File: gsl-ref.info,  Node: Multimin Examples,  Next: Multimin References and Further Reading,  Prev: Multimin Algorithms without Derivatives,  Up: Multidimensional Minimization

37.9 Examples
=============

This example program finds the minimum of the paraboloid function
defined earlier.  The location of the minimum is offset from the origin
in x and y, and the function value at the minimum is non-zero.  The main
program is given below, it requires the example function given earlier
in this chapter.

     int
     main (void)
     {
       size_t iter = 0;
       int status;

       const gsl_multimin_fdfminimizer_type *T;
       gsl_multimin_fdfminimizer *s;

       /* Position of the minimum (1,2), scale factors
          10,20, height 30. */
       double par[5] = { 1.0, 2.0, 10.0, 20.0, 30.0 };

       gsl_vector *x;
       gsl_multimin_function_fdf my_func;

       my_func.n = 2;
       my_func.f = my_f;
       my_func.df = my_df;
       my_func.fdf = my_fdf;
       my_func.params = par;

       /* Starting point, x = (5,7) */
       x = gsl_vector_alloc (2);
       gsl_vector_set (x, 0, 5.0);
       gsl_vector_set (x, 1, 7.0);

       T = gsl_multimin_fdfminimizer_conjugate_fr;
       s = gsl_multimin_fdfminimizer_alloc (T, 2);

       gsl_multimin_fdfminimizer_set (s, &my_func, x, 0.01, 1e-4);

       do
         {
           iter++;
           status = gsl_multimin_fdfminimizer_iterate (s);

           if (status)
             break;

           status = gsl_multimin_test_gradient (s->gradient, 1e-3);

           if (status == GSL_SUCCESS)
             printf ("Minimum found at:\n");

           printf ("%5d %.5f %.5f %10.5f\n", iter,
                   gsl_vector_get (s->x, 0),
                   gsl_vector_get (s->x, 1),
                   s->f);

         }
       while (status == GSL_CONTINUE && iter < 100);

       gsl_multimin_fdfminimizer_free (s);
       gsl_vector_free (x);

       return 0;
     }

The initial step-size is chosen as 0.01, a conservative estimate in this
case, and the line minimization parameter is set at 0.0001.  The program
terminates when the norm of the gradient has been reduced below 0.001.
The output of the program is shown below,

              x       y         f
         1 4.99629 6.99072  687.84780
         2 4.98886 6.97215  683.55456
         3 4.97400 6.93501  675.01278
         4 4.94429 6.86073  658.10798
         5 4.88487 6.71217  625.01340
         6 4.76602 6.41506  561.68440
         7 4.52833 5.82083  446.46694
         8 4.05295 4.63238  261.79422
         9 3.10219 2.25548   75.49762
        10 2.85185 1.62963   67.03704
        11 2.19088 1.76182   45.31640
        12 0.86892 2.02622   30.18555
     Minimum found at:
        13 1.00000 2.00000   30.00000

Note that the algorithm gradually increases the step size as it
successfully moves downhill, as can be seen by plotting the successive
points.

The conjugate gradient algorithm finds the minimum on its second
direction because the function is purely quadratic.  Additional
iterations would be needed for a more complicated function.

   Here is another example using the Nelder-Mead Simplex algorithm to
minimize the same example object function, as above.

     int
     main(void)
     {
       double par[5] = {1.0, 2.0, 10.0, 20.0, 30.0};

       const gsl_multimin_fminimizer_type *T =
         gsl_multimin_fminimizer_nmsimplex2;
       gsl_multimin_fminimizer *s = NULL;
       gsl_vector *ss, *x;
       gsl_multimin_function minex_func;

       size_t iter = 0;
       int status;
       double size;

       /* Starting point */
       x = gsl_vector_alloc (2);
       gsl_vector_set (x, 0, 5.0);
       gsl_vector_set (x, 1, 7.0);

       /* Set initial step sizes to 1 */
       ss = gsl_vector_alloc (2);
       gsl_vector_set_all (ss, 1.0);

       /* Initialize method and iterate */
       minex_func.n = 2;
       minex_func.f = my_f;
       minex_func.params = par;

       s = gsl_multimin_fminimizer_alloc (T, 2);
       gsl_multimin_fminimizer_set (s, &minex_func, x, ss);

       do
         {
           iter++;
           status = gsl_multimin_fminimizer_iterate(s);

           if (status)
             break;

           size = gsl_multimin_fminimizer_size (s);
           status = gsl_multimin_test_size (size, 1e-2);

           if (status == GSL_SUCCESS)
             {
               printf ("converged to minimum at\n");
             }

           printf ("%5d %10.3e %10.3e f() = %7.3f size = %.3f\n",
                   iter,
                   gsl_vector_get (s->x, 0),
                   gsl_vector_get (s->x, 1),
                   s->fval, size);
         }
       while (status == GSL_CONTINUE && iter < 100);

       gsl_vector_free(x);
       gsl_vector_free(ss);
       gsl_multimin_fminimizer_free (s);

       return status;
     }

The minimum search stops when the Simplex size drops to 0.01.  The
output is shown below.

         1  6.500e+00  5.000e+00 f() = 512.500 size = 1.130
         2  5.250e+00  4.000e+00 f() = 290.625 size = 1.409
         3  5.250e+00  4.000e+00 f() = 290.625 size = 1.409
         4  5.500e+00  1.000e+00 f() = 252.500 size = 1.409
         5  2.625e+00  3.500e+00 f() = 101.406 size = 1.847
         6  2.625e+00  3.500e+00 f() = 101.406 size = 1.847
         7  0.000e+00  3.000e+00 f() =  60.000 size = 1.847
         8  2.094e+00  1.875e+00 f() =  42.275 size = 1.321
         9  2.578e-01  1.906e+00 f() =  35.684 size = 1.069
        10  5.879e-01  2.445e+00 f() =  35.664 size = 0.841
        11  1.258e+00  2.025e+00 f() =  30.680 size = 0.476
        12  1.258e+00  2.025e+00 f() =  30.680 size = 0.367
        13  1.093e+00  1.849e+00 f() =  30.539 size = 0.300
        14  8.830e-01  2.004e+00 f() =  30.137 size = 0.172
        15  8.830e-01  2.004e+00 f() =  30.137 size = 0.126
        16  9.582e-01  2.060e+00 f() =  30.090 size = 0.106
        17  1.022e+00  2.004e+00 f() =  30.005 size = 0.063
        18  1.022e+00  2.004e+00 f() =  30.005 size = 0.043
        19  1.022e+00  2.004e+00 f() =  30.005 size = 0.043
        20  1.022e+00  2.004e+00 f() =  30.005 size = 0.027
        21  1.022e+00  2.004e+00 f() =  30.005 size = 0.022
        22  9.920e-01  1.997e+00 f() =  30.001 size = 0.016
        23  9.920e-01  1.997e+00 f() =  30.001 size = 0.013
     converged to minimum at
        24  9.920e-01  1.997e+00 f() =  30.001 size = 0.008

The simplex size first increases, while the simplex moves towards the
minimum.  After a while the size begins to decrease as the simplex
contracts around the minimum.


File: gsl-ref.info,  Node: Multimin References and Further Reading,  Prev: Multimin Examples,  Up: Multidimensional Minimization

37.10 References and Further Reading
====================================

The conjugate gradient and BFGS methods are described in detail in the
following book,

     R. Fletcher, 'Practical Methods of Optimization (Second Edition)'
     Wiley (1987), ISBN 0471915475.

   A brief description of multidimensional minimization algorithms and
more recent references can be found in,

     C.W. Ueberhuber, 'Numerical Computation (Volume 2)', Chapter 14,
     Section 4.4 "Minimization Methods", p. 325-335, Springer (1997),
     ISBN 3-540-62057-5.

The simplex algorithm is described in the following paper,

     J.A. Nelder and R. Mead, 'A simplex method for function
     minimization', Computer Journal vol. 7 (1965), 308-313.


File: gsl-ref.info,  Node: Least-Squares Fitting,  Next: Nonlinear Least-Squares Fitting,  Prev: Multidimensional Minimization,  Up: Top

38 Least-Squares Fitting
************************

This chapter describes routines for performing least squares fits to
experimental data using linear combinations of functions.  The data may
be weighted or unweighted, i.e.  with known or unknown errors.  For
weighted data the functions compute the best fit parameters and their
associated covariance matrix.  For unweighted data the covariance matrix
is estimated from the scatter of the points, giving a
variance-covariance matrix.

   The functions are divided into separate versions for simple one- or
two-parameter regression and multiple-parameter fits.

* Menu:

* Fitting Overview::            
* Linear regression::           
* Multi-parameter regression::
* Regularized regression::
* Robust linear regression::
* Large Dense Linear Systems::
* Troubleshooting::
* Fitting Examples::
* Fitting References and Further Reading::  


File: gsl-ref.info,  Node: Fitting Overview,  Next: Linear regression,  Up: Least-Squares Fitting

38.1 Overview
=============

Least-squares fits are found by minimizing \chi^2 (chi-squared), the
weighted sum of squared residuals over n experimental datapoints (x_i,
y_i) for the model Y(c,x),

     \chi^2 = \sum_i w_i (y_i - Y(c, x_i))^2

The p parameters of the model are c = {c_0, c_1, ...}.  The weight
factors w_i are given by w_i = 1/\sigma_i^2, where \sigma_i is the
experimental error on the data-point y_i.  The errors are assumed to be
Gaussian and uncorrelated.  For unweighted data the chi-squared sum is
computed without any weight factors.

   The fitting routines return the best-fit parameters c and their p
\times p covariance matrix.  The covariance matrix measures the
statistical errors on the best-fit parameters resulting from the errors
on the data, \sigma_i, and is defined as C_{ab} = <\delta c_a \delta
c_b> where < > denotes an average over the Gaussian error distributions
of the underlying datapoints.

   The covariance matrix is calculated by error propagation from the
data errors \sigma_i.  The change in a fitted parameter \delta c_a
caused by a small change in the data \delta y_i is given by

     \delta c_a = \sum_i (dc_a/dy_i) \delta y_i

allowing the covariance matrix to be written in terms of the errors on
the data,

     C_{ab} = \sum_{i,j} (dc_a/dy_i) (dc_b/dy_j) <\delta y_i \delta y_j>

For uncorrelated data the fluctuations of the underlying datapoints
satisfy <\delta y_i \delta y_j> = \sigma_i^2 \delta_{ij}, giving a
corresponding parameter covariance matrix of

     C_{ab} = \sum_i (1/w_i) (dc_a/dy_i) (dc_b/dy_i)

When computing the covariance matrix for unweighted data, i.e.  data
with unknown errors, the weight factors w_i in this sum are replaced by
the single estimate w = 1/\sigma^2, where \sigma^2 is the computed
variance of the residuals about the best-fit model, \sigma^2 = \sum (y_i
- Y(c,x_i))^2 / (n-p).  This is referred to as the "variance-covariance
matrix".

   The standard deviations of the best-fit parameters are given by the
square root of the corresponding diagonal elements of the covariance
matrix, \sigma_{c_a} = \sqrt{C_{aa}}.  The correlation coefficient of
the fit parameters c_a and c_b is given by \rho_{ab} = C_{ab} /
\sqrt{C_{aa} C_{bb}}.


File: gsl-ref.info,  Node: Linear regression,  Next: Multi-parameter regression,  Prev: Fitting Overview,  Up: Least-Squares Fitting

38.2 Linear regression
======================

The functions in this section are used to fit simple one or two
parameter linear regression models.  The functions are declared in the
header file 'gsl_fit.h'.

* Menu:

* Linear regression with a constant term::
* Linear regression without a constant term::


File: gsl-ref.info,  Node: Linear regression with a constant term,  Next: Linear regression without a constant term,  Up: Linear regression

38.2.1 Linear regression with a constant term
---------------------------------------------

The functions described in this section can be used to perform
least-squares fits to a straight line model, Y(c,x) = c_0 + c_1 x.

 -- Function: int gsl_fit_linear (const double * X, const size_t
          XSTRIDE, const double * Y, const size_t YSTRIDE, size_t N,
          double * C0, double * C1, double * COV00, double * COV01,
          double * COV11, double * SUMSQ)
     This function computes the best-fit linear regression coefficients
     (C0,C1) of the model Y = c_0 + c_1 X for the dataset (X, Y), two
     vectors of length N with strides XSTRIDE and YSTRIDE.  The errors
     on Y are assumed unknown so the variance-covariance matrix for the
     parameters (C0, C1) is estimated from the scatter of the points
     around the best-fit line and returned via the parameters (COV00,
     COV01, COV11).  The sum of squares of the residuals from the
     best-fit line is returned in SUMSQ.  Note: the correlation
     coefficient of the data can be computed using
     'gsl_stats_correlation' (*note Correlation::), it does not depend
     on the fit.

 -- Function: int gsl_fit_wlinear (const double * X, const size_t
          XSTRIDE, const double * W, const size_t WSTRIDE, const double
          * Y, const size_t YSTRIDE, size_t N, double * C0, double * C1,
          double * COV00, double * COV01, double * COV11, double *
          CHISQ)
     This function computes the best-fit linear regression coefficients
     (C0,C1) of the model Y = c_0 + c_1 X for the weighted dataset (X,
     Y), two vectors of length N with strides XSTRIDE and YSTRIDE.  The
     vector W, of length N and stride WSTRIDE, specifies the weight of
     each datapoint.  The weight is the reciprocal of the variance for
     each datapoint in Y.

     The covariance matrix for the parameters (C0, C1) is computed using
     the weights and returned via the parameters (COV00, COV01, COV11).
     The weighted sum of squares of the residuals from the best-fit
     line, \chi^2, is returned in CHISQ.

 -- Function: int gsl_fit_linear_est (double X, double C0, double C1,
          double COV00, double COV01, double COV11, double * Y, double *
          Y_ERR)
     This function uses the best-fit linear regression coefficients C0,
     C1 and their covariance COV00, COV01, COV11 to compute the fitted
     function Y and its standard deviation Y_ERR for the model Y = c_0 +
     c_1 X at the point X.


File: gsl-ref.info,  Node: Linear regression without a constant term,  Prev: Linear regression with a constant term,  Up: Linear regression

38.2.2 Linear regression without a constant term
------------------------------------------------

The functions described in this section can be used to perform
least-squares fits to a straight line model without a constant term, Y =
c_1 X.

 -- Function: int gsl_fit_mul (const double * X, const size_t XSTRIDE,
          const double * Y, const size_t YSTRIDE, size_t N, double * C1,
          double * COV11, double * SUMSQ)
     This function computes the best-fit linear regression coefficient
     C1 of the model Y = c_1 X for the datasets (X, Y), two vectors of
     length N with strides XSTRIDE and YSTRIDE.  The errors on Y are
     assumed unknown so the variance of the parameter C1 is estimated
     from the scatter of the points around the best-fit line and
     returned via the parameter COV11.  The sum of squares of the
     residuals from the best-fit line is returned in SUMSQ.

 -- Function: int gsl_fit_wmul (const double * X, const size_t XSTRIDE,
          const double * W, const size_t WSTRIDE, const double * Y,
          const size_t YSTRIDE, size_t N, double * C1, double * COV11,
          double * SUMSQ)
     This function computes the best-fit linear regression coefficient
     C1 of the model Y = c_1 X for the weighted datasets (X, Y), two
     vectors of length N with strides XSTRIDE and YSTRIDE.  The vector
     W, of length N and stride WSTRIDE, specifies the weight of each
     datapoint.  The weight is the reciprocal of the variance for each
     datapoint in Y.

     The variance of the parameter C1 is computed using the weights and
     returned via the parameter COV11.  The weighted sum of squares of
     the residuals from the best-fit line, \chi^2, is returned in CHISQ.

 -- Function: int gsl_fit_mul_est (double X, double C1, double COV11,
          double * Y, double * Y_ERR)
     This function uses the best-fit linear regression coefficient C1
     and its covariance COV11 to compute the fitted function Y and its
     standard deviation Y_ERR for the model Y = c_1 X at the point X.


File: gsl-ref.info,  Node: Multi-parameter regression,  Next: Regularized regression,  Prev: Linear regression,  Up: Least-Squares Fitting

38.3 Multi-parameter regression
===============================

This section describes routines which perform least squares fits to a
linear model by minimizing the cost function
     \chi^2 = \sum_i w_i (y_i - \sum_j X_ij c_j)^2 = || y - Xc ||_W^2
   where y is a vector of n observations, X is an n-by-p matrix of
predictor variables, c is a vector of the p unknown best-fit parameters
to be estimated, and ||r||_W^2 = r^T W r.  The matrix W =
diag(w_1,w_2,...,w_n) defines the weights or uncertainties of the
observation vector.

   This formulation can be used for fits to any number of functions
and/or variables by preparing the n-by-p matrix X appropriately.  For
example, to fit to a p-th order polynomial in X, use the following
matrix,

     X_{ij} = x_i^j

where the index i runs over the observations and the index j runs from 0
to p-1.

   To fit to a set of p sinusoidal functions with fixed frequencies
\omega_1, \omega_2, ..., \omega_p, use,

     X_{ij} = sin(\omega_j x_i)

To fit to p independent variables x_1, x_2, ..., x_p, use,

     X_{ij} = x_j(i)

where x_j(i) is the i-th value of the predictor variable x_j.

   The solution of the general linear least-squares system requires an
additional working space for intermediate results, such as the singular
value decomposition of the matrix X.

   These functions are declared in the header file 'gsl_multifit.h'.

 -- Function: gsl_multifit_linear_workspace * gsl_multifit_linear_alloc
          (const size_t N, const size_t P)
     This function allocates a workspace for fitting a model to a
     maximum of N observations using a maximum of P parameters.  The
     user may later supply a smaller least squares system if desired.
     The size of the workspace is O(np + p^2).

 -- Function: void gsl_multifit_linear_free
          (gsl_multifit_linear_workspace * WORK)
     This function frees the memory associated with the workspace W.

 -- Function: int gsl_multifit_linear_svd (const gsl_matrix * X,
          gsl_multifit_linear_workspace * WORK)
     This function performs a singular value decomposition of the matrix
     X and stores the SVD factors internally in WORK.

 -- Function: int gsl_multifit_linear_bsvd (const gsl_matrix * X,
          gsl_multifit_linear_workspace * WORK)
     This function performs a singular value decomposition of the matrix
     X and stores the SVD factors internally in WORK.  The matrix X is
     first balanced by applying column scaling factors to improve the
     accuracy of the singular values.

 -- Function: int gsl_multifit_linear (const gsl_matrix * X, const
          gsl_vector * Y, gsl_vector * C, gsl_matrix * COV, double *
          CHISQ, gsl_multifit_linear_workspace * WORK)
     This function computes the best-fit parameters C of the model y = X
     c for the observations Y and the matrix of predictor variables X,
     using the preallocated workspace provided in WORK.  The p-by-p
     variance-covariance matrix of the model parameters COV is set to
     \sigma^2 (X^T X)^{-1}, where \sigma is the standard deviation of
     the fit residuals.  The sum of squares of the residuals from the
     best-fit, \chi^2, is returned in CHISQ.  If the coefficient of
     determination is desired, it can be computed from the expression
     R^2 = 1 - \chi^2 / TSS, where the total sum of squares (TSS) of the
     observations Y may be computed from 'gsl_stats_tss'.

     The best-fit is found by singular value decomposition of the matrix
     X using the modified Golub-Reinsch SVD algorithm, with column
     scaling to improve the accuracy of the singular values.  Any
     components which have zero singular value (to machine precision)
     are discarded from the fit.

 -- Function: int gsl_multifit_linear_tsvd (const gsl_matrix * X, const
          gsl_vector * Y, const double TOL, gsl_vector * C, gsl_matrix *
          COV, double * CHISQ, size_t * RANK,
          gsl_multifit_linear_workspace * WORK)
     This function computes the best-fit parameters C of the model y = X
     c for the observations Y and the matrix of predictor variables X,
     using a truncated SVD expansion.  Singular values which satisfy s_i
     \le tol \times s_0 are discarded from the fit, where s_0 is the
     largest singular value.  The p-by-p variance-covariance matrix of
     the model parameters COV is set to \sigma^2 (X^T X)^{-1}, where
     \sigma is the standard deviation of the fit residuals.  The sum of
     squares of the residuals from the best-fit, \chi^2, is returned in
     CHISQ.  The effective rank (number of singular values used in
     solution) is returned in RANK.  If the coefficient of determination
     is desired, it can be computed from the expression R^2 = 1 - \chi^2
     / TSS, where the total sum of squares (TSS) of the observations Y
     may be computed from 'gsl_stats_tss'.

 -- Function: int gsl_multifit_wlinear (const gsl_matrix * X, const
          gsl_vector * W, const gsl_vector * Y, gsl_vector * C,
          gsl_matrix * COV, double * CHISQ,
          gsl_multifit_linear_workspace * WORK)
     This function computes the best-fit parameters C of the weighted
     model y = X c for the observations Y with weights W and the matrix
     of predictor variables X, using the preallocated workspace provided
     in WORK.  The p-by-p covariance matrix of the model parameters COV
     is computed as (X^T W X)^{-1}.  The weighted sum of squares of the
     residuals from the best-fit, \chi^2, is returned in CHISQ.  If the
     coefficient of determination is desired, it can be computed from
     the expression R^2 = 1 - \chi^2 / WTSS, where the weighted total
     sum of squares (WTSS) of the observations Y may be computed from
     'gsl_stats_wtss'.

 -- Function: int gsl_multifit_wlinear_tsvd (const gsl_matrix * X, const
          gsl_vector * W, const gsl_vector * Y, const double TOL,
          gsl_vector * C, gsl_matrix * COV, double * CHISQ, size_t *
          RANK, gsl_multifit_linear_workspace * WORK)
     This function computes the best-fit parameters C of the weighted
     model y = X c for the observations Y with weights W and the matrix
     of predictor variables X, using a truncated SVD expansion.
     Singular values which satisfy s_i \le tol \times s_0 are discarded
     from the fit, where s_0 is the largest singular value.  The p-by-p
     covariance matrix of the model parameters COV is computed as (X^T W
     X)^{-1}.  The weighted sum of squares of the residuals from the
     best-fit, \chi^2, is returned in CHISQ.  The effective rank of the
     system (number of singular values used in the solution) is returned
     in RANK.  If the coefficient of determination is desired, it can be
     computed from the expression R^2 = 1 - \chi^2 / WTSS, where the
     weighted total sum of squares (WTSS) of the observations Y may be
     computed from 'gsl_stats_wtss'.

 -- Function: int gsl_multifit_linear_est (const gsl_vector * X, const
          gsl_vector * C, const gsl_matrix * COV, double * Y, double *
          Y_ERR)
     This function uses the best-fit multilinear regression coefficients
     C and their covariance matrix COV to compute the fitted function
     value Y and its standard deviation Y_ERR for the model y = x.c at
     the point X.

 -- Function: int gsl_multifit_linear_residuals (const gsl_matrix * X,
          const gsl_vector * Y, const gsl_vector * C, gsl_vector * R)
     This function computes the vector of residuals r = y - X c for the
     observations Y, coefficients C and matrix of predictor variables X.

 -- Function: size_t gsl_multifit_linear_rank (const double TOL, const
          gsl_multifit_linear_workspace * WORK)
     This function returns the rank of the matrix X which must first
     have its singular value decomposition computed.  The rank is
     computed by counting the number of singular values \sigma_j which
     satisfy \sigma_j > tol \times \sigma_0, where \sigma_0 is the
     largest singular value.


File: gsl-ref.info,  Node: Regularized regression,  Next: Robust linear regression,  Prev: Multi-parameter regression,  Up: Least-Squares Fitting

38.4 Regularized regression
===========================

Ordinary weighted least squares models seek a solution vector c which
minimizes the residual
     \chi^2 = || y - Xc ||_W^2
   where y is the n-by-1 observation vector, X is the n-by-p design
matrix, c is the p-by-1 solution vector, W = diag(w_1,...,w_n) is the
data weighting matrix, and ||r||_W^2 = r^T W r.  In cases where the
least squares matrix X is ill-conditioned, small perturbations (ie:
noise) in the observation vector could lead to widely different solution
vectors c.  One way of dealing with ill-conditioned matrices is to use a
"truncated SVD" in which small singular values, below some given
tolerance, are discarded from the solution.  The truncated SVD method is
available using the functions 'gsl_multifit_linear_tsvd' and
'gsl_multifit_wlinear_tsvd'.  Another way to help solve ill-posed
problems is to include a regularization term in the least squares
minimization
     \chi^2 = || y - Xc ||_W^2 + \lambda^2 || L c ||^2
   for a suitably chosen regularization parameter \lambda and matrix L.
This type of regularization is known as Tikhonov, or ridge, regression.
In some applications, L is chosen as the identity matrix, giving
preference to solution vectors c with smaller norms.  Including this
regularization term leads to the explicit "normal equations" solution
     c = ( X^T W X + \lambda^2 L^T L )^-1 X^T W y
   which reduces to the ordinary least squares solution when L = 0.  In
practice, it is often advantageous to transform a regularized least
squares system into the form
     \chi^2 = || y~ - X~ c~ ||^2 + \lambda^2 || c~ ||^2
   This is known as the Tikhonov "standard form" and has the normal
equations solution \tilde{c} = \left( \tilde{X}^T \tilde{X} + \lambda^2
I \right)^{-1} \tilde{X}^T \tilde{y}.  For an m-by-p matrix L which is
full rank and has m >= p (ie: L is square or has more rows than
columns), we can calculate the "thin" QR decomposition of L, and note
that ||L c|| = ||R c|| since the Q factor will not change the norm.
Since R is p-by-p, we can then use the transformation
     X~ = sqrt(W) X R^-1
     y~ = sqrt(W) y
     c~ = R c
   to achieve the standard form.  For a rectangular matrix L with m < p,
a more sophisticated approach is needed (see Hansen 1998, chapter 2.3).
In practice, the normal equations solution above is not desirable due to
numerical instabilities, and so the system is solved using the singular
value decomposition of the matrix \tilde{X}.  The matrix L is often
chosen as the identity matrix, or as a first or second finite difference
operator, to ensure a smoothly varying coefficient vector c, or as a
diagonal matrix to selectively damp each model parameter differently.
If L \ne I, the user must first convert the least squares problem to
standard form using 'gsl_multifit_linear_stdform1' or
'gsl_multifit_linear_stdform2', solve the system, and then backtransform
the solution vector to recover the solution of the original problem (see
'gsl_multifit_linear_genform1' and 'gsl_multifit_linear_genform2').

   In many regularization problems, care must be taken when choosing the
regularization parameter \lambda.  Since both the residual norm ||y - X
c|| and solution norm ||L c|| are being minimized, the parameter \lambda
represents a tradeoff between minimizing either the residuals or the
solution vector.  A common tool for visualizing the comprimise between
the minimization of these two quantities is known as the L-curve.  The
L-curve is a log-log plot of the residual norm ||y - X c|| on the
horizontal axis and the solution norm ||L c|| on the vertical axis.
This curve nearly always as an L shaped appearance, with a distinct
corner separating the horizontal and vertical sections of the curve.
The regularization parameter corresponding to this corner is often
chosen as the optimal value.  GSL provides routines to calculate the
L-curve for all relevant regularization parameters as well as locating
the corner.

   Another method of choosing the regularization parameter is known as
Generalized Cross Validation (GCV). This method is based on the idea
that if an arbitrary element y_i is left out of the right hand side, the
resulting regularized solution should predict this element accurately.
This leads to choosing the parameter \lambda which minimizes the GCV
function

     G(\lambda) = (||y - X c_{\lambda}||^2) / Tr(I_n - X X^I)^2

   where X_{\lambda}^I is the matrix which relates the solution
c_{\lambda} to the right hand side y, ie: c_{\lambda} = X_{\lambda}^I y.
GSL provides routines to compute the GCV curve and its minimum.

For most applications, the steps required to solve a regularized least
squares problem are as follows:

  1. Construct the least squares system (X, y, W, L)

  2. Transform the system to standard form (\tilde{X},\tilde{y}).  This
     step can be skipped if L = I_p and W = I_n.

  3. Calculate the SVD of \tilde{X}.

  4. Determine an appropriate regularization parameter \lambda (using
     for example L-curve or GCV analysis).

  5. Solve the standard form system using the chosen \lambda and the SVD
     of \tilde{X}.

  6. Backtransform the standard form solution \tilde{c} to recover the
     original solution vector c.

 -- Function: int gsl_multifit_linear_stdform1 (const gsl_vector * L,
          const gsl_matrix * X, const gsl_vector * Y, gsl_matrix * XS,
          gsl_vector * YS, gsl_multifit_linear_workspace * WORK)
 -- Function: int gsl_multifit_linear_wstdform1 (const gsl_vector * L,
          const gsl_matrix * X, const gsl_vector * W, const gsl_vector *
          Y, gsl_matrix * XS, gsl_vector * YS,
          gsl_multifit_linear_workspace * WORK)
     These functions define a regularization matrix L =
     diag(l_0,l_1,...,l_{p-1}).  The diagonal matrix element l_i is
     provided by the ith element of the input vector L.  The n-by-p
     least squares matrix X and vector Y of length n are then converted
     to standard form as described above and the parameters
     (\tilde{X},\tilde{y}) are stored in XS and YS on output.  XS and YS
     have the same dimensions as X and Y.  Optional data weights may be
     supplied in the vector W of length n.  In order to apply this
     transformation, L^{-1} must exist and so none of the l_i may be
     zero.  After the standard form system has been solved, use
     'gsl_multifit_linear_genform1' to recover the original solution
     vector.  It is allowed to have X = XS and Y = YS for an in-place
     transform.  In order to perform a weighted regularized fit with L =
     I, the user may call 'gsl_multifit_linear_applyW' to convert to
     standard form.

 -- Function: int gsl_multifit_linear_L_decomp (gsl_matrix * L,
          gsl_vector * TAU)
     This function factors the m-by-p regularization matrix L into a
     form needed for the later transformation to standard form.  L may
     have any number of rows m.  If m \ge p the QR decomposition of L is
     computed and stored in L on output.  If m < p, the QR decomposition
     of L^T is computed and stored in L on output.  On output, the
     Householder scalars are stored in the vector TAU of size MIN(m,p).
     These outputs will be used by 'gsl_multifit_linear_wstdform2' to
     complete the transformation to standard form.

 -- Function: int gsl_multifit_linear_stdform2 (const gsl_matrix * LQR,
          const gsl_vector * LTAU, const gsl_matrix * X, const
          gsl_vector * Y, gsl_matrix * XS, gsl_vector * YS, gsl_matrix *
          M, gsl_multifit_linear_workspace * WORK)
 -- Function: int gsl_multifit_linear_wstdform2 (const gsl_matrix * LQR,
          const gsl_vector * LTAU, const gsl_matrix * X, const
          gsl_vector * W, const gsl_vector * Y, gsl_matrix * XS,
          gsl_vector * YS, gsl_matrix * M, gsl_multifit_linear_workspace
          * WORK)
     These functions convert the least squares system (X,Y,W,L) to
     standard form (\tilde{X},\tilde{y}) which are stored in XS and YS
     respectively.  The m-by-p regularization matrix L is specified by
     the inputs LQR and LTAU, which are outputs from
     'gsl_multifit_linear_L_decomp'.  The dimensions of the standard
     form parameters (\tilde{X},\tilde{y}) depend on whether m is larger
     or less than p.  For m \ge p, XS is n-by-p, YS is n-by-1, and M is
     not used.  For m < p, XS is (n - p + m)-by-m, YS is (n - p +
     m)-by-1, and M is additional n-by-p workspace, which is required to
     recover the original solution vector after the system has been
     solved (see 'gsl_multifit_linear_genform2').  Optional data weights
     may be supplied in the vector W of length n, where W = diag(w).

 -- Function: int gsl_multifit_linear_solve (const double LAMBDA, const
          gsl_matrix * XS, const gsl_vector * YS, gsl_vector * CS,
          double * RNORM, double * SNORM, gsl_multifit_linear_workspace
          * WORK)
     This function computes the regularized best-fit parameters
     \tilde{c} which minimize the cost function \chi^2 = || \tilde{y} -
     \tilde{X} \tilde{c} ||^2 + \lambda^2 || \tilde{c} ||^2 which is in
     standard form.  The least squares system must therefore be
     converted to standard form prior to calling this function.  The
     observation vector \tilde{y} is provided in YS and the matrix of
     predictor variables \tilde{X} in XS.  The solution vector \tilde{c}
     is returned in CS, which has length min(m,p).  The SVD of XS must
     be computed prior to calling this function, using
     'gsl_multifit_linear_svd'.  The regularization parameter \lambda is
     provided in LAMBDA.  The residual norm || \tilde{y} - \tilde{X}
     \tilde{c} || = ||y - X c||_W is returned in RNORM.  The solution
     norm || \tilde{c} || = ||L c|| is returned in SNORM.

 -- Function: int gsl_multifit_linear_genform1 (const gsl_vector * L,
          const gsl_vector * CS, gsl_vector * C,
          gsl_multifit_linear_workspace * WORK)
     After a regularized system has been solved with L =
     diag(\l_0,\l_1,...,\l_{p-1}), this function backtransforms the
     standard form solution vector CS to recover the solution vector of
     the original problem C.  The diagonal matrix elements l_i are
     provided in the vector L.  It is allowed to have C = CS for an
     in-place transform.

 -- Function: int gsl_multifit_linear_genform2 (const gsl_matrix * LQR,
          const gsl_vector * LTAU, const gsl_matrix * X, const
          gsl_vector * Y, const gsl_vector * CS, const gsl_matrix * M,
          gsl_vector * C, gsl_multifit_linear_workspace * WORK)
 -- Function: int gsl_multifit_linear_wgenform2 (const gsl_matrix * LQR,
          const gsl_vector * LTAU, const gsl_matrix * X, const
          gsl_vector * W, const gsl_vector * Y, const gsl_vector * CS,
          const gsl_matrix * M, gsl_vector * C,
          gsl_multifit_linear_workspace * WORK)
     After a regularized system has been solved with a general
     rectangular matrix L, specified by (LQR,LTAU), this function
     backtransforms the standard form solution CS to recover the
     solution vector of the original problem, which is stored in C, of
     length p.  The original least squares matrix and observation vector
     are provided in X and Y respectively.  M is the matrix computed by
     'gsl_multifit_linear_stdform2'.  For weighted fits, the weight
     vector W must also be supplied.

 -- Function: int gsl_multifit_linear_applyW (const gsl_matrix * X,
          const gsl_vector * W, const gsl_vector * Y, gsl_matrix * WX,
          gsl_vector * WY)
     For weighted least squares systems with L = I, this function may be
     used to convert the system to standard form by applying the weight
     matrix W = diag(W) to the least squares matrix X and observation
     vector Y.  On output, WX is equal to W^{1/2} X and WY is equal to
     W^{1/2} y.  It is allowed for WX = X and WY = Y for an in-place
     transform.

 -- Function: int gsl_multifit_linear_lcurve (const gsl_vector * Y,
          gsl_vector * REG_PARAM, gsl_vector * RHO, gsl_vector * ETA,
          gsl_multifit_linear_workspace * WORK)
     This function computes the L-curve for a least squares system using
     the right hand side vector Y and the SVD decomposition of the least
     squares matrix X, which must be provided to
     'gsl_multifit_linear_svd' prior to calling this function.  The
     output vectors REG_PARAM, RHO, and ETA must all be the same size,
     and will contain the regularization parameters \lambda_i, residual
     norms ||y - X c_i||, and solution norms || L c_i || which compose
     the L-curve, where c_i is the regularized solution vector
     corresponding to \lambda_i.  The user may determine the number of
     points on the L-curve by adjusting the size of these input arrays.
     The regularization parameters \lambda_i are estimated from the
     singular values of X, and chosen to represent the most relevant
     portion of the L-curve.

 -- Function: int gsl_multifit_linear_lcorner (const gsl_vector * RHO,
          const gsl_vector * ETA, size_t * IDX)
     This function attempts to locate the corner of the L-curve (||y - X
     c||, ||L c||) defined by the RHO and ETA input arrays respectively.
     The corner is defined as the point of maximum curvature of the
     L-curve in log-log scale.  The RHO and ETA arrays can be outputs of
     'gsl_multifit_linear_lcurve'.  The algorithm used simply fits a
     circle to 3 consecutive points on the L-curve and uses the circle's
     radius to determine the curvature at the middle point.  Therefore,
     the input array sizes must be \ge 3.  With more points provided for
     the L-curve, a better estimate of the curvature can be obtained.
     The array index corresponding to maximum curvature (ie: the corner)
     is returned in IDX.  If the input arrays contain colinear points,
     this function could fail and return 'GSL_EINVAL'.

 -- Function: int gsl_multifit_linear_lcorner2 (const gsl_vector *
          REG_PARAM, const gsl_vector * ETA, size_t * IDX)
     This function attempts to locate the corner of an alternate L-curve
     (\lambda^2, ||L c||^2) studied by Rezghi and Hosseini, 2009.  This
     alternate L-curve can provide better estimates of the
     regularization parameter for smooth solution vectors.  The
     regularization parameters \lambda and solution norms ||L c|| are
     provided in the REG_PARAM and ETA input arrays respectively.  The
     corner is defined as the point of maximum curvature of this
     alternate L-curve in linear scale.  The REG_PARAM and ETA arrays
     can be outputs of 'gsl_multifit_linear_lcurve'.  The algorithm used
     simply fits a circle to 3 consecutive points on the L-curve and
     uses the circle's radius to determine the curvature at the middle
     point.  Therefore, the input array sizes must be \ge 3.  With more
     points provided for the L-curve, a better estimate of the curvature
     can be obtained.  The array index corresponding to maximum
     curvature (ie: the corner) is returned in IDX.  If the input arrays
     contain colinear points, this function could fail and return
     'GSL_EINVAL'.

 -- Function: int gsl_multifit_linear_gcv_init(const gsl_vector * Y,
          gsl_vector * REG_PARAM, gsl_vector * UTY, double * DELTA0,
          gsl_multifit_linear_workspace * WORK)
     This function performs some initialization in preparation for
     computing the GCV curve and its minimum.  The right hand side
     vector is provided in Y.  On output, REG_PARAM is set to a vector
     of regularization parameters in decreasing order and may be of any
     size.  The vector UTY of size p is set to U^T y.  The parameter
     DELTA0 is needed for subsequent steps of the GCV calculation.

 -- Function: int gsl_multifit_linear_gcv_curve(const gsl_vector *
          REG_PARAM, const gsl_vector * UTY, const double DELTA0,
          gsl_vector * G, gsl_multifit_linear_workspace * WORK)
     This funtion calculates the GCV curve G(\lambda) and stores it in G
     on output, which must be the same size as REG_PARAM.  The inputs
     REG_PARAM, UTY and DELTA0 are computed in
     'gsl_multifit_linear_gcv_init'.

 -- Function: int gsl_multifit_linear_gcv_min(const gsl_vector *
          REG_PARAM, const gsl_vector * UTY, const gsl_vector * G, const
          double DELTA0, double * LAMBDA, gsl_multifit_linear_workspace
          * WORK)
     This function computes the value of the regularization parameter
     which minimizes the GCV curve G(\lambda) and stores it in LAMBDA.
     The input G is calculated by 'gsl_multifit_linear_gcv_curve' and
     the inputs REG_PARAM, UTY and DELTA0 are computed by
     'gsl_multifit_linear_gcv_init'.

 -- Function: double gsl_multifit_linear_gcv_calc(const double LAMBDA,
          const gsl_vector * UTY, const double DELTA0,
          gsl_multifit_linear_workspace * WORK)
     This function returns the value of the GCV curve G(\lambda)
     corresponding to the input LAMBDA.

 -- Function: int gsl_multifit_linear_gcv(const gsl_vector * Y,
          gsl_vector * REG_PARAM, gsl_vector * G, double * LAMBDA,
          double * G_LAMBDA, gsl_multifit_linear_workspace * WORK)
     This function combines the steps 'gcv_init', 'gcv_curve', and
     'gcv_min' defined above into a single function.  The input Y is the
     right hand side vector.  On output, REG_PARAM and G, which must be
     the same size, are set to vectors of \lambda and G(\lambda) values
     respectively.  The output LAMBDA is set to the optimal value of
     \lambda which minimizes the GCV curve.  The minimum value of the
     GCV curve is returned in G_LAMBDA.

 -- Function: int gsl_multifit_linear_Lk (const size_t P, const size_t
          K, gsl_matrix * L)
     This function computes the discrete approximation to the derivative
     operator L_k of order K on a regular grid of P points and stores it
     in L.  The dimensions of L are (p-k)-by-p.

 -- Function: int gsl_multifit_linear_Lsobolev (const size_t P, const
          size_t KMAX, const gsl_vector * ALPHA, gsl_matrix * L,
          gsl_multifit_linear_workspace * WORK)
     This function computes the regularization matrix L corresponding to
     the weighted Sobolov norm ||L c||^2 = \sum_k \alpha_k^2 ||L_k c||^2
     where L_k approximates the derivative operator of order k.  This
     regularization norm can be useful in applications where it is
     necessary to smooth several derivatives of the solution.  P is the
     number of model parameters, KMAX is the highest derivative to
     include in the summation above, and ALPHA is the vector of weights
     of size KMAX + 1, where ALPHA[k] = \alpha_k is the weight assigned
     to the derivative of order k.  The output matrix L is size P-by-P
     and upper triangular.

 -- Function: double gsl_multifit_linear_rcond (const
          gsl_multifit_linear_workspace * WORK)
     This function returns the reciprocal condition number of the least
     squares matrix X, defined as the ratio of the smallest and largest
     singular values, rcond = \sigma_{min}/\sigma_{max}.  The routine
     'gsl_multifit_linear_svd' must first be called to compute the SVD
     of X.


File: gsl-ref.info,  Node: Robust linear regression,  Next: Large Dense Linear Systems,  Prev: Regularized regression,  Up: Least-Squares Fitting

38.5 Robust linear regression
=============================

Ordinary least squares (OLS) models are often heavily influenced by the
presence of outliers.  Outliers are data points which do not follow the
general trend of the other observations, although there is strictly no
precise definition of an outlier.  Robust linear regression refers to
regression algorithms which are robust to outliers.  The most common
type of robust regression is M-estimation.  The general M-estimator
minimizes the objective function

     \sum_i \rho(e_i) = \sum_i \rho (y_i - Y(c, x_i))

   where e_i = y_i - Y(c, x_i) is the residual of the ith data point,
and \rho(e_i) is a function which should have the following properties:
     \rho(e) \ge 0
     \rho(0) = 0
     \rho(-e) = \rho(e)
     \rho(e_1) > \rho(e_2) for |e_1| > |e_2|
The special case of ordinary least squares is given by \rho(e_i) =
e_i^2.  Letting \psi = \rho' be the derivative of \rho, differentiating
the objective function with respect to the coefficients c and setting
the partial derivatives to zero produces the system of equations

     \sum_i \psi(e_i) X_i = 0

   where X_i is a vector containing row i of the design matrix X.  Next,
we define a weight function w(e) = \psi(e)/e, and let w_i = w(e_i):

     \sum_i w_i e_i X_i = 0

   This system of equations is equivalent to solving a weighted ordinary
least squares problem, minimizing \chi^2 = \sum_i w_i e_i^2.  The
weights however, depend on the residuals e_i, which depend on the
coefficients c, which depend on the weights.  Therefore, an iterative
solution is used, called Iteratively Reweighted Least Squares (IRLS).
  1. Compute initial estimates of the coefficients c^{(0)} using
     ordinary least squares

  2. For iteration k, form the residuals e_i^{(k)} = (y_i - X_i
     c^{(k-1)})/(t \sigma^{(k)} \sqrt{1 - h_i}), where t is a tuning
     constant depending on the choice of \psi, and h_i are the
     statistical leverages (diagonal elements of the matrix X (X^T
     X)^{-1} X^T).  Including t and h_i in the residual calculation has
     been shown to improve the convergence of the method.  The residual
     standard deviation is approximated as \sigma^{(k)} = MAD / 0.6745,
     where MAD is the Median-Absolute-Deviation of the n-p largest
     residuals from the previous iteration.

  3. Compute new weights w_i^{(k)} = \psi(e_i^{(k)})/e_i^{(k)}.

  4. Compute new coefficients c^{(k)} by solving the weighted least
     squares problem with weights w_i^{(k)}.

  5. Steps 2 through 4 are iterated until the coefficients converge or
     until some maximum iteration limit is reached.  Coefficients are
     tested for convergence using the critera:

          |c_i^(k) - c_i^(k-1)| \le \epsilon \times max(|c_i^(k)|, |c_i^(k-1)|)

     for all 0 \le i < p where \epsilon is a small tolerance factor.
The key to this method lies in selecting the function \psi(e_i) to
assign smaller weights to large residuals, and larger weights to smaller
residuals.  As the iteration proceeds, outliers are assigned smaller and
smaller weights, eventually having very little or no effect on the
fitted model.

 -- Function: gsl_multifit_robust_workspace * gsl_multifit_robust_alloc
          (const gsl_multifit_robust_type * T, const size_t N, const
          size_t P)
     This function allocates a workspace for fitting a model to N
     observations using P parameters.  The size of the workspace is O(np
     + p^2).  The type T specifies the function \psi and can be selected
     from the following choices.
      -- Robust type: gsl_multifit_robust_default
          This specifies the 'gsl_multifit_robust_bisquare' type (see
          below) and is a good general purpose choice for robust
          regression.

      -- Robust type: gsl_multifit_robust_bisquare
          This is Tukey's biweight (bisquare) function and is a good
          general purpose choice for robust regression.  The weight
          function is given by

               w(e) = (1 - e^2)^2

          and the default tuning constant is t = 4.685.

      -- Robust type: gsl_multifit_robust_cauchy
          This is Cauchy's function, also known as the Lorentzian
          function.  This function does not guarantee a unique solution,
          meaning different choices of the coefficient vector C could
          minimize the objective function.  Therefore this option should
          be used with care.  The weight function is given by

               w(e) = 1 / (1 + e^2)

          and the default tuning constant is t = 2.385.

      -- Robust type: gsl_multifit_robust_fair
          This is the fair \rho function, which guarantees a unique
          solution and has continuous derivatives to three orders.  The
          weight function is given by

               w(e) = 1 / (1 + |e|)

          and the default tuning constant is t = 1.400.

      -- Robust type: gsl_multifit_robust_huber
          This specifies Huber's \rho function, which is a parabola in
          the vicinity of zero and increases linearly for a given
          threshold |e| > t.  This function is also considered an
          excellent general purpose robust estimator, however,
          occasional difficulties can be encountered due to the
          discontinuous first derivative of the \psi function.  The
          weight function is given by

               w(e) = 1/max(1,|e|)

          and the default tuning constant is t = 1.345.

      -- Robust type: gsl_multifit_robust_ols
          This specifies the ordinary least squares solution, which can
          be useful for quickly checking the difference between the
          various robust and OLS solutions.  The weight function is
          given by

               w(e) = 1

          and the default tuning constant is t = 1.

      -- Robust type: gsl_multifit_robust_welsch
          This specifies the Welsch function which can perform well in
          cases where the residuals have an exponential distribution.
          The weight function is given by

               w(e) = \exp(-e^2)

          and the default tuning constant is t = 2.985.

 -- Function: void gsl_multifit_robust_free
          (gsl_multifit_robust_workspace * W)
     This function frees the memory associated with the workspace W.

 -- Function: const char * gsl_multifit_robust_name (const
          gsl_multifit_robust_workspace * W)
     This function returns the name of the robust type T specified to
     'gsl_multifit_robust_alloc'.

 -- Function: int gsl_multifit_robust_tune (const double TUNE,
          gsl_multifit_robust_workspace * W)
     This function sets the tuning constant t used to adjust the
     residuals at each iteration to TUNE.  Decreasing the tuning
     constant increases the downweight assigned to large residuals,
     while increasing the tuning constant decreases the downweight
     assigned to large residuals.

 -- Function: int gsl_multifit_robust_maxiter (const size_t MAXITER,
          gsl_multifit_robust_workspace * W)
     This function sets the maximum number of iterations in the
     iteratively reweighted least squares algorithm to MAXITER.  By
     default, this value is set to 100 by 'gsl_multifit_robust_alloc'.

 -- Function: int gsl_multifit_robust_weights (const gsl_vector * R,
          gsl_vector * WTS, gsl_multifit_robust_workspace * W)
     This function assigns weights to the vector WTS using the residual
     vector R and previously specified weighting function.  The output
     weights are given by wts_i = w(r_i / (t \sigma)), where the
     weighting functions w are detailed in 'gsl_multifit_robust_alloc'.
     \sigma is an estimate of the residual standard deviation based on
     the Median-Absolute-Deviation and t is the tuning constant.  This
     function is useful if the user wishes to implement their own robust
     regression rather than using the supplied 'gsl_multifit_robust'
     routine below.

 -- Function: int gsl_multifit_robust (const gsl_matrix * X, const
          gsl_vector * Y, gsl_vector * C, gsl_matrix * COV,
          gsl_multifit_robust_workspace * W)
     This function computes the best-fit parameters C of the model y = X
     c for the observations Y and the matrix of predictor variables X,
     attemping to reduce the influence of outliers using the algorithm
     outlined above.  The p-by-p variance-covariance matrix of the model
     parameters COV is estimated as \sigma^2 (X^T X)^{-1}, where \sigma
     is an approximation of the residual standard deviation using the
     theory of robust regression.  Special care must be taken when
     estimating \sigma and other statistics such as R^2, and so these
     are computed internally and are available by calling the function
     'gsl_multifit_robust_statistics'.

     If the coefficients do not converge within the maximum iteration
     limit, the function returns 'GSL_EMAXITER'.  In this case, the
     current estimates of the coefficients and covariance matrix are
     returned in C and COV and the internal fit statistics are computed
     with these estimates.

 -- Function: int gsl_multifit_robust_est (const gsl_vector * X, const
          gsl_vector * C, const gsl_matrix * COV, double * Y, double *
          Y_ERR)
     This function uses the best-fit robust regression coefficients C
     and their covariance matrix COV to compute the fitted function
     value Y and its standard deviation Y_ERR for the model y = x.c at
     the point X.

 -- Function: int gsl_multifit_robust_residuals (const gsl_matrix * X,
          const gsl_vector * Y, const gsl_vector * C, gsl_vector * R,
          gsl_multifit_robust_workspace * W)
     This function computes the vector of studentized residuals r_i =
     {y_i - (X c)_i \over \sigma \sqrt{1 - h_i}} for the observations Y,
     coefficients C and matrix of predictor variables X.  The routine
     'gsl_multifit_robust' must first be called to compute the
     statisical leverages h_i of the matrix X and residual standard
     deviation estimate \sigma.

 -- Function: gsl_multifit_robust_stats gsl_multifit_robust_statistics
          (const gsl_multifit_robust_workspace * W)
     This function returns a structure containing relevant statistics
     from a robust regression.  The function 'gsl_multifit_robust' must
     be called first to perform the regression and calculate these
     statistics.  The returned 'gsl_multifit_robust_stats' structure
     contains the following fields.
          double 'sigma_ols' This contains the standard deviation of the
          residuals as computed from ordinary least squares (OLS).

          double 'sigma_mad' This contains an estimate of the standard
          deviation of the final residuals using the
          Median-Absolute-Deviation statistic

          double 'sigma_rob' This contains an estimate of the standard
          deviation of the final residuals from the theory of robust
          regression (see Street et al, 1988).

          double 'sigma' This contains an estimate of the standard
          deviation of the final residuals by attemping to reconcile
          'sigma_rob' and 'sigma_ols' in a reasonable way.

          double 'Rsq' This contains the R^2 coefficient of
          determination statistic using the estimate 'sigma'.

          double 'adj_Rsq' This contains the adjusted R^2 coefficient of
          determination statistic using the estimate 'sigma'.

          double 'rmse' This contains the root mean squared error of the
          final residuals

          double 'sse' This contains the residual sum of squares taking
          into account the robust covariance matrix.

          size_t 'dof' This contains the number of degrees of freedom n
          - p

          size_t 'numit' Upon successful convergence, this contains the
          number of iterations performed

          gsl_vector * 'weights' This contains the final weight vector
          of length N

          gsl_vector * 'r' This contains the final residual vector of
          length N, r = y - X c


File: gsl-ref.info,  Node: Large Dense Linear Systems,  Next: Troubleshooting,  Prev: Robust linear regression,  Up: Least-Squares Fitting

38.6 Large dense linear systems
===============================

This module is concerned with solving large dense least squares systems
X c = y where the n-by-p matrix X has n >> p (ie: many more rows than
columns).  This type of matrix is called a "tall skinny" matrix, and for
some applications, it may not be possible to fit the entire matrix in
memory at once to use the standard SVD approach.  Therefore, the
algorithms in this module are designed to allow the user to construct
smaller blocks of the matrix X and accumulate those blocks into the
larger system one at a time.  The algorithms in this module never need
to store the entire matrix X in memory.  The large linear least squares
routines support data weights and Tikhonov regularization, and are
designed to minimize the residual
     \chi^2 = || y - Xc ||_W^2 + \lambda^2 || L c ||^2
   where y is the n-by-1 observation vector, X is the n-by-p design
matrix, c is the p-by-1 solution vector, W = diag(w_1,...,w_n) is the
data weighting matrix, L is an m-by-p regularization matrix, \lambda is
a regularization parameter, and ||r||_W^2 = r^T W r.  In the discussion
which follows, we will assume that the system has been converted into
Tikhonov standard form,
     \chi^2 = || y~ - X~ c~ ||^2 + \lambda^2 || c~ ||^2
   and we will drop the tilde characters from the various parameters.
For a discussion of the transformation to standard form *note
Regularized regression::.

   The basic idea is to partition the matrix X and observation vector y
as
     [ X_1 ] c = [ y_1 ]
     [ X_2 ]     [ y_2 ]
     [ X_3 ]     [ y_3 ]
     [ ... ]     [ ... ]
     [ X_k ]     [ y_k ]
   into k blocks, where each block (X_i,y_i) may have any number of
rows, but each X_i has p columns.  The sections below describe the
methods available for solving this partitioned system.  The functions
are declared in the header file 'gsl_multilarge.h'.

* Menu:

* Large Dense Linear Systems Normal Equations::
* Large Dense Linear Systems TSQR::
* Large Dense Linear Systems Solution Steps::
* Large Dense Linear Systems Routines::


File: gsl-ref.info,  Node: Large Dense Linear Systems Normal Equations,  Next: Large Dense Linear Systems TSQR,  Up: Large Dense Linear Systems

38.6.1 Normal Equations Approach
--------------------------------

The normal equations approach to the large linear least squares problem
described above is popular due to its speed and simplicity.  Since the
normal equations solution to the problem is given by
     c = ( X^T X + \lambda^2 I )^-1 X^T y
   only the p-by-p matrix X^T X and p-by-1 vector X^T y need to be
stored.  Using the partition scheme described above, these are given by
     X^T X = \sum_i X_i^T X_i
     X^T y = \sum_i X_i^T y_i
   Since the matrix X^T X is symmetric, only half of it needs to be
calculated.  Once all of the blocks (X_i,y_i) have been accumulated into
the final X^T X and X^T y, the system can be solved with a Cholesky
factorization of the X^T X matrix.  If the Cholesky factorization fails
(occasionally due to numerical rounding errors), a QR decomposition is
then used.  In both cases, the X^T X matrix is first transformed via a
diagonal scaling transformation to attempt to reduce its condition
number as much as possible to recover a more accurate solution vector.
The normal equations approach is the fastest method for solving the
large least squares problem, and is accurate for well-conditioned
matrices X.  However, for ill-conditioned matrices, as is often the case
for large systems, this method can suffer from numerical instabilities
(see Trefethen and Bau, 1997).  The number of operations for this method
is O(np^2 + {1 \over 3}p^3).


File: gsl-ref.info,  Node: Large Dense Linear Systems TSQR,  Next: Large Dense Linear Systems Solution Steps,  Prev: Large Dense Linear Systems Normal Equations,  Up: Large Dense Linear Systems

38.6.2 Tall Skinny QR (TSQR) Approach
-------------------------------------

An algorithm which has better numerical stability for ill-conditioned
problems is known as the Tall Skinny QR (TSQR) method.  This method is
based on computing the thin QR decomposition of the least squares matrix
X = Q R, where Q is an n-by-p matrix with orthogonal columns, and R is a
p-by-p upper triangular matrix.  Once these factors are calculated, the
residual becomes
     \chi^2 = || Q^T y - R c ||^2 + \lambda^2 || c ||^2
   which can be written as the matrix equation
     [ R ; \lambda I ] c = [ Q^T b ; 0 ]
   The matrix on the left hand side is now a much smaller 2p-by-p matrix
which can be solved with a standard SVD approach.  The Q matrix is just
as large as the original matrix X, however it does not need to be
explicitly constructed.  The TSQR algorithm computes only the p-by-p
matrix R and the p-by-1 vector Q^T y, and updates these quantities as
new blocks are added to the system.  Each time a new block of rows
(X_i,y_i) is added, the algorithm performs a QR decomposition of the
matrix
     [ R_(i-1) ; X_i ]
   where R_{i-1} is the upper triangular R factor for the matrix
     [ X_1 ; ... ; X_(i-1) ]
   This QR decomposition is done efficiently taking into account the
sparse structure of R_{i-1}.  See Demmel et al, 2008 for more details on
how this is accomplished.  The number of operations for this method is
O(2np^2 - {2 \over 3}p^3).


File: gsl-ref.info,  Node: Large Dense Linear Systems Solution Steps,  Next: Large Dense Linear Systems Routines,  Prev: Large Dense Linear Systems TSQR,  Up: Large Dense Linear Systems

38.6.3 Large Dense Linear Systems Solution Steps
------------------------------------------------

The typical steps required to solve large regularized linear least
squares problems are as follows:

  1. Choose the regularization matrix L.

  2. Construct a block of rows of the least squares matrix, right hand
     side vector, and weight vector (X_i, y_i, w_i).

  3. Transform the block to standard form (\tilde{X_i},\tilde{y_i}).
     This step can be skipped if L = I and W = I.

  4. Accumulate the standard form block (\tilde{X_i},\tilde{y_i}) into
     the system.

  5. Repeat steps 2-4 until the entire matrix and right hand side vector
     have been accumulated.

  6. Determine an appropriate regularization parameter \lambda (using
     for example L-curve analysis).

  7. Solve the standard form system using the chosen \lambda.

  8. Backtransform the standard form solution \tilde{c} to recover the
     original solution vector c.


File: gsl-ref.info,  Node: Large Dense Linear Systems Routines,  Prev: Large Dense Linear Systems Solution Steps,  Up: Large Dense Linear Systems

38.6.4 Large Dense Linear Least Squares Routines
------------------------------------------------

 -- Function: gsl_multilarge_linear_workspace *
          gsl_multilarge_linear_alloc (const gsl_multilarge_linear_type
          * T, const size_t P)
     This function allocates a workspace for solving large linear least
     squares systems.  The least squares matrix X has P columns, but may
     have any number of rows.  The parameter T specifies the method to
     be used for solving the large least squares system and may be
     selected from the following choices

      -- Multilarge type: gsl_multilarge_linear_normal
          This specifies the normal equations approach for solving the
          least squares system.  This method is suitable in cases where
          performance is critical and it is known that the least squares
          matrix X is well conditioned.  The size of this workspace is
          O(p^2).

      -- Multilarge type: gsl_multilarge_linear_tsqr
          This specifies the sequential Tall Skinny QR (TSQR) approach
          for solving the least squares system.  This method is a good
          general purpose choice for large systems, but requires about
          twice as many operations as the normal equations method for n
          >> p.  The size of this workspace is O(p^2).

 -- Function: void gsl_multilarge_linear_free
          (gsl_multilarge_linear_workspace * W)
     This function frees the memory associated with the workspace W.

 -- Function: const char * gsl_multilarge_linear_name
          (gsl_multilarge_linear_workspace * W)
     This function returns a string pointer to the name of the
     multilarge solver.

 -- Function: int gsl_multilarge_linear_reset
          (gsl_multilarge_linear_workspace * W)
     This function resets the workspace W so it can begin to accumulate
     a new least squares system.

 -- Function: int gsl_multilarge_linear_stdform1 (const gsl_vector * L,
          const gsl_matrix * X, const gsl_vector * Y, gsl_matrix * XS,
          gsl_vector * YS, gsl_multilarge_linear_workspace * WORK)
 -- Function: int gsl_multilarge_linear_wstdform1 (const gsl_vector * L,
          const gsl_matrix * X, const gsl_vector * W, const gsl_vector *
          Y, gsl_matrix * XS, gsl_vector * YS,
          gsl_multilarge_linear_workspace * WORK)
     These functions define a regularization matrix L =
     diag(l_0,l_1,...,l_{p-1}).  The diagonal matrix element l_i is
     provided by the ith element of the input vector L.  The block (X,Y)
     is converted to standard form and the parameters
     (\tilde{X},\tilde{y}) are stored in XS and YS on output.  XS and YS
     have the same dimensions as X and Y.  Optional data weights may be
     supplied in the vector W.  In order to apply this transformation,
     L^{-1} must exist and so none of the l_i may be zero.  After the
     standard form system has been solved, use
     'gsl_multilarge_linear_genform1' to recover the original solution
     vector.  It is allowed to have X = XS and Y = YS for an in-place
     transform.

 -- Function: int gsl_multilarge_linear_L_decomp (gsl_matrix * L,
          gsl_vector * TAU)
     This function calculates the QR decomposition of the m-by-p
     regularization matrix L.  L must have m \ge p.  On output, the
     Householder scalars are stored in the vector TAU of size p.  These
     outputs will be used by 'gsl_multilarge_linear_wstdform2' to
     complete the transformation to standard form.

 -- Function: int gsl_multilarge_linear_stdform2 (const gsl_matrix *
          LQR, const gsl_vector * LTAU, const gsl_matrix * X, const
          gsl_vector * Y, gsl_matrix * XS, gsl_vector * YS,
          gsl_multilarge_linear_workspace * WORK)
 -- Function: int gsl_multilarge_linear_wstdform2 (const gsl_matrix *
          LQR, const gsl_vector * LTAU, const gsl_matrix * X, const
          gsl_vector * W, const gsl_vector * Y, gsl_matrix * XS,
          gsl_vector * YS, gsl_multilarge_linear_workspace * WORK)
     These functions convert a block of rows (X,Y,W) to standard form
     (\tilde{X},\tilde{y}) which are stored in XS and YS respectively.
     X, Y, and W must all have the same number of rows.  The m-by-p
     regularization matrix L is specified by the inputs LQR and LTAU,
     which are outputs from 'gsl_multilarge_linear_L_decomp'.  XS and YS
     have the same dimensions as X and Y.  After the standard form
     system has been solved, use 'gsl_multilarge_linear_genform2' to
     recover the original solution vector.  Optional data weights may be
     supplied in the vector W, where W = diag(w).

 -- Function: int gsl_multilarge_linear_accumulate (gsl_matrix * X,
          gsl_vector * Y, gsl_multilarge_linear_workspace * W)
     This function accumulates the standard form block (X,y) into the
     current least squares system.  X and Y have the same number of
     rows, which can be arbitrary.  X must have p columns.  For the TSQR
     method, X and Y are destroyed on output.  For the normal equations
     method, they are both unchanged.

 -- Function: int gsl_multilarge_linear_solve (const double LAMBDA,
          gsl_vector * C, double * RNORM, double * SNORM,
          gsl_multilarge_linear_workspace * W)
     After all blocks (X_i,y_i) have been accumulated into the large
     least squares system, this function will compute the solution
     vector which is stored in C on output.  The regularization
     parameter \lambda is provided in LAMBDA.  On output, RNORM contains
     the residual norm ||y - X c||_W and SNORM contains the solution
     norm ||L c||.

 -- Function: int gsl_multilarge_linear_genform1 (const gsl_vector * L,
          const gsl_vector * CS, gsl_vector * C,
          gsl_multilarge_linear_workspace * WORK)
     After a regularized system has been solved with L =
     diag(\l_0,\l_1,...,\l_{p-1}), this function backtransforms the
     standard form solution vector CS to recover the solution vector of
     the original problem C.  The diagonal matrix elements l_i are
     provided in the vector L.  It is allowed to have C = CS for an
     in-place transform.

 -- Function: int gsl_multilarge_linear_genform2 (const gsl_matrix *
          LQR, const gsl_vector * LTAU, const gsl_vector * CS,
          gsl_vector * C, gsl_multilarge_linear_workspace * WORK)
     After a regularized system has been solved with a regularization
     matrix L, specified by (LQR,LTAU), this function backtransforms the
     standard form solution CS to recover the solution vector of the
     original problem, which is stored in C, of length p.

 -- Function: int gsl_multilarge_linear_lcurve (gsl_vector * REG_PARAM,
          gsl_vector * RHO, gsl_vector * ETA,
          gsl_multilarge_linear_workspace * WORK)
     This function computes the L-curve for a large least squares system
     after it has been fully accumulated into the workspace WORK.  The
     output vectors REG_PARAM, RHO, and ETA must all be the same size,
     and will contain the regularization parameters \lambda_i, residual
     norms ||y - X c_i||, and solution norms || L c_i || which compose
     the L-curve, where c_i is the regularized solution vector
     corresponding to \lambda_i.  The user may determine the number of
     points on the L-curve by adjusting the size of these input arrays.
     For the TSQR method, the regularization parameters \lambda_i are
     estimated from the singular values of the triangular R factor.  For
     the normal equations method, they are estimated from the
     eigenvalues of the X^T X matrix.

 -- Function: int gsl_multilarge_linear_rcond (double * RCOND,
          gsl_multilarge_linear_workspace * WORK)
     This function computes the reciprocal condition number, stored in
     RCOND, of the least squares matrix after it has been accumulated
     into the workspace WORK.  For the TSQR algorithm, this is
     accomplished by calculating the SVD of the R factor, which has the
     same singular values as the matrix X.  For the normal equations
     method, this is done by computing the eigenvalues of X^T X, which
     could be inaccurate for ill-conditioned matrices X.


File: gsl-ref.info,  Node: Troubleshooting,  Next: Fitting Examples,  Prev: Large Dense Linear Systems,  Up: Least-Squares Fitting

38.7 Troubleshooting
====================

When using models based on polynomials, care should be taken when
constructing the design matrix X.  If the x values are large, then the
matrix X could be ill-conditioned since its columns are powers of x,
leading to unstable least-squares solutions.  In this case it can often
help to center and scale the x values using the mean and standard
deviation:

     x' = (x - mu)/sigma

and then construct the X matrix using the transformed values x'.


File: gsl-ref.info,  Node: Fitting Examples,  Next: Fitting References and Further Reading,  Prev: Troubleshooting,  Up: Least-Squares Fitting

38.8 Examples
=============

The example programs in this section demonstrate the various linear
regression methods.

* Menu:

* Fitting linear regression example::
* Fitting multi-parameter linear regression example::
* Fitting regularized linear regression example 1::
* Fitting regularized linear regression example 2::
* Fitting robust linear regression example::
* Fitting large linear systems example::


File: gsl-ref.info,  Node: Fitting linear regression example,  Next: Fitting multi-parameter linear regression example,  Up: Fitting Examples

38.8.1 Simple Linear Regression Example
---------------------------------------

The following program computes a least squares straight-line fit to a
simple dataset, and outputs the best-fit line and its associated one
standard-deviation error bars.

     #include <stdio.h>
     #include <gsl/gsl_fit.h>

     int
     main (void)
     {
       int i, n = 4;
       double x[4] = { 1970, 1980, 1990, 2000 };
       double y[4] = {   12,   11,   14,   13 };
       double w[4] = {  0.1,  0.2,  0.3,  0.4 };

       double c0, c1, cov00, cov01, cov11, chisq;

       gsl_fit_wlinear (x, 1, w, 1, y, 1, n,
                        &c0, &c1, &cov00, &cov01, &cov11,
                        &chisq);

       printf ("# best fit: Y = %g + %g X\n", c0, c1);
       printf ("# covariance matrix:\n");
       printf ("# [ %g, %g\n#   %g, %g]\n",
               cov00, cov01, cov01, cov11);
       printf ("# chisq = %g\n", chisq);

       for (i = 0; i < n; i++)
         printf ("data: %g %g %g\n",
                        x[i], y[i], 1/sqrt(w[i]));

       printf ("\n");

       for (i = -30; i < 130; i++)
         {
           double xf = x[0] + (i/100.0) * (x[n-1] - x[0]);
           double yf, yf_err;

           gsl_fit_linear_est (xf,
                               c0, c1,
                               cov00, cov01, cov11,
                               &yf, &yf_err);

           printf ("fit: %g %g\n", xf, yf);
           printf ("hi : %g %g\n", xf, yf + yf_err);
           printf ("lo : %g %g\n", xf, yf - yf_err);
         }
       return 0;
     }

The following commands extract the data from the output of the program
and display it using the GNU plotutils 'graph' utility,

     $ ./demo > tmp
     $ more tmp
     # best fit: Y = -106.6 + 0.06 X
     # covariance matrix:
     # [ 39602, -19.9
     #   -19.9, 0.01]
     # chisq = 0.8

     $ for n in data fit hi lo ;
        do
          grep "^$n" tmp | cut -d: -f2 > $n ;
        done
     $ graph -T X -X x -Y y -y 0 20 -m 0 -S 2 -Ie data
          -S 0 -I a -m 1 fit -m 2 hi -m 2 lo


File: gsl-ref.info,  Node: Fitting multi-parameter linear regression example,  Next: Fitting regularized linear regression example 1,  Prev: Fitting linear regression example,  Up: Fitting Examples

38.8.2 Multi-parameter Linear Regression Example
------------------------------------------------

The following program performs a quadratic fit y = c_0 + c_1 x + c_2 x^2
to a weighted dataset using the generalised linear fitting function
'gsl_multifit_wlinear'.  The model matrix X for a quadratic fit is given
by,

     X = [ 1   , x_0  , x_0^2 ;
           1   , x_1  , x_1^2 ;
           1   , x_2  , x_2^2 ;
           ... , ...  , ...   ]

where the column of ones corresponds to the constant term c_0.  The two
remaining columns corresponds to the terms c_1 x and c_2 x^2.

   The program reads N lines of data in the format (X, Y, ERR) where ERR
is the error (standard deviation) in the value Y.

     #include <stdio.h>
     #include <gsl/gsl_multifit.h>

     int
     main (int argc, char **argv)
     {
       int i, n;
       double xi, yi, ei, chisq;
       gsl_matrix *X, *cov;
       gsl_vector *y, *w, *c;

       if (argc != 2)
         {
           fprintf (stderr,"usage: fit n < data\n");
           exit (-1);
         }

       n = atoi (argv[1]);

       X = gsl_matrix_alloc (n, 3);
       y = gsl_vector_alloc (n);
       w = gsl_vector_alloc (n);

       c = gsl_vector_alloc (3);
       cov = gsl_matrix_alloc (3, 3);

       for (i = 0; i < n; i++)
         {
           int count = fscanf (stdin, "%lg %lg %lg",
                               &xi, &yi, &ei);

           if (count != 3)
             {
               fprintf (stderr, "error reading file\n");
               exit (-1);
             }

           printf ("%g %g +/- %g\n", xi, yi, ei);

           gsl_matrix_set (X, i, 0, 1.0);
           gsl_matrix_set (X, i, 1, xi);
           gsl_matrix_set (X, i, 2, xi*xi);

           gsl_vector_set (y, i, yi);
           gsl_vector_set (w, i, 1.0/(ei*ei));
         }

       {
         gsl_multifit_linear_workspace * work
           = gsl_multifit_linear_alloc (n, 3);
         gsl_multifit_wlinear (X, w, y, c, cov,
                               &chisq, work);
         gsl_multifit_linear_free (work);
       }

     #define C(i) (gsl_vector_get(c,(i)))
     #define COV(i,j) (gsl_matrix_get(cov,(i),(j)))

       {
         printf ("# best fit: Y = %g + %g X + %g X^2\n",
                 C(0), C(1), C(2));

         printf ("# covariance matrix:\n");
         printf ("[ %+.5e, %+.5e, %+.5e  \n",
                    COV(0,0), COV(0,1), COV(0,2));
         printf ("  %+.5e, %+.5e, %+.5e  \n",
                    COV(1,0), COV(1,1), COV(1,2));
         printf ("  %+.5e, %+.5e, %+.5e ]\n",
                    COV(2,0), COV(2,1), COV(2,2));
         printf ("# chisq = %g\n", chisq);
       }

       gsl_matrix_free (X);
       gsl_vector_free (y);
       gsl_vector_free (w);
       gsl_vector_free (c);
       gsl_matrix_free (cov);

       return 0;
     }

A suitable set of data for fitting can be generated using the following
program.  It outputs a set of points with gaussian errors from the curve
y = e^x in the region 0 < x < 2.

     #include <stdio.h>
     #include <math.h>
     #include <gsl/gsl_randist.h>

     int
     main (void)
     {
       double x;
       const gsl_rng_type * T;
       gsl_rng * r;

       gsl_rng_env_setup ();

       T = gsl_rng_default;
       r = gsl_rng_alloc (T);

       for (x = 0.1; x < 2; x+= 0.1)
         {
           double y0 = exp (x);
           double sigma = 0.1 * y0;
           double dy = gsl_ran_gaussian (r, sigma);

           printf ("%g %g %g\n", x, y0 + dy, sigma);
         }

       gsl_rng_free(r);

       return 0;
     }

The data can be prepared by running the resulting executable program,

     $ GSL_RNG_TYPE=mt19937_1999 ./generate > exp.dat
     $ more exp.dat
     0.1 0.97935 0.110517
     0.2 1.3359 0.12214
     0.3 1.52573 0.134986
     0.4 1.60318 0.149182
     0.5 1.81731 0.164872
     0.6 1.92475 0.182212
     ....

To fit the data use the previous program, with the number of data points
given as the first argument.  In this case there are 19 data points.

     $ ./fit 19 < exp.dat
     0.1 0.97935 +/- 0.110517
     0.2 1.3359 +/- 0.12214
     ...
     # best fit: Y = 1.02318 + 0.956201 X + 0.876796 X^2
     # covariance matrix:
     [ +1.25612e-02, -3.64387e-02, +1.94389e-02
       -3.64387e-02, +1.42339e-01, -8.48761e-02
       +1.94389e-02, -8.48761e-02, +5.60243e-02 ]
     # chisq = 23.0987

The parameters of the quadratic fit match the coefficients of the
expansion of e^x, taking into account the errors on the parameters and
the O(x^3) difference between the exponential and quadratic functions
for the larger values of x.  The errors on the parameters are given by
the square-root of the corresponding diagonal elements of the covariance
matrix.  The chi-squared per degree of freedom is 1.4, indicating a
reasonable fit to the data.


File: gsl-ref.info,  Node: Fitting regularized linear regression example 1,  Next: Fitting regularized linear regression example 2,  Prev: Fitting multi-parameter linear regression example,  Up: Fitting Examples

38.8.3 Regularized Linear Regression Example 1
----------------------------------------------

The next program demonstrates the difference between ordinary and
regularized least squares when the design matrix is near-singular.  In
this program, we generate two random normally distributed variables u
and v, with v = u + noise so that u and v are nearly colinear.  We then
set a third dependent variable y = u + v + noise and solve for the
coefficients c_1,c_2 of the model Y(c_1,c_2) = c_1 u + c_2 v.  Since u
\approx v, the design matrix X is nearly singular, leading to unstable
ordinary least squares solutions.

Here is the program output:
     matrix condition number = 1.025113e+04
     === Unregularized fit ===
     best fit: y = -43.6588 u + 45.6636 v
     residual norm = 31.6248
     solution norm = 63.1764
     chisq/dof = 1.00213
     === Regularized fit (L-curve) ===
     optimal lambda: 4.51103
     best fit: y = 1.00113 u + 1.0032 v
     residual norm = 31.6547
     solution norm = 1.41728
     chisq/dof = 1.04499
     === Regularized fit (GCV) ===
     optimal lambda: 0.0232029
     best fit: y = -19.8367 u + 21.8417 v
     residual norm = 31.6332
     solution norm = 29.5051
     chisq/dof = 1.00314

We see that the ordinary least squares solution is completely wrong,
while the L-curve regularized method with the optimal \lambda = 4.51103
finds the correct solution c_1 \approx c_2 \approx 1.  The GCV
regularized method finds a regularization parameter \lambda = 0.0232029
which is too small to give an accurate solution, although it performs
better than OLS. The L-curve and its computed corner, as well as the GCV
curve and its minimum are plotted below.

The program is given below.
     #include <gsl/gsl_math.h>
     #include <gsl/gsl_vector.h>
     #include <gsl/gsl_matrix.h>
     #include <gsl/gsl_rng.h>
     #include <gsl/gsl_randist.h>
     #include <gsl/gsl_multifit.h>

     int
     main()
     {
       const size_t n = 1000; /* number of observations */
       const size_t p = 2;    /* number of model parameters */
       size_t i;
       gsl_rng *r = gsl_rng_alloc(gsl_rng_default);
       gsl_matrix *X = gsl_matrix_alloc(n, p);
       gsl_vector *y = gsl_vector_alloc(n);

       for (i = 0; i < n; ++i)
         {
           /* generate first random variable u */
           double ui = 5.0 * gsl_ran_gaussian(r, 1.0);

           /* set v = u + noise */
           double vi = ui + gsl_ran_gaussian(r, 0.001);

           /* set y = u + v + noise */
           double yi = ui + vi + gsl_ran_gaussian(r, 1.0);

           /* since u =~ v, the matrix X is ill-conditioned */
           gsl_matrix_set(X, i, 0, ui);
           gsl_matrix_set(X, i, 1, vi);

           /* rhs vector */
           gsl_vector_set(y, i, yi);
         }

       {
         const size_t npoints = 200;                   /* number of points on L-curve and GCV curve */
         gsl_multifit_linear_workspace *w =
           gsl_multifit_linear_alloc(n, p);
         gsl_vector *c = gsl_vector_alloc(p);          /* OLS solution */
         gsl_vector *c_lcurve = gsl_vector_alloc(p);   /* regularized solution (L-curve) */
         gsl_vector *c_gcv = gsl_vector_alloc(p);      /* regularized solution (GCV) */
         gsl_vector *reg_param = gsl_vector_alloc(npoints);
         gsl_vector *rho = gsl_vector_alloc(npoints);  /* residual norms */
         gsl_vector *eta = gsl_vector_alloc(npoints);  /* solution norms */
         gsl_vector *G = gsl_vector_alloc(npoints);    /* GCV function values */
         double lambda_l;                              /* optimal regularization parameter (L-curve) */
         double lambda_gcv;                            /* optimal regularization parameter (GCV) */
         double G_gcv;                                 /* G(lambda_gcv) */
         size_t reg_idx;                               /* index of optimal lambda */
         double rcond;                                 /* reciprocal condition number of X */
         double chisq, rnorm, snorm;

         /* compute SVD of X */
         gsl_multifit_linear_svd(X, w);

         rcond = gsl_multifit_linear_rcond(w);
         fprintf(stderr, "matrix condition number = %e\n", 1.0 / rcond);

         /* unregularized (standard) least squares fit, lambda = 0 */
         gsl_multifit_linear_solve(0.0, X, y, c, &rnorm, &snorm, w);
         chisq = pow(rnorm, 2.0);

         fprintf(stderr, "=== Unregularized fit ===\n");
         fprintf(stderr, "best fit: y = %g u + %g v\n",
           gsl_vector_get(c, 0), gsl_vector_get(c, 1));
         fprintf(stderr, "residual norm = %g\n", rnorm);
         fprintf(stderr, "solution norm = %g\n", snorm);
         fprintf(stderr, "chisq/dof = %g\n", chisq / (n - p));

         /* calculate L-curve and find its corner */
         gsl_multifit_linear_lcurve(y, reg_param, rho, eta, w);
         gsl_multifit_linear_lcorner(rho, eta, &reg_idx);

         /* store optimal regularization parameter */
         lambda_l = gsl_vector_get(reg_param, reg_idx);

         /* regularize with lambda_l */
         gsl_multifit_linear_solve(lambda_l, X, y, c_lcurve, &rnorm, &snorm, w);
         chisq = pow(rnorm, 2.0) + pow(lambda_l * snorm, 2.0);

         fprintf(stderr, "=== Regularized fit (L-curve) ===\n");
         fprintf(stderr, "optimal lambda: %g\n", lambda_l);
         fprintf(stderr, "best fit: y = %g u + %g v\n",
                 gsl_vector_get(c_lcurve, 0), gsl_vector_get(c_lcurve, 1));
         fprintf(stderr, "residual norm = %g\n", rnorm);
         fprintf(stderr, "solution norm = %g\n", snorm);
         fprintf(stderr, "chisq/dof = %g\n", chisq / (n - p));

         /* calculate GCV curve and find its minimum */
         gsl_multifit_linear_gcv(y, reg_param, G, &lambda_gcv, &G_gcv, w);

         /* regularize with lambda_gcv */
         gsl_multifit_linear_solve(lambda_gcv, X, y, c_gcv, &rnorm, &snorm, w);
         chisq = pow(rnorm, 2.0) + pow(lambda_gcv * snorm, 2.0);

         fprintf(stderr, "=== Regularized fit (GCV) ===\n");
         fprintf(stderr, "optimal lambda: %g\n", lambda_gcv);
         fprintf(stderr, "best fit: y = %g u + %g v\n",
                 gsl_vector_get(c_gcv, 0), gsl_vector_get(c_gcv, 1));
         fprintf(stderr, "residual norm = %g\n", rnorm);
         fprintf(stderr, "solution norm = %g\n", snorm);
         fprintf(stderr, "chisq/dof = %g\n", chisq / (n - p));

         /* output L-curve and GCV curve */
         for (i = 0; i < npoints; ++i)
           {
             printf("%e %e %e %e\n",
                    gsl_vector_get(reg_param, i),
                    gsl_vector_get(rho, i),
                    gsl_vector_get(eta, i),
                    gsl_vector_get(G, i));
           }

         /* output L-curve corner point */
         printf("\n\n%f %f\n",
                gsl_vector_get(rho, reg_idx),
                gsl_vector_get(eta, reg_idx));

         /* output GCV curve corner minimum */
         printf("\n\n%e %e\n",
                lambda_gcv,
                G_gcv);

         gsl_multifit_linear_free(w);
         gsl_vector_free(c);
         gsl_vector_free(c_lcurve);
         gsl_vector_free(reg_param);
         gsl_vector_free(rho);
         gsl_vector_free(eta);
         gsl_vector_free(G);
       }

       gsl_rng_free(r);
       gsl_matrix_free(X);
       gsl_vector_free(y);

       return 0;
     }


File: gsl-ref.info,  Node: Fitting regularized linear regression example 2,  Next: Fitting robust linear regression example,  Prev: Fitting regularized linear regression example 1,  Up: Fitting Examples

38.8.4 Regularized Linear Regression Example 2
----------------------------------------------

The following example program minimizes the cost function

     ||y - X c||^2 + \lambda^2 ||x||^2

where X is the 10-by-8 Hilbert matrix whose entries are given by

     X_{ij} = 1 / (i + j - 1)

and the right hand side vector is given by y =
[1,-1,1,-1,1,-1,1,-1,1,-1]^T.  Solutions are computed for \lambda = 0
(unregularized) as well as for optimal parameters \lambda chosen by
analyzing the L-curve and GCV curve.

Here is the program output:
     matrix condition number = 3.565872e+09
     === Unregularized fit ===
     residual norm = 2.15376
     solution norm = 2.92217e+09
     chisq/dof = 2.31934
     === Regularized fit (L-curve) ===
     optimal lambda: 7.11407e-07
     residual norm = 2.60386
     solution norm = 424507
     chisq/dof = 3.43565
     === Regularized fit (GCV) ===
     optimal lambda: 1.72278
     residual norm = 3.1375
     solution norm = 0.139357
     chisq/dof = 4.95076

Here we see the unregularized solution results in a large solution norm
due to the ill-conditioned matrix.  The L-curve solution finds a small
value of \lambda = 7.11e-7 which still results in a badly conditioned
system and a large solution norm.  The GCV method finds a parameter
\lambda = 1.72 which results in a well-conditioned system and small
solution norm.

The L-curve and its computed corner, as well as the GCV curve and its
minimum are plotted below.

The program is given below.
     #include <gsl/gsl_math.h>
     #include <gsl/gsl_vector.h>
     #include <gsl/gsl_matrix.h>
     #include <gsl/gsl_multifit.h>
     #include <gsl/gsl_blas.h>

     static int
     hilbert_matrix(gsl_matrix * m)
     {
       const size_t N = m->size1;
       const size_t M = m->size2;
       size_t i, j;

       for (i = 0; i < N; i++)
         {
           for (j = 0; j < M; j++)
             {
               gsl_matrix_set(m, i, j, 1.0/(i+j+1.0));
             }
         }

       return GSL_SUCCESS;
     }

     int
     main()
     {
       const size_t n = 10; /* number of observations */
       const size_t p = 8;  /* number of model parameters */
       size_t i;
       gsl_matrix *X = gsl_matrix_alloc(n, p);
       gsl_vector *y = gsl_vector_alloc(n);

       /* construct Hilbert matrix and rhs vector */
       hilbert_matrix(X);

       {
         double val = 1.0;
         for (i = 0; i < n; ++i)
           {
             gsl_vector_set(y, i, val);
             val *= -1.0;
           }
       }

       {
         const size_t npoints = 200;                   /* number of points on L-curve and GCV curve */
         gsl_multifit_linear_workspace *w =
           gsl_multifit_linear_alloc(n, p);
         gsl_vector *c = gsl_vector_alloc(p);          /* OLS solution */
         gsl_vector *c_lcurve = gsl_vector_alloc(p);   /* regularized solution (L-curve) */
         gsl_vector *c_gcv = gsl_vector_alloc(p);      /* regularized solution (GCV) */
         gsl_vector *reg_param = gsl_vector_alloc(npoints);
         gsl_vector *rho = gsl_vector_alloc(npoints);  /* residual norms */
         gsl_vector *eta = gsl_vector_alloc(npoints);  /* solution norms */
         gsl_vector *G = gsl_vector_alloc(npoints);    /* GCV function values */
         double lambda_l;                              /* optimal regularization parameter (L-curve) */
         double lambda_gcv;                            /* optimal regularization parameter (GCV) */
         double G_gcv;                                 /* G(lambda_gcv) */
         size_t reg_idx;                               /* index of optimal lambda */
         double rcond;                                 /* reciprocal condition number of X */
         double chisq, rnorm, snorm;

         /* compute SVD of X */
         gsl_multifit_linear_svd(X, w);

         rcond = gsl_multifit_linear_rcond(w);
         fprintf(stderr, "matrix condition number = %e\n", 1.0 / rcond);

         /* unregularized (standard) least squares fit, lambda = 0 */
         gsl_multifit_linear_solve(0.0, X, y, c, &rnorm, &snorm, w);
         chisq = pow(rnorm, 2.0);

         fprintf(stderr, "=== Unregularized fit ===\n");
         fprintf(stderr, "residual norm = %g\n", rnorm);
         fprintf(stderr, "solution norm = %g\n", snorm);
         fprintf(stderr, "chisq/dof = %g\n", chisq / (n - p));

         /* calculate L-curve and find its corner */
         gsl_multifit_linear_lcurve(y, reg_param, rho, eta, w);
         gsl_multifit_linear_lcorner(rho, eta, &reg_idx);

         /* store optimal regularization parameter */
         lambda_l = gsl_vector_get(reg_param, reg_idx);

         /* regularize with lambda_l */
         gsl_multifit_linear_solve(lambda_l, X, y, c_lcurve, &rnorm, &snorm, w);
         chisq = pow(rnorm, 2.0) + pow(lambda_l * snorm, 2.0);

         fprintf(stderr, "=== Regularized fit (L-curve) ===\n");
         fprintf(stderr, "optimal lambda: %g\n", lambda_l);
         fprintf(stderr, "residual norm = %g\n", rnorm);
         fprintf(stderr, "solution norm = %g\n", snorm);
         fprintf(stderr, "chisq/dof = %g\n", chisq / (n - p));

         /* calculate GCV curve and find its minimum */
         gsl_multifit_linear_gcv(y, reg_param, G, &lambda_gcv, &G_gcv, w);

         /* regularize with lambda_gcv */
         gsl_multifit_linear_solve(lambda_gcv, X, y, c_gcv, &rnorm, &snorm, w);
         chisq = pow(rnorm, 2.0) + pow(lambda_gcv * snorm, 2.0);

         fprintf(stderr, "=== Regularized fit (GCV) ===\n");
         fprintf(stderr, "optimal lambda: %g\n", lambda_gcv);
         fprintf(stderr, "residual norm = %g\n", rnorm);
         fprintf(stderr, "solution norm = %g\n", snorm);
         fprintf(stderr, "chisq/dof = %g\n", chisq / (n - p));

         /* output L-curve and GCV curve */
         for (i = 0; i < npoints; ++i)
           {
             printf("%e %e %e %e\n",
                    gsl_vector_get(reg_param, i),
                    gsl_vector_get(rho, i),
                    gsl_vector_get(eta, i),
                    gsl_vector_get(G, i));
           }

         /* output L-curve corner point */
         printf("\n\n%f %f\n",
                gsl_vector_get(rho, reg_idx),
                gsl_vector_get(eta, reg_idx));

         /* output GCV curve corner minimum */
         printf("\n\n%e %e\n",
                lambda_gcv,
                G_gcv);

         gsl_multifit_linear_free(w);
         gsl_vector_free(c);
         gsl_vector_free(c_lcurve);
         gsl_vector_free(reg_param);
         gsl_vector_free(rho);
         gsl_vector_free(eta);
         gsl_vector_free(G);
       }

       gsl_matrix_free(X);
       gsl_vector_free(y);

       return 0;
     }


File: gsl-ref.info,  Node: Fitting robust linear regression example,  Next: Fitting large linear systems example,  Prev: Fitting regularized linear regression example 2,  Up: Fitting Examples

38.8.5 Robust Linear Regression Example
---------------------------------------

The next program demonstrates the advantage of robust least squares on a
dataset with outliers.  The program generates linear (x,y) data pairs on
the line y = 1.45 x + 3.88, adds some random noise, and inserts 3
outliers into the dataset.  Both the robust and ordinary least squares
(OLS) coefficients are computed for comparison.

     #include <stdio.h>
     #include <gsl/gsl_multifit.h>
     #include <gsl/gsl_randist.h>

     int
     dofit(const gsl_multifit_robust_type *T,
           const gsl_matrix *X, const gsl_vector *y,
           gsl_vector *c, gsl_matrix *cov)
     {
       int s;
       gsl_multifit_robust_workspace * work
         = gsl_multifit_robust_alloc (T, X->size1, X->size2);

       s = gsl_multifit_robust (X, y, c, cov, work);
       gsl_multifit_robust_free (work);

       return s;
     }

     int
     main (int argc, char **argv)
     {
       size_t i;
       size_t n;
       const size_t p = 2; /* linear fit */
       gsl_matrix *X, *cov;
       gsl_vector *x, *y, *c, *c_ols;
       const double a = 1.45; /* slope */
       const double b = 3.88; /* intercept */
       gsl_rng *r;

       if (argc != 2)
         {
           fprintf (stderr,"usage: robfit n\n");
           exit (-1);
         }

       n = atoi (argv[1]);

       X = gsl_matrix_alloc (n, p);
       x = gsl_vector_alloc (n);
       y = gsl_vector_alloc (n);

       c = gsl_vector_alloc (p);
       c_ols = gsl_vector_alloc (p);
       cov = gsl_matrix_alloc (p, p);

       r = gsl_rng_alloc(gsl_rng_default);

       /* generate linear dataset */
       for (i = 0; i < n - 3; i++)
         {
           double dx = 10.0 / (n - 1.0);
           double ei = gsl_rng_uniform(r);
           double xi = -5.0 + i * dx;
           double yi = a * xi + b;

           gsl_vector_set (x, i, xi);
           gsl_vector_set (y, i, yi + ei);
         }

       /* add a few outliers */
       gsl_vector_set(x, n - 3, 4.7);
       gsl_vector_set(y, n - 3, -8.3);

       gsl_vector_set(x, n - 2, 3.5);
       gsl_vector_set(y, n - 2, -6.7);

       gsl_vector_set(x, n - 1, 4.1);
       gsl_vector_set(y, n - 1, -6.0);

       /* construct design matrix X for linear fit */
       for (i = 0; i < n; ++i)
         {
           double xi = gsl_vector_get(x, i);

           gsl_matrix_set (X, i, 0, 1.0);
           gsl_matrix_set (X, i, 1, xi);
         }

       /* perform robust and OLS fit */
       dofit(gsl_multifit_robust_ols, X, y, c_ols, cov);
       dofit(gsl_multifit_robust_bisquare, X, y, c, cov);

       /* output data and model */
       for (i = 0; i < n; ++i)
         {
           double xi = gsl_vector_get(x, i);
           double yi = gsl_vector_get(y, i);
           gsl_vector_view v = gsl_matrix_row(X, i);
           double y_ols, y_rob, y_err;

           gsl_multifit_robust_est(&v.vector, c, cov, &y_rob, &y_err);
           gsl_multifit_robust_est(&v.vector, c_ols, cov, &y_ols, &y_err);

           printf("%g %g %g %g\n", xi, yi, y_rob, y_ols);
         }

     #define C(i) (gsl_vector_get(c,(i)))
     #define COV(i,j) (gsl_matrix_get(cov,(i),(j)))

       {
         printf ("# best fit: Y = %g + %g X\n",
                 C(0), C(1));

         printf ("# covariance matrix:\n");
         printf ("# [ %+.5e, %+.5e\n",
                    COV(0,0), COV(0,1));
         printf ("#   %+.5e, %+.5e\n",
                    COV(1,0), COV(1,1));
       }

       gsl_matrix_free (X);
       gsl_vector_free (x);
       gsl_vector_free (y);
       gsl_vector_free (c);
       gsl_vector_free (c_ols);
       gsl_matrix_free (cov);
       gsl_rng_free(r);

       return 0;
     }

   The output from the program is shown in the following plot.


File: gsl-ref.info,  Node: Fitting large linear systems example,  Prev: Fitting robust linear regression example,  Up: Fitting Examples

38.8.6 Large Dense Linear Regression Example
--------------------------------------------

The following program demostrates the large dense linear least squares
solvers.  This example is adapted from Trefethen and Bau, and fits the
function f(t) = \exp{(\sin^3{(10t)}}) on the interval [0,1] with a
degree 15 polynomial.  The program generates n = 50000 equally spaced
points t_i on this interval, calculates the function value and adds
random noise to determine the observation value y_i.  The entries of the
least squares matrix are X_{ij} = t_i^j, representing a polynomial fit.
The matrix is highly ill-conditioned, with a condition number of about
1.4 \cdot 10^{11}.  The program accumulates the matrix into the least
squares system in 5 blocks, each with 10000 rows.  This way the full
matrix X is never stored in memory.  We solve the system with both the
normal equations and TSQR methods.  The results are shown in the plot
below.  In the top left plot, we see the unregularized normal equations
solution has larger error than TSQR due to the ill-conditioning of the
matrix.  In the bottom left plot, we show the L-curve, which exhibits
multiple corners.  In the top right panel, we plot a regularized
solution using \lambda = 10^{-6}.  The TSQR and normal solutions now
agree, however they are unable to provide a good fit due to the damping.
This indicates that for some ill-conditioned problems, regularizing the
normal equations does not improve the solution.  This is further
illustrated in the bottom right panel, where we plot the L-curve
calculated from the normal equations.  The curve agrees with the TSQR
curve for larger damping parameters, but for small \lambda, the normal
equations approach cannot provide accurate solution vectors leading to
numerical inaccuracies in the left portion of the curve.

     #include <gsl/gsl_math.h>
     #include <gsl/gsl_vector.h>
     #include <gsl/gsl_matrix.h>
     #include <gsl/gsl_rng.h>
     #include <gsl/gsl_randist.h>
     #include <gsl/gsl_multifit.h>
     #include <gsl/gsl_multilarge.h>
     #include <gsl/gsl_blas.h>

     /* function to be fitted */
     double
     func(const double t)
     {
       double x = sin(10.0 * t);
       return exp(x*x*x);
     }

     /* construct a row of the least squares matrix */
     int
     build_row(const double t, gsl_vector *row)
     {
       const size_t p = row->size;
       double Xj = 1.0;
       size_t j;

       for (j = 0; j < p; ++j)
         {
           gsl_vector_set(row, j, Xj);
           Xj *= t;
         }

       return 0;
     }

     int
     solve_system(const int print_data, const gsl_multilarge_linear_type * T,
                  const double lambda, const size_t n, const size_t p,
                  gsl_vector * c)
     {
       const size_t nblock = 5;         /* number of blocks to accumulate */
       const size_t nrows = n / nblock; /* number of rows per block */
       gsl_multilarge_linear_workspace * w =
         gsl_multilarge_linear_alloc(T, p);
       gsl_matrix *X = gsl_matrix_alloc(nrows, p);
       gsl_vector *y = gsl_vector_alloc(nrows);
       gsl_rng *r = gsl_rng_alloc(gsl_rng_default);
       const size_t nlcurve = 200;
       gsl_vector *reg_param = gsl_vector_alloc(nlcurve);
       gsl_vector *rho = gsl_vector_alloc(nlcurve);
       gsl_vector *eta = gsl_vector_alloc(nlcurve);
       size_t rowidx = 0;
       double rnorm, snorm, rcond;
       double t = 0.0;
       double dt = 1.0 / (n - 1.0);

       while (rowidx < n)
         {
           size_t nleft = n - rowidx;         /* number of rows left to accumulate */
           size_t nr = GSL_MIN(nrows, nleft); /* number of rows in this block */
           gsl_matrix_view Xv = gsl_matrix_submatrix(X, 0, 0, nr, p);
           gsl_vector_view yv = gsl_vector_subvector(y, 0, nr);
           size_t i;

           /* build (X,y) block with 'nr' rows */
           for (i = 0; i < nr; ++i)
             {
               gsl_vector_view row = gsl_matrix_row(&Xv.matrix, i);
               double fi = func(t);
               double ei = gsl_ran_gaussian (r, 0.1 * fi); /* noise */
               double yi = fi + ei;

               /* construct this row of LS matrix */
               build_row(t, &row.vector);

               /* set right hand side value with added noise */
               gsl_vector_set(&yv.vector, i, yi);

               if (print_data && (i % 100 == 0))
                 printf("%f %f\n", t, yi);

               t += dt;
             }

           /* accumulate (X,y) block into LS system */
           gsl_multilarge_linear_accumulate(&Xv.matrix, &yv.vector, w);

           rowidx += nr;
         }

       if (print_data)
         printf("\n\n");

       /* compute L-curve */
       gsl_multilarge_linear_lcurve(reg_param, rho, eta, w);

       /* solve large LS system and store solution in c */
       gsl_multilarge_linear_solve(lambda, c, &rnorm, &snorm, w);

       /* compute reciprocal condition number */
       gsl_multilarge_linear_rcond(&rcond, w);

       fprintf(stderr, "=== Method %s ===\n", gsl_multilarge_linear_name(w));
       fprintf(stderr, "condition number = %e\n", 1.0 / rcond);
       fprintf(stderr, "residual norm    = %e\n", rnorm);
       fprintf(stderr, "solution norm    = %e\n", snorm);

       /* output L-curve */
       {
         size_t i;
         for (i = 0; i < nlcurve; ++i)
           {
             printf("%.12e %.12e %.12e\n",
                    gsl_vector_get(reg_param, i),
                    gsl_vector_get(rho, i),
                    gsl_vector_get(eta, i));
           }
         printf("\n\n");
       }

       gsl_matrix_free(X);
       gsl_vector_free(y);
       gsl_multilarge_linear_free(w);
       gsl_rng_free(r);
       gsl_vector_free(reg_param);
       gsl_vector_free(rho);
       gsl_vector_free(eta);

       return 0;
     }

     int
     main(int argc, char *argv[])
     {
       const size_t n = 50000;   /* number of observations */
       const size_t p = 16;      /* polynomial order + 1 */
       double lambda = 0.0;      /* regularization parameter */
       gsl_vector *c_tsqr = gsl_vector_alloc(p);
       gsl_vector *c_normal = gsl_vector_alloc(p);

       if (argc > 1)
         lambda = atof(argv[1]);

       /* solve system with TSQR method */
       solve_system(1, gsl_multilarge_linear_tsqr, lambda, n, p, c_tsqr);

       /* solve system with Normal equations method */
       solve_system(0, gsl_multilarge_linear_normal, lambda, n, p, c_normal);

       /* output solutions */
       {
         gsl_vector *v = gsl_vector_alloc(p);
         double t;

         for (t = 0.0; t <= 1.0; t += 0.01)
           {
             double f_exact = func(t);
             double f_tsqr, f_normal;

             build_row(t, v);
             gsl_blas_ddot(v, c_tsqr, &f_tsqr);
             gsl_blas_ddot(v, c_normal, &f_normal);

             printf("%f %e %e %e\n", t, f_exact, f_tsqr, f_normal);
           }

         gsl_vector_free(v);
       }

       gsl_vector_free(c_tsqr);
       gsl_vector_free(c_normal);

       return 0;
     }


File: gsl-ref.info,  Node: Fitting References and Further Reading,  Prev: Fitting Examples,  Up: Least-Squares Fitting

38.9 References and Further Reading
===================================

A summary of formulas and techniques for least squares fitting can be
found in the "Statistics" chapter of the Annual Review of Particle
Physics prepared by the Particle Data Group,

     'Review of Particle Properties', R.M. Barnett et al., Physical
     Review D54, 1 (1996) <http://pdg.lbl.gov/>

The Review of Particle Physics is available online at the website given
above.

   The tests used to prepare these routines are based on the NIST
Statistical Reference Datasets.  The datasets and their documentation
are available from NIST at the following website,

           <http://www.nist.gov/itl/div898/strd/index.html>.

More information on Tikhonov regularization can be found in

     Hansen, P. C. (1998), Rank-Deficient and Discrete Ill-Posed
     Problems: Numerical Aspects of Linear Inversion.  SIAM Monogr.  on
     Mathematical Modeling and Computation, Society for Industrial and
     Applied Mathematics

     M. Rezghi and S. M. Hosseini (2009), A new variant of L-curve for
     Tikhonov regularization, Journal of Computational and Applied
     Mathematics, Volume 231, Issue 2, pages 914-924.

The GSL implementation of robust linear regression closely follows the
publications

     DuMouchel, W. and F. O'Brien (1989), "Integrating a robust option
     into a multiple regression computing environment," Computer Science
     and Statistics: Proceedings of the 21st Symposium on the Interface,
     American Statistical Association

     Street, J.O., R.J. Carroll, and D. Ruppert (1988), "A note on
     computing robust regression estimates via iteratively reweighted
     least squares," The American Statistician, v.  42, pp.  152-154.

More information about the normal equations and TSQR approach for
solving large linear least squares systems can be found in the
publications

     Trefethen, L. N. and Bau, D. (1997), "Numerical Linear Algebra",
     SIAM.

     Demmel, J., Grigori, L., Hoemmen, M. F., and Langou, J.
     "Communication-optimal parallel and sequential QR and LU
     factorizations", UCB Technical Report No.  UCB/EECS-2008-89, 2008.


File: gsl-ref.info,  Node: Nonlinear Least-Squares Fitting,  Next: Basis Splines,  Prev: Least-Squares Fitting,  Up: Top

39 Nonlinear Least-Squares Fitting
**********************************

This chapter describes functions for multidimensional nonlinear
least-squares fitting.  There are generally two classes of algorithms
for solving nonlinear least squares problems, which fall under line
search methods and trust region methods.  GSL currently implements only
trust region methods and provides the user with full access to
intermediate steps of the iteration.  The user also has the ability to
tune a number of parameters which affect low-level aspects of the
algorithm which can help to accelerate convergence for the specific
problem at hand.  GSL provides two separate interfaces for nonlinear
least squares fitting.  The first is designed for small to moderate
sized problems, and the second is designed for very large problems,
which may or may not have significant sparse structure.

   The header file 'gsl_multifit_nlinear.h' contains prototypes for the
multidimensional nonlinear fitting functions and related declarations
relating to the small to moderate sized systems.

   The header file 'gsl_multilarge_nlinear.h' contains prototypes for
the multidimensional nonlinear fitting functions and related
declarations relating to large systems.

* Menu:

* Nonlinear Least-Squares Overview::
* Nonlinear Least-Squares TRS Overview::
* Nonlinear Least-Squares Weighted Overview::
* Nonlinear Least-Squares Tunable Parameters::
* Nonlinear Least-Squares Initialization::
* Nonlinear Least-Squares Function Definition::
* Nonlinear Least-Squares Iteration::
* Nonlinear Least-Squares Testing for Convergence::
* Nonlinear Least-Squares High Level Driver::
* Nonlinear Least-Squares Covariance Matrix::
* Nonlinear Least-Squares Troubleshooting::
* Nonlinear Least-Squares Examples::
* Nonlinear Least-Squares References and Further Reading::


File: gsl-ref.info,  Node: Nonlinear Least-Squares Overview,  Next: Nonlinear Least-Squares TRS Overview,  Up: Nonlinear Least-Squares Fitting

39.1 Overview
=============

The problem of multidimensional nonlinear least-squares fitting requires
the minimization of the squared residuals of n functions, f_i, in p
parameters, x_i,

     \Phi(x) = (1/2) || f(x) ||^2
             = (1/2) \sum_{i=1}^{n} f_i(x_1, ..., x_p)^2

In trust region methods, the objective (or cost) function \Phi(x) is
approximated by a model function m_k(\delta) in the vicinity of some
point x_k.  The model function is often simply a second order Taylor
series expansion around the point x_k, ie:

     \Phi(x_k + \delta) ~=~ m_k(\delta) = \Phi(x_k) + g_k^T \delta + 1/2 \delta^T B_k \delta

   where g_k = \nabla \Phi(x_k) = J^T f is the gradient vector at the
point x_k, B_k = \nabla^2 \Phi(x_k) is the Hessian matrix at x_k, or
some approximation to it, and J is the n-by-p Jacobian matrix J_{ij} = d
f_i / d x_j.  In order to find the next step \delta, we minimize the
model function m_k(\delta), but search for solutions only within a
region where we trust that m_k(\delta) is a good approximation to the
objective function \Phi(x_k + \delta).  In other words, we seek a
solution of the trust region subproblem (TRS)

     \min_(\delta \in R^p) m_k(\delta), s.t. || D_k \delta || <= \Delta_k

   where \Delta_k > 0 is the trust region radius and D_k is a scaling
matrix.  If D_k = I, then the trust region is a ball of radius \Delta_k
centered at x_k.  In some applications, the parameter vector x may have
widely different scales.  For example, one parameter might be a
temperature on the order of 10^3 K, while another might be a length on
the order of 10^{-6} m.  In such cases, a spherical trust region may not
be the best choice, since if \Phi changes rapidly along directions with
one scale, and more slowly along directions with a different scale, the
model function m_k may be a poor approximation to \Phi along the rapidly
changing directions.  In such problems, it may be best to use an
elliptical trust region, by setting D_k to a diagonal matrix whose
entries are designed so that the scaled step D_k \delta has entries of
approximately the same order of magnitude.

   The trust region subproblem above normally amounts to solving a
linear least squares system (or multiple systems) for the step \delta.
Once \delta is computed, it is checked whether or not it reduces the
objective function \Phi(x).  A useful statistic for this is to look at
the ratio

     \rho_k = ( \Phi(x_k) - \Phi(x_k + \delta_k) / ( m_k(0) - m_k(\delta_k) )

   where the numerator is the actual reduction of the objective function
due to the step \delta_k, and the denominator is the predicted reduction
due to the model m_k.  If \rho_k is negative, it means that the step
\delta_k increased the objective function and so it is rejected.  If
\rho_k is positive, then we have found a step which reduced the
objective function and it is accepted.  Furthermore, if \rho_k is close
to 1, then this indicates that the model function is a good
approximation to the objective function in the trust region, and so on
the next iteration the trust region is enlarged in order to take more
ambitious steps.  When a step is rejected, the trust region is made
smaller and the TRS is solved again.  An outline for the general trust
region method used by GSL can now be given.

Trust Region Algorithm

  1. Initialize: given x_0, construct m_0(\delta), D_0 and \Delta_0 > 0

  2. For k = 0, 1, 2, ...

       a. If converged, then stop

       b. Solve TRS for trial step \delta_k

       c. Evaluate trial step by computing \rho_k

            1. if step is accepted, set x_{k+1} = x_k + \delta_k and
               increase radius, \Delta_{k+1} = \alpha \Delta_k
            2. if step is rejected, set x_{k+1} = x_k and decrease
               radius, \Delta_{k+1} = {\Delta_k \over \beta}; goto 2(b)

       d. Construct m_{k+1}(\delta) and D_{k+1}

GSL offers the user a number of different algorithms for solving the
trust region subproblem in 2(b), as well as different choices of scaling
matrices D_k and different methods of updating the trust region radius
\Delta_k.  Therefore, while reasonable default methods are provided, the
user has a lot of control to fine-tune the various steps of the
algorithm for their specific problem.


File: gsl-ref.info,  Node: Nonlinear Least-Squares TRS Overview,  Next: Nonlinear Least-Squares Weighted Overview,  Prev: Nonlinear Least-Squares Overview,  Up: Nonlinear Least-Squares Fitting

39.2 Solving the Trust Region Subproblem (TRS)
==============================================

* Menu:

* Nonlinear Least-Squares TRS Levenberg-Marquardt::
* Nonlinear Least-Squares TRS Levenberg-Marquardt with Geodesic Acceleration::
* Nonlinear Least-Squares TRS Dogleg::
* Nonlinear Least-Squares TRS Double Dogleg::
* Nonlinear Least-Squares TRS 2D Subspace::
* Nonlinear Least-Squares TRS Steihaug-Toint Conjugate Gradient::

Below we describe the methods available for solving the trust region
subproblem.  The methods available provide either exact or approximate
solutions to the trust region subproblem.  In all algorithms below, the
Hessian matrix B_k is approximated as B_k \approx J_k^T J_k, where J_k =
J(x_k).  In all methods, the solution of the TRS involves solving a
linear least squares system involving the Jacobian matrix.  For small to
moderate sized problems ('gsl_multifit_nlinear' interface), this is
accomplished by factoring the full Jacobian matrix, which is provided by
the user, with the Cholesky, QR, or SVD decompositions.  For large
systems ('gsl_multilarge_nlinear' interface), the user has two choices.
One is to solve the system iteratively, without needing to store the
full Jacobian matrix in memory.  With this method, the user must provide
a routine to calculate the matrix-vector products J u or J^T u for a
given vector u.  This iterative method is particularly useful for
systems where the Jacobian has sparse structure, since forming
matrix-vector products can be done cheaply.  The second option for large
systems involves forming the normal equations matrix J^T J and then
factoring it using a Cholesky decomposition.  The normal equations
matrix is p-by-p, typically much smaller than the full n-by-p Jacobian,
and can usually be stored in memory even if the full Jacobian matrix
cannot.  This option is useful for large, dense systems, or if the
iterative method has difficulty converging.


File: gsl-ref.info,  Node: Nonlinear Least-Squares TRS Levenberg-Marquardt,  Next: Nonlinear Least-Squares TRS Levenberg-Marquardt with Geodesic Acceleration,  Up: Nonlinear Least-Squares TRS Overview

39.2.1 Levenberg-Marquardt
--------------------------

There is a theorem which states that if \delta_k is a solution to the
trust region subproblem given above, then there exists \mu_k \ge 0 such
that

     ( B_k + \mu_k D_k^T D_k ) \delta_k = -g_k

   with \mu_k (\Delta_k - ||D_k \delta_k||) = 0.  This forms the basis
of the Levenberg-Marquardt algorithm, which controls the trust region
size by adjusting the parameter \mu_k rather than the radius \Delta_k
directly.  For each radius \Delta_k, there is a unique parameter \mu_k
which solves the TRS, and they have an inverse relationship, so that
large values of \mu_k correspond to smaller trust regions, while small
values of \mu_k correspond to larger trust regions.

With the approximation B_k \approx J_k^T J_k, on each iteration, in
order to calculate the step \delta_k, the following linear least squares
problem is solved:

     [J_k; sqrt(mu_k) D_k] \delta_k = - [f_k; 0]

If the step \delta_k is accepted, then \mu_k is decreased on the next
iteration in order to take a larger step, otherwise it is increased to
take a smaller step.  The Levenberg-Marquardt algorithm provides an
exact solution of the trust region subproblem, but typically has a
higher computational cost per iteration than the approximate methods
discussed below, since it may need to solve the least squares system
above several times for different values of \mu_k.


File: gsl-ref.info,  Node: Nonlinear Least-Squares TRS Levenberg-Marquardt with Geodesic Acceleration,  Next: Nonlinear Least-Squares TRS Dogleg,  Prev: Nonlinear Least-Squares TRS Levenberg-Marquardt,  Up: Nonlinear Least-Squares TRS Overview

39.2.2 Levenberg-Marquardt with Geodesic Acceleration
-----------------------------------------------------

This method applies a so-called geodesic acceleration correction to the
standard Levenberg-Marquardt step \delta_k (Transtrum et al, 2011).  By
interpreting \delta_k as a first order step along a geodesic in the
model parameter space (ie: a velocity \delta_k = v_k), the geodesic
acceleration a_k is a second order correction along the geodesic which
is determined by solving the linear least squares system

     [J_k; sqrt(mu_k) D_k] a_k = - [f_vv(x_k); 0]

where f_{vv} is the second directional derivative of the residual vector
in the velocity direction v, f_{vv}(x) = D_v^2 f = \sum_{\alpha\beta}
v_{\alpha} v_{\beta} \partial_{\alpha} \partial_{\beta} f(x), where
\alpha and \beta are summed over the p parameters.  The new total step
is then \delta_k' = v_k + {1 \over 2}a_k.  The second order correction
a_k can be calculated with a modest additional cost, and has been shown
to dramatically reduce the number of iterations (and expensive Jacobian
evaluations) required to reach convergence on a variety of different
problems.  In order to utilize the geodesic acceleration, the user must
supply a function which provides the second directional derivative
vector f_{vv}(x), or alternatively the library can use a finite
difference method to estimate this vector with one additional function
evaluation of f(x + h v) where h is a tunable step size (see the 'h_fvv'
parameter description).


File: gsl-ref.info,  Node: Nonlinear Least-Squares TRS Dogleg,  Next: Nonlinear Least-Squares TRS Double Dogleg,  Prev: Nonlinear Least-Squares TRS Levenberg-Marquardt with Geodesic Acceleration,  Up: Nonlinear Least-Squares TRS Overview

39.2.3 Dogleg
-------------

This is Powell's dogleg method, which finds an approximate solution to
the trust region subproblem, by restricting its search to a piecewise
linear "dogleg" path, composed of the origin, the Cauchy point which
represents the model minimizer along the steepest descent direction, and
the Gauss-Newton point, which is the overall minimizer of the
unconstrained model.  The Gauss-Newton step is calculated by solving

     J_k \delta_gn = -f_k

   which is the main computational task for each iteration, but only
needs to be performed once per iteration.  If the Gauss-Newton point is
inside the trust region, it is selected as the step.  If it is outside,
the method then calculates the Cauchy point, which is located along the
gradient direction.  If the Cauchy point is also outside the trust
region, the method assumes that it is still far from the minimum and so
proceeds along the gradient direction, truncating the step at the trust
region boundary.  If the Cauchy point is inside the trust region, with
the Gauss-Newton point outside, the method uses a dogleg step, which is
a linear combination of the gradient direction and the Gauss-Newton
direction, stopping at the trust region boundary.


File: gsl-ref.info,  Node: Nonlinear Least-Squares TRS Double Dogleg,  Next: Nonlinear Least-Squares TRS 2D Subspace,  Prev: Nonlinear Least-Squares TRS Dogleg,  Up: Nonlinear Least-Squares TRS Overview

39.2.4 Double Dogleg
--------------------

This method is an improvement over the classical dogleg algorithm, which
attempts to include information about the Gauss-Newton step while the
iteration is still far from the minimum.  When the Cauchy point is
inside the trust region and the Gauss-Newton point is outside, the
method computes a scaled Gauss-Newton point and then takes a dogleg step
between the Cauchy point and the scaled Gauss-Newton point.  The scaling
is calculated to ensure that the reduction in the model m_k is about the
same as the reduction provided by the Cauchy point.


File: gsl-ref.info,  Node: Nonlinear Least-Squares TRS 2D Subspace,  Next: Nonlinear Least-Squares TRS Steihaug-Toint Conjugate Gradient,  Prev: Nonlinear Least-Squares TRS Double Dogleg,  Up: Nonlinear Least-Squares TRS Overview

39.2.5 Two Dimensional Subspace
-------------------------------

The dogleg methods restrict the search for the TRS solution to a 1D
curve defined by the Cauchy and Gauss-Newton points.  An improvement to
this is to search for a solution using the full two dimensional subspace
spanned by the Cauchy and Gauss-Newton directions.  The dogleg path is
of course inside this subspace, and so this method solves the TRS at
least as accurately as the dogleg methods.  Since this method searches a
larger subspace for a solution, it can converge more quickly than dogleg
on some problems.  Because the subspace is only two dimensional, this
method is very efficient and the main computation per iteration is to
determine the Gauss-Newton point.


File: gsl-ref.info,  Node: Nonlinear Least-Squares TRS Steihaug-Toint Conjugate Gradient,  Prev: Nonlinear Least-Squares TRS 2D Subspace,  Up: Nonlinear Least-Squares TRS Overview

39.2.6 Steihaug-Toint Conjugate Gradient
----------------------------------------

One difficulty of the dogleg methods is calculating the Gauss-Newton
step when the Jacobian matrix is singular.  The Steihaug-Toint method
also computes a generalized dogleg step, but avoids solving for the
Gauss-Newton step directly, instead using an iterative conjugate
gradient algorithm.  This method performs well at points where the
Jacobian is singular, and is also suitable for large-scale problems
where factoring the Jacobian matrix could be prohibitively expensive.


File: gsl-ref.info,  Node: Nonlinear Least-Squares Weighted Overview,  Next: Nonlinear Least-Squares Tunable Parameters,  Prev: Nonlinear Least-Squares TRS Overview,  Up: Nonlinear Least-Squares Fitting

39.3 Weighted Nonlinear Least-Squares
=====================================

Weighted nonlinear least-squares fitting minimizes the function

     \Phi(x) = (1/2) || f(x) ||_W^2
             = (1/2) \sum_{i=1}^{n} f_i(x_1, ..., x_p)^2

   where W = diag(w_1,w_2,...,w_n) is the weighting matrix, and
||f||_W^2 = f^T W f.  The weights w_i are commonly defined as w_i =
1/\sigma_i^2, where \sigma_i is the error in the ith measurement.  A
simple change of variables \tilde{f} = W^{1 \over 2} f yields \Phi(x) =
{1 \over 2} ||\tilde{f}||^2, which is in the same form as the unweighted
case.  The user can either perform this transform directly on their
function residuals and Jacobian, or use the 'gsl_multifit_nlinear_winit'
interface which automatically performs the correct scaling.  To manually
perform this transformation, the residuals and Jacobian should be
modified according to

     f~_i = f_i / \sigma_i
     J~_ij = 1 / \sigma_i df_i/dx_j

For large systems, the user must perform their own weighting.


File: gsl-ref.info,  Node: Nonlinear Least-Squares Tunable Parameters,  Next: Nonlinear Least-Squares Initialization,  Prev: Nonlinear Least-Squares Weighted Overview,  Up: Nonlinear Least-Squares Fitting

39.4 Tunable Parameters
=======================

The user can tune nearly all aspects of the iteration at allocation
time.  For the 'gsl_multifit_nlinear' interface, the user may modify the
'gsl_multifit_nlinear_parameters' structure, which is defined as
follows:

     typedef struct
     {
       const gsl_multifit_nlinear_trs *trs;        /* trust region subproblem method */
       const gsl_multifit_nlinear_scale *scale;    /* scaling method */
       const gsl_multifit_nlinear_solver *solver;  /* solver method */
       gsl_multifit_nlinear_fdtype fdtype;         /* finite difference method */
       double factor_up;                           /* factor for increasing trust radius */
       double factor_down;                         /* factor for decreasing trust radius */
       double avmax;                               /* max allowed |a|/|v| */
       double h_df;                                /* step size for finite difference Jacobian */
       double h_fvv;                               /* step size for finite difference fvv */
     } gsl_multifit_nlinear_parameters;

For the 'gsl_multilarge_nlinear' interface, the user may modify the
'gsl_multilarge_nlinear_parameters' structure, which is defined as
follows:

     typedef struct
     {
       const gsl_multilarge_nlinear_trs *trs;       /* trust region subproblem method */
       const gsl_multilarge_nlinear_scale *scale;   /* scaling method */
       const gsl_multilarge_nlinear_solver *solver; /* solver method */
       gsl_multilarge_nlinear_fdtype fdtype;        /* finite difference method */
       double factor_up;                            /* factor for increasing trust radius */
       double factor_down;                          /* factor for decreasing trust radius */
       double avmax;                                /* max allowed |a|/|v| */
       double h_df;                                 /* step size for finite difference Jacobian */
       double h_fvv;                                /* step size for finite difference fvv */
       size_t max_iter;                             /* maximum iterations for trs method */
       double tol;                                  /* tolerance for solving trs */
     } gsl_multilarge_nlinear_parameters;

Each of these parameters is discussed in further detail below.

 -- Parameter: const gsl_multifit_nlinear_trs * trs
 -- Parameter: const gsl_multilarge_nlinear_trs * trs

     This parameter determines the method used to solve the trust region
     subproblem, and may be selected from the following choices,

      -- Default: gsl_multifit_nlinear_trs_lm
      -- Default: gsl_multilarge_nlinear_trs_lm
          This selects the Levenberg-Marquardt algorithm.

      -- Option: gsl_multifit_nlinear_trs_lmaccel
      -- Option: gsl_multilarge_nlinear_trs_lmaccel
          This selects the Levenberg-Marquardt algorithm with geodesic
          acceleration.

      -- Option: gsl_multifit_nlinear_trs_dogleg
      -- Option: gsl_multilarge_nlinear_trs_dogleg
          This selects the dogleg algorithm.

      -- Option: gsl_multifit_nlinear_trs_ddogleg
      -- Option: gsl_multilarge_nlinear_trs_ddogleg
          This selects the double dogleg algorithm.

      -- Option: gsl_multifit_nlinear_trs_subspace2D
      -- Option: gsl_multilarge_nlinear_trs_subspace2D
          This selects the 2D subspace algorithm.

      -- Option: gsl_multilarge_nlinear_trs_cgst
          This selects the Steihaug-Toint conjugate gradient algorithm.
          This method is available only for large systems.

 -- Parameter: const gsl_multifit_nlinear_scale * scale
 -- Parameter: const gsl_multilarge_nlinear_scale * scale

     This parameter determines the diagonal scaling matrix D and may be
     selected from the following choices,

      -- Default: gsl_multifit_nlinear_scale_more
      -- Default: gsl_multilarge_nlinear_scale_more
          This damping strategy was suggested by More', and corresponds
          to D^T D = max(diag(J^T J)), in other words the maximum
          elements of diag(J^T J) encountered thus far in the iteration.
          This choice of D makes the problem scale-invariant, so that if
          the model parameters x_i are each scaled by an arbitrary
          constant, \tilde{x}_i = a_i x_i, then the sequence of iterates
          produced by the algorithm would be unchanged.  This method can
          work very well in cases where the model parameters have widely
          different scales (ie: if some parameters are measured in
          nanometers, while others are measured in degrees Kelvin).
          This strategy has been proven effective on a large class of
          problems and so it is the library default, but it may not be
          the best choice for all problems.

      -- Option: gsl_multifit_nlinear_scale_levenberg
      -- Option: gsl_multilarge_nlinear_scale_levenberg
          This damping strategy was originally suggested by Levenberg,
          and corresponds to D^T D = I.  This method has also proven
          effective on a large class of problems, but is not
          scale-invariant.  However, some authors (e.g.  Transtrum and
          Sethna 2012) argue that this choice is better for problems
          which are susceptible to parameter evaporation (ie: parameters
          go to infinity)

      -- Option: gsl_multifit_nlinear_scale_marquardt
      -- Option: gsl_multilarge_nlinear_scale_marquardt
          This damping strategy was suggested by Marquardt, and
          corresponds to D^T D = diag(J^T J).  This method is
          scale-invariant, but it is generally considered inferior to
          both the Levenberg and More' strategies, though may work well
          on certain classes of problems.

 -- Parameter: const gsl_multifit_nlinear_solver * solver
 -- Parameter: const gsl_multilarge_nlinear_solver * solver

     Solving the trust region subproblem on each iteration almost always
     requires the solution of the following linear least squares system

          [J; sqrt(mu) D] \delta = - [f; 0]

     The SOLVER parameter determines how the system is solved and can be
     selected from the following choices:

      -- Default: gsl_multifit_nlinear_solver_qr
          This method solves the system using a rank revealing QR
          decomposition of the Jacobian J.  This method will produce
          reliable solutions in cases where the Jacobian is rank
          deficient or near-singular but does require about twice as
          many operations as the Cholesky method discussed below.

      -- Option: gsl_multifit_nlinear_solver_cholesky
      -- Default: gsl_multilarge_nlinear_solver_cholesky
          This method solves the alternate normal equations problem

               ( J^T J + \mu D^T D ) \delta = -J^T f

          by using a Cholesky decomposition of the matrix J^T J + \mu
          D^T D.  This method is faster than the QR approach, however it
          is susceptible to numerical instabilities if the Jacobian
          matrix is rank deficient or near-singular.  In these cases, an
          attempt is made to reduce the condition number of the matrix
          using Jacobi preconditioning, but for highly ill-conditioned
          problems the QR approach is better.  If it is known that the
          Jacobian matrix is well conditioned, this method is accurate
          and will perform faster than the QR approach.

      -- Option: gsl_multifit_nlinear_solver_svd
          This method solves the system using a singular value
          decomposition of the Jacobian J.  This method will produce the
          most reliable solutions for ill-conditioned Jacobians but is
          also the slowest solver method.

 -- Parameter: gsl_multifit_nlinear_fdtype fdtype

     This parameter specifies whether to use forward or centered
     differences when approximating the Jacobian.  This is only used
     when an analytic Jacobian is not provided to the solver.  This
     parameter may be set to one of the following choices.

      -- Default: GSL_MULTIFIT_NLINEAR_FWDIFF
          This specifies a forward finite difference to approximate the
          Jacobian matrix.  The Jacobian matrix will be calculated as

               J_ij = 1 / \Delta_j ( f_i(x + \Delta_j e_j) - f_i(x) )

          where \Delta_j = h |x_j| and e_j is the standard jth Cartesian
          unit basis vector so that x + \Delta_j e_j represents a small
          (forward) perturbation of the jth parameter by an amount
          \Delta_j.  The perturbation \Delta_j is proportional to the
          current value |x_j| which helps to calculate an accurate
          Jacobian when the various parameters have different scale
          sizes.  The value of h is specified by the 'h_df' parameter.
          The accuracy of this method is O(h), and evaluating this
          matrix requires an additional p function evaluations.

      -- Option: GSL_MULTIFIT_NLINEAR_CTRDIFF
          This specifies a centered finite difference to approximate the
          Jacobian matrix.  The Jacobian matrix will be calculated as

               J_ij = 1 / \Delta_j ( f_i(x + 1/2 \Delta_j e_j) - f_i(x - 1/2 \Delta_j e_j) )

          See above for a description of \Delta_j.  The accuracy of this
          method is O(h^2), but evaluating this matrix requires an
          additional 2p function evaluations.

 -- Parameter: double factor_up

     When a step is accepted, the trust region radius will be increased
     by this factor.  The default value is 3.

 -- Parameter: double factor_down

     When a step is rejected, the trust region radius will be decreased
     by this factor.  The default value is 2.

 -- Parameter: double avmax

     When using geodesic acceleration to solve a nonlinear least squares
     problem, an important parameter to monitor is the ratio of the
     acceleration term to the velocity term,

          |a| / |v|

     If this ratio is small, it means the acceleration correction is
     contributing very little to the step.  This could be because the
     problem is not "nonlinear" enough to benefit from the acceleration.
     If the ratio is large (> 1) it means that the acceleration is
     larger than the velocity, which shouldn't happen since the step
     represents a truncated series and so the second order term a should
     be smaller than the first order term v to guarantee convergence.
     Therefore any steps with a ratio larger than the parameter AVMAX
     are rejected.  AVMAX is set to 0.75 by default.  For problems which
     experience difficulty converging, this threshold could be lowered.

 -- Parameter: double h_df

     This parameter specifies the step size for approximating the
     Jacobian matrix with finite differences.  It is set to
     \sqrt{\epsilon} by default, where \epsilon is 'GSL_DBL_EPSILON'.

 -- Parameter: double h_fvv

     When using geodesic acceleration, the user must either supply a
     function to calculate f_{vv}(x) or the library can estimate this
     second directional derivative using a finite difference method.
     When using finite differences, the library must calculate f(x + h
     v) where h represents a small step in the velocity direction.  The
     parameter H_FVV defines this step size and is set to 0.02 by
     default.


File: gsl-ref.info,  Node: Nonlinear Least-Squares Initialization,  Next: Nonlinear Least-Squares Function Definition,  Prev: Nonlinear Least-Squares Tunable Parameters,  Up: Nonlinear Least-Squares Fitting

39.5 Initializing the Solver
============================

 -- Function: gsl_multifit_nlinear_workspace *
          gsl_multifit_nlinear_alloc (const gsl_multifit_nlinear_type *
          T, const gsl_multifit_nlinear_parameters * PARAMS, const
          size_t N, const size_t P)
 -- Function: gsl_multilarge_nlinear_workspace *
          gsl_multilarge_nlinear_alloc (const
          gsl_multilarge_nlinear_type * T, const
          gsl_multilarge_nlinear_parameters * PARAMS, const size_t N,
          const size_t P)
     These functions return a pointer to a newly allocated instance of a
     derivative solver of type T for N observations and P parameters.
     The PARAMS input specifies a tunable set of parameters which will
     affect important details in each iteration of the trust region
     subproblem algorithm.  It is recommended to start with the
     suggested default parameters (see
     'gsl_multifit_nlinear_default_parameters' and
     'gsl_multilarge_nlinear_default_parameters') and then tune the
     parameters once the code is working correctly.  See *note Nonlinear
     Least-Squares Tunable Parameters:: for descriptions of the various
     parameters.  For example, the following code creates an instance of
     a Levenberg-Marquardt solver for 100 data points and 3 parameters,
     using suggested defaults:

          const gsl_multifit_nlinear_type * T
              = gsl_multifit_nlinear_lm;
          gsl_multifit_nlinear_parameters params
              = gsl_multifit_nlinear_default_parameters();
          gsl_multifit_nlinear_workspace * w
              = gsl_multifit_nlinear_alloc (T, &params, 100, 3);

     The number of observations N must be greater than or equal to
     parameters P.

     If there is insufficient memory to create the solver then the
     function returns a null pointer and the error handler is invoked
     with an error code of 'GSL_ENOMEM'.

 -- Function: gsl_multifit_nlinear_parameters
          gsl_multifit_nlinear_default_parameters (void)
 -- Function: gsl_multilarge_nlinear_parameters
          gsl_multilarge_nlinear_default_parameters (void)
     These functions return a set of recommended default parameters for
     use in solving nonlinear least squares problems.  The user can tune
     each parameter to improve the performance on their particular
     problem, see *note Nonlinear Least-Squares Tunable Parameters::.

 -- Function: int gsl_multifit_nlinear_init (const gsl_vector * X,
          gsl_multifit_nlinear_fdf * FDF, gsl_multifit_nlinear_workspace
          * W)
 -- Function: int gsl_multifit_nlinear_winit (const gsl_vector * X,
          const gsl_vector * WTS, gsl_multifit_nlinear_fdf * FDF,
          gsl_multifit_nlinear_workspace * W)
 -- Function: int gsl_multilarge_nlinear_init (const gsl_vector * X,
          gsl_multilarge_nlinear_fdf * FDF,
          gsl_multilarge_nlinear_workspace * W)
 -- Function: int gsl_multilarge_nlinear_winit (const gsl_vector * X,
          const gsl_vector * WTS, gsl_multilarge_nlinear_fdf * FDF,
          gsl_multilarge_nlinear_workspace * W)
     These functions initialize, or reinitialize, an existing workspace
     W to use the system FDF and the initial guess X.  See *note
     Nonlinear Least-Squares Function Definition:: for a description of
     the FDF structure.

     Optionally, a weight vector WTS can be given to perform a weighted
     nonlinear regression.  Here, the weighting matrix is W =
     diag(w_1,w_2,...,w_n).

 -- Function: void gsl_multifit_nlinear_free
          (gsl_multifit_nlinear_workspace * W)
 -- Function: void gsl_multilarge_nlinear_free
          (gsl_multilarge_nlinear_workspace * W)
     These functions free all the memory associated with the workspace
     W.

 -- Function: const char * gsl_multifit_nlinear_name (const
          gsl_multifit_nlinear_workspace * W)
 -- Function: const char * gsl_multilarge_nlinear_name (const
          gsl_multilarge_nlinear_workspace * W)
     These functions return a pointer to the name of the solver.  For
     example,

          printf ("w is a '%s' solver\n",
                  gsl_multifit_nlinear_name (w));

     would print something like 'w is a 'trust-region' solver'.

 -- Function: const char * gsl_multifit_nlinear_trs_name (const
          gsl_multifit_nlinear_workspace * W)
 -- Function: const char * gsl_multilarge_nlinear_trs_name (const
          gsl_multilarge_nlinear_workspace * W)
     These functions return a pointer to the name of the trust region
     subproblem method.  For example,

          printf ("w is a '%s' solver\n",
                  gsl_multifit_nlinear_trs_name (w));

     would print something like 'w is a 'levenberg-marquardt' solver'.


File: gsl-ref.info,  Node: Nonlinear Least-Squares Function Definition,  Next: Nonlinear Least-Squares Iteration,  Prev: Nonlinear Least-Squares Initialization,  Up: Nonlinear Least-Squares Fitting

39.6 Providing the Function to be Minimized
===========================================

The user must provide n functions of p variables for the minimization
algorithm to operate on.  In order to allow for arbitrary parameters the
functions are defined by the following data types:

 -- Data Type: gsl_multifit_nlinear_fdf
     This data type defines a general system of functions with arbitrary
     parameters, the corresponding Jacobian matrix of derivatives, and
     optionally the second directional derivative of the functions for
     geodesic acceleration.

     'int (* f) (const gsl_vector * X, void * PARAMS, gsl_vector * F)'
          This function should store the n components of the vector f(x)
          in F for argument X and arbitrary parameters PARAMS, returning
          an appropriate error code if the function cannot be computed.

     'int (* df) (const gsl_vector * X, void * PARAMS, gsl_matrix * J)'
          This function should store the N-by-P matrix result J_ij = d
          f_i(x) / d x_j in J for argument X and arbitrary parameters
          PARAMS, returning an appropriate error code if the matrix
          cannot be computed.  If an analytic Jacobian is unavailable,
          or too expensive to compute, this function pointer may be set
          to NULL, in which case the Jacobian will be internally
          computed using finite difference approximations of the
          function F.

     'int (* fvv) (const gsl_vector * X, const gsl_vector * V, void * PARAMS, gsl_vector * FVV)'
          When geodesic acceleration is enabled, this function should
          store the n components of the vector f_{vv}(x) =
          \sum_{\alpha\beta} v_{\alpha} v_{\beta} {\partial \over
          \partial x_{\alpha}} {\partial \over \partial x_{\beta}} f(x),
          representing second directional derivatives of the function to
          be minimized, into the output FVV.  The parameter vector is
          provided in X and the velocity vector is provided in V, both
          of which have p components.  The arbitrary parameters are
          given in PARAMS.  If analytic expressions for f_{vv}(x) are
          unavailable or too difficult to compute, this function pointer
          may be set to NULL, in which case f_{vv}(x) will be computed
          internally using a finite difference approximation.

     'size_t n'
          the number of functions, i.e.  the number of components of the
          vector F.

     'size_t p'
          the number of independent variables, i.e.  the number of
          components of the vector X.

     'void * params'
          a pointer to the arbitrary parameters of the function.

     'size_t nevalf'
          This does not need to be set by the user.  It counts the
          number of function evaluations and is initialized by the
          '_init' function.

     'size_t nevaldf'
          This does not need to be set by the user.  It counts the
          number of Jacobian evaluations and is initialized by the
          '_init' function.

     'size_t nevalfvv'
          This does not need to be set by the user.  It counts the
          number of f_{vv}(x) evaluations and is initialized by the
          '_init' function.

 -- Data Type: gsl_multilarge_nlinear_fdf
     This data type defines a general system of functions with arbitrary
     parameters, a function to compute J u or J^T u for a given vector
     u, the normal equations matrix J^T J, and optionally the second
     directional derivative of the functions for geodesic acceleration.

     'int (* f) (const gsl_vector * X, void * PARAMS, gsl_vector * F)'
          This function should store the n components of the vector f(x)
          in F for argument X and arbitrary parameters PARAMS, returning
          an appropriate error code if the function cannot be computed.

     'int (* df) (CBLAS_TRANSPOSE_t TRANSJ, const gsl_vector * X, const gsl_vector * U, void * PARAMS, gsl_vector * V, gsl_matrix * JTJ)'
          If TRANSJ is equal to 'CblasNoTrans', then this function
          should compute the matrix-vector product J u and store the
          result in V.  If TRANSJ is equal to 'CblasTrans', then this
          function should compute the matrix-vector product J^T u and
          store the result in V.  Additionally, the normal equations
          matrix J^T J should be stored in the lower half of JTJ.  The
          input matrix JTJ could be set to NULL, for example by
          iterative methods which do not require this matrix, so the
          user should check for this prior to constructing the matrix.
          The input PARAMS contains the arbitrary parameters.

     'int (* fvv) (const gsl_vector * X, const gsl_vector * V, void * PARAMS, gsl_vector * FVV)'
          When geodesic acceleration is enabled, this function should
          store the n components of the vector f_{vv}(x) =
          \sum_{\alpha\beta} v_{\alpha} v_{\beta} {\partial \over
          \partial x_{\alpha}} {\partial \over \partial x_{\beta}} f(x),
          representing second directional derivatives of the function to
          be minimized, into the output FVV.  The parameter vector is
          provided in X and the velocity vector is provided in V, both
          of which have p components.  The arbitrary parameters are
          given in PARAMS.  If analytic expressions for f_{vv}(x) are
          unavailable or too difficult to compute, this function pointer
          may be set to NULL, in which case f_{vv}(x) will be computed
          internally using a finite difference approximation.

     'size_t n'
          the number of functions, i.e.  the number of components of the
          vector F.

     'size_t p'
          the number of independent variables, i.e.  the number of
          components of the vector X.

     'void * params'
          a pointer to the arbitrary parameters of the function.

     'size_t nevalf'
          This does not need to be set by the user.  It counts the
          number of function evaluations and is initialized by the
          '_init' function.

     'size_t nevaldfu'
          This does not need to be set by the user.  It counts the
          number of Jacobian matrix-vector evaluations (J u or J^T u)
          and is initialized by the '_init' function.

     'size_t nevaldf2'
          This does not need to be set by the user.  It counts the
          number of J^T J evaluations and is initialized by the '_init'
          function.

     'size_t nevalfvv'
          This does not need to be set by the user.  It counts the
          number of f_{vv}(x) evaluations and is initialized by the
          '_init' function.

Note that when fitting a non-linear model against experimental data, the
data is passed to the functions above using the PARAMS argument and the
trial best-fit parameters through the X argument.


File: gsl-ref.info,  Node: Nonlinear Least-Squares Iteration,  Next: Nonlinear Least-Squares Testing for Convergence,  Prev: Nonlinear Least-Squares Function Definition,  Up: Nonlinear Least-Squares Fitting

39.7 Iteration
==============

The following functions drive the iteration of each algorithm.  Each
function performs one iteration of the trust region method and updates
the state of the solver.

 -- Function: int gsl_multifit_nlinear_iterate
          (gsl_multifit_nlinear_workspace * W)
 -- Function: int gsl_multilarge_nlinear_iterate
          (gsl_multilarge_nlinear_workspace * W)
     These functions perform a single iteration of the solver W.  If the
     iteration encounters an unexpected problem then an error code will
     be returned.  The solver workspace maintains a current estimate of
     the best-fit parameters at all times.

The solver workspace W contains the following entries, which can be used
to track the progress of the solution:

'gsl_vector * x'
     The current position, length p.

'gsl_vector * f'
     The function residual vector at the current position f(x), length
     n.

'gsl_matrix * J'
     The Jacobian matrix at the current position J(x), size n-by-p (only
     for 'gsl_multifit_nlinear' interface).

'gsl_vector * dx'
     The difference between the current position and the previous
     position, i.e.  the last step \delta, taken as a vector, length p.

These quantities can be accessed with the following functions,

 -- Function: gsl_vector * gsl_multifit_nlinear_position (const
          gsl_multifit_nlinear_workspace * W)
 -- Function: gsl_vector * gsl_multilarge_nlinear_position (const
          gsl_multilarge_nlinear_workspace * W)
     These functions return the current position x (i.e.  best-fit
     parameters) of the solver W.

 -- Function: gsl_vector * gsl_multifit_nlinear_residual (const
          gsl_multifit_nlinear_workspace * W)
 -- Function: gsl_vector * gsl_multilarge_nlinear_residual (const
          gsl_multilarge_nlinear_workspace * W)
     These functions return the current residual vector f(x) of the
     solver W.  For weighted systems, the residual vector includes the
     weighting factor \sqrt{W}.

 -- Function: gsl_matrix * gsl_multifit_nlinear_jac (const
          gsl_multifit_nlinear_workspace * W)
     This function returns a pointer to the n-by-p Jacobian matrix for
     the current iteration of the solver W.  This function is available
     only for the 'gsl_multifit_nlinear' interface.

 -- Function: size_t gsl_multifit_nlinear_niter (const
          gsl_multifit_nlinear_workspace * W)
 -- Function: size_t gsl_multilarge_nlinear_niter (const
          gsl_multilarge_nlinear_workspace * W)
     These functions return the number of iterations performed so far.
     The iteration counter is updated on each call to the '_iterate'
     functions above, and reset to 0 in the '_init' functions.

 -- Function: int gsl_multifit_nlinear_rcond (double * RCOND, const
          gsl_multifit_nlinear_workspace * W)
 -- Function: int gsl_multilarge_nlinear_rcond (double * RCOND, const
          gsl_multilarge_nlinear_workspace * W)
     This function estimates the reciprocal condition number of the
     Jacobian matrix at the current position x and stores it in RCOND.
     The computed value is only an estimate to give the user a guideline
     as to the conditioning of their particular problem.  Its
     calculation is based on which factorization method is used
     (Cholesky, QR, or SVD).

        * For the Cholesky solver, the matrix J^T J is factored at each
          iteration.  Therefore this function will estimate the 1-norm
          condition number rcond^2 = 1/(||J^T J||_1 \cdot ||(J^T
          J)^{-1}||_1)

        * For the QR solver, J is factored as J = Q R at each iteration.
          For simplicity, this function calculates the 1-norm
          conditioning of only the R factor, rcond = 1 / (||R||_1 \cdot
          ||R^{-1}||_1).  This can be computed efficiently since R is
          upper triangular.

        * For the SVD solver, in order to efficiently solve the trust
          region subproblem, the matrix which is factored is J D^{-1},
          instead of J itself.  The resulting singular values are used
          to provide the 2-norm reciprocal condition number, as rcond =
          \sigma_{min} / \sigma_{max}.  Note that when using More'
          scaling, D \ne I and the resulting RCOND estimate may be
          significantly different from the true RCOND of J itself.


File: gsl-ref.info,  Node: Nonlinear Least-Squares Testing for Convergence,  Next: Nonlinear Least-Squares High Level Driver,  Prev: Nonlinear Least-Squares Iteration,  Up: Nonlinear Least-Squares Fitting

39.8 Testing for Convergence
============================

A minimization procedure should stop when one of the following
conditions is true:

   * A minimum has been found to within the user-specified precision.

   * A user-specified maximum number of iterations has been reached.

   * An error has occurred.

The handling of these conditions is under user control.  The functions
below allow the user to test the current estimate of the best-fit
parameters in several standard ways.

 -- Function: int gsl_multifit_nlinear_test (const double XTOL, const
          double GTOL, const double FTOL, int * INFO, const
          gsl_multifit_nlinear_workspace * W)
 -- Function: int gsl_multilarge_nlinear_test (const double XTOL, const
          double GTOL, const double FTOL, int * INFO, const
          gsl_multilarge_nlinear_workspace * W)
     These functions test for convergence of the minimization method
     using the following criteria:

        * Testing for a small step size relative to the current
          parameter vector
               |\delta_i| <= xtol (|x_i| + xtol)
          for each 0 <= i < p.  Each element of the step vector \delta
          is tested individually in case the different parameters have
          widely different scales.  Adding XTOL to |x_i| helps the test
          avoid breaking down in situations where the true solution
          value x_i = 0.  If this test succeeds, INFO is set to 1 and
          the function returns 'GSL_SUCCESS'.

          A general guideline for selecting the step tolerance is to
          choose xtol = 10^{-d} where d is the number of accurate
          decimal digits desired in the solution x.  See Dennis and
          Schnabel for more information.

        * Testing for a small gradient (g = \nabla \Phi(x) = J^T f)
          indicating a local function minimum:
               ||g||_inf <= gtol
          This expression tests whether the ratio (\nabla \Phi)_i x_i /
          \Phi is small.  Testing this scaled gradient is a better than
          \nabla \Phi alone since it is a dimensionless quantity and so
          independent of the scale of the problem.  The 'max' arguments
          help ensure the test doesn't break down in regions where x_i
          or \Phi(x) are close to 0.  If this test succeeds, INFO is set
          to 2 and the function returns 'GSL_SUCCESS'.

          A general guideline for choosing the gradient tolerance is to
          set 'gtol = GSL_DBL_EPSILON^(1/3)'.  See Dennis and Schnabel
          for more information.

     If none of the tests succeed, INFO is set to 0 and the function
     returns 'GSL_CONTINUE', indicating further iterations are required.


File: gsl-ref.info,  Node: Nonlinear Least-Squares High Level Driver,  Next: Nonlinear Least-Squares Covariance Matrix,  Prev: Nonlinear Least-Squares Testing for Convergence,  Up: Nonlinear Least-Squares Fitting

39.9 High Level Driver
======================

These routines provide a high level wrapper that combines the iteration
and convergence testing for easy use.

 -- Function: int gsl_multifit_nlinear_driver (const size_t MAXITER,
          const double XTOL, const double GTOL, const double FTOL, void
          (* CALLBACK)(const size_t ITER, void * PARAMS, const
          gsl_multifit_linear_workspace * W), void * CALLBACK_PARAMS,
          int * INFO, gsl_multifit_nlinear_workspace * W)
 -- Function: int gsl_multilarge_nlinear_driver (const size_t MAXITER,
          const double XTOL, const double GTOL, const double FTOL, void
          (* CALLBACK)(const size_t ITER, void * PARAMS, const
          gsl_multilarge_linear_workspace * W), void * CALLBACK_PARAMS,
          int * INFO, gsl_multilarge_nlinear_workspace * W)
     These functions iterate the nonlinear least squares solver W for a
     maximum of MAXITER iterations.  After each iteration, the system is
     tested for convergence with the error tolerances XTOL, GTOL and
     FTOL.  Additionally, the user may supply a callback function
     CALLBACK which is called after each iteration, so that the user may
     save or print relevant quantities for each iteration.  The
     parameter CALLBACK_PARAMS is passed to the CALLBACK function.  The
     parameters CALLBACK and CALLBACK_PARAMS may be set to NULL to
     disable this feature.  Upon successful convergence, the function
     returns 'GSL_SUCCESS' and sets INFO to the reason for convergence
     (see 'gsl_multifit_nlinear_test').  If the function has not
     converged after MAXITER iterations, 'GSL_EMAXITER' is returned.  In
     rare cases, during an iteration the algorithm may be unable to find
     a new acceptable step \delta to take.  In this case, 'GSL_ENOPROG'
     is returned indicating no further progress can be made.  If your
     problem is having difficulty converging, see *note Nonlinear
     Least-Squares Troubleshooting:: for further guidance.


File: gsl-ref.info,  Node: Nonlinear Least-Squares Covariance Matrix,  Next: Nonlinear Least-Squares Troubleshooting,  Prev: Nonlinear Least-Squares High Level Driver,  Up: Nonlinear Least-Squares Fitting

39.10 Covariance matrix of best fit parameters
==============================================

 -- Function: int gsl_multifit_nlinear_covar (const gsl_matrix * J,
          const double EPSREL, gsl_matrix * COVAR)
 -- Function: int gsl_multilarge_nlinear_covar (gsl_matrix * COVAR,
          gsl_multilarge_nlinear_workspace * W)
     This function computes the covariance matrix of best-fit parameters
     using the Jacobian matrix J and stores it in COVAR.  The parameter
     EPSREL is used to remove linear-dependent columns when J is rank
     deficient.

     The covariance matrix is given by,

          covar = (J^T J)^{-1}

     or in the weighted case,

          covar = (J^T W J)^{-1}

     and is computed using the factored form of the Jacobian (Cholesky,
     QR, or SVD). Any columns of R which satisfy

          |R_{kk}| <= epsrel |R_{11}|

     are considered linearly-dependent and are excluded from the
     covariance matrix (the corresponding rows and columns of the
     covariance matrix are set to zero).

     If the minimisation uses the weighted least-squares function f_i =
     (Y(x, t_i) - y_i) / \sigma_i then the covariance matrix above gives
     the statistical error on the best-fit parameters resulting from the
     Gaussian errors \sigma_i on the underlying data y_i.  This can be
     verified from the relation \delta f = J \delta c and the fact that
     the fluctuations in f from the data y_i are normalised by \sigma_i
     and so satisfy <\delta f \delta f^T> = I.

     For an unweighted least-squares function f_i = (Y(x, t_i) - y_i)
     the covariance matrix above should be multiplied by the variance of
     the residuals about the best-fit \sigma^2 = \sum (y_i - Y(x,t_i))^2
     / (n-p) to give the variance-covariance matrix \sigma^2 C.  This
     estimates the statistical error on the best-fit parameters from the
     scatter of the underlying data.

     For more information about covariance matrices see *note Fitting
     Overview::.


File: gsl-ref.info,  Node: Nonlinear Least-Squares Troubleshooting,  Next: Nonlinear Least-Squares Examples,  Prev: Nonlinear Least-Squares Covariance Matrix,  Up: Nonlinear Least-Squares Fitting

39.11 Troubleshooting
=====================

When developing a code to solve a nonlinear least squares problem, here
are a few considerations to keep in mind.

  1. The most common difficulty is the accurate implementation of the
     Jacobian matrix.  If the analytic Jacobian is not properly provided
     to the solver, this can hinder and many times prevent convergence
     of the method.  When developing a new nonlinear least squares code,
     it often helps to compare the program output with the internally
     computed finite difference Jacobian and the user supplied analytic
     Jacobian.  If there is a large difference in coefficients, it is
     likely the analytic Jacobian is incorrectly implemented.

  2. If your code is having difficulty converging, the next thing to
     check is the starting point provided to the solver.  The methods of
     this chapter are local methods, meaning if you provide a starting
     point far away from the true minimum, the method may converge to a
     local minimum or not converge at all.  Sometimes it is possible to
     solve a linearized approximation to the nonlinear problem, and use
     the linear solution as the starting point to the nonlinear problem.

  3. If the various parameters of the coefficient vector x vary widely
     in magnitude, then the problem is said to be badly scaled.  The
     methods of this chapter do attempt to automatically rescale the
     elements of x to have roughly the same order of magnitude, but in
     extreme cases this could still cause problems for convergence.  In
     these cases it is recommended for the user to scale their parameter
     vector x so that each parameter spans roughly the same range, say
     [-1,1].  The solution vector can be backscaled to recover the
     original units of the problem.


File: gsl-ref.info,  Node: Nonlinear Least-Squares Examples,  Next: Nonlinear Least-Squares References and Further Reading,  Prev: Nonlinear Least-Squares Troubleshooting,  Up: Nonlinear Least-Squares Fitting

39.12 Examples
==============

The following example programs demonstrate the nonlinear least squares
fitting capabilities.

* Menu:

* Nonlinear Least-Squares Exponential Fit Example::
* Nonlinear Least-Squares Geodesic Acceleration Example::
* Nonlinear Least-Squares Comparison Example::
* Nonlinear Least-Squares Large Example::


File: gsl-ref.info,  Node: Nonlinear Least-Squares Exponential Fit Example,  Next: Nonlinear Least-Squares Geodesic Acceleration Example,  Up: Nonlinear Least-Squares Examples

39.12.1 Exponential Fitting Example
-----------------------------------

The following example program fits a weighted exponential model with
background to experimental data, Y = A \exp(-\lambda t) + b.  The first
part of the program sets up the functions 'expb_f' and 'expb_df' to
calculate the model and its Jacobian.  The appropriate fitting function
is given by,

     f_i = (A \exp(-\lambda t_i) + b) - y_i

where we have chosen t_i = i.  The Jacobian matrix J is the derivative
of these functions with respect to the three parameters (A, \lambda, b).
It is given by,

     J_{ij} = d f_i / d x_j

where x_0 = A, x_1 = \lambda and x_2 = b.  The ith row of the Jacobian
is therefore

The main part of the program sets up a Levenberg-Marquardt solver and
some simulated random data.  The data uses the known parameters
(5.0,0.1,1.0) combined with Gaussian noise (standard deviation = 0.1)
over a range of 40 timesteps.  The initial guess for the parameters is
chosen as (1.0, 1.0, 0.0).  The iteration terminates when the relative
change in x is smaller than 10^{-8}, or when the magnitude of the
gradient falls below 10^{-8}.  Here are the results of running the
program:

     iter  0: A = 1.0000, lambda = 1.0000, b = 0.0000, cond(J) =      inf, |f(x)| = 62.2029
     iter  1: A = 1.2196, lambda = 0.3663, b = 0.0436, cond(J) =  53.6368, |f(x)| = 59.8062
     iter  2: A = 1.6062, lambda = 0.1506, b = 0.1054, cond(J) =  23.8178, |f(x)| = 53.9039
     iter  3: A = 2.4528, lambda = 0.0583, b = 0.2470, cond(J) =  20.0493, |f(x)| = 28.8039
     iter  4: A = 2.9723, lambda = 0.0494, b = 0.3727, cond(J) =  94.5601, |f(x)| = 15.3252
     iter  5: A = 3.3473, lambda = 0.0477, b = 0.4410, cond(J) = 229.3627, |f(x)| = 10.7511
     iter  6: A = 3.6690, lambda = 0.0508, b = 0.4617, cond(J) = 298.3589, |f(x)| = 9.7373
     iter  7: A = 3.9907, lambda = 0.0580, b = 0.5433, cond(J) = 250.0194, |f(x)| = 8.7661
     iter  8: A = 4.2353, lambda = 0.0731, b = 0.7989, cond(J) = 154.8571, |f(x)| = 7.4299
     iter  9: A = 4.6573, lambda = 0.0958, b = 1.0302, cond(J) = 140.2265, |f(x)| = 6.1893
     iter 10: A = 5.0138, lambda = 0.1060, b = 1.0329, cond(J) = 109.4141, |f(x)| = 5.4961
     iter 11: A = 5.1505, lambda = 0.1103, b = 1.0497, cond(J) = 100.8762, |f(x)| = 5.4552
     iter 12: A = 5.1724, lambda = 0.1110, b = 1.0526, cond(J) =  97.3403, |f(x)| = 5.4542
     iter 13: A = 5.1737, lambda = 0.1110, b = 1.0528, cond(J) =  96.7136, |f(x)| = 5.4542
     iter 14: A = 5.1738, lambda = 0.1110, b = 1.0528, cond(J) =  96.6678, |f(x)| = 5.4542
     iter 15: A = 5.1738, lambda = 0.1110, b = 1.0528, cond(J) =  96.6663, |f(x)| = 5.4542
     iter 16: A = 5.1738, lambda = 0.1110, b = 1.0528, cond(J) =  96.6663, |f(x)| = 5.4542
     summary from method 'trust-region/levenberg-marquardt'
     number of iterations: 16
     function evaluations: 23
     Jacobian evaluations: 17
     reason for stopping: small step size
     initial |f(x)| = 62.202928
     final   |f(x)| = 5.454180
     chisq/dof = 0.804002
     A      = 5.17379 +/- 0.27938
     lambda = 0.11104 +/- 0.00817
     b      = 1.05283 +/- 0.05365
     status = success

The approximate values of the parameters are found correctly, and the
chi-squared value indicates a good fit (the chi-squared per degree of
freedom is approximately 1).  In this case the errors on the parameters
can be estimated from the square roots of the diagonal elements of the
covariance matrix.  If the chi-squared value shows a poor fit (i.e.
chi^2/dof >> 1) then the error estimates obtained from the covariance
matrix will be too small.  In the example program the error estimates
are multiplied by \sqrt{\chi^2/dof} in this case, a common way of
increasing the errors for a poor fit.  Note that a poor fit will result
from the use of an inappropriate model, and the scaled error estimates
may then be outside the range of validity for Gaussian errors.

Additionally, we see that the condition number of J(x) stays reasonably
small throughout the iteration.  This indicates we could safely switch
to the Cholesky solver for speed improvement, although this particular
system is too small to really benefit.

     #include <stdlib.h>
     #include <stdio.h>
     #include <gsl/gsl_rng.h>
     #include <gsl/gsl_randist.h>
     #include <gsl/gsl_matrix.h>
     #include <gsl/gsl_vector.h>
     #include <gsl/gsl_blas.h>
     #include <gsl/gsl_multifit_nlinear.h>

     /* number of data points to fit */
     #define N 40

     struct data {
       size_t n;
       double * y;
     };

     int
     expb_f (const gsl_vector * x, void *data,
             gsl_vector * f)
     {
       size_t n = ((struct data *)data)->n;
       double *y = ((struct data *)data)->y;

       double A = gsl_vector_get (x, 0);
       double lambda = gsl_vector_get (x, 1);
       double b = gsl_vector_get (x, 2);

       size_t i;

       for (i = 0; i < n; i++)
         {
           /* Model Yi = A * exp(-lambda * i) + b */
           double t = i;
           double Yi = A * exp (-lambda * t) + b;
           gsl_vector_set (f, i, Yi - y[i]);
         }

       return GSL_SUCCESS;
     }

     int
     expb_df (const gsl_vector * x, void *data,
              gsl_matrix * J)
     {
       size_t n = ((struct data *)data)->n;

       double A = gsl_vector_get (x, 0);
       double lambda = gsl_vector_get (x, 1);

       size_t i;

       for (i = 0; i < n; i++)
         {
           /* Jacobian matrix J(i,j) = dfi / dxj, */
           /* where fi = (Yi - yi)/sigma[i],      */
           /*       Yi = A * exp(-lambda * i) + b  */
           /* and the xj are the parameters (A,lambda,b) */
           double t = i;
           double e = exp(-lambda * t);
           gsl_matrix_set (J, i, 0, e);
           gsl_matrix_set (J, i, 1, -t * A * e);
           gsl_matrix_set (J, i, 2, 1.0);
         }
       return GSL_SUCCESS;
     }

     void
     callback(const size_t iter, void *params,
              const gsl_multifit_nlinear_workspace *w)
     {
       gsl_vector *f = gsl_multifit_nlinear_residual(w);
       gsl_vector *x = gsl_multifit_nlinear_position(w);
       double rcond;

       /* compute reciprocal condition number of J(x) */
       gsl_multifit_nlinear_rcond(&rcond, w);

       fprintf(stderr, "iter %2zu: A = %.4f, lambda = %.4f, b = %.4f, cond(J) = %8.4f, |f(x)| = %.4f\n",
               iter,
               gsl_vector_get(x, 0),
               gsl_vector_get(x, 1),
               gsl_vector_get(x, 2),
               1.0 / rcond,
               gsl_blas_dnrm2(f));
     }

     int
     main (void)
     {
       const gsl_multifit_nlinear_type *T = gsl_multifit_nlinear_trust;
       gsl_multifit_nlinear_workspace *w;
       gsl_multifit_nlinear_fdf fdf;
       gsl_multifit_nlinear_parameters fdf_params =
         gsl_multifit_nlinear_default_parameters();
       const size_t n = N;
       const size_t p = 3;

       gsl_vector *f;
       gsl_matrix *J;
       gsl_matrix *covar = gsl_matrix_alloc (p, p);
       double y[N], weights[N];
       struct data d = { n, y };
       double x_init[3] = { 1.0, 1.0, 0.0 }; /* starting values */
       gsl_vector_view x = gsl_vector_view_array (x_init, p);
       gsl_vector_view wts = gsl_vector_view_array(weights, n);
       gsl_rng * r;
       double chisq, chisq0;
       int status, info;
       size_t i;

       const double xtol = 1e-8;
       const double gtol = 1e-8;
       const double ftol = 0.0;

       gsl_rng_env_setup();
       r = gsl_rng_alloc(gsl_rng_default);

       /* define the function to be minimized */
       fdf.f = expb_f;
       fdf.df = expb_df;   /* set to NULL for finite-difference Jacobian */
       fdf.fvv = NULL;     /* not using geodesic acceleration */
       fdf.n = n;
       fdf.p = p;
       fdf.params = &d;

       /* this is the data to be fitted */
       for (i = 0; i < n; i++)
         {
           double t = i;
           double yi = 1.0 + 5 * exp (-0.1 * t);
           double si = 0.1 * yi;
           double dy = gsl_ran_gaussian(r, si);

           weights[i] = 1.0 / (si * si);
           y[i] = yi + dy;
           printf ("data: %zu %g %g\n", i, y[i], si);
         };

       /* allocate workspace with default parameters */
       w = gsl_multifit_nlinear_alloc (T, &fdf_params, n, p);

       /* initialize solver with starting point and weights */
       gsl_multifit_nlinear_winit (&x.vector, &wts.vector, &fdf, w);

       /* compute initial cost function */
       f = gsl_multifit_nlinear_residual(w);
       gsl_blas_ddot(f, f, &chisq0);

       /* solve the system with a maximum of 20 iterations */
       status = gsl_multifit_nlinear_driver(20, xtol, gtol, ftol,
                                            callback, NULL, &info, w);

       /* compute covariance of best fit parameters */
       J = gsl_multifit_nlinear_jac(w);
       gsl_multifit_nlinear_covar (J, 0.0, covar);

       /* compute final cost */
       gsl_blas_ddot(f, f, &chisq);

     #define FIT(i) gsl_vector_get(w->x, i)
     #define ERR(i) sqrt(gsl_matrix_get(covar,i,i))

       fprintf(stderr, "summary from method '%s/%s'\n",
               gsl_multifit_nlinear_name(w),
               gsl_multifit_nlinear_trs_name(w));
       fprintf(stderr, "number of iterations: %zu\n",
               gsl_multifit_nlinear_niter(w));
       fprintf(stderr, "function evaluations: %zu\n", fdf.nevalf);
       fprintf(stderr, "Jacobian evaluations: %zu\n", fdf.nevaldf);
       fprintf(stderr, "reason for stopping: %s\n",
               (info == 1) ? "small step size" : "small gradient");
       fprintf(stderr, "initial |f(x)| = %f\n", sqrt(chisq0));
       fprintf(stderr, "final   |f(x)| = %f\n", sqrt(chisq));

       {
         double dof = n - p;
         double c = GSL_MAX_DBL(1, sqrt(chisq / dof));

         fprintf(stderr, "chisq/dof = %g\n", chisq / dof);

         fprintf (stderr, "A      = %.5f +/- %.5f\n", FIT(0), c*ERR(0));
         fprintf (stderr, "lambda = %.5f +/- %.5f\n", FIT(1), c*ERR(1));
         fprintf (stderr, "b      = %.5f +/- %.5f\n", FIT(2), c*ERR(2));
       }

       fprintf (stderr, "status = %s\n", gsl_strerror (status));

       gsl_multifit_nlinear_free (w);
       gsl_matrix_free (covar);
       gsl_rng_free (r);

       return 0;
     }

