Regression
==========

Ordinary Least Squares and Ridge Regression
-------------------------------------------

.. autoclass:: mlpy.RidgeRegression
   :members:

   .. versionadded:: 2.2.0

.. note::
   
   The predicted response is computed as:
      
   .. math:: \hat{y} = \beta_0 + X \boldsymbol\beta

Example (requires matplotlib module):

.. code-block:: python

   >>> import numpy as np
   >>> import mlpy
   >>> import matplotlib.pyplot as plt
   >>> x = np.array([[1], [2], [3], [4], [5], [6]]) # p = 1
   >>> y = np.array([0.13, 0.19, 0.31, 0.38, 0.49, 0.64])
   >>> rr = mlpy.RidgeRegression(alpha=0.0) # OLS
   >>> rr.learn(x, y)
   >>> y_hat = rr.pred(x)
   >>> plt.figure(1)
   >>> plt.plot(x[:, 0], y, 'o') # show y
   >>> plt.plot(x[:, 0], y_hat) # show y_hat
   >>> plt.show()

.. image:: images/ols.png

.. code-block:: python

   >>> rr.beta0()
   0.0046666666666667078
   >>> rr.beta()
   array([ 0.10057143])


Kernel Ridge Regression
-----------------------

.. autoclass:: mlpy.KernelRidgeRegression
   :members:

   .. versionadded:: 2.2.0

Example (requires matplotlib module):

.. code-block:: python

   >>> import numpy as np
   >>> import mlpy
   >>> import matplotlib.pyplot as plt
   >>> x = np.array([[1], [2], [3], [4], [5], [6]]) # p = 1
   >>> y = np.array([0.13, 0.19, 0.31, 0.38, 0.49, 0.64])
   >>> kernel = mlpy.KernelGaussian(sigma=0.01)
   >>> krr = mlpy.KernelRidgeRegression(kernel=kernel, alpha=0.01)
   >>> krr.learn(x,y)
   >>> y_hat = krr.pred(x)
   >>> plt.figure(1)
   >>> plt.plot(x[:, 0], y, 'o') # show y
   >>> plt.plot(x[:, 0], y_hat) # show y_hat
   >>> plt.show()

.. image:: images/krr.png


Least Angle Regression (LAR)
----------------------------
Least Angle Regression is described in [Efron04]_.

Covariates should be standardized to have mean 0 and unit length,
and the response should have mean 0:

.. math::
   
   \sum_{i=1}^n{x_{ij}} = 0, \hspace{1cm} \sum_{i=1}^n{x_{ij}^2} = 1, \hspace{1cm} \sum_{i=1}^n{y_i} = 0 \hspace{1cm} \mathrm{for} \hspace{0.2cm} j = 1, 2, \dots, p.


.. autoclass:: mlpy.Lar
   :members:

   .. versionadded:: 2.2.0

LASSO (LARS implementation)
---------------------------
It implements simple modifications of the LARS algorithm
that produces Lasso estimates. See [Efron04]_ and [Tibshirani96]_.

Covariates should be standardized to have mean 0 and unit length,
and the response should have mean 0:

.. math::
   
   \sum_{i=1}^n{x_{ij}} = 0, \hspace{1cm} \sum_{i=1}^n{x_{ij}^2} = 1, \hspace{1cm} \sum_{i=1}^n{y_i} = 0 \hspace{1cm} \mathrm{for} \hspace{0.2cm} j = 1, 2, \dots, p.

.. autoclass:: mlpy.Lasso
   :members:

   .. versionadded:: 2.2.0


Gradient Descent
----------------

.. autoclass:: mlpy.GradientDescent
   :members:

   .. versionadded:: 2.2.0
   

.. [Efron04] Bradley Efron, Trevor Hastie, Iain Johnstone and Robert Tibshirani. Least Angle Regression. Annals of Statistics, 2004, volume 32, pages 407-499.
.. [Tibshirani96] Robert Tibshirani. Regression shrinkage and selection via the lasso. J. Royal. Statist. Soc B., 1996, volume 58, number 1, pages 267-288.
