scikit-learn sklearn gaussianprocess to solve regression and probabilistic classification problems Gaussian Processes Gaussian Processes GP are a nonparametric supervised learning method used gaussianprocess

Answer:
.. _gaussian_process:

==================
Gaussian Processes
==================

.. currentmodule:: sklearn.gaussian_process

**Gaussian Processes (GP)** are a nonparametric supervised learning method used
to solve *regression* and *probabilistic classification* problems.

The advantages of Gaussian processes are:

- The prediction interpolates the observations (at least for regular
  kernels).

- The prediction is probabilistic (Gaussian) so that one can compute
  empirical confidence intervals and decide based on those if one should
  refit (online fitting, adaptive fitting) the prediction in some
  region of interest.

- Versatile: different :ref:`kernels
  <gp_kernels>` can be specified. Common kernels are provided, but
  it is also possible to specify custom kernels.

The disadvantages of Gaussian processes include:

- Our implementation is not sparse, i.e., they use the whole samples/features
  information to perform the prediction.

- They lose efficiency in high dimensional spaces -- namely when the number
  of features exceeds a few dozens.

.. _gpr:

Gaussian Process Regression (GPR)
=================================

.. currentmodule:: sklearn.gaussian_process

The :class:`GaussianProcessRegressor` implements Gaussian processes (GP) for
regression purposes. For this, the prior of the GP needs to be specified. GP
will combine this prior and the likelihood function based on training samples.
It allows to give a probabilistic approach to prediction by giving the mean and
standard deviation as output when predicting.

.. figure:: ../auto_examples/gaussian_process/images/sphx_glr_plot_gpr_noisy_targets_002.png
   :target: ../auto_examples/gaussian_process/plot_gpr_noisy_targets.html
   :align: center

The prior mean is assumed to be constant and zero (for `normalize_y=False`) or
the training data's mean (for `normalize_y=True`). The prior's covariance is
specified by passing a :ref:`kernel <gp_kernels>` object. The hyperparameters
of the kernel are optimized when fitting the :class:`GaussianProcessRegressor`
by maximizing the log-marginal-likelihood (LML) based on the passed
`optimizer`. As the LML may have multiple local optima, the optimizer can be
started repeatedly by specifying `n_restarts_optimizer`. The first run is
always conducted starting from the initial hyperparameter values of the kernel;
subsequent runs are conducted from hyperparameter values that have been chosen
randomly from the range of allowed values. If the initial hyperparameters
should be kept fixed, `None` can be passed as optimizer.

The noise level in the targets can be specified by passing it via the parameter
`alpha`, either globally as a scalar or per datapoint. Note that a moderate
noise level can also be helpful for dealing with numeric instabilities during
fitting as it is effectively implemented as Tikhonov regularization, i.e., by
adding it to the diagonal of the kernel matrix. An alternative to specifying
the noise level explicitly is to include a
:class:`~sklearn.gaussian_process.kernels.WhiteKernel` component into the
kernel, which can estimate the global noise level from the data (see example
below). The figure below shows the effect of noisy target handled by setting
the parameter `alpha`.

.. figure:: ../auto_examples/gaussian_process/images/sphx_glr_plot_gpr_noisy_targets_003.png
   :target: ../auto_examples/gaussian_process/plot_gpr_noisy_targets.html
   :align: center

The implementation is based on Algorithm 2.1 of [RW2006]_. In addition to
the API of standard scikit-learn estimators, :class:`GaussianProcessRegressor`:

* allows prediction without prior fitting (based on the GP prior)

* provides an additional method ``sample_y(X)``, which evaluates samples
  drawn from the GPR (prior or posterior) at given inputs

* exposes a method ``log_marginal_likelihood(theta)``, which can be used
  externally for other ways of selecting hyperparameters, e.g., via
  Markov chain Monte Carlo.

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_gaussian_process_plot_gpr_noisy_targets.py`
* :ref:`sphx_glr_auto_examples_gaussian_process_plot_gpr_noisy.py`
* :ref:`sphx_glr_auto_examples_gaussian_process_plot_compare_gpr_krr.py`
* :ref:`sphx_glr_auto_examples_gaussian_process_plot_gpr_co2.py`

.. _gpc:

Gaussian Process Classification (GPC)
=====================================

.. currentmodule:: sklearn.gaussian_process

The :class:`GaussianProcessClassifier` implements Gaussian processes (GP) for
classification purposes, more specifically for probabilistic classification,
where test predictions take the form of class probabilities.
GaussianProcessClassifier places a GP prior on a latent function :math:`f`,
which is then squashed through a link function to obtain the probabilistic
classification. The latent function :math:`f` is a so-called nuisance function,
whose values are not observed and are not relevant by themselves.
Its purpose is to allow a convenient formulation of the model, and :math:`f`
is removed (integrated out) during prediction. GaussianProcessClassifier
implements the logistic link function, for which the integral cannot be
computed analytically but is easily approximated in the binary case.

In contrast to the regression setting, the posterior of the latent function
:math:`f` is not Gaussian even for a GP prior since a Gaussian likelihood is
inappropriate for discrete class labels. Rather, a non-Gaussian likelihood
corresponding to the logistic link function (logit) is used.
GaussianProcessClassifier approximates the non-Gaussian posterior with a
Gaussian based on the Laplace approximation. More details can be found in
Chapter 3 of [RW2006]_.

The GP prior mean is assumed to be zero. The prior's
covariance is specified by passing a :ref:`kernel <gp_kernels>` object. The
hyperparameters of the kernel are optimized during fitting of
GaussianProcessRegressor by maximizing the log-marginal-likelihood (LML) based
on the passed ``optimizer``. As the LML may have multiple local optima, the
optimizer can be started repeatedly by specifying ``n_restarts_optimizer``. The
first run is always conducted starting from the initial hyperparameter values
of the kernel; subsequent runs are conducted from hyperparameter values
that have been chosen randomly from the range of allowed values.
If the initial hyperparameters should be kept fixed, `None` can be passed as
optimizer.

:class:`GaussianProcessClassifier` supports multi-class classification
by performing either one-versus-rest or one-versus-one based training and
prediction.  In one-versus-rest, one binary Gaussian process classifier is
fitted for each class, which is trained to separate this class from the rest.
In "one_vs_one", one binary Gaussian process classifier is fitted for each pair
of classes, which is trained to separate these two classes. The predictions of
these binary predictors are combined into multi-class predictions. See the
section on :ref:`multi-class classification <multiclass>` for more details.

In the case of Gaussian process classification, "one_vs_one" might be
computationally  cheaper since it has to solve many problems involving only a
subset of the whole training set rather than fewer problems on the whole
dataset. Since Gaussian process classification scales cubically with the size
of the dataset, this might be considerably faster. However, note that
"one_vs_one" does not support predicting probability estimates but only plain
predictions. Moreover, note that :class:`GaussianProcessClassifier` does not
(yet) implement a true multi-class Laplace approximation internally, but
as discussed above is based on solving several binary classification tasks
internally, which are combined using one-versus-rest or one-versus-one.

GPC examples
============

Probabilistic predictions with GPC
----------------------------------

This example illustrates the predicted probability of GPC for an RBF kernel
with different choices of the hyperparameters. The first figure shows the
predicted probability of GPC with arbitrarily chosen hyperparameters and with
the hyperparameters corresponding to the maximum log-marginal-likelihood (LML).

While the hyperparameters chosen by optimizing LML have a considerably larger
LML, they perform slightly worse according to the log-loss on test data. The
figure shows that this is because they exhibit a steep change of the class
probabilities at the class boundaries (which is good) but have predicted
probabilities close to 0.5 far away from the class boundaries (which is bad)
This undesirable effect is caused by the Laplace approximation used
internally by GPC.

The second figure shows the log-marginal-likelihood for different choices of
the kernel's hyperparameters, highlighting the two choices of the
hyperparameters used in the first figure by black dots.

.. figure:: ../auto_examples/gaussian_process/images/sphx_glr_plot_gpc_001.png
   :target: ../auto_examples/gaussian_process/plot_gpc.html
   :align: center

.. figure:: ../auto_examples/gaussian_process/images/sphx_glr_plot_gpc_002.png
   :target: ../auto_examples/gaussian_process/plot_gpc.html
   :align: center

Illustration of GPC on the XOR dataset
--------------------------------------

.. currentmodule:: sklearn.gaussian_process.kernels

This example illustrates GPC on XOR data. Compared are a stationary, isotropic
kernel (:class:`RBF`) and a non-stationary kernel (:class:`DotProduct`). On
this particular dataset, the :class:`DotProduct` kernel obtains considerably
better results because the class-boundaries are linear and coincide with the
coordinate axes. In practice, however, stationary kernels such as :class:`RBF`
often obtain better results.

.. figure:: ../auto_examples/gaussian_process/images/sphx_glr_plot_gpc_xor_001.png
   :target: ../auto_examples/gaussian_process/plot_gpc_xor.html
   :align: center

.. currentmodule:: sklearn.gaussian_process

Gaussian process classification (GPC) on iris dataset
-----------------------------------------------------

This example illustrates the predicted probability of GPC for an isotropic
and anisotropic RBF kernel on a two-dimensional version for the iris-dataset.
This illustrates the applicability of GPC to non-binary classification.
The anisotropic RBF kernel obtains slightly higher log-marginal-likelihood by
assigning different length-scales to the two feature dimensions.

.. figure:: ../auto_examples/gaussian_process/images/sphx_glr_plot_gpc_iris_001.png
   :target: ../auto_examples/gaussian_process/plot_gpc_iris.html
   :align: center

.. _gp_kernels:

Kernels for Gaussian Processes
==============================
.. currentmodule:: sklearn.gaussian_process.kernels

Kernels (also called "covariance functions" in the context of GPs) are a crucial
ingredient of GPs which determine the shape of prior and posterior of the GP.
They encode the assumptions on the function being learned by defining the "similarity"
of two datapoints combined with the assumption that similar datapoints should
have similar target values. Two categories of kernels can be distinguished:
stationary kernels depend only on the distance of two datapoints and not on their
absolute values :math:`k(x_i, x_j)= k(d(x_i, x_j))` and are thus invariant to
translations in the input space, while non-stationary kernels
depend also on the specific values of the datapoints. Stationary kernels can further
be subdivided into isotropic and anisotropic kernels, where isotropic kernels are
also invariant to rotations in the input space. For more details, we refer to
Chapter 4 of [RW2006]_. For guidance on how to best combine different kernels,
we refer to [Duv2014]_.

.. dropdown:: Gaussian Process Kernel API

   The main usage of a :class:`Kernel` is to compute the GP's covariance between
   datapoints. For this, the method ``__call__`` of the kernel can be called. This
   method can either be used to compute the "auto-covariance" of all pairs of
   datapoints in a 2d array X, or the "cross-covariance" of all combinations
   of datapoints of a 2d array X with datapoints in a 2d array Y. The following
   identity holds true for all kernels k (except for the :class:`WhiteKernel`):
   ``k(X) == K(X, Y=X)``

   If only the diagonal of the auto-covariance is being used, the method ``diag()``
   of a kernel can be called, which is more computationally efficient than the
   equivalent call to ``__call__``: ``np.diag(k(X, X)) == k.diag(X)``

   Kernels are parameterized by a vector :math:`\theta` of hyperparameters. These
   hyperparameters can for instance control length-scales or periodicity of a
   kernel (see below). All kernels support computing analytic gradients
   of the kernel's auto-covariance with respect to :math:`log(\theta)` via setting
   ``eval_gradient=True`` in the ``__call__`` method.
   That is, a ``(len(X), len(X), len(theta))`` array is returned where the entry
   ``[i, j, l]`` contains :math:`\frac{\partial k_\theta(x_i, x_j)}{\partial log(\theta_l)}`.
   This gradient is used by the Gaussian process (both regressor and classifier)
   in computing the gradient of the log-marginal-likelihood, which in turn is used
   to determine the value of :math:`\theta`, which maximizes the log-marginal-likelihood,
   via gradient ascent. For each hyperparameter, the initial value and the
   bounds need to be specified when creating an instance of the kernel. The
   current value of :math:`\theta` can be get and set via the property
   ``theta`` of the kernel object. Moreover, the bounds of the hyperparameters can be
   accessed by the property ``bounds`` of the kernel. Note that both properties
   (theta and bounds) return log-transformed values of the internally used values
   since those are typically more amenable to gradient-based optimization.
   The specification of each hyperparameter is stored in the form of an instance of
   :class:`Hyperparameter` in the respective kernel. Note that a kernel using a
   hyperparameter with name "x" must have the attributes self.x and self.x_bounds.

   The abstract base class for all kernels is :class:`Kernel`. Kernel implements a
   similar interface as :class:`~sklearn.base.BaseEstimator`, providing the
   methods ``get_params()``, ``set_params()``, and ``clone()``. This allows
   setting kernel values also via meta-estimators such as
   :class:`~sklearn.pipeline.Pipeline` or
   :class:`~sklearn.model_selection.GridSearchCV`. Note that due to the nested
   structure of kernels (by applying kernel operators, see below), the names of
   kernel parameters might become relatively complicated. In general, for a binary
   kernel operator, parameters of the left operand are prefixed with ``k1__`` and
   parameters of the right operand with ``k2__``. An additional convenience method
   is ``clone_with_theta(theta)``, which returns a cloned version of the kernel
   but with the hyperparameters set to ``theta``. An illustrative example:

      >>> from sklearn.gaussian_process.kernels import ConstantKernel, RBF
      >>> kernel = ConstantKernel(constant_value=1.0, constant_value_bounds=(0.0, 10.0)) * RBF(length_scale=0.5, length_scale_bounds=(0.0, 10.0)) + RBF(length_scale=2.0, length_scale_bounds=(0.0, 10.0))
      >>> for hyperparameter in kernel.hyperparameters: print(hyperparameter)
      Hyperparameter(name='k1__k1__constant_value', value_type='numeric', bounds=array([[ 0., 10.]]), n_elements=1, fixed=False)
      Hyperparameter(name='k1__k2__length_scale', value_type='numeric', bounds=array([[ 0., 10.]]), n_elements=1, fixed=False)
      Hyperparameter(name='k2__length_scale', value_type='numeric', bounds=array([[ 0., 10.]]), n_elements=1, fixed=False)
      >>> params = kernel.get_params()
      >>> for key in sorted(params): print("%s : %s" % (key, params[key]))
      k1 : 1**2 * RBF(length_scale=0.5)
      k1__k1 : 1**2
      k1__k1__constant_value : 1.0
      k1__k1__constant_value_bounds : (0.0, 10.0)
      k1__k2 : RBF(length_scale=0.5)
      k1__k2__length_scale : 0.5
      k1__k2__length_scale_bounds : (0.0, 10.0)
      k2 : RBF(length_scale=2)
      k2__length_scale : 2.0
      k2__length_scale_bounds : (0.0, 10.0)
      >>> print(kernel.theta)  # Note: log-transformed
      [ 0.         -0.69314718  0.69314718]
      >>> print(kernel.bounds)  # Note: log-transformed
      [[      -inf 2.30258509]
      [      -inf 2.30258509]
      [      -inf 2.30258509]]

   All Gaussian process kernels are interoperable with :mod:`sklearn.metrics.pairwise`
   and vice versa: instances of subclasses of :class:`Kernel` can be passed as
   ``metric`` to ``pairwise_kernels`` from :mod:`sklearn.metrics.pairwise`. Moreover,
   kernel functions from pairwise can be used as GP kernels by using the wrapper
   class :class:`PairwiseKernel`. The only caveat is that the gradient of
   the hyperparameters is not analytic but numeric and all those kernels support
   only isotropic distances. The parameter ``gamma`` is considered to be a
   hyperparameter and may be optimized. The other kernel parameters are set
   directly at initialization and are kept fixed.

Basic kernels
-------------
The :class:`ConstantKernel` kernel can be used as part of a :class:`Product`
kernel where it scales the magnitude of the other factor (kernel) or as part
of a :class:`Sum` kernel, where it modifies the mean of the Gaussian process.
It depends on a parameter :math:`constant\_value`. It is defined as:

.. math::
   k(x_i, x_j) = constant\_value \;\forall\; x_1, x_2

The main use-case of the :class:`WhiteKernel` kernel is as part of a
sum-kernel where it explains the noise-component of the signal. Tuning its
parameter :math:`noise\_level` corresponds to estimating the noise-level.
It is defined as:

.. math::
    k(x_i, x_j) = noise\_level \text{ if } x_i == x_j \text{ else } 0

Kernel operators
----------------
Kernel operators take one or two base kernels and combine them into a new
kernel. The :class:`Sum` kernel takes two kernels :math:`k_1` and :math:`k_2`
and combines them via :math:`k_{sum}(X, Y) = k_1(X, Y) + k_2(X, Y)`.
The  :class:`Product` kernel takes two kernels :math:`k_1` and :math:`k_2`
and combines them via :math:`k_{product}(X, Y) = k_1(X, Y) * k_2(X, Y)`.
The :class:`Exponentiation` kernel takes one base kernel and a scalar parameter
:math:`p` and combines them via
:math:`k_{exp}(X, Y) = k(X, Y)^p`.
Note that magic methods ``__add__``, ``__mul___`` and ``__pow__`` are
overridden on the Kernel objects, so one can use e.g. ``RBF() + RBF()`` as
a shortcut for ``Sum(RBF(), RBF())``.

Radial basis function (RBF) kernel
----------------------------------
The :class:`RBF` kernel is a stationary kernel. It is also known as the "squared
exponential" kernel. It is parameterized by a length-scale parameter :math:`l>0`, which
can either be a scalar (isotropic variant of the kernel) or a vector with the same
number of dimensions as the inputs :math:`x` (anisotropic variant of the kernel).
The kernel is given by:

.. math::
   k(x_i, x_j) = \text{exp}\left(- \frac{d(x_i, x_j)^2}{2l^2} \right)

where :math:`d(\cdot, \cdot)` is the Euclidean distance.
This kernel is infinitely differentiable, which implies that GPs with this
kernel as covariance function have mean square derivatives of all orders, and are thus
very smooth. The prior and posterior of a GP resulting from an RBF kernel are shown in
the following figure:

.. figure:: ../auto_examples/gaussian_process/images/sphx_glr_plot_gpr_prior_posterior_001.png
   :target: ../auto_examples/gaussian_process/plot_gpr_prior_posterior.html
   :align: center

Matérn kernel
-------------
The :class:`Matern` kernel is a stationary kernel and a generalization of the
:class:`RBF` kernel. It has an additional parameter :math:`\nu` which controls
the smoothness of the resulting function. It is parameterized by a length-scale parameter :math:`l>0`, which can either be a scalar (isotropic variant of the kernel) or a vector with the same number of dimensions as the inputs :math:`x` (anisotropic variant of the kernel).

.. dropdown:: Mathematical implementation of Matérn kernel

   The kernel is given by:

   .. math::

      k(x_i, x_j) = \frac{1}{\Gamma(\nu)2^{\nu-1}}\Bigg(\frac{\sqrt{2\nu}}{l} d(x_i , x_j )\Bigg)^\nu K_\nu\Bigg(\frac{\sqrt{2\nu}}{l} d(x_i , x_j )\Bigg),

   where :math:`d(\cdot,\cdot)` is the Euclidean distance, :math:`K_\nu(\cdot)` is a modified Bessel function and :math:`\Gamma(\cdot)` is the gamma function.
   As :math:`\nu\rightarrow\infty`, the Matérn kernel converges to the RBF kernel.
   When :math:`\nu = 1/2`, the Matérn kernel becomes identical to the absolute
   exponential kernel, i.e.,

   .. math::
      k(x_i, x_j) = \exp \Bigg(- \frac{1}{l} d(x_i , x_j ) \Bigg) \quad \quad \nu= \tfrac{1}{2}

   In particular, :math:`\nu = 3/2`:

   .. math::
      k(x_i, x_j) =  \Bigg(1 + \frac{\sqrt{3}}{l} d(x_i , x_j )\Bigg) \exp \Bigg(-\frac{\sqrt{3}}{l} d(x_i , x_j ) \Bigg) \quad \quad \nu= \tfrac{3}{2}

   and :math:`\nu = 5/2`:

   .. math::
      k(x_i, x_j) = \Bigg(1 + \frac{\sqrt{5}}{l} d(x_i , x_j ) +\frac{5}{3l} d(x_i , x_j )^2 \Bigg) \exp \Bigg(-\frac{\sqrt{5}}{l} d(x_i , x_j ) \Bigg) \quad \quad \nu= \tfrac{5}{2}

   are popular choices for learning functions that are not infinitely
   differentiable (as assumed by the RBF kernel) but at least once (:math:`\nu =
   3/2`) or twice differentiable (:math:`\nu = 5/2`).

   The flexibility of controlling the smoothness of the learned function via :math:`\nu`
   allows adapting to the properties of the true underlying functional relation.

The prior and posterior of a GP resulting from a Matérn kernel are shown in
the following figure:

.. figure:: ../auto_examples/gaussian_process/images/sphx_glr_plot_gpr_prior_posterior_005.png
   :target: ../auto_examples/gaussian_process/plot_gpr_prior_posterior.html
   :align: center

See [RW2006]_, pp84 for further details regarding the
different variants of the Matérn kernel.

Rational quadratic kernel
-------------------------

The :class:`RationalQuadratic` kernel can be seen as a scale mixture (an infinite sum)
of :class:`RBF` kernels with different characteristic length-scales. It is parameterized
by a length-scale parameter :math:`l>0` and a scale mixture parameter  :math:`\alpha>0`
Only the isotropic variant where :math:`l` is a scalar is supported at the moment.
The kernel is given by:

.. math::
   k(x_i, x_j) = \left(1 + \frac{d(x_i, x_j)^2}{2\alpha l^2}\right)^{-\alpha}

The prior and posterior of a GP resulting from a :class:`RationalQuadratic` kernel are shown in
the following figure:

.. figure:: ../auto_examples/gaussian_process/images/sphx_glr_plot_gpr_prior_posterior_002.png
   :target: ../auto_examples/gaussian_process/plot_gpr_prior_posterior.html
   :align: center

Exp-Sine-Squared kernel
-----------------------

The :class:`ExpSineSquared` kernel allows modeling periodic functions.
It is parameterized by a length-scale parameter :math:`l>0` and a periodicity parameter
:math:`p>0`. Only the isotropic variant where :math:`l` is a scalar is supported at the moment.
The kernel is given by:

.. math::
   k(x_i, x_j) = \text{exp}\left(- \frac{ 2\sin^2(\pi d(x_i, x_j) / p) }{ l^ 2} \right)

The prior and posterior of a GP resulting from an ExpSineSquared kernel are shown in
the following figure:

.. figure:: ../auto_examples/gaussian_process/images/sphx_glr_plot_gpr_prior_posterior_003.png
   :target: ../auto_examples/gaussian_process/plot_gpr_prior_posterior.html
   :align: center

Dot-Product kernel
------------------

The :class:`DotProduct` kernel is non-stationary and can be obtained from linear regression
by putting :math:`N(0, 1)` priors on the coefficients of :math:`x_d (d = 1, . . . , D)` and
a prior of :math:`N(0, \sigma_0^2)` on the bias. The :class:`DotProduct` kernel is invariant to a rotation
of the coordinates about the origin, but not translations.
It is parameterized by a parameter :math:`\sigma_0^2`. For :math:`\sigma_0^2 = 0`, the kernel
is called the homogeneous linear kernel, otherwise it is inhomogeneous. The kernel is given by

.. math::
   k(x_i, x_j) = \sigma_0 ^ 2 + x_i \cdot x_j

The :class:`DotProduct` kernel is commonly combined with exponentiation. An example with exponent 2 is
shown in the following figure:

.. figure:: ../auto_examples/gaussian_process/images/sphx_glr_plot_gpr_prior_posterior_004.png
   :target: ../auto_examples/gaussian_process/plot_gpr_prior_posterior.html
   :align: center

References
----------

.. [RW2006] `Carl E. Rasmussen and Christopher K.I. Williams,
   "Gaussian Processes for Machine Learning",
   MIT Press 2006 <https://www.gaussianprocess.org/gpml/chapters/RW.pdf>`_

.. [Duv2014] `David Duvenaud, "The Kernel Cookbook: Advice on Covariance functions", 2014
   <https://www.cs.toronto.edu/~duvenaud/cookbook/>`_

.. currentmodule:: sklearn.gaussian_process

gaussian process                      Gaussian Processes                        currentmodule   sklearn gaussian process    Gaussian Processes  GP    are a nonparametric supervised learning method used to solve  regression  and  probabilistic classification  problems   The advantages of Gaussian processes are     The prediction interpolates the observations  at least for regular   kernels      The prediction is probabilistic  Gaussian  so that one can compute   empirical confidence intervals and decide based on those if one should   refit  online fitting  adaptive fitting  the prediction in some   region of interest     Versatile  different  ref  kernels    gp kernels   can be specified  Common kernels are provided  but   it is also possible to specify custom kernels   The disadvantages of Gaussian processes include     Our implementation is not sparse  i e   they use the whole samples features   information to perform the prediction     They lose efficiency in high dimensional spaces    namely when the number   of features exceeds a few dozens        gpr   Gaussian Process Regression  GPR                                        currentmodule   sklearn gaussian process  The  class  GaussianProcessRegressor  implements Gaussian processes  GP  for regression purposes  For this  the prior of the GP needs to be specified  GP will combine this prior and the likelihood function based on training samples  It allows to give a probabilistic approach to prediction by giving the mean and standard deviation as output when predicting      figure      auto examples gaussian process images sphx glr plot gpr noisy targets 002 png     target     auto examples gaussian process plot gpr noisy targets html     align  center  The prior mean is assumed to be constant and zero  for  normalize y False   or the training data s mean  for  normalize y True    The prior s covariance is specified by passing a  ref  kernel  gp kernels   object  The hyperparameters of the kernel are optimized when fitting the  class  GaussianProcessRegressor  by maximizing the log marginal likelihood  LML  based on the passed  optimizer   As the LML may have multiple local optima  the optimizer can be started repeatedly by specifying  n restarts optimizer   The first run is always conducted starting from the initial hyperparameter values of the kernel  subsequent runs are conducted from hyperparameter values that have been chosen randomly from the range of allowed values  If the initial hyperparameters should be kept fixed   None  can be passed as optimizer   The noise level in the targets can be specified by passing it via the parameter  alpha   either globally as a scalar or per datapoint  Note that a moderate noise level can also be helpful for dealing with numeric instabilities during fitting as it is effectively implemented as Tikhonov regularization  i e   by adding it to the diagonal of the kernel matrix  An alternative to specifying the noise level explicitly is to include a  class   sklearn gaussian process kernels WhiteKernel  component into the kernel  which can estimate the global noise level from the data  see example below   The figure below shows the effect of noisy target handled by setting the parameter  alpha       figure      auto examples gaussian process images sphx glr plot gpr noisy targets 003 png     target     auto examples gaussian process plot gpr noisy targets html     align  center  The implementation is based on Algorithm 2 1 of  RW2006    In addition to the API of standard scikit learn estimators   class  GaussianProcessRegressor      allows prediction without prior fitting  based on the GP prior     provides an additional method   sample y X     which evaluates samples   drawn from the GPR  prior or posterior  at given inputs    exposes a method   log marginal likelihood theta     which can be used   externally for other ways of selecting hyperparameters  e g   via   Markov chain Monte Carlo      rubric   Examples     ref  sphx glr auto examples gaussian process plot gpr noisy targets py     ref  sphx glr auto examples gaussian process plot gpr noisy py     ref  sphx glr auto examples gaussian process plot compare gpr krr py     ref  sphx glr auto examples gaussian process plot gpr co2 py       gpc   Gaussian Process Classification  GPC                                            currentmodule   sklearn gaussian process  The  class  GaussianProcessClassifier  implements Gaussian processes  GP  for classification purposes  more specifically for probabilistic classification  where test predictions take the form of class probabilities  GaussianProcessClassifier places a GP prior on a latent function  math  f   which is then squashed through a link function to obtain the probabilistic classification  The latent function  math  f  is a so called nuisance function  whose values are not observed and are not relevant by themselves  Its purpose is to allow a convenient formulation of the model  and  math  f  is removed  integrated out  during prediction  GaussianProcessClassifier implements the logistic link function  for which the integral cannot be computed analytically but is easily approximated in the binary case   In contrast to the regression setting  the posterior of the latent function  math  f  is not Gaussian even for a GP prior since a Gaussian likelihood is inappropriate for discrete class labels  Rather  a non Gaussian likelihood corresponding to the logistic link function  logit  is used  GaussianProcessClassifier approximates the non Gaussian posterior with a Gaussian based on the Laplace approximation  More details can be found in Chapter 3 of  RW2006     The GP prior mean is assumed to be zero  The prior s covariance is specified by passing a  ref  kernel  gp kernels   object  The hyperparameters of the kernel are optimized during fitting of GaussianProcessRegressor by maximizing the log marginal likelihood  LML  based on the passed   optimizer    As the LML may have multiple local optima  the optimizer can be started repeatedly by specifying   n restarts optimizer    The first run is always conducted starting from the initial hyperparameter values of the kernel  subsequent runs are conducted from hyperparameter values that have been chosen randomly from the range of allowed values  If the initial hyperparameters should be kept fixed   None  can be passed as optimizer    class  GaussianProcessClassifier  supports multi class classification by performing either one versus rest or one versus one based training and prediction   In one versus rest  one binary Gaussian process classifier is fitted for each class  which is trained to separate this class from the rest  In  one vs one   one binary Gaussian process classifier is fitted for each pair of classes  which is trained to separate these two classes  The predictions of these binary predictors are combined into multi class predictions  See the section on  ref  multi class classification  multiclass   for more details   In the case of Gaussian process classification   one vs one  might be computationally  cheaper since it has to solve many problems involving only a subset of the whole training set rather than fewer problems on the whole dataset  Since Gaussian process classification scales cubically with the size of the dataset  this might be considerably faster  However  note that  one vs one  does not support predicting probability estimates but only plain predictions  Moreover  note that  class  GaussianProcessClassifier  does not  yet  implement a true multi class Laplace approximation internally  but as discussed above is based on solving several binary classification tasks internally  which are combined using one versus rest or one versus one   GPC examples               Probabilistic predictions with GPC                                     This example illustrates the predicted probability of GPC for an RBF kernel with different choices of the hyperparameters  The first figure shows the predicted probability of GPC with arbitrarily chosen hyperparameters and with the hyperparameters corresponding to the maximum log marginal likelihood  LML    While the hyperparameters chosen by optimizing LML have a considerably larger LML  they perform slightly worse according to the log loss on test data  The figure shows that this is because they exhibit a steep change of the class probabilities at the class boundaries  which is good  but have predicted probabilities close to 0 5 far away from the class boundaries  which is bad  This undesirable effect is caused by the Laplace approximation used internally by GPC   The second figure shows the log marginal likelihood for different choices of the kernel s hyperparameters  highlighting the two choices of the hyperparameters used in the first figure by black dots      figure      auto examples gaussian process images sphx glr plot gpc 001 png     target     auto examples gaussian process plot gpc html     align  center     figure      auto examples gaussian process images sphx glr plot gpc 002 png     target     auto examples gaussian process plot gpc html     align  center   Illustration of GPC on the XOR dataset                                            currentmodule   sklearn gaussian process kernels  This example illustrates GPC on XOR data  Compared are a stationary  isotropic kernel   class  RBF   and a non stationary kernel   class  DotProduct    On this particular dataset  the  class  DotProduct  kernel obtains considerably better results because the class boundaries are linear and coincide with the coordinate axes  In practice  however  stationary kernels such as  class  RBF  often obtain better results      figure      auto examples gaussian process images sphx glr plot gpc xor 001 png     target     auto examples gaussian process plot gpc xor html     align  center     currentmodule   sklearn gaussian process   Gaussian process classification  GPC  on iris dataset                                                        This example illustrates the predicted probability of GPC for an isotropic and anisotropic RBF kernel on a two dimensional version for the iris dataset  This illustrates the applicability of GPC to non binary classification  The anisotropic RBF kernel obtains slightly higher log marginal likelihood by assigning different length scales to the two feature dimensions      figure      auto examples gaussian process images sphx glr plot gpc iris 001 png     target     auto examples gaussian process plot gpc iris html     align  center       gp kernels   Kernels for Gaussian Processes                                   currentmodule   sklearn gaussian process kernels  Kernels  also called  covariance functions  in the context of GPs  are a crucial ingredient of GPs which determine the shape of prior and posterior of the GP  They encode the assumptions on the function being learned by defining the  similarity  of two datapoints combined with the assumption that similar datapoints should have similar target values  Two categories of kernels can be distinguished  stationary kernels depend only on the distance of two datapoints and not on their absolute values  math  k x i  x j   k d x i  x j    and are thus invariant to translations in the input space  while non stationary kernels depend also on the specific values of the datapoints  Stationary kernels can further be subdivided into isotropic and anisotropic kernels  where isotropic kernels are also invariant to rotations in the input space  For more details  we refer to Chapter 4 of  RW2006    For guidance on how to best combine different kernels  we refer to  Duv2014        dropdown   Gaussian Process Kernel API     The main usage of a  class  Kernel  is to compute the GP s covariance between    datapoints  For this  the method     call     of the kernel can be called  This    method can either be used to compute the  auto covariance  of all pairs of    datapoints in a 2d array X  or the  cross covariance  of all combinations    of datapoints of a 2d array X with datapoints in a 2d array Y  The following    identity holds true for all kernels k  except for the  class  WhiteKernel         k X     K X  Y X        If only the diagonal of the auto covariance is being used  the method   diag        of a kernel can be called  which is more computationally efficient than the    equivalent call to     call        np diag k X  X      k diag X        Kernels are parameterized by a vector  math   theta  of hyperparameters  These    hyperparameters can for instance control length scales or periodicity of a    kernel  see below   All kernels support computing analytic gradients    of the kernel s auto covariance with respect to  math  log  theta   via setting      eval gradient True   in the     call     method     That is  a    len X   len X   len theta     array is returned where the entry       i  j  l    contains  math   frac  partial k  theta x i  x j    partial log  theta l        This gradient is used by the Gaussian process  both regressor and classifier     in computing the gradient of the log marginal likelihood  which in turn is used    to determine the value of  math   theta   which maximizes the log marginal likelihood     via gradient ascent  For each hyperparameter  the initial value and the    bounds need to be specified when creating an instance of the kernel  The    current value of  math   theta  can be get and set via the property      theta   of the kernel object  Moreover  the bounds of the hyperparameters can be    accessed by the property   bounds   of the kernel  Note that both properties     theta and bounds  return log transformed values of the internally used values    since those are typically more amenable to gradient based optimization     The specification of each hyperparameter is stored in the form of an instance of     class  Hyperparameter  in the respective kernel  Note that a kernel using a    hyperparameter with name  x  must have the attributes self x and self x bounds      The abstract base class for all kernels is  class  Kernel   Kernel implements a    similar interface as  class   sklearn base BaseEstimator   providing the    methods   get params        set params      and   clone      This allows    setting kernel values also via meta estimators such as     class   sklearn pipeline Pipeline  or     class   sklearn model selection GridSearchCV   Note that due to the nested    structure of kernels  by applying kernel operators  see below   the names of    kernel parameters might become relatively complicated  In general  for a binary    kernel operator  parameters of the left operand are prefixed with   k1     and    parameters of the right operand with   k2      An additional convenience method    is   clone with theta theta     which returns a cloned version of the kernel    but with the hyperparameters set to   theta    An illustrative example             from sklearn gaussian process kernels import ConstantKernel  RBF           kernel   ConstantKernel constant value 1 0  constant value bounds  0 0  10 0     RBF length scale 0 5  length scale bounds  0 0  10 0     RBF length scale 2 0  length scale bounds  0 0  10 0             for hyperparameter in kernel hyperparameters  print hyperparameter        Hyperparameter name  k1  k1  constant value   value type  numeric   bounds array    0   10      n elements 1  fixed False        Hyperparameter name  k1  k2  length scale   value type  numeric   bounds array    0   10      n elements 1  fixed False        Hyperparameter name  k2  length scale   value type  numeric   bounds array    0   10      n elements 1  fixed False            params   kernel get params             for key in sorted params   print   s    s     key  params key          k1   1  2   RBF length scale 0 5        k1  k1   1  2       k1  k1  constant value   1 0       k1  k1  constant value bounds    0 0  10 0        k1  k2   RBF length scale 0 5        k1  k2  length scale   0 5       k1  k2  length scale bounds    0 0  10 0        k2   RBF length scale 2        k2  length scale   2 0       k2  length scale bounds    0 0  10 0            print kernel theta     Note  log transformed         0           0 69314718  0 69314718            print kernel bounds     Note  log transformed                inf 2 30258509                inf 2 30258509                inf 2 30258509       All Gaussian process kernels are interoperable with  mod  sklearn metrics pairwise     and vice versa  instances of subclasses of  class  Kernel  can be passed as      metric   to   pairwise kernels   from  mod  sklearn metrics pairwise   Moreover     kernel functions from pairwise can be used as GP kernels by using the wrapper    class  class  PairwiseKernel   The only caveat is that the gradient of    the hyperparameters is not analytic but numeric and all those kernels support    only isotropic distances  The parameter   gamma   is considered to be a    hyperparameter and may be optimized  The other kernel parameters are set    directly at initialization and are kept fixed   Basic kernels               The  class  ConstantKernel  kernel can be used as part of a  class  Product  kernel where it scales the magnitude of the other factor  kernel  or as part of a  class  Sum  kernel  where it modifies the mean of the Gaussian process  It depends on a parameter  math  constant  value   It is defined as      math      k x i  x j    constant  value    forall   x 1  x 2  The main use case of the  class  WhiteKernel  kernel is as part of a sum kernel where it explains the noise component of the signal  Tuning its parameter  math  noise  level  corresponds to estimating the noise level  It is defined as      math       k x i  x j    noise  level  text  if   x i    x j  text  else   0   Kernel operators                  Kernel operators take one or two base kernels and combine them into a new kernel  The  class  Sum  kernel takes two kernels  math  k 1  and  math  k 2  and combines them via  math  k  sum  X  Y    k 1 X  Y    k 2 X  Y    The   class  Product  kernel takes two kernels  math  k 1  and  math  k 2  and combines them via  math  k  product  X  Y    k 1 X  Y    k 2 X  Y    The  class  Exponentiation  kernel takes one base kernel and a scalar parameter  math  p  and combines them via  math  k  exp  X  Y    k X  Y  p   Note that magic methods     add          mul      and     pow     are overridden on the Kernel objects  so one can use e g    RBF     RBF     as a shortcut for   Sum RBF    RBF        Radial basis function  RBF  kernel                                    The  class  RBF  kernel is a stationary kernel  It is also known as the  squared exponential  kernel  It is parameterized by a length scale parameter  math  l 0   which can either be a scalar  isotropic variant of the kernel  or a vector with the same number of dimensions as the inputs  math  x   anisotropic variant of the kernel   The kernel is given by      math      k x i  x j     text exp  left    frac d x i  x j  2  2l 2   right   where  math  d  cdot   cdot   is the Euclidean distance  This kernel is infinitely differentiable  which implies that GPs with this kernel as covariance function have mean square derivatives of all orders  and are thus very smooth  The prior and posterior of a GP resulting from an RBF kernel are shown in the following figure      figure      auto examples gaussian process images sphx glr plot gpr prior posterior 001 png     target     auto examples gaussian process plot gpr prior posterior html     align  center   Mat rn kernel               The  class  Matern  kernel is a stationary kernel and a generalization of the  class  RBF  kernel  It has an additional parameter  math   nu  which controls the smoothness of the resulting function  It is parameterized by a length scale parameter  math  l 0   which can either be a scalar  isotropic variant of the kernel  or a vector with the same number of dimensions as the inputs  math  x   anisotropic variant of the kernel       dropdown   Mathematical implementation of Mat rn kernel     The kernel is given by         math          k x i  x j     frac 1   Gamma  nu 2   nu 1   Bigg  frac  sqrt 2 nu   l  d x i   x j   Bigg   nu K  nu Bigg  frac  sqrt 2 nu   l  d x i   x j   Bigg       where  math  d  cdot  cdot   is the Euclidean distance   math  K  nu  cdot   is a modified Bessel function and  math   Gamma  cdot   is the gamma function     As  math   nu rightarrow infty   the Mat rn kernel converges to the RBF kernel     When  math   nu   1 2   the Mat rn kernel becomes identical to the absolute    exponential kernel  i e          math         k x i  x j     exp  Bigg    frac 1  l  d x i   x j    Bigg   quad  quad  nu   tfrac 1  2      In particular   math   nu   3 2          math         k x i  x j      Bigg 1    frac  sqrt 3   l  d x i   x j   Bigg   exp  Bigg   frac  sqrt 3   l  d x i   x j    Bigg   quad  quad  nu   tfrac 3  2      and  math   nu   5 2          math         k x i  x j     Bigg 1    frac  sqrt 5   l  d x i   x j     frac 5  3l  d x i   x j   2  Bigg   exp  Bigg   frac  sqrt 5   l  d x i   x j    Bigg   quad  quad  nu   tfrac 5  2      are popular choices for learning functions that are not infinitely    differentiable  as assumed by the RBF kernel  but at least once   math   nu      3 2   or twice differentiable   math   nu   5 2        The flexibility of controlling the smoothness of the learned function via  math   nu     allows adapting to the properties of the true underlying functional relation   The prior and posterior of a GP resulting from a Mat rn kernel are shown in the following figure      figure      auto examples gaussian process images sphx glr plot gpr prior posterior 005 png     target     auto examples gaussian process plot gpr prior posterior html     align  center  See  RW2006    pp84 for further details regarding the different variants of the Mat rn kernel   Rational quadratic kernel                            The  class  RationalQuadratic  kernel can be seen as a scale mixture  an infinite sum  of  class  RBF  kernels with different characteristic length scales  It is parameterized by a length scale parameter  math  l 0  and a scale mixture parameter   math   alpha 0  Only the isotropic variant where  math  l  is a scalar is supported at the moment  The kernel is given by      math      k x i  x j     left 1    frac d x i  x j  2  2 alpha l 2  right     alpha   The prior and posterior of a GP resulting from a  class  RationalQuadratic  kernel are shown in the following figure      figure      auto examples gaussian process images sphx glr plot gpr prior posterior 002 png     target     auto examples gaussian process plot gpr prior posterior html     align  center  Exp Sine Squared kernel                          The  class  ExpSineSquared  kernel allows modeling periodic functions  It is parameterized by a length scale parameter  math  l 0  and a periodicity parameter  math  p 0   Only the isotropic variant where  math  l  is a scalar is supported at the moment  The kernel is given by      math      k x i  x j     text exp  left    frac  2 sin 2  pi d x i  x j    p     l  2   right   The prior and posterior of a GP resulting from an ExpSineSquared kernel are shown in the following figure      figure      auto examples gaussian process images sphx glr plot gpr prior posterior 003 png     target     auto examples gaussian process plot gpr prior posterior html     align  center  Dot Product kernel                     The  class  DotProduct  kernel is non stationary and can be obtained from linear regression by putting  math  N 0  1   priors on the coefficients of  math  x d  d   1          D   and a prior of  math  N 0   sigma 0 2   on the bias  The  class  DotProduct  kernel is invariant to a rotation of the coordinates about the origin  but not translations  It is parameterized by a parameter  math   sigma 0 2   For  math   sigma 0 2   0   the kernel is called the homogeneous linear kernel  otherwise it is inhomogeneous  The kernel is given by     math      k x i  x j     sigma 0   2   x i  cdot x j  The  class  DotProduct  kernel is commonly combined with exponentiation  An example with exponent 2 is shown in the following figure      figure      auto examples gaussian process images sphx glr plot gpr prior posterior 004 png     target     auto examples gaussian process plot gpr prior posterior html     align  center  References                 RW2006   Carl E  Rasmussen and Christopher K I  Williams      Gaussian Processes for Machine Learning      MIT Press 2006  https   www gaussianprocess org gpml chapters RW pdf         Duv2014   David Duvenaud   The Kernel Cookbook  Advice on Covariance functions   2014     https   www cs toronto edu  duvenaud cookbook         currentmodule   sklearn gaussian process


scikit-learn sklearn and Quadratic Linear and Quadratic Discriminant Analysis ldaqda Linear Discriminant Analysis

Answer:
.. _lda_qda:

==========================================
Linear and Quadratic Discriminant Analysis
==========================================

.. currentmodule:: sklearn

Linear Discriminant Analysis
(:class:`~discriminant_analysis.LinearDiscriminantAnalysis`) and Quadratic
Discriminant Analysis
(:class:`~discriminant_analysis.QuadraticDiscriminantAnalysis`) are two classic
classifiers, with, as their names suggest, a linear and a quadratic decision
surface, respectively.

These classifiers are attractive because they have closed-form solutions that
can be easily computed, are inherently multiclass, have proven to work well in
practice, and have no hyperparameters to tune.

.. |ldaqda| image:: ../auto_examples/classification/images/sphx_glr_plot_lda_qda_001.png
        :target: ../auto_examples/classification/plot_lda_qda.html
        :scale: 80

.. centered:: |ldaqda|

The plot shows decision boundaries for Linear Discriminant Analysis and
Quadratic Discriminant Analysis. The bottom row demonstrates that Linear
Discriminant Analysis can only learn linear boundaries, while Quadratic
Discriminant Analysis can learn quadratic boundaries and is therefore more
flexible.

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_classification_plot_lda_qda.py`: Comparison of LDA and
  QDA on synthetic data.

Dimensionality reduction using Linear Discriminant Analysis
===========================================================

:class:`~discriminant_analysis.LinearDiscriminantAnalysis` can be used to
perform supervised dimensionality reduction, by projecting the input data to a
linear subspace consisting of the directions which maximize the separation
between classes (in a precise sense discussed in the mathematics section
below). The dimension of the output is necessarily less than the number of
classes, so this is in general a rather strong dimensionality reduction, and
only makes sense in a multiclass setting.

This is implemented in the `transform` method. The desired dimensionality can
be set using the ``n_components`` parameter. This parameter has no influence
on the `fit` and `predict` methods.

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_decomposition_plot_pca_vs_lda.py`: Comparison of LDA and
  PCA for dimensionality reduction of the Iris dataset

.. _lda_qda_math:

Mathematical formulation of the LDA and QDA classifiers
=======================================================

Both LDA and QDA can be derived from simple probabilistic models which model
the class conditional distribution of the data :math:`P(X|y=k)` for each class
:math:`k`. Predictions can then be obtained by using Bayes' rule, for each
training sample :math:`x \in \mathcal{R}^d`:

.. math::
    P(y=k | x) = \frac{P(x | y=k) P(y=k)}{P(x)} = \frac{P(x | y=k) P(y = k)}{ \sum_{l} P(x | y=l) \cdot P(y=l)}

and we select the class :math:`k` which maximizes this posterior probability.

More specifically, for linear and quadratic discriminant analysis,
:math:`P(x|y)` is modeled as a multivariate Gaussian distribution with
density:

.. math:: P(x | y=k) = \frac{1}{(2\pi)^{d/2} |\Sigma_k|^{1/2}}\exp\left(-\frac{1}{2} (x-\mu_k)^t \Sigma_k^{-1} (x-\mu_k)\right)

where :math:`d` is the number of features.

QDA
---

According to the model above, the log of the posterior is:

.. math::

    \log P(y=k | x) &= \log P(x | y=k) + \log P(y = k) + Cst \\
    &= -\frac{1}{2} \log |\Sigma_k| -\frac{1}{2} (x-\mu_k)^t \Sigma_k^{-1} (x-\mu_k) + \log P(y = k) + Cst,

where the constant term :math:`Cst` corresponds to the denominator
:math:`P(x)`, in addition to other constant terms from the Gaussian. The
predicted class is the one that maximises this log-posterior.

.. note:: **Relation with Gaussian Naive Bayes**

	  If in the QDA model one assumes that the covariance matrices are diagonal,
	  then the inputs are assumed to be conditionally independent in each class,
	  and the resulting classifier is equivalent to the Gaussian Naive Bayes
	  classifier :class:`naive_bayes.GaussianNB`.

LDA
---

LDA is a special case of QDA, where the Gaussians for each class are assumed
to share the same covariance matrix: :math:`\Sigma_k = \Sigma` for all
:math:`k`. This reduces the log posterior to:

.. math:: \log P(y=k | x) = -\frac{1}{2} (x-\mu_k)^t \Sigma^{-1} (x-\mu_k) + \log P(y = k) + Cst.

The term :math:`(x-\mu_k)^t \Sigma^{-1} (x-\mu_k)` corresponds to the
`Mahalanobis Distance <https://en.wikipedia.org/wiki/Mahalanobis_distance>`_
between the sample :math:`x` and the mean :math:`\mu_k`. The Mahalanobis
distance tells how close :math:`x` is from :math:`\mu_k`, while also
accounting for the variance of each feature. We can thus interpret LDA as
assigning :math:`x` to the class whose mean is the closest in terms of
Mahalanobis distance, while also accounting for the class prior
probabilities.

The log-posterior of LDA can also be written [3]_ as:

.. math::

    \log P(y=k | x) = \omega_k^t x + \omega_{k0} + Cst.

where :math:`\omega_k = \Sigma^{-1} \mu_k` and :math:`\omega_{k0} =
-\frac{1}{2} \mu_k^t\Sigma^{-1}\mu_k + \log P (y = k)`. These quantities
correspond to the `coef_` and `intercept_` attributes, respectively.

From the above formula, it is clear that LDA has a linear decision surface.
In the case of QDA, there are no assumptions on the covariance matrices
:math:`\Sigma_k` of the Gaussians, leading to quadratic decision surfaces.
See [1]_ for more details.

Mathematical formulation of LDA dimensionality reduction
========================================================

First note that the K means :math:`\mu_k` are vectors in
:math:`\mathcal{R}^d`, and they lie in an affine subspace :math:`H` of
dimension at most :math:`K - 1` (2 points lie on a line, 3 points lie on a
plane, etc.).

As mentioned above, we can interpret LDA as assigning :math:`x` to the class
whose mean :math:`\mu_k` is the closest in terms of Mahalanobis distance,
while also accounting for the class prior probabilities. Alternatively, LDA
is equivalent to first *sphering* the data so that the covariance matrix is
the identity, and then assigning :math:`x` to the closest mean in terms of
Euclidean distance (still accounting for the class priors).

Computing Euclidean distances in this d-dimensional space is equivalent to
first projecting the data points into :math:`H`, and computing the distances
there (since the other dimensions will contribute equally to each class in
terms of distance). In other words, if :math:`x` is closest to :math:`\mu_k`
in the original space, it will also be the case in :math:`H`.
This shows that, implicit in the LDA
classifier, there is a dimensionality reduction by linear projection onto a
:math:`K-1` dimensional space.

We can reduce the dimension even more, to a chosen :math:`L`, by projecting
onto the linear subspace :math:`H_L` which maximizes the variance of the
:math:`\mu^*_k` after projection (in effect, we are doing a form of PCA for the
transformed class means :math:`\mu^*_k`). This :math:`L` corresponds to the
``n_components`` parameter used in the
:func:`~discriminant_analysis.LinearDiscriminantAnalysis.transform` method. See
[1]_ for more details.

Shrinkage and Covariance Estimator
==================================

Shrinkage is a form of regularization used to improve the estimation of
covariance matrices in situations where the number of training samples is
small compared to the number of features.
In this scenario, the empirical sample covariance is a poor
estimator, and shrinkage helps improving the generalization performance of
the classifier.
Shrinkage LDA can be used by setting the ``shrinkage`` parameter of
the :class:`~discriminant_analysis.LinearDiscriminantAnalysis` class to 'auto'.
This automatically determines the optimal shrinkage parameter in an analytic
way following the lemma introduced by Ledoit and Wolf [2]_. Note that
currently shrinkage only works when setting the ``solver`` parameter to 'lsqr'
or 'eigen'.

The ``shrinkage`` parameter can also be manually set between 0 and 1. In
particular, a value of 0 corresponds to no shrinkage (which means the empirical
covariance matrix will be used) and a value of 1 corresponds to complete
shrinkage (which means that the diagonal matrix of variances will be used as
an estimate for the covariance matrix). Setting this parameter to a value
between these two extrema will estimate a shrunk version of the covariance
matrix.

The shrunk Ledoit and Wolf estimator of covariance may not always be the
best choice. For example if the distribution of the data
is normally distributed, the
Oracle Approximating Shrinkage estimator :class:`sklearn.covariance.OAS`
yields a smaller Mean Squared Error than the one given by Ledoit and Wolf's
formula used with shrinkage="auto". In LDA, the data are assumed to be gaussian
conditionally to the class. If these assumptions hold, using LDA with
the OAS estimator of covariance will yield a better classification
accuracy than if Ledoit and Wolf or the empirical covariance estimator is used.

The covariance estimator can be chosen using with the ``covariance_estimator``
parameter of the :class:`discriminant_analysis.LinearDiscriminantAnalysis`
class. A covariance estimator should have a :term:`fit` method and a
``covariance_`` attribute like all covariance estimators in the
:mod:`sklearn.covariance` module.

.. |shrinkage| image:: ../auto_examples/classification/images/sphx_glr_plot_lda_001.png
        :target: ../auto_examples/classification/plot_lda.html
        :scale: 75

.. centered:: |shrinkage|

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_classification_plot_lda.py`: Comparison of LDA classifiers
  with Empirical, Ledoit Wolf and OAS covariance estimator.

Estimation algorithms
=====================

Using LDA and QDA requires computing the log-posterior which depends on the
class priors :math:`P(y=k)`, the class means :math:`\mu_k`, and the
covariance matrices.

The 'svd' solver is the default solver used for
:class:`~sklearn.discriminant_analysis.LinearDiscriminantAnalysis`, and it is
the only available solver for
:class:`~sklearn.discriminant_analysis.QuadraticDiscriminantAnalysis`.
It can perform both classification and transform (for LDA).
As it does not rely on the calculation of the covariance matrix, the 'svd'
solver may be preferable in situations where the number of features is large.
The 'svd' solver cannot be used with shrinkage.
For QDA, the use of the SVD solver relies on the fact that the covariance
matrix :math:`\Sigma_k` is, by definition, equal to :math:`\frac{1}{n - 1}
X_k^tX_k = \frac{1}{n - 1} V S^2 V^t` where :math:`V` comes from the SVD of the (centered)
matrix: :math:`X_k = U S V^t`. It turns out that we can compute the
log-posterior above without having to explicitly compute :math:`\Sigma`:
computing :math:`S` and :math:`V` via the SVD of :math:`X` is enough. For
LDA, two SVDs are computed: the SVD of the centered input matrix :math:`X`
and the SVD of the class-wise mean vectors.

The 'lsqr' solver is an efficient algorithm that only works for
classification. It needs to explicitly compute the covariance matrix
:math:`\Sigma`, and supports shrinkage and custom covariance estimators.
This solver computes the coefficients
:math:`\omega_k = \Sigma^{-1}\mu_k` by solving for :math:`\Sigma \omega =
\mu_k`, thus avoiding the explicit computation of the inverse
:math:`\Sigma^{-1}`.

The 'eigen' solver is based on the optimization of the between class scatter to
within class scatter ratio. It can be used for both classification and
transform, and it supports shrinkage. However, the 'eigen' solver needs to
compute the covariance matrix, so it might not be suitable for situations with
a high number of features.

.. rubric:: References

.. [1] "The Elements of Statistical Learning", Hastie T., Tibshirani R.,
    Friedman J., Section 4.3, p.106-119, 2008.

.. [2] Ledoit O, Wolf M. Honey, I Shrunk the Sample Covariance Matrix.
    The Journal of Portfolio Management 30(4), 110-119, 2004.

.. [3] R. O. Duda, P. E. Hart, D. G. Stork. Pattern Classification
    (Second Edition), section 2.6.2.

lda qda                                              Linear and Quadratic Discriminant Analysis                                                currentmodule   sklearn  Linear Discriminant Analysis   class   discriminant analysis LinearDiscriminantAnalysis   and Quadratic Discriminant Analysis   class   discriminant analysis QuadraticDiscriminantAnalysis   are two classic classifiers  with  as their names suggest  a linear and a quadratic decision surface  respectively   These classifiers are attractive because they have closed form solutions that can be easily computed  are inherently multiclass  have proven to work well in practice  and have no hyperparameters to tune       ldaqda  image      auto examples classification images sphx glr plot lda qda 001 png          target     auto examples classification plot lda qda html          scale  80     centered    ldaqda   The plot shows decision boundaries for Linear Discriminant Analysis and Quadratic Discriminant Analysis  The bottom row demonstrates that Linear Discriminant Analysis can only learn linear boundaries  while Quadratic Discriminant Analysis can learn quadratic boundaries and is therefore more flexible      rubric   Examples     ref  sphx glr auto examples classification plot lda qda py   Comparison of LDA and   QDA on synthetic data   Dimensionality reduction using Linear Discriminant Analysis                                                               class   discriminant analysis LinearDiscriminantAnalysis  can be used to perform supervised dimensionality reduction  by projecting the input data to a linear subspace consisting of the directions which maximize the separation between classes  in a precise sense discussed in the mathematics section below   The dimension of the output is necessarily less than the number of classes  so this is in general a rather strong dimensionality reduction  and only makes sense in a multiclass setting   This is implemented in the  transform  method  The desired dimensionality can be set using the   n components   parameter  This parameter has no influence on the  fit  and  predict  methods      rubric   Examples     ref  sphx glr auto examples decomposition plot pca vs lda py   Comparison of LDA and   PCA for dimensionality reduction of the Iris dataset      lda qda math   Mathematical formulation of the LDA and QDA classifiers                                                          Both LDA and QDA can be derived from simple probabilistic models which model the class conditional distribution of the data  math  P X y k   for each class  math  k   Predictions can then be obtained by using Bayes  rule  for each training sample  math  x  in  mathcal R  d       math       P y k   x     frac P x   y k  P y k   P x      frac P x   y k  P y   k     sum  l  P x   y l   cdot P y l    and we select the class  math  k  which maximizes this posterior probability   More specifically  for linear and quadratic discriminant analysis   math  P x y   is modeled as a multivariate Gaussian distribution with density      math   P x   y k     frac 1   2 pi   d 2    Sigma k   1 2   exp left   frac 1  2   x  mu k  t  Sigma k   1   x  mu k  right   where  math  d  is the number of features   QDA      According to the model above  the log of the posterior is      math         log P y k   x      log P x   y k     log P y   k    Cst             frac 1  2   log   Sigma k    frac 1  2   x  mu k  t  Sigma k   1   x  mu k     log P y   k    Cst   where the constant term  math  Cst  corresponds to the denominator  math  P x    in addition to other constant terms from the Gaussian  The predicted class is the one that maximises this log posterior      note     Relation with Gaussian Naive Bayes       If in the QDA model one assumes that the covariance matrices are diagonal     then the inputs are assumed to be conditionally independent in each class     and the resulting classifier is equivalent to the Gaussian Naive Bayes    classifier  class  naive bayes GaussianNB    LDA      LDA is a special case of QDA  where the Gaussians for each class are assumed to share the same covariance matrix   math   Sigma k    Sigma  for all  math  k   This reduces the log posterior to      math    log P y k   x      frac 1  2   x  mu k  t  Sigma   1   x  mu k     log P y   k    Cst   The term  math   x  mu k  t  Sigma   1   x  mu k   corresponds to the  Mahalanobis Distance  https   en wikipedia org wiki Mahalanobis distance    between the sample  math  x  and the mean  math   mu k   The Mahalanobis distance tells how close  math  x  is from  math   mu k   while also accounting for the variance of each feature  We can thus interpret LDA as assigning  math  x  to the class whose mean is the closest in terms of Mahalanobis distance  while also accounting for the class prior probabilities   The log posterior of LDA can also be written  3   as      math         log P y k   x     omega k t x    omega  k0    Cst   where  math   omega k    Sigma   1   mu k  and  math   omega  k0      frac 1  2   mu k t Sigma   1  mu k    log P  y   k    These quantities correspond to the  coef   and  intercept   attributes  respectively   From the above formula  it is clear that LDA has a linear decision surface  In the case of QDA  there are no assumptions on the covariance matrices  math   Sigma k  of the Gaussians  leading to quadratic decision surfaces  See  1   for more details   Mathematical formulation of LDA dimensionality reduction                                                           First note that the K means  math   mu k  are vectors in  math   mathcal R  d   and they lie in an affine subspace  math  H  of dimension at most  math  K   1   2 points lie on a line  3 points lie on a plane  etc     As mentioned above  we can interpret LDA as assigning  math  x  to the class whose mean  math   mu k  is the closest in terms of Mahalanobis distance  while also accounting for the class prior probabilities  Alternatively  LDA is equivalent to first  sphering  the data so that the covariance matrix is the identity  and then assigning  math  x  to the closest mean in terms of Euclidean distance  still accounting for the class priors    Computing Euclidean distances in this d dimensional space is equivalent to first projecting the data points into  math  H   and computing the distances there  since the other dimensions will contribute equally to each class in terms of distance   In other words  if  math  x  is closest to  math   mu k  in the original space  it will also be the case in  math  H   This shows that  implicit in the LDA classifier  there is a dimensionality reduction by linear projection onto a  math  K 1  dimensional space   We can reduce the dimension even more  to a chosen  math  L   by projecting onto the linear subspace  math  H L  which maximizes the variance of the  math   mu   k  after projection  in effect  we are doing a form of PCA for the transformed class means  math   mu   k    This  math  L  corresponds to the   n components   parameter used in the  func   discriminant analysis LinearDiscriminantAnalysis transform  method  See  1   for more details   Shrinkage and Covariance Estimator                                     Shrinkage is a form of regularization used to improve the estimation of covariance matrices in situations where the number of training samples is small compared to the number of features  In this scenario  the empirical sample covariance is a poor estimator  and shrinkage helps improving the generalization performance of the classifier  Shrinkage LDA can be used by setting the   shrinkage   parameter of the  class   discriminant analysis LinearDiscriminantAnalysis  class to  auto   This automatically determines the optimal shrinkage parameter in an analytic way following the lemma introduced by Ledoit and Wolf  2    Note that currently shrinkage only works when setting the   solver   parameter to  lsqr  or  eigen    The   shrinkage   parameter can also be manually set between 0 and 1  In particular  a value of 0 corresponds to no shrinkage  which means the empirical covariance matrix will be used  and a value of 1 corresponds to complete shrinkage  which means that the diagonal matrix of variances will be used as an estimate for the covariance matrix   Setting this parameter to a value between these two extrema will estimate a shrunk version of the covariance matrix   The shrunk Ledoit and Wolf estimator of covariance may not always be the best choice  For example if the distribution of the data is normally distributed  the Oracle Approximating Shrinkage estimator  class  sklearn covariance OAS  yields a smaller Mean Squared Error than the one given by Ledoit and Wolf s formula used with shrinkage  auto   In LDA  the data are assumed to be gaussian conditionally to the class  If these assumptions hold  using LDA with the OAS estimator of covariance will yield a better classification accuracy than if Ledoit and Wolf or the empirical covariance estimator is used   The covariance estimator can be chosen using with the   covariance estimator   parameter of the  class  discriminant analysis LinearDiscriminantAnalysis  class  A covariance estimator should have a  term  fit  method and a   covariance    attribute like all covariance estimators in the  mod  sklearn covariance  module        shrinkage  image      auto examples classification images sphx glr plot lda 001 png          target     auto examples classification plot lda html          scale  75     centered    shrinkage      rubric   Examples     ref  sphx glr auto examples classification plot lda py   Comparison of LDA classifiers   with Empirical  Ledoit Wolf and OAS covariance estimator   Estimation algorithms                        Using LDA and QDA requires computing the log posterior which depends on the class priors  math  P y k    the class means  math   mu k   and the covariance matrices   The  svd  solver is the default solver used for  class   sklearn discriminant analysis LinearDiscriminantAnalysis   and it is the only available solver for  class   sklearn discriminant analysis QuadraticDiscriminantAnalysis   It can perform both classification and transform  for LDA   As it does not rely on the calculation of the covariance matrix  the  svd  solver may be preferable in situations where the number of features is large  The  svd  solver cannot be used with shrinkage  For QDA  the use of the SVD solver relies on the fact that the covariance matrix  math   Sigma k  is  by definition  equal to  math   frac 1  n   1  X k tX k    frac 1  n   1  V S 2 V t  where  math  V  comes from the SVD of the  centered  matrix   math  X k   U S V t   It turns out that we can compute the log posterior above without having to explicitly compute  math   Sigma   computing  math  S  and  math  V  via the SVD of  math  X  is enough  For LDA  two SVDs are computed  the SVD of the centered input matrix  math  X  and the SVD of the class wise mean vectors   The  lsqr  solver is an efficient algorithm that only works for classification  It needs to explicitly compute the covariance matrix  math   Sigma   and supports shrinkage and custom covariance estimators  This solver computes the coefficients  math   omega k    Sigma   1  mu k  by solving for  math   Sigma  omega    mu k   thus avoiding the explicit computation of the inverse  math   Sigma   1     The  eigen  solver is based on the optimization of the between class scatter to within class scatter ratio  It can be used for both classification and transform  and it supports shrinkage  However  the  eigen  solver needs to compute the covariance matrix  so it might not be suitable for situations with a high number of features      rubric   References      1   The Elements of Statistical Learning   Hastie T   Tibshirani R       Friedman J   Section 4 3  p 106 119  2008       2  Ledoit O  Wolf M  Honey  I Shrunk the Sample Covariance Matrix      The Journal of Portfolio Management 30 4   110 119  2004       3  R  O  Duda  P  E  Hart  D  G  Stork  Pattern Classification      Second Edition   section 2 6 2


scikit-learn semisupervised Semi supervised learning sklearn semisupervised https en wikipedia org wiki Semi supervisedlearning is a situation

Answer:
.. _semi_supervised:

===================================================
Semi-supervised learning
===================================================

.. currentmodule:: sklearn.semi_supervised

`Semi-supervised learning
<https://en.wikipedia.org/wiki/Semi-supervised_learning>`_ is a situation
in which in your training data some of the samples are not labeled. The
semi-supervised estimators in :mod:`sklearn.semi_supervised` are able to
make use of this additional unlabeled data to better capture the shape of
the underlying data distribution and generalize better to new samples.
These algorithms can perform well when we have a very small amount of
labeled points and a large amount of unlabeled points.

.. topic:: Unlabeled entries in `y`

   It is important to assign an identifier to unlabeled points along with the
   labeled data when training the model with the ``fit`` method. The
   identifier that this implementation uses is the integer value :math:`-1`.
   Note that for string labels, the dtype of `y` should be object so that it
   can contain both strings and integers.

.. note::

   Semi-supervised algorithms need to make assumptions about the distribution
   of the dataset in order to achieve performance gains. See `here
   <https://en.wikipedia.org/wiki/Semi-supervised_learning#Assumptions>`_
   for more details.

.. _self_training:

Self Training
=============

This self-training implementation is based on Yarowsky's [1]_ algorithm. Using
this algorithm, a given supervised classifier can function as a semi-supervised
classifier, allowing it to learn from unlabeled data.

:class:`SelfTrainingClassifier` can be called with any classifier that
implements `predict_proba`, passed as the parameter `base_classifier`. In
each iteration, the `base_classifier` predicts labels for the unlabeled
samples and adds a subset of these labels to the labeled dataset.

The choice of this subset is determined by the selection criterion. This
selection can be done using a `threshold` on the prediction probabilities, or
by choosing the `k_best` samples according to the prediction probabilities.

The labels used for the final fit as well as the iteration in which each sample
was labeled are available as attributes. The optional `max_iter` parameter
specifies how many times the loop is executed at most.

The `max_iter` parameter may be set to `None`, causing the algorithm to iterate
until all samples have labels or no new samples are selected in that iteration.

.. note::

   When using the self-training classifier, the
   :ref:`calibration <calibration>` of the classifier is important.

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_semi_supervised_plot_self_training_varying_threshold.py`
* :ref:`sphx_glr_auto_examples_semi_supervised_plot_semi_supervised_versus_svm_iris.py`

.. rubric:: References

.. [1] :doi:`"Unsupervised word sense disambiguation rivaling supervised methods"
    <10.3115/981658.981684>`
    David Yarowsky, Proceedings of the 33rd annual meeting on Association for
    Computational Linguistics (ACL '95). Association for Computational Linguistics,
    Stroudsburg, PA, USA, 189-196.

.. _label_propagation:

Label Propagation
=================

Label propagation denotes a few variations of semi-supervised graph
inference algorithms.

A few features available in this model:
  * Used for classification tasks
  * Kernel methods to project data into alternate dimensional spaces

`scikit-learn` provides two label propagation models:
:class:`LabelPropagation` and :class:`LabelSpreading`. Both work by
constructing a similarity graph over all items in the input dataset.

.. figure:: ../auto_examples/semi_supervised/images/sphx_glr_plot_label_propagation_structure_001.png
    :target: ../auto_examples/semi_supervised/plot_label_propagation_structure.html
    :align: center
    :scale: 60%

    **An illustration of label-propagation:** *the structure of unlabeled
    observations is consistent with the class structure, and thus the
    class label can be propagated to the unlabeled observations of the
    training set.*

:class:`LabelPropagation` and :class:`LabelSpreading`
differ in modifications to the similarity matrix that graph and the
clamping effect on the label distributions.
Clamping allows the algorithm to change the weight of the true ground labeled
data to some degree. The :class:`LabelPropagation` algorithm performs hard
clamping of input labels, which means :math:`\alpha=0`. This clamping factor
can be relaxed, to say :math:`\alpha=0.2`, which means that we will always
retain 80 percent of our original label distribution, but the algorithm gets to
change its confidence of the distribution within 20 percent.

:class:`LabelPropagation` uses the raw similarity matrix constructed from
the data with no modifications. In contrast, :class:`LabelSpreading`
minimizes a loss function that has regularization properties, as such it
is often more robust to noise. The algorithm iterates on a modified
version of the original graph and normalizes the edge weights by
computing the normalized graph Laplacian matrix. This procedure is also
used in :ref:`spectral_clustering`.

Label propagation models have two built-in kernel methods. Choice of kernel
effects both scalability and performance of the algorithms. The following are
available:

* rbf (:math:`\exp(-\gamma |x-y|^2), \gamma > 0`). :math:`\gamma` is
  specified by keyword gamma.

* knn (:math:`1[x' \in kNN(x)]`). :math:`k` is specified by keyword
  n_neighbors.

The RBF kernel will produce a fully connected graph which is represented in memory
by a dense matrix. This matrix may be very large and combined with the cost of
performing a full matrix multiplication calculation for each iteration of the
algorithm can lead to prohibitively long running times. On the other hand,
the KNN kernel will produce a much more memory-friendly sparse matrix
which can drastically reduce running times.

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_semi_supervised_plot_semi_supervised_versus_svm_iris.py`
* :ref:`sphx_glr_auto_examples_semi_supervised_plot_label_propagation_structure.py`
* :ref:`sphx_glr_auto_examples_semi_supervised_plot_label_propagation_digits.py`
* :ref:`sphx_glr_auto_examples_semi_supervised_plot_label_propagation_digits_active_learning.py`

.. rubric:: References

[2] Yoshua Bengio, Olivier Delalleau, Nicolas Le Roux. In Semi-Supervised
Learning (2006), pp. 193-216

[3] Olivier Delalleau, Yoshua Bengio, Nicolas Le Roux. Efficient
Non-Parametric Function Induction in Semi-Supervised Learning. AISTAT 2005
https://www.gatsby.ucl.ac.uk/aistats/fullpapers/204.pdf

semi supervised                                                       Semi supervised learning                                                         currentmodule   sklearn semi supervised   Semi supervised learning  https   en wikipedia org wiki Semi supervised learning    is a situation in which in your training data some of the samples are not labeled  The semi supervised estimators in  mod  sklearn semi supervised  are able to make use of this additional unlabeled data to better capture the shape of the underlying data distribution and generalize better to new samples  These algorithms can perform well when we have a very small amount of labeled points and a large amount of unlabeled points      topic   Unlabeled entries in  y      It is important to assign an identifier to unlabeled points along with the    labeled data when training the model with the   fit   method  The    identifier that this implementation uses is the integer value  math   1      Note that for string labels  the dtype of  y  should be object so that it    can contain both strings and integers      note       Semi supervised algorithms need to make assumptions about the distribution    of the dataset in order to achieve performance gains  See  here     https   en wikipedia org wiki Semi supervised learning Assumptions       for more details       self training   Self Training                This self training implementation is based on Yarowsky s  1   algorithm  Using this algorithm  a given supervised classifier can function as a semi supervised classifier  allowing it to learn from unlabeled data    class  SelfTrainingClassifier  can be called with any classifier that implements  predict proba   passed as the parameter  base classifier   In each iteration  the  base classifier  predicts labels for the unlabeled samples and adds a subset of these labels to the labeled dataset   The choice of this subset is determined by the selection criterion  This selection can be done using a  threshold  on the prediction probabilities  or by choosing the  k best  samples according to the prediction probabilities   The labels used for the final fit as well as the iteration in which each sample was labeled are available as attributes  The optional  max iter  parameter specifies how many times the loop is executed at most   The  max iter  parameter may be set to  None   causing the algorithm to iterate until all samples have labels or no new samples are selected in that iteration      note       When using the self training classifier  the     ref  calibration  calibration   of the classifier is important      rubric   Examples     ref  sphx glr auto examples semi supervised plot self training varying threshold py     ref  sphx glr auto examples semi supervised plot semi supervised versus svm iris py      rubric   References      1   doi   Unsupervised word sense disambiguation rivaling supervised methods       10 3115 981658 981684       David Yarowsky  Proceedings of the 33rd annual meeting on Association for     Computational Linguistics  ACL  95   Association for Computational Linguistics      Stroudsburg  PA  USA  189 196       label propagation   Label Propagation                    Label propagation denotes a few variations of semi supervised graph inference algorithms   A few features available in this model      Used for classification tasks     Kernel methods to project data into alternate dimensional spaces   scikit learn  provides two label propagation models   class  LabelPropagation  and  class  LabelSpreading   Both work by constructing a similarity graph over all items in the input dataset      figure      auto examples semi supervised images sphx glr plot label propagation structure 001 png      target     auto examples semi supervised plot label propagation structure html      align  center      scale  60         An illustration of label propagation     the structure of unlabeled     observations is consistent with the class structure  and thus the     class label can be propagated to the unlabeled observations of the     training set     class  LabelPropagation  and  class  LabelSpreading  differ in modifications to the similarity matrix that graph and the clamping effect on the label distributions  Clamping allows the algorithm to change the weight of the true ground labeled data to some degree  The  class  LabelPropagation  algorithm performs hard clamping of input labels  which means  math   alpha 0   This clamping factor can be relaxed  to say  math   alpha 0 2   which means that we will always retain 80 percent of our original label distribution  but the algorithm gets to change its confidence of the distribution within 20 percent    class  LabelPropagation  uses the raw similarity matrix constructed from the data with no modifications  In contrast   class  LabelSpreading  minimizes a loss function that has regularization properties  as such it is often more robust to noise  The algorithm iterates on a modified version of the original graph and normalizes the edge weights by computing the normalized graph Laplacian matrix  This procedure is also used in  ref  spectral clustering    Label propagation models have two built in kernel methods  Choice of kernel effects both scalability and performance of the algorithms  The following are available     rbf   math   exp   gamma  x y  2    gamma   0     math   gamma  is   specified by keyword gamma     knn   math  1 x   in kNN x       math  k  is specified by keyword   n neighbors   The RBF kernel will produce a fully connected graph which is represented in memory by a dense matrix  This matrix may be very large and combined with the cost of performing a full matrix multiplication calculation for each iteration of the algorithm can lead to prohibitively long running times  On the other hand  the KNN kernel will produce a much more memory friendly sparse matrix which can drastically reduce running times      rubric   Examples     ref  sphx glr auto examples semi supervised plot semi supervised versus svm iris py     ref  sphx glr auto examples semi supervised plot label propagation structure py     ref  sphx glr auto examples semi supervised plot label propagation digits py     ref  sphx glr auto examples semi supervised plot label propagation digits active learning py      rubric   References   2  Yoshua Bengio  Olivier Delalleau  Nicolas Le Roux  In Semi Supervised Learning  2006   pp  193 216   3  Olivier Delalleau  Yoshua Bengio  Nicolas Le Roux  Efficient Non Parametric Function Induction in Semi Supervised Learning  AISTAT 2005 https   www gatsby ucl ac uk aistats fullpapers 204 pdf