scikit-learn sklearn covariance Many statistical problems require the estimation of a Covariance estimation covariance

Answer:
.. _covariance:

===================================================
Covariance estimation
===================================================

.. currentmodule:: sklearn.covariance

Many statistical problems require the estimation of a
population's covariance matrix, which can be seen as an estimation of
data set scatter plot shape. Most of the time, such an estimation has
to be done on a sample whose properties (size, structure, homogeneity)
have a large influence on the estimation's quality. The
:mod:`sklearn.covariance` package provides tools for accurately estimating
a population's covariance matrix under various settings.

We assume that the observations are independent and identically
distributed (i.i.d.).

Empirical covariance
====================

The covariance matrix of a data set is known to be well approximated
by the classical *maximum likelihood estimator* (or "empirical
covariance"), provided the number of observations is large enough
compared to the number of features (the variables describing the
observations). More precisely, the Maximum Likelihood Estimator of a
sample is an asymptotically unbiased estimator of the corresponding
population's covariance matrix.

The empirical covariance matrix of a sample can be computed using the
:func:`empirical_covariance` function of the package, or by fitting an
:class:`EmpiricalCovariance` object to the data sample with the
:meth:`EmpiricalCovariance.fit` method. Be careful that results depend
on whether the data are centered, so one may want to use the
``assume_centered`` parameter accurately. More precisely, if
``assume_centered=False``, then the test set is supposed to have the
same mean vector as the training set. If not, both should be centered
by the user, and ``assume_centered=True`` should be used.

.. rubric:: Examples

* See :ref:`sphx_glr_auto_examples_covariance_plot_covariance_estimation.py` for
  an example on how to fit an :class:`EmpiricalCovariance` object to data.

.. _shrunk_covariance:

Shrunk Covariance
=================

Basic shrinkage
---------------

Despite being an asymptotically unbiased estimator of the covariance matrix,
the Maximum Likelihood Estimator is not a good estimator of the
eigenvalues of the covariance matrix, so the precision matrix obtained
from its inversion is not accurate. Sometimes, it even occurs that the
empirical covariance matrix cannot be inverted for numerical
reasons. To avoid such an inversion problem, a transformation of the
empirical covariance matrix has been introduced: the ``shrinkage``.

In scikit-learn, this transformation (with a user-defined shrinkage
coefficient) can be directly applied to a pre-computed covariance with
the :func:`shrunk_covariance` method. Also, a shrunk estimator of the
covariance can be fitted to data with a :class:`ShrunkCovariance` object
and its :meth:`ShrunkCovariance.fit` method. Again, results depend on
whether the data are centered, so one may want to use the
``assume_centered`` parameter accurately.

Mathematically, this shrinkage consists in reducing the ratio between the
smallest and the largest eigenvalues of the empirical covariance matrix.
It can be done by simply shifting every eigenvalue according to a given
offset, which is equivalent of finding the l2-penalized Maximum
Likelihood Estimator of the covariance matrix. In practice, shrinkage
boils down to a simple a convex transformation : :math:`\Sigma_{\rm
shrunk} = (1-\alpha)\hat{\Sigma} + \alpha\frac{{\rm
Tr}\hat{\Sigma}}{p}\rm Id`.

Choosing the amount of shrinkage, :math:`\alpha` amounts to setting a
bias/variance trade-off, and is discussed below.

.. rubric:: Examples

* See :ref:`sphx_glr_auto_examples_covariance_plot_covariance_estimation.py` for
  an example on how to fit a :class:`ShrunkCovariance` object to data.

Ledoit-Wolf shrinkage
---------------------

In their 2004 paper [1]_, O. Ledoit and M. Wolf propose a formula
to compute the optimal shrinkage coefficient :math:`\alpha` that
minimizes the Mean Squared Error between the estimated and the real
covariance matrix.

The Ledoit-Wolf estimator of the covariance matrix can be computed on
a sample with the :meth:`ledoit_wolf` function of the
:mod:`sklearn.covariance` package, or it can be otherwise obtained by
fitting a :class:`LedoitWolf` object to the same sample.

.. note:: **Case when population covariance matrix is isotropic**

    It is important to note that when the number of samples is much larger than
    the number of features, one would expect that no shrinkage would be
    necessary. The intuition behind this is that if the population covariance
    is full rank, when the number of sample grows, the sample covariance will
    also become positive definite. As a result, no shrinkage would necessary
    and the method should automatically do this.

    This, however, is not the case in the Ledoit-Wolf procedure when the
    population covariance happens to be a multiple of the identity matrix. In
    this case, the Ledoit-Wolf shrinkage estimate approaches 1 as the number of
    samples increases. This indicates that the optimal estimate of the
    covariance matrix in the Ledoit-Wolf sense is multiple of the identity.
    Since the population covariance is already a multiple of the identity
    matrix, the Ledoit-Wolf solution is indeed a reasonable estimate.

.. rubric:: Examples

* See :ref:`sphx_glr_auto_examples_covariance_plot_covariance_estimation.py` for
  an example on how to fit a :class:`LedoitWolf` object to data and
  for visualizing the performances of the Ledoit-Wolf estimator in
  terms of likelihood.

.. rubric:: References

.. [1] O. Ledoit and M. Wolf, "A Well-Conditioned Estimator for Large-Dimensional
       Covariance Matrices", Journal of Multivariate Analysis, Volume 88, Issue 2,
       February 2004, pages 365-411.

.. _oracle_approximating_shrinkage:

Oracle Approximating Shrinkage
------------------------------

Under the assumption that the data are Gaussian distributed, Chen et
al. [2]_ derived a formula aimed at choosing a shrinkage coefficient that
yields a smaller Mean Squared Error than the one given by Ledoit and
Wolf's formula. The resulting estimator is known as the Oracle
Shrinkage Approximating estimator of the covariance.

The OAS estimator of the covariance matrix can be computed on a sample
with the :meth:`oas` function of the :mod:`sklearn.covariance`
package, or it can be otherwise obtained by fitting an :class:`OAS`
object to the same sample.

.. figure:: ../auto_examples/covariance/images/sphx_glr_plot_covariance_estimation_001.png
   :target: ../auto_examples/covariance/plot_covariance_estimation.html
   :align: center
   :scale: 65%

   Bias-variance trade-off when setting the shrinkage: comparing the
   choices of Ledoit-Wolf and OAS estimators

.. rubric:: References

.. [2] :arxiv:`"Shrinkage algorithms for MMSE covariance estimation.",
       Chen, Y., Wiesel, A., Eldar, Y. C., & Hero, A. O.
       IEEE Transactions on Signal Processing, 58(10), 5016-5029, 2010.
       <0907.4698>`

.. rubric:: Examples

* See :ref:`sphx_glr_auto_examples_covariance_plot_covariance_estimation.py` for
  an example on how to fit an :class:`OAS` object to data.

* See :ref:`sphx_glr_auto_examples_covariance_plot_lw_vs_oas.py` to visualize the
  Mean Squared Error difference between a :class:`LedoitWolf` and
  an :class:`OAS` estimator of the covariance.

.. figure:: ../auto_examples/covariance/images/sphx_glr_plot_lw_vs_oas_001.png
   :target: ../auto_examples/covariance/plot_lw_vs_oas.html
   :align: center
   :scale: 75%

.. _sparse_inverse_covariance:

Sparse inverse covariance
==========================

The matrix inverse of the covariance matrix, often called the precision
matrix, is proportional to the partial correlation matrix. It gives the
partial independence relationship. In other words, if two features are
independent conditionally on the others, the corresponding coefficient in
the precision matrix will be zero. This is why it makes sense to
estimate a sparse precision matrix: the estimation of the covariance
matrix is better conditioned by learning independence relations from
the data. This is known as *covariance selection*.

In the small-samples situation, in which ``n_samples`` is on the order
of ``n_features`` or smaller, sparse inverse covariance estimators tend to work
better than shrunk covariance estimators. However, in the opposite
situation, or for very correlated data, they can be numerically unstable.
In addition, unlike shrinkage estimators, sparse estimators are able to
recover off-diagonal structure.

The :class:`GraphicalLasso` estimator uses an l1 penalty to enforce sparsity on
the precision matrix: the higher its ``alpha`` parameter, the more sparse
the precision matrix. The corresponding :class:`GraphicalLassoCV` object uses
cross-validation to automatically set the ``alpha`` parameter.

.. figure:: ../auto_examples/covariance/images/sphx_glr_plot_sparse_cov_001.png
   :target: ../auto_examples/covariance/plot_sparse_cov.html
   :align: center
   :scale: 60%

   *A comparison of maximum likelihood, shrinkage and sparse estimates of
   the covariance and precision matrix in the very small samples
   settings.*

.. note:: **Structure recovery**

   Recovering a graphical structure from correlations in the data is a
   challenging thing. If you are interested in such recovery keep in mind
   that:

   * Recovery is easier from a correlation matrix than a covariance
     matrix: standardize your observations before running :class:`GraphicalLasso`

   * If the underlying graph has nodes with much more connections than
     the average node, the algorithm will miss some of these connections.

   * If your number of observations is not large compared to the number
     of edges in your underlying graph, you will not recover it.

   * Even if you are in favorable recovery conditions, the alpha
     parameter chosen by cross-validation (e.g. using the
     :class:`GraphicalLassoCV` object) will lead to selecting too many edges.
     However, the relevant edges will have heavier weights than the
     irrelevant ones.

The mathematical formulation is the following:

.. math::

    \hat{K} = \mathrm{argmin}_K \big(
                \mathrm{tr} S K - \mathrm{log} \mathrm{det} K
                + \alpha \|K\|_1
                \big)

Where :math:`K` is the precision matrix to be estimated, and :math:`S` is the
sample covariance matrix. :math:`\|K\|_1` is the sum of the absolute values of
off-diagonal coefficients of :math:`K`. The algorithm employed to solve this
problem is the GLasso algorithm, from the Friedman 2008 Biostatistics
paper. It is the same algorithm as in the R ``glasso`` package.

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_covariance_plot_sparse_cov.py`: example on synthetic
  data showing some recovery of a structure, and comparing to other
  covariance estimators.

* :ref:`sphx_glr_auto_examples_applications_plot_stock_market.py`: example on real
  stock market data, finding which symbols are most linked.

.. rubric:: References

* Friedman et al, `"Sparse inverse covariance estimation with the
  graphical lasso" <https://biostatistics.oxfordjournals.org/content/9/3/432.short>`_,
  Biostatistics 9, pp 432, 2008

.. _robust_covariance:

Robust Covariance Estimation
============================

Real data sets are often subject to measurement or recording
errors. Regular but uncommon observations may also appear for a variety
of reasons. Observations which are very uncommon are called
outliers.
The empirical covariance estimator and the shrunk covariance
estimators presented above are very sensitive to the presence of
outliers in the data. Therefore, one should use robust
covariance estimators to estimate the covariance of its real data
sets. Alternatively, robust covariance estimators can be used to
perform outlier detection and discard/downweight some observations
according to further processing of the data.

The ``sklearn.covariance`` package implements a robust estimator of covariance,
the Minimum Covariance Determinant [3]_.

Minimum Covariance Determinant
------------------------------

The Minimum Covariance Determinant estimator is a robust estimator of
a data set's covariance introduced by P.J. Rousseeuw in [3]_.  The idea
is to find a given proportion (h) of "good" observations which are not
outliers and compute their empirical covariance matrix.  This
empirical covariance matrix is then rescaled to compensate the
performed selection of observations ("consistency step").  Having
computed the Minimum Covariance Determinant estimator, one can give
weights to observations according to their Mahalanobis distance,
leading to a reweighted estimate of the covariance matrix of the data
set ("reweighting step").

Rousseeuw and Van Driessen [4]_ developed the FastMCD algorithm in order
to compute the Minimum Covariance Determinant. This algorithm is used
in scikit-learn when fitting an MCD object to data. The FastMCD
algorithm also computes a robust estimate of the data set location at
the same time.

Raw estimates can be accessed as ``raw_location_`` and ``raw_covariance_``
attributes of a :class:`MinCovDet` robust covariance estimator object.

.. rubric:: References

.. [3] P. J. Rousseeuw. Least median of squares regression.
       J. Am Stat Ass, 79:871, 1984.
.. [4] A Fast Algorithm for the Minimum Covariance Determinant Estimator,
       1999, American Statistical Association and the American Society
       for Quality, TECHNOMETRICS.

.. rubric:: Examples

* See :ref:`sphx_glr_auto_examples_covariance_plot_robust_vs_empirical_covariance.py` for
  an example on how to fit a :class:`MinCovDet` object to data and see how
  the estimate remains accurate despite the presence of outliers.

* See :ref:`sphx_glr_auto_examples_covariance_plot_mahalanobis_distances.py` to
  visualize the difference between :class:`EmpiricalCovariance` and
  :class:`MinCovDet` covariance estimators in terms of Mahalanobis distance
  (so we get a better estimate of the precision matrix too).

.. |robust_vs_emp| image:: ../auto_examples/covariance/images/sphx_glr_plot_robust_vs_empirical_covariance_001.png
   :target: ../auto_examples/covariance/plot_robust_vs_empirical_covariance.html
   :scale: 49%

.. |mahalanobis| image:: ../auto_examples/covariance/images/sphx_glr_plot_mahalanobis_distances_001.png
   :target: ../auto_examples/covariance/plot_mahalanobis_distances.html
   :scale: 49%

____

.. list-table::
    :header-rows: 1

    * - Influence of outliers on location and covariance estimates
      - Separating inliers from outliers using a Mahalanobis distance

    * - |robust_vs_emp|
      - |mahalanobis|

covariance                                                       Covariance estimation                                                         currentmodule   sklearn covariance   Many statistical problems require the estimation of a population s covariance matrix  which can be seen as an estimation of data set scatter plot shape  Most of the time  such an estimation has to be done on a sample whose properties  size  structure  homogeneity  have a large influence on the estimation s quality  The  mod  sklearn covariance  package provides tools for accurately estimating a population s covariance matrix under various settings   We assume that the observations are independent and identically distributed  i i d      Empirical covariance                       The covariance matrix of a data set is known to be well approximated by the classical  maximum likelihood estimator   or  empirical covariance    provided the number of observations is large enough compared to the number of features  the variables describing the observations   More precisely  the Maximum Likelihood Estimator of a sample is an asymptotically unbiased estimator of the corresponding population s covariance matrix   The empirical covariance matrix of a sample can be computed using the  func  empirical covariance  function of the package  or by fitting an  class  EmpiricalCovariance  object to the data sample with the  meth  EmpiricalCovariance fit  method  Be careful that results depend on whether the data are centered  so one may want to use the   assume centered   parameter accurately  More precisely  if   assume centered False    then the test set is supposed to have the same mean vector as the training set  If not  both should be centered by the user  and   assume centered True   should be used      rubric   Examples    See  ref  sphx glr auto examples covariance plot covariance estimation py  for   an example on how to fit an  class  EmpiricalCovariance  object to data        shrunk covariance   Shrunk Covariance                    Basic shrinkage                  Despite being an asymptotically unbiased estimator of the covariance matrix  the Maximum Likelihood Estimator is not a good estimator of the eigenvalues of the covariance matrix  so the precision matrix obtained from its inversion is not accurate  Sometimes  it even occurs that the empirical covariance matrix cannot be inverted for numerical reasons  To avoid such an inversion problem  a transformation of the empirical covariance matrix has been introduced  the   shrinkage     In scikit learn  this transformation  with a user defined shrinkage coefficient  can be directly applied to a pre computed covariance with the  func  shrunk covariance  method  Also  a shrunk estimator of the covariance can be fitted to data with a  class  ShrunkCovariance  object and its  meth  ShrunkCovariance fit  method  Again  results depend on whether the data are centered  so one may want to use the   assume centered   parameter accurately    Mathematically  this shrinkage consists in reducing the ratio between the smallest and the largest eigenvalues of the empirical covariance matrix  It can be done by simply shifting every eigenvalue according to a given offset  which is equivalent of finding the l2 penalized Maximum Likelihood Estimator of the covariance matrix  In practice  shrinkage boils down to a simple a convex transformation    math   Sigma   rm shrunk     1  alpha  hat  Sigma     alpha frac   rm Tr  hat  Sigma   p  rm Id    Choosing the amount of shrinkage   math   alpha  amounts to setting a bias variance trade off  and is discussed below      rubric   Examples    See  ref  sphx glr auto examples covariance plot covariance estimation py  for   an example on how to fit a  class  ShrunkCovariance  object to data    Ledoit Wolf shrinkage                        In their 2004 paper  1    O  Ledoit and M  Wolf propose a formula to compute the optimal shrinkage coefficient  math   alpha  that minimizes the Mean Squared Error between the estimated and the real covariance matrix   The Ledoit Wolf estimator of the covariance matrix can be computed on a sample with the  meth  ledoit wolf  function of the  mod  sklearn covariance  package  or it can be otherwise obtained by fitting a  class  LedoitWolf  object to the same sample      note     Case when population covariance matrix is isotropic        It is important to note that when the number of samples is much larger than     the number of features  one would expect that no shrinkage would be     necessary  The intuition behind this is that if the population covariance     is full rank  when the number of sample grows  the sample covariance will     also become positive definite  As a result  no shrinkage would necessary     and the method should automatically do this       This  however  is not the case in the Ledoit Wolf procedure when the     population covariance happens to be a multiple of the identity matrix  In     this case  the Ledoit Wolf shrinkage estimate approaches 1 as the number of     samples increases  This indicates that the optimal estimate of the     covariance matrix in the Ledoit Wolf sense is multiple of the identity      Since the population covariance is already a multiple of the identity     matrix  the Ledoit Wolf solution is indeed a reasonable estimate      rubric   Examples    See  ref  sphx glr auto examples covariance plot covariance estimation py  for   an example on how to fit a  class  LedoitWolf  object to data and   for visualizing the performances of the Ledoit Wolf estimator in   terms of likelihood      rubric   References      1  O  Ledoit and M  Wolf   A Well Conditioned Estimator for Large Dimensional        Covariance Matrices   Journal of Multivariate Analysis  Volume 88  Issue 2         February 2004  pages 365 411       oracle approximating shrinkage   Oracle Approximating Shrinkage                                 Under the assumption that the data are Gaussian distributed  Chen et al   2   derived a formula aimed at choosing a shrinkage coefficient that yields a smaller Mean Squared Error than the one given by Ledoit and Wolf s formula  The resulting estimator is known as the Oracle Shrinkage Approximating estimator of the covariance   The OAS estimator of the covariance matrix can be computed on a sample with the  meth  oas  function of the  mod  sklearn covariance  package  or it can be otherwise obtained by fitting an  class  OAS  object to the same sample      figure      auto examples covariance images sphx glr plot covariance estimation 001 png     target     auto examples covariance plot covariance estimation html     align  center     scale  65      Bias variance trade off when setting the shrinkage  comparing the    choices of Ledoit Wolf and OAS estimators     rubric   References      2   arxiv   Shrinkage algorithms for MMSE covariance estimation           Chen  Y   Wiesel  A   Eldar  Y  C     Hero  A  O         IEEE Transactions on Signal Processing  58 10   5016 5029  2010          0907 4698       rubric   Examples    See  ref  sphx glr auto examples covariance plot covariance estimation py  for   an example on how to fit an  class  OAS  object to data     See  ref  sphx glr auto examples covariance plot lw vs oas py  to visualize the   Mean Squared Error difference between a  class  LedoitWolf  and   an  class  OAS  estimator of the covariance       figure      auto examples covariance images sphx glr plot lw vs oas 001 png     target     auto examples covariance plot lw vs oas html     align  center     scale  75        sparse inverse covariance   Sparse inverse covariance                             The matrix inverse of the covariance matrix  often called the precision matrix  is proportional to the partial correlation matrix  It gives the partial independence relationship  In other words  if two features are independent conditionally on the others  the corresponding coefficient in the precision matrix will be zero  This is why it makes sense to estimate a sparse precision matrix  the estimation of the covariance matrix is better conditioned by learning independence relations from the data  This is known as  covariance selection    In the small samples situation  in which   n samples   is on the order of   n features   or smaller  sparse inverse covariance estimators tend to work better than shrunk covariance estimators  However  in the opposite situation  or for very correlated data  they can be numerically unstable  In addition  unlike shrinkage estimators  sparse estimators are able to recover off diagonal structure   The  class  GraphicalLasso  estimator uses an l1 penalty to enforce sparsity on the precision matrix  the higher its   alpha   parameter  the more sparse the precision matrix  The corresponding  class  GraphicalLassoCV  object uses cross validation to automatically set the   alpha   parameter      figure      auto examples covariance images sphx glr plot sparse cov 001 png     target     auto examples covariance plot sparse cov html     align  center     scale  60       A comparison of maximum likelihood  shrinkage and sparse estimates of    the covariance and precision matrix in the very small samples    settings       note     Structure recovery       Recovering a graphical structure from correlations in the data is a    challenging thing  If you are interested in such recovery keep in mind    that        Recovery is easier from a correlation matrix than a covariance      matrix  standardize your observations before running  class  GraphicalLasso        If the underlying graph has nodes with much more connections than      the average node  the algorithm will miss some of these connections        If your number of observations is not large compared to the number      of edges in your underlying graph  you will not recover it        Even if you are in favorable recovery conditions  the alpha      parameter chosen by cross validation  e g  using the       class  GraphicalLassoCV  object  will lead to selecting too many edges       However  the relevant edges will have heavier weights than the      irrelevant ones   The mathematical formulation is the following      math         hat K     mathrm argmin  K  big                   mathrm tr  S K    mathrm log   mathrm det  K                    alpha   K   1                  big   Where  math  K  is the precision matrix to be estimated  and  math  S  is the sample covariance matrix   math    K   1  is the sum of the absolute values of off diagonal coefficients of  math  K   The algorithm employed to solve this problem is the GLasso algorithm  from the Friedman 2008 Biostatistics paper  It is the same algorithm as in the R   glasso   package       rubric   Examples     ref  sphx glr auto examples covariance plot sparse cov py   example on synthetic   data showing some recovery of a structure  and comparing to other   covariance estimators      ref  sphx glr auto examples applications plot stock market py   example on real   stock market data  finding which symbols are most linked      rubric   References    Friedman et al    Sparse inverse covariance estimation with the   graphical lasso   https   biostatistics oxfordjournals org content 9 3 432 short       Biostatistics 9  pp 432  2008      robust covariance   Robust Covariance Estimation                               Real data sets are often subject to measurement or recording errors  Regular but uncommon observations may also appear for a variety of reasons  Observations which are very uncommon are called outliers  The empirical covariance estimator and the shrunk covariance estimators presented above are very sensitive to the presence of outliers in the data  Therefore  one should use robust covariance estimators to estimate the covariance of its real data sets  Alternatively  robust covariance estimators can be used to perform outlier detection and discard downweight some observations according to further processing of the data   The   sklearn covariance   package implements a robust estimator of covariance  the Minimum Covariance Determinant  3      Minimum Covariance Determinant                                 The Minimum Covariance Determinant estimator is a robust estimator of a data set s covariance introduced by P J  Rousseeuw in  3     The idea is to find a given proportion  h  of  good  observations which are not outliers and compute their empirical covariance matrix   This empirical covariance matrix is then rescaled to compensate the performed selection of observations   consistency step     Having computed the Minimum Covariance Determinant estimator  one can give weights to observations according to their Mahalanobis distance  leading to a reweighted estimate of the covariance matrix of the data set   reweighting step     Rousseeuw and Van Driessen  4   developed the FastMCD algorithm in order to compute the Minimum Covariance Determinant  This algorithm is used in scikit learn when fitting an MCD object to data  The FastMCD algorithm also computes a robust estimate of the data set location at the same time   Raw estimates can be accessed as   raw location    and   raw covariance    attributes of a  class  MinCovDet  robust covariance estimator object      rubric   References      3  P  J  Rousseeuw  Least median of squares regression         J  Am Stat Ass  79 871  1984      4  A Fast Algorithm for the Minimum Covariance Determinant Estimator         1999  American Statistical Association and the American Society        for Quality  TECHNOMETRICS      rubric   Examples    See  ref  sphx glr auto examples covariance plot robust vs empirical covariance py  for   an example on how to fit a  class  MinCovDet  object to data and see how   the estimate remains accurate despite the presence of outliers     See  ref  sphx glr auto examples covariance plot mahalanobis distances py  to   visualize the difference between  class  EmpiricalCovariance  and    class  MinCovDet  covariance estimators in terms of Mahalanobis distance    so we get a better estimate of the precision matrix too        robust vs emp  image      auto examples covariance images sphx glr plot robust vs empirical covariance 001 png     target     auto examples covariance plot robust vs empirical covariance html     scale  49       mahalanobis  image      auto examples covariance images sphx glr plot mahalanobis distances 001 png     target     auto examples covariance plot mahalanobis distances html     scale  49              list table        header rows  1          Influence of outliers on location and covariance estimates         Separating inliers from outliers using a Mahalanobis distance           robust vs emp           mahalanobis


scikit-learn classification and regression This section of the user guide covers functionality related to multi learning problems including and Multiclass and multioutput algorithms multiclass

Answer:
.. _multiclass:

=====================================
Multiclass and multioutput algorithms
=====================================

This section of the user guide covers functionality related to multi-learning
problems, including :term:`multiclass`, :term:`multilabel`, and
:term:`multioutput` classification and regression.

The modules in this section implement :term:`meta-estimators`, which require a
base estimator to be provided in their constructor. Meta-estimators extend the
functionality of the base estimator to support multi-learning problems, which
is accomplished by transforming the multi-learning problem into a set of
simpler problems, then fitting one estimator per problem.

This section covers two modules: :mod:`sklearn.multiclass` and
:mod:`sklearn.multioutput`. The chart below demonstrates the problem types
that each module is responsible for, and the corresponding meta-estimators
that each module provides.

.. image:: ../images/multi_org_chart.png
   :align: center

The table below provides a quick reference on the differences between problem
types. More detailed explanations can be found in subsequent sections of this
guide.

+------------------------------+-----------------------+-------------------------+--------------------------------------------------+
|                              | Number of targets     | Target cardinality      | Valid                                            |
|                              |                       |                         | :func:`~sklearn.utils.multiclass.type_of_target` |
+==============================+=======================+=========================+==================================================+
| Multiclass                   |  1                    | >2                      | 'multiclass'                                     |
| classification               |                       |                         |                                                  |
+------------------------------+-----------------------+-------------------------+--------------------------------------------------+
| Multilabel                   | >1                    |  2 (0 or 1)             | 'multilabel-indicator'                           |
| classification               |                       |                         |                                                  |
+------------------------------+-----------------------+-------------------------+--------------------------------------------------+
| Multiclass-multioutput       | >1                    | >2                      | 'multiclass-multioutput'                         |
| classification               |                       |                         |                                                  |
+------------------------------+-----------------------+-------------------------+--------------------------------------------------+
| Multioutput                  | >1                    | Continuous              | 'continuous-multioutput'                         |
| regression                   |                       |                         |                                                  |
+------------------------------+-----------------------+-------------------------+--------------------------------------------------+

Below is a summary of scikit-learn estimators that have multi-learning support
built-in, grouped by strategy. You don't need the meta-estimators provided by
this section if you're using one of these estimators. However, meta-estimators
can provide additional strategies beyond what is built-in:

.. currentmodule:: sklearn

- **Inherently multiclass:**

  - :class:`naive_bayes.BernoulliNB`
  - :class:`tree.DecisionTreeClassifier`
  - :class:`tree.ExtraTreeClassifier`
  - :class:`ensemble.ExtraTreesClassifier`
  - :class:`naive_bayes.GaussianNB`
  - :class:`neighbors.KNeighborsClassifier`
  - :class:`semi_supervised.LabelPropagation`
  - :class:`semi_supervised.LabelSpreading`
  - :class:`discriminant_analysis.LinearDiscriminantAnalysis`
  - :class:`svm.LinearSVC` (setting multi_class="crammer_singer")
  - :class:`linear_model.LogisticRegression` (with most solvers)
  - :class:`linear_model.LogisticRegressionCV` (with most solvers)
  - :class:`neural_network.MLPClassifier`
  - :class:`neighbors.NearestCentroid`
  - :class:`discriminant_analysis.QuadraticDiscriminantAnalysis`
  - :class:`neighbors.RadiusNeighborsClassifier`
  - :class:`ensemble.RandomForestClassifier`
  - :class:`linear_model.RidgeClassifier`
  - :class:`linear_model.RidgeClassifierCV`

- **Multiclass as One-Vs-One:**

  - :class:`svm.NuSVC`
  - :class:`svm.SVC`.
  - :class:`gaussian_process.GaussianProcessClassifier` (setting multi_class = "one_vs_one")

- **Multiclass as One-Vs-The-Rest:**

  - :class:`ensemble.GradientBoostingClassifier`
  - :class:`gaussian_process.GaussianProcessClassifier` (setting multi_class = "one_vs_rest")
  - :class:`svm.LinearSVC` (setting multi_class="ovr")
  - :class:`linear_model.LogisticRegression` (most solvers)
  - :class:`linear_model.LogisticRegressionCV` (most solvers)
  - :class:`linear_model.SGDClassifier`
  - :class:`linear_model.Perceptron`
  - :class:`linear_model.PassiveAggressiveClassifier`

- **Support multilabel:**

  - :class:`tree.DecisionTreeClassifier`
  - :class:`tree.ExtraTreeClassifier`
  - :class:`ensemble.ExtraTreesClassifier`
  - :class:`neighbors.KNeighborsClassifier`
  - :class:`neural_network.MLPClassifier`
  - :class:`neighbors.RadiusNeighborsClassifier`
  - :class:`ensemble.RandomForestClassifier`
  - :class:`linear_model.RidgeClassifier`
  - :class:`linear_model.RidgeClassifierCV`

- **Support multiclass-multioutput:**

  - :class:`tree.DecisionTreeClassifier`
  - :class:`tree.ExtraTreeClassifier`
  - :class:`ensemble.ExtraTreesClassifier`
  - :class:`neighbors.KNeighborsClassifier`
  - :class:`neighbors.RadiusNeighborsClassifier`
  - :class:`ensemble.RandomForestClassifier`

.. _multiclass_classification:

Multiclass classification
=========================

.. warning::
    All classifiers in scikit-learn do multiclass classification
    out-of-the-box. You don't need to use the :mod:`sklearn.multiclass` module
    unless you want to experiment with different multiclass strategies.

**Multiclass classification** is a classification task with more than two
classes. Each sample can only be labeled as one class.

For example, classification using features extracted from a set of images of
fruit, where each image may either be of an orange, an apple, or a pear.
Each image is one sample and is labeled as one of the 3 possible classes.
Multiclass classification makes the assumption that each sample is assigned
to one and only one label - one sample cannot, for example, be both a pear
and an apple.

While all scikit-learn classifiers are capable of multiclass classification,
the meta-estimators offered by :mod:`sklearn.multiclass`
permit changing the way they handle more than two classes
because this may have an effect on classifier performance
(either in terms of generalization error or required computational resources).

Target format
-------------

Valid :term:`multiclass` representations for
:func:`~sklearn.utils.multiclass.type_of_target` (`y`) are:

- 1d or column vector containing more than two discrete values. An
  example of a vector ``y`` for 4 samples:

    >>> import numpy as np
    >>> y = np.array(['apple', 'pear', 'apple', 'orange'])
    >>> print(y)
    ['apple' 'pear' 'apple' 'orange']

- Dense or sparse :term:`binary` matrix of shape ``(n_samples, n_classes)``
  with a single sample per row, where each column represents one class. An
  example of both a dense and sparse :term:`binary` matrix ``y`` for 4
  samples, where the columns, in order, are apple, orange, and pear:

    >>> import numpy as np
    >>> from sklearn.preprocessing import LabelBinarizer
    >>> y = np.array(['apple', 'pear', 'apple', 'orange'])
    >>> y_dense = LabelBinarizer().fit_transform(y)
    >>> print(y_dense)
    [[1 0 0]
     [0 0 1]
     [1 0 0]
     [0 1 0]]
    >>> from scipy import sparse
    >>> y_sparse = sparse.csr_matrix(y_dense)
    >>> print(y_sparse)
    <Compressed Sparse Row sparse matrix of dtype 'int64'
    	with 4 stored elements and shape (4, 3)>
      Coords	Values
      (0, 0)	1
      (1, 2)	1
      (2, 0)	1
      (3, 1)	1

For more information about :class:`~sklearn.preprocessing.LabelBinarizer`,
refer to :ref:`preprocessing_targets`.

.. _ovr_classification:

OneVsRestClassifier
-------------------

The **one-vs-rest** strategy, also known as **one-vs-all**, is implemented in
:class:`~sklearn.multiclass.OneVsRestClassifier`.  The strategy consists in
fitting one classifier per class. For each classifier, the class is fitted
against all the other classes. In addition to its computational efficiency
(only `n_classes` classifiers are needed), one advantage of this approach is
its interpretability. Since each class is represented by one and only one
classifier, it is possible to gain knowledge about the class by inspecting its
corresponding classifier. This is the most commonly used strategy and is a fair
default choice.

Below is an example of multiclass learning using OvR::

  >>> from sklearn import datasets
  >>> from sklearn.multiclass import OneVsRestClassifier
  >>> from sklearn.svm import LinearSVC
  >>> X, y = datasets.load_iris(return_X_y=True)
  >>> OneVsRestClassifier(LinearSVC(random_state=0)).fit(X, y).predict(X)
  array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
         2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 1, 2, 2, 2, 2,
         2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

:class:`~sklearn.multiclass.OneVsRestClassifier` also supports multilabel
classification. To use this feature, feed the classifier an indicator matrix,
in which cell [i, j] indicates the presence of label j in sample i.

.. figure:: ../auto_examples/miscellaneous/images/sphx_glr_plot_multilabel_001.png
    :target: ../auto_examples/miscellaneous/plot_multilabel.html
    :align: center
    :scale: 75%

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_miscellaneous_plot_multilabel.py`
* :ref:`sphx_glr_auto_examples_classification_plot_classification_probability.py`

.. _ovo_classification:

OneVsOneClassifier
------------------

:class:`~sklearn.multiclass.OneVsOneClassifier` constructs one classifier per
pair of classes. At prediction time, the class which received the most votes
is selected. In the event of a tie (among two classes with an equal number of
votes), it selects the class with the highest aggregate classification
confidence by summing over the pair-wise classification confidence levels
computed by the underlying binary classifiers.

Since it requires to fit ``n_classes * (n_classes - 1) / 2`` classifiers,
this method is usually slower than one-vs-the-rest, due to its
O(n_classes^2) complexity. However, this method may be advantageous for
algorithms such as kernel algorithms which don't scale well with
``n_samples``. This is because each individual learning problem only involves
a small subset of the data whereas, with one-vs-the-rest, the complete
dataset is used ``n_classes`` times. The decision function is the result
of a monotonic transformation of the one-versus-one classification.

Below is an example of multiclass learning using OvO::

  >>> from sklearn import datasets
  >>> from sklearn.multiclass import OneVsOneClassifier
  >>> from sklearn.svm import LinearSVC
  >>> X, y = datasets.load_iris(return_X_y=True)
  >>> OneVsOneClassifier(LinearSVC(random_state=0)).fit(X, y).predict(X)
  array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 2, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
         2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
         2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

.. rubric:: References

* "Pattern Recognition and Machine Learning. Springer",
  Christopher M. Bishop, page 183, (First Edition)

.. _ecoc:

OutputCodeClassifier
--------------------

Error-Correcting Output Code-based strategies are fairly different from
one-vs-the-rest and one-vs-one. With these strategies, each class is
represented in a Euclidean space, where each dimension can only be 0 or 1.
Another way to put it is that each class is represented by a binary code (an
array of 0 and 1). The matrix which keeps track of the location/code of each
class is called the code book. The code size is the dimensionality of the
aforementioned space. Intuitively, each class should be represented by a code
as unique as possible and a good code book should be designed to optimize
classification accuracy. In this implementation, we simply use a
randomly-generated code book as advocated in [3]_ although more elaborate
methods may be added in the future.

At fitting time, one binary classifier per bit in the code book is fitted.
At prediction time, the classifiers are used to project new points in the
class space and the class closest to the points is chosen.

In :class:`~sklearn.multiclass.OutputCodeClassifier`, the ``code_size``
attribute allows the user to control the number of classifiers which will be
used. It is a percentage of the total number of classes.

A number between 0 and 1 will require fewer classifiers than
one-vs-the-rest. In theory, ``log2(n_classes) / n_classes`` is sufficient to
represent each class unambiguously. However, in practice, it may not lead to
good accuracy since ``log2(n_classes)`` is much smaller than `n_classes`.

A number greater than 1 will require more classifiers than
one-vs-the-rest. In this case, some classifiers will in theory correct for
the mistakes made by other classifiers, hence the name "error-correcting".
In practice, however, this may not happen as classifier mistakes will
typically be correlated. The error-correcting output codes have a similar
effect to bagging.

Below is an example of multiclass learning using Output-Codes::

  >>> from sklearn import datasets
  >>> from sklearn.multiclass import OutputCodeClassifier
  >>> from sklearn.svm import LinearSVC
  >>> X, y = datasets.load_iris(return_X_y=True)
  >>> clf = OutputCodeClassifier(LinearSVC(random_state=0), code_size=2, random_state=0)
  >>> clf.fit(X, y).predict(X)
  array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1,
         1, 2, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
         2, 2, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 1, 1, 2, 2, 2,
         2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

.. rubric:: References

* "Solving multiclass learning problems via error-correcting output codes",
  Dietterich T., Bakiri G., Journal of Artificial Intelligence Research 2, 1995.

.. [3] "The error coding method and PICTs", James G., Hastie T.,
  Journal of Computational and Graphical statistics 7, 1998.

* "The Elements of Statistical Learning",
  Hastie T., Tibshirani R., Friedman J., page 606 (second-edition), 2008.

.. _multilabel_classification:

Multilabel classification
=========================

**Multilabel classification** (closely related to **multioutput**
**classification**) is a classification task labeling each sample with ``m``
labels from ``n_classes`` possible classes, where ``m`` can be 0 to
``n_classes`` inclusive. This can be thought of as predicting properties of a
sample that are not mutually exclusive. Formally, a binary output is assigned
to each class, for every sample. Positive classes are indicated with 1 and
negative classes with 0 or -1. It is thus comparable to running ``n_classes``
binary classification tasks, for example with
:class:`~sklearn.multioutput.MultiOutputClassifier`. This approach treats
each label independently whereas multilabel classifiers *may* treat the
multiple classes simultaneously, accounting for correlated behavior among
them.

For example, prediction of the topics relevant to a text document or video.
The document or video may be about one of 'religion', 'politics', 'finance'
or 'education', several of the topic classes or all of the topic classes.

Target format
-------------

A valid representation of :term:`multilabel` `y` is an either dense or sparse
:term:`binary` matrix of shape ``(n_samples, n_classes)``. Each column
represents a class. The ``1``'s in each row denote the positive classes a
sample has been labeled with. An example of a dense matrix ``y`` for 3
samples:

  >>> y = np.array([[1, 0, 0, 1], [0, 0, 1, 1], [0, 0, 0, 0]])
  >>> print(y)
  [[1 0 0 1]
   [0 0 1 1]
   [0 0 0 0]]

Dense binary matrices can also be created using
:class:`~sklearn.preprocessing.MultiLabelBinarizer`. For more information,
refer to :ref:`preprocessing_targets`.

An example of the same ``y`` in sparse matrix form:

  >>> y_sparse = sparse.csr_matrix(y)
  >>> print(y_sparse)
  <Compressed Sparse Row sparse matrix of dtype 'int64'
    with 4 stored elements and shape (3, 4)>
    Coords	Values
    (0, 0)	1
    (0, 3)	1
    (1, 2)	1
    (1, 3)	1

.. _multioutputclassfier:

MultiOutputClassifier
---------------------

Multilabel classification support can be added to any classifier with
:class:`~sklearn.multioutput.MultiOutputClassifier`. This strategy consists of
fitting one classifier per target.  This allows multiple target variable
classifications. The purpose of this class is to extend estimators
to be able to estimate a series of target functions (f1,f2,f3...,fn)
that are trained on a single X predictor matrix to predict a series
of responses (y1,y2,y3...,yn).

You can find a usage example for
:class:`~sklearn.multioutput.MultiOutputClassifier`
as part of the section on :ref:`multiclass_multioutput_classification`
since it is a generalization of multilabel classification to
multiclass outputs instead of binary outputs.

.. _classifierchain:

ClassifierChain
---------------

Classifier chains (see :class:`~sklearn.multioutput.ClassifierChain`) are a way
of combining a number of binary classifiers into a single multi-label model
that is capable of exploiting correlations among targets.

For a multi-label classification problem with N classes, N binary
classifiers are assigned an integer between 0 and N-1. These integers
define the order of models in the chain. Each classifier is then fit on the
available training data plus the true labels of the classes whose
models were assigned a lower number.

When predicting, the true labels will not be available. Instead the
predictions of each model are passed on to the subsequent models in the
chain to be used as features.

Clearly the order of the chain is important. The first model in the chain
has no information about the other labels while the last model in the chain
has features indicating the presence of all of the other labels. In general
one does not know the optimal ordering of the models in the chain so
typically many randomly ordered chains are fit and their predictions are
averaged together.

.. rubric:: References

* Jesse Read, Bernhard Pfahringer, Geoff Holmes, Eibe Frank,
  "Classifier Chains for Multi-label Classification", 2009.

.. _multiclass_multioutput_classification:

Multiclass-multioutput classification
=====================================

**Multiclass-multioutput classification**
(also known as **multitask classification**) is a
classification task which labels each sample with a set of **non-binary**
properties. Both the number of properties and the number of
classes per property is greater than 2. A single estimator thus
handles several joint classification tasks. This is both a generalization of
the multi\ *label* classification task, which only considers binary
attributes, as well as a generalization of the multi\ *class* classification
task, where only one property is considered.

For example, classification of the properties "type of fruit" and "colour"
for a set of images of fruit. The property "type of fruit" has the possible
classes: "apple", "pear" and "orange". The property "colour" has the
possible classes: "green", "red", "yellow" and "orange". Each sample is an
image of a fruit, a label is output for both properties and each label is
one of the possible classes of the corresponding property.

Note that all classifiers handling multiclass-multioutput (also known as
multitask classification) tasks, support the multilabel classification task
as a special case. Multitask classification is similar to the multioutput
classification task with different model formulations. For more information,
see the relevant estimator documentation.

Below is an example of multiclass-multioutput classification:

    >>> from sklearn.datasets import make_classification
    >>> from sklearn.multioutput import MultiOutputClassifier
    >>> from sklearn.ensemble import RandomForestClassifier
    >>> from sklearn.utils import shuffle
    >>> import numpy as np
    >>> X, y1 = make_classification(n_samples=10, n_features=100,
    ...                             n_informative=30, n_classes=3,
    ...                             random_state=1)
    >>> y2 = shuffle(y1, random_state=1)
    >>> y3 = shuffle(y1, random_state=2)
    >>> Y = np.vstack((y1, y2, y3)).T
    >>> n_samples, n_features = X.shape # 10,100
    >>> n_outputs = Y.shape[1] # 3
    >>> n_classes = 3
    >>> forest = RandomForestClassifier(random_state=1)
    >>> multi_target_forest = MultiOutputClassifier(forest, n_jobs=2)
    >>> multi_target_forest.fit(X, Y).predict(X)
    array([[2, 2, 0],
           [1, 2, 1],
           [2, 1, 0],
           [0, 0, 2],
           [0, 2, 1],
           [0, 0, 2],
           [1, 1, 0],
           [1, 1, 1],
           [0, 0, 2],
           [2, 0, 0]])

.. warning::
    At present, no metric in :mod:`sklearn.metrics`
    supports the multiclass-multioutput classification task.

Target format
-------------

A valid representation of :term:`multioutput` `y` is a dense matrix of shape
``(n_samples, n_classes)`` of class labels. A column wise concatenation of 1d
:term:`multiclass` variables. An example of ``y`` for 3 samples:

  >>> y = np.array([['apple', 'green'], ['orange', 'orange'], ['pear', 'green']])
  >>> print(y)
  [['apple' 'green']
   ['orange' 'orange']
   ['pear' 'green']]

.. _multioutput_regression:

Multioutput regression
======================

**Multioutput regression** predicts multiple numerical properties for each
sample. Each property is a numerical variable and the number of properties
to be predicted for each sample is greater than or equal to 2. Some estimators
that support multioutput regression are faster than just running ``n_output``
estimators.

For example, prediction of both wind speed and wind direction, in degrees,
using data obtained at a certain location. Each sample would be data
obtained at one location and both wind speed and direction would be
output for each sample.

The following regressors natively support multioutput regression:

- :class:`cross_decomposition.CCA`
- :class:`tree.DecisionTreeRegressor`
- :class:`dummy.DummyRegressor`
- :class:`linear_model.ElasticNet`
- :class:`tree.ExtraTreeRegressor`
- :class:`ensemble.ExtraTreesRegressor`
- :class:`gaussian_process.GaussianProcessRegressor`
- :class:`neighbors.KNeighborsRegressor`
- :class:`kernel_ridge.KernelRidge`
- :class:`linear_model.Lars`
- :class:`linear_model.Lasso`
- :class:`linear_model.LassoLars`
- :class:`linear_model.LinearRegression`
- :class:`multioutput.MultiOutputRegressor`
- :class:`linear_model.MultiTaskElasticNet`
- :class:`linear_model.MultiTaskElasticNetCV`
- :class:`linear_model.MultiTaskLasso`
- :class:`linear_model.MultiTaskLassoCV`
- :class:`linear_model.OrthogonalMatchingPursuit`
- :class:`cross_decomposition.PLSCanonical`
- :class:`cross_decomposition.PLSRegression`
- :class:`linear_model.RANSACRegressor`
- :class:`neighbors.RadiusNeighborsRegressor`
- :class:`ensemble.RandomForestRegressor`
- :class:`multioutput.RegressorChain`
- :class:`linear_model.Ridge`
- :class:`linear_model.RidgeCV`
- :class:`compose.TransformedTargetRegressor`

Target format
-------------

A valid representation of :term:`multioutput` `y` is a dense matrix of shape
``(n_samples, n_output)`` of floats. A column wise concatenation of
:term:`continuous` variables. An example of ``y`` for 3 samples:

  >>> y = np.array([[31.4, 94], [40.5, 109], [25.0, 30]])
  >>> print(y)
  [[ 31.4  94. ]
   [ 40.5 109. ]
   [ 25.   30. ]]

.. _multioutputregressor:

MultiOutputRegressor
--------------------

Multioutput regression support can be added to any regressor with
:class:`~sklearn.multioutput.MultiOutputRegressor`.  This strategy consists of
fitting one regressor per target. Since each target is represented by exactly
one regressor it is possible to gain knowledge about the target by
inspecting its corresponding regressor. As
:class:`~sklearn.multioutput.MultiOutputRegressor` fits one regressor per
target it can not take advantage of correlations between targets.

Below is an example of multioutput regression:

  >>> from sklearn.datasets import make_regression
  >>> from sklearn.multioutput import MultiOutputRegressor
  >>> from sklearn.ensemble import GradientBoostingRegressor
  >>> X, y = make_regression(n_samples=10, n_targets=3, random_state=1)
  >>> MultiOutputRegressor(GradientBoostingRegressor(random_state=0)).fit(X, y).predict(X)
  array([[-154.75474165, -147.03498585,  -50.03812219],
         [   7.12165031,    5.12914884,  -81.46081961],
         [-187.8948621 , -100.44373091,   13.88978285],
         [-141.62745778,   95.02891072, -191.48204257],
         [  97.03260883,  165.34867495,  139.52003279],
         [ 123.92529176,   21.25719016,   -7.84253   ],
         [-122.25193977,  -85.16443186, -107.12274212],
         [ -30.170388  ,  -94.80956739,   12.16979946],
         [ 140.72667194,  176.50941682,  -17.50447799],
         [ 149.37967282,  -81.15699552,   -5.72850319]])

.. _regressorchain:

RegressorChain
--------------

Regressor chains (see :class:`~sklearn.multioutput.RegressorChain`) is
analogous to :class:`~sklearn.multioutput.ClassifierChain` as a way of
combining a number of regressions into a single multi-target model that is
capable of exploiting correlations among targets.

multiclass                                         Multiclass and multioutput algorithms                                        This section of the user guide covers functionality related to multi learning problems  including  term  multiclass    term  multilabel   and  term  multioutput  classification and regression   The modules in this section implement  term  meta estimators   which require a base estimator to be provided in their constructor  Meta estimators extend the functionality of the base estimator to support multi learning problems  which is accomplished by transforming the multi learning problem into a set of simpler problems  then fitting one estimator per problem   This section covers two modules   mod  sklearn multiclass  and  mod  sklearn multioutput   The chart below demonstrates the problem types that each module is responsible for  and the corresponding meta estimators that each module provides      image      images multi org chart png     align  center  The table below provides a quick reference on the differences between problem types  More detailed explanations can be found in subsequent sections of this guide                                                                                Number of targets       Target cardinality        Valid                                                                                func   sklearn utils multiclass type of target                                                                                Multiclass                      1                       2                         multiclass                                          classification                                                                                Multilabel                      1                       2  0 or 1                 multilabel indicator                                classification                                                                                Multiclass multioutput          1                       2                         multiclass multioutput                              classification                                                                                Multioutput                     1                      Continuous                 continuous multioutput                              regression                                                                                Below is a summary of scikit learn estimators that have multi learning support built in  grouped by strategy  You don t need the meta estimators provided by this section if you re using one of these estimators  However  meta estimators can provide additional strategies beyond what is built in      currentmodule   sklearn      Inherently multiclass          class  naive bayes BernoulliNB       class  tree DecisionTreeClassifier       class  tree ExtraTreeClassifier       class  ensemble ExtraTreesClassifier       class  naive bayes GaussianNB       class  neighbors KNeighborsClassifier       class  semi supervised LabelPropagation       class  semi supervised LabelSpreading       class  discriminant analysis LinearDiscriminantAnalysis       class  svm LinearSVC   setting multi class  crammer singer        class  linear model LogisticRegression   with most solvers       class  linear model LogisticRegressionCV   with most solvers       class  neural network MLPClassifier       class  neighbors NearestCentroid       class  discriminant analysis QuadraticDiscriminantAnalysis       class  neighbors RadiusNeighborsClassifier       class  ensemble RandomForestClassifier       class  linear model RidgeClassifier       class  linear model RidgeClassifierCV        Multiclass as One Vs One          class  svm NuSVC       class  svm SVC        class  gaussian process GaussianProcessClassifier   setting multi class    one vs one         Multiclass as One Vs The Rest          class  ensemble GradientBoostingClassifier       class  gaussian process GaussianProcessClassifier   setting multi class    one vs rest        class  svm LinearSVC   setting multi class  ovr        class  linear model LogisticRegression   most solvers       class  linear model LogisticRegressionCV   most solvers       class  linear model SGDClassifier       class  linear model Perceptron       class  linear model PassiveAggressiveClassifier        Support multilabel          class  tree DecisionTreeClassifier       class  tree ExtraTreeClassifier       class  ensemble ExtraTreesClassifier       class  neighbors KNeighborsClassifier       class  neural network MLPClassifier       class  neighbors RadiusNeighborsClassifier       class  ensemble RandomForestClassifier       class  linear model RidgeClassifier       class  linear model RidgeClassifierCV        Support multiclass multioutput          class  tree DecisionTreeClassifier       class  tree ExtraTreeClassifier       class  ensemble ExtraTreesClassifier       class  neighbors KNeighborsClassifier       class  neighbors RadiusNeighborsClassifier       class  ensemble RandomForestClassifier       multiclass classification   Multiclass classification                               warning       All classifiers in scikit learn do multiclass classification     out of the box  You don t need to use the  mod  sklearn multiclass  module     unless you want to experiment with different multiclass strategies     Multiclass classification   is a classification task with more than two classes  Each sample can only be labeled as one class   For example  classification using features extracted from a set of images of fruit  where each image may either be of an orange  an apple  or a pear  Each image is one sample and is labeled as one of the 3 possible classes  Multiclass classification makes the assumption that each sample is assigned to one and only one label   one sample cannot  for example  be both a pear and an apple   While all scikit learn classifiers are capable of multiclass classification  the meta estimators offered by  mod  sklearn multiclass  permit changing the way they handle more than two classes because this may have an effect on classifier performance  either in terms of generalization error or required computational resources    Target format                Valid  term  multiclass  representations for  func   sklearn utils multiclass type of target    y   are     1d or column vector containing more than two discrete values  An   example of a vector   y   for 4 samples           import numpy as np         y   np array   apple    pear    apple    orange            print y        apple   pear   apple   orange      Dense or sparse  term  binary  matrix of shape    n samples  n classes      with a single sample per row  where each column represents one class  An   example of both a dense and sparse  term  binary  matrix   y   for 4   samples  where the columns  in order  are apple  orange  and pear           import numpy as np         from sklearn preprocessing import LabelBinarizer         y   np array   apple    pear    apple    orange            y dense   LabelBinarizer   fit transform y          print y dense        1 0 0        0 0 1        1 0 0        0 1 0           from scipy import sparse         y sparse   sparse csr matrix y dense          print y sparse       Compressed Sparse Row sparse matrix of dtype  int64       with 4 stored elements and shape  4  3         Coords Values        0  0  1        1  2  1        2  0  1        3  1  1  For more information about  class   sklearn preprocessing LabelBinarizer   refer to  ref  preprocessing targets        ovr classification   OneVsRestClassifier                      The   one vs rest   strategy  also known as   one vs all    is implemented in  class   sklearn multiclass OneVsRestClassifier    The strategy consists in fitting one classifier per class  For each classifier  the class is fitted against all the other classes  In addition to its computational efficiency  only  n classes  classifiers are needed   one advantage of this approach is its interpretability  Since each class is represented by one and only one classifier  it is possible to gain knowledge about the class by inspecting its corresponding classifier  This is the most commonly used strategy and is a fair default choice   Below is an example of multiclass learning using OvR          from sklearn import datasets       from sklearn multiclass import OneVsRestClassifier       from sklearn svm import LinearSVC       X  y   datasets load iris return X y True        OneVsRestClassifier LinearSVC random state 0   fit X  y  predict X    array  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0           0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0           0  0  0  0  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1           1  2  1  1  1  1  1  1  1  1  1  1  1  1  2  2  1  1  1  1  1  1  1           1  1  1  1  1  1  1  1  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2           2  2  2  2  2  2  2  2  2  2  2  2  2  2  1  2  2  2  1  2  2  2  2           2  2  2  2  2  2  2  2  2  2  2  2      class   sklearn multiclass OneVsRestClassifier  also supports multilabel classification  To use this feature  feed the classifier an indicator matrix  in which cell  i  j  indicates the presence of label j in sample i       figure      auto examples miscellaneous images sphx glr plot multilabel 001 png      target     auto examples miscellaneous plot multilabel html      align  center      scale  75       rubric   Examples     ref  sphx glr auto examples miscellaneous plot multilabel py     ref  sphx glr auto examples classification plot classification probability py       ovo classification   OneVsOneClassifier                      class   sklearn multiclass OneVsOneClassifier  constructs one classifier per pair of classes  At prediction time  the class which received the most votes is selected  In the event of a tie  among two classes with an equal number of votes   it selects the class with the highest aggregate classification confidence by summing over the pair wise classification confidence levels computed by the underlying binary classifiers   Since it requires to fit   n classes    n classes   1    2   classifiers  this method is usually slower than one vs the rest  due to its O n classes 2  complexity  However  this method may be advantageous for algorithms such as kernel algorithms which don t scale well with   n samples    This is because each individual learning problem only involves a small subset of the data whereas  with one vs the rest  the complete dataset is used   n classes   times  The decision function is the result of a monotonic transformation of the one versus one classification   Below is an example of multiclass learning using OvO          from sklearn import datasets       from sklearn multiclass import OneVsOneClassifier       from sklearn svm import LinearSVC       X  y   datasets load iris return X y True        OneVsOneClassifier LinearSVC random state 0   fit X  y  predict X    array  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0           0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0           0  0  0  0  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1           1  2  1  2  1  1  1  1  1  1  1  1  1  1  2  1  1  1  1  1  1  1  1           1  1  1  1  1  1  1  1  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2           2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2           2  2  2  2  2  2  2  2  2  2  2  2        rubric   References     Pattern Recognition and Machine Learning  Springer     Christopher M  Bishop  page 183   First Edition       ecoc   OutputCodeClassifier                       Error Correcting Output Code based strategies are fairly different from one vs the rest and one vs one  With these strategies  each class is represented in a Euclidean space  where each dimension can only be 0 or 1  Another way to put it is that each class is represented by a binary code  an array of 0 and 1   The matrix which keeps track of the location code of each class is called the code book  The code size is the dimensionality of the aforementioned space  Intuitively  each class should be represented by a code as unique as possible and a good code book should be designed to optimize classification accuracy  In this implementation  we simply use a randomly generated code book as advocated in  3   although more elaborate methods may be added in the future   At fitting time  one binary classifier per bit in the code book is fitted  At prediction time  the classifiers are used to project new points in the class space and the class closest to the points is chosen   In  class   sklearn multiclass OutputCodeClassifier   the   code size   attribute allows the user to control the number of classifiers which will be used  It is a percentage of the total number of classes   A number between 0 and 1 will require fewer classifiers than one vs the rest  In theory    log2 n classes    n classes   is sufficient to represent each class unambiguously  However  in practice  it may not lead to good accuracy since   log2 n classes    is much smaller than  n classes    A number greater than 1 will require more classifiers than one vs the rest  In this case  some classifiers will in theory correct for the mistakes made by other classifiers  hence the name  error correcting   In practice  however  this may not happen as classifier mistakes will typically be correlated  The error correcting output codes have a similar effect to bagging   Below is an example of multiclass learning using Output Codes          from sklearn import datasets       from sklearn multiclass import OutputCodeClassifier       from sklearn svm import LinearSVC       X  y   datasets load iris return X y True        clf   OutputCodeClassifier LinearSVC random state 0   code size 2  random state 0        clf fit X  y  predict X    array  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0           0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0           0  0  0  0  1  1  1  1  1  1  2  1  1  1  1  1  1  1  1  1  2  1  1           1  2  1  1  1  1  1  1  2  1  1  1  1  1  2  2  2  1  1  1  1  1  1           1  1  1  1  1  1  1  1  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2           2  2  2  2  1  2  2  2  2  2  2  2  2  2  1  2  2  2  1  1  2  2  2           2  2  2  2  2  2  2  2  2  2  2  2       rubric   References     Solving multiclass learning problems via error correcting output codes     Dietterich T   Bakiri G   Journal of Artificial Intelligence Research 2  1995       3   The error coding method and PICTs   James G   Hastie T     Journal of Computational and Graphical statistics 7  1998      The Elements of Statistical Learning     Hastie T   Tibshirani R   Friedman J   page 606  second edition   2008       multilabel classification   Multilabel classification                              Multilabel classification    closely related to   multioutput     classification    is a classification task labeling each sample with   m   labels from   n classes   possible classes  where   m   can be 0 to   n classes   inclusive  This can be thought of as predicting properties of a sample that are not mutually exclusive  Formally  a binary output is assigned to each class  for every sample  Positive classes are indicated with 1 and negative classes with 0 or  1  It is thus comparable to running   n classes   binary classification tasks  for example with  class   sklearn multioutput MultiOutputClassifier   This approach treats each label independently whereas multilabel classifiers  may  treat the multiple classes simultaneously  accounting for correlated behavior among them   For example  prediction of the topics relevant to a text document or video  The document or video may be about one of  religion    politics    finance  or  education   several of the topic classes or all of the topic classes   Target format                A valid representation of  term  multilabel   y  is an either dense or sparse  term  binary  matrix of shape    n samples  n classes     Each column represents a class  The   1   s in each row denote the positive classes a sample has been labeled with  An example of a dense matrix   y   for 3 samples         y   np array   1  0  0  1    0  0  1  1    0  0  0  0          print y      1 0 0 1      0 0 1 1      0 0 0 0    Dense binary matrices can also be created using  class   sklearn preprocessing MultiLabelBinarizer   For more information  refer to  ref  preprocessing targets    An example of the same   y   in sparse matrix form         y sparse   sparse csr matrix y        print y sparse     Compressed Sparse Row sparse matrix of dtype  int64      with 4 stored elements and shape  3  4       Coords Values      0  0  1      0  3  1      1  2  1      1  3  1      multioutputclassfier   MultiOutputClassifier                        Multilabel classification support can be added to any classifier with  class   sklearn multioutput MultiOutputClassifier   This strategy consists of fitting one classifier per target   This allows multiple target variable classifications  The purpose of this class is to extend estimators to be able to estimate a series of target functions  f1 f2 f3    fn  that are trained on a single X predictor matrix to predict a series of responses  y1 y2 y3    yn    You can find a usage example for  class   sklearn multioutput MultiOutputClassifier  as part of the section on  ref  multiclass multioutput classification  since it is a generalization of multilabel classification to multiclass outputs instead of binary outputs       classifierchain   ClassifierChain                  Classifier chains  see  class   sklearn multioutput ClassifierChain   are a way of combining a number of binary classifiers into a single multi label model that is capable of exploiting correlations among targets   For a multi label classification problem with N classes  N binary classifiers are assigned an integer between 0 and N 1  These integers define the order of models in the chain  Each classifier is then fit on the available training data plus the true labels of the classes whose models were assigned a lower number   When predicting  the true labels will not be available  Instead the predictions of each model are passed on to the subsequent models in the chain to be used as features   Clearly the order of the chain is important  The first model in the chain has no information about the other labels while the last model in the chain has features indicating the presence of all of the other labels  In general one does not know the optimal ordering of the models in the chain so typically many randomly ordered chains are fit and their predictions are averaged together      rubric   References    Jesse Read  Bernhard Pfahringer  Geoff Holmes  Eibe Frank     Classifier Chains for Multi label Classification   2009       multiclass multioutput classification   Multiclass multioutput classification                                          Multiclass multioutput classification    also known as   multitask classification    is a classification task which labels each sample with a set of   non binary   properties  Both the number of properties and the number of classes per property is greater than 2  A single estimator thus handles several joint classification tasks  This is both a generalization of the multi   label  classification task  which only considers binary attributes  as well as a generalization of the multi   class  classification task  where only one property is considered   For example  classification of the properties  type of fruit  and  colour  for a set of images of fruit  The property  type of fruit  has the possible classes   apple    pear  and  orange   The property  colour  has the possible classes   green    red    yellow  and  orange   Each sample is an image of a fruit  a label is output for both properties and each label is one of the possible classes of the corresponding property   Note that all classifiers handling multiclass multioutput  also known as multitask classification  tasks  support the multilabel classification task as a special case  Multitask classification is similar to the multioutput classification task with different model formulations  For more information  see the relevant estimator documentation   Below is an example of multiclass multioutput classification           from sklearn datasets import make classification         from sklearn multioutput import MultiOutputClassifier         from sklearn ensemble import RandomForestClassifier         from sklearn utils import shuffle         import numpy as np         X  y1   make classification n samples 10  n features 100                                      n informative 30  n classes 3                                      random state 1          y2   shuffle y1  random state 1          y3   shuffle y1  random state 2          Y   np vstack  y1  y2  y3   T         n samples  n features   X shape   10 100         n outputs   Y shape 1    3         n classes   3         forest   RandomForestClassifier random state 1          multi target forest   MultiOutputClassifier forest  n jobs 2          multi target forest fit X  Y  predict X      array   2  2  0               1  2  1               2  1  0               0  0  2               0  2  1               0  0  2               1  1  0               1  1  1               0  0  2               2  0  0        warning       At present  no metric in  mod  sklearn metrics      supports the multiclass multioutput classification task   Target format                A valid representation of  term  multioutput   y  is a dense matrix of shape    n samples  n classes    of class labels  A column wise concatenation of 1d  term  multiclass  variables  An example of   y   for 3 samples         y   np array    apple    green      orange    orange      pear    green           print y       apple   green        orange   orange        pear   green         multioutput regression   Multioutput regression                           Multioutput regression   predicts multiple numerical properties for each sample  Each property is a numerical variable and the number of properties to be predicted for each sample is greater than or equal to 2  Some estimators that support multioutput regression are faster than just running   n output   estimators   For example  prediction of both wind speed and wind direction  in degrees  using data obtained at a certain location  Each sample would be data obtained at one location and both wind speed and direction would be output for each sample   The following regressors natively support multioutput regression      class  cross decomposition CCA     class  tree DecisionTreeRegressor     class  dummy DummyRegressor     class  linear model ElasticNet     class  tree ExtraTreeRegressor     class  ensemble ExtraTreesRegressor     class  gaussian process GaussianProcessRegressor     class  neighbors KNeighborsRegressor     class  kernel ridge KernelRidge     class  linear model Lars     class  linear model Lasso     class  linear model LassoLars     class  linear model LinearRegression     class  multioutput MultiOutputRegressor     class  linear model MultiTaskElasticNet     class  linear model MultiTaskElasticNetCV     class  linear model MultiTaskLasso     class  linear model MultiTaskLassoCV     class  linear model OrthogonalMatchingPursuit     class  cross decomposition PLSCanonical     class  cross decomposition PLSRegression     class  linear model RANSACRegressor     class  neighbors RadiusNeighborsRegressor     class  ensemble RandomForestRegressor     class  multioutput RegressorChain     class  linear model Ridge     class  linear model RidgeCV     class  compose TransformedTargetRegressor   Target format                A valid representation of  term  multioutput   y  is a dense matrix of shape    n samples  n output    of floats  A column wise concatenation of  term  continuous  variables  An example of   y   for 3 samples         y   np array   31 4  94    40 5  109    25 0  30          print y       31 4  94         40 5 109         25    30          multioutputregressor   MultiOutputRegressor                       Multioutput regression support can be added to any regressor with  class   sklearn multioutput MultiOutputRegressor    This strategy consists of fitting one regressor per target  Since each target is represented by exactly one regressor it is possible to gain knowledge about the target by inspecting its corresponding regressor  As  class   sklearn multioutput MultiOutputRegressor  fits one regressor per target it can not take advantage of correlations between targets   Below is an example of multioutput regression         from sklearn datasets import make regression       from sklearn multioutput import MultiOutputRegressor       from sklearn ensemble import GradientBoostingRegressor       X  y   make regression n samples 10  n targets 3  random state 1        MultiOutputRegressor GradientBoostingRegressor random state 0   fit X  y  predict X    array    154 75474165   147 03498585    50 03812219                7 12165031     5 12914884    81 46081961              187 8948621    100 44373091    13 88978285              141 62745778    95 02891072   191 48204257               97 03260883   165 34867495   139 52003279              123 92529176    21 25719016     7 84253                 122 25193977    85 16443186   107 12274212               30 170388      94 80956739    12 16979946              140 72667194   176 50941682    17 50447799              149 37967282    81 15699552     5 72850319         regressorchain   RegressorChain                 Regressor chains  see  class   sklearn multioutput RegressorChain   is analogous to  class   sklearn multioutput ClassifierChain  as a way of combining a number of regressions into a single multi target model that is capable of exploiting correlations among targets