scikit-learn sklearn naivebayes naivebayes Naive Bayes Naive Bayes methods are a set of supervised learning algorithms

Answer:
.. _naive_bayes:

===========
Naive Bayes
===========

.. currentmodule:: sklearn.naive_bayes

Naive Bayes methods are a set of supervised learning algorithms
based on applying Bayes' theorem with the "naive" assumption of
conditional independence between every pair of features given the
value of the class variable. Bayes' theorem states the following
relationship, given class variable :math:`y` and dependent feature
vector :math:`x_1` through :math:`x_n`, :

.. math::

   P(y \mid x_1, \dots, x_n) = \frac{P(y) P(x_1, \dots, x_n \mid y)}
                                    {P(x_1, \dots, x_n)}

Using the naive conditional independence assumption that

.. math::

   P(x_i | y, x_1, \dots, x_{i-1}, x_{i+1}, \dots, x_n) = P(x_i | y),

for all :math:`i`, this relationship is simplified to

.. math::

   P(y \mid x_1, \dots, x_n) = \frac{P(y) \prod_{i=1}^{n} P(x_i \mid y)}
                                    {P(x_1, \dots, x_n)}

Since :math:`P(x_1, \dots, x_n)` is constant given the input,
we can use the following classification rule:

.. math::

   P(y \mid x_1, \dots, x_n) \propto P(y) \prod_{i=1}^{n} P(x_i \mid y)

   \Downarrow

   \hat{y} = \arg\max_y P(y) \prod_{i=1}^{n} P(x_i \mid y),

and we can use Maximum A Posteriori (MAP) estimation to estimate
:math:`P(y)` and :math:`P(x_i \mid y)`;
the former is then the relative frequency of class :math:`y`
in the training set.

The different naive Bayes classifiers differ mainly by the assumptions they
make regarding the distribution of :math:`P(x_i \mid y)`.

In spite of their apparently over-simplified assumptions, naive Bayes
classifiers have worked quite well in many real-world situations, famously
document classification and spam filtering. They require a small amount
of training data to estimate the necessary parameters. (For theoretical
reasons why naive Bayes works well, and on which types of data it does, see
the references below.)

Naive Bayes learners and classifiers can be extremely fast compared to more
sophisticated methods.
The decoupling of the class conditional feature distributions means that each
distribution can be independently estimated as a one dimensional distribution.
This in turn helps to alleviate problems stemming from the curse of
dimensionality.

On the flip side, although naive Bayes is known as a decent classifier,
it is known to be a bad estimator, so the probability outputs from
``predict_proba`` are not to be taken too seriously.

.. dropdown:: References

   * H. Zhang (2004). `The optimality of Naive Bayes.
     <https://www.cs.unb.ca/~hzhang/publications/FLAIRS04ZhangH.pdf>`_
     Proc. FLAIRS.

.. _gaussian_naive_bayes:

Gaussian Naive Bayes
--------------------

:class:`GaussianNB` implements the Gaussian Naive Bayes algorithm for
classification. The likelihood of the features is assumed to be Gaussian:

.. math::

   P(x_i \mid y) = \frac{1}{\sqrt{2\pi\sigma^2_y}} \exp\left(-\frac{(x_i - \mu_y)^2}{2\sigma^2_y}\right)

The parameters :math:`\sigma_y` and :math:`\mu_y`
are estimated using maximum likelihood.

   >>> from sklearn.datasets import load_iris
   >>> from sklearn.model_selection import train_test_split
   >>> from sklearn.naive_bayes import GaussianNB
   >>> X, y = load_iris(return_X_y=True)
   >>> X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)
   >>> gnb = GaussianNB()
   >>> y_pred = gnb.fit(X_train, y_train).predict(X_test)
   >>> print("Number of mislabeled points out of a total %d points : %d"
   ...       % (X_test.shape[0], (y_test != y_pred).sum()))
   Number of mislabeled points out of a total 75 points : 4

.. _multinomial_naive_bayes:

Multinomial Naive Bayes
-----------------------

:class:`MultinomialNB` implements the naive Bayes algorithm for multinomially
distributed data, and is one of the two classic naive Bayes variants used in
text classification (where the data are typically represented as word vector
counts, although tf-idf vectors are also known to work well in practice).
The distribution is parametrized by vectors
:math:`\theta_y = (\theta_{y1},\ldots,\theta_{yn})`
for each class :math:`y`, where :math:`n` is the number of features
(in text classification, the size of the vocabulary)
and :math:`\theta_{yi}` is the probability :math:`P(x_i \mid y)`
of feature :math:`i` appearing in a sample belonging to class :math:`y`.

The parameters :math:`\theta_y` is estimated by a smoothed
version of maximum likelihood, i.e. relative frequency counting:

.. math::

    \hat{\theta}_{yi} = \frac{ N_{yi} + \alpha}{N_y + \alpha n}

where :math:`N_{yi} = \sum_{x \in T} x_i` is
the number of times feature :math:`i` appears in all samples of class :math:`y`
in the training set :math:`T`,
and :math:`N_{y} = \sum_{i=1}^{n} N_{yi}` is the total count of
all features for class :math:`y`.

The smoothing priors :math:`\alpha \ge 0` accounts for
features not present in the learning samples and prevents zero probabilities
in further computations.
Setting :math:`\alpha = 1` is called Laplace smoothing,
while :math:`\alpha < 1` is called Lidstone smoothing.

.. _complement_naive_bayes:

Complement Naive Bayes
----------------------

:class:`ComplementNB` implements the complement naive Bayes (CNB) algorithm.
CNB is an adaptation of the standard multinomial naive Bayes (MNB) algorithm
that is particularly suited for imbalanced data sets. Specifically, CNB uses
statistics from the *complement* of each class to compute the model's weights.
The inventors of CNB show empirically that the parameter estimates for CNB are
more stable than those for MNB. Further, CNB regularly outperforms MNB (often
by a considerable margin) on text classification tasks.

.. dropdown:: Weights calculation

   The procedure for calculating the weights is as follows:

   .. math::

      \hat{\theta}_{ci} = \frac{\alpha_i + \sum_{j:y_j \neq c} d_{ij}}
                              {\alpha + \sum_{j:y_j \neq c} \sum_{k} d_{kj}}

      w_{ci} = \log \hat{\theta}_{ci}

      w_{ci} = \frac{w_{ci}}{\sum_{j} |w_{cj}|}

   where the summations are over all documents :math:`j` not in class :math:`c`,
   :math:`d_{ij}` is either the count or tf-idf value of term :math:`i` in document
   :math:`j`, :math:`\alpha_i` is a smoothing hyperparameter like that found in
   MNB, and :math:`\alpha = \sum_{i} \alpha_i`. The second normalization addresses
   the tendency for longer documents to dominate parameter estimates in MNB. The
   classification rule is:

   .. math::

      \hat{c} = \arg\min_c \sum_{i} t_i w_{ci}

   i.e., a document is assigned to the class that is the *poorest* complement
   match.

.. dropdown:: References

   * Rennie, J. D., Shih, L., Teevan, J., & Karger, D. R. (2003).
     `Tackling the poor assumptions of naive bayes text classifiers.
     <https://people.csail.mit.edu/jrennie/papers/icml03-nb.pdf>`_
     In ICML (Vol. 3, pp. 616-623).

.. _bernoulli_naive_bayes:

Bernoulli Naive Bayes
---------------------

:class:`BernoulliNB` implements the naive Bayes training and classification
algorithms for data that is distributed according to multivariate Bernoulli
distributions; i.e., there may be multiple features but each one is assumed
to be a binary-valued (Bernoulli, boolean) variable.
Therefore, this class requires samples to be represented as binary-valued
feature vectors; if handed any other kind of data, a :class:`BernoulliNB` instance
may binarize its input (depending on the ``binarize`` parameter).

The decision rule for Bernoulli naive Bayes is based on

.. math::

    P(x_i \mid y) = P(x_i = 1 \mid y) x_i + (1 - P(x_i = 1 \mid y)) (1 - x_i)

which differs from multinomial NB's rule
in that it explicitly penalizes the non-occurrence of a feature :math:`i`
that is an indicator for class :math:`y`,
where the multinomial variant would simply ignore a non-occurring feature.

In the case of text classification, word occurrence vectors (rather than word
count vectors) may be used to train and use this classifier. :class:`BernoulliNB`
might perform better on some datasets, especially those with shorter documents.
It is advisable to evaluate both models, if time permits.

.. dropdown:: References

   * C.D. Manning, P. Raghavan and H. Schütze (2008). Introduction to
     Information Retrieval. Cambridge University Press, pp. 234-265.

   * A. McCallum and K. Nigam (1998).
     `A comparison of event models for Naive Bayes text classification.
     <https://citeseerx.ist.psu.edu/doc_view/pid/04ce064505b1635583fa0d9cc07cac7e9ea993cc>`_
     Proc. AAAI/ICML-98 Workshop on Learning for Text Categorization, pp. 41-48.

   * V. Metsis, I. Androutsopoulos and G. Paliouras (2006).
     `Spam filtering with Naive Bayes -- Which Naive Bayes?
     <https://citeseerx.ist.psu.edu/doc_view/pid/8bd0934b366b539ec95e683ae39f8abb29ccc757>`_
     3rd Conf. on Email and Anti-Spam (CEAS).

.. _categorical_naive_bayes:

Categorical Naive Bayes
-----------------------

:class:`CategoricalNB` implements the categorical naive Bayes
algorithm for categorically distributed data. It assumes that each feature,
which is described by the index :math:`i`, has its own categorical
distribution.

For each feature :math:`i` in the training set :math:`X`,
:class:`CategoricalNB` estimates a categorical distribution for each feature i
of X conditioned on the class y. The index set of the samples is defined as
:math:`J = \{ 1, \dots, m \}`, with :math:`m` as the number of samples.

.. dropdown:: Probability calculation

   The probability of category :math:`t` in feature :math:`i` given class
   :math:`c` is estimated as:

   .. math::

      P(x_i = t \mid y = c \: ;\, \alpha) = \frac{ N_{tic} + \alpha}{N_{c} +
                                             \alpha n_i},

   where :math:`N_{tic} = |\{j \in J \mid x_{ij} = t, y_j = c\}|` is the number
   of times category :math:`t` appears in the samples :math:`x_{i}`, which belong
   to class :math:`c`, :math:`N_{c} = |\{ j \in J\mid y_j = c\}|` is the number
   of samples with class c, :math:`\alpha` is a smoothing parameter and
   :math:`n_i` is the number of available categories of feature :math:`i`.

:class:`CategoricalNB` assumes that the sample matrix :math:`X` is encoded (for
instance with the help of :class:`~sklearn.preprocessing.OrdinalEncoder`) such
that all categories for each feature :math:`i` are represented with numbers
:math:`0, ..., n_i - 1` where :math:`n_i` is the number of available categories
of feature :math:`i`.

Out-of-core naive Bayes model fitting
-------------------------------------

Naive Bayes models can be used to tackle large scale classification problems
for which the full training set might not fit in memory. To handle this case,
:class:`MultinomialNB`, :class:`BernoulliNB`, and :class:`GaussianNB`
expose a ``partial_fit`` method that can be used
incrementally as done with other classifiers as demonstrated in
:ref:`sphx_glr_auto_examples_applications_plot_out_of_core_classification.py`. All naive Bayes
classifiers support sample weighting.

Contrary to the ``fit`` method, the first call to ``partial_fit`` needs to be
passed the list of all the expected class labels.

For an overview of available strategies in scikit-learn, see also the
:ref:`out-of-core learning <scaling_strategies>` documentation.

.. note::

   The ``partial_fit`` method call of naive Bayes models introduces some
   computational overhead. It is recommended to use data chunk sizes that are as
   large as possible, that is as the available RAM allows.

naive bayes               Naive Bayes                 currentmodule   sklearn naive bayes   Naive Bayes methods are a set of supervised learning algorithms based on applying Bayes  theorem with the  naive  assumption of conditional independence between every pair of features given the value of the class variable  Bayes  theorem states the following relationship  given class variable  math  y  and dependent feature vector  math  x 1  through  math  x n         math       P y  mid x 1   dots  x n     frac P y  P x 1   dots  x n  mid y                                        P x 1   dots  x n    Using the naive conditional independence assumption that     math       P x i   y  x 1   dots  x  i 1   x  i 1    dots  x n    P x i   y    for all  math  i   this relationship is simplified to     math       P y  mid x 1   dots  x n     frac P y   prod  i 1   n  P x i  mid y                                        P x 1   dots  x n    Since  math  P x 1   dots  x n   is constant given the input  we can use the following classification rule      math       P y  mid x 1   dots  x n   propto P y   prod  i 1   n  P x i  mid y       Downarrow      hat y     arg max y P y   prod  i 1   n  P x i  mid y    and we can use Maximum A Posteriori  MAP  estimation to estimate  math  P y   and  math  P x i  mid y    the former is then the relative frequency of class  math  y  in the training set   The different naive Bayes classifiers differ mainly by the assumptions they make regarding the distribution of  math  P x i  mid y     In spite of their apparently over simplified assumptions  naive Bayes classifiers have worked quite well in many real world situations  famously document classification and spam filtering  They require a small amount of training data to estimate the necessary parameters   For theoretical reasons why naive Bayes works well  and on which types of data it does  see the references below    Naive Bayes learners and classifiers can be extremely fast compared to more sophisticated methods  The decoupling of the class conditional feature distributions means that each distribution can be independently estimated as a one dimensional distribution  This in turn helps to alleviate problems stemming from the curse of dimensionality   On the flip side  although naive Bayes is known as a decent classifier  it is known to be a bad estimator  so the probability outputs from   predict proba   are not to be taken too seriously      dropdown   References       H  Zhang  2004    The optimality of Naive Bayes        https   www cs unb ca  hzhang publications FLAIRS04ZhangH pdf         Proc  FLAIRS       gaussian naive bayes   Gaussian Naive Bayes                        class  GaussianNB  implements the Gaussian Naive Bayes algorithm for classification  The likelihood of the features is assumed to be Gaussian      math       P x i  mid y     frac 1   sqrt 2 pi sigma 2 y    exp left   frac  x i    mu y  2  2 sigma 2 y  right   The parameters  math   sigma y  and  math   mu y  are estimated using maximum likelihood          from sklearn datasets import load iris        from sklearn model selection import train test split        from sklearn naive bayes import GaussianNB        X  y   load iris return X y True         X train  X test  y train  y test   train test split X  y  test size 0 5  random state 0         gnb   GaussianNB          y pred   gnb fit X train  y train  predict X test         print  Number of mislabeled points out of a total  d points    d                  X test shape 0    y test    y pred  sum        Number of mislabeled points out of a total 75 points   4      multinomial naive bayes   Multinomial Naive Bayes                           class  MultinomialNB  implements the naive Bayes algorithm for multinomially distributed data  and is one of the two classic naive Bayes variants used in text classification  where the data are typically represented as word vector counts  although tf idf vectors are also known to work well in practice   The distribution is parametrized by vectors  math   theta y     theta  y1   ldots  theta  yn    for each class  math  y   where  math  n  is the number of features  in text classification  the size of the vocabulary  and  math   theta  yi   is the probability  math  P x i  mid y   of feature  math  i  appearing in a sample belonging to class  math  y    The parameters  math   theta y  is estimated by a smoothed version of maximum likelihood  i e  relative frequency counting      math         hat  theta   yi     frac  N  yi     alpha  N y    alpha n   where  math  N  yi     sum  x  in T  x i  is the number of times feature  math  i  appears in all samples of class  math  y  in the training set  math  T   and  math  N  y     sum  i 1   n  N  yi   is the total count of all features for class  math  y    The smoothing priors  math   alpha  ge 0  accounts for features not present in the learning samples and prevents zero probabilities in further computations  Setting  math   alpha   1  is called Laplace smoothing  while  math   alpha   1  is called Lidstone smoothing       complement naive bayes   Complement Naive Bayes                          class  ComplementNB  implements the complement naive Bayes  CNB  algorithm  CNB is an adaptation of the standard multinomial naive Bayes  MNB  algorithm that is particularly suited for imbalanced data sets  Specifically  CNB uses statistics from the  complement  of each class to compute the model s weights  The inventors of CNB show empirically that the parameter estimates for CNB are more stable than those for MNB  Further  CNB regularly outperforms MNB  often by a considerable margin  on text classification tasks      dropdown   Weights calculation     The procedure for calculating the weights is as follows         math           hat  theta   ci     frac  alpha i    sum  j y j  neq c  d  ij                                   alpha    sum  j y j  neq c   sum  k  d  kj          w  ci     log  hat  theta   ci         w  ci     frac w  ci    sum  j   w  cj        where the summations are over all documents  math  j  not in class  math  c       math  d  ij   is either the count or tf idf value of term  math  i  in document     math  j    math   alpha i  is a smoothing hyperparameter like that found in    MNB  and  math   alpha    sum  i   alpha i   The second normalization addresses    the tendency for longer documents to dominate parameter estimates in MNB  The    classification rule is         math           hat c     arg min c  sum  i  t i w  ci      i e   a document is assigned to the class that is the  poorest  complement    match      dropdown   References       Rennie  J  D   Shih  L   Teevan  J     Karger  D  R   2003         Tackling the poor assumptions of naive bayes text classifiers        https   people csail mit edu jrennie papers icml03 nb pdf         In ICML  Vol  3  pp  616 623         bernoulli naive bayes   Bernoulli Naive Bayes                         class  BernoulliNB  implements the naive Bayes training and classification algorithms for data that is distributed according to multivariate Bernoulli distributions  i e   there may be multiple features but each one is assumed to be a binary valued  Bernoulli  boolean  variable  Therefore  this class requires samples to be represented as binary valued feature vectors  if handed any other kind of data  a  class  BernoulliNB  instance may binarize its input  depending on the   binarize   parameter    The decision rule for Bernoulli naive Bayes is based on     math        P x i  mid y    P x i   1  mid y  x i    1   P x i   1  mid y    1   x i   which differs from multinomial NB s rule in that it explicitly penalizes the non occurrence of a feature  math  i  that is an indicator for class  math  y   where the multinomial variant would simply ignore a non occurring feature   In the case of text classification  word occurrence vectors  rather than word count vectors  may be used to train and use this classifier   class  BernoulliNB  might perform better on some datasets  especially those with shorter documents  It is advisable to evaluate both models  if time permits      dropdown   References       C D  Manning  P  Raghavan and H  Sch tze  2008   Introduction to      Information Retrieval  Cambridge University Press  pp  234 265        A  McCallum and K  Nigam  1998         A comparison of event models for Naive Bayes text classification        https   citeseerx ist psu edu doc view pid 04ce064505b1635583fa0d9cc07cac7e9ea993cc         Proc  AAAI ICML 98 Workshop on Learning for Text Categorization  pp  41 48        V  Metsis  I  Androutsopoulos and G  Paliouras  2006         Spam filtering with Naive Bayes    Which Naive Bayes        https   citeseerx ist psu edu doc view pid 8bd0934b366b539ec95e683ae39f8abb29ccc757         3rd Conf  on Email and Anti Spam  CEAS         categorical naive bayes   Categorical Naive Bayes                           class  CategoricalNB  implements the categorical naive Bayes algorithm for categorically distributed data  It assumes that each feature  which is described by the index  math  i   has its own categorical distribution   For each feature  math  i  in the training set  math  X    class  CategoricalNB  estimates a categorical distribution for each feature i of X conditioned on the class y  The index set of the samples is defined as  math  J      1   dots  m      with  math  m  as the number of samples      dropdown   Probability calculation     The probability of category  math  t  in feature  math  i  given class     math  c  is estimated as         math          P x i   t  mid y   c         alpha     frac  N  tic     alpha  N  c                                                  alpha n i       where  math  N  tic       j  in J  mid x  ij    t  y j   c     is the number    of times category  math  t  appears in the samples  math  x  i    which belong    to class  math  c    math  N  c        j  in J mid y j   c     is the number    of samples with class c   math   alpha  is a smoothing parameter and     math  n i  is the number of available categories of feature  math  i     class  CategoricalNB  assumes that the sample matrix  math  X  is encoded  for instance with the help of  class   sklearn preprocessing OrdinalEncoder   such that all categories for each feature  math  i  are represented with numbers  math  0       n i   1  where  math  n i  is the number of available categories of feature  math  i    Out of core naive Bayes model fitting                                        Naive Bayes models can be used to tackle large scale classification problems for which the full training set might not fit in memory  To handle this case   class  MultinomialNB    class  BernoulliNB   and  class  GaussianNB  expose a   partial fit   method that can be used incrementally as done with other classifiers as demonstrated in  ref  sphx glr auto examples applications plot out of core classification py   All naive Bayes classifiers support sample weighting   Contrary to the   fit   method  the first call to   partial fit   needs to be passed the list of all the expected class labels   For an overview of available strategies in scikit learn  see also the  ref  out of core learning  scaling strategies   documentation      note       The   partial fit   method call of naive Bayes models introduces some    computational overhead  It is recommended to use data chunk sizes that are as    large as possible  that is as the available RAM allows


scikit-learn The cross decomposition module contains supervised estimators for crossdecomposition Cross decomposition sklearn crossdecomposition dimensionality reduction and regression belonging to the Partial Least

Answer:
.. _cross_decomposition:

===================
Cross decomposition
===================

.. currentmodule:: sklearn.cross_decomposition

The cross decomposition module contains **supervised** estimators for
dimensionality reduction and regression, belonging to the "Partial Least
Squares" family.

.. figure:: ../auto_examples/cross_decomposition/images/sphx_glr_plot_compare_cross_decomposition_001.png
   :target: ../auto_examples/cross_decomposition/plot_compare_cross_decomposition.html
   :scale: 75%
   :align: center

Cross decomposition algorithms find the fundamental relations between two
matrices (X and Y). They are latent variable approaches to modeling the
covariance structures in these two spaces. They will try to find the
multidimensional direction in the X space that explains the maximum
multidimensional variance direction in the Y space. In other words, PLS
projects both `X` and `Y` into a lower-dimensional subspace such that the
covariance between `transformed(X)` and `transformed(Y)` is maximal.

PLS draws similarities with `Principal Component Regression
<https://en.wikipedia.org/wiki/Principal_component_regression>`_ (PCR), where
the samples are first projected into a lower-dimensional subspace, and the
targets `y` are predicted using `transformed(X)`. One issue with PCR is that
the dimensionality reduction is unsupervised, and may lose some important
variables: PCR would keep the features with the most variance, but it's
possible that features with a small variances are relevant from predicting
the target. In a way, PLS allows for the same kind of dimensionality
reduction, but by taking into account the targets `y`. An illustration of
this fact is given in the following example:
* :ref:`sphx_glr_auto_examples_cross_decomposition_plot_pcr_vs_pls.py`.

Apart from CCA, the PLS estimators are particularly suited when the matrix of
predictors has more variables than observations, and when there is
multicollinearity among the features. By contrast, standard linear regression
would fail in these cases unless it is regularized.

Classes included in this module are :class:`PLSRegression`,
:class:`PLSCanonical`, :class:`CCA` and :class:`PLSSVD`

PLSCanonical
------------

We here describe the algorithm used in :class:`PLSCanonical`. The other
estimators use variants of this algorithm, and are detailed below.
We recommend section [1]_ for more details and comparisons between these
algorithms. In [1]_, :class:`PLSCanonical` corresponds to "PLSW2A".

Given two centered matrices :math:`X \in \mathbb{R}^{n \times d}` and
:math:`Y \in \mathbb{R}^{n \times t}`, and a number of components :math:`K`,
:class:`PLSCanonical` proceeds as follows:

Set :math:`X_1` to :math:`X` and :math:`Y_1` to :math:`Y`. Then, for each
:math:`k \in [1, K]`:

- a) compute :math:`u_k \in \mathbb{R}^d` and :math:`v_k \in \mathbb{R}^t`,
  the first left and right singular vectors of the cross-covariance matrix
  :math:`C = X_k^T Y_k`.
  :math:`u_k` and :math:`v_k` are called the *weights*.
  By definition, :math:`u_k` and :math:`v_k` are
  chosen so that they maximize the covariance between the projected
  :math:`X_k` and the projected target, that is :math:`\text{Cov}(X_k u_k,
  Y_k v_k)`.
- b) Project :math:`X_k` and :math:`Y_k` on the singular vectors to obtain
  *scores*: :math:`\xi_k = X_k u_k` and :math:`\omega_k = Y_k v_k`
- c) Regress :math:`X_k` on :math:`\xi_k`, i.e. find a vector :math:`\gamma_k
  \in \mathbb{R}^d` such that the rank-1 matrix :math:`\xi_k \gamma_k^T`
  is as close as possible to :math:`X_k`. Do the same on :math:`Y_k` with
  :math:`\omega_k` to obtain :math:`\delta_k`. The vectors
  :math:`\gamma_k` and :math:`\delta_k` are called the *loadings*.
- d) *deflate* :math:`X_k` and :math:`Y_k`, i.e. subtract the rank-1
  approximations: :math:`X_{k+1} = X_k - \xi_k \gamma_k^T`, and
  :math:`Y_{k + 1} = Y_k - \omega_k \delta_k^T`.

At the end, we have approximated :math:`X` as a sum of rank-1 matrices:
:math:`X = \Xi \Gamma^T` where :math:`\Xi \in \mathbb{R}^{n \times K}`
contains the scores in its columns, and :math:`\Gamma^T \in \mathbb{R}^{K
\times d}` contains the loadings in its rows. Similarly for :math:`Y`, we
have :math:`Y = \Omega \Delta^T`.

Note that the scores matrices :math:`\Xi` and :math:`\Omega` correspond to
the projections of the training data :math:`X` and :math:`Y`, respectively.

Step *a)* may be performed in two ways: either by computing the whole SVD of
:math:`C` and only retain the singular vectors with the biggest singular
values, or by directly computing the singular vectors using the power method (cf section 11.3 in [1]_),
which corresponds to the `'nipals'` option of the `algorithm` parameter.

.. dropdown:: Transforming data

  To transform :math:`X` into :math:`\bar{X}`, we need to find a projection
  matrix :math:`P` such that :math:`\bar{X} = XP`. We know that for the
  training data, :math:`\Xi = XP`, and :math:`X = \Xi \Gamma^T`. Setting
  :math:`P = U(\Gamma^T U)^{-1}` where :math:`U` is the matrix with the
  :math:`u_k` in the columns, we have :math:`XP = X U(\Gamma^T U)^{-1} = \Xi
  (\Gamma^T U) (\Gamma^T U)^{-1} = \Xi` as desired. The rotation matrix
  :math:`P` can be accessed from the `x_rotations_` attribute.

  Similarly, :math:`Y` can be transformed using the rotation matrix
  :math:`V(\Delta^T V)^{-1}`, accessed via the `y_rotations_` attribute.

.. dropdown:: Predicting the targets `Y`

  To predict the targets of some data :math:`X`, we are looking for a
  coefficient matrix :math:`\beta \in R^{d \times t}` such that :math:`Y =
  X\beta`.

  The idea is to try to predict the transformed targets :math:`\Omega` as a
  function of the transformed samples :math:`\Xi`, by computing :math:`\alpha
  \in \mathbb{R}` such that :math:`\Omega = \alpha \Xi`.

  Then, we have :math:`Y = \Omega \Delta^T = \alpha \Xi \Delta^T`, and since
  :math:`\Xi` is the transformed training data we have that :math:`Y = X \alpha
  P \Delta^T`, and as a result the coefficient matrix :math:`\beta = \alpha P
  \Delta^T`.

  :math:`\beta` can be accessed through the `coef_` attribute.

PLSSVD
------

:class:`PLSSVD` is a simplified version of :class:`PLSCanonical`
described earlier: instead of iteratively deflating the matrices :math:`X_k`
and :math:`Y_k`, :class:`PLSSVD` computes the SVD of :math:`C = X^TY`
only *once*, and stores the `n_components` singular vectors corresponding to
the biggest singular values in the matrices `U` and `V`, corresponding to the
`x_weights_` and `y_weights_` attributes. Here, the transformed data is
simply `transformed(X) = XU` and `transformed(Y) = YV`.

If `n_components == 1`, :class:`PLSSVD` and :class:`PLSCanonical` are
strictly equivalent.

PLSRegression
-------------

The :class:`PLSRegression` estimator is similar to
:class:`PLSCanonical` with `algorithm='nipals'`, with 2 significant
differences:

- at step a) in the power method to compute :math:`u_k` and :math:`v_k`,
  :math:`v_k` is never normalized.
- at step c), the targets :math:`Y_k` are approximated using the projection
  of :math:`X_k` (i.e. :math:`\xi_k`) instead of the projection of
  :math:`Y_k` (i.e. :math:`\omega_k`). In other words, the loadings
  computation is different. As a result, the deflation in step d) will also
  be affected.

These two modifications affect the output of `predict` and `transform`,
which are not the same as for :class:`PLSCanonical`. Also, while the number
of components is limited by `min(n_samples, n_features, n_targets)` in
:class:`PLSCanonical`, here the limit is the rank of :math:`X^TX`, i.e.
`min(n_samples, n_features)`.

:class:`PLSRegression` is also known as PLS1 (single targets) and PLS2
(multiple targets). Much like :class:`~sklearn.linear_model.Lasso`,
:class:`PLSRegression` is a form of regularized linear regression where the
number of components controls the strength of the regularization.

Canonical Correlation Analysis
------------------------------

Canonical Correlation Analysis was developed prior and independently to PLS.
But it turns out that :class:`CCA` is a special case of PLS, and corresponds
to PLS in "Mode B" in the literature.

:class:`CCA` differs from :class:`PLSCanonical` in the way the weights
:math:`u_k` and :math:`v_k` are computed in the power method of step a).
Details can be found in section 10 of [1]_.

Since :class:`CCA` involves the inversion of :math:`X_k^TX_k` and
:math:`Y_k^TY_k`, this estimator can be unstable if the number of features or
targets is greater than the number of samples.

.. rubric:: References

.. [1] `A survey of Partial Least Squares (PLS) methods, with emphasis on the two-block
  case <https://stat.uw.edu/sites/default/files/files/reports/2000/tr371.pdf>`_,
  JA Wegelin

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_cross_decomposition_plot_compare_cross_decomposition.py`
* :ref:`sphx_glr_auto_examples_cross_decomposition_plot_pcr_vs_pls.py`

cross decomposition                       Cross decomposition                         currentmodule   sklearn cross decomposition  The cross decomposition module contains   supervised   estimators for dimensionality reduction and regression  belonging to the  Partial Least Squares  family      figure      auto examples cross decomposition images sphx glr plot compare cross decomposition 001 png     target     auto examples cross decomposition plot compare cross decomposition html     scale  75      align  center   Cross decomposition algorithms find the fundamental relations between two matrices  X and Y   They are latent variable approaches to modeling the covariance structures in these two spaces  They will try to find the multidimensional direction in the X space that explains the maximum multidimensional variance direction in the Y space  In other words  PLS projects both  X  and  Y  into a lower dimensional subspace such that the covariance between  transformed X   and  transformed Y   is maximal   PLS draws similarities with  Principal Component Regression  https   en wikipedia org wiki Principal component regression     PCR   where the samples are first projected into a lower dimensional subspace  and the targets  y  are predicted using  transformed X    One issue with PCR is that the dimensionality reduction is unsupervised  and may lose some important variables  PCR would keep the features with the most variance  but it s possible that features with a small variances are relevant from predicting the target  In a way  PLS allows for the same kind of dimensionality reduction  but by taking into account the targets  y   An illustration of this fact is given in the following example     ref  sphx glr auto examples cross decomposition plot pcr vs pls py    Apart from CCA  the PLS estimators are particularly suited when the matrix of predictors has more variables than observations  and when there is multicollinearity among the features  By contrast  standard linear regression would fail in these cases unless it is regularized   Classes included in this module are  class  PLSRegression    class  PLSCanonical    class  CCA  and  class  PLSSVD   PLSCanonical               We here describe the algorithm used in  class  PLSCanonical   The other estimators use variants of this algorithm  and are detailed below  We recommend section  1   for more details and comparisons between these algorithms  In  1     class  PLSCanonical  corresponds to  PLSW2A    Given two centered matrices  math  X  in  mathbb R   n  times d   and  math  Y  in  mathbb R   n  times t    and a number of components  math  K    class  PLSCanonical  proceeds as follows   Set  math  X 1  to  math  X  and  math  Y 1  to  math  Y   Then  for each  math  k  in  1  K       a  compute  math  u k  in  mathbb R  d  and  math  v k  in  mathbb R  t     the first left and right singular vectors of the cross covariance matrix    math  C   X k T Y k      math  u k  and  math  v k  are called the  weights     By definition   math  u k  and  math  v k  are   chosen so that they maximize the covariance between the projected    math  X k  and the projected target  that is  math   text Cov  X k u k    Y k v k      b  Project  math  X k  and  math  Y k  on the singular vectors to obtain    scores    math   xi k   X k u k  and  math   omega k   Y k v k    c  Regress  math  X k  on  math   xi k   i e  find a vector  math   gamma k    in  mathbb R  d  such that the rank 1 matrix  math   xi k  gamma k T    is as close as possible to  math  X k   Do the same on  math  Y k  with    math   omega k  to obtain  math   delta k   The vectors    math   gamma k  and  math   delta k  are called the  loadings     d   deflate   math  X k  and  math  Y k   i e  subtract the rank 1   approximations   math  X  k 1    X k    xi k  gamma k T   and    math  Y  k   1    Y k    omega k  delta k T    At the end  we have approximated  math  X  as a sum of rank 1 matrices   math  X    Xi  Gamma T  where  math   Xi  in  mathbb R   n  times K   contains the scores in its columns  and  math   Gamma T  in  mathbb R   K  times d   contains the loadings in its rows  Similarly for  math  Y   we have  math  Y    Omega  Delta T    Note that the scores matrices  math   Xi  and  math   Omega  correspond to the projections of the training data  math  X  and  math  Y   respectively   Step  a   may be performed in two ways  either by computing the whole SVD of  math  C  and only retain the singular vectors with the biggest singular values  or by directly computing the singular vectors using the power method  cf section 11 3 in  1     which corresponds to the   nipals   option of the  algorithm  parameter      dropdown   Transforming data    To transform  math  X  into  math   bar X    we need to find a projection   matrix  math  P  such that  math   bar X    XP   We know that for the   training data   math   Xi   XP   and  math  X    Xi  Gamma T   Setting    math  P   U  Gamma T U    1   where  math  U  is the matrix with the    math  u k  in the columns  we have  math  XP   X U  Gamma T U    1     Xi     Gamma T U    Gamma T U    1     Xi  as desired  The rotation matrix    math  P  can be accessed from the  x rotations   attribute     Similarly   math  Y  can be transformed using the rotation matrix    math  V  Delta T V    1    accessed via the  y rotations   attribute      dropdown   Predicting the targets  Y     To predict the targets of some data  math  X   we are looking for a   coefficient matrix  math   beta  in R  d  times t   such that  math  Y     X beta      The idea is to try to predict the transformed targets  math   Omega  as a   function of the transformed samples  math   Xi   by computing  math   alpha    in  mathbb R   such that  math   Omega    alpha  Xi      Then  we have  math  Y    Omega  Delta T    alpha  Xi  Delta T   and since    math   Xi  is the transformed training data we have that  math  Y   X  alpha   P  Delta T   and as a result the coefficient matrix  math   beta    alpha P    Delta T       math   beta  can be accessed through the  coef   attribute   PLSSVD          class  PLSSVD  is a simplified version of  class  PLSCanonical  described earlier  instead of iteratively deflating the matrices  math  X k  and  math  Y k    class  PLSSVD  computes the SVD of  math  C   X TY  only  once   and stores the  n components  singular vectors corresponding to the biggest singular values in the matrices  U  and  V   corresponding to the  x weights   and  y weights   attributes  Here  the transformed data is simply  transformed X    XU  and  transformed Y    YV    If  n components    1    class  PLSSVD  and  class  PLSCanonical  are strictly equivalent   PLSRegression                The  class  PLSRegression  estimator is similar to  class  PLSCanonical  with  algorithm  nipals    with 2 significant differences     at step a  in the power method to compute  math  u k  and  math  v k      math  v k  is never normalized    at step c   the targets  math  Y k  are approximated using the projection   of  math  X k   i e   math   xi k   instead of the projection of    math  Y k   i e   math   omega k    In other words  the loadings   computation is different  As a result  the deflation in step d  will also   be affected   These two modifications affect the output of  predict  and  transform   which are not the same as for  class  PLSCanonical   Also  while the number of components is limited by  min n samples  n features  n targets   in  class  PLSCanonical   here the limit is the rank of  math  X TX   i e   min n samples  n features      class  PLSRegression  is also known as PLS1  single targets  and PLS2  multiple targets   Much like  class   sklearn linear model Lasso    class  PLSRegression  is a form of regularized linear regression where the number of components controls the strength of the regularization   Canonical Correlation Analysis                                 Canonical Correlation Analysis was developed prior and independently to PLS  But it turns out that  class  CCA  is a special case of PLS  and corresponds to PLS in  Mode B  in the literature    class  CCA  differs from  class  PLSCanonical  in the way the weights  math  u k  and  math  v k  are computed in the power method of step a   Details can be found in section 10 of  1     Since  class  CCA  involves the inversion of  math  X k TX k  and  math  Y k TY k   this estimator can be unstable if the number of features or targets is greater than the number of samples      rubric   References      1   A survey of Partial Least Squares  PLS  methods  with emphasis on the two block   case  https   stat uw edu sites default files files reports 2000 tr371 pdf       JA Wegelin     rubric   Examples     ref  sphx glr auto examples cross decomposition plot compare cross decomposition py     ref  sphx glr auto examples cross decomposition plot pcr vs pls py


scikit-learn for and ref regression tree Decision Trees Decision Trees DTs are a non parametric supervised learning method used sklearn tree

Answer:
.. _tree:

==============
Decision Trees
==============

.. currentmodule:: sklearn.tree

**Decision Trees (DTs)** are a non-parametric supervised learning method used
for :ref:`classification <tree_classification>` and :ref:`regression
<tree_regression>`. The goal is to create a model that predicts the value of a
target variable by learning simple decision rules inferred from the data
features. A tree can be seen as a piecewise constant approximation.

For instance, in the example below, decision trees learn from data to
approximate a sine curve with a set of if-then-else decision rules. The deeper
the tree, the more complex the decision rules and the fitter the model.

.. figure:: ../auto_examples/tree/images/sphx_glr_plot_tree_regression_001.png
   :target: ../auto_examples/tree/plot_tree_regression.html
   :scale: 75
   :align: center

Some advantages of decision trees are:

- Simple to understand and to interpret. Trees can be visualized.

- Requires little data preparation. Other techniques often require data
  normalization, dummy variables need to be created and blank values to
  be removed. Some tree and algorithm combinations support
  :ref:`missing values <tree_missing_value_support>`.

- The cost of using the tree (i.e., predicting data) is logarithmic in the
  number of data points used to train the tree.

- Able to handle both numerical and categorical data. However, the scikit-learn
  implementation does not support categorical variables for now. Other
  techniques are usually specialized in analyzing datasets that have only one type
  of variable. See :ref:`algorithms <tree_algorithms>` for more
  information.

- Able to handle multi-output problems.

- Uses a white box model. If a given situation is observable in a model,
  the explanation for the condition is easily explained by boolean logic.
  By contrast, in a black box model (e.g., in an artificial neural
  network), results may be more difficult to interpret.

- Possible to validate a model using statistical tests. That makes it
  possible to account for the reliability of the model.

- Performs well even if its assumptions are somewhat violated by
  the true model from which the data were generated.

The disadvantages of decision trees include:

- Decision-tree learners can create over-complex trees that do not
  generalize the data well. This is called overfitting. Mechanisms
  such as pruning, setting the minimum number of samples required
  at a leaf node or setting the maximum depth of the tree are
  necessary to avoid this problem.

- Decision trees can be unstable because small variations in the
  data might result in a completely different tree being generated.
  This problem is mitigated by using decision trees within an
  ensemble.

- Predictions of decision trees are neither smooth nor continuous, but
  piecewise constant approximations as seen in the above figure. Therefore,
  they are not good at extrapolation.

- The problem of learning an optimal decision tree is known to be
  NP-complete under several aspects of optimality and even for simple
  concepts. Consequently, practical decision-tree learning algorithms
  are based on heuristic algorithms such as the greedy algorithm where
  locally optimal decisions are made at each node. Such algorithms
  cannot guarantee to return the globally optimal decision tree.  This
  can be mitigated by training multiple trees in an ensemble learner,
  where the features and samples are randomly sampled with replacement.

- There are concepts that are hard to learn because decision trees
  do not express them easily, such as XOR, parity or multiplexer problems.

- Decision tree learners create biased trees if some classes dominate.
  It is therefore recommended to balance the dataset prior to fitting
  with the decision tree.

.. _tree_classification:

Classification
==============

:class:`DecisionTreeClassifier` is a class capable of performing multi-class
classification on a dataset.

As with other classifiers, :class:`DecisionTreeClassifier` takes as input two arrays:
an array X, sparse or dense, of shape ``(n_samples, n_features)`` holding the
training samples, and an array Y of integer values, shape ``(n_samples,)``,
holding the class labels for the training samples::

    >>> from sklearn import tree
    >>> X = [[0, 0], [1, 1]]
    >>> Y = [0, 1]
    >>> clf = tree.DecisionTreeClassifier()
    >>> clf = clf.fit(X, Y)

After being fitted, the model can then be used to predict the class of samples::

    >>> clf.predict([[2., 2.]])
    array([1])

In case that there are multiple classes with the same and highest
probability, the classifier will predict the class with the lowest index
amongst those classes.

As an alternative to outputting a specific class, the probability of each class
can be predicted, which is the fraction of training samples of the class in a
leaf::

    >>> clf.predict_proba([[2., 2.]])
    array([[0., 1.]])

:class:`DecisionTreeClassifier` is capable of both binary (where the
labels are [-1, 1]) classification and multiclass (where the labels are
[0, ..., K-1]) classification.

Using the Iris dataset, we can construct a tree as follows::

    >>> from sklearn.datasets import load_iris
    >>> from sklearn import tree
    >>> iris = load_iris()
    >>> X, y = iris.data, iris.target
    >>> clf = tree.DecisionTreeClassifier()
    >>> clf = clf.fit(X, y)

Once trained, you can plot the tree with the :func:`plot_tree` function::

    >>> tree.plot_tree(clf)
    [...]

.. figure:: ../auto_examples/tree/images/sphx_glr_plot_iris_dtc_002.png
   :target: ../auto_examples/tree/plot_iris_dtc.html
   :scale: 75
   :align: center

.. dropdown:: Alternative ways to export trees

  We can also export the tree in `Graphviz
  <https://www.graphviz.org/>`_ format using the :func:`export_graphviz`
  exporter. If you use the `conda <https://conda.io>`_ package manager, the graphviz binaries
  and the python package can be installed with `conda install python-graphviz`.

  Alternatively binaries for graphviz can be downloaded from the graphviz project homepage,
  and the Python wrapper installed from pypi with `pip install graphviz`.

  Below is an example graphviz export of the above tree trained on the entire
  iris dataset; the results are saved in an output file `iris.pdf`::

      >>> import graphviz # doctest: +SKIP
      >>> dot_data = tree.export_graphviz(clf, out_file=None) # doctest: +SKIP
      >>> graph = graphviz.Source(dot_data) # doctest: +SKIP
      >>> graph.render("iris") # doctest: +SKIP

  The :func:`export_graphviz` exporter also supports a variety of aesthetic
  options, including coloring nodes by their class (or value for regression) and
  using explicit variable and class names if desired. Jupyter notebooks also
  render these plots inline automatically::

      >>> dot_data = tree.export_graphviz(clf, out_file=None, # doctest: +SKIP
      ...                      feature_names=iris.feature_names,  # doctest: +SKIP
      ...                      class_names=iris.target_names,  # doctest: +SKIP
      ...                      filled=True, rounded=True,  # doctest: +SKIP
      ...                      special_characters=True)  # doctest: +SKIP
      >>> graph = graphviz.Source(dot_data)  # doctest: +SKIP
      >>> graph # doctest: +SKIP

  .. only:: html

      .. figure:: ../images/iris.svg
        :align: center

  .. only:: latex

      .. figure:: ../images/iris.pdf
        :align: center

  .. figure:: ../auto_examples/tree/images/sphx_glr_plot_iris_dtc_001.png
    :target: ../auto_examples/tree/plot_iris_dtc.html
    :align: center
    :scale: 75

  Alternatively, the tree can also be exported in textual format with the
  function :func:`export_text`. This method doesn't require the installation
  of external libraries and is more compact:

      >>> from sklearn.datasets import load_iris
      >>> from sklearn.tree import DecisionTreeClassifier
      >>> from sklearn.tree import export_text
      >>> iris = load_iris()
      >>> decision_tree = DecisionTreeClassifier(random_state=0, max_depth=2)
      >>> decision_tree = decision_tree.fit(iris.data, iris.target)
      >>> r = export_text(decision_tree, feature_names=iris['feature_names'])
      >>> print(r)
      |--- petal width (cm) <= 0.80
      |   |--- class: 0
      |--- petal width (cm) >  0.80
      |   |--- petal width (cm) <= 1.75
      |   |   |--- class: 1
      |   |--- petal width (cm) >  1.75
      |   |   |--- class: 2
      <BLANKLINE>

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_tree_plot_iris_dtc.py`
* :ref:`sphx_glr_auto_examples_tree_plot_unveil_tree_structure.py`

.. _tree_regression:

Regression
==========

.. figure:: ../auto_examples/tree/images/sphx_glr_plot_tree_regression_001.png
   :target: ../auto_examples/tree/plot_tree_regression.html
   :scale: 75
   :align: center

Decision trees can also be applied to regression problems, using the
:class:`DecisionTreeRegressor` class.

As in the classification setting, the fit method will take as argument arrays X
and y, only that in this case y is expected to have floating point values
instead of integer values::

    >>> from sklearn import tree
    >>> X = [[0, 0], [2, 2]]
    >>> y = [0.5, 2.5]
    >>> clf = tree.DecisionTreeRegressor()
    >>> clf = clf.fit(X, y)
    >>> clf.predict([[1, 1]])
    array([0.5])

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_tree_plot_tree_regression.py`

.. _tree_multioutput:

Multi-output problems
=====================

A multi-output problem is a supervised learning problem with several outputs
to predict, that is when Y is a 2d array of shape ``(n_samples, n_outputs)``.

When there is no correlation between the outputs, a very simple way to solve
this kind of problem is to build n independent models, i.e. one for each
output, and then to use those models to independently predict each one of the n
outputs. However, because it is likely that the output values related to the
same input are themselves correlated, an often better way is to build a single
model capable of predicting simultaneously all n outputs. First, it requires
lower training time since only a single estimator is built. Second, the
generalization accuracy of the resulting estimator may often be increased.

With regard to decision trees, this strategy can readily be used to support
multi-output problems. This requires the following changes:

- Store n output values in leaves, instead of 1;
- Use splitting criteria that compute the average reduction across all
  n outputs.

This module offers support for multi-output problems by implementing this
strategy in both :class:`DecisionTreeClassifier` and
:class:`DecisionTreeRegressor`. If a decision tree is fit on an output array Y
of shape ``(n_samples, n_outputs)`` then the resulting estimator will:

* Output n_output values upon ``predict``;

* Output a list of n_output arrays of class probabilities upon
  ``predict_proba``.

The use of multi-output trees for regression is demonstrated in
:ref:`sphx_glr_auto_examples_tree_plot_tree_regression.py`. In this example, the input
X is a single real value and the outputs Y are the sine and cosine of X.

.. figure:: ../auto_examples/tree/images/sphx_glr_plot_tree_regression_002.png
   :target: ../auto_examples/tree/plot_tree_regression.html
   :scale: 75
   :align: center

The use of multi-output trees for classification is demonstrated in
:ref:`sphx_glr_auto_examples_miscellaneous_plot_multioutput_face_completion.py`. In this example, the inputs
X are the pixels of the upper half of faces and the outputs Y are the pixels of
the lower half of those faces.

.. figure:: ../auto_examples/miscellaneous/images/sphx_glr_plot_multioutput_face_completion_001.png
   :target: ../auto_examples/miscellaneous/plot_multioutput_face_completion.html
   :scale: 75
   :align: center

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_miscellaneous_plot_multioutput_face_completion.py`

.. rubric:: References

* M. Dumont et al,  `Fast multi-class image annotation with random subwindows
  and multiple output randomized trees
  <http://www.montefiore.ulg.ac.be/services/stochastic/pubs/2009/DMWG09/dumont-visapp09-shortpaper.pdf>`_,
  International Conference on Computer Vision Theory and Applications 2009

.. _tree_complexity:

Complexity
==========

In general, the run time cost to construct a balanced binary tree is
:math:`O(n_{samples}n_{features}\log(n_{samples}))` and query time
:math:`O(\log(n_{samples}))`.  Although the tree construction algorithm attempts
to generate balanced trees, they will not always be balanced.  Assuming that the
subtrees remain approximately balanced, the cost at each node consists of
searching through :math:`O(n_{features})` to find the feature that offers the
largest reduction in the impurity criterion, e.g. log loss (which is equivalent to an
information gain). This has a cost of
:math:`O(n_{features}n_{samples}\log(n_{samples}))` at each node, leading to a
total cost over the entire trees (by summing the cost at each node) of
:math:`O(n_{features}n_{samples}^{2}\log(n_{samples}))`.

Tips on practical use
=====================

* Decision trees tend to overfit on data with a large number of features.
  Getting the right ratio of samples to number of features is important, since
  a tree with few samples in high dimensional space is very likely to overfit.

* Consider performing  dimensionality reduction (:ref:`PCA <PCA>`,
  :ref:`ICA <ICA>`, or :ref:`feature_selection`) beforehand to
  give your tree a better chance of finding features that are discriminative.

* :ref:`sphx_glr_auto_examples_tree_plot_unveil_tree_structure.py` will help
  in gaining more insights about how the decision tree makes predictions, which is
  important for understanding the important features in the data.

* Visualize your tree as you are training by using the ``export``
  function.  Use ``max_depth=3`` as an initial tree depth to get a feel for
  how the tree is fitting to your data, and then increase the depth.

* Remember that the number of samples required to populate the tree doubles
  for each additional level the tree grows to.  Use ``max_depth`` to control
  the size of the tree to prevent overfitting.

* Use ``min_samples_split`` or ``min_samples_leaf`` to ensure that multiple
  samples inform every decision in the tree, by controlling which splits will
  be considered. A very small number will usually mean the tree will overfit,
  whereas a large number will prevent the tree from learning the data. Try
  ``min_samples_leaf=5`` as an initial value. If the sample size varies
  greatly, a float number can be used as percentage in these two parameters.
  While ``min_samples_split`` can create arbitrarily small leaves,
  ``min_samples_leaf`` guarantees that each leaf has a minimum size, avoiding
  low-variance, over-fit leaf nodes in regression problems.  For
  classification with few classes, ``min_samples_leaf=1`` is often the best
  choice.

  Note that ``min_samples_split`` considers samples directly and independent of
  ``sample_weight``, if provided (e.g. a node with m weighted samples is still
  treated as having exactly m samples). Consider ``min_weight_fraction_leaf`` or
  ``min_impurity_decrease`` if accounting for sample weights is required at splits.

* Balance your dataset before training to prevent the tree from being biased
  toward the classes that are dominant. Class balancing can be done by
  sampling an equal number of samples from each class, or preferably by
  normalizing the sum of the sample weights (``sample_weight``) for each
  class to the same value. Also note that weight-based pre-pruning criteria,
  such as ``min_weight_fraction_leaf``, will then be less biased toward
  dominant classes than criteria that are not aware of the sample weights,
  like ``min_samples_leaf``.

* If the samples are weighted, it will be easier to optimize the tree
  structure using weight-based pre-pruning criterion such as
  ``min_weight_fraction_leaf``, which ensure that leaf nodes contain at least
  a fraction of the overall sum of the sample weights.

* All decision trees use ``np.float32`` arrays internally.
  If training data is not in this format, a copy of the dataset will be made.

* If the input matrix X is very sparse, it is recommended to convert to sparse
  ``csc_matrix`` before calling fit and sparse ``csr_matrix`` before calling
  predict. Training time can be orders of magnitude faster for a sparse
  matrix input compared to a dense matrix when features have zero values in
  most of the samples.

.. _tree_algorithms:

Tree algorithms: ID3, C4.5, C5.0 and CART
==========================================

What are all the various decision tree algorithms and how do they differ
from each other? Which one is implemented in scikit-learn?

.. dropdown:: Various decision tree algorithms

  ID3_ (Iterative Dichotomiser 3) was developed in 1986 by Ross Quinlan.
  The algorithm creates a multiway tree, finding for each node (i.e. in
  a greedy manner) the categorical feature that will yield the largest
  information gain for categorical targets. Trees are grown to their
  maximum size and then a pruning step is usually applied to improve the
  ability of the tree to generalize to unseen data.

  C4.5 is the successor to ID3 and removed the restriction that features
  must be categorical by dynamically defining a discrete attribute (based
  on numerical variables) that partitions the continuous attribute value
  into a discrete set of intervals. C4.5 converts the trained trees
  (i.e. the output of the ID3 algorithm) into sets of if-then rules.
  The accuracy of each rule is then evaluated to determine the order
  in which they should be applied. Pruning is done by removing a rule's
  precondition if the accuracy of the rule improves without it.

  C5.0 is Quinlan's latest version release under a proprietary license.
  It uses less memory and builds smaller rulesets than C4.5 while being
  more accurate.

  CART (Classification and Regression Trees) is very similar to C4.5, but
  it differs in that it supports numerical target variables (regression) and
  does not compute rule sets. CART constructs binary trees using the feature
  and threshold that yield the largest information gain at each node.

scikit-learn uses an optimized version of the CART algorithm; however, the
scikit-learn implementation does not support categorical variables for now.

.. _ID3: https://en.wikipedia.org/wiki/ID3_algorithm

.. _tree_mathematical_formulation:

Mathematical formulation
========================

Given training vectors :math:`x_i \in R^n`, i=1,..., l and a label vector
:math:`y \in R^l`, a decision tree recursively partitions the feature space
such that the samples with the same labels or similar target values are grouped
together.

Let the data at node :math:`m` be represented by :math:`Q_m` with :math:`n_m`
samples. For each candidate split :math:`\theta = (j, t_m)` consisting of a
feature :math:`j` and threshold :math:`t_m`, partition the data into
:math:`Q_m^{left}(\theta)` and :math:`Q_m^{right}(\theta)` subsets

.. math::

    Q_m^{left}(\theta) = \{(x, y) | x_j \leq t_m\}

    Q_m^{right}(\theta) = Q_m \setminus Q_m^{left}(\theta)

The quality of a candidate split of node :math:`m` is then computed using an
impurity function or loss function :math:`H()`, the choice of which depends on
the task being solved (classification or regression)

.. math::

   G(Q_m, \theta) = \frac{n_m^{left}}{n_m} H(Q_m^{left}(\theta))
   + \frac{n_m^{right}}{n_m} H(Q_m^{right}(\theta))

Select the parameters that minimises the impurity

.. math::

    \theta^* = \operatorname{argmin}_\theta  G(Q_m, \theta)

Recurse for subsets :math:`Q_m^{left}(\theta^*)` and
:math:`Q_m^{right}(\theta^*)` until the maximum allowable depth is reached,
:math:`n_m < \min_{samples}` or :math:`n_m = 1`.

Classification criteria
-----------------------

If a target is a classification outcome taking on values 0,1,...,K-1,
for node :math:`m`, let

.. math::

    p_{mk} = \frac{1}{n_m} \sum_{y \in Q_m} I(y = k)

be the proportion of class k observations in node :math:`m`. If :math:`m` is a
terminal node, `predict_proba` for this region is set to :math:`p_{mk}`.
Common measures of impurity are the following.

Gini:

.. math::

    H(Q_m) = \sum_k p_{mk} (1 - p_{mk})

Log Loss or Entropy:

.. math::

    H(Q_m) = - \sum_k p_{mk} \log(p_{mk})

.. dropdown:: Shannon entropy

  The entropy criterion computes the Shannon entropy of the possible classes. It
  takes the class frequencies of the training data points that reached a given
  leaf :math:`m` as their probability. Using the **Shannon entropy as tree node
  splitting criterion is equivalent to minimizing the log loss** (also known as
  cross-entropy and multinomial deviance) between the true labels :math:`y_i`
  and the probabilistic predictions :math:`T_k(x_i)` of the tree model :math:`T` for class :math:`k`.

  To see this, first recall that the log loss of a tree model :math:`T`
  computed on a dataset :math:`D` is defined as follows:

  .. math::

      \mathrm{LL}(D, T) = -\frac{1}{n} \sum_{(x_i, y_i) \in D} \sum_k I(y_i = k) \log(T_k(x_i))

  where :math:`D` is a training dataset of :math:`n` pairs :math:`(x_i, y_i)`.

  In a classification tree, the predicted class probabilities within leaf nodes
  are constant, that is: for all :math:`(x_i, y_i) \in Q_m`, one has:
  :math:`T_k(x_i) = p_{mk}` for each class :math:`k`.

  This property makes it possible to rewrite :math:`\mathrm{LL}(D, T)` as the
  sum of the Shannon entropies computed for each leaf of :math:`T` weighted by
  the number of training data points that reached each leaf:

  .. math::

      \mathrm{LL}(D, T) = \sum_{m \in T} \frac{n_m}{n} H(Q_m)

Regression criteria
-------------------

If the target is a continuous value, then for node :math:`m`, common
criteria to minimize as for determining locations for future splits are Mean
Squared Error (MSE or L2 error), Poisson deviance as well as Mean Absolute
Error (MAE or L1 error). MSE and Poisson deviance both set the predicted value
of terminal nodes to the learned mean value :math:`\bar{y}_m` of the node
whereas the MAE sets the predicted value of terminal nodes to the median
:math:`median(y)_m`.

Mean Squared Error:

.. math::

    \bar{y}_m = \frac{1}{n_m} \sum_{y \in Q_m} y

    H(Q_m) = \frac{1}{n_m} \sum_{y \in Q_m} (y - \bar{y}_m)^2

Mean Poisson deviance:

.. math::

    H(Q_m) = \frac{2}{n_m} \sum_{y \in Q_m} (y \log\frac{y}{\bar{y}_m}
    - y + \bar{y}_m)

Setting `criterion="poisson"` might be a good choice if your target is a count
or a frequency (count per some unit). In any case, :math:`y >= 0` is a
necessary condition to use this criterion. Note that it fits much slower than
the MSE criterion. For performance reasons the actual implementation minimizes
the half mean poisson deviance, i.e. the mean poisson deviance divided by 2.

Mean Absolute Error:

.. math::

    median(y)_m = \underset{y \in Q_m}{\mathrm{median}}(y)

    H(Q_m) = \frac{1}{n_m} \sum_{y \in Q_m} |y - median(y)_m|

Note that it fits much slower than the MSE criterion.

.. _tree_missing_value_support:

Missing Values Support
======================

:class:`DecisionTreeClassifier`, :class:`DecisionTreeRegressor`
have built-in support for missing values using `splitter='best'`, where
the splits are determined in a greedy fashion.
:class:`ExtraTreeClassifier`, and :class:`ExtraTreeRegressor` have built-in
support for missing values for `splitter='random'`, where the splits
are determined randomly. For more details on how the splitter differs on
non-missing values, see the :ref:`Forest section <forest>`.

The criterion supported when there are missing-values are
`'gini'`, `'entropy`', or `'log_loss'`, for classification or
`'squared_error'`, `'friedman_mse'`, or `'poisson'` for regression.

First we will describe how :class:`DecisionTreeClassifier`, :class:`DecisionTreeRegressor`
handle missing-values in the data.

For each potential threshold on the non-missing data, the splitter will evaluate
the split with all the missing values going to the left node or the right node.

Decisions are made as follows:

- By default when predicting, the samples with missing values are classified
  with the class used in the split found during training::

    >>> from sklearn.tree import DecisionTreeClassifier
    >>> import numpy as np

    >>> X = np.array([0, 1, 6, np.nan]).reshape(-1, 1)
    >>> y = [0, 0, 1, 1]

    >>> tree = DecisionTreeClassifier(random_state=0).fit(X, y)
    >>> tree.predict(X)
    array([0, 0, 1, 1])

- If the criterion evaluation is the same for both nodes,
  then the tie for missing value at predict time is broken by going to the
  right node. The splitter also checks the split where all the missing
  values go to one child and non-missing values go to the other::

    >>> from sklearn.tree import DecisionTreeClassifier
    >>> import numpy as np

    >>> X = np.array([np.nan, -1, np.nan, 1]).reshape(-1, 1)
    >>> y = [0, 0, 1, 1]

    >>> tree = DecisionTreeClassifier(random_state=0).fit(X, y)

    >>> X_test = np.array([np.nan]).reshape(-1, 1)
    >>> tree.predict(X_test)
    array([1])

- If no missing values are seen during training for a given feature, then during
  prediction missing values are mapped to the child with the most samples::

    >>> from sklearn.tree import DecisionTreeClassifier
    >>> import numpy as np

    >>> X = np.array([0, 1, 2, 3]).reshape(-1, 1)
    >>> y = [0, 1, 1, 1]

    >>> tree = DecisionTreeClassifier(random_state=0).fit(X, y)

    >>> X_test = np.array([np.nan]).reshape(-1, 1)
    >>> tree.predict(X_test)
    array([1])

:class:`ExtraTreeClassifier`, and :class:`ExtraTreeRegressor` handle missing values
in a slightly different way. When splitting a node, a random threshold will be chosen
to split the non-missing values on. Then the non-missing values will be sent to the
left and right child based on the randomly selected threshold, while the missing
values will also be randomly sent to the left or right child. This is repeated for
every feature considered at each split. The best split among these is chosen.

During prediction, the treatment of missing-values is the same as that of the
decision tree:

- By default when predicting, the samples with missing values are classified
  with the class used in the split found during training.

- If no missing values are seen during training for a given feature, then during
  prediction missing values are mapped to the child with the most samples.

.. _minimal_cost_complexity_pruning:

Minimal Cost-Complexity Pruning
===============================

Minimal cost-complexity pruning is an algorithm used to prune a tree to avoid
over-fitting, described in Chapter 3 of [BRE]_. This algorithm is parameterized
by :math:`\alpha\ge0` known as the complexity parameter. The complexity
parameter is used to define the cost-complexity measure, :math:`R_\alpha(T)` of
a given tree :math:`T`:

.. math::

  R_\alpha(T) = R(T) + \alpha|\widetilde{T}|

where :math:`|\widetilde{T}|` is the number of terminal nodes in :math:`T` and :math:`R(T)`
is traditionally defined as the total misclassification rate of the terminal
nodes. Alternatively, scikit-learn uses the total sample weighted impurity of
the terminal nodes for :math:`R(T)`. As shown above, the impurity of a node
depends on the criterion. Minimal cost-complexity pruning finds the subtree of
:math:`T` that minimizes :math:`R_\alpha(T)`.

The cost complexity measure of a single node is
:math:`R_\alpha(t)=R(t)+\alpha`. The branch, :math:`T_t`, is defined to be a
tree where node :math:`t` is its root. In general, the impurity of a node
is greater than the sum of impurities of its terminal nodes,
:math:`R(T_t)<R(t)`. However, the cost complexity measure of a node,
:math:`t`, and its branch, :math:`T_t`, can be equal depending on
:math:`\alpha`. We define the effective :math:`\alpha` of a node to be the
value where they are equal, :math:`R_\alpha(T_t)=R_\alpha(t)` or
:math:`\alpha_{eff}(t)=\frac{R(t)-R(T_t)}{|T|-1}`. A non-terminal node
with the smallest value of :math:`\alpha_{eff}` is the weakest link and will
be pruned. This process stops when the pruned tree's minimal
:math:`\alpha_{eff}` is greater than the ``ccp_alpha`` parameter.

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_tree_plot_cost_complexity_pruning.py`

.. rubric:: References

.. [BRE] L. Breiman, J. Friedman, R. Olshen, and C. Stone. Classification
  and Regression Trees. Wadsworth, Belmont, CA, 1984.

* https://en.wikipedia.org/wiki/Decision_tree_learning

* https://en.wikipedia.org/wiki/Predictive_analytics

* J.R. Quinlan. C4. 5: programs for machine learning. Morgan
  Kaufmann, 1993.

* T. Hastie, R. Tibshirani and J. Friedman. Elements of Statistical
  Learning, Springer, 2009.

tree                  Decision Trees                    currentmodule   sklearn tree    Decision Trees  DTs    are a non parametric supervised learning method used for  ref  classification  tree classification   and  ref  regression  tree regression    The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features  A tree can be seen as a piecewise constant approximation   For instance  in the example below  decision trees learn from data to approximate a sine curve with a set of if then else decision rules  The deeper the tree  the more complex the decision rules and the fitter the model      figure      auto examples tree images sphx glr plot tree regression 001 png     target     auto examples tree plot tree regression html     scale  75     align  center  Some advantages of decision trees are     Simple to understand and to interpret  Trees can be visualized     Requires little data preparation  Other techniques often require data   normalization  dummy variables need to be created and blank values to   be removed  Some tree and algorithm combinations support    ref  missing values  tree missing value support       The cost of using the tree  i e   predicting data  is logarithmic in the   number of data points used to train the tree     Able to handle both numerical and categorical data  However  the scikit learn   implementation does not support categorical variables for now  Other   techniques are usually specialized in analyzing datasets that have only one type   of variable  See  ref  algorithms  tree algorithms   for more   information     Able to handle multi output problems     Uses a white box model  If a given situation is observable in a model    the explanation for the condition is easily explained by boolean logic    By contrast  in a black box model  e g   in an artificial neural   network   results may be more difficult to interpret     Possible to validate a model using statistical tests  That makes it   possible to account for the reliability of the model     Performs well even if its assumptions are somewhat violated by   the true model from which the data were generated    The disadvantages of decision trees include     Decision tree learners can create over complex trees that do not   generalize the data well  This is called overfitting  Mechanisms   such as pruning  setting the minimum number of samples required   at a leaf node or setting the maximum depth of the tree are   necessary to avoid this problem     Decision trees can be unstable because small variations in the   data might result in a completely different tree being generated    This problem is mitigated by using decision trees within an   ensemble     Predictions of decision trees are neither smooth nor continuous  but   piecewise constant approximations as seen in the above figure  Therefore    they are not good at extrapolation     The problem of learning an optimal decision tree is known to be   NP complete under several aspects of optimality and even for simple   concepts  Consequently  practical decision tree learning algorithms   are based on heuristic algorithms such as the greedy algorithm where   locally optimal decisions are made at each node  Such algorithms   cannot guarantee to return the globally optimal decision tree   This   can be mitigated by training multiple trees in an ensemble learner    where the features and samples are randomly sampled with replacement     There are concepts that are hard to learn because decision trees   do not express them easily  such as XOR  parity or multiplexer problems     Decision tree learners create biased trees if some classes dominate    It is therefore recommended to balance the dataset prior to fitting   with the decision tree        tree classification   Classification                  class  DecisionTreeClassifier  is a class capable of performing multi class classification on a dataset   As with other classifiers   class  DecisionTreeClassifier  takes as input two arrays  an array X  sparse or dense  of shape    n samples  n features    holding the training samples  and an array Y of integer values  shape    n samples      holding the class labels for the training samples            from sklearn import tree         X     0  0    1  1           Y    0  1          clf   tree DecisionTreeClassifier           clf   clf fit X  Y   After being fitted  the model can then be used to predict the class of samples            clf predict   2   2         array  1    In case that there are multiple classes with the same and highest probability  the classifier will predict the class with the lowest index amongst those classes   As an alternative to outputting a specific class  the probability of each class can be predicted  which is the fraction of training samples of the class in a leaf            clf predict proba   2   2         array   0   1       class  DecisionTreeClassifier  is capable of both binary  where the labels are   1  1   classification and multiclass  where the labels are  0       K 1   classification   Using the Iris dataset  we can construct a tree as follows            from sklearn datasets import load iris         from sklearn import tree         iris   load iris           X  y   iris data  iris target         clf   tree DecisionTreeClassifier           clf   clf fit X  y   Once trained  you can plot the tree with the  func  plot tree  function             tree plot tree clf                figure      auto examples tree images sphx glr plot iris dtc 002 png     target     auto examples tree plot iris dtc html     scale  75     align  center     dropdown   Alternative ways to export trees    We can also export the tree in  Graphviz    https   www graphviz org     format using the  func  export graphviz    exporter  If you use the  conda  https   conda io    package manager  the graphviz binaries   and the python package can be installed with  conda install python graphviz      Alternatively binaries for graphviz can be downloaded from the graphviz project homepage    and the Python wrapper installed from pypi with  pip install graphviz      Below is an example graphviz export of the above tree trained on the entire   iris dataset  the results are saved in an output file  iris pdf                import graphviz   doctest   SKIP           dot data   tree export graphviz clf  out file None    doctest   SKIP           graph   graphviz Source dot data    doctest   SKIP           graph render  iris     doctest   SKIP    The  func  export graphviz  exporter also supports a variety of aesthetic   options  including coloring nodes by their class  or value for regression  and   using explicit variable and class names if desired  Jupyter notebooks also   render these plots inline automatically              dot data   tree export graphviz clf  out file None    doctest   SKIP                                feature names iris feature names     doctest   SKIP                                class names iris target names     doctest   SKIP                                filled True  rounded True     doctest   SKIP                                special characters True     doctest   SKIP           graph   graphviz Source dot data     doctest   SKIP           graph   doctest   SKIP       only   html           figure      images iris svg          align  center       only   latex           figure      images iris pdf          align  center       figure      auto examples tree images sphx glr plot iris dtc 001 png      target     auto examples tree plot iris dtc html      align  center      scale  75    Alternatively  the tree can also be exported in textual format with the   function  func  export text   This method doesn t require the installation   of external libraries and is more compact             from sklearn datasets import load iris           from sklearn tree import DecisionTreeClassifier           from sklearn tree import export text           iris   load iris             decision tree   DecisionTreeClassifier random state 0  max depth 2            decision tree   decision tree fit iris data  iris target            r   export text decision tree  feature names iris  feature names              print r             petal width  cm     0 80                class  0            petal width  cm     0 80                petal width  cm     1 75                    class  1                petal width  cm     1 75                    class  2        BLANKLINE      rubric   Examples     ref  sphx glr auto examples tree plot iris dtc py     ref  sphx glr auto examples tree plot unveil tree structure py       tree regression   Regression                figure      auto examples tree images sphx glr plot tree regression 001 png     target     auto examples tree plot tree regression html     scale  75     align  center  Decision trees can also be applied to regression problems  using the  class  DecisionTreeRegressor  class   As in the classification setting  the fit method will take as argument arrays X and y  only that in this case y is expected to have floating point values instead of integer values            from sklearn import tree         X     0  0    2  2           y    0 5  2 5          clf   tree DecisionTreeRegressor           clf   clf fit X  y          clf predict   1  1        array  0 5       rubric   Examples     ref  sphx glr auto examples tree plot tree regression py        tree multioutput   Multi output problems                        A multi output problem is a supervised learning problem with several outputs to predict  that is when Y is a 2d array of shape    n samples  n outputs      When there is no correlation between the outputs  a very simple way to solve this kind of problem is to build n independent models  i e  one for each output  and then to use those models to independently predict each one of the n outputs  However  because it is likely that the output values related to the same input are themselves correlated  an often better way is to build a single model capable of predicting simultaneously all n outputs  First  it requires lower training time since only a single estimator is built  Second  the generalization accuracy of the resulting estimator may often be increased   With regard to decision trees  this strategy can readily be used to support multi output problems  This requires the following changes     Store n output values in leaves  instead of 1    Use splitting criteria that compute the average reduction across all   n outputs   This module offers support for multi output problems by implementing this strategy in both  class  DecisionTreeClassifier  and  class  DecisionTreeRegressor   If a decision tree is fit on an output array Y of shape    n samples  n outputs    then the resulting estimator will     Output n output values upon   predict       Output a list of n output arrays of class probabilities upon     predict proba     The use of multi output trees for regression is demonstrated in  ref  sphx glr auto examples tree plot tree regression py   In this example  the input X is a single real value and the outputs Y are the sine and cosine of X      figure      auto examples tree images sphx glr plot tree regression 002 png     target     auto examples tree plot tree regression html     scale  75     align  center  The use of multi output trees for classification is demonstrated in  ref  sphx glr auto examples miscellaneous plot multioutput face completion py   In this example  the inputs X are the pixels of the upper half of faces and the outputs Y are the pixels of the lower half of those faces      figure      auto examples miscellaneous images sphx glr plot multioutput face completion 001 png     target     auto examples miscellaneous plot multioutput face completion html     scale  75     align  center     rubric   Examples     ref  sphx glr auto examples miscellaneous plot multioutput face completion py      rubric   References    M  Dumont et al    Fast multi class image annotation with random subwindows   and multiple output randomized trees    http   www montefiore ulg ac be services stochastic pubs 2009 DMWG09 dumont visapp09 shortpaper pdf       International Conference on Computer Vision Theory and Applications 2009      tree complexity   Complexity             In general  the run time cost to construct a balanced binary tree is  math  O n  samples n  features  log n  samples     and query time  math  O  log n  samples       Although the tree construction algorithm attempts to generate balanced trees  they will not always be balanced   Assuming that the subtrees remain approximately balanced  the cost at each node consists of searching through  math  O n  features    to find the feature that offers the largest reduction in the impurity criterion  e g  log loss  which is equivalent to an information gain   This has a cost of  math  O n  features n  samples  log n  samples     at each node  leading to a total cost over the entire trees  by summing the cost at each node  of  math  O n  features n  samples   2  log n  samples        Tips on practical use                          Decision trees tend to overfit on data with a large number of features    Getting the right ratio of samples to number of features is important  since   a tree with few samples in high dimensional space is very likely to overfit     Consider performing  dimensionality reduction   ref  PCA  PCA       ref  ICA  ICA    or  ref  feature selection   beforehand to   give your tree a better chance of finding features that are discriminative      ref  sphx glr auto examples tree plot unveil tree structure py  will help   in gaining more insights about how the decision tree makes predictions  which is   important for understanding the important features in the data     Visualize your tree as you are training by using the   export     function   Use   max depth 3   as an initial tree depth to get a feel for   how the tree is fitting to your data  and then increase the depth     Remember that the number of samples required to populate the tree doubles   for each additional level the tree grows to   Use   max depth   to control   the size of the tree to prevent overfitting     Use   min samples split   or   min samples leaf   to ensure that multiple   samples inform every decision in the tree  by controlling which splits will   be considered  A very small number will usually mean the tree will overfit    whereas a large number will prevent the tree from learning the data  Try     min samples leaf 5   as an initial value  If the sample size varies   greatly  a float number can be used as percentage in these two parameters    While   min samples split   can create arbitrarily small leaves      min samples leaf   guarantees that each leaf has a minimum size  avoiding   low variance  over fit leaf nodes in regression problems   For   classification with few classes    min samples leaf 1   is often the best   choice     Note that   min samples split   considers samples directly and independent of     sample weight    if provided  e g  a node with m weighted samples is still   treated as having exactly m samples   Consider   min weight fraction leaf   or     min impurity decrease   if accounting for sample weights is required at splits     Balance your dataset before training to prevent the tree from being biased   toward the classes that are dominant  Class balancing can be done by   sampling an equal number of samples from each class  or preferably by   normalizing the sum of the sample weights    sample weight    for each   class to the same value  Also note that weight based pre pruning criteria    such as   min weight fraction leaf    will then be less biased toward   dominant classes than criteria that are not aware of the sample weights    like   min samples leaf       If the samples are weighted  it will be easier to optimize the tree   structure using weight based pre pruning criterion such as     min weight fraction leaf    which ensure that leaf nodes contain at least   a fraction of the overall sum of the sample weights     All decision trees use   np float32   arrays internally    If training data is not in this format  a copy of the dataset will be made     If the input matrix X is very sparse  it is recommended to convert to sparse     csc matrix   before calling fit and sparse   csr matrix   before calling   predict  Training time can be orders of magnitude faster for a sparse   matrix input compared to a dense matrix when features have zero values in   most of the samples        tree algorithms   Tree algorithms  ID3  C4 5  C5 0 and CART                                             What are all the various decision tree algorithms and how do they differ from each other  Which one is implemented in scikit learn      dropdown   Various decision tree algorithms    ID3   Iterative Dichotomiser 3  was developed in 1986 by Ross Quinlan    The algorithm creates a multiway tree  finding for each node  i e  in   a greedy manner  the categorical feature that will yield the largest   information gain for categorical targets  Trees are grown to their   maximum size and then a pruning step is usually applied to improve the   ability of the tree to generalize to unseen data     C4 5 is the successor to ID3 and removed the restriction that features   must be categorical by dynamically defining a discrete attribute  based   on numerical variables  that partitions the continuous attribute value   into a discrete set of intervals  C4 5 converts the trained trees    i e  the output of the ID3 algorithm  into sets of if then rules    The accuracy of each rule is then evaluated to determine the order   in which they should be applied  Pruning is done by removing a rule s   precondition if the accuracy of the rule improves without it     C5 0 is Quinlan s latest version release under a proprietary license    It uses less memory and builds smaller rulesets than C4 5 while being   more accurate     CART  Classification and Regression Trees  is very similar to C4 5  but   it differs in that it supports numerical target variables  regression  and   does not compute rule sets  CART constructs binary trees using the feature   and threshold that yield the largest information gain at each node   scikit learn uses an optimized version of the CART algorithm  however  the scikit learn implementation does not support categorical variables for now       ID3  https   en wikipedia org wiki ID3 algorithm       tree mathematical formulation   Mathematical formulation                           Given training vectors  math  x i  in R n   i 1      l and a label vector  math  y  in R l   a decision tree recursively partitions the feature space such that the samples with the same labels or similar target values are grouped together   Let the data at node  math  m  be represented by  math  Q m  with  math  n m  samples  For each candidate split  math   theta    j  t m   consisting of a feature  math  j  and threshold  math  t m   partition the data into  math  Q m  left   theta   and  math  Q m  right   theta   subsets     math        Q m  left   theta       x  y    x j  leq t m        Q m  right   theta    Q m  setminus Q m  left   theta   The quality of a candidate split of node  math  m  is then computed using an impurity function or loss function  math  H     the choice of which depends on the task being solved  classification or regression      math       G Q m   theta     frac n m  left   n m  H Q m  left   theta         frac n m  right   n m  H Q m  right   theta    Select the parameters that minimises the impurity     math         theta      operatorname argmin   theta  G Q m   theta   Recurse for subsets  math  Q m  left   theta     and  math  Q m  right   theta     until the maximum allowable depth is reached   math  n m    min  samples   or  math  n m   1    Classification criteria                          If a target is a classification outcome taking on values 0 1     K 1  for node  math  m   let     math        p  mk     frac 1  n m   sum  y  in Q m  I y   k   be the proportion of class k observations in node  math  m   If  math  m  is a terminal node   predict proba  for this region is set to  math  p  mk    Common measures of impurity are the following   Gini      math        H Q m     sum k p  mk   1   p  mk    Log Loss or Entropy      math        H Q m       sum k p  mk   log p  mk       dropdown   Shannon entropy    The entropy criterion computes the Shannon entropy of the possible classes  It   takes the class frequencies of the training data points that reached a given   leaf  math  m  as their probability  Using the   Shannon entropy as tree node   splitting criterion is equivalent to minimizing the log loss    also known as   cross entropy and multinomial deviance  between the true labels  math  y i    and the probabilistic predictions  math  T k x i   of the tree model  math  T  for class  math  k      To see this  first recall that the log loss of a tree model  math  T    computed on a dataset  math  D  is defined as follows        math           mathrm LL  D  T      frac 1  n   sum   x i  y i   in D   sum k I y i   k   log T k x i      where  math  D  is a training dataset of  math  n  pairs  math   x i  y i       In a classification tree  the predicted class probabilities within leaf nodes   are constant  that is  for all  math   x i  y i   in Q m   one has     math  T k x i    p  mk   for each class  math  k      This property makes it possible to rewrite  math   mathrm LL  D  T   as the   sum of the Shannon entropies computed for each leaf of  math  T  weighted by   the number of training data points that reached each leaf        math           mathrm LL  D  T     sum  m  in T   frac n m  n  H Q m   Regression criteria                      If the target is a continuous value  then for node  math  m   common criteria to minimize as for determining locations for future splits are Mean Squared Error  MSE or L2 error   Poisson deviance as well as Mean Absolute Error  MAE or L1 error   MSE and Poisson deviance both set the predicted value of terminal nodes to the learned mean value  math   bar y  m  of the node whereas the MAE sets the predicted value of terminal nodes to the median  math  median y  m    Mean Squared Error      math         bar y  m    frac 1  n m   sum  y  in Q m  y      H Q m     frac 1  n m   sum  y  in Q m   y    bar y  m  2  Mean Poisson deviance      math        H Q m     frac 2  n m   sum  y  in Q m   y  log frac y   bar y  m        y    bar y  m   Setting  criterion  poisson   might be a good choice if your target is a count or a frequency  count per some unit   In any case   math  y    0  is a necessary condition to use this criterion  Note that it fits much slower than the MSE criterion  For performance reasons the actual implementation minimizes the half mean poisson deviance  i e  the mean poisson deviance divided by 2   Mean Absolute Error      math        median y  m    underset y  in Q m   mathrm median   y       H Q m     frac 1  n m   sum  y  in Q m   y   median y  m   Note that it fits much slower than the MSE criterion       tree missing value support   Missing Values Support                          class  DecisionTreeClassifier    class  DecisionTreeRegressor  have built in support for missing values using  splitter  best    where the splits are determined in a greedy fashion   class  ExtraTreeClassifier   and  class  ExtraTreeRegressor  have built in support for missing values for  splitter  random    where the splits are determined randomly  For more details on how the splitter differs on non missing values  see the  ref  Forest section  forest     The criterion supported when there are missing values are   gini      entropy    or   log loss    for classification or   squared error      friedman mse    or   poisson   for regression   First we will describe how  class  DecisionTreeClassifier    class  DecisionTreeRegressor  handle missing values in the data   For each potential threshold on the non missing data  the splitter will evaluate the split with all the missing values going to the left node or the right node   Decisions are made as follows     By default when predicting  the samples with missing values are classified   with the class used in the split found during training            from sklearn tree import DecisionTreeClassifier         import numpy as np          X   np array  0  1  6  np nan   reshape  1  1          y    0  0  1  1           tree   DecisionTreeClassifier random state 0  fit X  y          tree predict X      array  0  0  1  1      If the criterion evaluation is the same for both nodes    then the tie for missing value at predict time is broken by going to the   right node  The splitter also checks the split where all the missing   values go to one child and non missing values go to the other            from sklearn tree import DecisionTreeClassifier         import numpy as np          X   np array  np nan   1  np nan  1   reshape  1  1          y    0  0  1  1           tree   DecisionTreeClassifier random state 0  fit X  y           X test   np array  np nan   reshape  1  1          tree predict X test      array  1      If no missing values are seen during training for a given feature  then during   prediction missing values are mapped to the child with the most samples            from sklearn tree import DecisionTreeClassifier         import numpy as np          X   np array  0  1  2  3   reshape  1  1          y    0  1  1  1           tree   DecisionTreeClassifier random state 0  fit X  y           X test   np array  np nan   reshape  1  1          tree predict X test      array  1     class  ExtraTreeClassifier   and  class  ExtraTreeRegressor  handle missing values in a slightly different way  When splitting a node  a random threshold will be chosen to split the non missing values on  Then the non missing values will be sent to the left and right child based on the randomly selected threshold  while the missing values will also be randomly sent to the left or right child  This is repeated for every feature considered at each split  The best split among these is chosen   During prediction  the treatment of missing values is the same as that of the decision tree     By default when predicting  the samples with missing values are classified   with the class used in the split found during training     If no missing values are seen during training for a given feature  then during   prediction missing values are mapped to the child with the most samples       minimal cost complexity pruning   Minimal Cost Complexity Pruning                                  Minimal cost complexity pruning is an algorithm used to prune a tree to avoid over fitting  described in Chapter 3 of  BRE    This algorithm is parameterized by  math   alpha ge0  known as the complexity parameter  The complexity parameter is used to define the cost complexity measure   math  R  alpha T   of a given tree  math  T       math      R  alpha T    R T     alpha  widetilde T    where  math    widetilde T    is the number of terminal nodes in  math  T  and  math  R T   is traditionally defined as the total misclassification rate of the terminal nodes  Alternatively  scikit learn uses the total sample weighted impurity of the terminal nodes for  math  R T    As shown above  the impurity of a node depends on the criterion  Minimal cost complexity pruning finds the subtree of  math  T  that minimizes  math  R  alpha T     The cost complexity measure of a single node is  math  R  alpha t  R t   alpha   The branch   math  T t   is defined to be a tree where node  math  t  is its root  In general  the impurity of a node is greater than the sum of impurities of its terminal nodes   math  R T t  R t    However  the cost complexity measure of a node   math  t   and its branch   math  T t   can be equal depending on  math   alpha   We define the effective  math   alpha  of a node to be the value where they are equal   math  R  alpha T t  R  alpha t   or  math   alpha  eff  t   frac R t  R T t    T  1    A non terminal node with the smallest value of  math   alpha  eff   is the weakest link and will be pruned  This process stops when the pruned tree s minimal  math   alpha  eff   is greater than the   ccp alpha   parameter      rubric   Examples     ref  sphx glr auto examples tree plot cost complexity pruning py      rubric   References      BRE  L  Breiman  J  Friedman  R  Olshen  and C  Stone  Classification   and Regression Trees  Wadsworth  Belmont  CA  1984     https   en wikipedia org wiki Decision tree learning    https   en wikipedia org wiki Predictive analytics    J R  Quinlan  C4  5  programs for machine learning  Morgan   Kaufmann  1993     T  Hastie  R  Tibshirani and J  Friedman  Elements of Statistical   Learning  Springer  2009