scikit-learn In addition scikit learn includes various random sample generators that samplegenerators sklearn datasets Generated datasets can be used to build artificial datasets of controlled size and complexity

Answer:
.. _sample_generators:

Generated datasets
==================

.. currentmodule:: sklearn.datasets

In addition, scikit-learn includes various random sample generators that
can be used to build artificial datasets of controlled size and complexity.

Generators for classification and clustering
--------------------------------------------

These generators produce a matrix of features and corresponding discrete
targets.

Single label
~~~~~~~~~~~~

:func:`make_blobs` creates a multiclass dataset by allocating each class to one
normally-distributed cluster of points. It provides control over the centers and
standard deviations of each cluster. This dataset is used to demonstrate clustering.

.. plot::
   :context: close-figs
   :scale: 70
   :align: center

   import matplotlib.pyplot as plt
   from sklearn.datasets import make_blobs

   X, y = make_blobs(centers=3, cluster_std=0.5, random_state=0)

   plt.scatter(X[:, 0], X[:, 1], c=y)
   plt.title("Three normally-distributed clusters")
   plt.show()

:func:`make_classification` also creates multiclass datasets but specializes in
introducing noise by way of: correlated, redundant and uninformative features; multiple
Gaussian clusters per class; and linear transformations of the feature space.

.. plot::
   :context: close-figs
   :scale: 70
   :align: center

   import matplotlib.pyplot as plt
   from sklearn.datasets import make_classification

   fig, axs = plt.subplots(1, 3, figsize=(12, 4), sharey=True, sharex=True)
   titles = ["Two classes,\none informative feature,\none cluster per class",
             "Two classes,\ntwo informative features,\ntwo clusters per class",
             "Three classes,\ntwo informative features,\none cluster per class"]
   params = [
       {"n_informative": 1, "n_clusters_per_class": 1, "n_classes": 2},
       {"n_informative": 2, "n_clusters_per_class": 2, "n_classes": 2},
       {"n_informative": 2, "n_clusters_per_class": 1, "n_classes": 3}
   ]

   for i, param in enumerate(params):
       X, Y = make_classification(n_features=2, n_redundant=0, random_state=1, **param)
       axs[i].scatter(X[:, 0], X[:, 1], c=Y)
       axs[i].set_title(titles[i])

   plt.tight_layout()
   plt.show()

:func:`make_gaussian_quantiles` divides a single Gaussian cluster into
near-equal-size classes separated by concentric hyperspheres.

.. plot::
   :context: close-figs
   :scale: 70
   :align: center

   import matplotlib.pyplot as plt
   from sklearn.datasets import make_gaussian_quantiles

   X, Y = make_gaussian_quantiles(n_features=2, n_classes=3, random_state=0)
   plt.scatter(X[:, 0], X[:, 1], c=Y)
   plt.title("Gaussian divided into three quantiles")
   plt.show()

:func:`make_hastie_10_2` generates a similar binary, 10-dimensional problem.

:func:`make_circles` and :func:`make_moons` generate 2D binary classification
datasets that are challenging to certain algorithms (e.g., centroid-based
clustering or linear classification), including optional Gaussian noise.
They are useful for visualization. :func:`make_circles` produces Gaussian data
with a spherical decision boundary for binary classification, while
:func:`make_moons` produces two interleaving half-circles.

.. plot::
   :context: close-figs
   :scale: 70
   :align: center

   import matplotlib.pyplot as plt
   from sklearn.datasets import make_circles, make_moons

   fig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2, figsize=(8, 4))

   X, Y = make_circles(noise=0.1, factor=0.3, random_state=0)
   ax1.scatter(X[:, 0], X[:, 1], c=Y)
   ax1.set_title("make_circles")

   X, Y = make_moons(noise=0.1, random_state=0)
   ax2.scatter(X[:, 0], X[:, 1], c=Y)
   ax2.set_title("make_moons")

   plt.tight_layout()
   plt.show()

Multilabel
~~~~~~~~~~

:func:`make_multilabel_classification` generates random samples with multiple
labels, reflecting a bag of words drawn from a mixture of topics. The number of
topics for each document is drawn from a Poisson distribution, and the topics
themselves are drawn from a fixed random distribution. Similarly, the number of
words is drawn from Poisson, with words drawn from a multinomial, where each
topic defines a probability distribution over words. Simplifications with
respect to true bag-of-words mixtures include:

* Per-topic word distributions are independently drawn, where in reality all
  would be affected by a sparse base distribution, and would be correlated.
* For a document generated from multiple topics, all topics are weighted
  equally in generating its bag of words.
* Documents without labels words at random, rather than from a base
  distribution.

.. image:: ../auto_examples/datasets/images/sphx_glr_plot_random_multilabel_dataset_001.png
   :target: ../auto_examples/datasets/plot_random_multilabel_dataset.html
   :scale: 50
   :align: center

Biclustering
~~~~~~~~~~~~

.. autosummary::

   make_biclusters
   make_checkerboard

Generators for regression
-------------------------

:func:`make_regression` produces regression targets as an optionally-sparse
random linear combination of random features, with noise. Its informative
features may be uncorrelated, or low rank (few features account for most of the
variance).

Other regression generators generate functions deterministically from
randomized features.  :func:`make_sparse_uncorrelated` produces a target as a
linear combination of four features with fixed coefficients.
Others encode explicitly non-linear relations:
:func:`make_friedman1` is related by polynomial and sine transforms;
:func:`make_friedman2` includes feature multiplication and reciprocation; and
:func:`make_friedman3` is similar with an arctan transformation on the target.

Generators for manifold learning
--------------------------------

.. autosummary::

   make_s_curve
   make_swiss_roll

Generators for decomposition
----------------------------

.. autosummary::

   make_low_rank_matrix
   make_sparse_coded_signal
   make_spd_matrix
   make_sparse_spd_matrix

sample generators   Generated datasets                        currentmodule   sklearn datasets  In addition  scikit learn includes various random sample generators that can be used to build artificial datasets of controlled size and complexity   Generators for classification and clustering                                               These generators produce a matrix of features and corresponding discrete targets   Single label                func  make blobs  creates a multiclass dataset by allocating each class to one normally distributed cluster of points  It provides control over the centers and standard deviations of each cluster  This dataset is used to demonstrate clustering      plot       context  close figs     scale  70     align  center     import matplotlib pyplot as plt    from sklearn datasets import make blobs     X  y   make blobs centers 3  cluster std 0 5  random state 0      plt scatter X    0   X    1   c y     plt title  Three normally distributed clusters      plt show     func  make classification  also creates multiclass datasets but specializes in introducing noise by way of  correlated  redundant and uninformative features  multiple Gaussian clusters per class  and linear transformations of the feature space      plot       context  close figs     scale  70     align  center     import matplotlib pyplot as plt    from sklearn datasets import make classification     fig  axs   plt subplots 1  3  figsize  12  4   sharey True  sharex True     titles     Two classes  none informative feature  none cluster per class                 Two classes  ntwo informative features  ntwo clusters per class                 Three classes  ntwo informative features  none cluster per class      params              n informative   1   n clusters per class   1   n classes   2            n informative   2   n clusters per class   2   n classes   2            n informative   2   n clusters per class   1   n classes   3           for i  param in enumerate params          X  Y   make classification n features 2  n redundant 0  random state 1    param         axs i  scatter X    0   X    1   c Y         axs i  set title titles i       plt tight layout      plt show     func  make gaussian quantiles  divides a single Gaussian cluster into near equal size classes separated by concentric hyperspheres      plot       context  close figs     scale  70     align  center     import matplotlib pyplot as plt    from sklearn datasets import make gaussian quantiles     X  Y   make gaussian quantiles n features 2  n classes 3  random state 0     plt scatter X    0   X    1   c Y     plt title  Gaussian divided into three quantiles      plt show     func  make hastie 10 2  generates a similar binary  10 dimensional problem    func  make circles  and  func  make moons  generate 2D binary classification datasets that are challenging to certain algorithms  e g   centroid based clustering or linear classification   including optional Gaussian noise  They are useful for visualization   func  make circles  produces Gaussian data with a spherical decision boundary for binary classification  while  func  make moons  produces two interleaving half circles       plot       context  close figs     scale  70     align  center     import matplotlib pyplot as plt    from sklearn datasets import make circles  make moons     fig   ax1  ax2    plt subplots nrows 1  ncols 2  figsize  8  4       X  Y   make circles noise 0 1  factor 0 3  random state 0     ax1 scatter X    0   X    1   c Y     ax1 set title  make circles       X  Y   make moons noise 0 1  random state 0     ax2 scatter X    0   X    1   c Y     ax2 set title  make moons       plt tight layout      plt show      Multilabel              func  make multilabel classification  generates random samples with multiple labels  reflecting a bag of words drawn from a mixture of topics  The number of topics for each document is drawn from a Poisson distribution  and the topics themselves are drawn from a fixed random distribution  Similarly  the number of words is drawn from Poisson  with words drawn from a multinomial  where each topic defines a probability distribution over words  Simplifications with respect to true bag of words mixtures include     Per topic word distributions are independently drawn  where in reality all   would be affected by a sparse base distribution  and would be correlated    For a document generated from multiple topics  all topics are weighted   equally in generating its bag of words    Documents without labels words at random  rather than from a base   distribution      image      auto examples datasets images sphx glr plot random multilabel dataset 001 png     target     auto examples datasets plot random multilabel dataset html     scale  50     align  center  Biclustering                  autosummary       make biclusters    make checkerboard   Generators for regression                             func  make regression  produces regression targets as an optionally sparse random linear combination of random features  with noise  Its informative features may be uncorrelated  or low rank  few features account for most of the variance    Other regression generators generate functions deterministically from randomized features    func  make sparse uncorrelated  produces a target as a linear combination of four features with fixed coefficients  Others encode explicitly non linear relations   func  make friedman1  is related by polynomial and sine transforms   func  make friedman2  includes feature multiplication and reciprocation  and  func  make friedman3  is similar with an arctan transformation on the target   Generators for manifold learning                                      autosummary       make s curve    make swiss roll  Generators for decomposition                                  autosummary       make low rank matrix    make sparse coded signal    make spd matrix    make sparse spd matrix


scikit-learn scalingstrategies For some applications the amount of examples features or both and or the speed at which they need to be processed are challenging for traditional approaches In these cases scikit learn has a number of options you can consider to make your system scale Strategies to scale computationally bigger data

Answer:
.. _scaling_strategies:

Strategies to scale computationally: bigger data
=================================================

For some applications the amount of examples, features (or both) and/or the
speed at which they need to be processed are challenging for traditional
approaches. In these cases scikit-learn has a number of options you can
consider to make your system scale.

Scaling with instances using out-of-core learning
--------------------------------------------------

Out-of-core (or "external memory") learning is a technique used to learn from
data that cannot fit in a computer's main memory (RAM).

Here is a sketch of a system designed to achieve this goal:

1. a way to stream instances
2. a way to extract features from instances
3. an incremental algorithm

Streaming instances
....................

Basically, 1. may be a reader that yields instances from files on a
hard drive, a database, from a network stream etc. However,
details on how to achieve this are beyond the scope of this documentation.

Extracting features
...................

\2. could be any relevant way to extract features among the
different :ref:`feature extraction <feature_extraction>` methods supported by
scikit-learn. However, when working with data that needs vectorization and
where the set of features or values is not known in advance one should take
explicit care. A good example is text classification where unknown terms are
likely to be found during training. It is possible to use a stateful
vectorizer if making multiple passes over the data is reasonable from an
application point of view. Otherwise, one can turn up the difficulty by using
a stateless feature extractor. Currently the preferred way to do this is to
use the so-called :ref:`hashing trick<feature_hashing>` as implemented by
:class:`sklearn.feature_extraction.FeatureHasher` for datasets with categorical
variables represented as list of Python dicts or
:class:`sklearn.feature_extraction.text.HashingVectorizer` for text documents.

Incremental learning
.....................

Finally, for 3. we have a number of options inside scikit-learn. Although not
all algorithms can learn incrementally (i.e. without seeing all the instances
at once), all estimators implementing the ``partial_fit`` API are candidates.
Actually, the ability to learn incrementally from a mini-batch of instances
(sometimes called "online learning") is key to out-of-core learning as it
guarantees that at any given time there will be only a small amount of
instances in the main memory. Choosing a good size for the mini-batch that
balances relevancy and memory footprint could involve some tuning [1]_.

Here is a list of incremental estimators for different tasks:

- Classification
    + :class:`sklearn.naive_bayes.MultinomialNB`
    + :class:`sklearn.naive_bayes.BernoulliNB`
    + :class:`sklearn.linear_model.Perceptron`
    + :class:`sklearn.linear_model.SGDClassifier`
    + :class:`sklearn.linear_model.PassiveAggressiveClassifier`
    + :class:`sklearn.neural_network.MLPClassifier`
- Regression
    + :class:`sklearn.linear_model.SGDRegressor`
    + :class:`sklearn.linear_model.PassiveAggressiveRegressor`
    + :class:`sklearn.neural_network.MLPRegressor`
- Clustering
    + :class:`sklearn.cluster.MiniBatchKMeans`
    + :class:`sklearn.cluster.Birch`
- Decomposition / feature Extraction
    + :class:`sklearn.decomposition.MiniBatchDictionaryLearning`
    + :class:`sklearn.decomposition.IncrementalPCA`
    + :class:`sklearn.decomposition.LatentDirichletAllocation`
    + :class:`sklearn.decomposition.MiniBatchNMF`
- Preprocessing
    + :class:`sklearn.preprocessing.StandardScaler`
    + :class:`sklearn.preprocessing.MinMaxScaler`
    + :class:`sklearn.preprocessing.MaxAbsScaler`

For classification, a somewhat important thing to note is that although a
stateless feature extraction routine may be able to cope with new/unseen
attributes, the incremental learner itself may be unable to cope with
new/unseen targets classes. In this case you have to pass all the possible
classes to the first ``partial_fit`` call using the ``classes=`` parameter.

Another aspect to consider when choosing a proper algorithm is that not all of
them put the same importance on each example over time. Namely, the
``Perceptron`` is still sensitive to badly labeled examples even after many
examples whereas the ``SGD*`` and ``PassiveAggressive*`` families are more
robust to this kind of artifacts. Conversely, the latter also tend to give less
importance to remarkably different, yet properly labeled examples when they
come late in the stream as their learning rate decreases over time.

Examples
..........

Finally, we have a full-fledged example of
:ref:`sphx_glr_auto_examples_applications_plot_out_of_core_classification.py`. It is aimed at
providing a starting point for people wanting to build out-of-core learning
systems and demonstrates most of the notions discussed above.

Furthermore, it also shows the evolution of the performance of different
algorithms with the number of processed examples.

.. |accuracy_over_time| image::  ../auto_examples/applications/images/sphx_glr_plot_out_of_core_classification_001.png
    :target: ../auto_examples/applications/plot_out_of_core_classification.html
    :scale: 80

.. centered:: |accuracy_over_time|

Now looking at the computation time of the different parts, we see that the
vectorization is much more expensive than learning itself. From the different
algorithms, ``MultinomialNB`` is the most expensive, but its overhead can be
mitigated by increasing the size of the mini-batches (exercise: change
``minibatch_size`` to 100 and 10000 in the program and compare).

.. |computation_time| image::  ../auto_examples/applications/images/sphx_glr_plot_out_of_core_classification_003.png
    :target: ../auto_examples/applications/plot_out_of_core_classification.html
    :scale: 80

.. centered:: |computation_time|

Notes
......

.. [1] Depending on the algorithm the mini-batch size can influence results or
       not. SGD*, PassiveAggressive*, and discrete NaiveBayes are truly online
       and are not affected by batch size. Conversely, MiniBatchKMeans
       convergence rate is affected by the batch size. Also, its memory
       footprint can vary dramatically with batch size.

scaling strategies   Strategies to scale computationally  bigger data                                                    For some applications the amount of examples  features  or both  and or the speed at which they need to be processed are challenging for traditional approaches  In these cases scikit learn has a number of options you can consider to make your system scale   Scaling with instances using out of core learning                                                     Out of core  or  external memory   learning is a technique used to learn from data that cannot fit in a computer s main memory  RAM    Here is a sketch of a system designed to achieve this goal   1  a way to stream instances 2  a way to extract features from instances 3  an incremental algorithm  Streaming instances                       Basically  1  may be a reader that yields instances from files on a hard drive  a database  from a network stream etc  However  details on how to achieve this are beyond the scope of this documentation   Extracting features                       2  could be any relevant way to extract features among the different  ref  feature extraction  feature extraction   methods supported by scikit learn  However  when working with data that needs vectorization and where the set of features or values is not known in advance one should take explicit care  A good example is text classification where unknown terms are likely to be found during training  It is possible to use a stateful vectorizer if making multiple passes over the data is reasonable from an application point of view  Otherwise  one can turn up the difficulty by using a stateless feature extractor  Currently the preferred way to do this is to use the so called  ref  hashing trick feature hashing   as implemented by  class  sklearn feature extraction FeatureHasher  for datasets with categorical variables represented as list of Python dicts or  class  sklearn feature extraction text HashingVectorizer  for text documents   Incremental learning                        Finally  for 3  we have a number of options inside scikit learn  Although not all algorithms can learn incrementally  i e  without seeing all the instances at once   all estimators implementing the   partial fit   API are candidates  Actually  the ability to learn incrementally from a mini batch of instances  sometimes called  online learning   is key to out of core learning as it guarantees that at any given time there will be only a small amount of instances in the main memory  Choosing a good size for the mini batch that balances relevancy and memory footprint could involve some tuning  1     Here is a list of incremental estimators for different tasks     Classification        class  sklearn naive bayes MultinomialNB         class  sklearn naive bayes BernoulliNB         class  sklearn linear model Perceptron         class  sklearn linear model SGDClassifier         class  sklearn linear model PassiveAggressiveClassifier         class  sklearn neural network MLPClassifier    Regression        class  sklearn linear model SGDRegressor         class  sklearn linear model PassiveAggressiveRegressor         class  sklearn neural network MLPRegressor    Clustering        class  sklearn cluster MiniBatchKMeans         class  sklearn cluster Birch    Decomposition   feature Extraction        class  sklearn decomposition MiniBatchDictionaryLearning         class  sklearn decomposition IncrementalPCA         class  sklearn decomposition LatentDirichletAllocation         class  sklearn decomposition MiniBatchNMF    Preprocessing        class  sklearn preprocessing StandardScaler         class  sklearn preprocessing MinMaxScaler         class  sklearn preprocessing MaxAbsScaler   For classification  a somewhat important thing to note is that although a stateless feature extraction routine may be able to cope with new unseen attributes  the incremental learner itself may be unable to cope with new unseen targets classes  In this case you have to pass all the possible classes to the first   partial fit   call using the   classes    parameter   Another aspect to consider when choosing a proper algorithm is that not all of them put the same importance on each example over time  Namely  the   Perceptron   is still sensitive to badly labeled examples even after many examples whereas the   SGD    and   PassiveAggressive    families are more robust to this kind of artifacts  Conversely  the latter also tend to give less importance to remarkably different  yet properly labeled examples when they come late in the stream as their learning rate decreases over time   Examples             Finally  we have a full fledged example of  ref  sphx glr auto examples applications plot out of core classification py   It is aimed at providing a starting point for people wanting to build out of core learning systems and demonstrates most of the notions discussed above   Furthermore  it also shows the evolution of the performance of different algorithms with the number of processed examples       accuracy over time  image       auto examples applications images sphx glr plot out of core classification 001 png      target     auto examples applications plot out of core classification html      scale  80     centered    accuracy over time   Now looking at the computation time of the different parts  we see that the vectorization is much more expensive than learning itself  From the different algorithms    MultinomialNB   is the most expensive  but its overhead can be mitigated by increasing the size of the mini batches  exercise  change   minibatch size   to 100 and 10000 in the program and compare        computation time  image       auto examples applications images sphx glr plot out of core classification 003 png      target     auto examples applications plot out of core classification html      scale  80     centered    computation time    Notes             1  Depending on the algorithm the mini batch size can influence results or        not  SGD   PassiveAggressive   and discrete NaiveBayes are truly online        and are not affected by batch size  Conversely  MiniBatchKMeans        convergence rate is affected by the batch size  Also  its memory        footprint can vary dramatically with batch size


scikit-learn correspond to certain kernels as they are used for example in support vector machines see kernelapproximation This submodule contains functions that approximate the feature mappings that The following feature functions perform non linear transformations of the Kernel Approximation input which can serve as a basis for linear classification or other

Answer:
.. _kernel_approximation:

Kernel Approximation
====================

This submodule contains functions that approximate the feature mappings that
correspond to certain kernels, as they are used for example in support vector
machines (see :ref:`svm`).
The following feature functions perform non-linear transformations of the
input, which can serve as a basis for linear classification or other
algorithms.

.. currentmodule:: sklearn.linear_model

The advantage of using approximate explicit feature maps compared to the
`kernel trick <https://en.wikipedia.org/wiki/Kernel_trick>`_,
which makes use of feature maps implicitly, is that explicit mappings
can be better suited for online learning and can significantly reduce the cost
of learning with very large datasets.
Standard kernelized SVMs do not scale well to large datasets, but using an
approximate kernel map it is possible to use much more efficient linear SVMs.
In particular, the combination of kernel map approximations with
:class:`SGDClassifier` can make non-linear learning on large datasets possible.

Since there has not been much empirical work using approximate embeddings, it
is advisable to compare results against exact kernel methods when possible.

.. seealso::

   :ref:`polynomial_regression` for an exact polynomial transformation.

.. currentmodule:: sklearn.kernel_approximation

.. _nystroem_kernel_approx:

Nystroem Method for Kernel Approximation
----------------------------------------
The Nystroem method, as implemented in :class:`Nystroem` is a general method for
reduced rank approximations of kernels. It achieves this by subsampling without
replacement rows/columns of the data on which the kernel is evaluated. While the
computational complexity of the exact method is
:math:`\mathcal{O}(n^3_{\text{samples}})`, the complexity of the approximation
is :math:`\mathcal{O}(n^2_{\text{components}} \cdot n_{\text{samples}})`, where
one can set :math:`n_{\text{components}} \ll n_{\text{samples}}` without a
significative decrease in performance [WS2001]_.

We can construct the eigendecomposition of the kernel matrix :math:`K`, based
on the features of the data, and then split it into sampled and unsampled data
points.

.. math::

        K = U \Lambda U^T
        = \begin{bmatrix} U_1 \\ U_2\end{bmatrix} \Lambda \begin{bmatrix} U_1 \\ U_2 \end{bmatrix}^T
        = \begin{bmatrix} U_1 \Lambda U_1^T & U_1 \Lambda U_2^T \\ U_2 \Lambda U_1^T & U_2 \Lambda U_2^T \end{bmatrix}
        \equiv \begin{bmatrix} K_{11} & K_{12} \\ K_{21} & K_{22} \end{bmatrix}

where:

* :math:`U` is orthonormal
* :math:`\Lambda` is diagonal matrix of eigenvalues
* :math:`U_1` is orthonormal matrix of samples that were chosen
* :math:`U_2` is orthonormal matrix of samples that were not chosen

Given that :math:`U_1 \Lambda U_1^T` can be obtained by orthonormalization of
the matrix :math:`K_{11}`, and :math:`U_2 \Lambda U_1^T` can be evaluated (as
well as its transpose), the only remaining term to elucidate is
:math:`U_2 \Lambda U_2^T`. To do this we can express it in terms of the already
evaluated matrices:

.. math::

         \begin{align} U_2 \Lambda U_2^T &= \left(K_{21} U_1 \Lambda^{-1}\right) \Lambda \left(K_{21} U_1 \Lambda^{-1}\right)^T
         \\&= K_{21} U_1 (\Lambda^{-1} \Lambda) \Lambda^{-1} U_1^T K_{21}^T
         \\&= K_{21} U_1 \Lambda^{-1} U_1^T K_{21}^T
         \\&= K_{21} K_{11}^{-1} K_{21}^T
         \\&= \left( K_{21} K_{11}^{-\frac12} \right) \left( K_{21} K_{11}^{-\frac12} \right)^T
         .\end{align}

During ``fit``, the class :class:`Nystroem` evaluates the basis :math:`U_1`, and
computes the normalization constant, :math:`K_{11}^{-\frac12}`. Later, during
``transform``, the kernel matrix is determined between the basis (given by the
`components_` attribute) and the new data points, ``X``. This matrix is then
multiplied by the ``normalization_`` matrix for the final result.

By default :class:`Nystroem` uses the ``rbf`` kernel, but it can use any kernel
function or a precomputed kernel matrix. The number of samples used - which is
also the dimensionality of the features computed - is given by the parameter
``n_components``.

.. rubric:: Examples

* See the example entitled
  :ref:`sphx_glr_auto_examples_applications_plot_cyclical_feature_engineering.py`,
  that shows an efficient machine learning pipeline that uses a
  :class:`Nystroem` kernel.

.. _rbf_kernel_approx:

Radial Basis Function Kernel
----------------------------

The :class:`RBFSampler` constructs an approximate mapping for the radial basis
function kernel, also known as *Random Kitchen Sinks* [RR2007]_. This
transformation can be used to explicitly model a kernel map, prior to applying
a linear algorithm, for example a linear SVM::

    >>> from sklearn.kernel_approximation import RBFSampler
    >>> from sklearn.linear_model import SGDClassifier
    >>> X = [[0, 0], [1, 1], [1, 0], [0, 1]]
    >>> y = [0, 0, 1, 1]
    >>> rbf_feature = RBFSampler(gamma=1, random_state=1)
    >>> X_features = rbf_feature.fit_transform(X)
    >>> clf = SGDClassifier(max_iter=5)
    >>> clf.fit(X_features, y)
    SGDClassifier(max_iter=5)
    >>> clf.score(X_features, y)
    1.0

The mapping relies on a Monte Carlo approximation to the
kernel values. The ``fit`` function performs the Monte Carlo sampling, whereas
the ``transform`` method performs the mapping of the data.  Because of the
inherent randomness of the process, results may vary between different calls to
the ``fit`` function.

The ``fit`` function takes two arguments:
``n_components``, which is the target dimensionality of the feature transform,
and ``gamma``, the parameter of the RBF-kernel.  A higher ``n_components`` will
result in a better approximation of the kernel and will yield results more
similar to those produced by a kernel SVM. Note that "fitting" the feature
function does not actually depend on the data given to the ``fit`` function.
Only the dimensionality of the data is used.
Details on the method can be found in [RR2007]_.

For a given value of ``n_components`` :class:`RBFSampler` is often less accurate
as :class:`Nystroem`. :class:`RBFSampler` is cheaper to compute, though, making
use of larger feature spaces more efficient.

.. figure:: ../auto_examples/miscellaneous/images/sphx_glr_plot_kernel_approximation_002.png
    :target: ../auto_examples/miscellaneous/plot_kernel_approximation.html
    :scale: 50%
    :align: center

    Comparing an exact RBF kernel (left) with the approximation (right)

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_miscellaneous_plot_kernel_approximation.py`

.. _additive_chi_kernel_approx:

Additive Chi Squared Kernel
---------------------------

The additive chi squared kernel is a kernel on histograms, often used in computer vision.

The additive chi squared kernel as used here is given by

.. math::

        k(x, y) = \sum_i \frac{2x_iy_i}{x_i+y_i}

This is not exactly the same as :func:`sklearn.metrics.pairwise.additive_chi2_kernel`.
The authors of [VZ2010]_ prefer the version above as it is always positive
definite.
Since the kernel is additive, it is possible to treat all components
:math:`x_i` separately for embedding. This makes it possible to sample
the Fourier transform in regular intervals, instead of approximating
using Monte Carlo sampling.

The class :class:`AdditiveChi2Sampler` implements this component wise
deterministic sampling. Each component is sampled :math:`n` times, yielding
:math:`2n+1` dimensions per input dimension (the multiple of two stems
from the real and complex part of the Fourier transform).
In the literature, :math:`n` is usually chosen to be 1 or 2, transforming
the dataset to size ``n_samples * 5 * n_features`` (in the case of :math:`n=2`).

The approximate feature map provided by :class:`AdditiveChi2Sampler` can be combined
with the approximate feature map provided by :class:`RBFSampler` to yield an approximate
feature map for the exponentiated chi squared kernel.
See the [VZ2010]_ for details and [VVZ2010]_ for combination with the :class:`RBFSampler`.

.. _skewed_chi_kernel_approx:

Skewed Chi Squared Kernel
-------------------------

The skewed chi squared kernel is given by:

.. math::

        k(x,y) = \prod_i \frac{2\sqrt{x_i+c}\sqrt{y_i+c}}{x_i + y_i + 2c}

It has properties that are similar to the exponentiated chi squared kernel
often used in computer vision, but allows for a simple Monte Carlo
approximation of the feature map.

The usage of the :class:`SkewedChi2Sampler` is the same as the usage described
above for the :class:`RBFSampler`. The only difference is in the free
parameter, that is called :math:`c`.
For a motivation for this mapping and the mathematical details see [LS2010]_.

.. _polynomial_kernel_approx:

Polynomial Kernel Approximation via Tensor Sketch
-------------------------------------------------

The :ref:`polynomial kernel <polynomial_kernel>` is a popular type of kernel
function given by:

.. math::

        k(x, y) = (\gamma x^\top y +c_0)^d

where:

* ``x``, ``y`` are the input vectors
* ``d`` is the kernel degree

Intuitively, the feature space of the polynomial kernel of degree `d`
consists of all possible degree-`d` products among input features, which enables
learning algorithms using this kernel to account for interactions between features.

The TensorSketch [PP2013]_ method, as implemented in :class:`PolynomialCountSketch`, is a
scalable, input data independent method for polynomial kernel approximation.
It is based on the concept of Count sketch [WIKICS]_ [CCF2002]_ , a dimensionality
reduction technique similar to feature hashing, which instead uses several
independent hash functions. TensorSketch obtains a Count Sketch of the outer product
of two vectors (or a vector with itself), which can be used as an approximation of the
polynomial kernel feature space. In particular, instead of explicitly computing
the outer product, TensorSketch computes the Count Sketch of the vectors and then
uses polynomial multiplication via the Fast Fourier Transform to compute the
Count Sketch of their outer product.

Conveniently, the training phase of TensorSketch simply consists of initializing
some random variables. It is thus independent of the input data, i.e. it only
depends on the number of input features, but not the data values.
In addition, this method can transform samples in
:math:`\mathcal{O}(n_{\text{samples}}(n_{\text{features}} + n_{\text{components}} \log(n_{\text{components}})))`
time, where :math:`n_{\text{components}}` is the desired output dimension,
determined by ``n_components``.

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_kernel_approximation_plot_scalable_poly_kernels.py`

.. _tensor_sketch_kernel_approx:

Mathematical Details
--------------------

Kernel methods like support vector machines or kernelized
PCA rely on a property of reproducing kernel Hilbert spaces.
For any positive definite kernel function :math:`k` (a so called Mercer kernel),
it is guaranteed that there exists a mapping :math:`\phi`
into a Hilbert space :math:`\mathcal{H}`, such that

.. math::

        k(x,y) = \langle \phi(x), \phi(y) \rangle

Where :math:`\langle \cdot, \cdot \rangle` denotes the inner product in the
Hilbert space.

If an algorithm, such as a linear support vector machine or PCA,
relies only on the scalar product of data points :math:`x_i`, one may use
the value of :math:`k(x_i, x_j)`, which corresponds to applying the algorithm
to the mapped data points :math:`\phi(x_i)`.
The advantage of using :math:`k` is that the mapping :math:`\phi` never has
to be calculated explicitly, allowing for arbitrary large
features (even infinite).

One drawback of kernel methods is, that it might be necessary
to store many kernel values :math:`k(x_i, x_j)` during optimization.
If a kernelized classifier is applied to new data :math:`y_j`,
:math:`k(x_i, y_j)` needs to be computed to make predictions,
possibly for many different :math:`x_i` in the training set.

The classes in this submodule allow to approximate the embedding
:math:`\phi`, thereby working explicitly with the representations
:math:`\phi(x_i)`, which obviates the need to apply the kernel
or store training examples.

.. rubric:: References

.. [WS2001] `"Using the Nyström method to speed up kernel machines"
  <https://papers.nips.cc/paper_files/paper/2000/hash/19de10adbaa1b2ee13f77f679fa1483a-Abstract.html>`_
  Williams, C.K.I.; Seeger, M. - 2001.
.. [RR2007] `"Random features for large-scale kernel machines"
  <https://papers.nips.cc/paper/2007/hash/013a006f03dbc5392effeb8f18fda755-Abstract.html>`_
  Rahimi, A. and Recht, B. - Advances in neural information processing 2007,
.. [LS2010] `"Random Fourier approximations for skewed multiplicative histogram kernels"
  <https://www.researchgate.net/publication/221114584_Random_Fourier_Approximations_for_Skewed_Multiplicative_Histogram_Kernels>`_
  Li, F., Ionescu, C., and Sminchisescu, C.
  - Pattern Recognition,  DAGM 2010, Lecture Notes in Computer Science.
.. [VZ2010] `"Efficient additive kernels via explicit feature maps"
  <https://www.robots.ox.ac.uk/~vgg/publications/2011/Vedaldi11/vedaldi11.pdf>`_
  Vedaldi, A. and Zisserman, A. - Computer Vision and Pattern Recognition 2010
.. [VVZ2010] `"Generalized RBF feature maps for Efficient Detection"
  <https://www.robots.ox.ac.uk/~vgg/publications/2010/Sreekanth10/sreekanth10.pdf>`_
  Vempati, S. and Vedaldi, A. and Zisserman, A. and Jawahar, CV - 2010
.. [PP2013] :doi:`"Fast and scalable polynomial kernels via explicit feature maps"
  <10.1145/2487575.2487591>`
  Pham, N., & Pagh, R. - 2013
.. [CCF2002] `"Finding frequent items in data streams"
  <https://www.cs.princeton.edu/courses/archive/spring04/cos598B/bib/CharikarCF.pdf>`_
  Charikar, M., Chen, K., & Farach-Colton - 2002
.. [WIKICS] `"Wikipedia: Count sketch"
  <https://en.wikipedia.org/wiki/Count_sketch>`_

kernel approximation   Kernel Approximation                       This submodule contains functions that approximate the feature mappings that correspond to certain kernels  as they are used for example in support vector machines  see  ref  svm    The following feature functions perform non linear transformations of the input  which can serve as a basis for linear classification or other algorithms      currentmodule   sklearn linear model  The advantage of using approximate explicit feature maps compared to the  kernel trick  https   en wikipedia org wiki Kernel trick     which makes use of feature maps implicitly  is that explicit mappings can be better suited for online learning and can significantly reduce the cost of learning with very large datasets  Standard kernelized SVMs do not scale well to large datasets  but using an approximate kernel map it is possible to use much more efficient linear SVMs  In particular  the combination of kernel map approximations with  class  SGDClassifier  can make non linear learning on large datasets possible   Since there has not been much empirical work using approximate embeddings  it is advisable to compare results against exact kernel methods when possible      seealso        ref  polynomial regression  for an exact polynomial transformation      currentmodule   sklearn kernel approximation      nystroem kernel approx   Nystroem Method for Kernel Approximation                                          The Nystroem method  as implemented in  class  Nystroem  is a general method for reduced rank approximations of kernels  It achieves this by subsampling without replacement rows columns of the data on which the kernel is evaluated  While the computational complexity of the exact method is  math   mathcal O  n 3   text samples      the complexity of the approximation is  math   mathcal O  n 2   text components    cdot n   text samples      where one can set  math  n   text components    ll n   text samples    without a significative decrease in performance  WS2001     We can construct the eigendecomposition of the kernel matrix  math  K   based on the features of the data  and then split it into sampled and unsampled data points      math            K   U  Lambda U T            begin bmatrix  U 1    U 2 end bmatrix   Lambda  begin bmatrix  U 1    U 2  end bmatrix  T            begin bmatrix  U 1  Lambda U 1 T   U 1  Lambda U 2 T    U 2  Lambda U 1 T   U 2  Lambda U 2 T  end bmatrix           equiv  begin bmatrix  K  11    K  12     K  21    K  22   end bmatrix   where      math  U  is orthonormal    math   Lambda  is diagonal matrix of eigenvalues    math  U 1  is orthonormal matrix of samples that were chosen    math  U 2  is orthonormal matrix of samples that were not chosen  Given that  math  U 1  Lambda U 1 T  can be obtained by orthonormalization of the matrix  math  K  11    and  math  U 2  Lambda U 1 T  can be evaluated  as well as its transpose   the only remaining term to elucidate is  math  U 2  Lambda U 2 T   To do this we can express it in terms of the already evaluated matrices      math              begin align  U 2  Lambda U 2 T     left K  21  U 1  Lambda   1  right   Lambda  left K  21  U 1  Lambda   1  right  T               K  21  U 1   Lambda   1   Lambda   Lambda   1  U 1 T K  21  T               K  21  U 1  Lambda   1  U 1 T K  21  T               K  21  K  11    1  K  21  T                left  K  21  K  11     frac12   right   left  K  21  K  11     frac12   right  T            end align   During   fit    the class  class  Nystroem  evaluates the basis  math  U 1   and computes the normalization constant   math  K  11     frac12    Later  during   transform    the kernel matrix is determined between the basis  given by the  components   attribute  and the new data points    X    This matrix is then multiplied by the   normalization    matrix for the final result   By default  class  Nystroem  uses the   rbf   kernel  but it can use any kernel function or a precomputed kernel matrix  The number of samples used   which is also the dimensionality of the features computed   is given by the parameter   n components        rubric   Examples    See the example entitled    ref  sphx glr auto examples applications plot cyclical feature engineering py     that shows an efficient machine learning pipeline that uses a    class  Nystroem  kernel       rbf kernel approx   Radial Basis Function Kernel                               The  class  RBFSampler  constructs an approximate mapping for the radial basis function kernel  also known as  Random Kitchen Sinks   RR2007    This transformation can be used to explicitly model a kernel map  prior to applying a linear algorithm  for example a linear SVM            from sklearn kernel approximation import RBFSampler         from sklearn linear model import SGDClassifier         X     0  0    1  1    1  0    0  1           y    0  0  1  1          rbf feature   RBFSampler gamma 1  random state 1          X features   rbf feature fit transform X          clf   SGDClassifier max iter 5          clf fit X features  y      SGDClassifier max iter 5          clf score X features  y      1 0  The mapping relies on a Monte Carlo approximation to the kernel values  The   fit   function performs the Monte Carlo sampling  whereas the   transform   method performs the mapping of the data   Because of the inherent randomness of the process  results may vary between different calls to the   fit   function   The   fit   function takes two arguments    n components    which is the target dimensionality of the feature transform  and   gamma    the parameter of the RBF kernel   A higher   n components   will result in a better approximation of the kernel and will yield results more similar to those produced by a kernel SVM  Note that  fitting  the feature function does not actually depend on the data given to the   fit   function  Only the dimensionality of the data is used  Details on the method can be found in  RR2007     For a given value of   n components    class  RBFSampler  is often less accurate as  class  Nystroem    class  RBFSampler  is cheaper to compute  though  making use of larger feature spaces more efficient      figure      auto examples miscellaneous images sphx glr plot kernel approximation 002 png      target     auto examples miscellaneous plot kernel approximation html      scale  50       align  center      Comparing an exact RBF kernel  left  with the approximation  right      rubric   Examples     ref  sphx glr auto examples miscellaneous plot kernel approximation py       additive chi kernel approx   Additive Chi Squared Kernel                              The additive chi squared kernel is a kernel on histograms  often used in computer vision   The additive chi squared kernel as used here is given by     math            k x  y     sum i  frac 2x iy i  x i y i   This is not exactly the same as  func  sklearn metrics pairwise additive chi2 kernel   The authors of  VZ2010   prefer the version above as it is always positive definite  Since the kernel is additive  it is possible to treat all components  math  x i  separately for embedding  This makes it possible to sample the Fourier transform in regular intervals  instead of approximating using Monte Carlo sampling   The class  class  AdditiveChi2Sampler  implements this component wise deterministic sampling  Each component is sampled  math  n  times  yielding  math  2n 1  dimensions per input dimension  the multiple of two stems from the real and complex part of the Fourier transform   In the literature   math  n  is usually chosen to be 1 or 2  transforming the dataset to size   n samples   5   n features    in the case of  math  n 2     The approximate feature map provided by  class  AdditiveChi2Sampler  can be combined with the approximate feature map provided by  class  RBFSampler  to yield an approximate feature map for the exponentiated chi squared kernel  See the  VZ2010   for details and  VVZ2010   for combination with the  class  RBFSampler        skewed chi kernel approx   Skewed Chi Squared Kernel                            The skewed chi squared kernel is given by      math            k x y     prod i  frac 2 sqrt x i c  sqrt y i c   x i   y i   2c    It has properties that are similar to the exponentiated chi squared kernel often used in computer vision  but allows for a simple Monte Carlo approximation of the feature map   The usage of the  class  SkewedChi2Sampler  is the same as the usage described above for the  class  RBFSampler   The only difference is in the free parameter  that is called  math  c   For a motivation for this mapping and the mathematical details see  LS2010         polynomial kernel approx   Polynomial Kernel Approximation via Tensor Sketch                                                    The  ref  polynomial kernel  polynomial kernel   is a popular type of kernel function given by      math            k x  y      gamma x  top y  c 0  d  where       x      y   are the input vectors     d   is the kernel degree  Intuitively  the feature space of the polynomial kernel of degree  d  consists of all possible degree  d  products among input features  which enables learning algorithms using this kernel to account for interactions between features   The TensorSketch  PP2013   method  as implemented in  class  PolynomialCountSketch   is a scalable  input data independent method for polynomial kernel approximation  It is based on the concept of Count sketch  WIKICS    CCF2002     a dimensionality reduction technique similar to feature hashing  which instead uses several independent hash functions  TensorSketch obtains a Count Sketch of the outer product of two vectors  or a vector with itself   which can be used as an approximation of the polynomial kernel feature space  In particular  instead of explicitly computing the outer product  TensorSketch computes the Count Sketch of the vectors and then uses polynomial multiplication via the Fast Fourier Transform to compute the Count Sketch of their outer product   Conveniently  the training phase of TensorSketch simply consists of initializing some random variables  It is thus independent of the input data  i e  it only depends on the number of input features  but not the data values  In addition  this method can transform samples in  math   mathcal O  n   text samples   n   text features     n   text components    log n   text components       time  where  math  n   text components    is the desired output dimension  determined by   n components        rubric   Examples     ref  sphx glr auto examples kernel approximation plot scalable poly kernels py       tensor sketch kernel approx   Mathematical Details                       Kernel methods like support vector machines or kernelized PCA rely on a property of reproducing kernel Hilbert spaces  For any positive definite kernel function  math  k   a so called Mercer kernel   it is guaranteed that there exists a mapping  math   phi  into a Hilbert space  math   mathcal H    such that     math            k x y     langle  phi x    phi y   rangle  Where  math   langle  cdot   cdot  rangle  denotes the inner product in the Hilbert space   If an algorithm  such as a linear support vector machine or PCA  relies only on the scalar product of data points  math  x i   one may use the value of  math  k x i  x j    which corresponds to applying the algorithm to the mapped data points  math   phi x i    The advantage of using  math  k  is that the mapping  math   phi  never has to be calculated explicitly  allowing for arbitrary large features  even infinite    One drawback of kernel methods is  that it might be necessary to store many kernel values  math  k x i  x j   during optimization  If a kernelized classifier is applied to new data  math  y j    math  k x i  y j   needs to be computed to make predictions  possibly for many different  math  x i  in the training set   The classes in this submodule allow to approximate the embedding  math   phi   thereby working explicitly with the representations  math   phi x i    which obviates the need to apply the kernel or store training examples       rubric   References      WS2001    Using the Nystr m method to speed up kernel machines     https   papers nips cc paper files paper 2000 hash 19de10adbaa1b2ee13f77f679fa1483a Abstract html      Williams  C K I   Seeger  M    2001      RR2007    Random features for large scale kernel machines     https   papers nips cc paper 2007 hash 013a006f03dbc5392effeb8f18fda755 Abstract html      Rahimi  A  and Recht  B    Advances in neural information processing 2007      LS2010    Random Fourier approximations for skewed multiplicative histogram kernels     https   www researchgate net publication 221114584 Random Fourier Approximations for Skewed Multiplicative Histogram Kernels      Li  F   Ionescu  C   and Sminchisescu  C      Pattern Recognition   DAGM 2010  Lecture Notes in Computer Science      VZ2010    Efficient additive kernels via explicit feature maps     https   www robots ox ac uk  vgg publications 2011 Vedaldi11 vedaldi11 pdf      Vedaldi  A  and Zisserman  A    Computer Vision and Pattern Recognition 2010     VVZ2010    Generalized RBF feature maps for Efficient Detection     https   www robots ox ac uk  vgg publications 2010 Sreekanth10 sreekanth10 pdf      Vempati  S  and Vedaldi  A  and Zisserman  A  and Jawahar  CV   2010     PP2013   doi   Fast and scalable polynomial kernels via explicit feature maps     10 1145 2487575 2487591     Pham  N     Pagh  R    2013     CCF2002    Finding frequent items in data streams     https   www cs princeton edu courses archive spring04 cos598B bib CharikarCF pdf      Charikar  M   Chen  K     Farach Colton   2002     WIKICS    Wikipedia  Count sketch     https   en wikipedia org wiki Count sketch


scikit-learn neuralnetworksunsupervised Neural network models unsupervised sklearn neuralnetwork rbm

Answer:
.. _neural_networks_unsupervised:

====================================
Neural network models (unsupervised)
====================================

.. currentmodule:: sklearn.neural_network

.. _rbm:

Restricted Boltzmann machines
=============================

Restricted Boltzmann machines (RBM) are unsupervised nonlinear feature learners
based on a probabilistic model. The features extracted by an RBM or a hierarchy
of RBMs often give good results when fed into a linear classifier such as a
linear SVM or a perceptron.

The model makes assumptions regarding the distribution of inputs. At the moment,
scikit-learn only provides :class:`BernoulliRBM`, which assumes the inputs are
either binary values or values between 0 and 1, each encoding the probability
that the specific feature would be turned on.

The RBM tries to maximize the likelihood of the data using a particular
graphical model. The parameter learning algorithm used (:ref:`Stochastic
Maximum Likelihood <sml>`) prevents the representations from straying far
from the input data, which makes them capture interesting regularities, but
makes the model less useful for small datasets, and usually not useful for
density estimation.

The method gained popularity for initializing deep neural networks with the
weights of independent RBMs. This method is known as unsupervised pre-training.

.. figure:: ../auto_examples/neural_networks/images/sphx_glr_plot_rbm_logistic_classification_001.png
   :target: ../auto_examples/neural_networks/plot_rbm_logistic_classification.html
   :align: center
   :scale: 100%

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_neural_networks_plot_rbm_logistic_classification.py`

Graphical model and parametrization
-----------------------------------

The graphical model of an RBM is a fully-connected bipartite graph.

.. image:: ../images/rbm_graph.png
   :align: center

The nodes are random variables whose states depend on the state of the other
nodes they are connected to. The model is therefore parameterized by the
weights of the connections, as well as one intercept (bias) term for each
visible and hidden unit, omitted from the image for simplicity.

The energy function measures the quality of a joint assignment:

.. math::

   E(\mathbf{v}, \mathbf{h}) = -\sum_i \sum_j w_{ij}v_ih_j - \sum_i b_iv_i
     - \sum_j c_jh_j

In the formula above, :math:`\mathbf{b}` and :math:`\mathbf{c}` are the
intercept vectors for the visible and hidden layers, respectively. The
joint probability of the model is defined in terms of the energy:

.. math::

   P(\mathbf{v}, \mathbf{h}) = \frac{e^{-E(\mathbf{v}, \mathbf{h})}}{Z}

The word *restricted* refers to the bipartite structure of the model, which
prohibits direct interaction between hidden units, or between visible units.
This means that the following conditional independencies are assumed:

.. math::

   h_i \bot h_j | \mathbf{v} \\
   v_i \bot v_j | \mathbf{h}

The bipartite structure allows for the use of efficient block Gibbs sampling for
inference.

Bernoulli Restricted Boltzmann machines
---------------------------------------

In the :class:`BernoulliRBM`, all units are binary stochastic units. This
means that the input data should either be binary, or real-valued between 0 and
1 signifying the probability that the visible unit would turn on or off. This
is a good model for character recognition, where the interest is on which
pixels are active and which aren't. For images of natural scenes it no longer
fits because of background, depth and the tendency of neighbouring pixels to
take the same values.

The conditional probability distribution of each unit is given by the
logistic sigmoid activation function of the input it receives:

.. math::

   P(v_i=1|\mathbf{h}) = \sigma(\sum_j w_{ij}h_j + b_i) \\
   P(h_i=1|\mathbf{v}) = \sigma(\sum_i w_{ij}v_i + c_j)

where :math:`\sigma` is the logistic sigmoid function:

.. math::

   \sigma(x) = \frac{1}{1 + e^{-x}}

.. _sml:

Stochastic Maximum Likelihood learning
--------------------------------------

The training algorithm implemented in :class:`BernoulliRBM` is known as
Stochastic Maximum Likelihood (SML) or Persistent Contrastive Divergence
(PCD). Optimizing maximum likelihood directly is infeasible because of
the form of the data likelihood:

.. math::

   \log P(v) = \log \sum_h e^{-E(v, h)} - \log \sum_{x, y} e^{-E(x, y)}

For simplicity the equation above is written for a single training example.
The gradient with respect to the weights is formed of two terms corresponding to
the ones above. They are usually known as the positive gradient and the negative
gradient, because of their respective signs.  In this implementation, the
gradients are estimated over mini-batches of samples.

In maximizing the log-likelihood, the positive gradient makes the model prefer
hidden states that are compatible with the observed training data. Because of
the bipartite structure of RBMs, it can be computed efficiently. The
negative gradient, however, is intractable. Its goal is to lower the energy of
joint states that the model prefers, therefore making it stay true to the data.
It can be approximated by Markov chain Monte Carlo using block Gibbs sampling by
iteratively sampling each of :math:`v` and :math:`h` given the other, until the
chain mixes. Samples generated in this way are sometimes referred as fantasy
particles. This is inefficient and it is difficult to determine whether the
Markov chain mixes.

The Contrastive Divergence method suggests to stop the chain after a small
number of iterations, :math:`k`, usually even 1. This method is fast and has
low variance, but the samples are far from the model distribution.

Persistent Contrastive Divergence addresses this. Instead of starting a new
chain each time the gradient is needed, and performing only one Gibbs sampling
step, in PCD we keep a number of chains (fantasy particles) that are updated
:math:`k` Gibbs steps after each weight update. This allows the particles to
explore the space more thoroughly.

.. rubric:: References

* `"A fast learning algorithm for deep belief nets"
  <https://www.cs.toronto.edu/~hinton/absps/fastnc.pdf>`_,
  G. Hinton, S. Osindero, Y.-W. Teh, 2006

* `"Training Restricted Boltzmann Machines using Approximations to
  the Likelihood Gradient"
  <https://www.cs.toronto.edu/~tijmen/pcd/pcd.pdf>`_,
  T. Tieleman, 2008

neural networks unsupervised                                        Neural network models  unsupervised                                           currentmodule   sklearn neural network       rbm   Restricted Boltzmann machines                                Restricted Boltzmann machines  RBM  are unsupervised nonlinear feature learners based on a probabilistic model  The features extracted by an RBM or a hierarchy of RBMs often give good results when fed into a linear classifier such as a linear SVM or a perceptron   The model makes assumptions regarding the distribution of inputs  At the moment  scikit learn only provides  class  BernoulliRBM   which assumes the inputs are either binary values or values between 0 and 1  each encoding the probability that the specific feature would be turned on   The RBM tries to maximize the likelihood of the data using a particular graphical model  The parameter learning algorithm used   ref  Stochastic Maximum Likelihood  sml    prevents the representations from straying far from the input data  which makes them capture interesting regularities  but makes the model less useful for small datasets  and usually not useful for density estimation   The method gained popularity for initializing deep neural networks with the weights of independent RBMs  This method is known as unsupervised pre training      figure      auto examples neural networks images sphx glr plot rbm logistic classification 001 png     target     auto examples neural networks plot rbm logistic classification html     align  center     scale  100      rubric   Examples     ref  sphx glr auto examples neural networks plot rbm logistic classification py    Graphical model and parametrization                                      The graphical model of an RBM is a fully connected bipartite graph      image      images rbm graph png     align  center  The nodes are random variables whose states depend on the state of the other nodes they are connected to  The model is therefore parameterized by the weights of the connections  as well as one intercept  bias  term for each visible and hidden unit  omitted from the image for simplicity   The energy function measures the quality of a joint assignment      math       E  mathbf v    mathbf h       sum i  sum j w  ij v ih j    sum i b iv i         sum j c jh j  In the formula above   math   mathbf b   and  math   mathbf c   are the intercept vectors for the visible and hidden layers  respectively  The joint probability of the model is defined in terms of the energy      math       P  mathbf v    mathbf h      frac e   E  mathbf v    mathbf h     Z    The word  restricted  refers to the bipartite structure of the model  which prohibits direct interaction between hidden units  or between visible units  This means that the following conditional independencies are assumed      math       h i  bot h j    mathbf v        v i  bot v j    mathbf h   The bipartite structure allows for the use of efficient block Gibbs sampling for inference   Bernoulli Restricted Boltzmann machines                                          In the  class  BernoulliRBM   all units are binary stochastic units  This means that the input data should either be binary  or real valued between 0 and 1 signifying the probability that the visible unit would turn on or off  This is a good model for character recognition  where the interest is on which pixels are active and which aren t  For images of natural scenes it no longer fits because of background  depth and the tendency of neighbouring pixels to take the same values   The conditional probability distribution of each unit is given by the logistic sigmoid activation function of the input it receives      math       P v i 1  mathbf h      sigma  sum j w  ij h j   b i        P h i 1  mathbf v      sigma  sum i w  ij v i   c j   where  math   sigma  is the logistic sigmoid function      math        sigma x     frac 1  1   e   x        sml   Stochastic Maximum Likelihood learning                                         The training algorithm implemented in  class  BernoulliRBM  is known as Stochastic Maximum Likelihood  SML  or Persistent Contrastive Divergence  PCD   Optimizing maximum likelihood directly is infeasible because of the form of the data likelihood      math        log P v     log  sum h e   E v  h      log  sum  x  y  e   E x  y    For simplicity the equation above is written for a single training example  The gradient with respect to the weights is formed of two terms corresponding to the ones above  They are usually known as the positive gradient and the negative gradient  because of their respective signs   In this implementation  the gradients are estimated over mini batches of samples   In maximizing the log likelihood  the positive gradient makes the model prefer hidden states that are compatible with the observed training data  Because of the bipartite structure of RBMs  it can be computed efficiently  The negative gradient  however  is intractable  Its goal is to lower the energy of joint states that the model prefers  therefore making it stay true to the data  It can be approximated by Markov chain Monte Carlo using block Gibbs sampling by iteratively sampling each of  math  v  and  math  h  given the other  until the chain mixes  Samples generated in this way are sometimes referred as fantasy particles  This is inefficient and it is difficult to determine whether the Markov chain mixes   The Contrastive Divergence method suggests to stop the chain after a small number of iterations   math  k   usually even 1  This method is fast and has low variance  but the samples are far from the model distribution   Persistent Contrastive Divergence addresses this  Instead of starting a new chain each time the gradient is needed  and performing only one Gibbs sampling step  in PCD we keep a number of chains  fantasy particles  that are updated  math  k  Gibbs steps after each weight update  This allows the particles to explore the space more thoroughly      rubric   References      A fast learning algorithm for deep belief nets     https   www cs toronto edu  hinton absps fastnc pdf       G  Hinton  S  Osindero  Y  W  Teh  2006      Training Restricted Boltzmann Machines using Approximations to   the Likelihood Gradient     https   www cs toronto edu  tijmen pcd pcd pdf       T  Tieleman  2008


scikit-learn Density estimation walks the line between unsupervised learning feature Density Estimation density estimation techniques are mixture models such as Jake Vanderplas vanderplas astro washington edu densityestimation engineering and data modeling Some of the most popular and useful

Answer:
.. _density_estimation:

==================
Density Estimation
==================
.. sectionauthor:: Jake Vanderplas <vanderplas@astro.washington.edu>

Density estimation walks the line between unsupervised learning, feature
engineering, and data modeling.  Some of the most popular and useful
density estimation techniques are mixture models such as
Gaussian Mixtures (:class:`~sklearn.mixture.GaussianMixture`), and
neighbor-based approaches such as the kernel density estimate
(:class:`~sklearn.neighbors.KernelDensity`).
Gaussian Mixtures are discussed more fully in the context of
:ref:`clustering <clustering>`, because the technique is also useful as
an unsupervised clustering scheme.

Density estimation is a very simple concept, and most people are already
familiar with one common density estimation technique: the histogram.

Density Estimation: Histograms
==============================
A histogram is a simple visualization of data where bins are defined, and the
number of data points within each bin is tallied.  An example of a histogram
can be seen in the upper-left panel of the following figure:

.. |hist_to_kde| image:: ../auto_examples/neighbors/images/sphx_glr_plot_kde_1d_001.png
   :target: ../auto_examples/neighbors/plot_kde_1d.html
   :scale: 80

.. centered:: |hist_to_kde|

A major problem with histograms, however, is that the choice of binning can
have a disproportionate effect on the resulting visualization.  Consider the
upper-right panel of the above figure.  It shows a histogram over the same
data, with the bins shifted right.  The results of the two visualizations look
entirely different, and might lead to different interpretations of the data.

Intuitively, one can also think of a histogram as a stack of blocks, one block
per point.  By stacking the blocks in the appropriate grid space, we recover
the histogram.  But what if, instead of stacking the blocks on a regular grid,
we center each block on the point it represents, and sum the total height at
each location?  This idea leads to the lower-left visualization.  It is perhaps
not as clean as a histogram, but the fact that the data drive the block
locations mean that it is a much better representation of the underlying
data.

This visualization is an example of a *kernel density estimation*, in this case
with a top-hat kernel (i.e. a square block at each point).  We can recover a
smoother distribution by using a smoother kernel.  The bottom-right plot shows
a Gaussian kernel density estimate, in which each point contributes a Gaussian
curve to the total.  The result is a smooth density estimate which is derived
from the data, and functions as a powerful non-parametric model of the
distribution of points.

.. _kernel_density:

Kernel Density Estimation
=========================
Kernel density estimation in scikit-learn is implemented in the
:class:`~sklearn.neighbors.KernelDensity` estimator, which uses the
Ball Tree or KD Tree for efficient queries (see :ref:`neighbors` for
a discussion of these).  Though the above example
uses a 1D data set for simplicity, kernel density estimation can be
performed in any number of dimensions, though in practice the curse of
dimensionality causes its performance to degrade in high dimensions.

In the following figure, 100 points are drawn from a bimodal distribution,
and the kernel density estimates are shown for three choices of kernels:

.. |kde_1d_distribution| image:: ../auto_examples/neighbors/images/sphx_glr_plot_kde_1d_003.png
   :target: ../auto_examples/neighbors/plot_kde_1d.html
   :scale: 80

.. centered:: |kde_1d_distribution|

It's clear how the kernel shape affects the smoothness of the resulting
distribution.  The scikit-learn kernel density estimator can be used as
follows:

   >>> from sklearn.neighbors import KernelDensity
   >>> import numpy as np
   >>> X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
   >>> kde = KernelDensity(kernel='gaussian', bandwidth=0.2).fit(X)
   >>> kde.score_samples(X)
   array([-0.41075698, -0.41075698, -0.41076071, -0.41075698, -0.41075698,
          -0.41076071])

Here we have used ``kernel='gaussian'``, as seen above.
Mathematically, a kernel is a positive function :math:`K(x;h)`
which is controlled by the bandwidth parameter :math:`h`.
Given this kernel form, the density estimate at a point :math:`y` within
a group of points :math:`x_i; i=1\cdots N` is given by:

.. math::
    \rho_K(y) = \sum_{i=1}^{N} K(y - x_i; h)

The bandwidth here acts as a smoothing parameter, controlling the tradeoff
between bias and variance in the result.  A large bandwidth leads to a very
smooth (i.e. high-bias) density distribution.  A small bandwidth leads
to an unsmooth (i.e. high-variance) density distribution.

The parameter `bandwidth` controls this smoothing. One can either set
manually this parameter or use Scott's and Silverman's estimation
methods.

:class:`~sklearn.neighbors.KernelDensity` implements several common kernel
forms, which are shown in the following figure:

.. |kde_kernels| image:: ../auto_examples/neighbors/images/sphx_glr_plot_kde_1d_002.png
   :target: ../auto_examples/neighbors/plot_kde_1d.html
   :scale: 80

.. centered:: |kde_kernels|

.. dropdown:: Kernels' mathematical expressions

  The form of these kernels is as follows:

  * Gaussian kernel (``kernel = 'gaussian'``)

    :math:`K(x; h) \propto \exp(- \frac{x^2}{2h^2} )`

  * Tophat kernel (``kernel = 'tophat'``)

    :math:`K(x; h) \propto 1` if :math:`x < h`

  * Epanechnikov kernel (``kernel = 'epanechnikov'``)

    :math:`K(x; h) \propto 1 - \frac{x^2}{h^2}`

  * Exponential kernel (``kernel = 'exponential'``)

    :math:`K(x; h) \propto \exp(-x/h)`

  * Linear kernel (``kernel = 'linear'``)

    :math:`K(x; h) \propto 1 - x/h` if :math:`x < h`

  * Cosine kernel (``kernel = 'cosine'``)

    :math:`K(x; h) \propto \cos(\frac{\pi x}{2h})` if :math:`x < h`

The kernel density estimator can be used with any of the valid distance
metrics (see :class:`~sklearn.metrics.DistanceMetric` for a list of
available metrics), though the results are properly normalized only
for the Euclidean metric.  One particularly useful metric is the
`Haversine distance <https://en.wikipedia.org/wiki/Haversine_formula>`_
which measures the angular distance between points on a sphere.  Here
is an example of using a kernel density estimate for a visualization
of geospatial data, in this case the distribution of observations of two
different species on the South American continent:

.. |species_kde| image:: ../auto_examples/neighbors/images/sphx_glr_plot_species_kde_001.png
   :target: ../auto_examples/neighbors/plot_species_kde.html
   :scale: 80

.. centered:: |species_kde|

One other useful application of kernel density estimation is to learn a
non-parametric generative model of a dataset in order to efficiently
draw new samples from this generative model.
Here is an example of using this process to
create a new set of hand-written digits, using a Gaussian kernel learned
on a PCA projection of the data:

.. |digits_kde| image:: ../auto_examples/neighbors/images/sphx_glr_plot_digits_kde_sampling_001.png
   :target: ../auto_examples/neighbors/plot_digits_kde_sampling.html
   :scale: 80

.. centered:: |digits_kde|

The "new" data consists of linear combinations of the input data, with weights
probabilistically drawn given the KDE model.

.. rubric:: Examples

* :ref:`sphx_glr_auto_examples_neighbors_plot_kde_1d.py`: computation of simple kernel
  density estimates in one dimension.

* :ref:`sphx_glr_auto_examples_neighbors_plot_digits_kde_sampling.py`: an example of using
  Kernel Density estimation to learn a generative model of the hand-written
  digits data, and drawing new samples from this model.

* :ref:`sphx_glr_auto_examples_neighbors_plot_species_kde.py`: an example of Kernel Density
  estimation using the Haversine distance metric to visualize geospatial data

density estimation                      Density Estimation                       sectionauthor   Jake Vanderplas  vanderplas astro washington edu   Density estimation walks the line between unsupervised learning  feature engineering  and data modeling   Some of the most popular and useful density estimation techniques are mixture models such as Gaussian Mixtures   class   sklearn mixture GaussianMixture    and neighbor based approaches such as the kernel density estimate   class   sklearn neighbors KernelDensity    Gaussian Mixtures are discussed more fully in the context of  ref  clustering  clustering    because the technique is also useful as an unsupervised clustering scheme   Density estimation is a very simple concept  and most people are already familiar with one common density estimation technique  the histogram   Density Estimation  Histograms                                A histogram is a simple visualization of data where bins are defined  and the number of data points within each bin is tallied   An example of a histogram can be seen in the upper left panel of the following figure       hist to kde  image      auto examples neighbors images sphx glr plot kde 1d 001 png     target     auto examples neighbors plot kde 1d html     scale  80     centered    hist to kde   A major problem with histograms  however  is that the choice of binning can have a disproportionate effect on the resulting visualization   Consider the upper right panel of the above figure   It shows a histogram over the same data  with the bins shifted right   The results of the two visualizations look entirely different  and might lead to different interpretations of the data   Intuitively  one can also think of a histogram as a stack of blocks  one block per point   By stacking the blocks in the appropriate grid space  we recover the histogram   But what if  instead of stacking the blocks on a regular grid  we center each block on the point it represents  and sum the total height at each location   This idea leads to the lower left visualization   It is perhaps not as clean as a histogram  but the fact that the data drive the block locations mean that it is a much better representation of the underlying data   This visualization is an example of a  kernel density estimation   in this case with a top hat kernel  i e  a square block at each point    We can recover a smoother distribution by using a smoother kernel   The bottom right plot shows a Gaussian kernel density estimate  in which each point contributes a Gaussian curve to the total   The result is a smooth density estimate which is derived from the data  and functions as a powerful non parametric model of the distribution of points       kernel density   Kernel Density Estimation                           Kernel density estimation in scikit learn is implemented in the  class   sklearn neighbors KernelDensity  estimator  which uses the Ball Tree or KD Tree for efficient queries  see  ref  neighbors  for a discussion of these    Though the above example uses a 1D data set for simplicity  kernel density estimation can be performed in any number of dimensions  though in practice the curse of dimensionality causes its performance to degrade in high dimensions   In the following figure  100 points are drawn from a bimodal distribution  and the kernel density estimates are shown for three choices of kernels       kde 1d distribution  image      auto examples neighbors images sphx glr plot kde 1d 003 png     target     auto examples neighbors plot kde 1d html     scale  80     centered    kde 1d distribution   It s clear how the kernel shape affects the smoothness of the resulting distribution   The scikit learn kernel density estimator can be used as follows          from sklearn neighbors import KernelDensity        import numpy as np        X   np array    1   1     2   1     3   2    1  1    2  1    3  2           kde   KernelDensity kernel  gaussian   bandwidth 0 2  fit X         kde score samples X     array   0 41075698   0 41075698   0 41076071   0 41075698   0 41075698             0 41076071    Here we have used   kernel  gaussian     as seen above  Mathematically  a kernel is a positive function  math  K x h   which is controlled by the bandwidth parameter  math  h   Given this kernel form  the density estimate at a point  math  y  within a group of points  math  x i  i 1 cdots N  is given by      math        rho K y     sum  i 1   N  K y   x i  h   The bandwidth here acts as a smoothing parameter  controlling the tradeoff between bias and variance in the result   A large bandwidth leads to a very smooth  i e  high bias  density distribution   A small bandwidth leads to an unsmooth  i e  high variance  density distribution   The parameter  bandwidth  controls this smoothing  One can either set manually this parameter or use Scott s and Silverman s estimation methods    class   sklearn neighbors KernelDensity  implements several common kernel forms  which are shown in the following figure       kde kernels  image      auto examples neighbors images sphx glr plot kde 1d 002 png     target     auto examples neighbors plot kde 1d html     scale  80     centered    kde kernels      dropdown   Kernels  mathematical expressions    The form of these kernels is as follows       Gaussian kernel    kernel    gaussian           math  K x  h   propto  exp    frac x 2  2h 2          Tophat kernel    kernel    tophat           math  K x  h   propto 1  if  math  x   h       Epanechnikov kernel    kernel    epanechnikov           math  K x  h   propto 1    frac x 2  h 2        Exponential kernel    kernel    exponential           math  K x  h   propto  exp  x h        Linear kernel    kernel    linear           math  K x  h   propto 1   x h  if  math  x   h       Cosine kernel    kernel    cosine           math  K x  h   propto  cos  frac  pi x  2h    if  math  x   h    The kernel density estimator can be used with any of the valid distance metrics  see  class   sklearn metrics DistanceMetric  for a list of available metrics   though the results are properly normalized only for the Euclidean metric   One particularly useful metric is the  Haversine distance  https   en wikipedia org wiki Haversine formula    which measures the angular distance between points on a sphere   Here is an example of using a kernel density estimate for a visualization of geospatial data  in this case the distribution of observations of two different species on the South American continent       species kde  image      auto examples neighbors images sphx glr plot species kde 001 png     target     auto examples neighbors plot species kde html     scale  80     centered    species kde   One other useful application of kernel density estimation is to learn a non parametric generative model of a dataset in order to efficiently draw new samples from this generative model  Here is an example of using this process to create a new set of hand written digits  using a Gaussian kernel learned on a PCA projection of the data       digits kde  image      auto examples neighbors images sphx glr plot digits kde sampling 001 png     target     auto examples neighbors plot digits kde sampling html     scale  80     centered    digits kde   The  new  data consists of linear combinations of the input data  with weights probabilistically drawn given the KDE model      rubric   Examples     ref  sphx glr auto examples neighbors plot kde 1d py   computation of simple kernel   density estimates in one dimension      ref  sphx glr auto examples neighbors plot digits kde sampling py   an example of using   Kernel Density estimation to learn a generative model of the hand written   digits data  and drawing new samples from this model      ref  sphx glr auto examples neighbors plot species kde py   an example of Kernel Density   estimation using the Haversine distance metric to visualize geospatial data