redis Using the redis benchmark utility on a Redis server docs reference optimization benchmarks md title Redis benchmark aliases Benchmarking topics benchmarks weight 1 docs reference optimization benchmarks

Answer:
---
title: "Redis benchmark"
linkTitle: "Benchmarking"
weight: 1
description: >
    Using the redis-benchmark utility on a Redis server
aliases: [
    /topics/benchmarks,
    /docs/reference/optimization/benchmarks,
    /docs/reference/optimization/benchmarks.md
]
---

Redis includes the `redis-benchmark` utility that simulates running commands done
by N clients while at the same time sending M total queries. The utility provides
a default set of tests, or you can supply a custom set of tests.

The following options are supported:

    Usage: redis-benchmark [-h <host>] [-p <port>] [-c <clients>] [-n <requests]> [-k <boolean>]

     -h <hostname>      Server hostname (default 127.0.0.1)
     -p <port>          Server port (default 6379)
     -s <socket>        Server socket (overrides host and port)
     -a <password>      Password for Redis Auth
     -c <clients>       Number of parallel connections (default 50)
     -n <requests>      Total number of requests (default 100000)
     -d <size>          Data size of SET/GET value in bytes (default 3)
     --dbnum <db>       SELECT the specified db number (default 0)
     -k <boolean>       1=keep alive 0=reconnect (default 1)
     -r <keyspacelen>   Use random keys for SET/GET/INCR, random values for SADD
      Using this option the benchmark will expand the string __rand_int__
      inside an argument with a 12 digits number in the specified range
      from 0 to keyspacelen-1. The substitution changes every time a command
      is executed. Default tests use this to hit random keys in the
      specified range.
     -P <numreq>        Pipeline <numreq> requests. Default 1 (no pipeline).
     -q                 Quiet. Just show query/sec values
     --csv              Output in CSV format
     -l                 Loop. Run the tests forever
     -t <tests>         Only run the comma separated list of tests. The test
                        names are the same as the ones produced as output.
     -I                 Idle mode. Just open N idle connections and wait.

You need to have a running Redis instance before launching the benchmark.
You can run the benchmarking utility like so:

    redis-benchmark -q -n 100000

### Running only a subset of the tests

You don't need to run all the default tests every time you execute `redis-benchmark`.
For example, to select only a subset of tests, use the `-t` option
as in the following example:

    $ redis-benchmark -t set,lpush -n 100000 -q
    SET: 74239.05 requests per second
    LPUSH: 79239.30 requests per second

This example runs the tests for the `SET` and `LPUSH` commands and uses quiet mode (see the `-q` switch).

You can even benchmark a specific command:

    $ redis-benchmark -n 100000 -q script load "redis.call('set','foo','bar')"
    script load redis.call('set','foo','bar'): 69881.20 requests per second

### Selecting the size of the key space

By default, the benchmark runs against a single key. In Redis the difference
between such a synthetic benchmark and a real one is not huge since it is an
in-memory system, however it is possible to stress cache misses and in general
to simulate a more real-world work load by using a large key space.

This is obtained by using the `-r` switch. For instance if I want to run
one million SET operations, using a random key for every operation out of
100k possible keys, I'll use the following command line:

    $ redis-cli flushall
    OK

    $ redis-benchmark -t set -r 100000 -n 1000000
    ====== SET ======
      1000000 requests completed in 13.86 seconds
      50 parallel clients
      3 bytes payload
      keep alive: 1

    99.76% `<=` 1 milliseconds
    99.98% `<=` 2 milliseconds
    100.00% `<=` 3 milliseconds
    100.00% `<=` 3 milliseconds
    72144.87 requests per second

    $ redis-cli dbsize
    (integer) 99993

### Using pipelining

By default every client (the benchmark simulates 50 clients if not otherwise
specified with `-c`) sends the next command only when the reply of the previous
command is received, this means that the server will likely need a read call
in order to read each command from every client. Also RTT is paid as well.

Redis supports [pipelining](/topics/pipelining), so it is possible to send
multiple commands at once, a feature often exploited by real world applications.
Redis pipelining is able to dramatically improve the number of operations per
second a server is able do deliver.

Consider this example of running the benchmark using a
pipelining of 16 commands:

    $ redis-benchmark -n 1000000 -t set,get -P 16 -q
    SET: 403063.28 requests per second
    GET: 508388.41 requests per second

Using pipelining results in a significant increase in performance.

### Pitfalls and misconceptions

The first point is obvious: the golden rule of a useful benchmark is to
only compare apples and apples. You can compare different versions of Redis on the same workload or the same version of Redis, but with
different options. If you plan to compare Redis to something else, then it is
important to evaluate the functional and technical differences, and take them
in account.

+ Redis is a server: all commands involve network or IPC round trips. It is meaningless to compare it to embedded data stores, because the cost of most operations is primarily in network/protocol management.
+ Redis commands return an acknowledgment for all usual commands. Some other data stores do not. Comparing Redis to stores involving one-way queries is only mildly useful.
+ Naively iterating on synchronous Redis commands does not benchmark Redis itself, but rather measure your network (or IPC) latency and the client library intrinsic latency. To really test Redis, you need multiple connections (like redis-benchmark) and/or to use pipelining to aggregate several commands and/or multiple threads or processes.
+ Redis is an in-memory data store with some optional persistence options. If you plan to compare it to transactional servers (MySQL, PostgreSQL, etc ...), then you should consider activating AOF and decide on a suitable fsync policy.
+ Redis is, mostly, a single-threaded server from the POV of commands execution (actually modern versions of Redis use threads for different things). It is not designed to benefit from multiple CPU cores. People are supposed to launch several Redis instances to scale out on several cores if needed. It is not really fair to compare one single Redis instance to a multi-threaded data store.

The `redis-benchmark` program is a quick and useful way to get some figures and
evaluate the performance of a Redis instance on a given hardware. However,
by default, it does not represent the maximum throughput a Redis instance can
sustain. Actually, by using pipelining and a fast client (hiredis), it is fairly
easy to write a program generating more throughput than redis-benchmark. The
default behavior of redis-benchmark is to achieve throughput by exploiting
concurrency only (i.e. it creates several connections to the server).
It does not use pipelining or any parallelism at all (one pending query per
connection at most, and no multi-threading), if not explicitly enabled via
the `-P` parameter. So in some way using `redis-benchmark` and, triggering, for
example, a `BGSAVE` operation in the background at the same time, will provide
the user with numbers more near to the *worst case* than to the best case.

To run a benchmark using pipelining mode (and achieve higher throughput),
you need to explicitly use the -P option. Please note that it is still a
realistic behavior since a lot of Redis based applications actively use
pipelining to improve performance. However you should use a pipeline size that
is more or less the average pipeline length you'll be able to use in your
application in order to get realistic numbers.

The benchmark should apply the same operations, and work in the same way
with the multiple data stores you want to compare. It is absolutely pointless to
compare the result of redis-benchmark to the result of another benchmark
program and extrapolate.

For instance, Redis and memcached in single-threaded mode can be compared on
GET/SET operations. Both are in-memory data stores, working mostly in the same
way at the protocol level. Provided their respective benchmark application is
aggregating queries in the same way (pipelining) and use a similar number of
connections, the comparison is actually meaningful.

When you're benchmarking a high-performance, in-memory database like Redis,
it may be difficult to saturate
the server. Sometimes, the performance bottleneck is on the client side,
and not the server-side. In that case, the client (i.e., the benchmarking program itself)
must be fixed, or perhaps scaled out, to reach the maximum throughput.

### Factors impacting Redis performance

There are multiple factors having direct consequences on Redis performance.
We mention them here, since they can alter the result of any benchmarks.
Please note however, that a typical Redis instance running on a low end,
untuned box usually provides good enough performance for most applications.

+ Network bandwidth and latency usually have a direct impact on the performance.
It is a good practice to use the ping program to quickly check the latency
between the client and server hosts is normal before launching the benchmark.
Regarding the bandwidth, it is generally useful to estimate
the throughput in Gbit/s and compare it to the theoretical bandwidth
of the network. For instance a benchmark setting 4 KB strings
in Redis at 100000 q/s, would actually consume 3.2 Gbit/s of bandwidth
and probably fit within a 10 Gbit/s link, but not a 1 Gbit/s one. In many real
world scenarios, Redis throughput is limited by the network well before being
limited by the CPU. To consolidate several high-throughput Redis instances
on a single server, it worth considering putting a 10 Gbit/s NIC
or multiple 1 Gbit/s NICs with TCP/IP bonding.
+ CPU is another very important factor. Being single-threaded, Redis favors
fast CPUs with large caches and not many cores. At this game, Intel CPUs are
currently the winners. It is not uncommon to get only half the performance on
an AMD Opteron CPU compared to similar Nehalem EP/Westmere EP/Sandy Bridge
Intel CPUs with Redis. When client and server run on the same box, the CPU is
the limiting factor with redis-benchmark.
+ Speed of RAM and memory bandwidth seem less critical for global performance
especially for small objects. For large objects (>10 KB), it may become
noticeable though. Usually, it is not really cost-effective to buy expensive
fast memory modules to optimize Redis.
+ Redis runs slower on a VM compared to running without virtualization using
the same hardware. If you have the chance to run Redis on a physical machine
this is preferred. However this does not mean that Redis is slow in
virtualized environments, the delivered performances are still very good
and most of the serious performance issues you may incur in virtualized
environments are due to over-provisioning, non-local disks with high latency,
or old hypervisor software that have slow `fork` syscall implementation.
+ When the server and client benchmark programs run on the same box, both
the TCP/IP loopback and unix domain sockets can be used. Depending on the
platform, unix domain sockets can achieve around 50% more throughput than
the TCP/IP loopback (on Linux for instance). The default behavior of
redis-benchmark is to use the TCP/IP loopback.
+ The performance benefit of unix domain sockets compared to TCP/IP loopback
tends to decrease when pipelining is heavily used (i.e. long pipelines).
+ When an ethernet network is used to access Redis, aggregating commands using
pipelining is especially efficient when the size of the data is kept under
the ethernet packet size (about 1500 bytes). Actually, processing 10 bytes,
100 bytes, or 1000 bytes queries almost result in the same throughput.
See the graph below.

    ![Data size impact](Data_size.png)

+ On multi CPU sockets servers, Redis performance becomes dependent on the
NUMA configuration and process location. The most visible effect is that
redis-benchmark results seem non-deterministic because client and server
processes are distributed randomly on the cores. To get deterministic results,
it is required to use process placement tools (on Linux: taskset or numactl).
The most efficient combination is always to put the client and server on two
different cores of the same CPU to benefit from the L3 cache.
Here are some results of 4 KB SET benchmark for 3 server CPUs (AMD Istanbul,
Intel Nehalem EX, and Intel Westmere) with different relative placements.
Please note this benchmark is not meant to compare CPU models between themselves
(CPUs exact model and frequency are therefore not disclosed).

    ![NUMA chart](NUMA_chart.gif)

+ With high-end configurations, the number of client connections is also an
important factor. Being based on epoll/kqueue, the Redis event loop is quite
scalable. Redis has already been benchmarked at more than 60000 connections,
and was still able to sustain 50000 q/s in these conditions. As a rule of thumb,
an instance with 30000 connections can only process half the throughput
achievable with 100 connections. Here is an example showing the throughput of
a Redis instance per number of connections:

    ![connections chart](Connections_chart.png)

+ With high-end configurations, it is possible to achieve higher throughput by
tuning the NIC(s) configuration and associated interruptions. Best throughput
is achieved by setting an affinity between Rx/Tx NIC queues and CPU cores,
and activating RPS (Receive Packet Steering) support. More information in this
[thread](https://groups.google.com/forum/#!msg/redis-db/gUhc19gnYgc/BruTPCOroiMJ).
Jumbo frames may also provide a performance boost when large objects are used.
+ Depending on the platform, Redis can be compiled against different memory
allocators (libc malloc, jemalloc, tcmalloc), which may have different behaviors
in term of raw speed, internal and external fragmentation.
If you did not compile Redis yourself, you can use the INFO command to check
the `mem_allocator` field. Please note most benchmarks do not run long enough to
generate significant external fragmentation (contrary to production Redis
instances).

### Other things to consider

One important goal of any benchmark is to get reproducible results, so they
can be compared to the results of other tests.

+ A good practice is to try to run tests on isolated hardware as much as possible.
If it is not possible, then the system must be monitored to check the benchmark
is not impacted by some external activity.
+ Some configurations (desktops and laptops for sure, some servers as well)
have a variable CPU core frequency mechanism. The policy controlling this
mechanism can be set at the OS level. Some CPU models are more aggressive than
others at adapting the frequency of the CPU cores to the workload. To get
reproducible results, it is better to set the highest possible fixed frequency
for all the CPU cores involved in the benchmark.
+ An important point is to size the system accordingly to the benchmark.
The system must have enough RAM and must not swap. On Linux, do not forget
to set the `overcommit_memory` parameter correctly. Please note 32 and 64 bit
Redis instances do not have the same memory footprint.
+ If you plan to use RDB or AOF for your benchmark, please check there is no other
I/O activity in the system. Avoid putting RDB or AOF files on NAS or NFS shares,
or on any other devices impacting your network bandwidth and/or latency
(for instance, EBS on Amazon EC2).
+ Set Redis logging level (loglevel parameter) to warning or notice. Avoid putting
the generated log file on a remote filesystem.
+ Avoid using monitoring tools which can alter the result of the benchmark. For
instance using INFO at regular interval to gather statistics is probably fine,
but MONITOR will impact the measured performance significantly.

### Other Redis benchmarking tools

There are several third-party tools that can be used for benchmarking Redis. Refer to each tool's
documentation for more information about its goals and capabilities.

* [memtier_benchmark](https://github.com/redislabs/memtier_benchmark) from [Redis Ltd.](https://twitter.com/RedisInc) is a NoSQL Redis and Memcache traffic generation and benchmarking tool.
* [rpc-perf](https://github.com/twitter/rpc-perf) from [Twitter](https://twitter.com/twitter) is a tool for benchmarking RPC services that supports Redis and Memcache.
* [YCSB](https://github.com/brianfrankcooper/YCSB) from [Yahoo @Yahoo](https://twitter.com/Yahoo) is a benchmarking framework with clients to many databases, including Redis.

title   Redis benchmark  linkTitle   Benchmarking  weight  1 description        Using the redis benchmark utility on a Redis server aliases         topics benchmarks       docs reference optimization benchmarks       docs reference optimization benchmarks md        Redis includes the  redis benchmark  utility that simulates running commands done by N clients while at the same time sending M total queries  The utility provides a default set of tests  or you can supply a custom set of tests   The following options are supported       Usage  redis benchmark   h  host     p  port     c  clients     n  requests     k  boolean          h  hostname       Server hostname  default 127 0 0 1        p  port           Server port  default 6379        s  socket         Server socket  overrides host and port        a  password       Password for Redis Auth       c  clients        Number of parallel connections  default 50        n  requests       Total number of requests  default 100000        d  size           Data size of SET GET value in bytes  default 3         dbnum  db        SELECT the specified db number  default 0        k  boolean        1 keep alive 0 reconnect  default 1        r  keyspacelen    Use random keys for SET GET INCR  random values for SADD       Using this option the benchmark will expand the string   rand int         inside an argument with a 12 digits number in the specified range       from 0 to keyspacelen 1  The substitution changes every time a command       is executed  Default tests use this to hit random keys in the       specified range        P  numreq         Pipeline  numreq  requests  Default 1  no pipeline         q                 Quiet  Just show query sec values        csv              Output in CSV format       l                 Loop  Run the tests forever       t  tests          Only run the comma separated list of tests  The test                         names are the same as the ones produced as output        I                 Idle mode  Just open N idle connections and wait   You need to have a running Redis instance before launching the benchmark  You can run the benchmarking utility like so       redis benchmark  q  n 100000      Running only a subset of the tests  You don t need to run all the default tests every time you execute  redis benchmark   For example  to select only a subset of tests  use the   t  option as in the following example         redis benchmark  t set lpush  n 100000  q     SET  74239 05 requests per second     LPUSH  79239 30 requests per second  This example runs the tests for the  SET  and  LPUSH  commands and uses quiet mode  see the   q  switch    You can even benchmark a specific command         redis benchmark  n 100000  q script load  redis call  set   foo   bar        script load redis call  set   foo   bar    69881 20 requests per second      Selecting the size of the key space  By default  the benchmark runs against a single key  In Redis the difference between such a synthetic benchmark and a real one is not huge since it is an in memory system  however it is possible to stress cache misses and in general to simulate a more real world work load by using a large key space   This is obtained by using the   r  switch  For instance if I want to run one million SET operations  using a random key for every operation out of 100k possible keys  I ll use the following command line         redis cli flushall     OK        redis benchmark  t set  r 100000  n 1000000            SET              1000000 requests completed in 13 86 seconds       50 parallel clients       3 bytes payload       keep alive  1      99 76       1 milliseconds     99 98       2 milliseconds     100 00       3 milliseconds     100 00       3 milliseconds     72144 87 requests per second        redis cli dbsize      integer  99993      Using pipelining  By default every client  the benchmark simulates 50 clients if not otherwise specified with   c   sends the next command only when the reply of the previous command is received  this means that the server will likely need a read call in order to read each command from every client  Also RTT is paid as well   Redis supports  pipelining   topics pipelining   so it is possible to send multiple commands at once  a feature often exploited by real world applications  Redis pipelining is able to dramatically improve the number of operations per second a server is able do deliver   Consider this example of running the benchmark using a pipelining of 16 commands         redis benchmark  n 1000000  t set get  P 16  q     SET  403063 28 requests per second     GET  508388 41 requests per second  Using pipelining results in a significant increase in performance       Pitfalls and misconceptions  The first point is obvious  the golden rule of a useful benchmark is to only compare apples and apples  You can compare different versions of Redis on the same workload or the same version of Redis  but with different options  If you plan to compare Redis to something else  then it is important to evaluate the functional and technical differences  and take them in account     Redis is a server  all commands involve network or IPC round trips  It is meaningless to compare it to embedded data stores  because the cost of most operations is primarily in network protocol management    Redis commands return an acknowledgment for all usual commands  Some other data stores do not  Comparing Redis to stores involving one way queries is only mildly useful    Naively iterating on synchronous Redis commands does not benchmark Redis itself  but rather measure your network  or IPC  latency and the client library intrinsic latency  To really test Redis  you need multiple connections  like redis benchmark  and or to use pipelining to aggregate several commands and or multiple threads or processes    Redis is an in memory data store with some optional persistence options  If you plan to compare it to transactional servers  MySQL  PostgreSQL  etc       then you should consider activating AOF and decide on a suitable fsync policy    Redis is  mostly  a single threaded server from the POV of commands execution  actually modern versions of Redis use threads for different things   It is not designed to benefit from multiple CPU cores  People are supposed to launch several Redis instances to scale out on several cores if needed  It is not really fair to compare one single Redis instance to a multi threaded data store   The  redis benchmark  program is a quick and useful way to get some figures and evaluate the performance of a Redis instance on a given hardware  However  by default  it does not represent the maximum throughput a Redis instance can sustain  Actually  by using pipelining and a fast client  hiredis   it is fairly easy to write a program generating more throughput than redis benchmark  The default behavior of redis benchmark is to achieve throughput by exploiting concurrency only  i e  it creates several connections to the server   It does not use pipelining or any parallelism at all  one pending query per connection at most  and no multi threading   if not explicitly enabled via the   P  parameter  So in some way using  redis benchmark  and  triggering  for example  a  BGSAVE  operation in the background at the same time  will provide the user with numbers more near to the  worst case  than to the best case   To run a benchmark using pipelining mode  and achieve higher throughput   you need to explicitly use the  P option  Please note that it is still a realistic behavior since a lot of Redis based applications actively use pipelining to improve performance  However you should use a pipeline size that is more or less the average pipeline length you ll be able to use in your application in order to get realistic numbers   The benchmark should apply the same operations  and work in the same way with the multiple data stores you want to compare  It is absolutely pointless to compare the result of redis benchmark to the result of another benchmark program and extrapolate   For instance  Redis and memcached in single threaded mode can be compared on GET SET operations  Both are in memory data stores  working mostly in the same way at the protocol level  Provided their respective benchmark application is aggregating queries in the same way  pipelining  and use a similar number of connections  the comparison is actually meaningful   When you re benchmarking a high performance  in memory database like Redis  it may be difficult to saturate the server  Sometimes  the performance bottleneck is on the client side  and not the server side  In that case  the client  i e   the benchmarking program itself  must be fixed  or perhaps scaled out  to reach the maximum throughput       Factors impacting Redis performance  There are multiple factors having direct consequences on Redis performance  We mention them here  since they can alter the result of any benchmarks  Please note however  that a typical Redis instance running on a low end  untuned box usually provides good enough performance for most applications     Network bandwidth and latency usually have a direct impact on the performance  It is a good practice to use the ping program to quickly check the latency between the client and server hosts is normal before launching the benchmark  Regarding the bandwidth  it is generally useful to estimate the throughput in Gbit s and compare it to the theoretical bandwidth of the network  For instance a benchmark setting 4 KB strings in Redis at 100000 q s  would actually consume 3 2 Gbit s of bandwidth and probably fit within a 10 Gbit s link  but not a 1 Gbit s one  In many real world scenarios  Redis throughput is limited by the network well before being limited by the CPU  To consolidate several high throughput Redis instances on a single server  it worth considering putting a 10 Gbit s NIC or multiple 1 Gbit s NICs with TCP IP bonding    CPU is another very important factor  Being single threaded  Redis favors fast CPUs with large caches and not many cores  At this game  Intel CPUs are currently the winners  It is not uncommon to get only half the performance on an AMD Opteron CPU compared to similar Nehalem EP Westmere EP Sandy Bridge Intel CPUs with Redis  When client and server run on the same box  the CPU is the limiting factor with redis benchmark    Speed of RAM and memory bandwidth seem less critical for global performance especially for small objects  For large objects   10 KB   it may become noticeable though  Usually  it is not really cost effective to buy expensive fast memory modules to optimize Redis    Redis runs slower on a VM compared to running without virtualization using the same hardware  If you have the chance to run Redis on a physical machine this is preferred  However this does not mean that Redis is slow in virtualized environments  the delivered performances are still very good and most of the serious performance issues you may incur in virtualized environments are due to over provisioning  non local disks with high latency  or old hypervisor software that have slow  fork  syscall implementation    When the server and client benchmark programs run on the same box  both the TCP IP loopback and unix domain sockets can be used  Depending on the platform  unix domain sockets can achieve around 50  more throughput than the TCP IP loopback  on Linux for instance   The default behavior of redis benchmark is to use the TCP IP loopback    The performance benefit of unix domain sockets compared to TCP IP loopback tends to decrease when pipelining is heavily used  i e  long pipelines     When an ethernet network is used to access Redis  aggregating commands using pipelining is especially efficient when the size of the data is kept under the ethernet packet size  about 1500 bytes   Actually  processing 10 bytes  100 bytes  or 1000 bytes queries almost result in the same throughput  See the graph below         Data size impact  Data size png     On multi CPU sockets servers  Redis performance becomes dependent on the NUMA configuration and process location  The most visible effect is that redis benchmark results seem non deterministic because client and server processes are distributed randomly on the cores  To get deterministic results  it is required to use process placement tools  on Linux  taskset or numactl   The most efficient combination is always to put the client and server on two different cores of the same CPU to benefit from the L3 cache  Here are some results of 4 KB SET benchmark for 3 server CPUs  AMD Istanbul  Intel Nehalem EX  and Intel Westmere  with different relative placements  Please note this benchmark is not meant to compare CPU models between themselves  CPUs exact model and frequency are therefore not disclosed          NUMA chart  NUMA chart gif     With high end configurations  the number of client connections is also an important factor  Being based on epoll kqueue  the Redis event loop is quite scalable  Redis has already been benchmarked at more than 60000 connections  and was still able to sustain 50000 q s in these conditions  As a rule of thumb  an instance with 30000 connections can only process half the throughput achievable with 100 connections  Here is an example showing the throughput of a Redis instance per number of connections         connections chart  Connections chart png     With high end configurations  it is possible to achieve higher throughput by tuning the NIC s  configuration and associated interruptions  Best throughput is achieved by setting an affinity between Rx Tx NIC queues and CPU cores  and activating RPS  Receive Packet Steering  support  More information in this  thread  https   groups google com forum   msg redis db gUhc19gnYgc BruTPCOroiMJ   Jumbo frames may also provide a performance boost when large objects are used    Depending on the platform  Redis can be compiled against different memory allocators  libc malloc  jemalloc  tcmalloc   which may have different behaviors in term of raw speed  internal and external fragmentation  If you did not compile Redis yourself  you can use the INFO command to check the  mem allocator  field  Please note most benchmarks do not run long enough to generate significant external fragmentation  contrary to production Redis instances        Other things to consider  One important goal of any benchmark is to get reproducible results  so they can be compared to the results of other tests     A good practice is to try to run tests on isolated hardware as much as possible  If it is not possible  then the system must be monitored to check the benchmark is not impacted by some external activity    Some configurations  desktops and laptops for sure  some servers as well  have a variable CPU core frequency mechanism  The policy controlling this mechanism can be set at the OS level  Some CPU models are more aggressive than others at adapting the frequency of the CPU cores to the workload  To get reproducible results  it is better to set the highest possible fixed frequency for all the CPU cores involved in the benchmark    An important point is to size the system accordingly to the benchmark  The system must have enough RAM and must not swap  On Linux  do not forget to set the  overcommit memory  parameter correctly  Please note 32 and 64 bit Redis instances do not have the same memory footprint    If you plan to use RDB or AOF for your benchmark  please check there is no other I O activity in the system  Avoid putting RDB or AOF files on NAS or NFS shares  or on any other devices impacting your network bandwidth and or latency  for instance  EBS on Amazon EC2     Set Redis logging level  loglevel parameter  to warning or notice  Avoid putting the generated log file on a remote filesystem    Avoid using monitoring tools which can alter the result of the benchmark  For instance using INFO at regular interval to gather statistics is probably fine  but MONITOR will impact the measured performance significantly       Other Redis benchmarking tools  There are several third party tools that can be used for benchmarking Redis  Refer to each tool s documentation for more information about its goals and capabilities      memtier benchmark  https   github com redislabs memtier benchmark  from  Redis Ltd   https   twitter com RedisInc  is a NoSQL Redis and Memcache traffic generation and benchmarking tool     rpc perf  https   github com twitter rpc perf  from  Twitter  https   twitter com twitter  is a tool for benchmarking RPC services that supports Redis and Memcache     YCSB  https   github com brianfrankcooper YCSB  from  Yahoo  Yahoo  https   twitter com Yahoo  is a benchmarking framework with clients to many databases  including Redis


scikit-learn is the best approach for most users It will provide a stable version Installing scikit learn There are different ways to install scikit learn This installation instructions

Answer:
.. _installation-instructions:

=======================
Installing scikit-learn
=======================

There are different ways to install scikit-learn:

* :ref:`Install the latest official release <install_official_release>`. This
  is the best approach for most users. It will provide a stable version
  and pre-built packages are available for most platforms.

* Install the version of scikit-learn provided by your
  :ref:`operating system or Python distribution <install_by_distribution>`.
  This is a quick option for those who have operating systems or Python
  distributions that distribute scikit-learn.
  It might not provide the latest release version.

* :ref:`Building the package from source
  <install_bleeding_edge>`. This is best for users who want the
  latest-and-greatest features and aren't afraid of running
  brand-new code. This is also needed for users who wish to contribute to the
  project.

.. _install_official_release:

Installing the latest release
=============================

.. raw:: html

  <style>
    /* Show caption on large screens */
    @media screen and (min-width: 960px) {
      .install-instructions .sd-tab-set {
        --tab-caption-width: 20%;
      }

      .install-instructions .sd-tab-set.tabs-os::before {
        content: "Operating System";
      }

      .install-instructions .sd-tab-set.tabs-package-manager::before {
        content: "Package Manager";
      }
    }
  </style>

.. div:: install-instructions

  .. tab-set::
    :class: tabs-os

    .. tab-item:: Windows
      :class-label: tab-4

      .. tab-set::
        :class: tabs-package-manager

        .. tab-item:: pip
          :class-label: tab-6
          :sync: package-manager-pip

          Install the 64-bit version of Python 3, for instance from the
          `official website <https://www.python.org/downloads/windows/>`__.

          Now create a `virtual environment (venv)
          <https://docs.python.org/3/tutorial/venv.html>`_ and install scikit-learn.
          Note that the virtual environment is optional but strongly recommended, in
          order to avoid potential conflicts with other packages.

          .. prompt:: powershell

            python -m venv sklearn-env
            sklearn-env\Scripts\activate  # activate
            pip install -U scikit-learn

          In order to check your installation, you can use:

          .. prompt:: powershell

            python -m pip show scikit-learn  # show scikit-learn version and location
            python -m pip freeze             # show all installed packages in the environment
            python -c "import sklearn; sklearn.show_versions()"

        .. tab-item:: conda
          :class-label: tab-6
          :sync: package-manager-conda

          .. include:: ./install_instructions_conda.rst

    .. tab-item:: MacOS
      :class-label: tab-4

      .. tab-set::
        :class: tabs-package-manager

        .. tab-item:: pip
          :class-label: tab-6
          :sync: package-manager-pip

          Install Python 3 using `homebrew <https://brew.sh/>`_ (`brew install python`)
          or by manually installing the package from the `official website
          <https://www.python.org/downloads/macos/>`__.

          Now create a `virtual environment (venv)
          <https://docs.python.org/3/tutorial/venv.html>`_ and install scikit-learn.
          Note that the virtual environment is optional but strongly recommended, in
          order to avoid potential conflicts with other packges.

          .. prompt:: bash

            python -m venv sklearn-env
            source sklearn-env/bin/activate  # activate
            pip install -U scikit-learn

          In order to check your installation, you can use:

          .. prompt:: bash

            python -m pip show scikit-learn  # show scikit-learn version and location
            python -m pip freeze             # show all installed packages in the environment
            python -c "import sklearn; sklearn.show_versions()"

        .. tab-item:: conda
          :class-label: tab-6
          :sync: package-manager-conda

          .. include:: ./install_instructions_conda.rst

    .. tab-item:: Linux
      :class-label: tab-4

      .. tab-set::
        :class: tabs-package-manager

        .. tab-item:: pip
          :class-label: tab-6
          :sync: package-manager-pip

          Python 3 is usually installed by default on most Linux distributions. To
          check if you have it installed, try:

          .. prompt:: bash

            python3 --version
            pip3 --version

          If you don't have Python 3 installed, please install `python3` and
          `python3-pip` from your distribution's package manager.

          Now create a `virtual environment (venv)
          <https://docs.python.org/3/tutorial/venv.html>`_ and install scikit-learn.
          Note that the virtual environment is optional but strongly recommended, in
          order to avoid potential conflicts with other packages.

          .. prompt:: bash

            python3 -m venv sklearn-env
            source sklearn-env/bin/activate  # activate
            pip3 install -U scikit-learn

          In order to check your installation, you can use:

          .. prompt:: bash

            python3 -m pip show scikit-learn  # show scikit-learn version and location
            python3 -m pip freeze             # show all installed packages in the environment
            python3 -c "import sklearn; sklearn.show_versions()"

        .. tab-item:: conda
          :class-label: tab-6
          :sync: package-manager-conda

          .. include:: ./install_instructions_conda.rst

Using an isolated environment such as pip venv or conda makes it possible to
install a specific version of scikit-learn with pip or conda and its dependencies
independently of any previously installed Python packages. In particular under Linux
it is discouraged to install pip packages alongside the packages managed by the
package manager of the distribution (apt, dnf, pacman...).

Note that you should always remember to activate the environment of your choice
prior to running any Python command whenever you start a new terminal session.

If you have not installed NumPy or SciPy yet, you can also install these using
conda or pip. When using pip, please ensure that *binary wheels* are used,
and NumPy and SciPy are not recompiled from source, which can happen when using
particular configurations of operating system and hardware (such as Linux on
a Raspberry Pi).

Scikit-learn plotting capabilities (i.e., functions starting with `plot\_`
and classes ending with `Display`) require Matplotlib. The examples require
Matplotlib and some examples require scikit-image, pandas, or seaborn. The
minimum version of scikit-learn dependencies are listed below along with its
purpose.

.. include:: min_dependency_table.rst

.. warning::

    Scikit-learn 0.20 was the last version to support Python 2.7 and Python 3.4.
    Scikit-learn 0.21 supported Python 3.5-3.7.
    Scikit-learn 0.22 supported Python 3.5-3.8.
    Scikit-learn 0.23-0.24 required Python 3.6 or newer.
    Scikit-learn 1.0 supported Python 3.7-3.10.
    Scikit-learn 1.1, 1.2 and 1.3 support Python 3.8-3.12
    Scikit-learn 1.4 requires Python 3.9 or newer.

.. _install_by_distribution:

Third party distributions of scikit-learn
=========================================

Some third-party distributions provide versions of
scikit-learn integrated with their package-management systems.

These can make installation and upgrading much easier for users since
the integration includes the ability to automatically install
dependencies (numpy, scipy) that scikit-learn requires.

The following is an incomplete list of OS and python distributions
that provide their own version of scikit-learn.

Alpine Linux
------------

Alpine Linux's package is provided through the `official repositories
<https://pkgs.alpinelinux.org/packages?name=py3-scikit-learn>`__ as
``py3-scikit-learn`` for Python.
It can be installed by typing the following command:

.. prompt:: bash

  sudo apk add py3-scikit-learn

Arch Linux
----------

Arch Linux's package is provided through the `official repositories
<https://www.archlinux.org/packages/?q=scikit-learn>`_ as
``python-scikit-learn`` for Python.
It can be installed by typing the following command:

.. prompt:: bash

  sudo pacman -S python-scikit-learn

Debian/Ubuntu
-------------

The Debian/Ubuntu package is split in three different packages called
``python3-sklearn`` (python modules), ``python3-sklearn-lib`` (low-level
implementations and bindings), ``python-sklearn-doc`` (documentation).
Note that scikit-learn requires Python 3, hence the need to use the `python3-`
suffixed package names.
Packages can be installed using ``apt-get``:

.. prompt:: bash

  sudo apt-get install python3-sklearn python3-sklearn-lib python-sklearn-doc

Fedora
------

The Fedora package is called ``python3-scikit-learn`` for the python 3 version,
the only one available in Fedora.
It can be installed using ``dnf``:

.. prompt:: bash

  sudo dnf install python3-scikit-learn

NetBSD
------

scikit-learn is available via `pkgsrc-wip <http://pkgsrc-wip.sourceforge.net/>`_:
https://pkgsrc.se/math/py-scikit-learn

MacPorts for Mac OSX
--------------------

The MacPorts package is named ``py<XY>-scikits-learn``,
where ``XY`` denotes the Python version.
It can be installed by typing the following
command:

.. prompt:: bash

  sudo port install py39-scikit-learn

Anaconda and Enthought Deployment Manager for all supported platforms
---------------------------------------------------------------------

`Anaconda <https://www.anaconda.com/download>`_ and
`Enthought Deployment Manager <https://assets.enthought.com/downloads/>`_
both ship with scikit-learn in addition to a large set of scientific
python library for Windows, Mac OSX and Linux.

Anaconda offers scikit-learn as part of its free distribution.

Intel Extension for Scikit-learn
--------------------------------

Intel maintains an optimized x86_64 package, available in PyPI (via `pip`),
and in the `main`, `conda-forge` and `intel` conda channels:

.. prompt:: bash

  conda install scikit-learn-intelex

This package has an Intel optimized version of many estimators. Whenever
an alternative implementation doesn't exist, scikit-learn implementation
is used as a fallback. Those optimized solvers come from the oneDAL
C++ library and are optimized for the x86_64 architecture, and are
optimized for multi-core Intel CPUs.

Note that those solvers are not enabled by default, please refer to the
`scikit-learn-intelex <https://intel.github.io/scikit-learn-intelex/latest/what-is-patching.html>`_
documentation for more details on usage scenarios. Direct export example:

.. prompt:: python >>>

  from sklearnex.neighbors import NearestNeighbors

Compatibility with the standard scikit-learn solvers is checked by running the
full scikit-learn test suite via automated continuous integration as reported
on https://github.com/intel/scikit-learn-intelex. If you observe any issue
with `scikit-learn-intelex`, please report the issue on their
`issue tracker <https://github.com/intel/scikit-learn-intelex/issues>`__.

WinPython for Windows
---------------------

The `WinPython <https://winpython.github.io/>`_ project distributes
scikit-learn as an additional plugin.

Troubleshooting
===============

If you encounter unexpected failures when installing scikit-learn, you may submit
an issue to the `issue tracker <https://github.com/scikit-learn/scikit-learn/issues>`_.
Before that, please also make sure to check the following common issues.

.. _windows_longpath:

Error caused by file path length limit on Windows
-------------------------------------------------

It can happen that pip fails to install packages when reaching the default path
size limit of Windows if Python is installed in a nested location such as the
`AppData` folder structure under the user home directory, for instance::

    C:\Users\username>C:\Users\username\AppData\Local\Microsoft\WindowsApps\python.exe -m pip install scikit-learn
    Collecting scikit-learn
    ...
    Installing collected packages: scikit-learn
    ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'C:\\Users\\username\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python37\\site-packages\\sklearn\\datasets\\tests\\data\\openml\\292\\api-v1-json-data-list-data_name-australian-limit-2-data_version-1-status-deactivated.json.gz'

In this case it is possible to lift that limit in the Windows registry by
using the ``regedit`` tool:

#. Type "regedit" in the Windows start menu to launch ``regedit``.

#. Go to the
   ``Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem``
   key.

#. Edit the value of the ``LongPathsEnabled`` property of that key and set
   it to 1.

#. Reinstall scikit-learn (ignoring the previous broken installation):

   .. prompt:: powershell

      pip install --exists-action=i scikit-learn

installation instructions                           Installing scikit learn                          There are different ways to install scikit learn      ref  Install the latest official release  install official release    This   is the best approach for most users  It will provide a stable version   and pre built packages are available for most platforms     Install the version of scikit learn provided by your    ref  operating system or Python distribution  install by distribution      This is a quick option for those who have operating systems or Python   distributions that distribute scikit learn    It might not provide the latest release version      ref  Building the package from source    install bleeding edge    This is best for users who want the   latest and greatest features and aren t afraid of running   brand new code  This is also needed for users who wish to contribute to the   project        install official release   Installing the latest release                                   raw   html     style         Show caption on large screens         media screen and  min width  960px           install instructions  sd tab set             tab caption width  20                   install instructions  sd tab set tabs os  before           content   Operating System                   install instructions  sd tab set tabs package manager  before           content   Package Manager                     style      div   install instructions       tab set        class  tabs os         tab item   Windows        class label  tab 4           tab set            class  tabs package manager             tab item   pip            class label  tab 6            sync  package manager pip            Install the 64 bit version of Python 3  for instance from the            official website  https   www python org downloads windows                  Now create a  virtual environment  venv             https   docs python org 3 tutorial venv html    and install scikit learn            Note that the virtual environment is optional but strongly recommended  in           order to avoid potential conflicts with other packages                prompt   powershell              python  m venv sklearn env             sklearn env Scripts activate    activate             pip install  U scikit learn            In order to check your installation  you can use                prompt   powershell              python  m pip show scikit learn    show scikit learn version and location             python  m pip freeze               show all installed packages in the environment             python  c  import sklearn  sklearn show versions                tab item   conda            class label  tab 6            sync  package manager conda               include     install instructions conda rst         tab item   MacOS        class label  tab 4           tab set            class  tabs package manager             tab item   pip            class label  tab 6            sync  package manager pip            Install Python 3 using  homebrew  https   brew sh       brew install python             or by manually installing the package from the  official website            https   www python org downloads macos                  Now create a  virtual environment  venv             https   docs python org 3 tutorial venv html    and install scikit learn            Note that the virtual environment is optional but strongly recommended  in           order to avoid potential conflicts with other packges                prompt   bash              python  m venv sklearn env             source sklearn env bin activate    activate             pip install  U scikit learn            In order to check your installation  you can use                prompt   bash              python  m pip show scikit learn    show scikit learn version and location             python  m pip freeze               show all installed packages in the environment             python  c  import sklearn  sklearn show versions                tab item   conda            class label  tab 6            sync  package manager conda               include     install instructions conda rst         tab item   Linux        class label  tab 4           tab set            class  tabs package manager             tab item   pip            class label  tab 6            sync  package manager pip            Python 3 is usually installed by default on most Linux distributions  To           check if you have it installed  try                prompt   bash              python3   version             pip3   version            If you don t have Python 3 installed  please install  python3  and            python3 pip  from your distribution s package manager             Now create a  virtual environment  venv             https   docs python org 3 tutorial venv html    and install scikit learn            Note that the virtual environment is optional but strongly recommended  in           order to avoid potential conflicts with other packages                prompt   bash              python3  m venv sklearn env             source sklearn env bin activate    activate             pip3 install  U scikit learn            In order to check your installation  you can use                prompt   bash              python3  m pip show scikit learn    show scikit learn version and location             python3  m pip freeze               show all installed packages in the environment             python3  c  import sklearn  sklearn show versions                tab item   conda            class label  tab 6            sync  package manager conda               include     install instructions conda rst   Using an isolated environment such as pip venv or conda makes it possible to install a specific version of scikit learn with pip or conda and its dependencies independently of any previously installed Python packages  In particular under Linux it is discouraged to install pip packages alongside the packages managed by the package manager of the distribution  apt  dnf  pacman       Note that you should always remember to activate the environment of your choice prior to running any Python command whenever you start a new terminal session   If you have not installed NumPy or SciPy yet  you can also install these using conda or pip  When using pip  please ensure that  binary wheels  are used  and NumPy and SciPy are not recompiled from source  which can happen when using particular configurations of operating system and hardware  such as Linux on a Raspberry Pi    Scikit learn plotting capabilities  i e   functions starting with  plot    and classes ending with  Display   require Matplotlib  The examples require Matplotlib and some examples require scikit image  pandas  or seaborn  The minimum version of scikit learn dependencies are listed below along with its purpose      include   min dependency table rst     warning        Scikit learn 0 20 was the last version to support Python 2 7 and Python 3 4      Scikit learn 0 21 supported Python 3 5 3 7      Scikit learn 0 22 supported Python 3 5 3 8      Scikit learn 0 23 0 24 required Python 3 6 or newer      Scikit learn 1 0 supported Python 3 7 3 10      Scikit learn 1 1  1 2 and 1 3 support Python 3 8 3 12     Scikit learn 1 4 requires Python 3 9 or newer       install by distribution   Third party distributions of scikit learn                                            Some third party distributions provide versions of scikit learn integrated with their package management systems   These can make installation and upgrading much easier for users since the integration includes the ability to automatically install dependencies  numpy  scipy  that scikit learn requires   The following is an incomplete list of OS and python distributions that provide their own version of scikit learn   Alpine Linux               Alpine Linux s package is provided through the  official repositories  https   pkgs alpinelinux org packages name py3 scikit learn     as   py3 scikit learn   for Python  It can be installed by typing the following command      prompt   bash    sudo apk add py3 scikit learn   Arch Linux             Arch Linux s package is provided through the  official repositories  https   www archlinux org packages  q scikit learn    as   python scikit learn   for Python  It can be installed by typing the following command      prompt   bash    sudo pacman  S python scikit learn   Debian Ubuntu                The Debian Ubuntu package is split in three different packages called   python3 sklearn    python modules     python3 sklearn lib    low level implementations and bindings     python sklearn doc    documentation   Note that scikit learn requires Python 3  hence the need to use the  python3   suffixed package names  Packages can be installed using   apt get        prompt   bash    sudo apt get install python3 sklearn python3 sklearn lib python sklearn doc   Fedora         The Fedora package is called   python3 scikit learn   for the python 3 version  the only one available in Fedora  It can be installed using   dnf        prompt   bash    sudo dnf install python3 scikit learn   NetBSD         scikit learn is available via  pkgsrc wip  http   pkgsrc wip sourceforge net      https   pkgsrc se math py scikit learn   MacPorts for Mac OSX                       The MacPorts package is named   py XY  scikits learn    where   XY   denotes the Python version  It can be installed by typing the following command      prompt   bash    sudo port install py39 scikit learn   Anaconda and Enthought Deployment Manager for all supported platforms                                                                         Anaconda  https   www anaconda com download    and  Enthought Deployment Manager  https   assets enthought com downloads     both ship with scikit learn in addition to a large set of scientific python library for Windows  Mac OSX and Linux   Anaconda offers scikit learn as part of its free distribution    Intel Extension for Scikit learn                                   Intel maintains an optimized x86 64 package  available in PyPI  via  pip    and in the  main    conda forge  and  intel  conda channels      prompt   bash    conda install scikit learn intelex  This package has an Intel optimized version of many estimators  Whenever an alternative implementation doesn t exist  scikit learn implementation is used as a fallback  Those optimized solvers come from the oneDAL C   library and are optimized for the x86 64 architecture  and are optimized for multi core Intel CPUs   Note that those solvers are not enabled by default  please refer to the  scikit learn intelex  https   intel github io scikit learn intelex latest what is patching html    documentation for more details on usage scenarios  Direct export example      prompt   python        from sklearnex neighbors import NearestNeighbors  Compatibility with the standard scikit learn solvers is checked by running the full scikit learn test suite via automated continuous integration as reported on https   github com intel scikit learn intelex  If you observe any issue with  scikit learn intelex   please report the issue on their  issue tracker  https   github com intel scikit learn intelex issues        WinPython for Windows                        The  WinPython  https   winpython github io     project distributes scikit learn as an additional plugin    Troubleshooting                  If you encounter unexpected failures when installing scikit learn  you may submit an issue to the  issue tracker  https   github com scikit learn scikit learn issues     Before that  please also make sure to check the following common issues       windows longpath   Error caused by file path length limit on Windows                                                    It can happen that pip fails to install packages when reaching the default path size limit of Windows if Python is installed in a nested location such as the  AppData  folder structure under the user home directory  for instance        C  Users username C  Users username AppData Local Microsoft WindowsApps python exe  m pip install scikit learn     Collecting scikit learn             Installing collected packages  scikit learn     ERROR  Could not install packages due to an OSError   Errno 2  No such file or directory   C   Users  username  AppData  Local  Packages  PythonSoftwareFoundation Python 3 7 qbz5n2kfra8p0  LocalCache  local packages  Python37  site packages  sklearn  datasets  tests  data  openml  292  api v1 json data list data name australian limit 2 data version 1 status deactivated json gz   In this case it is possible to lift that limit in the Windows registry by using the   regedit   tool      Type  regedit  in the Windows start menu to launch   regedit        Go to the      Computer HKEY LOCAL MACHINE SYSTEM CurrentControlSet Control FileSystem      key      Edit the value of the   LongPathsEnabled   property of that key and set    it to 1      Reinstall scikit learn  ignoring the previous broken installation          prompt   powershell        pip install   exists action i scikit learn


Answer:
.. |ss| raw:: html

   <strike>

.. |se| raw:: html

   </strike>

.. _roadmap:

Roadmap
=======

Purpose of this document
------------------------
This document list general directions that core contributors are interested
to see developed in scikit-learn. The fact that an item is listed here is in
no way a promise that it will happen, as resources are limited. Rather, it
is an indication that help is welcomed on this topic.

Statement of purpose: Scikit-learn in 2018
------------------------------------------
Eleven years after the inception of Scikit-learn, much has changed in the
world of machine learning. Key changes include:

* Computational tools: The exploitation of GPUs, distributed programming
  frameworks like Scala/Spark, etc.
* High-level Python libraries for experimentation, processing and data
  management: Jupyter notebook, Cython, Pandas, Dask, Numba...
* Changes in the focus of machine learning research: artificial intelligence
  applications (where input structure is key) with deep learning,
  representation learning, reinforcement learning, domain transfer, etc.

A more subtle change over the last decade is that, due to changing interests
in ML, PhD students in machine learning are more likely to contribute to
PyTorch, Dask, etc. than to Scikit-learn, so our contributor pool is very
different to a decade ago.

Scikit-learn remains very popular in practice for trying out canonical
machine learning techniques, particularly for applications in experimental
science and in data science. A lot of what we provide is now very mature.
But it can be costly to maintain, and we cannot therefore include arbitrary
new implementations. Yet Scikit-learn is also essential in defining an API
framework for the development of interoperable machine learning components
external to the core library.

**Thus our main goals in this era are to**:

* continue maintaining a high-quality, well-documented collection of canonical
  tools for data processing and machine learning within the current scope
  (i.e. rectangular data largely invariant to column and row order;
  predicting targets with simple structure)
* improve the ease for users to develop and publish external components
* improve interoperability with modern data science tools (e.g. Pandas, Dask)
  and infrastructures (e.g. distributed processing)

Many of the more fine-grained goals can be found under the `API tag
<https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3AAPI>`_
on the issue tracker.

Architectural / general goals
-----------------------------
The list is numbered not as an indication of the order of priority, but to
make referring to specific points easier. Please add new entries only at the
bottom. Note that the crossed out entries are already done, and we try to keep
the document up to date as we work on these issues.

#. Improved handling of Pandas DataFrames

   * document current handling

#. Improved handling of categorical features

   * Tree-based models should be able to handle both continuous and categorical
     features :issue:`29437`.
   * Handling mixtures of categorical and continuous variables

#. Improved handling of missing data

   * Making sure meta-estimators are lenient towards missing data by implementing
     a common test.
   * An amputation sample generator to make parts of a dataset go missing
     :issue:`6284`

#. More didactic documentation

   * More and more options have been added to scikit-learn. As a result, the
     documentation is crowded which makes it hard for beginners to get the big
     picture. Some work could be done in prioritizing the information.

#. Passing around information that is not (X, y): Feature properties

   * Per-feature handling (e.g. "is this a nominal / ordinal / English language
     text?") should also not need to be provided to estimator constructors,
     ideally, but should be available as metadata alongside X. :issue:`8480`

#. Passing around information that is not (X, y): Target information

   * We have problems getting the full set of classes to all components when
     the data is split/sampled. :issue:`6231` :issue:`8100`
   * We have no way to handle a mixture of categorical and continuous targets.

#. Make it easier for external users to write Scikit-learn-compatible
   components

   * More self-sufficient running of scikit-learn-contrib or a similar resource

#. Support resampling and sample reduction

   * Allow subsampling of majority classes (in a pipeline?) :issue:`3855`

#. Better interfaces for interactive development

   * Improve the HTML visualisations of estimators via the `estimator_html_repr`.
   * Include more plotting tools, not just as examples.

#. Improved tools for model diagnostics and basic inference

   * work on a unified interface for "feature importance"
   * better ways to handle validation sets when fitting

#. Better tools for selecting hyperparameters with transductive estimators

   * Grid search and cross validation are not applicable to most clustering
     tasks. Stability-based selection is more relevant.

#. Better support for manual and automatic pipeline building

   * Easier way to construct complex pipelines and valid search spaces
     :issue:`7608` :issue:`5082` :issue:`8243`
   * provide search ranges for common estimators??
   * cf. `searchgrid <https://searchgrid.readthedocs.io/en/latest/>`_

#. Improved tracking of fitting

   * Verbose is not very friendly and should use a standard logging library
     :issue:`6929`, :issue:`78`
   * Callbacks or a similar system would facilitate logging and early stopping

#. Distributed parallelism

   * Accept data which complies with ``__array_function__``

#. A way forward for more out of core

   * Dask enables easy out-of-core computation. While the Dask model probably
     cannot be adaptable to all machine-learning algorithms, most machine
     learning is on smaller data than ETL, hence we can maybe adapt to very
     large scale while supporting only a fraction of the patterns.

#. Backwards-compatible de/serialization of some estimators

   * Currently serialization (with pickle) breaks across versions. While we may
     not be able to get around other limitations of pickle re security etc, it
     would be great to offer cross-version safety from version 1.0. Note: Gael
     and Olivier think that this can cause heavy maintenance burden and we
     should manage the trade-offs. A possible alternative is presented in the
     following point.

#. Documentation and tooling for model lifecycle management

   * Document good practices for model deployments and lifecycle: before
     deploying a model: snapshot the code versions (numpy, scipy, scikit-learn,
     custom code repo), the training script and an alias on how to retrieve
     historical training data + snapshot a copy of a small validation set +
     snapshot of the predictions (predicted probabilities for classifiers)
     on that validation set.
   * Document and tools to make it easy to manage upgrade of scikit-learn
     versions:

     * Try to load the old pickle, if it works, use the validation set
       prediction snapshot to detect that the serialized model still behave
       the same;
     * If joblib.load / pickle.load not work, use the versioned control
       training script + historical training set to retrain the model and use
       the validation set prediction snapshot to assert that it is possible to
       recover the previous predictive performance: if this is not the case
       there is probably a bug in scikit-learn that needs to be reported.

#. Everything in scikit-learn should probably conform to our API contract.
   We are still in the process of making decisions on some of these related
   issues.

   * `Pipeline <pipeline.Pipeline>` and `FeatureUnion` modify their input
     parameters in fit. Fixing this requires making sure we have a good
     grasp of their use cases to make sure all current functionality is
     maintained. :issue:`8157` :issue:`7382`

#. (Optional) Improve scikit-learn common tests suite to make sure that (at
   least for frequently used) models have stable predictions across-versions
   (to be discussed);

   * Extend documentation to mention how to deploy models in Python-free
     environments for instance `ONNX <https://github.com/onnx/sklearn-onnx>`_.
     and use the above best practices to assess predictive consistency between
     scikit-learn and ONNX prediction functions on validation set.
   * Document good practices to detect temporal distribution drift for deployed
     model and good practices for re-training on fresh data without causing
     catastrophic predictive performance regressions.

ss  raw   html      strike       se  raw   html       strike       roadmap   Roadmap          Purpose of this document                          This document list general directions that core contributors are interested to see developed in scikit learn  The fact that an item is listed here is in no way a promise that it will happen  as resources are limited  Rather  it is an indication that help is welcomed on this topic   Statement of purpose  Scikit learn in 2018                                            Eleven years after the inception of Scikit learn  much has changed in the world of machine learning  Key changes include     Computational tools  The exploitation of GPUs  distributed programming   frameworks like Scala Spark  etc    High level Python libraries for experimentation  processing and data   management  Jupyter notebook  Cython  Pandas  Dask  Numba      Changes in the focus of machine learning research  artificial intelligence   applications  where input structure is key  with deep learning    representation learning  reinforcement learning  domain transfer  etc   A more subtle change over the last decade is that  due to changing interests in ML  PhD students in machine learning are more likely to contribute to PyTorch  Dask  etc  than to Scikit learn  so our contributor pool is very different to a decade ago   Scikit learn remains very popular in practice for trying out canonical machine learning techniques  particularly for applications in experimental science and in data science  A lot of what we provide is now very mature  But it can be costly to maintain  and we cannot therefore include arbitrary new implementations  Yet Scikit learn is also essential in defining an API framework for the development of interoperable machine learning components external to the core library     Thus our main goals in this era are to       continue maintaining a high quality  well documented collection of canonical   tools for data processing and machine learning within the current scope    i e  rectangular data largely invariant to column and row order    predicting targets with simple structure    improve the ease for users to develop and publish external components   improve interoperability with modern data science tools  e g  Pandas  Dask    and infrastructures  e g  distributed processing   Many of the more fine grained goals can be found under the  API tag  https   github com scikit learn scikit learn issues q is 3Aissue is 3Aopen sort 3Aupdated desc label 3AAPI    on the issue tracker   Architectural   general goals                               The list is numbered not as an indication of the order of priority  but to make referring to specific points easier  Please add new entries only at the bottom  Note that the crossed out entries are already done  and we try to keep the document up to date as we work on these issues       Improved handling of Pandas DataFrames       document current handling     Improved handling of categorical features       Tree based models should be able to handle both continuous and categorical      features  issue  29437        Handling mixtures of categorical and continuous variables     Improved handling of missing data       Making sure meta estimators are lenient towards missing data by implementing      a common test       An amputation sample generator to make parts of a dataset go missing       issue  6284      More didactic documentation       More and more options have been added to scikit learn  As a result  the      documentation is crowded which makes it hard for beginners to get the big      picture  Some work could be done in prioritizing the information      Passing around information that is not  X  y   Feature properties       Per feature handling  e g   is this a nominal   ordinal   English language      text    should also not need to be provided to estimator constructors       ideally  but should be available as metadata alongside X   issue  8480      Passing around information that is not  X  y   Target information       We have problems getting the full set of classes to all components when      the data is split sampled   issue  6231   issue  8100       We have no way to handle a mixture of categorical and continuous targets      Make it easier for external users to write Scikit learn compatible    components       More self sufficient running of scikit learn contrib or a similar resource     Support resampling and sample reduction       Allow subsampling of majority classes  in a pipeline    issue  3855      Better interfaces for interactive development       Improve the HTML visualisations of estimators via the  estimator html repr        Include more plotting tools  not just as examples      Improved tools for model diagnostics and basic inference       work on a unified interface for  feature importance       better ways to handle validation sets when fitting     Better tools for selecting hyperparameters with transductive estimators       Grid search and cross validation are not applicable to most clustering      tasks  Stability based selection is more relevant      Better support for manual and automatic pipeline building       Easier way to construct complex pipelines and valid search spaces       issue  7608   issue  5082   issue  8243       provide search ranges for common estimators        cf   searchgrid  https   searchgrid readthedocs io en latest         Improved tracking of fitting       Verbose is not very friendly and should use a standard logging library       issue  6929    issue  78       Callbacks or a similar system would facilitate logging and early stopping     Distributed parallelism       Accept data which complies with     array function         A way forward for more out of core       Dask enables easy out of core computation  While the Dask model probably      cannot be adaptable to all machine learning algorithms  most machine      learning is on smaller data than ETL  hence we can maybe adapt to very      large scale while supporting only a fraction of the patterns      Backwards compatible de serialization of some estimators       Currently serialization  with pickle  breaks across versions  While we may      not be able to get around other limitations of pickle re security etc  it      would be great to offer cross version safety from version 1 0  Note  Gael      and Olivier think that this can cause heavy maintenance burden and we      should manage the trade offs  A possible alternative is presented in the      following point      Documentation and tooling for model lifecycle management       Document good practices for model deployments and lifecycle  before      deploying a model  snapshot the code versions  numpy  scipy  scikit learn       custom code repo   the training script and an alias on how to retrieve      historical training data   snapshot a copy of a small validation set        snapshot of the predictions  predicted probabilities for classifiers       on that validation set       Document and tools to make it easy to manage upgrade of scikit learn      versions          Try to load the old pickle  if it works  use the validation set        prediction snapshot to detect that the serialized model still behave        the same         If joblib load   pickle load not work  use the versioned control        training script   historical training set to retrain the model and use        the validation set prediction snapshot to assert that it is possible to        recover the previous predictive performance  if this is not the case        there is probably a bug in scikit learn that needs to be reported      Everything in scikit learn should probably conform to our API contract     We are still in the process of making decisions on some of these related    issues         Pipeline  pipeline Pipeline   and  FeatureUnion  modify their input      parameters in fit  Fixing this requires making sure we have a good      grasp of their use cases to make sure all current functionality is      maintained   issue  8157   issue  7382       Optional  Improve scikit learn common tests suite to make sure that  at    least for frequently used  models have stable predictions across versions     to be discussed         Extend documentation to mention how to deploy models in Python free      environments for instance  ONNX  https   github com onnx sklearn onnx          and use the above best practices to assess predictive consistency between      scikit learn and ONNX prediction functions on validation set       Document good practices to detect temporal distribution drift for deployed      model and good practices for re training on fresh data without causing      catastrophic predictive performance regressions


scikit-learn There are several channels to connect with scikit learn developers for assistance feedback or contributions Note Communications on all channels should respect our announcementsandnotification Support

Answer:
=======
Support
=======

There are several channels to connect with scikit-learn developers for assistance, feedback, or contributions.

**Note**: Communications on all channels should respect our `Code of Conduct <https://github.com/scikit-learn/scikit-learn/blob/main/CODE_OF_CONDUCT.md>`_.

.. _announcements_and_notification:

Mailing Lists
=============

- **Main Mailing List**: Join the primary discussion
  platform for scikit-learn at `scikit-learn Mailing List
  <https://mail.python.org/mailman/listinfo/scikitlearn>`_.

- **Commit Updates**: Stay informed about repository
  updates and test failures on the `scikit-learn-commits list
  <https://lists.sourceforge.net/lists/listinfo/scikit-learn-commits>`_.

.. _user_questions:

User Questions
==============

If you have questions, this is our general workflow.

- **Stack Overflow**: Some scikit-learn developers support users using the
  `[scikit-learn] <https://stackoverflow.com/questions/tagged/scikit-learn>`_
  tag.

- **General Machine Learning Queries**: For broader machine learning
  discussions, visit `Stack Exchange <https://stats.stackexchange.com/>`_.

When posting questions:

- Please use a descriptive question in the title field (e.g. no "Please
  help with scikit-learn!" as this is not a question)

- Provide detailed context, expected results, and actual observations.

- Include code and data snippets (preferably minimalistic scripts,
  up to ~20 lines).

- Describe your data and preprocessing steps, including sample size,
  feature types (categorical or numerical), and the target for supervised
  learning tasks (classification type or regression).

**Note**: Avoid asking user questions on the bug tracker to keep
the focus on development.

- `GitHub Discussions <https://github.com/scikit-learn/scikit-learn/discussions>`_
  Usage questions such as methodological

- `Stack Overflow <https://stackoverflow.com/questions/tagged/scikit-learn>`_
  Programming/user questions with `[scikit-learn]` tag

- `GitHub Bug Tracker <https://github.com/scikit-learn/scikit-learn/issues>`_
  Bug reports - Please do not ask usage questions on the issue tracker.

- `Discord Server <https://discord.gg/h9qyrK8Jc8>`_
  Current pull requests - Post any specific PR-related questions on your PR,
  and you can share a link to your PR on this server.

.. _bug_tracker:

Bug Tracker
===========

Encountered a bug? Report it on our `issue tracker
<https://github.com/scikit-learn/scikit-learn/issues>`_

Include in your report:

- Steps or scripts to reproduce the bug.

- Expected and observed outcomes.

- Python or gdb tracebacks, if applicable.

- The ideal bug report contains a :ref:`short reproducible code snippet
  <minimal_reproducer>`, this way anyone can try to reproduce the bug easily.

- If your snippet is longer than around 50 lines, please link to a
  `gist <https://gist.github.com>`_ or a github repo.

**Tip**: Gists are Git repositories; you can push data files to them using Git.

.. _social_media:

Social Media
============

scikit-learn has presence on various social media platforms to share
updates with the community. The platforms are not monitored for user
questions.

.. _gitter:

Gitter
======

**Note**: The scikit-learn Gitter room is no longer an active community.
For live discussions and support, please refer to the other channels
mentioned in this document.

.. _documentation_resources:

Documentation Resources
=======================

This documentation is for |release|. Find documentation for other versions
`here <https://scikit-learn.org/dev/versions.html>`__.

Older versions' printable PDF documentation is available `here
<https://sourceforge.net/projects/scikit-learn/files/documentation/>`_.
Building the PDF documentation is no longer supported in the website,
but you can still generate it locally by following the
:ref:`building documentation instructions <building_documentation>`.

Support          There are several channels to connect with scikit learn developers for assistance  feedback  or contributions     Note    Communications on all channels should respect our  Code of Conduct  https   github com scikit learn scikit learn blob main CODE OF CONDUCT md           announcements and notification   Mailing Lists                    Main Mailing List    Join the primary discussion    platform for scikit learn at  scikit learn Mailing List           https   mail python org mailman listinfo scikitlearn          Commit Updates    Stay informed about repository    updates and test failures on the  scikit learn commits list     https   lists sourceforge net lists listinfo scikit learn commits          user questions   User Questions                 If you have questions  this is our general workflow       Stack Overflow    Some scikit learn developers support users using the      scikit learn   https   stackoverflow com questions tagged scikit learn       tag       General Machine Learning Queries    For broader machine learning    discussions  visit  Stack Exchange  https   stats stackexchange com       When posting questions     Please use a descriptive question in the title field  e g  no  Please    help with scikit learn   as this is not a question      Provide detailed context  expected results  and actual observations     Include code and data snippets  preferably minimalistic scripts     up to  20 lines      Describe your data and preprocessing steps  including sample size     feature types  categorical or numerical   and the target for supervised    learning tasks  classification type or regression      Note    Avoid asking user questions on the bug tracker to keep  the focus on development      GitHub Discussions  https   github com scikit learn scikit learn discussions      Usage questions such as methodological     Stack Overflow  https   stackoverflow com questions tagged scikit learn      Programming user questions with   scikit learn   tag     GitHub Bug Tracker  https   github com scikit learn scikit learn issues      Bug reports   Please do not ask usage questions on the issue tracker      Discord Server  https   discord gg h9qyrK8Jc8      Current pull requests   Post any specific PR related questions on your PR     and you can share a link to your PR on this server       bug tracker   Bug Tracker              Encountered a bug  Report it on our  issue tracker  https   github com scikit learn scikit learn issues     Include in your report     Steps or scripts to reproduce the bug     Expected and observed outcomes     Python or gdb tracebacks  if applicable     The ideal bug report contains a  ref  short reproducible code snippet    minimal reproducer    this way anyone can try to reproduce the bug easily     If your snippet is longer than around 50 lines  please link to a     gist  https   gist github com    or a github repo     Tip    Gists are Git repositories  you can push data files to them using Git       social media   Social Media               scikit learn has presence on various social media platforms to share updates with the community  The platforms are not monitored for user questions       gitter   Gitter           Note    The scikit learn Gitter room is no longer an active community   For live discussions and support  please refer to the other channels  mentioned in this document       documentation resources   Documentation Resources                          This documentation is for  release   Find documentation for other versions   here  https   scikit learn org dev versions html       Older versions  printable PDF documentation is available  here  https   sourceforge net projects scikit learn files documentation      Building the PDF documentation is no longer supported in the website  but you can still generate it locally by following the  ref  building documentation instructions  building documentation