redis title Redis security Security model and features in Redis aliases docs manual security Security weight 1 topics security docs manual security md

title   Redis security  linkTitle   Security  weight  1 description  Security model and features in Redis aliases         topics security       docs manual security       docs manual security md        This document provides an introduction to the topic of security from the point of view of Redis  It covers the access control provided by Redis  code security concerns  attacks that can be triggered from the outside by selecting malicious inputs  and other similar topics   You can learn more about access control  data protection and encryption  secure Redis architectures  and secure deployment techniques by taking the  Redis University security course  https   university redis com courses ru330     For security related contacts  open an issue on GitHub  or when you feel it is really important to preserve the security of the communication  use the GPG key at the end of this document      Security model  Redis is designed to be accessed by trusted clients inside trusted environments  This means that usually it is not a good idea to expose the Redis instance directly to the internet or  in general  to an environment where untrusted clients can directly access the Redis TCP port or UNIX socket   For instance  in the common context of a web application implemented using Redis as a database  cache  or messaging system  the clients inside the front end  web side  of the application will query Redis to generate pages or to perform operations requested or triggered by the web application user   In this case  the web application mediates access between Redis and untrusted clients  the user browsers accessing the web application    In general  untrusted access to Redis should always be mediated by a layer implementing ACLs  validating user input  and deciding what operations to perform against the Redis instance      Network security  Access to the Redis port should be denied to everybody but trusted clients in the network  so the servers running Redis should be directly accessible only by the computers implementing the application using Redis   In the common case of a single computer directly exposed to the internet  such as a virtualized Linux instance  Linode  EC2        the Redis port should be firewalled to prevent access from the outside  Clients will still be able to access Redis using the loopback interface   Note that it is possible to bind Redis to a single interface by adding a line like the following to the   redis conf   file       bind 127 0 0 1  Failing to protect the Redis port from the outside can have a big security impact because of the nature of Redis  For instance  a single  FLUSHALL  command can be used by an external attacker to delete the whole data set      Protected mode  Unfortunately  many users fail to protect Redis instances from being accessed from external networks  Many instances are simply left exposed on the internet with public IPs  Since version 3 2 0  Redis enters a special mode called   protected mode   when it is executed with the default configuration  binding all the interfaces  and without any password in order to access it  In this mode  Redis only replies to queries from the loopback interfaces  and replies to clients connecting from other addresses with an error that explains the problem and how to configure Redis properly   We expect protected mode to seriously decrease the security issues caused by unprotected Redis instances executed without proper administration  However  the system administrator can still ignore the error given by Redis and disable protected mode or manually bind all the interfaces      Authentication  Redis provides two ways to authenticate clients  The recommended authentication method  introduced in Redis 6  is via Access Control Lists  allowing named users to be created and assigned fine grained permissions  Read more about Access Control Lists  here   docs management security acl     The legacy authentication method is enabled by editing the   redis conf   file  and providing a database password using the  requirepass  setting  This password is then used by all clients   When the  requirepass  setting is enabled  Redis will refuse any query by unauthenticated clients  A client can authenticate itself by sending the   AUTH   command followed by the password   The password is set by the system administrator in clear text inside the redis conf file  It should be long enough to prevent brute force attacks for two reasons     Redis is very fast at serving queries  Many passwords per second can be tested by an external client    The Redis password is stored in the   redis conf   file and inside the client configuration  Since the system administrator does not need to remember it  the password can be very long   The goal of the authentication layer is to optionally provide a layer of redundancy  If firewalling or any other system implemented to protect Redis from external attackers fail  an external client will still not be able to access the Redis instance without knowledge of the authentication password   Since the  AUTH  command  like every other Redis command  is sent unencrypted  it does not protect against an attacker that has enough access to the network to perform eavesdropping      TLS support  Redis has optional support for TLS on all communication channels  including client connections  replication links  and the Redis Cluster bus protocol      Disallowing specific commands  It is possible to disallow commands in Redis or to rename them as an unguessable name  so that normal clients are limited to a specified set of commands   For instance  a virtualized server provider may offer a managed Redis instance service  In this context  normal users should probably not be able to call the Redis   CONFIG   command to alter the configuration of the instance  but the systems that provide and remove instances should be able to do so   In this case  it is possible to either rename or completely shadow commands from the command table  This feature is available as a statement that can be used inside the redis conf configuration file  For example       rename command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52  In the above example  the   CONFIG   command was renamed into an unguessable name   It is also possible to completely disallow it  or any other command  by renaming it to the empty string  like in the following example       rename command CONFIG        Attacks triggered by malicious inputs from external clients  There is a class of attacks that an attacker can trigger from the outside even without external access to the instance  For example  an attacker might insert data into Redis that triggers pathological  worst case  algorithm complexity on data structures implemented inside Redis internals   An attacker could supply  via a web form  a set of strings that are known to hash to the same bucket in a hash table in order to turn the O 1  expected time  the average time  to the O N  worst case  This can consume more CPU than expected and ultimately cause a Denial of Service   To prevent this specific attack  Redis uses a per execution  pseudo random seed to the hash function   Redis implements the SORT command using the qsort algorithm  Currently  the algorithm is not randomized  so it is possible to trigger a quadratic worst case behavior by carefully selecting the right set of inputs      String escaping and NoSQL injection  The Redis protocol has no concept of string escaping  so injection is impossible under normal circumstances using a normal client library  The protocol uses prefixed length strings and is completely binary safe   Since Lua scripts executed by the  EVAL  and  EVALSHA  commands follow the same rules  those commands are also safe   While it would be a strange use case  the application should avoid composing the body of the Lua script from strings obtained from untrusted sources      Code security  In a classical Redis setup  clients are allowed full access to the command set  but accessing the instance should never result in the ability to control the system where Redis is running   Internally  Redis uses all the well known practices for writing secure code to prevent buffer overflows  format bugs  and other memory corruption issues  However  the ability to control the server configuration using the   CONFIG   command allows the client to change the working directory of the program and the name of the dump file  This allows clients to write RDB Redis files to random paths  This is  a security issue  http   antirez com news 96  that may lead to the ability to compromise the system and or run untrusted code as the same user as Redis is running   Redis does not require root privileges to run  It is recommended to run it as an unprivileged  redis  user that is only used for this purpose      GPG key           BEGIN PGP PUBLIC KEY BLOCK       mQINBF9FWioBEADfBiOE iKpj2EF cJ KzFX jSBKa8SKrE 9RE0faVF6OYnqstL S5ox o yT45FdfFiRNDflKenjFbOmCbAdIys9Ta0iq6I9hs4sKfkNfNVlKZWtSVG W4lI6zO2Zyc2wLZonI Q32dDiXWNcCEsmajFcddukPevj9vKMTJZtF79P2SylEPq mUuhMy jOt7q1ibJCj5srtaureBH9662t4IJMFjsEe hiZ5v071UiQA6Tp7rxLqZ O6ZRzuamFP3xfy2Lz5NQ7QwnBH1ROabhJPoBOKCATCbfgFcM1Rj 9AOGfoDCOJKH 7yiEezMqr9VbDrEmYSmCO4KheqwC0T06lOLIQC4nnwKopNO PN21mirCLHvfo01O H NUG1LZifOwAURbiFNF8Z3 L0csdhD8JnO 1nphjDHr0Xn9Vff2Vej030pRI 9C SJ2s5fZUq8jK4n06sKCbqA4pekpbKyhRy3iuITKv7Nxesl4T uhkc9ccpAvbuD1E NczN1IH05jiMUMM3lC1A9TSvxSqflqI46TZU3qWLa9yg45kDC8Ryr39TY37LscQk 9x3WwLLkuHeUurnwAk46fSj7 FCKTGTdPVw8v7XbvNOTDf8vJ3o2PxX1uh2P2BHs 9L E1P96oMkiEy1ug7gu8V mKu5PAuD3QFzU3XCB93DpDakgtznRRXCkAQARAQAB tBtSZWRpcyBMYWJzIDxyZWRpc0ByZWRpcy5pbz6JAk4EEwEKADgWIQR5sNCo1OBf WO913l22qvOUq0evbgUCX0VaKgIbAwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAK CRC2qvOUq0evbpZaD 4rN7xesDcAG4ec895Fqzk3w74W1 K9lzRKZDwRsAqI sAz ZXvQMtWSxLfF2BITxLnHJXK5P 2Y6XlNgrn1GYwC1MsARyM9e1AzwDJHcXFkHU82 2aALIMXGtiZs ejFh9ZSs5cgRlxBSqot uxXm9AvKEByhmIeHPZse Rc6e3qa57v OhCkVZB4ETx5iZrgA gdmS8N7MXG0cEu5gJLacG57MHi 2WMOCU9Xfj6 Pqhw3qc E6lBinKcA LdgUJ1onK0JCnOG1YVHjuFtaisfPXvEmUBGaSGE6lM4J7lass OWps Dd oHCGI VOGNx6AiBDZG8mZacu0 7goRnOTdljJ93rKkj31I 6 j4xzkAC0IXW8 LAP9Mmo9TGx0L5CaljykhW6z RK3qd7dAYE i7e8J9PuQaGG5pjFzuW4vY45j0V  9JUMKDaGbU5choGqsCpAVtAMFfIBj3UQ5LCt5zKyescKCUb9uifOLeeQ1vay3R9o eRSD52YpRBpor0AyYxcLur pkHB0sSvXEfRZENQTohpY71rHSaFd3q1Hkk7lZl95 m24NRlrJnjFmeSPKP22vqUYIwoGNUF D38UzvqHD8ltTPgkZc Y RRbVNqkQYiwW GH DigNB8r2sdkt 1EUu YkYosxtzxpxxpYGKXYXx0uf EZmRqRt OSHKnf2GLkC DQRfRVoqARAApffsrDNo4JWjX3r6wHJJ8IpwnGEJ2IzGkg8f1Ofk2uKrjkII oIx sXC3EeauC1Plhs m9GP SPY0LXmZ0OzGD S1yMpmBeBuXJ0gONDo xCg1pKGshPs 75XzpbggSOtEYR5S8Z46yCu7TGJRXBMGBhDgCfPVFBBNsnG5B0EeHXM4trqqlN6d PAcwtLnKPz Z lloKR6bFXvYGuN5vjRXjcVYZLLCEwdV9iY5 Opqk9sCluasb3t  c2gcsLWWFnNz2desvb Y4ADJzxY Um848DSR8IcdoArSsqmcCTiYvYC UU7XPVNk Jrx HwgTVYiLGbtMB3u3fUpHW8SabdHc4xG3sx0LeIvl JwHgx7yVhNYJEyOQfnE mfS97x6surXgTVLbWVjXKIJhoWnWbLP4NkBc27H4qo8wM IWH4SSXYNzFLlCDPnw vQZSel21qxdqAWaSxkKcymfMS4nVDhVj0jhlcTY3aZcHMjqoUB07p5 laJr9CCGv 0Y0j0qT2aUO22A3kbv6H9c1Yjv8EI7eNz07aoH1oYU6ShsiaLfIqPfGYb7LwOFWi PSl0dCY7WJg2H6UHsV y2DwRr 3oH0a9hv cvcMneMi3tpIkRwYFBPXEsIcoD9xr RI5dp8BBdO Nt puoQq9oyialWnQK5 AY7ErW1yxjgie4PQ XtN 85UAEQEAAYkC NgQYAQoAIBYhBHmw0KjU4F9Y73XeXbaq85SrR69uBQJfRVoqAhsMAAoJELaq85Sr R69uoV0QAIvlxAHYTjvH1lt5KbpVGs5gwIAnCMPxmaOXcaZ8V0Z1GEU  IztwV N MYCBv1tYa7OppNs1pn75DhzoNAi XQOVvU0OZgVJutthZe0fNDFGG9B4i cxRscI Ld8TPQQNiZPBZ4ubcxbZyBinE9HsYUM49otHjsyFZ0GqTpyne zBf1GAQoekxlKo tWSkkmW0x4qW6eiAmyo5lPS1bBjvaSc67i 6Bv5QkZa0UIkRqAzKN4zVvc2FyILz  7wVLCzWcXrJt8dOeS6Y Fjbhb6m7dtapUSETAKu6wJvSd9ndDUjFHD33NQIZ nL WaPbn01 e PHtUDmyZ2W2KbcdlIT9nb2uHrruqdCN04sXkID8E2m2gYMA TjhC0Q JBJ9WPmdBeKH91R6wWDq6 HwOpgc 9na BHZXMG qyEcvNHB5RJdiu2r1Haf6gHi Fd6rJ6VzaVwnmKmUSKA2wHUuUJ6oxVJ1nFb7Aaschq8F79TAfee0iaGe9cP xUHL zBDKwZ9PtyGfdBp1qNOb94sfEasWPftT26rLgKPFcroCSR2QCK5qHsMNCZL u71w NnTtq9YZDRaQ2JAc6VDZCcgu dLiFxVIi1PFcJQ31rVe16 AQ9zsafiNsxkPdZcY U9XKndQE028dGZv1E3S5BwpnikrUkWdxcYrVZ4fiNIy5I3My2yCe  J9BD      END PGP PUBLIC KEY BLOCK


redis title Redis CPU profiling topics performance on cpu Performance engineering guide for on CPU profiling and tracing CPU profiling aliases docs reference optimization cpu profiling weight 1

Answer:
---
title: "Redis CPU profiling"
linkTitle: "CPU profiling"
weight: 1
description: >
    Performance engineering guide for on-CPU profiling and tracing
aliases: [
    /topics/performance-on-cpu,
    /docs/reference/optimization/cpu-profiling
]
---

## Filling the performance checklist

Redis is developed with a great emphasis on performance. We do our best with
every release to make sure you'll experience a very stable and fast product.

Nevertheless, if you're finding room to improve the efficiency of Redis or
are pursuing a performance regression investigation you will need a concise
methodical way of monitoring and analyzing Redis performance.

To do so you can rely on different methodologies (some more suited than other
depending on the class of issues/analysis we intend to make). A curated list
of methodologies and their steps are enumerated by Brendan Greg at the
[following link](http://www.brendangregg.com/methodology.html).

We recommend the Utilization Saturation and Errors (USE) Method for answering
the question of what is your bottleneck. Check the following mapping between
system resource, metric, and tools for a practical deep dive:
[USE method](http://www.brendangregg.com/USEmethod/use-rosetta.html).

### Ensuring the CPU is your bottleneck

This guide assumes you've followed one of the above methodologies to perform a
complete check of system health, and identified the bottleneck being the CPU.
**If you have identified that most of the time is spent blocked on I/O, locks,
timers, paging/swapping, etc., this guide is not for you**.

### Build Prerequisites

For a proper On-CPU analysis, Redis (and any dynamically loaded library like
Redis Modules) requires stack traces to be available to tracers, which you may
need to fix first.

By default, Redis is compiled with the `-O2` switch (which we intent to keep
during profiling). This means that compiler optimizations are enabled. Many
compilers omit the frame pointer as a runtime optimization (saving a register),
thus breaking frame pointer-based stack walking. This makes the Redis
executable faster, but at the same time it makes Redis (like any other program)
harder to trace, potentially wrongfully pinpointing on-CPU time to the last
available frame pointer of a call stack that can get a lot deeper (but
impossible to trace).

It's important that you ensure that:
- debug information is present: compile option `-g`
- frame pointer register is present: `-fno-omit-frame-pointer`
- we still run with optimizations to get an accurate representation of production run times, meaning we will keep: `-O2`

You can do it as follows within redis main repo:

    $ make REDIS_CFLAGS="-g -fno-omit-frame-pointer"

## A set of instruments to identify performance regressions and/or potential **on-CPU performance** improvements

This document focuses specifically on **on-CPU** resource bottlenecks analysis,
meaning we're interested in understanding where threads are spending CPU cycles
while running on-CPU and, as importantly, whether those cycles are effectively
being used for computation or stalled waiting (not blocked!) for memory I/O,
and cache misses, etc.

For that we will rely on toolkits (perf, bcc tools), and hardware specific PMCs
(Performance Monitoring Counters), to proceed with:

- Hotspot analysis (perf or bcc tools): to profile code execution and determine which functions are consuming the most time and thus are targets for optimization. We'll present two options to collect, report, and visualize hotspots either with perf or bcc/BPF tracing tools.

- Call counts analysis: to count events including function calls, enabling us to correlate several calls/components at once, relying on bcc/BPF tracing tools.

- Hardware event sampling: crucial for understanding CPU behavior, including memory I/O, stall cycles, and cache misses.

### Tool prerequisites

The following steps rely on Linux perf_events (aka ["perf"](https://man7.org/linux/man-pages/man1/perf.1.html)), [bcc/BPF tracing tools](https://github.com/iovisor/bcc), and Brendan Greg’s [FlameGraph repo](https://github.com/brendangregg/FlameGraph).

We assume beforehand you have:

- Installed the perf tool on your system. Most Linux distributions will likely package this as a package related to the kernel. More information about the perf tool can be found at perf [wiki](https://perf.wiki.kernel.org/).
- Followed the install [bcc/BPF](https://github.com/iovisor/bcc/blob/master/INSTALL.md#installing-bcc) instructions to install bcc toolkit on your machine.
- Cloned Brendan Greg’s [FlameGraph repo](https://github.com/brendangregg/FlameGraph) and made accessible the `difffolded.pl` and `flamegraph.pl` files, to generated the collapsed stack traces and Flame Graphs.

## Hotspot analysis with perf or eBPF (stack traces sampling)

Profiling CPU usage by sampling stack traces at a timed interval is a fast and
easy way to identify performance-critical code sections (hotspots).

### Sampling stack traces using perf

To profile both user- and kernel-level stacks of redis-server for a specific
length of time, for example 60 seconds, at a sampling frequency of 999 samples
per second:

    $ perf record -g --pid $(pgrep redis-server) -F 999 -- sleep 60

#### Displaying the recorded profile information using perf report

By default perf record will generate a perf.data file in the current working
directory.

You can then report with a call-graph output (call chain, stack backtrace),
with a minimum call graph inclusion threshold of 0.5%, with:

    $ perf report -g "graph,0.5,caller"

See the [perf report](https://man7.org/linux/man-pages/man1/perf-report.1.html)
documentation for advanced filtering, sorting and aggregation capabilities.

#### Visualizing the recorded profile information using Flame Graphs

[Flame graphs](http://www.brendangregg.com/flamegraphs.html) allow for a quick
and accurate visualization of frequent code-paths. They can be generated using
Brendan Greg's open source programs on [github](https://github.com/brendangregg/FlameGraph),
which create interactive SVGs from folded stack files.

Specifically, for perf we need to convert the generated perf.data into the
captured stacks, and fold each of them into single lines. You can then render
the on-CPU flame graph with:

    $ perf script > redis.perf.stacks
    $ stackcollapse-perf.pl redis.perf.stacks > redis.folded.stacks
    $ flamegraph.pl redis.folded.stacks > redis.svg

By default, perf script will generate a perf.data file in the current working
directory. See the [perf script](https://linux.die.net/man/1/perf-script)
documentation for advanced usage.

See [FlameGraph usage options](https://github.com/brendangregg/FlameGraph#options)
for more advanced stack trace visualizations (like the differential one).

#### Archiving and sharing recorded profile information

So that analysis of the perf.data contents can be possible on a machine other
than the one on which collection happened, you need to export along with the
perf.data file all object files with build-ids found in the record data file.
This can be easily done with the help of
[perf-archive.sh](https://github.com/torvalds/linux/blob/master/tools/perf/perf-archive.sh)
script:

    $ perf-archive.sh perf.data

Now please run:

    $ tar xvf perf.data.tar.bz2 -C ~/.debug

on the machine where you need to run `perf report`.

### Sampling stack traces using bcc/BPF's profile

Similarly to perf, as of Linux kernel 4.9, BPF-optimized profiling is now fully
available with the promise of lower overhead on CPU (as stack traces are
frequency counted in kernel context) and disk I/O resources during profiling.

Apart from that, and relying solely on bcc/BPF's profile tool, we have also
removed the perf.data and intermediate steps if stack traces analysis is our
main goal. You can use bcc's profile tool to output folded format directly, for
flame graph generation:

    $ /usr/share/bcc/tools/profile -F 999 -f --pid $(pgrep redis-server) --duration 60 > redis.folded.stacks

In that manner, we've remove any preprocessing and can render the on-CPU flame
graph with a single command:

    $ flamegraph.pl redis.folded.stacks > redis.svg

### Visualizing the recorded profile information using Flame Graphs

## Call counts analysis with bcc/BPF

A function may consume significant CPU cycles either because its code is slow
or because it's frequently called. To answer at what rate functions are being
called, you can rely upon call counts analysis using BCC's `funccount` tool:

    $ /usr/share/bcc/tools/funccount 'redis-server:(call*|*Read*|*Write*)' --pid $(pgrep redis-server) --duration 60
    Tracing 64 functions for "redis-server:(call*|*Read*|*Write*)"... Hit Ctrl-C to end.

    FUNC                                    COUNT
    call                                      334
    handleClientsWithPendingWrites            388
    clientInstallWriteHandler                 388
    postponeClientRead                        514
    handleClientsWithPendingReadsUsingThreads      735
    handleClientsWithPendingWritesUsingThreads      735
    prepareClientToWrite                     1442
    Detaching...

The above output shows that, while tracing, the Redis's call() function was
called 334 times, handleClientsWithPendingWrites() 388 times, etc.

## Hardware event counting with Performance Monitoring Counters (PMCs)

Many modern processors contain a performance monitoring unit (PMU) exposing
Performance Monitoring Counters (PMCs). PMCs are crucial for understanding CPU
behavior, including memory I/O, stall cycles, and cache misses, and provide
low-level CPU performance statistics that aren't available anywhere else.

The design and functionality of a PMU is CPU-specific and you should assess
your CPU supported counters and features by using `perf list`.

To calculate the number of instructions per cycle, the number of micro ops
executed, the number of cycles during which no micro ops were dispatched, the
number stalled cycles on memory, including a per memory type stalls, for the
duration of 60s, specifically for redis process:

    $ perf stat -e "cpu-clock,cpu-cycles,instructions,uops_executed.core,uops_executed.stall_cycles,cache-references,cache-misses,cycle_activity.stalls_total,cycle_activity.stalls_mem_any,cycle_activity.stalls_l3_miss,cycle_activity.stalls_l2_miss,cycle_activity.stalls_l1d_miss" --pid $(pgrep redis-server) -- sleep 60

    Performance counter stats for process id '3038':

      60046.411437      cpu-clock (msec)          #    1.001 CPUs utilized
      168991975443      cpu-cycles                #    2.814 GHz                      (36.40%)
      388248178431      instructions              #    2.30  insn per cycle           (45.50%)
      443134227322      uops_executed.core        # 7379.862 M/sec                    (45.51%)
       30317116399      uops_executed.stall_cycles #  504.895 M/sec                    (45.51%)
         670821512      cache-references          #   11.172 M/sec                    (45.52%)
          23727619      cache-misses              #    3.537 % of all cache refs      (45.43%)
       30278479141      cycle_activity.stalls_total #  504.251 M/sec                    (36.33%)
       19981138777      cycle_activity.stalls_mem_any #  332.762 M/sec                    (36.33%)
         725708324      cycle_activity.stalls_l3_miss #   12.086 M/sec                    (36.33%)
        8487905659      cycle_activity.stalls_l2_miss #  141.356 M/sec                    (36.32%)
       10011909368      cycle_activity.stalls_l1d_miss #  166.736 M/sec                    (36.31%)

      60.002765665 seconds time elapsed

It's important to know that there are two very different ways in which PMCs can
be used (counting and sampling), and we've focused solely on PMCs counting for
the sake of this analysis. Brendan Greg clearly explains it on the following
[link](http://www.brendangregg.com/blog/2017-05-04/the-pmcs-of-ec2.html).

title   Redis CPU profiling  linkTitle   CPU profiling  weight  1 description        Performance engineering guide for on CPU profiling and tracing aliases         topics performance on cpu       docs reference optimization cpu profiling           Filling the performance checklist  Redis is developed with a great emphasis on performance  We do our best with every release to make sure you ll experience a very stable and fast product    Nevertheless  if you re finding room to improve the efficiency of Redis or are pursuing a performance regression investigation you will need a concise methodical way of monitoring and analyzing Redis performance    To do so you can rely on different methodologies  some more suited than other  depending on the class of issues analysis we intend to make   A curated list of methodologies and their steps are enumerated by Brendan Greg at the  following link  http   www brendangregg com methodology html     We recommend the Utilization Saturation and Errors  USE  Method for answering the question of what is your bottleneck  Check the following mapping between system resource  metric  and tools for a practical deep dive   USE method  http   www brendangregg com USEmethod use rosetta html         Ensuring the CPU is your bottleneck  This guide assumes you ve followed one of the above methodologies to perform a  complete check of system health  and identified the bottleneck being the CPU     If you have identified that most of the time is spent blocked on I O  locks  timers  paging swapping  etc   this guide is not for you          Build Prerequisites  For a proper On CPU analysis  Redis  and any dynamically loaded library like Redis Modules  requires stack traces to be available to tracers  which you may need to fix first    By default  Redis is compiled with the   O2  switch  which we intent to keep during profiling   This means that compiler optimizations are enabled  Many compilers omit the frame pointer as a runtime optimization  saving a register   thus breaking frame pointer based stack walking  This makes the Redis executable faster  but at the same time it makes Redis  like any other program  harder to trace  potentially wrongfully pinpointing on CPU time to the last available frame pointer of a call stack that can get a lot deeper  but impossible to trace    It s important that you ensure that    debug information is present  compile option   g    frame pointer register is present    fno omit frame pointer    we still run with optimizations to get an accurate representation of production run times  meaning we will keep    O2   You can do it as follows within redis main repo         make REDIS CFLAGS   g  fno omit frame pointer      A set of instruments to identify performance regressions and or potential   on CPU performance   improvements   This document focuses specifically on   on CPU   resource bottlenecks analysis  meaning we re interested in understanding where threads are spending CPU cycles while running on CPU and  as importantly  whether those cycles are effectively being used for computation or stalled waiting  not blocked   for memory I O  and cache misses  etc   For that we will rely on toolkits  perf  bcc tools   and hardware specific PMCs  Performance Monitoring Counters   to proceed with     Hotspot analysis  perf or bcc tools   to profile code execution and determine which functions are consuming the most time and thus are targets for optimization  We ll present two options to collect  report  and visualize hotspots either with perf or bcc BPF tracing tools     Call counts analysis  to count events including function calls  enabling us to correlate several calls components at once  relying on bcc BPF tracing tools     Hardware event sampling  crucial for understanding CPU behavior  including memory I O  stall cycles  and cache misses       Tool prerequisites  The following steps rely on Linux perf events  aka   perf   https   man7 org linux man pages man1 perf 1 html     bcc BPF tracing tools  https   github com iovisor bcc   and Brendan Greg s  FlameGraph repo  https   github com brendangregg FlameGraph    We assume beforehand you have     Installed the perf tool on your system  Most Linux distributions will likely package this as a package related to the kernel  More information about the perf tool can be found at perf  wiki  https   perf wiki kernel org      Followed the install  bcc BPF  https   github com iovisor bcc blob master INSTALL md installing bcc  instructions to install bcc toolkit on your machine    Cloned Brendan Greg s  FlameGraph repo  https   github com brendangregg FlameGraph  and made accessible the  difffolded pl  and  flamegraph pl  files  to generated the collapsed stack traces and Flame Graphs      Hotspot analysis with perf or eBPF  stack traces sampling   Profiling CPU usage by sampling stack traces at a timed interval is a fast and easy way to identify performance critical code sections  hotspots        Sampling stack traces using perf  To profile both user  and kernel level stacks of redis server for a specific length of time  for example 60 seconds  at a sampling frequency of 999 samples per second         perf record  g   pid   pgrep redis server   F 999    sleep 60       Displaying the recorded profile information using perf report  By default perf record will generate a perf data file in the current working directory    You can then report with a call graph output  call chain  stack backtrace   with a minimum call graph inclusion threshold of 0 5   with         perf report  g  graph 0 5 caller   See the  perf report  https   man7 org linux man pages man1 perf report 1 html  documentation for advanced filtering  sorting and aggregation capabilities        Visualizing the recorded profile information using Flame Graphs   Flame graphs  http   www brendangregg com flamegraphs html  allow for a quick and accurate visualization of frequent code paths  They can be generated using Brendan Greg s open source programs on  github  https   github com brendangregg FlameGraph   which create interactive SVGs from folded stack files   Specifically  for perf we need to convert the generated perf data into the captured stacks  and fold each of them into single lines  You can then render the on CPU flame graph with         perf script   redis perf stacks       stackcollapse perf pl redis perf stacks   redis folded stacks       flamegraph pl redis folded stacks   redis svg  By default  perf script will generate a perf data file in the current working directory  See the  perf script  https   linux die net man 1 perf script  documentation for advanced usage   See  FlameGraph usage options  https   github com brendangregg FlameGraph options  for more advanced stack trace visualizations  like the differential one         Archiving and sharing recorded profile information  So that analysis of the perf data contents can be possible on a machine other than the one on which collection happened  you need to export along with the perf data file all object files with build ids found in the record data file  This can be easily done with the help of   perf archive sh  https   github com torvalds linux blob master tools perf perf archive sh  script         perf archive sh perf data  Now please run         tar xvf perf data tar bz2  C    debug  on the machine where you need to run  perf report        Sampling stack traces using bcc BPF s profile      Similarly to perf  as of Linux kernel 4 9  BPF optimized profiling is now fully available with the promise of lower overhead on CPU  as stack traces are frequency counted in kernel context  and disk I O resources during profiling    Apart from that  and relying solely on bcc BPF s profile tool  we have also removed the perf data and intermediate steps if stack traces analysis is our main goal  You can use bcc s profile tool to output folded format directly  for flame graph generation          usr share bcc tools profile  F 999  f   pid   pgrep redis server    duration 60   redis folded stacks  In that manner  we ve remove any preprocessing and can render the on CPU flame graph with a single command         flamegraph pl redis folded stacks   redis svg      Visualizing the recorded profile information using Flame Graphs     Call counts analysis with bcc BPF  A function may consume significant CPU cycles either because its code is slow or because it s frequently called  To answer at what rate functions are being called  you can rely upon call counts analysis using BCC s  funccount  tool          usr share bcc tools funccount  redis server  call   Read   Write      pid   pgrep redis server    duration 60     Tracing 64 functions for  redis server  call   Read   Write       Hit Ctrl C to end       FUNC                                    COUNT     call                                      334     handleClientsWithPendingWrites            388     clientInstallWriteHandler                 388     postponeClientRead                        514     handleClientsWithPendingReadsUsingThreads      735     handleClientsWithPendingWritesUsingThreads      735     prepareClientToWrite                     1442     Detaching      The above output shows that  while tracing  the Redis s call   function was called 334 times  handleClientsWithPendingWrites   388 times  etc      Hardware event counting with Performance Monitoring Counters  PMCs   Many modern processors contain a performance monitoring unit  PMU  exposing Performance Monitoring Counters  PMCs   PMCs are crucial for understanding CPU behavior  including memory I O  stall cycles  and cache misses  and provide low level CPU performance statistics that aren t available anywhere else   The design and functionality of a PMU is CPU specific and you should assess your CPU supported counters and features by using  perf list     To calculate the number of instructions per cycle  the number of micro ops executed  the number of cycles during which no micro ops were dispatched  the number stalled cycles on memory  including a per memory type stalls  for the duration of 60s  specifically for redis process          perf stat  e  cpu clock cpu cycles instructions uops executed core uops executed stall cycles cache references cache misses cycle activity stalls total cycle activity stalls mem any cycle activity stalls l3 miss cycle activity stalls l2 miss cycle activity stalls l1d miss    pid   pgrep redis server     sleep 60      Performance counter stats for process id  3038          60046 411437      cpu clock  msec                1 001 CPUs utilized                 168991975443      cpu cycles                     2 814 GHz                       36 40         388248178431      instructions                   2 30  insn per cycle            45 50         443134227322      uops executed core          7379 862 M sec                     45 51          30317116399      uops executed stall cycles    504 895 M sec                     45 51            670821512      cache references              11 172 M sec                     45 52             23727619      cache misses                   3 537   of all cache refs       45 43          30278479141      cycle activity stalls total    504 251 M sec                     36 33          19981138777      cycle activity stalls mem any    332 762 M sec                     36 33            725708324      cycle activity stalls l3 miss     12 086 M sec                     36 33           8487905659      cycle activity stalls l2 miss    141 356 M sec                     36 32          10011909368      cycle activity stalls l1d miss    166 736 M sec                     36 31          60 002765665 seconds time elapsed  It s important to know that there are two very different ways in which PMCs can be used  counting and sampling   and we ve focused solely on PMCs counting for the sake of this analysis  Brendan Greg clearly explains it on the following  link  http   www brendangregg com blog 2017 05 04 the pmcs of ec2 html


redis docs reference optimization latency monitor Latency monitoring topics latency monitor aliases Discovering slow server events in Redis weight 1 title Redis latency monitoring

Answer:
---
title: "Redis latency monitoring"
linkTitle: "Latency monitoring"
weight: 1
description: Discovering slow server events in Redis
aliases: [
    /topics/latency-monitor,
    /docs/reference/optimization/latency-monitor
]
---

Redis is often used for demanding use cases, where it
serves a large number of queries per second per instance, but also has strict latency requirements for the average response
time and the worst-case latency.

While Redis is an in-memory system, it deals with the operating system in
different ways, for example, in the context of persisting to disk.
Moreover Redis implements a rich set of commands. Certain commands
are fast and run in constant or logarithmic time. Other commands are slower
O(N) commands that can cause latency spikes.

Finally, Redis is single threaded. This is usually an advantage
from the point of view of the amount of work it can perform per core, and in
the latency figures it is able to provide. However, it poses
a challenge for latency, since the single
thread must be able to perform certain tasks incrementally, for
example key expiration, in a way that does not impact the other clients
that are served.

For all these reasons, Redis 2.8.13 introduced a new feature called
**Latency Monitoring**, that helps the user to check and troubleshoot possible
latency problems. Latency monitoring is composed of the following conceptual
parts:

* Latency hooks that sample different latency-sensitive code paths.
* Time series recording of latency spikes, split by different events.
* Reporting engine to fetch raw data from the time series.
* Analysis engine to provide human-readable reports and hints according to the measurements.

The rest of this document covers the latency monitoring subsystem
details. For more information about the general topic of Redis
and latency, see [Redis latency problems troubleshooting](/topics/latency).

## Events and time series

Different monitored code paths have different names and are called *events*.
For example, `command` is an event that measures latency spikes of possibly slow
command executions, while `fast-command` is the event name for the monitoring
of the O(1) and O(log N) commands. Other events are less generic and monitor
specific operations performed by Redis. For example, the `fork` event
only monitors the time taken by Redis to execute the `fork(2)` system call.

A latency spike is an event that takes more time to run than the configured latency
threshold. There is a separate time series associated with every monitored
event. This is how the time series work:

* Every time a latency spike happens, it is logged in the appropriate time series.
* Every time series is composed of 160 elements.
* Each element is a pair made of a Unix timestamp of the time the latency spike was measured and the number of milliseconds the event took to execute.
* Latency spikes for the same event that occur in the same second are merged by taking the maximum latency. Even if continuous latency spikes are measured for a given event, which could happen with a low threshold, at least 160 seconds of history are available.
* Records the all-time maximum latency for every element.

The framework monitors and logs latency spikes in the execution time of these events:

* `command`: regular commands.
* `fast-command`: O(1) and O(log N) commands.
* `fork`: the `fork(2)` system call.
* `rdb-unlink-temp-file`: the `unlink(2)` system call.
* `aof-fsync-always`: the `fsync(2)` system call when invoked by the `appendfsync allways` policy.
* `aof-write`: writing to the AOF - a catchall event for `write(2)` system calls.
* `aof-write-pending-fsync`: the `write(2)` system call when there is a pending fsync.
* `aof-write-active-child`: the `write(2)` system call when there are active child processes.
* `aof-write-alone`: the `write(2)` system call when no pending fsync and no active child process.
* `aof-fstat`: the `fstat(2)` system call.
* `aof-rename`: the `rename(2)` system call for renaming the temporary file after completing `BGREWRITEAOF`.
* `aof-rewrite-diff-write`: writing the differences accumulated while performing `BGREWRITEAOF`.
* `active-defrag-cycle`: the active defragmentation cycle.
* `expire-cycle`: the expiration cycle.
* `eviction-cycle`: the eviction cycle.
* `eviction-del`: deletes during the eviction cycle.

## How to enable latency monitoring

What is high latency for one use case may not be considered high latency for another. Some applications may require that all queries be served in less than 1 millisecond. For other applications, it may be acceptable for a small amount of clients to experience a 2 second latency on occasion.

The first step to enable the latency monitor is to set a **latency threshold** in milliseconds. Only events that take longer than the specified threshold will be logged as latency spikes. The user should set the threshold according to their needs. For example, if the application requires a maximum acceptable latency of 100 milliseconds, the threshold should be set to log all the events blocking the server for a time equal or greater to 100 milliseconds.

Enable the latency monitor at runtime in a production server
with the following command:

    CONFIG SET latency-monitor-threshold 100

Monitoring is turned off by default (threshold set to 0), even if the actual cost of latency monitoring is near zero. While the memory requirements of latency monitoring are very small, there is no good reason to raise the baseline memory usage of a Redis instance that is working well.

## Report information with the LATENCY command

The user interface to the latency monitoring subsystem is the `LATENCY` command.
Like many other Redis commands, `LATENCY` accepts subcommands that modify its behavior. These subcommands are:

* `LATENCY LATEST` - returns the latest latency samples for all events.
* `LATENCY HISTORY` - returns latency time series for a given event.
* `LATENCY RESET` - resets latency time series data for one or more events.
* `LATENCY GRAPH` - renders an ASCII-art graph of an event's latency samples.
* `LATENCY DOCTOR` - replies with a human-readable latency analysis report.

Refer to each subcommand's documentation page for further information.

title   Redis latency monitoring  linkTitle   Latency monitoring  weight  1 description  Discovering slow server events in Redis aliases         topics latency monitor       docs reference optimization latency monitor        Redis is often used for demanding use cases  where it serves a large number of queries per second per instance  but also has strict latency requirements for the average response time and the worst case latency   While Redis is an in memory system  it deals with the operating system in different ways  for example  in the context of persisting to disk  Moreover Redis implements a rich set of commands  Certain commands are fast and run in constant or logarithmic time  Other commands are slower O N  commands that can cause latency spikes   Finally  Redis is single threaded  This is usually an advantage from the point of view of the amount of work it can perform per core  and in the latency figures it is able to provide  However  it poses a challenge for latency  since the single thread must be able to perform certain tasks incrementally  for example key expiration  in a way that does not impact the other clients that are served   For all these reasons  Redis 2 8 13 introduced a new feature called   Latency Monitoring    that helps the user to check and troubleshoot possible latency problems  Latency monitoring is composed of the following conceptual parts     Latency hooks that sample different latency sensitive code paths    Time series recording of latency spikes  split by different events    Reporting engine to fetch raw data from the time series    Analysis engine to provide human readable reports and hints according to the measurements   The rest of this document covers the latency monitoring subsystem details  For more information about the general topic of Redis and latency  see  Redis latency problems troubleshooting   topics latency       Events and time series  Different monitored code paths have different names and are called  events   For example   command  is an event that measures latency spikes of possibly slow command executions  while  fast command  is the event name for the monitoring of the O 1  and O log N  commands  Other events are less generic and monitor specific operations performed by Redis  For example  the  fork  event only monitors the time taken by Redis to execute the  fork 2   system call   A latency spike is an event that takes more time to run than the configured latency threshold  There is a separate time series associated with every monitored event  This is how the time series work     Every time a latency spike happens  it is logged in the appropriate time series    Every time series is composed of 160 elements    Each element is a pair made of a Unix timestamp of the time the latency spike was measured and the number of milliseconds the event took to execute    Latency spikes for the same event that occur in the same second are merged by taking the maximum latency  Even if continuous latency spikes are measured for a given event  which could happen with a low threshold  at least 160 seconds of history are available    Records the all time maximum latency for every element   The framework monitors and logs latency spikes in the execution time of these events      command   regular commands     fast command   O 1  and O log N  commands     fork   the  fork 2   system call     rdb unlink temp file   the  unlink 2   system call     aof fsync always   the  fsync 2   system call when invoked by the  appendfsync allways  policy     aof write   writing to the AOF   a catchall event for  write 2   system calls     aof write pending fsync   the  write 2   system call when there is a pending fsync     aof write active child   the  write 2   system call when there are active child processes     aof write alone   the  write 2   system call when no pending fsync and no active child process     aof fstat   the  fstat 2   system call     aof rename   the  rename 2   system call for renaming the temporary file after completing  BGREWRITEAOF      aof rewrite diff write   writing the differences accumulated while performing  BGREWRITEAOF      active defrag cycle   the active defragmentation cycle     expire cycle   the expiration cycle     eviction cycle   the eviction cycle     eviction del   deletes during the eviction cycle      How to enable latency monitoring  What is high latency for one use case may not be considered high latency for another  Some applications may require that all queries be served in less than 1 millisecond  For other applications  it may be acceptable for a small amount of clients to experience a 2 second latency on occasion   The first step to enable the latency monitor is to set a   latency threshold   in milliseconds  Only events that take longer than the specified threshold will be logged as latency spikes  The user should set the threshold according to their needs  For example  if the application requires a maximum acceptable latency of 100 milliseconds  the threshold should be set to log all the events blocking the server for a time equal or greater to 100 milliseconds   Enable the latency monitor at runtime in a production server with the following command       CONFIG SET latency monitor threshold 100  Monitoring is turned off by default  threshold set to 0   even if the actual cost of latency monitoring is near zero  While the memory requirements of latency monitoring are very small  there is no good reason to raise the baseline memory usage of a Redis instance that is working well      Report information with the LATENCY command  The user interface to the latency monitoring subsystem is the  LATENCY  command  Like many other Redis commands   LATENCY  accepts subcommands that modify its behavior  These subcommands are      LATENCY LATEST    returns the latest latency samples for all events     LATENCY HISTORY    returns latency time series for a given event     LATENCY RESET    resets latency time series data for one or more events     LATENCY GRAPH    renders an ASCII art graph of an event s latency samples     LATENCY DOCTOR    replies with a human readable latency analysis report   Refer to each subcommand s documentation page for further information