linux Reporting issues See the bottom of this file for additional redistribution information SPDX License Identifier GPL 2 0 OR CC BY 4 0 The short guide aka TL DR

Answer:
.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
.. See the bottom of this file for additional redistribution information.

Reporting issues
++++++++++++++++

The short guide (aka TL;DR)
===========================

Are you facing a regression with vanilla kernels from the same stable or
longterm series? One still supported? Then search the `LKML
<https://lore.kernel.org/lkml/>`_ and the `Linux stable mailing list
<https://lore.kernel.org/stable/>`_ archives for matching reports to join. If
you don't find any, install `the latest release from that series
<https://kernel.org/>`_. If it still shows the issue, report it to the stable
mailing list (stable@vger.kernel.org) and CC the regressions list
(regressions@lists.linux.dev); ideally also CC the maintainer and the mailing
list for the subsystem in question.

In all other cases try your best guess which kernel part might be causing the
issue. Check the :ref:`MAINTAINERS <maintainers>` file for how its developers
expect to be told about problems, which most of the time will be by email with a
mailing list in CC. Check the destination's archives for matching reports;
search the `LKML <https://lore.kernel.org/lkml/>`_ and the web, too. If you
don't find any to join, install `the latest mainline kernel
<https://kernel.org/>`_. If the issue is present there, send a report.

The issue was fixed there, but you would like to see it resolved in a still
supported stable or longterm series as well? Then install its latest release.
If it shows the problem, search for the change that fixed it in mainline and
check if backporting is in the works or was discarded; if it's neither, ask
those who handled the change for it.

**General remarks**: When installing and testing a kernel as outlined above,
ensure it's vanilla (IOW: not patched and not using add-on modules). Also make
sure it's built and running in a healthy environment and not already tainted
before the issue occurs.

If you are facing multiple issues with the Linux kernel at once, report each
separately. While writing your report, include all information relevant to the
issue, like the kernel and the distro used. In case of a regression, CC the
regressions mailing list (regressions@lists.linux.dev) to your report. Also try
to pin-point the culprit with a bisection; if you succeed, include its
commit-id and CC everyone in the sign-off-by chain.

Once the report is out, answer any questions that come up and help where you
can. That includes keeping the ball rolling by occasionally retesting with newer
releases and sending a status update afterwards.

Step-by-step guide how to report issues to the kernel maintainers
=================================================================

The above TL;DR outlines roughly how to report issues to the Linux kernel
developers. It might be all that's needed for people already familiar with
reporting issues to Free/Libre & Open Source Software (FLOSS) projects. For
everyone else there is this section. It is more detailed and uses a
step-by-step approach. It still tries to be brief for readability and leaves
out a lot of details; those are described below the step-by-step guide in a
reference section, which explains each of the steps in more detail.

Note: this section covers a few more aspects than the TL;DR and does things in
a slightly different order. That's in your interest, to make sure you notice
early if an issue that looks like a Linux kernel problem is actually caused by
something else. These steps thus help to ensure the time you invest in this
process won't feel wasted in the end:

 * Are you facing an issue with a Linux kernel a hardware or software vendor
   provided? Then in almost all cases you are better off to stop reading this
   document and reporting the issue to your vendor instead, unless you are
   willing to install the latest Linux version yourself. Be aware the latter
   will often be needed anyway to hunt down and fix issues.

 * Perform a rough search for existing reports with your favorite internet
   search engine; additionally, check the archives of the `Linux Kernel Mailing
   List (LKML) <https://lore.kernel.org/lkml/>`_. If you find matching reports,
   join the discussion instead of sending a new one.

 * See if the issue you are dealing with qualifies as regression, security
   issue, or a really severe problem: those are 'issues of high priority' that
   need special handling in some steps that are about to follow.

 * Make sure it's not the kernel's surroundings that are causing the issue
   you face.

 * Create a fresh backup and put system repair and restore tools at hand.

 * Ensure your system does not enhance its kernels by building additional
   kernel modules on-the-fly, which solutions like DKMS might be doing locally
   without your knowledge.

 * Check if your kernel was 'tainted' when the issue occurred, as the event
   that made the kernel set this flag might be causing the issue you face.

 * Write down coarsely how to reproduce the issue. If you deal with multiple
   issues at once, create separate notes for each of them and make sure they
   work independently on a freshly booted system. That's needed, as each issue
   needs to get reported to the kernel developers separately, unless they are
   strongly entangled.

 * If you are facing a regression within a stable or longterm version line
   (say something broke when updating from 5.10.4 to 5.10.5), scroll down to
   'Dealing with regressions within a stable and longterm kernel line'.

 * Locate the driver or kernel subsystem that seems to be causing the issue.
   Find out how and where its developers expect reports. Note: most of the
   time this won't be bugzilla.kernel.org, as issues typically need to be sent
   by mail to a maintainer and a public mailing list.

 * Search the archives of the bug tracker or mailing list in question
   thoroughly for reports that might match your issue. If you find anything,
   join the discussion instead of sending a new report.

After these preparations you'll now enter the main part:

 * Unless you are already running the latest 'mainline' Linux kernel, better
   go and install it for the reporting process. Testing and reporting with
   the latest 'stable' Linux can be an acceptable alternative in some
   situations; during the merge window that actually might be even the best
   approach, but in that development phase it can be an even better idea to
   suspend your efforts for a few days anyway. Whatever version you choose,
   ideally use a 'vanilla' build. Ignoring these advices will dramatically
   increase the risk your report will be rejected or ignored.

 * Ensure the kernel you just installed does not 'taint' itself when
   running.

 * Reproduce the issue with the kernel you just installed. If it doesn't show
   up there, scroll down to the instructions for issues only happening with
   stable and longterm kernels.

 * Optimize your notes: try to find and write the most straightforward way to
   reproduce your issue. Make sure the end result has all the important
   details, and at the same time is easy to read and understand for others
   that hear about it for the first time. And if you learned something in this
   process, consider searching again for existing reports about the issue.

 * If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider
   decoding the kernel log to find the line of code that triggered the error.

 * If your problem is a regression, try to narrow down when the issue was
   introduced as much as possible.

 * Start to compile the report by writing a detailed description about the
   issue. Always mention a few things: the latest kernel version you installed
   for reproducing, the Linux Distribution used, and your notes on how to
   reproduce the issue. Ideally, make the kernel's build configuration
   (.config) and the output from ``dmesg`` available somewhere on the net and
   link to it. Include or upload all other information that might be relevant,
   like the output/screenshot of an Oops or the output from ``lspci``. Once
   you wrote this main part, insert a normal length paragraph on top of it
   outlining the issue and the impact quickly. On top of this add one sentence
   that briefly describes the problem and gets people to read on. Now give the
   thing a descriptive title or subject that yet again is shorter. Then you're
   ready to send or file the report like the MAINTAINERS file told you, unless
   you are dealing with one of those 'issues of high priority': they need
   special care which is explained in 'Special handling for high priority
   issues' below.

 * Wait for reactions and keep the thing rolling until you can accept the
   outcome in one way or the other. Thus react publicly and in a timely manner
   to any inquiries. Test proposed fixes. Do proactive testing: retest with at
   least every first release candidate (RC) of a new mainline version and
   report your results. Send friendly reminders if things stall. And try to
   help yourself, if you don't get any help or if it's unsatisfying.

Reporting regressions within a stable and longterm kernel line
--------------------------------------------------------------

This subsection is for you, if you followed above process and got sent here at
the point about regression within a stable or longterm kernel version line. You
face one of those if something breaks when updating from 5.10.4 to 5.10.5 (a
switch from 5.9.15 to 5.10.5 does not qualify). The developers want to fix such
regressions as quickly as possible, hence there is a streamlined process to
report them:

 * Check if the kernel developers still maintain the Linux kernel version
   line you care about: go to the  `front page of kernel.org
   <https://kernel.org/>`_ and make sure it mentions
   the latest release of the particular version line without an '[EOL]' tag.

 * Check the archives of the `Linux stable mailing list
   <https://lore.kernel.org/stable/>`_ for existing reports.

 * Install the latest release from the particular version line as a vanilla
   kernel. Ensure this kernel is not tainted and still shows the problem, as
   the issue might have already been fixed there. If you first noticed the
   problem with a vendor kernel, check a vanilla build of the last version
   known to work performs fine as well.

 * Send a short problem report to the Linux stable mailing list
   (stable@vger.kernel.org) and CC the Linux regressions mailing list
   (regressions@lists.linux.dev); if you suspect the cause in a particular
   subsystem, CC its maintainer and its mailing list. Roughly describe the
   issue and ideally explain how to reproduce it. Mention the first version
   that shows the problem and the last version that's working fine. Then
   wait for further instructions.

The reference section below explains each of these steps in more detail.

Reporting issues only occurring in older kernel version lines
-------------------------------------------------------------

This subsection is for you, if you tried the latest mainline kernel as outlined
above, but failed to reproduce your issue there; at the same time you want to
see the issue fixed in a still supported stable or longterm series or vendor
kernels regularly rebased on those. If that the case, follow these steps:

 * Prepare yourself for the possibility that going through the next few steps
   might not get the issue solved in older releases: the fix might be too big
   or risky to get backported there.

 * Perform the first three steps in the section "Dealing with regressions
   within a stable and longterm kernel line" above.

 * Search the Linux kernel version control system for the change that fixed
   the issue in mainline, as its commit message might tell you if the fix is
   scheduled for backporting already. If you don't find anything that way,
   search the appropriate mailing lists for posts that discuss such an issue
   or peer-review possible fixes; then check the discussions if the fix was
   deemed unsuitable for backporting. If backporting was not considered at
   all, join the newest discussion, asking if it's in the cards.

 * One of the former steps should lead to a solution. If that doesn't work
   out, ask the maintainers for the subsystem that seems to be causing the
   issue for advice; CC the mailing list for the particular subsystem as well
   as the stable mailing list.

The reference section below explains each of these steps in more detail.

Reference section: Reporting issues to the kernel maintainers
=============================================================

The detailed guides above outline all the major steps in brief fashion, which
should be enough for most people. But sometimes there are situations where even
experienced users might wonder how to actually do one of those steps. That's
what this section is for, as it will provide a lot more details on each of the
above steps. Consider this as reference documentation: it's possible to read it
from top to bottom. But it's mainly meant to skim over and a place to look up
details how to actually perform those steps.

A few words of general advice before digging into the details:

 * The Linux kernel developers are well aware this process is complicated and
   demands more than other FLOSS projects. We'd love to make it simpler. But
   that would require work in various places as well as some infrastructure,
   which would need constant maintenance; nobody has stepped up to do that
   work, so that's just how things are for now.

 * A warranty or support contract with some vendor doesn't entitle you to
   request fixes from developers in the upstream Linux kernel community: such
   contracts are completely outside the scope of the Linux kernel, its
   development community, and this document. That's why you can't demand
   anything such a contract guarantees in this context, not even if the
   developer handling the issue works for the vendor in question. If you want
   to claim your rights, use the vendor's support channel instead. When doing
   so, you might want to mention you'd like to see the issue fixed in the
   upstream Linux kernel; motivate them by saying it's the only way to ensure
   the fix in the end will get incorporated in all Linux distributions.

 * If you never reported an issue to a FLOSS project before you should consider
   reading `How to Report Bugs Effectively
   <https://www.chiark.greenend.org.uk/~sgtatham/bugs.html>`_, `How To Ask
   Questions The Smart Way
   <http://www.catb.org/esr/faqs/smart-questions.html>`_, and `How to ask good
   questions <https://jvns.ca/blog/good-questions/>`_.

With that off the table, find below the details on how to properly report
issues to the Linux kernel developers.

Make sure you're using the upstream Linux kernel
------------------------------------------------

   *Are you facing an issue with a Linux kernel a hardware or software vendor
   provided? Then in almost all cases you are better off to stop reading this
   document and reporting the issue to your vendor instead, unless you are
   willing to install the latest Linux version yourself. Be aware the latter
   will often be needed anyway to hunt down and fix issues.*

Like most programmers, Linux kernel developers don't like to spend time dealing
with reports for issues that don't even happen with their current code. It's
just a waste everybody's time, especially yours. Unfortunately such situations
easily happen when it comes to the kernel and often leads to frustration on both
sides. That's because almost all Linux-based kernels pre-installed on devices
(Computers, Laptops, Smartphones, Routers, …) and most shipped by Linux
distributors are quite distant from the official Linux kernel as distributed by
kernel.org: these kernels from these vendors are often ancient from the point of
Linux development or heavily modified, often both.

Most of these vendor kernels are quite unsuitable for reporting issues to the
Linux kernel developers: an issue you face with one of them might have been
fixed by the Linux kernel developers months or years ago already; additionally,
the modifications and enhancements by the vendor might be causing the issue you
face, even if they look small or totally unrelated. That's why you should report
issues with these kernels to the vendor. Its developers should look into the
report and, in case it turns out to be an upstream issue, fix it directly
upstream or forward the report there. In practice that often does not work out
or might not what you want. You thus might want to consider circumventing the
vendor by installing the very latest Linux kernel core yourself. If that's an
option for you move ahead in this process, as a later step in this guide will
explain how to do that once it rules out other potential causes for your issue.

Note, the previous paragraph is starting with the word 'most', as sometimes
developers in fact are willing to handle reports about issues occurring with
vendor kernels. If they do in the end highly depends on the developers and the
issue in question. Your chances are quite good if the distributor applied only
small modifications to a kernel based on a recent Linux version; that for
example often holds true for the mainline kernels shipped by Debian GNU/Linux
Sid or Fedora Rawhide. Some developers will also accept reports about issues
with kernels from distributions shipping the latest stable kernel, as long as
its only slightly modified; that for example is often the case for Arch Linux,
regular Fedora releases, and openSUSE Tumbleweed. But keep in mind, you better
want to use a mainline Linux and avoid using a stable kernel for this
process, as outlined in the section 'Install a fresh kernel for testing' in more
detail.

Obviously you are free to ignore all this advice and report problems with an old
or heavily modified vendor kernel to the upstream Linux developers. But note,
those often get rejected or ignored, so consider yourself warned. But it's still
better than not reporting the issue at all: sometimes such reports directly or
indirectly will help to get the issue fixed over time.

Search for existing reports, first run
--------------------------------------

   *Perform a rough search for existing reports with your favorite internet
   search engine; additionally, check the archives of the Linux Kernel Mailing
   List (LKML). If you find matching reports, join the discussion instead of
   sending a new one.*

Reporting an issue that someone else already brought forward is often a waste of
time for everyone involved, especially you as the reporter. So it's in your own
interest to thoroughly check if somebody reported the issue already. At this
step of the process it's okay to just perform a rough search: a later step will
tell you to perform a more detailed search once you know where your issue needs
to be reported to. Nevertheless, do not hurry with this step of the reporting
process, it can save you time and trouble.

Simply search the internet with your favorite search engine first. Afterwards,
search the `Linux Kernel Mailing List (LKML) archives
<https://lore.kernel.org/lkml/>`_.

If you get flooded with results consider telling your search engine to limit
search timeframe to the past month or year. And wherever you search, make sure
to use good search terms; vary them a few times, too. While doing so try to
look at the issue from the perspective of someone else: that will help you to
come up with other words to use as search terms. Also make sure not to use too
many search terms at once. Remember to search with and without information like
the name of the kernel driver or the name of the affected hardware component.
But its exact brand name (say 'ASUS Red Devil Radeon RX 5700 XT Gaming OC')
often is not much helpful, as it is too specific. Instead try search terms like
the model line (Radeon 5700 or Radeon 5000) and the code name of the main chip
('Navi' or 'Navi10') with and without its manufacturer ('AMD').

In case you find an existing report about your issue, join the discussion, as
you might be able to provide valuable additional information. That can be
important even when a fix is prepared or in its final stages already, as
developers might look for people that can provide additional information or
test a proposed fix. Jump to the section 'Duties after the report went out' for
details on how to get properly involved.

Note, searching `bugzilla.kernel.org <https://bugzilla.kernel.org/>`_ might also
be a good idea, as that might provide valuable insights or turn up matching
reports. If you find the latter, just keep in mind: most subsystems expect
reports in different places, as described below in the section "Check where you
need to report your issue". The developers that should take care of the issue
thus might not even be aware of the bugzilla ticket. Hence, check the ticket if
the issue already got reported as outlined in this document and if not consider
doing so.

Issue of high priority?
-----------------------

    *See if the issue you are dealing with qualifies as regression, security
    issue, or a really severe problem: those are 'issues of high priority' that
    need special handling in some steps that are about to follow.*

Linus Torvalds and the leading Linux kernel developers want to see some issues
fixed as soon as possible, hence there are 'issues of high priority' that get
handled slightly differently in the reporting process. Three type of cases
qualify: regressions, security issues, and really severe problems.

You deal with a regression if some application or practical use case running
fine with one Linux kernel works worse or not at all with a newer version
compiled using a similar configuration. The document
Documentation/admin-guide/reporting-regressions.rst explains this in more
detail. It also provides a good deal of other information about regressions you
might want to be aware of; it for example explains how to add your issue to the
list of tracked regressions, to ensure it won't fall through the cracks.

What qualifies as security issue is left to your judgment. Consider reading
Documentation/process/security-bugs.rst before proceeding, as it
provides additional details how to best handle security issues.

An issue is a 'really severe problem' when something totally unacceptably bad
happens. That's for example the case when a Linux kernel corrupts the data it's
handling or damages hardware it's running on. You're also dealing with a severe
issue when the kernel suddenly stops working with an error message ('kernel
panic') or without any farewell note at all. Note: do not confuse a 'panic' (a
fatal error where the kernel stop itself) with a 'Oops' (a recoverable error),
as the kernel remains running after the latter.

Ensure a healthy environment
----------------------------

    *Make sure it's not the kernel's surroundings that are causing the issue
    you face.*

Problems that look a lot like a kernel issue are sometimes caused by build or
runtime environment. It's hard to rule out that problem completely, but you
should minimize it:

 * Use proven tools when building your kernel, as bugs in the compiler or the
   binutils can cause the resulting kernel to misbehave.

 * Ensure your computer components run within their design specifications;
   that's especially important for the main processor, the main memory, and the
   motherboard. Therefore, stop undervolting or overclocking when facing a
   potential kernel issue.

 * Try to make sure it's not faulty hardware that is causing your issue. Bad
   main memory for example can result in a multitude of issues that will
   manifest itself in problems looking like kernel issues.

 * If you're dealing with a filesystem issue, you might want to check the file
   system in question with ``fsck``, as it might be damaged in a way that leads
   to unexpected kernel behavior.

 * When dealing with a regression, make sure it's not something else that
   changed in parallel to updating the kernel. The problem for example might be
   caused by other software that was updated at the same time. It can also
   happen that a hardware component coincidentally just broke when you rebooted
   into a new kernel for the first time. Updating the systems BIOS or changing
   something in the BIOS Setup can also lead to problems that on look a lot
   like a kernel regression.

Prepare for emergencies
-----------------------

    *Create a fresh backup and put system repair and restore tools at hand.*

Reminder, you are dealing with computers, which sometimes do unexpected things,
especially if you fiddle with crucial parts like the kernel of its operating
system. That's what you are about to do in this process. Thus, make sure to
create a fresh backup; also ensure you have all tools at hand to repair or
reinstall the operating system as well as everything you need to restore the
backup.

Make sure your kernel doesn't get enhanced
------------------------------------------

    *Ensure your system does not enhance its kernels by building additional
    kernel modules on-the-fly, which solutions like DKMS might be doing locally
    without your knowledge.*

The risk your issue report gets ignored or rejected dramatically increases if
your kernel gets enhanced in any way. That's why you should remove or disable
mechanisms like akmods and DKMS: those build add-on kernel modules
automatically, for example when you install a new Linux kernel or boot it for
the first time. Also remove any modules they might have installed. Then reboot
before proceeding.

Note, you might not be aware that your system is using one of these solutions:
they often get set up silently when you install Nvidia's proprietary graphics
driver, VirtualBox, or other software that requires a some support from a
module not part of the Linux kernel. That why your might need to uninstall the
packages with such software to get rid of any 3rd party kernel module.

Check 'taint' flag
------------------

    *Check if your kernel was 'tainted' when the issue occurred, as the event
    that made the kernel set this flag might be causing the issue you face.*

The kernel marks itself with a 'taint' flag when something happens that might
lead to follow-up errors that look totally unrelated. The issue you face might
be such an error if your kernel is tainted. That's why it's in your interest to
rule this out early before investing more time into this process. This is the
only reason why this step is here, as this process later will tell you to
install the latest mainline kernel; you will need to check the taint flag again
then, as that's when it matters because it's the kernel the report will focus
on.

On a running system is easy to check if the kernel tainted itself: if ``cat
/proc/sys/kernel/tainted`` returns '0' then the kernel is not tainted and
everything is fine. Checking that file is impossible in some situations; that's
why the kernel also mentions the taint status when it reports an internal
problem (a 'kernel bug'), a recoverable error (a 'kernel Oops') or a
non-recoverable error before halting operation (a 'kernel panic'). Look near
the top of the error messages printed when one of these occurs and search for a
line starting with 'CPU:'. It should end with 'Not tainted' if the kernel was
not tainted when it noticed the problem; it was tainted if you see 'Tainted:'
followed by a few spaces and some letters.

If your kernel is tainted, study Documentation/admin-guide/tainted-kernels.rst
to find out why. Try to eliminate the reason. Often it's caused by one these
three things:

 1. A recoverable error (a 'kernel Oops') occurred and the kernel tainted
    itself, as the kernel knows it might misbehave in strange ways after that
    point. In that case check your kernel or system log and look for a section
    that starts with this::

       Oops: 0000 [#1] SMP

    That's the first Oops since boot-up, as the '#1' between the brackets shows.
    Every Oops and any other problem that happens after that point might be a
    follow-up problem to that first Oops, even if both look totally unrelated.
    Rule this out by getting rid of the cause for the first Oops and reproducing
    the issue afterwards. Sometimes simply restarting will be enough, sometimes
    a change to the configuration followed by a reboot can eliminate the Oops.
    But don't invest too much time into this at this point of the process, as
    the cause for the Oops might already be fixed in the newer Linux kernel
    version you are going to install later in this process.

 2. Your system uses a software that installs its own kernel modules, for
    example Nvidia's proprietary graphics driver or VirtualBox. The kernel
    taints itself when it loads such module from external sources (even if
    they are Open Source): they sometimes cause errors in unrelated kernel
    areas and thus might be causing the issue you face. You therefore have to
    prevent those modules from loading when you want to report an issue to the
    Linux kernel developers. Most of the time the easiest way to do that is:
    temporarily uninstall such software including any modules they might have
    installed. Afterwards reboot.

 3. The kernel also taints itself when it's loading a module that resides in
    the staging tree of the Linux kernel source. That's a special area for
    code (mostly drivers) that does not yet fulfill the normal Linux kernel
    quality standards. When you report an issue with such a module it's
    obviously okay if the kernel is tainted; just make sure the module in
    question is the only reason for the taint. If the issue happens in an
    unrelated area reboot and temporarily block the module from being loaded
    by specifying ``foo.blacklist=1`` as kernel parameter (replace 'foo' with
    the name of the module in question).

Document how to reproduce issue
-------------------------------

    *Write down coarsely how to reproduce the issue. If you deal with multiple
    issues at once, create separate notes for each of them and make sure they
    work independently on a freshly booted system. That's needed, as each issue
    needs to get reported to the kernel developers separately, unless they are
    strongly entangled.*

If you deal with multiple issues at once, you'll have to report each of them
separately, as they might be handled by different developers. Describing
various issues in one report also makes it quite difficult for others to tear
it apart. Hence, only combine issues in one report if they are very strongly
entangled.

Additionally, during the reporting process you will have to test if the issue
happens with other kernel versions. Therefore, it will make your work easier if
you know exactly how to reproduce an issue quickly on a freshly booted system.

Note: it's often fruitless to report issues that only happened once, as they
might be caused by a bit flip due to cosmic radiation. That's why you should
try to rule that out by reproducing the issue before going further. Feel free
to ignore this advice if you are experienced enough to tell a one-time error
due to faulty hardware apart from a kernel issue that rarely happens and thus
is hard to reproduce.

Regression in stable or longterm kernel?
----------------------------------------

    *If you are facing a regression within a stable or longterm version line
    (say something broke when updating from 5.10.4 to 5.10.5), scroll down to
    'Dealing with regressions within a stable and longterm kernel line'.*

Regression within a stable and longterm kernel version line are something the
Linux developers want to fix badly, as such issues are even more unwanted than
regression in the main development branch, as they can quickly affect a lot of
people. The developers thus want to learn about such issues as quickly as
possible, hence there is a streamlined process to report them. Note,
regressions with newer kernel version line (say something broke when switching
from 5.9.15 to 5.10.5) do not qualify.

Check where you need to report your issue
-----------------------------------------

    *Locate the driver or kernel subsystem that seems to be causing the issue.
    Find out how and where its developers expect reports. Note: most of the
    time this won't be bugzilla.kernel.org, as issues typically need to be sent
    by mail to a maintainer and a public mailing list.*

It's crucial to send your report to the right people, as the Linux kernel is a
big project and most of its developers are only familiar with a small subset of
it. Quite a few programmers for example only care for just one driver, for
example one for a WiFi chip; its developer likely will only have small or no
knowledge about the internals of remote or unrelated "subsystems", like the TCP
stack, the PCIe/PCI subsystem, memory management or file systems.

Problem is: the Linux kernel lacks a central bug tracker where you can simply
file your issue and make it reach the developers that need to know about it.
That's why you have to find the right place and way to report issues yourself.
You can do that with the help of a script (see below), but it mainly targets
kernel developers and experts. For everybody else the MAINTAINERS file is the
better place.

How to read the MAINTAINERS file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To illustrate how to use the :ref:`MAINTAINERS <maintainers>` file, lets assume
the WiFi in your Laptop suddenly misbehaves after updating the kernel. In that
case it's likely an issue in the WiFi driver. Obviously it could also be some
code it builds upon, but unless you suspect something like that stick to the
driver. If it's really something else, the driver's developers will get the
right people involved.

Sadly, there is no way to check which code is driving a particular hardware
component that is both universal and easy.

In case of a problem with the WiFi driver you for example might want to look at
the output of ``lspci -k``, as it lists devices on the PCI/PCIe bus and the
kernel module driving it::

       [user@something ~]$ lspci -k
       [...]
       3a:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32)
         Subsystem: Bigfoot Networks, Inc. Device 1535
         Kernel driver in use: ath10k_pci
         Kernel modules: ath10k_pci
       [...]

But this approach won't work if your WiFi chip is connected over USB or some
other internal bus. In those cases you might want to check your WiFi manager or
the output of ``ip link``. Look for the name of the problematic network
interface, which might be something like 'wlp58s0'. This name can be used like
this to find the module driving it::

       [user@something ~]$ realpath --relative-to=/sys/module/ /sys/class/net/wlp58s0/device/driver/module
       ath10k_pci

In case tricks like these don't bring you any further, try to search the
internet on how to narrow down the driver or subsystem in question. And if you
are unsure which it is: just try your best guess, somebody will help you if you
guessed poorly.

Once you know the driver or subsystem, you want to search for it in the
MAINTAINERS file. In the case of 'ath10k_pci' you won't find anything, as the
name is too specific. Sometimes you will need to search on the net for help;
but before doing so, try a somewhat shorted or modified name when searching the
MAINTAINERS file, as then you might find something like this::

       QUALCOMM ATHEROS ATH10K WIRELESS DRIVER
       Mail:          A. Some Human <shuman@example.com>
       Mailing list:  ath10k@lists.infradead.org
       Status:        Supported
       Web-page:      https://wireless.wiki.kernel.org/en/users/Drivers/ath10k
       SCM:           git git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git
       Files:         drivers/net/wireless/ath/ath10k/

Note: the line description will be abbreviations, if you read the plain
MAINTAINERS file found in the root of the Linux source tree. 'Mail:' for
example will be 'M:', 'Mailing list:' will be 'L', and 'Status:' will be 'S:'.
A section near the top of the file explains these and other abbreviations.

First look at the line 'Status'. Ideally it should be 'Supported' or
'Maintained'. If it states 'Obsolete' then you are using some outdated approach
that was replaced by a newer solution you need to switch to. Sometimes the code
only has someone who provides 'Odd Fixes' when feeling motivated. And with
'Orphan' you are totally out of luck, as nobody takes care of the code anymore.
That only leaves these options: arrange yourself to live with the issue, fix it
yourself, or find a programmer somewhere willing to fix it.

After checking the status, look for a line starting with 'bugs:': it will tell
you where to find a subsystem specific bug tracker to file your issue. The
example above does not have such a line. That is the case for most sections, as
Linux kernel development is completely driven by mail. Very few subsystems use
a bug tracker, and only some of those rely on bugzilla.kernel.org.

In this and many other cases you thus have to look for lines starting with
'Mail:' instead. Those mention the name and the email addresses for the
maintainers of the particular code. Also look for a line starting with 'Mailing
list:', which tells you the public mailing list where the code is developed.
Your report later needs to go by mail to those addresses. Additionally, for all
issue reports sent by email, make sure to add the Linux Kernel Mailing List
(LKML) <linux-kernel@vger.kernel.org> to CC. Don't omit either of the mailing
lists when sending your issue report by mail later! Maintainers are busy people
and might leave some work for other developers on the subsystem specific list;
and LKML is important to have one place where all issue reports can be found.

Finding the maintainers with the help of a script
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For people that have the Linux sources at hand there is a second option to find
the proper place to report: the script 'scripts/get_maintainer.pl' which tries
to find all people to contact. It queries the MAINTAINERS file and needs to be
called with a path to the source code in question. For drivers compiled as
module if often can be found with a command like this::

       $ modinfo ath10k_pci | grep filename | sed 's!/lib/modules/.*/kernel/!!; s!filename:!!; s!\.ko\(\|\.xz\)!!'
       drivers/net/wireless/ath/ath10k/ath10k_pci.ko

Pass parts of this to the script::

       $ ./scripts/get_maintainer.pl -f drivers/net/wireless/ath/ath10k*
       Some Human <shuman@example.com> (supporter:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER)
       Another S. Human <asomehuman@example.com> (maintainer:NETWORKING DRIVERS)
       ath10k@lists.infradead.org (open list:QUALCOMM ATHEROS ATH10K WIRELESS DRIVER)
       linux-wireless@vger.kernel.org (open list:NETWORKING DRIVERS (WIRELESS))
       netdev@vger.kernel.org (open list:NETWORKING DRIVERS)
       linux-kernel@vger.kernel.org (open list)

Don't sent your report to all of them. Send it to the maintainers, which the
script calls "supporter:"; additionally CC the most specific mailing list for
the code as well as the Linux Kernel Mailing List (LKML). In this case you thus
would need to send the report to 'Some Human <shuman@example.com>' with
'ath10k@lists.infradead.org' and 'linux-kernel@vger.kernel.org' in CC.

Note: in case you cloned the Linux sources with git you might want to call
``get_maintainer.pl`` a second time with ``--git``. The script then will look
at the commit history to find which people recently worked on the code in
question, as they might be able to help. But use these results with care, as it
can easily send you in a wrong direction. That for example happens quickly in
areas rarely changed (like old or unmaintained drivers): sometimes such code is
modified during tree-wide cleanups by developers that do not care about the
particular driver at all.

Search for existing reports, second run
---------------------------------------

    *Search the archives of the bug tracker or mailing list in question
    thoroughly for reports that might match your issue. If you find anything,
    join the discussion instead of sending a new report.*

As mentioned earlier already: reporting an issue that someone else already
brought forward is often a waste of time for everyone involved, especially you
as the reporter. That's why you should search for existing report again, now
that you know where they need to be reported to. If it's mailing list, you will
often find its archives on `lore.kernel.org <https://lore.kernel.org/>`_.

But some list are hosted in different places. That for example is the case for
the ath10k WiFi driver used as example in the previous step. But you'll often
find the archives for these lists easily on the net. Searching for 'archive
ath10k@lists.infradead.org' for example will lead you to the `Info page for the
ath10k mailing list <https://lists.infradead.org/mailman/listinfo/ath10k>`_,
which at the top links to its
`list archives <https://lists.infradead.org/pipermail/ath10k/>`_. Sadly this and
quite a few other lists miss a way to search the archives. In those cases use a
regular internet search engine and add something like
'site:lists.infradead.org/pipermail/ath10k/' to your search terms, which limits
the results to the archives at that URL.

It's also wise to check the internet, LKML and maybe bugzilla.kernel.org again
at this point. If your report needs to be filed in a bug tracker, you may want
to check the mailing list archives for the subsystem as well, as someone might
have reported it only there.

For details how to search and what to do if you find matching reports see
"Search for existing reports, first run" above.

Do not hurry with this step of the reporting process: spending 30 to 60 minutes
or even more time can save you and others quite a lot of time and trouble.

Install a fresh kernel for testing
----------------------------------

    *Unless you are already running the latest 'mainline' Linux kernel, better
    go and install it for the reporting process. Testing and reporting with
    the latest 'stable' Linux can be an acceptable alternative in some
    situations; during the merge window that actually might be even the best
    approach, but in that development phase it can be an even better idea to
    suspend your efforts for a few days anyway. Whatever version you choose,
    ideally use a 'vanilla' built. Ignoring these advices will dramatically
    increase the risk your report will be rejected or ignored.*

As mentioned in the detailed explanation for the first step already: Like most
programmers, Linux kernel developers don't like to spend time dealing with
reports for issues that don't even happen with the current code. It's just a
waste everybody's time, especially yours. That's why it's in everybody's
interest that you confirm the issue still exists with the latest upstream code
before reporting it. You are free to ignore this advice, but as outlined
earlier: doing so dramatically increases the risk that your issue report might
get rejected or simply ignored.

In the scope of the kernel "latest upstream" normally means:

 * Install a mainline kernel; the latest stable kernel can be an option, but
   most of the time is better avoided. Longterm kernels (sometimes called 'LTS
   kernels') are unsuitable at this point of the process. The next subsection
   explains all of this in more detail.

 * The over next subsection describes way to obtain and install such a kernel.
   It also outlines that using a pre-compiled kernel are fine, but better are
   vanilla, which means: it was built using Linux sources taken straight `from
   kernel.org <https://kernel.org/>`_ and not modified or enhanced in any way.

Choosing the right version for testing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Head over to `kernel.org <https://kernel.org/>`_ to find out which version you
want to use for testing. Ignore the big yellow button that says 'Latest release'
and look a little lower at the table. At its top you'll see a line starting with
mainline, which most of the time will point to a pre-release with a version
number like '5.8-rc2'. If that's the case, you'll want to use this mainline
kernel for testing, as that where all fixes have to be applied first. Do not let
that 'rc' scare you, these 'development kernels' are pretty reliable — and you
made a backup, as you were instructed above, didn't you?

In about two out of every nine to ten weeks, mainline might point you to a
proper release with a version number like '5.7'. If that happens, consider
suspending the reporting process until the first pre-release of the next
version (5.8-rc1) shows up on kernel.org. That's because the Linux development
cycle then is in its two-week long 'merge window'. The bulk of the changes and
all intrusive ones get merged for the next release during this time. It's a bit
more risky to use mainline during this period. Kernel developers are also often
quite busy then and might have no spare time to deal with issue reports. It's
also quite possible that one of the many changes applied during the merge
window fixes the issue you face; that's why you soon would have to retest with
a newer kernel version anyway, as outlined below in the section 'Duties after
the report went out'.

That's why it might make sense to wait till the merge window is over. But don't
to that if you're dealing with something that shouldn't wait. In that case
consider obtaining the latest mainline kernel via git (see below) or use the
latest stable version offered on kernel.org. Using that is also acceptable in
case mainline for some reason does currently not work for you. An in general:
using it for reproducing the issue is also better than not reporting it issue
at all.

Better avoid using the latest stable kernel outside merge windows, as all fixes
must be applied to mainline first. That's why checking the latest mainline
kernel is so important: any issue you want to see fixed in older version lines
needs to be fixed in mainline first before it can get backported, which can
take a few days or weeks. Another reason: the fix you hope for might be too
hard or risky for backporting; reporting the issue again hence is unlikely to
change anything.

These aspects are also why longterm kernels (sometimes called "LTS kernels")
are unsuitable for this part of the reporting process: they are to distant from
the current code. Hence go and test mainline first and follow the process
further: if the issue doesn't occur with mainline it will guide you how to get
it fixed in older version lines, if that's in the cards for the fix in question.

How to obtain a fresh Linux kernel
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Using a pre-compiled kernel**: This is often the quickest, easiest, and safest
way for testing — especially is you are unfamiliar with the Linux kernel. The
problem: most of those shipped by distributors or add-on repositories are build
from modified Linux sources. They are thus not vanilla and therefore often
unsuitable for testing and issue reporting: the changes might cause the issue
you face or influence it somehow.

But you are in luck if you are using a popular Linux distribution: for quite a
few of them you'll find repositories on the net that contain packages with the
latest mainline or stable Linux built as vanilla kernel. It's totally okay to
use these, just make sure from the repository's description they are vanilla or
at least close to it. Additionally ensure the packages contain the latest
versions as offered on kernel.org. The packages are likely unsuitable if they
are older than a week, as new mainline and stable kernels typically get released
at least once a week.

Please note that you might need to build your own kernel manually later: that's
sometimes needed for debugging or testing fixes, as described later in this
document. Also be aware that pre-compiled kernels might lack debug symbols that
are needed to decode messages the kernel prints when a panic, Oops, warning, or
BUG occurs; if you plan to decode those, you might be better off compiling a
kernel yourself (see the end of this subsection and the section titled 'Decode
failure messages' for details).

**Using git**: Developers and experienced Linux users familiar with git are
often best served by obtaining the latest Linux kernel sources straight from the
`official development repository on kernel.org
<https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/>`_.
Those are likely a bit ahead of the latest mainline pre-release. Don't worry
about it: they are as reliable as a proper pre-release, unless the kernel's
development cycle is currently in the middle of a merge window. But even then
they are quite reliable.

**Conventional**: People unfamiliar with git are often best served by
downloading the sources as tarball from `kernel.org <https://kernel.org/>`_.

How to actually build a kernel is not described here, as many websites explain
the necessary steps already. If you are new to it, consider following one of
those how-to's that suggest to use ``make localmodconfig``, as that tries to
pick up the configuration of your current kernel and then tries to adjust it
somewhat for your system. That does not make the resulting kernel any better,
but quicker to compile.

Note: If you are dealing with a panic, Oops, warning, or BUG from the kernel,
please try to enable CONFIG_KALLSYMS when configuring your kernel.
Additionally, enable CONFIG_DEBUG_KERNEL and CONFIG_DEBUG_INFO, too; the
latter is the relevant one of those two, but can only be reached if you enable
the former. Be aware CONFIG_DEBUG_INFO increases the storage space required to
build a kernel by quite a bit. But that's worth it, as these options will allow
you later to pinpoint the exact line of code that triggers your issue. The
section 'Decode failure messages' below explains this in more detail.

But keep in mind: Always keep a record of the issue encountered in case it is
hard to reproduce. Sending an undecoded report is better than not reporting
the issue at all.

Check 'taint' flag
------------------

    *Ensure the kernel you just installed does not 'taint' itself when
    running.*

As outlined above in more detail already: the kernel sets a 'taint' flag when
something happens that can lead to follow-up errors that look totally
unrelated. That's why you need to check if the kernel you just installed does
not set this flag. And if it does, you in almost all the cases needs to
eliminate the reason for it before you reporting issues that occur with it. See
the section above for details how to do that.

Reproduce issue with the fresh kernel
-------------------------------------

    *Reproduce the issue with the kernel you just installed. If it doesn't show
    up there, scroll down to the instructions for issues only happening with
    stable and longterm kernels.*

Check if the issue occurs with the fresh Linux kernel version you just
installed. If it was fixed there already, consider sticking with this version
line and abandoning your plan to report the issue. But keep in mind that other
users might still be plagued by it, as long as it's not fixed in either stable
and longterm version from kernel.org (and thus vendor kernels derived from
those). If you prefer to use one of those or just want to help their users,
head over to the section "Details about reporting issues only occurring in
older kernel version lines" below.

Optimize description to reproduce issue
---------------------------------------

    *Optimize your notes: try to find and write the most straightforward way to
    reproduce your issue. Make sure the end result has all the important
    details, and at the same time is easy to read and understand for others
    that hear about it for the first time. And if you learned something in this
    process, consider searching again for existing reports about the issue.*

An unnecessarily complex report will make it hard for others to understand your
report. Thus try to find a reproducer that's straight forward to describe and
thus easy to understand in written form. Include all important details, but at
the same time try to keep it as short as possible.

In this in the previous steps you likely have learned a thing or two about the
issue you face. Use this knowledge and search again for existing reports
instead you can join.

Decode failure messages
-----------------------

    *If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider
    decoding the kernel log to find the line of code that triggered the error.*

When the kernel detects an internal problem, it will log some information about
the executed code. This makes it possible to pinpoint the exact line in the
source code that triggered the issue and shows how it was called. But that only
works if you enabled CONFIG_DEBUG_INFO and CONFIG_KALLSYMS when configuring
your kernel. If you did so, consider to decode the information from the
kernel's log. That will make it a lot easier to understand what lead to the
'panic', 'Oops', 'warning', or 'BUG', which increases the chances that someone
can provide a fix.

Decoding can be done with a script you find in the Linux source tree. If you
are running a kernel you compiled yourself earlier, call it like this::

       [user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh ./linux-5.10.5/vmlinux

If you are running a packaged vanilla kernel, you will likely have to install
the corresponding packages with debug symbols. Then call the script (which you
might need to get from the Linux sources if your distro does not package it)
like this::

       [user@something ~]$ sudo dmesg | ./linux-5.10.5/scripts/decode_stacktrace.sh \
        /usr/lib/debug/lib/modules/5.10.10-4.1.x86_64/vmlinux /usr/src/kernels/5.10.10-4.1.x86_64/

The script will work on log lines like the following, which show the address of
the code the kernel was executing when the error occurred::

       [   68.387301] RIP: 0010:test_module_init+0x5/0xffa [test_module]

Once decoded, these lines will look like this::

       [   68.387301] RIP: 0010:test_module_init (/home/username/linux-5.10.5/test-module/test-module.c:16) test_module

In this case the executed code was built from the file
'~/linux-5.10.5/test-module/test-module.c' and the error occurred by the
instructions found in line '16'.

The script will similarly decode the addresses mentioned in the section
starting with 'Call trace', which show the path to the function where the
problem occurred. Additionally, the script will show the assembler output for
the code section the kernel was executing.

Note, if you can't get this to work, simply skip this step and mention the
reason for it in the report. If you're lucky, it might not be needed. And if it
is, someone might help you to get things going. Also be aware this is just one
of several ways to decode kernel stack traces. Sometimes different steps will
be required to retrieve the relevant details. Don't worry about that, if that's
needed in your case, developers will tell you what to do.

Special care for regressions
----------------------------

    *If your problem is a regression, try to narrow down when the issue was
    introduced as much as possible.*

Linux lead developer Linus Torvalds insists that the Linux kernel never
worsens, that's why he deems regressions as unacceptable and wants to see them
fixed quickly. That's why changes that introduced a regression are often
promptly reverted if the issue they cause can't get solved quickly any other
way. Reporting a regression is thus a bit like playing a kind of trump card to
get something quickly fixed. But for that to happen the change that's causing
the regression needs to be known. Normally it's up to the reporter to track
down the culprit, as maintainers often won't have the time or setup at hand to
reproduce it themselves.

To find the change there is a process called 'bisection' which the document
Documentation/admin-guide/bug-bisect.rst describes in detail. That process
will often require you to build about ten to twenty kernel images, trying to
reproduce the issue with each of them before building the next. Yes, that takes
some time, but don't worry, it works a lot quicker than most people assume.
Thanks to a 'binary search' this will lead you to the one commit in the source
code management system that's causing the regression. Once you find it, search
the net for the subject of the change, its commit id and the shortened commit id
(the first 12 characters of the commit id). This will lead you to existing
reports about it, if there are any.

Note, a bisection needs a bit of know-how, which not everyone has, and quite a
bit of effort, which not everyone is willing to invest. Nevertheless, it's
highly recommended performing a bisection yourself. If you really can't or
don't want to go down that route at least find out which mainline kernel
introduced the regression. If something for example breaks when switching from
5.5.15 to 5.8.4, then try at least all the mainline releases in that area (5.6,
5.7 and 5.8) to check when it first showed up. Unless you're trying to find a
regression in a stable or longterm kernel, avoid testing versions which number
has three sections (5.6.12, 5.7.8), as that makes the outcome hard to
interpret, which might render your testing useless. Once you found the major
version which introduced the regression, feel free to move on in the reporting
process. But keep in mind: it depends on the issue at hand if the developers
will be able to help without knowing the culprit. Sometimes they might
recognize from the report want went wrong and can fix it; other times they will
be unable to help unless you perform a bisection.

When dealing with regressions make sure the issue you face is really caused by
the kernel and not by something else, as outlined above already.

In the whole process keep in mind: an issue only qualifies as regression if the
older and the newer kernel got built with a similar configuration. This can be
achieved by using ``make olddefconfig``, as explained in more detail by
Documentation/admin-guide/reporting-regressions.rst; that document also
provides a good deal of other information about regressions you might want to be
aware of.

Write and send the report
-------------------------

    *Start to compile the report by writing a detailed description about the
    issue. Always mention a few things: the latest kernel version you installed
    for reproducing, the Linux Distribution used, and your notes on how to
    reproduce the issue. Ideally, make the kernel's build configuration
    (.config) and the output from ``dmesg`` available somewhere on the net and
    link to it. Include or upload all other information that might be relevant,
    like the output/screenshot of an Oops or the output from ``lspci``. Once
    you wrote this main part, insert a normal length paragraph on top of it
    outlining the issue and the impact quickly. On top of this add one sentence
    that briefly describes the problem and gets people to read on. Now give the
    thing a descriptive title or subject that yet again is shorter. Then you're
    ready to send or file the report like the MAINTAINERS file told you, unless
    you are dealing with one of those 'issues of high priority': they need
    special care which is explained in 'Special handling for high priority
    issues' below.*

Now that you have prepared everything it's time to write your report. How to do
that is partly explained by the three documents linked to in the preface above.
That's why this text will only mention a few of the essentials as well as
things specific to the Linux kernel.

There is one thing that fits both categories: the most crucial parts of your
report are the title/subject, the first sentence, and the first paragraph.
Developers often get quite a lot of mail. They thus often just take a few
seconds to skim a mail before deciding to move on or look closer. Thus: the
better the top section of your report, the higher are the chances that someone
will look into it and help you. And that is why you should ignore them for now
and write the detailed report first. ;-)

Things each report should mention
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Describe in detail how your issue happens with the fresh vanilla kernel you
installed. Try to include the step-by-step instructions you wrote and optimized
earlier that outline how you and ideally others can reproduce the issue; in
those rare cases where that's impossible try to describe what you did to
trigger it.

Also include all the relevant information others might need to understand the
issue and its environment. What's actually needed depends a lot on the issue,
but there are some things you should include always:

 * the output from ``cat /proc/version``, which contains the Linux kernel
   version number and the compiler it was built with.

 * the Linux distribution the machine is running (``hostnamectl | grep
   "Operating System"``)

 * the architecture of the CPU and the operating system (``uname -mi``)

 * if you are dealing with a regression and performed a bisection, mention the
   subject and the commit-id of the change that is causing it.

In a lot of cases it's also wise to make two more things available to those
that read your report:

 * the configuration used for building your Linux kernel (the '.config' file)

 * the kernel's messages that you get from ``dmesg`` written to a file. Make
   sure that it starts with a line like 'Linux version 5.8-1
   (foobar@example.com) (gcc (GCC) 10.2.1, GNU ld version 2.34) #1 SMP Mon Aug
   3 14:54:37 UTC 2020' If it's missing, then important messages from the first
   boot phase already got discarded. In this case instead consider using
   ``journalctl -b 0 -k``; alternatively you can also reboot, reproduce the
   issue and call ``dmesg`` right afterwards.

These two files are big, that's why it's a bad idea to put them directly into
your report. If you are filing the issue in a bug tracker then attach them to
the ticket. If you report the issue by mail do not attach them, as that makes
the mail too large; instead do one of these things:

 * Upload the files somewhere public (your website, a public file paste
   service, a ticket created just for this purpose on `bugzilla.kernel.org
   <https://bugzilla.kernel.org/>`_, ...) and include a link to them in your
   report. Ideally use something where the files stay available for years, as
   they could be useful to someone many years from now; this for example can
   happen if five or ten years from now a developer works on some code that was
   changed just to fix your issue.

 * Put the files aside and mention you will send them later in individual
   replies to your own mail. Just remember to actually do that once the report
   went out. ;-)

Things that might be wise to provide
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Depending on the issue you might need to add more background data. Here are a
few suggestions what often is good to provide:

 * If you are dealing with a 'warning', an 'OOPS' or a 'panic' from the kernel,
   include it. If you can't copy'n'paste it, try to capture a netconsole trace
   or at least take a picture of the screen.

 * If the issue might be related to your computer hardware, mention what kind
   of system you use. If you for example have problems with your graphics card,
   mention its manufacturer, the card's model, and what chip is uses. If it's a
   laptop mention its name, but try to make sure it's meaningful. 'Dell XPS 13'
   for example is not, because it might be the one from 2012; that one looks
   not that different from the one sold today, but apart from that the two have
   nothing in common. Hence, in such cases add the exact model number, which
   for example are '9380' or '7390' for XPS 13 models introduced during 2019.
   Names like 'Lenovo Thinkpad T590' are also somewhat ambiguous: there are
   variants of this laptop with and without a dedicated graphics chip, so try
   to find the exact model name or specify the main components.

 * Mention the relevant software in use. If you have problems with loading
   modules, you want to mention the versions of kmod, systemd, and udev in use.
   If one of the DRM drivers misbehaves, you want to state the versions of
   libdrm and Mesa; also specify your Wayland compositor or the X-Server and
   its driver. If you have a filesystem issue, mention the version of
   corresponding filesystem utilities (e2fsprogs, btrfs-progs, xfsprogs, ...).

 * Gather additional information from the kernel that might be of interest. The
   output from ``lspci -nn`` will for example help others to identify what
   hardware you use. If you have a problem with hardware you even might want to
   make the output from ``sudo lspci -vvv`` available, as that provides
   insights how the components were configured. For some issues it might be
   good to include the contents of files like ``/proc/cpuinfo``,
   ``/proc/ioports``, ``/proc/iomem``, ``/proc/modules``, or
   ``/proc/scsi/scsi``. Some subsystem also offer tools to collect relevant
   information. One such tool is ``alsa-info.sh`` `which the audio/sound
   subsystem developers provide <https://www.alsa-project.org/wiki/AlsaInfo>`_.

Those examples should give your some ideas of what data might be wise to
attach, but you have to think yourself what will be helpful for others to know.
Don't worry too much about forgetting something, as developers will ask for
additional details they need. But making everything important available from
the start increases the chance someone will take a closer look.

The important part: the head of your report
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Now that you have the detailed part of the report prepared let's get to the
most important section: the first few sentences. Thus go to the top, add
something like 'The detailed description:' before the part you just wrote and
insert two newlines at the top. Now write one normal length paragraph that
describes the issue roughly. Leave out all boring details and focus on the
crucial parts readers need to know to understand what this is all about; if you
think this bug affects a lot of users, mention this to get people interested.

Once you did that insert two more lines at the top and write a one sentence
summary that explains quickly what the report is about. After that you have to
get even more abstract and write an even shorter subject/title for the report.

Now that you have written this part take some time to optimize it, as it is the
most important parts of your report: a lot of people will only read this before
they decide if reading the rest is time well spent.

Now send or file the report like the :ref:`MAINTAINERS <maintainers>` file told
you, unless it's one of those 'issues of high priority' outlined earlier: in
that case please read the next subsection first before sending the report on
its way.

Special handling for high priority issues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Reports for high priority issues need special handling.

**Severe issues**: make sure the subject or ticket title as well as the first
paragraph makes the severeness obvious.

**Regressions**: make the report's subject start with '[REGRESSION]'.

In case you performed a successful bisection, use the title of the change that
introduced the regression as the second part of your subject. Make the report
also mention the commit id of the culprit. In case of an unsuccessful bisection,
make your report mention the latest tested version that's working fine (say 5.7)
and the oldest where the issue occurs (say 5.8-rc1).

When sending the report by mail, CC the Linux regressions mailing list
(regressions@lists.linux.dev). In case the report needs to be filed to some web
tracker, proceed to do so. Once filed, forward the report by mail to the
regressions list; CC the maintainer and the mailing list for the subsystem in
question. Make sure to inline the forwarded report, hence do not attach it.
Also add a short note at the top where you mention the URL to the ticket.

When mailing or forwarding the report, in case of a successful bisection add the
author of the culprit to the recipients; also CC everyone in the signed-off-by
chain, which you find at the end of its commit message.

**Security issues**: for these issues your will have to evaluate if a
short-term risk to other users would arise if details were publicly disclosed.
If that's not the case simply proceed with reporting the issue as described.
For issues that bear such a risk you will need to adjust the reporting process
slightly:

 * If the MAINTAINERS file instructed you to report the issue by mail, do not
   CC any public mailing lists.

 * If you were supposed to file the issue in a bug tracker make sure to mark
   the ticket as 'private' or 'security issue'. If the bug tracker does not
   offer a way to keep reports private, forget about it and send your report as
   a private mail to the maintainers instead.

In both cases make sure to also mail your report to the addresses the
MAINTAINERS file lists in the section 'security contact'. Ideally directly CC
them when sending the report by mail. If you filed it in a bug tracker, forward
the report's text to these addresses; but on top of it put a small note where
you mention that you filed it with a link to the ticket.

See Documentation/process/security-bugs.rst for more information.

Duties after the report went out
--------------------------------

    *Wait for reactions and keep the thing rolling until you can accept the
    outcome in one way or the other. Thus react publicly and in a timely manner
    to any inquiries. Test proposed fixes. Do proactive testing: retest with at
    least every first release candidate (RC) of a new mainline version and
    report your results. Send friendly reminders if things stall. And try to
    help yourself, if you don't get any help or if it's unsatisfying.*

If your report was good and you are really lucky then one of the developers
might immediately spot what's causing the issue; they then might write a patch
to fix it, test it, and send it straight for integration in mainline while
tagging it for later backport to stable and longterm kernels that need it. Then
all you need to do is reply with a 'Thank you very much' and switch to a version
with the fix once it gets released.

But this ideal scenario rarely happens. That's why the job is only starting
once you got the report out. What you'll have to do depends on the situations,
but often it will be the things listed below. But before digging into the
details, here are a few important things you need to keep in mind for this part
of the process.

General advice for further interactions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Always reply in public**: When you filed the issue in a bug tracker, always
reply there and do not contact any of the developers privately about it. For
mailed reports always use the 'Reply-all' function when replying to any mails
you receive. That includes mails with any additional data you might want to add
to your report: go to your mail applications 'Sent' folder and use 'reply-all'
on your mail with the report. This approach will make sure the public mailing
list(s) and everyone else that gets involved over time stays in the loop; it
also keeps the mail thread intact, which among others is really important for
mailing lists to group all related mails together.

There are just two situations where a comment in a bug tracker or a 'Reply-all'
is unsuitable:

 * Someone tells you to send something privately.

 * You were told to send something, but noticed it contains sensitive
   information that needs to be kept private. In that case it's okay to send it
   in private to the developer that asked for it. But note in the ticket or a
   mail that you did that, so everyone else knows you honored the request.

**Do research before asking for clarifications or help**: In this part of the
process someone might tell you to do something that requires a skill you might
not have mastered yet. For example, you might be asked to use some test tools
you never have heard of yet; or you might be asked to apply a patch to the
Linux kernel sources to test if it helps. In some cases it will be fine sending
a reply asking for instructions how to do that. But before going that route try
to find the answer own your own by searching the internet; alternatively
consider asking in other places for advice. For example ask a friend or post
about it to a chatroom or forum you normally hang out.

**Be patient**: If you are really lucky you might get a reply to your report
within a few hours. But most of the time it will take longer, as maintainers
are scattered around the globe and thus might be in a different time zone – one
where they already enjoy their night away from keyboard.

In general, kernel developers will take one to five business days to respond to
reports. Sometimes it will take longer, as they might be busy with the merge
windows, other work, visiting developer conferences, or simply enjoying a long
summer holiday.

The 'issues of high priority' (see above for an explanation) are an exception
here: maintainers should address them as soon as possible; that's why you
should wait a week at maximum (or just two days if it's something urgent)
before sending a friendly reminder.

Sometimes the maintainer might not be responding in a timely manner; other
times there might be disagreements, for example if an issue qualifies as
regression or not. In such cases raise your concerns on the mailing list and
ask others for public or private replies how to move on. If that fails, it
might be appropriate to get a higher authority involved. In case of a WiFi
driver that would be the wireless maintainers; if there are no higher level
maintainers or all else fails, it might be one of those rare situations where
it's okay to get Linus Torvalds involved.

**Proactive testing**: Every time the first pre-release (the 'rc1') of a new
mainline kernel version gets released, go and check if the issue is fixed there
or if anything of importance changed. Mention the outcome in the ticket or in a
mail you sent as reply to your report (make sure it has all those in the CC
that up to that point participated in the discussion). This will show your
commitment and that you are willing to help. It also tells developers if the
issue persists and makes sure they do not forget about it. A few other
occasional retests (for example with rc3, rc5 and the final) are also a good
idea, but only report your results if something relevant changed or if you are
writing something anyway.

With all these general things off the table let's get into the details of how
to help to get issues resolved once they were reported.

Inquires and testing request
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Here are your duties in case you got replies to your report:

**Check who you deal with**: Most of the time it will be the maintainer or a
developer of the particular code area that will respond to your report. But as
issues are normally reported in public it could be anyone that's replying —
including people that want to help, but in the end might guide you totally off
track with their questions or requests. That rarely happens, but it's one of
many reasons why it's wise to quickly run an internet search to see who you're
interacting with. By doing this you also get aware if your report was heard by
the right people, as a reminder to the maintainer (see below) might be in order
later if discussion fades out without leading to a satisfying solution for the
issue.

**Inquiries for data**: Often you will be asked to test something or provide
additional details. Try to provide the requested information soon, as you have
the attention of someone that might help and risk losing it the longer you
wait; that outcome is even likely if you do not provide the information within
a few business days.

**Requests for testing**: When you are asked to test a diagnostic patch or a
possible fix, try to test it in timely manner, too. But do it properly and make
sure to not rush it: mixing things up can happen easily and can lead to a lot
of confusion for everyone involved. A common mistake for example is thinking a
proposed patch with a fix was applied, but in fact wasn't. Things like that
happen even to experienced testers occasionally, but they most of the time will
notice when the kernel with the fix behaves just as one without it.

What to do when nothing of substance happens
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Some reports will not get any reaction from the responsible Linux kernel
developers; or a discussion around the issue evolved, but faded out with
nothing of substance coming out of it.

In these cases wait two (better: three) weeks before sending a friendly
reminder: maybe the maintainer was just away from keyboard for a while when
your report arrived or had something more important to take care of. When
writing the reminder, kindly ask if anything else from your side is needed to
get the ball running somehow. If the report got out by mail, do that in the
first lines of a mail that is a reply to your initial mail (see above) which
includes a full quote of the original report below: that's on of those few
situations where such a 'TOFU' (Text Over, Fullquote Under) is the right
approach, as then all the recipients will have the details at hand immediately
in the proper order.

After the reminder wait three more weeks for replies. If you still don't get a
proper reaction, you first should reconsider your approach. Did you maybe try
to reach out to the wrong people? Was the report maybe offensive or so
confusing that people decided to completely stay away from it? The best way to
rule out such factors: show the report to one or two people familiar with FLOSS
issue reporting and ask for their opinion. Also ask them for their advice how
to move forward. That might mean: prepare a better report and make those people
review it before you send it out. Such an approach is totally fine; just
mention that this is the second and improved report on the issue and include a
link to the first report.

If the report was proper you can send a second reminder; in it ask for advice
why the report did not get any replies. A good moment for this second reminder
mail is shortly after the first pre-release (the 'rc1') of a new Linux kernel
version got published, as you should retest and provide a status update at that
point anyway (see above).

If the second reminder again results in no reaction within a week, try to
contact a higher-level maintainer asking for advice: even busy maintainers by
then should at least have sent some kind of acknowledgment.

Remember to prepare yourself for a disappointment: maintainers ideally should
react somehow to every issue report, but they are only obliged to fix those
'issues of high priority' outlined earlier. So don't be too devastating if you
get a reply along the lines of 'thanks for the report, I have more important
issues to deal with currently and won't have time to look into this for the
foreseeable future'.

It's also possible that after some discussion in the bug tracker or on a list
nothing happens anymore and reminders don't help to motivate anyone to work out
a fix. Such situations can be devastating, but is within the cards when it
comes to Linux kernel development. This and several other reasons for not
getting help are explained in 'Why some issues won't get any reaction or remain
unfixed after being reported' near the end of this document.

Don't get devastated if you don't find any help or if the issue in the end does
not get solved: the Linux kernel is FLOSS and thus you can still help yourself.
You for example could try to find others that are affected and team up with
them to get the issue resolved. Such a team could prepare a fresh report
together that mentions how many you are and why this is something that in your
option should get fixed. Maybe together you can also narrow down the root cause
or the change that introduced a regression, which often makes developing a fix
easier. And with a bit of luck there might be someone in the team that knows a
bit about programming and might be able to write a fix.

Reference for "Reporting regressions within a stable and longterm kernel line"
------------------------------------------------------------------------------

This subsection provides details for the steps you need to perform if you face
a regression within a stable and longterm kernel line.

Make sure the particular version line still gets support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    *Check if the kernel developers still maintain the Linux kernel version
    line you care about: go to the front page of kernel.org and make sure it
    mentions the latest release of the particular version line without an
    '[EOL]' tag.*

Most kernel version lines only get supported for about three months, as
maintaining them longer is quite a lot of work. Hence, only one per year is
chosen and gets supported for at least two years (often six). That's why you
need to check if the kernel developers still support the version line you care
for.

Note, if kernel.org lists two stable version lines on the front page, you
should consider switching to the newer one and forget about the older one:
support for it is likely to be abandoned soon. Then it will get a "end-of-life"
(EOL) stamp. Version lines that reached that point still get mentioned on the
kernel.org front page for a week or two, but are unsuitable for testing and
reporting.

Search stable mailing list
~~~~~~~~~~~~~~~~~~~~~~~~~~

    *Check the archives of the Linux stable mailing list for existing reports.*

Maybe the issue you face is already known and was fixed or is about to. Hence,
`search the archives of the Linux stable mailing list
<https://lore.kernel.org/stable/>`_ for reports about an issue like yours. If
you find any matches, consider joining the discussion, unless the fix is
already finished and scheduled to get applied soon.

Reproduce issue with the newest release
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    *Install the latest release from the particular version line as a vanilla
    kernel. Ensure this kernel is not tainted and still shows the problem, as
    the issue might have already been fixed there. If you first noticed the
    problem with a vendor kernel, check a vanilla build of the last version
    known to work performs fine as well.*

Before investing any more time in this process you want to check if the issue
was already fixed in the latest release of version line you're interested in.
This kernel needs to be vanilla and shouldn't be tainted before the issue
happens, as detailed outlined already above in the section "Install a fresh
kernel for testing".

Did you first notice the regression with a vendor kernel? Then changes the
vendor applied might be interfering. You need to rule that out by performing
a recheck. Say something broke when you updated from 5.10.4-vendor.42 to
5.10.5-vendor.43. Then after testing the latest 5.10 release as outlined in
the previous paragraph check if a vanilla build of Linux 5.10.4 works fine as
well. If things are broken there, the issue does not qualify as upstream
regression and you need switch back to the main step-by-step guide to report
the issue.

Report the regression
~~~~~~~~~~~~~~~~~~~~~

    *Send a short problem report to the Linux stable mailing list
    (stable@vger.kernel.org) and CC the Linux regressions mailing list
    (regressions@lists.linux.dev); if you suspect the cause in a particular
    subsystem, CC its maintainer and its mailing list. Roughly describe the
    issue and ideally explain how to reproduce it. Mention the first version
    that shows the problem and the last version that's working fine. Then
    wait for further instructions.*

When reporting a regression that happens within a stable or longterm kernel
line (say when updating from 5.10.4 to 5.10.5) a brief report is enough for
the start to get the issue reported quickly. Hence a rough description to the
stable and regressions mailing list is all it takes; but in case you suspect
the cause in a particular subsystem, CC its maintainers and its mailing list
as well, because that will speed things up.

And note, it helps developers a great deal if you can specify the exact version
that introduced the problem. Hence if possible within a reasonable time frame,
try to find that version using vanilla kernels. Lets assume something broke when
your distributor released a update from Linux kernel 5.10.5 to 5.10.8. Then as
instructed above go and check the latest kernel from that version line, say
5.10.9. If it shows the problem, try a vanilla 5.10.5 to ensure that no patches
the distributor applied interfere. If the issue doesn't manifest itself there,
try 5.10.7 and then (depending on the outcome) 5.10.8 or 5.10.6 to find the
first version where things broke. Mention it in the report and state that 5.10.9
is still broken.

What the previous paragraph outlines is basically a rough manual 'bisection'.
Once your report is out your might get asked to do a proper one, as it allows to
pinpoint the exact change that causes the issue (which then can easily get
reverted to fix the issue quickly). Hence consider to do a proper bisection
right away if time permits. See the section 'Special care for regressions' and
the document Documentation/admin-guide/bug-bisect.rst for details how to
perform one. In case of a successful bisection add the author of the culprit to
the recipients; also CC everyone in the signed-off-by chain, which you find at
the end of its commit message.

Reference for "Reporting issues only occurring in older kernel version lines"
-----------------------------------------------------------------------------

This section provides details for the steps you need to take if you could not
reproduce your issue with a mainline kernel, but want to see it fixed in older
version lines (aka stable and longterm kernels).

Some fixes are too complex
~~~~~~~~~~~~~~~~~~~~~~~~~~

    *Prepare yourself for the possibility that going through the next few steps
    might not get the issue solved in older releases: the fix might be too big
    or risky to get backported there.*

Even small and seemingly obvious code-changes sometimes introduce new and
totally unexpected problems. The maintainers of the stable and longterm kernels
are very aware of that and thus only apply changes to these kernels that are
within rules outlined in Documentation/process/stable-kernel-rules.rst.

Complex or risky changes for example do not qualify and thus only get applied
to mainline. Other fixes are easy to get backported to the newest stable and
longterm kernels, but too risky to integrate into older ones. So be aware the
fix you are hoping for might be one of those that won't be backported to the
version line your care about. In that case you'll have no other choice then to
live with the issue or switch to a newer Linux version, unless you want to
patch the fix into your kernels yourself.

Common preparations
~~~~~~~~~~~~~~~~~~~

    *Perform the first three steps in the section "Reporting issues only
    occurring in older kernel version lines" above.*

You need to carry out a few steps already described in another section of this
guide. Those steps will let you:

 * Check if the kernel developers still maintain the Linux kernel version line
   you care about.

 * Search the Linux stable mailing list for exiting reports.

 * Check with the latest release.

Check code history and search for existing discussions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    *Search the Linux kernel version control system for the change that fixed
    the issue in mainline, as its commit message might tell you if the fix is
    scheduled for backporting already. If you don't find anything that way,
    search the appropriate mailing lists for posts that discuss such an issue
    or peer-review possible fixes; then check the discussions if the fix was
    deemed unsuitable for backporting. If backporting was not considered at
    all, join the newest discussion, asking if it's in the cards.*

In a lot of cases the issue you deal with will have happened with mainline, but
got fixed there. The commit that fixed it would need to get backported as well
to get the issue solved. That's why you want to search for it or any
discussions abound it.

 * First try to find the fix in the Git repository that holds the Linux kernel
   sources. You can do this with the web interfaces `on kernel.org
   <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/>`_
   or its mirror `on GitHub <https://github.com/torvalds/linux>`_; if you have
   a local clone you alternatively can search on the command line with ``git
   log --grep=<pattern>``.

   If you find the fix, look if the commit message near the end contains a
   'stable tag' that looks like this:

          Cc: <stable@vger.kernel.org> # 5.4+

   If that's case the developer marked the fix safe for backporting to version
   line 5.4 and later. Most of the time it's getting applied there within two
   weeks, but sometimes it takes a bit longer.

 * If the commit doesn't tell you anything or if you can't find the fix, look
   again for discussions about the issue. Search the net with your favorite
   internet search engine as well as the archives for the `Linux kernel
   developers mailing list <https://lore.kernel.org/lkml/>`_. Also read the
   section `Locate kernel area that causes the issue` above and follow the
   instructions to find the subsystem in question: its bug tracker or mailing
   list archive might have the answer you are looking for.

 * If you see a proposed fix, search for it in the version control system as
   outlined above, as the commit might tell you if a backport can be expected.

   * Check the discussions for any indicators the fix might be too risky to get
     backported to the version line you care about. If that's the case you have
     to live with the issue or switch to the kernel version line where the fix
     got applied.

   * If the fix doesn't contain a stable tag and backporting was not discussed,
     join the discussion: mention the version where you face the issue and that
     you would like to see it fixed, if suitable.

Ask for advice
~~~~~~~~~~~~~~

    *One of the former steps should lead to a solution. If that doesn't work
    out, ask the maintainers for the subsystem that seems to be causing the
    issue for advice; CC the mailing list for the particular subsystem as well
    as the stable mailing list.*

If the previous three steps didn't get you closer to a solution there is only
one option left: ask for advice. Do that in a mail you sent to the maintainers
for the subsystem where the issue seems to have its roots; CC the mailing list
for the subsystem as well as the stable mailing list (stable@vger.kernel.org).

Why some issues won't get any reaction or remain unfixed after being reported
=============================================================================

When reporting a problem to the Linux developers, be aware only 'issues of high
priority' (regressions, security issues, severe problems) are definitely going
to get resolved. The maintainers or if all else fails Linus Torvalds himself
will make sure of that. They and the other kernel developers will fix a lot of
other issues as well. But be aware that sometimes they can't or won't help; and
sometimes there isn't even anyone to send a report to.

This is best explained with kernel developers that contribute to the Linux
kernel in their spare time. Quite a few of the drivers in the kernel were
written by such programmers, often because they simply wanted to make their
hardware usable on their favorite operating system.

These programmers most of the time will happily fix problems other people
report. But nobody can force them to do, as they are contributing voluntarily.

Then there are situations where such developers really want to fix an issue,
but can't: sometimes they lack hardware programming documentation to do so.
This often happens when the publicly available docs are superficial or the
driver was written with the help of reverse engineering.

Sooner or later spare time developers will also stop caring for the driver.
Maybe their test hardware broke, got replaced by something more fancy, or is so
old that it's something you don't find much outside of computer museums
anymore. Sometimes developer stops caring for their code and Linux at all, as
something different in their life became way more important. In some cases
nobody is willing to take over the job as maintainer – and nobody can be forced
to, as contributing to the Linux kernel is done on a voluntary basis. Abandoned
drivers nevertheless remain in the kernel: they are still useful for people and
removing would be a regression.

The situation is not that different with developers that are paid for their
work on the Linux kernel. Those contribute most changes these days. But their
employers sooner or later also stop caring for their code or make its
programmer focus on other things. Hardware vendors for example earn their money
mainly by selling new hardware; quite a few of them hence are not investing
much time and energy in maintaining a Linux kernel driver for something they
stopped selling years ago. Enterprise Linux distributors often care for a
longer time period, but in new versions often leave support for old and rare
hardware aside to limit the scope. Often spare time contributors take over once
a company orphans some code, but as mentioned above: sooner or later they will
leave the code behind, too.

Priorities are another reason why some issues are not fixed, as maintainers
quite often are forced to set those, as time to work on Linux is limited.
That's true for spare time or the time employers grant their developers to
spend on maintenance work on the upstream kernel. Sometimes maintainers also
get overwhelmed with reports, even if a driver is working nearly perfectly. To
not get completely stuck, the programmer thus might have no other choice than
to prioritize issue reports and reject some of them.

But don't worry too much about all of this, a lot of drivers have active
maintainers who are quite interested in fixing as many issues as possible.

Closing words
=============

Compared with other Free/Libre & Open Source Software it's hard to report
issues to the Linux kernel developers: the length and complexity of this
document and the implications between the lines illustrate that. But that's how
it is for now. The main author of this text hopes documenting the state of the
art will lay some groundwork to improve the situation over time.

..
   end-of-content
..
   This document is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If
   you spot a typo or small mistake, feel free to let him know directly and
   he'll fix it. You are free to do the same in a mostly informal way if you
   want to contribute changes to the text, but for copyright reasons please CC
   linux-doc@vger.kernel.org and "sign-off" your contribution as
   Documentation/process/submitting-patches.rst outlines in the section "Sign
   your work - the Developer's Certificate of Origin".
..
   This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
   of the file. If you want to distribute this text under CC-BY-4.0 only,
   please use "The Linux kernel developers" for author attribution and link
   this as source:
   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/reporting-issues.rst
..
   Note: Only the content of this RST file as found in the Linux kernel sources
   is available under CC-BY-4.0, as versions of this text that were processed
   (for example by the kernel's build system) might contain content taken from
   files which use a more restrictive license.

SPDX License Identifier   GPL 2 0  OR CC BY 4 0     See the bottom of this file for additional redistribution information   Reporting issues                    The short guide  aka TL DR                               Are you facing a regression with vanilla kernels from the same stable or longterm series  One still supported  Then search the  LKML  https   lore kernel org lkml     and the  Linux stable mailing list  https   lore kernel org stable     archives for matching reports to join  If you don t find any  install  the latest release from that series  https   kernel org      If it still shows the issue  report it to the stable mailing list  stable vger kernel org  and CC the regressions list  regressions lists linux dev   ideally also CC the maintainer and the mailing list for the subsystem in question   In all other cases try your best guess which kernel part might be causing the issue  Check the  ref  MAINTAINERS  maintainers   file for how its developers expect to be told about problems  which most of the time will be by email with a mailing list in CC  Check the destination s archives for matching reports  search the  LKML  https   lore kernel org lkml     and the web  too  If you don t find any to join  install  the latest mainline kernel  https   kernel org      If the issue is present there  send a report   The issue was fixed there  but you would like to see it resolved in a still supported stable or longterm series as well  Then install its latest release  If it shows the problem  search for the change that fixed it in mainline and check if backporting is in the works or was discarded  if it s neither  ask those who handled the change for it     General remarks    When installing and testing a kernel as outlined above  ensure it s vanilla  IOW  not patched and not using add on modules   Also make sure it s built and running in a healthy environment and not already tainted before the issue occurs   If you are facing multiple issues with the Linux kernel at once  report each separately  While writing your report  include all information relevant to the issue  like the kernel and the distro used  In case of a regression  CC the regressions mailing list  regressions lists linux dev  to your report  Also try to pin point the culprit with a bisection  if you succeed  include its commit id and CC everyone in the sign off by chain   Once the report is out  answer any questions that come up and help where you can  That includes keeping the ball rolling by occasionally retesting with newer releases and sending a status update afterwards   Step by step guide how to report issues to the kernel maintainers                                                                    The above TL DR outlines roughly how to report issues to the Linux kernel developers  It might be all that s needed for people already familiar with reporting issues to Free Libre   Open Source Software  FLOSS  projects  For everyone else there is this section  It is more detailed and uses a step by step approach  It still tries to be brief for readability and leaves out a lot of details  those are described below the step by step guide in a reference section  which explains each of the steps in more detail   Note  this section covers a few more aspects than the TL DR and does things in a slightly different order  That s in your interest  to make sure you notice early if an issue that looks like a Linux kernel problem is actually caused by something else  These steps thus help to ensure the time you invest in this process won t feel wasted in the end      Are you facing an issue with a Linux kernel a hardware or software vendor    provided  Then in almost all cases you are better off to stop reading this    document and reporting the issue to your vendor instead  unless you are    willing to install the latest Linux version yourself  Be aware the latter    will often be needed anyway to hunt down and fix issues      Perform a rough search for existing reports with your favorite internet    search engine  additionally  check the archives of the  Linux Kernel Mailing    List  LKML   https   lore kernel org lkml      If you find matching reports     join the discussion instead of sending a new one      See if the issue you are dealing with qualifies as regression  security    issue  or a really severe problem  those are  issues of high priority  that    need special handling in some steps that are about to follow      Make sure it s not the kernel s surroundings that are causing the issue    you face      Create a fresh backup and put system repair and restore tools at hand      Ensure your system does not enhance its kernels by building additional    kernel modules on the fly  which solutions like DKMS might be doing locally    without your knowledge      Check if your kernel was  tainted  when the issue occurred  as the event    that made the kernel set this flag might be causing the issue you face      Write down coarsely how to reproduce the issue  If you deal with multiple    issues at once  create separate notes for each of them and make sure they    work independently on a freshly booted system  That s needed  as each issue    needs to get reported to the kernel developers separately  unless they are    strongly entangled      If you are facing a regression within a stable or longterm version line     say something broke when updating from 5 10 4 to 5 10 5   scroll down to     Dealing with regressions within a stable and longterm kernel line       Locate the driver or kernel subsystem that seems to be causing the issue     Find out how and where its developers expect reports  Note  most of the    time this won t be bugzilla kernel org  as issues typically need to be sent    by mail to a maintainer and a public mailing list      Search the archives of the bug tracker or mailing list in question    thoroughly for reports that might match your issue  If you find anything     join the discussion instead of sending a new report   After these preparations you ll now enter the main part      Unless you are already running the latest  mainline  Linux kernel  better    go and install it for the reporting process  Testing and reporting with    the latest  stable  Linux can be an acceptable alternative in some    situations  during the merge window that actually might be even the best    approach  but in that development phase it can be an even better idea to    suspend your efforts for a few days anyway  Whatever version you choose     ideally use a  vanilla  build  Ignoring these advices will dramatically    increase the risk your report will be rejected or ignored      Ensure the kernel you just installed does not  taint  itself when    running      Reproduce the issue with the kernel you just installed  If it doesn t show    up there  scroll down to the instructions for issues only happening with    stable and longterm kernels      Optimize your notes  try to find and write the most straightforward way to    reproduce your issue  Make sure the end result has all the important    details  and at the same time is easy to read and understand for others    that hear about it for the first time  And if you learned something in this    process  consider searching again for existing reports about the issue      If your failure involves a  panic    Oops    warning   or  BUG   consider    decoding the kernel log to find the line of code that triggered the error      If your problem is a regression  try to narrow down when the issue was    introduced as much as possible      Start to compile the report by writing a detailed description about the    issue  Always mention a few things  the latest kernel version you installed    for reproducing  the Linux Distribution used  and your notes on how to    reproduce the issue  Ideally  make the kernel s build configuration      config  and the output from   dmesg   available somewhere on the net and    link to it  Include or upload all other information that might be relevant     like the output screenshot of an Oops or the output from   lspci    Once    you wrote this main part  insert a normal length paragraph on top of it    outlining the issue and the impact quickly  On top of this add one sentence    that briefly describes the problem and gets people to read on  Now give the    thing a descriptive title or subject that yet again is shorter  Then you re    ready to send or file the report like the MAINTAINERS file told you  unless    you are dealing with one of those  issues of high priority   they need    special care which is explained in  Special handling for high priority    issues  below      Wait for reactions and keep the thing rolling until you can accept the    outcome in one way or the other  Thus react publicly and in a timely manner    to any inquiries  Test proposed fixes  Do proactive testing  retest with at    least every first release candidate  RC  of a new mainline version and    report your results  Send friendly reminders if things stall  And try to    help yourself  if you don t get any help or if it s unsatisfying    Reporting regressions within a stable and longterm kernel line                                                                 This subsection is for you  if you followed above process and got sent here at the point about regression within a stable or longterm kernel version line  You face one of those if something breaks when updating from 5 10 4 to 5 10 5  a switch from 5 9 15 to 5 10 5 does not qualify   The developers want to fix such regressions as quickly as possible  hence there is a streamlined process to report them      Check if the kernel developers still maintain the Linux kernel version    line you care about  go to the   front page of kernel org     https   kernel org     and make sure it mentions    the latest release of the particular version line without an   EOL   tag      Check the archives of the  Linux stable mailing list     https   lore kernel org stable     for existing reports      Install the latest release from the particular version line as a vanilla    kernel  Ensure this kernel is not tainted and still shows the problem  as    the issue might have already been fixed there  If you first noticed the    problem with a vendor kernel  check a vanilla build of the last version    known to work performs fine as well      Send a short problem report to the Linux stable mailing list     stable vger kernel org  and CC the Linux regressions mailing list     regressions lists linux dev   if you suspect the cause in a particular    subsystem  CC its maintainer and its mailing list  Roughly describe the    issue and ideally explain how to reproduce it  Mention the first version    that shows the problem and the last version that s working fine  Then    wait for further instructions   The reference section below explains each of these steps in more detail    Reporting issues only occurring in older kernel version lines                                                                This subsection is for you  if you tried the latest mainline kernel as outlined above  but failed to reproduce your issue there  at the same time you want to see the issue fixed in a still supported stable or longterm series or vendor kernels regularly rebased on those  If that the case  follow these steps      Prepare yourself for the possibility that going through the next few steps    might not get the issue solved in older releases  the fix might be too big    or risky to get backported there      Perform the first three steps in the section  Dealing with regressions    within a stable and longterm kernel line  above      Search the Linux kernel version control system for the change that fixed    the issue in mainline  as its commit message might tell you if the fix is    scheduled for backporting already  If you don t find anything that way     search the appropriate mailing lists for posts that discuss such an issue    or peer review possible fixes  then check the discussions if the fix was    deemed unsuitable for backporting  If backporting was not considered at    all  join the newest discussion  asking if it s in the cards      One of the former steps should lead to a solution  If that doesn t work    out  ask the maintainers for the subsystem that seems to be causing the    issue for advice  CC the mailing list for the particular subsystem as well    as the stable mailing list   The reference section below explains each of these steps in more detail    Reference section  Reporting issues to the kernel maintainers                                                                The detailed guides above outline all the major steps in brief fashion  which should be enough for most people  But sometimes there are situations where even experienced users might wonder how to actually do one of those steps  That s what this section is for  as it will provide a lot more details on each of the above steps  Consider this as reference documentation  it s possible to read it from top to bottom  But it s mainly meant to skim over and a place to look up details how to actually perform those steps   A few words of general advice before digging into the details      The Linux kernel developers are well aware this process is complicated and    demands more than other FLOSS projects  We d love to make it simpler  But    that would require work in various places as well as some infrastructure     which would need constant maintenance  nobody has stepped up to do that    work  so that s just how things are for now      A warranty or support contract with some vendor doesn t entitle you to    request fixes from developers in the upstream Linux kernel community  such    contracts are completely outside the scope of the Linux kernel  its    development community  and this document  That s why you can t demand    anything such a contract guarantees in this context  not even if the    developer handling the issue works for the vendor in question  If you want    to claim your rights  use the vendor s support channel instead  When doing    so  you might want to mention you d like to see the issue fixed in the    upstream Linux kernel  motivate them by saying it s the only way to ensure    the fix in the end will get incorporated in all Linux distributions      If you never reported an issue to a FLOSS project before you should consider    reading  How to Report Bugs Effectively     https   www chiark greenend org uk  sgtatham bugs html      How To Ask    Questions The Smart Way     http   www catb org esr faqs smart questions html     and  How to ask good    questions  https   jvns ca blog good questions       With that off the table  find below the details on how to properly report issues to the Linux kernel developers    Make sure you re using the upstream Linux kernel                                                       Are you facing an issue with a Linux kernel a hardware or software vendor    provided  Then in almost all cases you are better off to stop reading this    document and reporting the issue to your vendor instead  unless you are    willing to install the latest Linux version yourself  Be aware the latter    will often be needed anyway to hunt down and fix issues    Like most programmers  Linux kernel developers don t like to spend time dealing with reports for issues that don t even happen with their current code  It s just a waste everybody s time  especially yours  Unfortunately such situations easily happen when it comes to the kernel and often leads to frustration on both sides  That s because almost all Linux based kernels pre installed on devices  Computers  Laptops  Smartphones  Routers     and most shipped by Linux distributors are quite distant from the official Linux kernel as distributed by kernel org  these kernels from these vendors are often ancient from the point of Linux development or heavily modified  often both   Most of these vendor kernels are quite unsuitable for reporting issues to the Linux kernel developers  an issue you face with one of them might have been fixed by the Linux kernel developers months or years ago already  additionally  the modifications and enhancements by the vendor might be causing the issue you face  even if they look small or totally unrelated  That s why you should report issues with these kernels to the vendor  Its developers should look into the report and  in case it turns out to be an upstream issue  fix it directly upstream or forward the report there  In practice that often does not work out or might not what you want  You thus might want to consider circumventing the vendor by installing the very latest Linux kernel core yourself  If that s an option for you move ahead in this process  as a later step in this guide will explain how to do that once it rules out other potential causes for your issue   Note  the previous paragraph is starting with the word  most   as sometimes developers in fact are willing to handle reports about issues occurring with vendor kernels  If they do in the end highly depends on the developers and the issue in question  Your chances are quite good if the distributor applied only small modifications to a kernel based on a recent Linux version  that for example often holds true for the mainline kernels shipped by Debian GNU Linux Sid or Fedora Rawhide  Some developers will also accept reports about issues with kernels from distributions shipping the latest stable kernel  as long as its only slightly modified  that for example is often the case for Arch Linux  regular Fedora releases  and openSUSE Tumbleweed  But keep in mind  you better want to use a mainline Linux and avoid using a stable kernel for this process  as outlined in the section  Install a fresh kernel for testing  in more detail   Obviously you are free to ignore all this advice and report problems with an old or heavily modified vendor kernel to the upstream Linux developers  But note  those often get rejected or ignored  so consider yourself warned  But it s still better than not reporting the issue at all  sometimes such reports directly or indirectly will help to get the issue fixed over time    Search for existing reports  first run                                             Perform a rough search for existing reports with your favorite internet    search engine  additionally  check the archives of the Linux Kernel Mailing    List  LKML   If you find matching reports  join the discussion instead of    sending a new one    Reporting an issue that someone else already brought forward is often a waste of time for everyone involved  especially you as the reporter  So it s in your own interest to thoroughly check if somebody reported the issue already  At this step of the process it s okay to just perform a rough search  a later step will tell you to perform a more detailed search once you know where your issue needs to be reported to  Nevertheless  do not hurry with this step of the reporting process  it can save you time and trouble   Simply search the internet with your favorite search engine first  Afterwards  search the  Linux Kernel Mailing List  LKML  archives  https   lore kernel org lkml       If you get flooded with results consider telling your search engine to limit search timeframe to the past month or year  And wherever you search  make sure to use good search terms  vary them a few times  too  While doing so try to look at the issue from the perspective of someone else  that will help you to come up with other words to use as search terms  Also make sure not to use too many search terms at once  Remember to search with and without information like the name of the kernel driver or the name of the affected hardware component  But its exact brand name  say  ASUS Red Devil Radeon RX 5700 XT Gaming OC   often is not much helpful  as it is too specific  Instead try search terms like the model line  Radeon 5700 or Radeon 5000  and the code name of the main chip   Navi  or  Navi10   with and without its manufacturer   AMD     In case you find an existing report about your issue  join the discussion  as you might be able to provide valuable additional information  That can be important even when a fix is prepared or in its final stages already  as developers might look for people that can provide additional information or test a proposed fix  Jump to the section  Duties after the report went out  for details on how to get properly involved   Note  searching  bugzilla kernel org  https   bugzilla kernel org     might also be a good idea  as that might provide valuable insights or turn up matching reports  If you find the latter  just keep in mind  most subsystems expect reports in different places  as described below in the section  Check where you need to report your issue   The developers that should take care of the issue thus might not even be aware of the bugzilla ticket  Hence  check the ticket if the issue already got reported as outlined in this document and if not consider doing so    Issue of high priority                                See if the issue you are dealing with qualifies as regression  security     issue  or a really severe problem  those are  issues of high priority  that     need special handling in some steps that are about to follow    Linus Torvalds and the leading Linux kernel developers want to see some issues fixed as soon as possible  hence there are  issues of high priority  that get handled slightly differently in the reporting process  Three type of cases qualify  regressions  security issues  and really severe problems   You deal with a regression if some application or practical use case running fine with one Linux kernel works worse or not at all with a newer version compiled using a similar configuration  The document Documentation admin guide reporting regressions rst explains this in more detail  It also provides a good deal of other information about regressions you might want to be aware of  it for example explains how to add your issue to the list of tracked regressions  to ensure it won t fall through the cracks   What qualifies as security issue is left to your judgment  Consider reading Documentation process security bugs rst before proceeding  as it provides additional details how to best handle security issues   An issue is a  really severe problem  when something totally unacceptably bad happens  That s for example the case when a Linux kernel corrupts the data it s handling or damages hardware it s running on  You re also dealing with a severe issue when the kernel suddenly stops working with an error message   kernel panic   or without any farewell note at all  Note  do not confuse a  panic   a fatal error where the kernel stop itself  with a  Oops   a recoverable error   as the kernel remains running after the latter    Ensure a healthy environment                                    Make sure it s not the kernel s surroundings that are causing the issue     you face    Problems that look a lot like a kernel issue are sometimes caused by build or runtime environment  It s hard to rule out that problem completely  but you should minimize it      Use proven tools when building your kernel  as bugs in the compiler or the    binutils can cause the resulting kernel to misbehave      Ensure your computer components run within their design specifications     that s especially important for the main processor  the main memory  and the    motherboard  Therefore  stop undervolting or overclocking when facing a    potential kernel issue      Try to make sure it s not faulty hardware that is causing your issue  Bad    main memory for example can result in a multitude of issues that will    manifest itself in problems looking like kernel issues      If you re dealing with a filesystem issue  you might want to check the file    system in question with   fsck    as it might be damaged in a way that leads    to unexpected kernel behavior      When dealing with a regression  make sure it s not something else that    changed in parallel to updating the kernel  The problem for example might be    caused by other software that was updated at the same time  It can also    happen that a hardware component coincidentally just broke when you rebooted    into a new kernel for the first time  Updating the systems BIOS or changing    something in the BIOS Setup can also lead to problems that on look a lot    like a kernel regression    Prepare for emergencies                               Create a fresh backup and put system repair and restore tools at hand    Reminder  you are dealing with computers  which sometimes do unexpected things  especially if you fiddle with crucial parts like the kernel of its operating system  That s what you are about to do in this process  Thus  make sure to create a fresh backup  also ensure you have all tools at hand to repair or reinstall the operating system as well as everything you need to restore the backup    Make sure your kernel doesn t get enhanced                                                  Ensure your system does not enhance its kernels by building additional     kernel modules on the fly  which solutions like DKMS might be doing locally     without your knowledge    The risk your issue report gets ignored or rejected dramatically increases if your kernel gets enhanced in any way  That s why you should remove or disable mechanisms like akmods and DKMS  those build add on kernel modules automatically  for example when you install a new Linux kernel or boot it for the first time  Also remove any modules they might have installed  Then reboot before proceeding   Note  you might not be aware that your system is using one of these solutions  they often get set up silently when you install Nvidia s proprietary graphics driver  VirtualBox  or other software that requires a some support from a module not part of the Linux kernel  That why your might need to uninstall the packages with such software to get rid of any 3rd party kernel module    Check  taint  flag                          Check if your kernel was  tainted  when the issue occurred  as the event     that made the kernel set this flag might be causing the issue you face    The kernel marks itself with a  taint  flag when something happens that might lead to follow up errors that look totally unrelated  The issue you face might be such an error if your kernel is tainted  That s why it s in your interest to rule this out early before investing more time into this process  This is the only reason why this step is here  as this process later will tell you to install the latest mainline kernel  you will need to check the taint flag again then  as that s when it matters because it s the kernel the report will focus on   On a running system is easy to check if the kernel tainted itself  if   cat  proc sys kernel tainted   returns  0  then the kernel is not tainted and everything is fine  Checking that file is impossible in some situations  that s why the kernel also mentions the taint status when it reports an internal problem  a  kernel bug    a recoverable error  a  kernel Oops   or a non recoverable error before halting operation  a  kernel panic    Look near the top of the error messages printed when one of these occurs and search for a line starting with  CPU    It should end with  Not tainted  if the kernel was not tainted when it noticed the problem  it was tainted if you see  Tainted   followed by a few spaces and some letters   If your kernel is tainted  study Documentation admin guide tainted kernels rst to find out why  Try to eliminate the reason  Often it s caused by one these three things    1  A recoverable error  a  kernel Oops   occurred and the kernel tainted     itself  as the kernel knows it might misbehave in strange ways after that     point  In that case check your kernel or system log and look for a section     that starts with this           Oops  0000   1  SMP      That s the first Oops since boot up  as the   1  between the brackets shows      Every Oops and any other problem that happens after that point might be a     follow up problem to that first Oops  even if both look totally unrelated      Rule this out by getting rid of the cause for the first Oops and reproducing     the issue afterwards  Sometimes simply restarting will be enough  sometimes     a change to the configuration followed by a reboot can eliminate the Oops      But don t invest too much time into this at this point of the process  as     the cause for the Oops might already be fixed in the newer Linux kernel     version you are going to install later in this process    2  Your system uses a software that installs its own kernel modules  for     example Nvidia s proprietary graphics driver or VirtualBox  The kernel     taints itself when it loads such module from external sources  even if     they are Open Source   they sometimes cause errors in unrelated kernel     areas and thus might be causing the issue you face  You therefore have to     prevent those modules from loading when you want to report an issue to the     Linux kernel developers  Most of the time the easiest way to do that is      temporarily uninstall such software including any modules they might have     installed  Afterwards reboot    3  The kernel also taints itself when it s loading a module that resides in     the staging tree of the Linux kernel source  That s a special area for     code  mostly drivers  that does not yet fulfill the normal Linux kernel     quality standards  When you report an issue with such a module it s     obviously okay if the kernel is tainted  just make sure the module in     question is the only reason for the taint  If the issue happens in an     unrelated area reboot and temporarily block the module from being loaded     by specifying   foo blacklist 1   as kernel parameter  replace  foo  with     the name of the module in question     Document how to reproduce issue                                       Write down coarsely how to reproduce the issue  If you deal with multiple     issues at once  create separate notes for each of them and make sure they     work independently on a freshly booted system  That s needed  as each issue     needs to get reported to the kernel developers separately  unless they are     strongly entangled    If you deal with multiple issues at once  you ll have to report each of them separately  as they might be handled by different developers  Describing various issues in one report also makes it quite difficult for others to tear it apart  Hence  only combine issues in one report if they are very strongly entangled   Additionally  during the reporting process you will have to test if the issue happens with other kernel versions  Therefore  it will make your work easier if you know exactly how to reproduce an issue quickly on a freshly booted system   Note  it s often fruitless to report issues that only happened once  as they might be caused by a bit flip due to cosmic radiation  That s why you should try to rule that out by reproducing the issue before going further  Feel free to ignore this advice if you are experienced enough to tell a one time error due to faulty hardware apart from a kernel issue that rarely happens and thus is hard to reproduce    Regression in stable or longterm kernel                                                 If you are facing a regression within a stable or longterm version line      say something broke when updating from 5 10 4 to 5 10 5   scroll down to      Dealing with regressions within a stable and longterm kernel line     Regression within a stable and longterm kernel version line are something the Linux developers want to fix badly  as such issues are even more unwanted than regression in the main development branch  as they can quickly affect a lot of people  The developers thus want to learn about such issues as quickly as possible  hence there is a streamlined process to report them  Note  regressions with newer kernel version line  say something broke when switching from 5 9 15 to 5 10 5  do not qualify    Check where you need to report your issue                                                 Locate the driver or kernel subsystem that seems to be causing the issue      Find out how and where its developers expect reports  Note  most of the     time this won t be bugzilla kernel org  as issues typically need to be sent     by mail to a maintainer and a public mailing list    It s crucial to send your report to the right people  as the Linux kernel is a big project and most of its developers are only familiar with a small subset of it  Quite a few programmers for example only care for just one driver  for example one for a WiFi chip  its developer likely will only have small or no knowledge about the internals of remote or unrelated  subsystems   like the TCP stack  the PCIe PCI subsystem  memory management or file systems   Problem is  the Linux kernel lacks a central bug tracker where you can simply file your issue and make it reach the developers that need to know about it  That s why you have to find the right place and way to report issues yourself  You can do that with the help of a script  see below   but it mainly targets kernel developers and experts  For everybody else the MAINTAINERS file is the better place   How to read the MAINTAINERS file                                  To illustrate how to use the  ref  MAINTAINERS  maintainers   file  lets assume the WiFi in your Laptop suddenly misbehaves after updating the kernel  In that case it s likely an issue in the WiFi driver  Obviously it could also be some code it builds upon  but unless you suspect something like that stick to the driver  If it s really something else  the driver s developers will get the right people involved   Sadly  there is no way to check which code is driving a particular hardware component that is both universal and easy   In case of a problem with the WiFi driver you for example might want to look at the output of   lspci  k    as it lists devices on the PCI PCIe bus and the kernel module driving it            user something     lspci  k                     3a 00 0 Network controller  Qualcomm Atheros QCA6174 802 11ac Wireless Network Adapter  rev 32           Subsystem  Bigfoot Networks  Inc  Device 1535          Kernel driver in use  ath10k pci          Kernel modules  ath10k pci               But this approach won t work if your WiFi chip is connected over USB or some other internal bus  In those cases you might want to check your WiFi manager or the output of   ip link    Look for the name of the problematic network interface  which might be something like  wlp58s0   This name can be used like this to find the module driving it            user something     realpath   relative to  sys module   sys class net wlp58s0 device driver module        ath10k pci  In case tricks like these don t bring you any further  try to search the internet on how to narrow down the driver or subsystem in question  And if you are unsure which it is  just try your best guess  somebody will help you if you guessed poorly   Once you know the driver or subsystem  you want to search for it in the MAINTAINERS file  In the case of  ath10k pci  you won t find anything  as the name is too specific  Sometimes you will need to search on the net for help  but before doing so  try a somewhat shorted or modified name when searching the MAINTAINERS file  as then you might find something like this           QUALCOMM ATHEROS ATH10K WIRELESS DRIVER        Mail           A  Some Human  shuman example com         Mailing list   ath10k lists infradead org        Status         Supported        Web page       https   wireless wiki kernel org en users Drivers ath10k        SCM            git git   git kernel org pub scm linux kernel git kvalo ath git        Files          drivers net wireless ath ath10k   Note  the line description will be abbreviations  if you read the plain MAINTAINERS file found in the root of the Linux source tree   Mail   for example will be  M     Mailing list   will be  L   and  Status   will be  S    A section near the top of the file explains these and other abbreviations   First look at the line  Status   Ideally it should be  Supported  or  Maintained   If it states  Obsolete  then you are using some outdated approach that was replaced by a newer solution you need to switch to  Sometimes the code only has someone who provides  Odd Fixes  when feeling motivated  And with  Orphan  you are totally out of luck  as nobody takes care of the code anymore  That only leaves these options  arrange yourself to live with the issue  fix it yourself  or find a programmer somewhere willing to fix it   After checking the status  look for a line starting with  bugs    it will tell you where to find a subsystem specific bug tracker to file your issue  The example above does not have such a line  That is the case for most sections  as Linux kernel development is completely driven by mail  Very few subsystems use a bug tracker  and only some of those rely on bugzilla kernel org   In this and many other cases you thus have to look for lines starting with  Mail   instead  Those mention the name and the email addresses for the maintainers of the particular code  Also look for a line starting with  Mailing list    which tells you the public mailing list where the code is developed  Your report later needs to go by mail to those addresses  Additionally  for all issue reports sent by email  make sure to add the Linux Kernel Mailing List  LKML   linux kernel vger kernel org  to CC  Don t omit either of the mailing lists when sending your issue report by mail later  Maintainers are busy people and might leave some work for other developers on the subsystem specific list  and LKML is important to have one place where all issue reports can be found    Finding the maintainers with the help of a script                                                    For people that have the Linux sources at hand there is a second option to find the proper place to report  the script  scripts get maintainer pl  which tries to find all people to contact  It queries the MAINTAINERS file and needs to be called with a path to the source code in question  For drivers compiled as module if often can be found with a command like this             modinfo ath10k pci   grep filename   sed  s  lib modules    kernel     s filename     s   ko      xz             drivers net wireless ath ath10k ath10k pci ko  Pass parts of this to the script               scripts get maintainer pl  f drivers net wireless ath ath10k         Some Human  shuman example com   supporter QUALCOMM ATHEROS ATH10K WIRELESS DRIVER         Another S  Human  asomehuman example com   maintainer NETWORKING DRIVERS         ath10k lists infradead org  open list QUALCOMM ATHEROS ATH10K WIRELESS DRIVER         linux wireless vger kernel org  open list NETWORKING DRIVERS  WIRELESS          netdev vger kernel org  open list NETWORKING DRIVERS         linux kernel vger kernel org  open list   Don t sent your report to all of them  Send it to the maintainers  which the script calls  supporter    additionally CC the most specific mailing list for the code as well as the Linux Kernel Mailing List  LKML   In this case you thus would need to send the report to  Some Human  shuman example com   with  ath10k lists infradead org  and  linux kernel vger kernel org  in CC   Note  in case you cloned the Linux sources with git you might want to call   get maintainer pl   a second time with     git    The script then will look at the commit history to find which people recently worked on the code in question  as they might be able to help  But use these results with care  as it can easily send you in a wrong direction  That for example happens quickly in areas rarely changed  like old or unmaintained drivers   sometimes such code is modified during tree wide cleanups by developers that do not care about the particular driver at all    Search for existing reports  second run                                               Search the archives of the bug tracker or mailing list in question     thoroughly for reports that might match your issue  If you find anything      join the discussion instead of sending a new report    As mentioned earlier already  reporting an issue that someone else already brought forward is often a waste of time for everyone involved  especially you as the reporter  That s why you should search for existing report again  now that you know where they need to be reported to  If it s mailing list  you will often find its archives on  lore kernel org  https   lore kernel org       But some list are hosted in different places  That for example is the case for the ath10k WiFi driver used as example in the previous step  But you ll often find the archives for these lists easily on the net  Searching for  archive ath10k lists infradead org  for example will lead you to the  Info page for the ath10k mailing list  https   lists infradead org mailman listinfo ath10k     which at the top links to its  list archives  https   lists infradead org pipermail ath10k      Sadly this and quite a few other lists miss a way to search the archives  In those cases use a regular internet search engine and add something like  site lists infradead org pipermail ath10k   to your search terms  which limits the results to the archives at that URL   It s also wise to check the internet  LKML and maybe bugzilla kernel org again at this point  If your report needs to be filed in a bug tracker  you may want to check the mailing list archives for the subsystem as well  as someone might have reported it only there   For details how to search and what to do if you find matching reports see  Search for existing reports  first run  above   Do not hurry with this step of the reporting process  spending 30 to 60 minutes or even more time can save you and others quite a lot of time and trouble    Install a fresh kernel for testing                                          Unless you are already running the latest  mainline  Linux kernel  better     go and install it for the reporting process  Testing and reporting with     the latest  stable  Linux can be an acceptable alternative in some     situations  during the merge window that actually might be even the best     approach  but in that development phase it can be an even better idea to     suspend your efforts for a few days anyway  Whatever version you choose      ideally use a  vanilla  built  Ignoring these advices will dramatically     increase the risk your report will be rejected or ignored    As mentioned in the detailed explanation for the first step already  Like most programmers  Linux kernel developers don t like to spend time dealing with reports for issues that don t even happen with the current code  It s just a waste everybody s time  especially yours  That s why it s in everybody s interest that you confirm the issue still exists with the latest upstream code before reporting it  You are free to ignore this advice  but as outlined earlier  doing so dramatically increases the risk that your issue report might get rejected or simply ignored   In the scope of the kernel  latest upstream  normally means      Install a mainline kernel  the latest stable kernel can be an option  but    most of the time is better avoided  Longterm kernels  sometimes called  LTS    kernels   are unsuitable at this point of the process  The next subsection    explains all of this in more detail      The over next subsection describes way to obtain and install such a kernel     It also outlines that using a pre compiled kernel are fine  but better are    vanilla  which means  it was built using Linux sources taken straight  from    kernel org  https   kernel org     and not modified or enhanced in any way   Choosing the right version for testing                                         Head over to  kernel org  https   kernel org     to find out which version you want to use for testing  Ignore the big yellow button that says  Latest release  and look a little lower at the table  At its top you ll see a line starting with mainline  which most of the time will point to a pre release with a version number like  5 8 rc2   If that s the case  you ll want to use this mainline kernel for testing  as that where all fixes have to be applied first  Do not let that  rc  scare you  these  development kernels  are pretty reliable   and you made a backup  as you were instructed above  didn t you   In about two out of every nine to ten weeks  mainline might point you to a proper release with a version number like  5 7   If that happens  consider suspending the reporting process until the first pre release of the next version  5 8 rc1  shows up on kernel org  That s because the Linux development cycle then is in its two week long  merge window   The bulk of the changes and all intrusive ones get merged for the next release during this time  It s a bit more risky to use mainline during this period  Kernel developers are also often quite busy then and might have no spare time to deal with issue reports  It s also quite possible that one of the many changes applied during the merge window fixes the issue you face  that s why you soon would have to retest with a newer kernel version anyway  as outlined below in the section  Duties after the report went out    That s why it might make sense to wait till the merge window is over  But don t to that if you re dealing with something that shouldn t wait  In that case consider obtaining the latest mainline kernel via git  see below  or use the latest stable version offered on kernel org  Using that is also acceptable in case mainline for some reason does currently not work for you  An in general  using it for reproducing the issue is also better than not reporting it issue at all   Better avoid using the latest stable kernel outside merge windows  as all fixes must be applied to mainline first  That s why checking the latest mainline kernel is so important  any issue you want to see fixed in older version lines needs to be fixed in mainline first before it can get backported  which can take a few days or weeks  Another reason  the fix you hope for might be too hard or risky for backporting  reporting the issue again hence is unlikely to change anything   These aspects are also why longterm kernels  sometimes called  LTS kernels   are unsuitable for this part of the reporting process  they are to distant from the current code  Hence go and test mainline first and follow the process further  if the issue doesn t occur with mainline it will guide you how to get it fixed in older version lines  if that s in the cards for the fix in question   How to obtain a fresh Linux kernel                                       Using a pre compiled kernel    This is often the quickest  easiest  and safest way for testing   especially is you are unfamiliar with the Linux kernel  The problem  most of those shipped by distributors or add on repositories are build from modified Linux sources  They are thus not vanilla and therefore often unsuitable for testing and issue reporting  the changes might cause the issue you face or influence it somehow   But you are in luck if you are using a popular Linux distribution  for quite a few of them you ll find repositories on the net that contain packages with the latest mainline or stable Linux built as vanilla kernel  It s totally okay to use these  just make sure from the repository s description they are vanilla or at least close to it  Additionally ensure the packages contain the latest versions as offered on kernel org  The packages are likely unsuitable if they are older than a week  as new mainline and stable kernels typically get released at least once a week   Please note that you might need to build your own kernel manually later  that s sometimes needed for debugging or testing fixes  as described later in this document  Also be aware that pre compiled kernels might lack debug symbols that are needed to decode messages the kernel prints when a panic  Oops  warning  or BUG occurs  if you plan to decode those  you might be better off compiling a kernel yourself  see the end of this subsection and the section titled  Decode failure messages  for details      Using git    Developers and experienced Linux users familiar with git are often best served by obtaining the latest Linux kernel sources straight from the  official development repository on kernel org  https   git kernel org pub scm linux kernel git torvalds linux git tree      Those are likely a bit ahead of the latest mainline pre release  Don t worry about it  they are as reliable as a proper pre release  unless the kernel s development cycle is currently in the middle of a merge window  But even then they are quite reliable     Conventional    People unfamiliar with git are often best served by downloading the sources as tarball from  kernel org  https   kernel org       How to actually build a kernel is not described here  as many websites explain the necessary steps already  If you are new to it  consider following one of those how to s that suggest to use   make localmodconfig    as that tries to pick up the configuration of your current kernel and then tries to adjust it somewhat for your system  That does not make the resulting kernel any better  but quicker to compile   Note  If you are dealing with a panic  Oops  warning  or BUG from the kernel  please try to enable CONFIG KALLSYMS when configuring your kernel  Additionally  enable CONFIG DEBUG KERNEL and CONFIG DEBUG INFO  too  the latter is the relevant one of those two  but can only be reached if you enable the former  Be aware CONFIG DEBUG INFO increases the storage space required to build a kernel by quite a bit  But that s worth it  as these options will allow you later to pinpoint the exact line of code that triggers your issue  The section  Decode failure messages  below explains this in more detail   But keep in mind  Always keep a record of the issue encountered in case it is hard to reproduce  Sending an undecoded report is better than not reporting the issue at all    Check  taint  flag                          Ensure the kernel you just installed does not  taint  itself when     running    As outlined above in more detail already  the kernel sets a  taint  flag when something happens that can lead to follow up errors that look totally unrelated  That s why you need to check if the kernel you just installed does not set this flag  And if it does  you in almost all the cases needs to eliminate the reason for it before you reporting issues that occur with it  See the section above for details how to do that    Reproduce issue with the fresh kernel                                             Reproduce the issue with the kernel you just installed  If it doesn t show     up there  scroll down to the instructions for issues only happening with     stable and longterm kernels    Check if the issue occurs with the fresh Linux kernel version you just installed  If it was fixed there already  consider sticking with this version line and abandoning your plan to report the issue  But keep in mind that other users might still be plagued by it  as long as it s not fixed in either stable and longterm version from kernel org  and thus vendor kernels derived from those   If you prefer to use one of those or just want to help their users  head over to the section  Details about reporting issues only occurring in older kernel version lines  below    Optimize description to reproduce issue                                               Optimize your notes  try to find and write the most straightforward way to     reproduce your issue  Make sure the end result has all the important     details  and at the same time is easy to read and understand for others     that hear about it for the first time  And if you learned something in this     process  consider searching again for existing reports about the issue    An unnecessarily complex report will make it hard for others to understand your report  Thus try to find a reproducer that s straight forward to describe and thus easy to understand in written form  Include all important details  but at the same time try to keep it as short as possible   In this in the previous steps you likely have learned a thing or two about the issue you face  Use this knowledge and search again for existing reports instead you can join    Decode failure messages                               If your failure involves a  panic    Oops    warning   or  BUG   consider     decoding the kernel log to find the line of code that triggered the error    When the kernel detects an internal problem  it will log some information about the executed code  This makes it possible to pinpoint the exact line in the source code that triggered the issue and shows how it was called  But that only works if you enabled CONFIG DEBUG INFO and CONFIG KALLSYMS when configuring your kernel  If you did so  consider to decode the information from the kernel s log  That will make it a lot easier to understand what lead to the  panic    Oops    warning   or  BUG   which increases the chances that someone can provide a fix   Decoding can be done with a script you find in the Linux source tree  If you are running a kernel you compiled yourself earlier  call it like this            user something     sudo dmesg     linux 5 10 5 scripts decode stacktrace sh   linux 5 10 5 vmlinux  If you are running a packaged vanilla kernel  you will likely have to install the corresponding packages with debug symbols  Then call the script  which you might need to get from the Linux sources if your distro does not package it  like this            user something     sudo dmesg     linux 5 10 5 scripts decode stacktrace sh            usr lib debug lib modules 5 10 10 4 1 x86 64 vmlinux  usr src kernels 5 10 10 4 1 x86 64   The script will work on log lines like the following  which show the address of the code the kernel was executing when the error occurred               68 387301  RIP  0010 test module init 0x5 0xffa  test module   Once decoded  these lines will look like this               68 387301  RIP  0010 test module init   home username linux 5 10 5 test module test module c 16  test module  In this case the executed code was built from the file    linux 5 10 5 test module test module c  and the error occurred by the instructions found in line  16    The script will similarly decode the addresses mentioned in the section starting with  Call trace   which show the path to the function where the problem occurred  Additionally  the script will show the assembler output for the code section the kernel was executing   Note  if you can t get this to work  simply skip this step and mention the reason for it in the report  If you re lucky  it might not be needed  And if it is  someone might help you to get things going  Also be aware this is just one of several ways to decode kernel stack traces  Sometimes different steps will be required to retrieve the relevant details  Don t worry about that  if that s needed in your case  developers will tell you what to do    Special care for regressions                                    If your problem is a regression  try to narrow down when the issue was     introduced as much as possible    Linux lead developer Linus Torvalds insists that the Linux kernel never worsens  that s why he deems regressions as unacceptable and wants to see them fixed quickly  That s why changes that introduced a regression are often promptly reverted if the issue they cause can t get solved quickly any other way  Reporting a regression is thus a bit like playing a kind of trump card to get something quickly fixed  But for that to happen the change that s causing the regression needs to be known  Normally it s up to the reporter to track down the culprit  as maintainers often won t have the time or setup at hand to reproduce it themselves   To find the change there is a process called  bisection  which the document Documentation admin guide bug bisect rst describes in detail  That process will often require you to build about ten to twenty kernel images  trying to reproduce the issue with each of them before building the next  Yes  that takes some time  but don t worry  it works a lot quicker than most people assume  Thanks to a  binary search  this will lead you to the one commit in the source code management system that s causing the regression  Once you find it  search the net for the subject of the change  its commit id and the shortened commit id  the first 12 characters of the commit id   This will lead you to existing reports about it  if there are any   Note  a bisection needs a bit of know how  which not everyone has  and quite a bit of effort  which not everyone is willing to invest  Nevertheless  it s highly recommended performing a bisection yourself  If you really can t or don t want to go down that route at least find out which mainline kernel introduced the regression  If something for example breaks when switching from 5 5 15 to 5 8 4  then try at least all the mainline releases in that area  5 6  5 7 and 5 8  to check when it first showed up  Unless you re trying to find a regression in a stable or longterm kernel  avoid testing versions which number has three sections  5 6 12  5 7 8   as that makes the outcome hard to interpret  which might render your testing useless  Once you found the major version which introduced the regression  feel free to move on in the reporting process  But keep in mind  it depends on the issue at hand if the developers will be able to help without knowing the culprit  Sometimes they might recognize from the report want went wrong and can fix it  other times they will be unable to help unless you perform a bisection   When dealing with regressions make sure the issue you face is really caused by the kernel and not by something else  as outlined above already   In the whole process keep in mind  an issue only qualifies as regression if the older and the newer kernel got built with a similar configuration  This can be achieved by using   make olddefconfig    as explained in more detail by Documentation admin guide reporting regressions rst  that document also provides a good deal of other information about regressions you might want to be aware of    Write and send the report                                 Start to compile the report by writing a detailed description about the     issue  Always mention a few things  the latest kernel version you installed     for reproducing  the Linux Distribution used  and your notes on how to     reproduce the issue  Ideally  make the kernel s build configuration       config  and the output from   dmesg   available somewhere on the net and     link to it  Include or upload all other information that might be relevant      like the output screenshot of an Oops or the output from   lspci    Once     you wrote this main part  insert a normal length paragraph on top of it     outlining the issue and the impact quickly  On top of this add one sentence     that briefly describes the problem and gets people to read on  Now give the     thing a descriptive title or subject that yet again is shorter  Then you re     ready to send or file the report like the MAINTAINERS file told you  unless     you are dealing with one of those  issues of high priority   they need     special care which is explained in  Special handling for high priority     issues  below    Now that you have prepared everything it s time to write your report  How to do that is partly explained by the three documents linked to in the preface above  That s why this text will only mention a few of the essentials as well as things specific to the Linux kernel   There is one thing that fits both categories  the most crucial parts of your report are the title subject  the first sentence  and the first paragraph  Developers often get quite a lot of mail  They thus often just take a few seconds to skim a mail before deciding to move on or look closer  Thus  the better the top section of your report  the higher are the chances that someone will look into it and help you  And that is why you should ignore them for now and write the detailed report first       Things each report should mention                                    Describe in detail how your issue happens with the fresh vanilla kernel you installed  Try to include the step by step instructions you wrote and optimized earlier that outline how you and ideally others can reproduce the issue  in those rare cases where that s impossible try to describe what you did to trigger it   Also include all the relevant information others might need to understand the issue and its environment  What s actually needed depends a lot on the issue  but there are some things you should include always      the output from   cat  proc version    which contains the Linux kernel    version number and the compiler it was built with      the Linux distribution the machine is running    hostnamectl   grep     Operating System         the architecture of the CPU and the operating system    uname  mi        if you are dealing with a regression and performed a bisection  mention the    subject and the commit id of the change that is causing it   In a lot of cases it s also wise to make two more things available to those that read your report      the configuration used for building your Linux kernel  the   config  file      the kernel s messages that you get from   dmesg   written to a file  Make    sure that it starts with a line like  Linux version 5 8 1     foobar example com   gcc  GCC  10 2 1  GNU ld version 2 34   1 SMP Mon Aug    3 14 54 37 UTC 2020  If it s missing  then important messages from the first    boot phase already got discarded  In this case instead consider using      journalctl  b 0  k    alternatively you can also reboot  reproduce the    issue and call   dmesg   right afterwards   These two files are big  that s why it s a bad idea to put them directly into your report  If you are filing the issue in a bug tracker then attach them to the ticket  If you report the issue by mail do not attach them  as that makes the mail too large  instead do one of these things      Upload the files somewhere public  your website  a public file paste    service  a ticket created just for this purpose on  bugzilla kernel org     https   bugzilla kernel org           and include a link to them in your    report  Ideally use something where the files stay available for years  as    they could be useful to someone many years from now  this for example can    happen if five or ten years from now a developer works on some code that was    changed just to fix your issue      Put the files aside and mention you will send them later in individual    replies to your own mail  Just remember to actually do that once the report    went out       Things that might be wise to provide                                       Depending on the issue you might need to add more background data  Here are a few suggestions what often is good to provide      If you are dealing with a  warning   an  OOPS  or a  panic  from the kernel     include it  If you can t copy n paste it  try to capture a netconsole trace    or at least take a picture of the screen      If the issue might be related to your computer hardware  mention what kind    of system you use  If you for example have problems with your graphics card     mention its manufacturer  the card s model  and what chip is uses  If it s a    laptop mention its name  but try to make sure it s meaningful   Dell XPS 13     for example is not  because it might be the one from 2012  that one looks    not that different from the one sold today  but apart from that the two have    nothing in common  Hence  in such cases add the exact model number  which    for example are  9380  or  7390  for XPS 13 models introduced during 2019     Names like  Lenovo Thinkpad T590  are also somewhat ambiguous  there are    variants of this laptop with and without a dedicated graphics chip  so try    to find the exact model name or specify the main components      Mention the relevant software in use  If you have problems with loading    modules  you want to mention the versions of kmod  systemd  and udev in use     If one of the DRM drivers misbehaves  you want to state the versions of    libdrm and Mesa  also specify your Wayland compositor or the X Server and    its driver  If you have a filesystem issue  mention the version of    corresponding filesystem utilities  e2fsprogs  btrfs progs  xfsprogs            Gather additional information from the kernel that might be of interest  The    output from   lspci  nn   will for example help others to identify what    hardware you use  If you have a problem with hardware you even might want to    make the output from   sudo lspci  vvv   available  as that provides    insights how the components were configured  For some issues it might be    good to include the contents of files like    proc cpuinfo          proc ioports       proc iomem       proc modules    or       proc scsi scsi    Some subsystem also offer tools to collect relevant    information  One such tool is   alsa info sh    which the audio sound    subsystem developers provide  https   www alsa project org wiki AlsaInfo      Those examples should give your some ideas of what data might be wise to attach  but you have to think yourself what will be helpful for others to know  Don t worry too much about forgetting something  as developers will ask for additional details they need  But making everything important available from the start increases the chance someone will take a closer look    The important part  the head of your report                                              Now that you have the detailed part of the report prepared let s get to the most important section  the first few sentences  Thus go to the top  add something like  The detailed description   before the part you just wrote and insert two newlines at the top  Now write one normal length paragraph that describes the issue roughly  Leave out all boring details and focus on the crucial parts readers need to know to understand what this is all about  if you think this bug affects a lot of users  mention this to get people interested   Once you did that insert two more lines at the top and write a one sentence summary that explains quickly what the report is about  After that you have to get even more abstract and write an even shorter subject title for the report   Now that you have written this part take some time to optimize it  as it is the most important parts of your report  a lot of people will only read this before they decide if reading the rest is time well spent   Now send or file the report like the  ref  MAINTAINERS  maintainers   file told you  unless it s one of those  issues of high priority  outlined earlier  in that case please read the next subsection first before sending the report on its way   Special handling for high priority issues                                            Reports for high priority issues need special handling     Severe issues    make sure the subject or ticket title as well as the first paragraph makes the severeness obvious     Regressions    make the report s subject start with   REGRESSION     In case you performed a successful bisection  use the title of the change that introduced the regression as the second part of your subject  Make the report also mention the commit id of the culprit  In case of an unsuccessful bisection  make your report mention the latest tested version that s working fine  say 5 7  and the oldest where the issue occurs  say 5 8 rc1    When sending the report by mail  CC the Linux regressions mailing list  regressions lists linux dev   In case the report needs to be filed to some web tracker  proceed to do so  Once filed  forward the report by mail to the regressions list  CC the maintainer and the mailing list for the subsystem in question  Make sure to inline the forwarded report  hence do not attach it  Also add a short note at the top where you mention the URL to the ticket   When mailing or forwarding the report  in case of a successful bisection add the author of the culprit to the recipients  also CC everyone in the signed off by chain  which you find at the end of its commit message     Security issues    for these issues your will have to evaluate if a short term risk to other users would arise if details were publicly disclosed  If that s not the case simply proceed with reporting the issue as described  For issues that bear such a risk you will need to adjust the reporting process slightly      If the MAINTAINERS file instructed you to report the issue by mail  do not    CC any public mailing lists      If you were supposed to file the issue in a bug tracker make sure to mark    the ticket as  private  or  security issue   If the bug tracker does not    offer a way to keep reports private  forget about it and send your report as    a private mail to the maintainers instead   In both cases make sure to also mail your report to the addresses the MAINTAINERS file lists in the section  security contact   Ideally directly CC them when sending the report by mail  If you filed it in a bug tracker  forward the report s text to these addresses  but on top of it put a small note where you mention that you filed it with a link to the ticket   See Documentation process security bugs rst for more information    Duties after the report went out                                        Wait for reactions and keep the thing rolling until you can accept the     outcome in one way or the other  Thus react publicly and in a timely manner     to any inquiries  Test proposed fixes  Do proactive testing  retest with at     least every first release candidate  RC  of a new mainline version and     report your results  Send friendly reminders if things stall  And try to     help yourself  if you don t get any help or if it s unsatisfying    If your report was good and you are really lucky then one of the developers might immediately spot what s causing the issue  they then might write a patch to fix it  test it  and send it straight for integration in mainline while tagging it for later backport to stable and longterm kernels that need it  Then all you need to do is reply with a  Thank you very much  and switch to a version with the fix once it gets released   But this ideal scenario rarely happens  That s why the job is only starting once you got the report out  What you ll have to do depends on the situations  but often it will be the things listed below  But before digging into the details  here are a few important things you need to keep in mind for this part of the process    General advice for further interactions                                            Always reply in public    When you filed the issue in a bug tracker  always reply there and do not contact any of the developers privately about it  For mailed reports always use the  Reply all  function when replying to any mails you receive  That includes mails with any additional data you might want to add to your report  go to your mail applications  Sent  folder and use  reply all  on your mail with the report  This approach will make sure the public mailing list s  and everyone else that gets involved over time stays in the loop  it also keeps the mail thread intact  which among others is really important for mailing lists to group all related mails together   There are just two situations where a comment in a bug tracker or a  Reply all  is unsuitable      Someone tells you to send something privately      You were told to send something  but noticed it contains sensitive    information that needs to be kept private  In that case it s okay to send it    in private to the developer that asked for it  But note in the ticket or a    mail that you did that  so everyone else knows you honored the request     Do research before asking for clarifications or help    In this part of the process someone might tell you to do something that requires a skill you might not have mastered yet  For example  you might be asked to use some test tools you never have heard of yet  or you might be asked to apply a patch to the Linux kernel sources to test if it helps  In some cases it will be fine sending a reply asking for instructions how to do that  But before going that route try to find the answer own your own by searching the internet  alternatively consider asking in other places for advice  For example ask a friend or post about it to a chatroom or forum you normally hang out     Be patient    If you are really lucky you might get a reply to your report within a few hours  But most of the time it will take longer  as maintainers are scattered around the globe and thus might be in a different time zone   one where they already enjoy their night away from keyboard   In general  kernel developers will take one to five business days to respond to reports  Sometimes it will take longer  as they might be busy with the merge windows  other work  visiting developer conferences  or simply enjoying a long summer holiday   The  issues of high priority   see above for an explanation  are an exception here  maintainers should address them as soon as possible  that s why you should wait a week at maximum  or just two days if it s something urgent  before sending a friendly reminder   Sometimes the maintainer might not be responding in a timely manner  other times there might be disagreements  for example if an issue qualifies as regression or not  In such cases raise your concerns on the mailing list and ask others for public or private replies how to move on  If that fails  it might be appropriate to get a higher authority involved  In case of a WiFi driver that would be the wireless maintainers  if there are no higher level maintainers or all else fails  it might be one of those rare situations where it s okay to get Linus Torvalds involved     Proactive testing    Every time the first pre release  the  rc1   of a new mainline kernel version gets released  go and check if the issue is fixed there or if anything of importance changed  Mention the outcome in the ticket or in a mail you sent as reply to your report  make sure it has all those in the CC that up to that point participated in the discussion   This will show your commitment and that you are willing to help  It also tells developers if the issue persists and makes sure they do not forget about it  A few other occasional retests  for example with rc3  rc5 and the final  are also a good idea  but only report your results if something relevant changed or if you are writing something anyway   With all these general things off the table let s get into the details of how to help to get issues resolved once they were reported   Inquires and testing request                               Here are your duties in case you got replies to your report     Check who you deal with    Most of the time it will be the maintainer or a developer of the particular code area that will respond to your report  But as issues are normally reported in public it could be anyone that s replying   including people that want to help  but in the end might guide you totally off track with their questions or requests  That rarely happens  but it s one of many reasons why it s wise to quickly run an internet search to see who you re interacting with  By doing this you also get aware if your report was heard by the right people  as a reminder to the maintainer  see below  might be in order later if discussion fades out without leading to a satisfying solution for the issue     Inquiries for data    Often you will be asked to test something or provide additional details  Try to provide the requested information soon  as you have the attention of someone that might help and risk losing it the longer you wait  that outcome is even likely if you do not provide the information within a few business days     Requests for testing    When you are asked to test a diagnostic patch or a possible fix  try to test it in timely manner  too  But do it properly and make sure to not rush it  mixing things up can happen easily and can lead to a lot of confusion for everyone involved  A common mistake for example is thinking a proposed patch with a fix was applied  but in fact wasn t  Things like that happen even to experienced testers occasionally  but they most of the time will notice when the kernel with the fix behaves just as one without it   What to do when nothing of substance happens                                               Some reports will not get any reaction from the responsible Linux kernel developers  or a discussion around the issue evolved  but faded out with nothing of substance coming out of it   In these cases wait two  better  three  weeks before sending a friendly reminder  maybe the maintainer was just away from keyboard for a while when your report arrived or had something more important to take care of  When writing the reminder  kindly ask if anything else from your side is needed to get the ball running somehow  If the report got out by mail  do that in the first lines of a mail that is a reply to your initial mail  see above  which includes a full quote of the original report below  that s on of those few situations where such a  TOFU   Text Over  Fullquote Under  is the right approach  as then all the recipients will have the details at hand immediately in the proper order   After the reminder wait three more weeks for replies  If you still don t get a proper reaction  you first should reconsider your approach  Did you maybe try to reach out to the wrong people  Was the report maybe offensive or so confusing that people decided to completely stay away from it  The best way to rule out such factors  show the report to one or two people familiar with FLOSS issue reporting and ask for their opinion  Also ask them for their advice how to move forward  That might mean  prepare a better report and make those people review it before you send it out  Such an approach is totally fine  just mention that this is the second and improved report on the issue and include a link to the first report   If the report was proper you can send a second reminder  in it ask for advice why the report did not get any replies  A good moment for this second reminder mail is shortly after the first pre release  the  rc1   of a new Linux kernel version got published  as you should retest and provide a status update at that point anyway  see above    If the second reminder again results in no reaction within a week  try to contact a higher level maintainer asking for advice  even busy maintainers by then should at least have sent some kind of acknowledgment   Remember to prepare yourself for a disappointment  maintainers ideally should react somehow to every issue report  but they are only obliged to fix those  issues of high priority  outlined earlier  So don t be too devastating if you get a reply along the lines of  thanks for the report  I have more important issues to deal with currently and won t have time to look into this for the foreseeable future    It s also possible that after some discussion in the bug tracker or on a list nothing happens anymore and reminders don t help to motivate anyone to work out a fix  Such situations can be devastating  but is within the cards when it comes to Linux kernel development  This and several other reasons for not getting help are explained in  Why some issues won t get any reaction or remain unfixed after being reported  near the end of this document   Don t get devastated if you don t find any help or if the issue in the end does not get solved  the Linux kernel is FLOSS and thus you can still help yourself  You for example could try to find others that are affected and team up with them to get the issue resolved  Such a team could prepare a fresh report together that mentions how many you are and why this is something that in your option should get fixed  Maybe together you can also narrow down the root cause or the change that introduced a regression  which often makes developing a fix easier  And with a bit of luck there might be someone in the team that knows a bit about programming and might be able to write a fix    Reference for  Reporting regressions within a stable and longterm kernel line                                                                                This subsection provides details for the steps you need to perform if you face a regression within a stable and longterm kernel line   Make sure the particular version line still gets support                                                                Check if the kernel developers still maintain the Linux kernel version     line you care about  go to the front page of kernel org and make sure it     mentions the latest release of the particular version line without an       EOL   tag    Most kernel version lines only get supported for about three months  as maintaining them longer is quite a lot of work  Hence  only one per year is chosen and gets supported for at least two years  often six   That s why you need to check if the kernel developers still support the version line you care for   Note  if kernel org lists two stable version lines on the front page  you should consider switching to the newer one and forget about the older one  support for it is likely to be abandoned soon  Then it will get a  end of life   EOL  stamp  Version lines that reached that point still get mentioned on the kernel org front page for a week or two  but are unsuitable for testing and reporting   Search stable mailing list                                  Check the archives of the Linux stable mailing list for existing reports    Maybe the issue you face is already known and was fixed or is about to  Hence   search the archives of the Linux stable mailing list  https   lore kernel org stable     for reports about an issue like yours  If you find any matches  consider joining the discussion  unless the fix is already finished and scheduled to get applied soon   Reproduce issue with the newest release                                               Install the latest release from the particular version line as a vanilla     kernel  Ensure this kernel is not tainted and still shows the problem  as     the issue might have already been fixed there  If you first noticed the     problem with a vendor kernel  check a vanilla build of the last version     known to work performs fine as well    Before investing any more time in this process you want to check if the issue was already fixed in the latest release of version line you re interested in  This kernel needs to be vanilla and shouldn t be tainted before the issue happens  as detailed outlined already above in the section  Install a fresh kernel for testing    Did you first notice the regression with a vendor kernel  Then changes the vendor applied might be interfering  You need to rule that out by performing a recheck  Say something broke when you updated from 5 10 4 vendor 42 to 5 10 5 vendor 43  Then after testing the latest 5 10 release as outlined in the previous paragraph check if a vanilla build of Linux 5 10 4 works fine as well  If things are broken there  the issue does not qualify as upstream regression and you need switch back to the main step by step guide to report the issue   Report the regression                             Send a short problem report to the Linux stable mailing list      stable vger kernel org  and CC the Linux regressions mailing list      regressions lists linux dev   if you suspect the cause in a particular     subsystem  CC its maintainer and its mailing list  Roughly describe the     issue and ideally explain how to reproduce it  Mention the first version     that shows the problem and the last version that s working fine  Then     wait for further instructions    When reporting a regression that happens within a stable or longterm kernel line  say when updating from 5 10 4 to 5 10 5  a brief report is enough for the start to get the issue reported quickly  Hence a rough description to the stable and regressions mailing list is all it takes  but in case you suspect the cause in a particular subsystem  CC its maintainers and its mailing list as well  because that will speed things up   And note  it helps developers a great deal if you can specify the exact version that introduced the problem  Hence if possible within a reasonable time frame  try to find that version using vanilla kernels  Lets assume something broke when your distributor released a update from Linux kernel 5 10 5 to 5 10 8  Then as instructed above go and check the latest kernel from that version line  say 5 10 9  If it shows the problem  try a vanilla 5 10 5 to ensure that no patches the distributor applied interfere  If the issue doesn t manifest itself there  try 5 10 7 and then  depending on the outcome  5 10 8 or 5 10 6 to find the first version where things broke  Mention it in the report and state that 5 10 9 is still broken   What the previous paragraph outlines is basically a rough manual  bisection   Once your report is out your might get asked to do a proper one  as it allows to pinpoint the exact change that causes the issue  which then can easily get reverted to fix the issue quickly   Hence consider to do a proper bisection right away if time permits  See the section  Special care for regressions  and the document Documentation admin guide bug bisect rst for details how to perform one  In case of a successful bisection add the author of the culprit to the recipients  also CC everyone in the signed off by chain  which you find at the end of its commit message    Reference for  Reporting issues only occurring in older kernel version lines                                                                                This section provides details for the steps you need to take if you could not reproduce your issue with a mainline kernel  but want to see it fixed in older version lines  aka stable and longterm kernels    Some fixes are too complex                                  Prepare yourself for the possibility that going through the next few steps     might not get the issue solved in older releases  the fix might be too big     or risky to get backported there    Even small and seemingly obvious code changes sometimes introduce new and totally unexpected problems  The maintainers of the stable and longterm kernels are very aware of that and thus only apply changes to these kernels that are within rules outlined in Documentation process stable kernel rules rst   Complex or risky changes for example do not qualify and thus only get applied to mainline  Other fixes are easy to get backported to the newest stable and longterm kernels  but too risky to integrate into older ones  So be aware the fix you are hoping for might be one of those that won t be backported to the version line your care about  In that case you ll have no other choice then to live with the issue or switch to a newer Linux version  unless you want to patch the fix into your kernels yourself   Common preparations                           Perform the first three steps in the section  Reporting issues only     occurring in older kernel version lines  above    You need to carry out a few steps already described in another section of this guide  Those steps will let you      Check if the kernel developers still maintain the Linux kernel version line    you care about      Search the Linux stable mailing list for exiting reports      Check with the latest release    Check code history and search for existing discussions                                                              Search the Linux kernel version control system for the change that fixed     the issue in mainline  as its commit message might tell you if the fix is     scheduled for backporting already  If you don t find anything that way      search the appropriate mailing lists for posts that discuss such an issue     or peer review possible fixes  then check the discussions if the fix was     deemed unsuitable for backporting  If backporting was not considered at     all  join the newest discussion  asking if it s in the cards    In a lot of cases the issue you deal with will have happened with mainline  but got fixed there  The commit that fixed it would need to get backported as well to get the issue solved  That s why you want to search for it or any discussions abound it      First try to find the fix in the Git repository that holds the Linux kernel    sources  You can do this with the web interfaces  on kernel org     https   git kernel org pub scm linux kernel git torvalds linux git tree        or its mirror  on GitHub  https   github com torvalds linux     if you have    a local clone you alternatively can search on the command line with   git    log   grep  pattern         If you find the fix  look if the commit message near the end contains a     stable tag  that looks like this             Cc   stable vger kernel org    5 4      If that s case the developer marked the fix safe for backporting to version    line 5 4 and later  Most of the time it s getting applied there within two    weeks  but sometimes it takes a bit longer      If the commit doesn t tell you anything or if you can t find the fix  look    again for discussions about the issue  Search the net with your favorite    internet search engine as well as the archives for the  Linux kernel    developers mailing list  https   lore kernel org lkml      Also read the    section  Locate kernel area that causes the issue  above and follow the    instructions to find the subsystem in question  its bug tracker or mailing    list archive might have the answer you are looking for      If you see a proposed fix  search for it in the version control system as    outlined above  as the commit might tell you if a backport can be expected        Check the discussions for any indicators the fix might be too risky to get      backported to the version line you care about  If that s the case you have      to live with the issue or switch to the kernel version line where the fix      got applied        If the fix doesn t contain a stable tag and backporting was not discussed       join the discussion  mention the version where you face the issue and that      you would like to see it fixed  if suitable    Ask for advice                      One of the former steps should lead to a solution  If that doesn t work     out  ask the maintainers for the subsystem that seems to be causing the     issue for advice  CC the mailing list for the particular subsystem as well     as the stable mailing list    If the previous three steps didn t get you closer to a solution there is only one option left  ask for advice  Do that in a mail you sent to the maintainers for the subsystem where the issue seems to have its roots  CC the mailing list for the subsystem as well as the stable mailing list  stable vger kernel org     Why some issues won t get any reaction or remain unfixed after being reported                                                                                When reporting a problem to the Linux developers  be aware only  issues of high priority   regressions  security issues  severe problems  are definitely going to get resolved  The maintainers or if all else fails Linus Torvalds himself will make sure of that  They and the other kernel developers will fix a lot of other issues as well  But be aware that sometimes they can t or won t help  and sometimes there isn t even anyone to send a report to   This is best explained with kernel developers that contribute to the Linux kernel in their spare time  Quite a few of the drivers in the kernel were written by such programmers  often because they simply wanted to make their hardware usable on their favorite operating system   These programmers most of the time will happily fix problems other people report  But nobody can force them to do  as they are contributing voluntarily   Then there are situations where such developers really want to fix an issue  but can t  sometimes they lack hardware programming documentation to do so  This often happens when the publicly available docs are superficial or the driver was written with the help of reverse engineering   Sooner or later spare time developers will also stop caring for the driver  Maybe their test hardware broke  got replaced by something more fancy  or is so old that it s something you don t find much outside of computer museums anymore  Sometimes developer stops caring for their code and Linux at all  as something different in their life became way more important  In some cases nobody is willing to take over the job as maintainer   and nobody can be forced to  as contributing to the Linux kernel is done on a voluntary basis  Abandoned drivers nevertheless remain in the kernel  they are still useful for people and removing would be a regression   The situation is not that different with developers that are paid for their work on the Linux kernel  Those contribute most changes these days  But their employers sooner or later also stop caring for their code or make its programmer focus on other things  Hardware vendors for example earn their money mainly by selling new hardware  quite a few of them hence are not investing much time and energy in maintaining a Linux kernel driver for something they stopped selling years ago  Enterprise Linux distributors often care for a longer time period  but in new versions often leave support for old and rare hardware aside to limit the scope  Often spare time contributors take over once a company orphans some code  but as mentioned above  sooner or later they will leave the code behind  too   Priorities are another reason why some issues are not fixed  as maintainers quite often are forced to set those  as time to work on Linux is limited  That s true for spare time or the time employers grant their developers to spend on maintenance work on the upstream kernel  Sometimes maintainers also get overwhelmed with reports  even if a driver is working nearly perfectly  To not get completely stuck  the programmer thus might have no other choice than to prioritize issue reports and reject some of them   But don t worry too much about all of this  a lot of drivers have active maintainers who are quite interested in fixing as many issues as possible    Closing words                Compared with other Free Libre   Open Source Software it s hard to report issues to the Linux kernel developers  the length and complexity of this document and the implications between the lines illustrate that  But that s how it is for now  The main author of this text hopes documenting the state of the art will lay some groundwork to improve the situation over time          end of content       This document is maintained by Thorsten Leemhuis  linux leemhuis info   If    you spot a typo or small mistake  feel free to let him know directly and    he ll fix it  You are free to do the same in a mostly informal way if you    want to contribute changes to the text  but for copyright reasons please CC    linux doc vger kernel org and  sign off  your contribution as    Documentation process submitting patches rst outlines in the section  Sign    your work   the Developer s Certificate of Origin         This text is available under GPL 2 0  or CC BY 4 0  as stated at the top    of the file  If you want to distribute this text under CC BY 4 0 only     please use  The Linux kernel developers  for author attribution and link    this as source     https   git kernel org pub scm linux kernel git torvalds linux git plain Documentation admin guide reporting issues rst       Note  Only the content of this RST file as found in the Linux kernel sources    is available under CC BY 4 0  as versions of this text that were processed     for example by the kernel s build system  might contain content taken from    files which use a more restrictive license