diff options
Diffstat (limited to 'Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst')
| -rw-r--r-- | Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst | 940 |
1 files changed, 605 insertions, 335 deletions
diff --git a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst index 58211840ac6f..c389d4fd7599 100644 --- a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst +++ b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst @@ -29,7 +29,7 @@ The essence of the process (aka 'TL;DR') ======================================== *[If you are new to building or bisecting Linux, ignore this section and head -over to the* ":ref:`step-by-step guide<introguide_bissbs>`" *below. It utilizes +over to the* ':ref:`step-by-step guide <introguide_bissbs>`' *below. It utilizes the same commands as this section while describing them in brief fashion. The steps are nevertheless easy to follow and together with accompanying entries in a reference section mention many alternatives, pitfalls, and additional @@ -38,14 +38,14 @@ aspects, all of which might be essential in your present case.]* **In case you want to check if a bug is present in code currently supported by developers**, execute just the *preparations* and *segment 1*; while doing so, consider the newest Linux kernel you regularly use to be the 'working' kernel. -In the following example that's assumed to be 6.0.13, which is why the sources -of v6.0 will be used to prepare the .config file. +In the following example that's assumed to be 6.0, which is why its sources +will be used to prepare the .config file. **In case you face a regression**, follow the steps at least till the end of *segment 2*. Then you can submit a preliminary report -- or continue with *segment 3*, which describes how to perform a bisection needed for a full-fledged regression report. In the following example 6.0.13 is assumed to be -the 'working' kernel and 6.1.5 to be the first 'broken', which is why v6.0 +the 'working' kernel and 6.1.5 to be the first 'broken', which is why 6.0 will be considered the 'good' release and used to prepare the .config file. * **Preparations**: set up everything to build your own kernels:: @@ -61,7 +61,7 @@ will be considered the 'good' release and used to prepare the .config file. cd ~/linux/ git remote add -t master stable \ https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git - git checkout --detach v6.0 + git switch --detach v6.0 # * Hint: if you used an existing clone, ensure no stale .config is around. make olddefconfig # * Ensure the former command picked the .config of the 'working' kernel. @@ -87,7 +87,7 @@ will be considered the 'good' release and used to prepare the .config file. a) Checking out latest mainline code:: cd ~/linux/ - git checkout --force --detach mainline/master + git switch --discard-changes --detach mainline/master b) Build, install, and boot a kernel:: @@ -99,7 +99,8 @@ will be considered the 'good' release and used to prepare the .config file. # * Note: on Arch Linux, its derivatives and a few other distributions # the following commands will do nothing at all or only part of the # job. See the step-by-step guide for further details. - command -v installkernel && sudo make modules_install install + sudo make modules_install + command -v installkernel && sudo make install # * Check how much space your self-built kernel actually needs, which # enables you to make better estimates later: du -ch /boot/*$(make -s kernelrelease)* | tail -n 1 @@ -112,6 +113,7 @@ will be considered the 'good' release and used to prepare the .config file. # checking if the output of the next two commands matches: tail -n 1 ~/kernels-built uname -r + cat /proc/sys/kernel/tainted c) Check if the problem occurs with this kernel as well. @@ -123,7 +125,7 @@ will be considered the 'good' release and used to prepare the .config file. a) Start by checking out the sources of the 'good' version:: cd ~/linux/ - git checkout --force --detach v6.0 + git switch --discard-changes --detach v6.0 b) Build, install, and boot a kernel as described earlier in *segment 1, section b* -- just feel free to skip the 'du' commands, as you have a rough @@ -134,8 +136,7 @@ will be considered the 'good' release and used to prepare the .config file. * **Segment 3**: perform and validate the bisection. - a) In case your 'broken' version is a stable/longterm release, add the Git - branch holding it:: + a) Retrieve the sources for your 'bad' version:: git remote set-branches --add stable linux-6.1.y git fetch stable @@ -155,11 +156,12 @@ will be considered the 'good' release and used to prepare the .config file. works with the newly built kernel. If it does, tell Git by executing ``git bisect good``; if it does not, run ``git bisect bad`` instead. - All three commands will make Git checkout another commit; then re-execute + All three commands will make Git check out another commit; then re-execute this step (e.g. build, install, boot, and test a kernel to then tell Git the outcome). Do so again and again until Git shows which commit broke things. If you run short of disk space during this process, check the - "Supplementary tasks" section below. + section 'Complementary tasks: cleanup during and after the process' + below. d) Once your finished the bisection, put a few things away:: @@ -170,14 +172,17 @@ will be considered the 'good' release and used to prepare the .config file. e) Try to verify the bisection result:: - git checkout --force --detach mainline/master + git switch --discard-changes --detach mainline/master git revert --no-edit cafec0cacaca0 + cp ~/kernel-config-working .config + ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted' This is optional, as some commits are impossible to revert. But if the second command worked flawlessly, build, install, and boot one more kernel - kernel, which should not show the regression. + kernel; just this time skip the first command copying the base .config file + over, as that already has been taken care off. -* **Supplementary tasks**: cleanup during and after the process. +* **Complementary tasks**: cleanup during and after the process. a) To avoid running out of disk space during a bisection, you might need to remove some kernels you built earlier. You most likely want to keep those @@ -200,13 +205,25 @@ will be considered the 'good' release and used to prepare the .config file. the kernels you built earlier and later you might want to keep around for a week or two. +* **Optional task**: test a debug patch or a proposed fix later:: + + git fetch mainline + git switch --discard-changes --detach mainline/master + git apply /tmp/foobars-proposed-fix-v1.patch + cp ~/kernel-config-working .config + ./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1' + + Build, install, and boot a kernel as described in *segment 1, section b* -- + but this time omit the first command copying the build configuration over, + as that has been taken care of already. + .. _introguide_bissbs: Step-by-step guide on how to verify bugs and bisect regressions =============================================================== This guide describes how to set up your own Linux kernels for investigating bugs -or regressions you intent to report. How far you want to follow the instructions +or regressions you intend to report. How far you want to follow the instructions depends on your issue: Execute all steps till the end of *segment 1* to **verify if your kernel problem @@ -219,42 +236,56 @@ report; instead of the latter your could also head straight on and follow *segment 3* to **perform a bisection** for a full-fledged regression report developers are obliged to act upon. - :ref:`Preparations: set up everything to build your own kernels.<introprep_bissbs>` + :ref:`Preparations: set up everything to build your own kernels <introprep_bissbs>`. + + :ref:`Segment 1: try to reproduce the problem with the latest codebase <introlatestcheck_bissbs>`. - :ref:`Segment 1: try to reproduce the problem with the latest codebase.<introlatestcheck_bissbs>` + :ref:`Segment 2: check if the kernels you build work fine <introworkingcheck_bissbs>`. - :ref:`Segment 2: check if the kernels you build work fine.<introworkingcheck_bissbs>` + :ref:`Segment 3: perform a bisection and validate the result <introbisect_bissbs>`. - :ref:`Segment 3: perform a bisection and validate the result.<introbisect_bissbs>` + :ref:`Complementary tasks: cleanup during and after following this guide <introclosure_bissbs>`. - :ref:`Supplementary tasks: cleanup during and after following this guide.<introclosure_bissbs>` + :ref:`Optional tasks: test reverts, patches, or later versions <introoptional_bissbs>`. The steps in each segment illustrate the important aspects of the process, while -a comprehensive reference section holds additional details. The latter sometimes -also outlines alternative approaches, pitfalls, as well as problems that might -occur at the particular step -- and how to get things rolling again. +a comprehensive reference section holds additional details for almost all of the +steps. The reference section sometimes also outlines alternative approaches, +pitfalls, as well as problems that might occur at the particular step -- and how +to get things rolling again. For further details on how to report Linux kernel issues or regressions check out Documentation/admin-guide/reporting-issues.rst, which works in conjunction with this document. It among others explains why you need to verify bugs with -the latest 'mainline' kernel, even if you face a problem with a kernel from a -'stable/longterm' series; for users facing a regression it also explains that -sending a preliminary report after finishing segment 2 might be wise, as the -regression and its culprit might be known already. For further details on -what actually qualifies as a regression check out -Documentation/admin-guide/reporting-regressions.rst. +the latest 'mainline' kernel (e.g. versions like 6.0, 6.1-rc1, or 6.1-rc6), +even if you face a problem with a kernel from a 'stable/longterm' series +(say 6.0.13). + +For users facing a regression that document also explains why sending a +preliminary report after segment 2 might be wise, as the regression and its +culprit might be known already. For further details on what actually qualifies +as a regression check out Documentation/admin-guide/reporting-regressions.rst. + +If you run into any problems while following this guide or have ideas how to +improve it, :ref:`please let the kernel developers know <submit_improvements>`. .. _introprep_bissbs: Preparations: set up everything to build your own kernels --------------------------------------------------------- +The following steps lay the groundwork for all further tasks. + +Note: the instructions assume you are building and testing on the same +machine; if you want to compile the kernel on another system, check +:ref:`Build kernels on a different machine <buildhost_bis>` below. + .. _backup_bissbs: * Create a fresh backup and put system repair and restore tools at hand, just to be prepared for the unlikely case of something going sideways. - [:ref:`details<backup_bisref>`] + [:ref:`details <backup_bisref>`] .. _vanilla_bissbs: @@ -262,7 +293,7 @@ Preparations: set up everything to build your own kernels builds them automatically. That includes but is not limited to DKMS, openZFS, VirtualBox, and Nvidia's graphics drivers (including the GPLed kernel module). - [:ref:`details<vanilla_bisref>`] + [:ref:`details <vanilla_bisref>`] .. _secureboot_bissbs: @@ -273,48 +304,49 @@ Preparations: set up everything to build your own kernels their restrictions through a process initiated by ``mokutil --disable-validation``. - [:ref:`details<secureboot_bisref>`] + [:ref:`details <secureboot_bisref>`] .. _rangecheck_bissbs: * Determine the kernel versions considered 'good' and 'bad' throughout this - guide. + guide: - Do you follow this guide to verify if a bug is present in the code developers - care for? Then consider the mainline release your 'working' kernel (the newest - one you regularly use) is based on to be the 'good' version; if your 'working' - kernel for example is '6.0.11', then your 'good' kernel is 'v6.0'. + * Do you follow this guide to verify if a bug is present in the code the + primary developers care for? Then consider the version of the newest kernel + you regularly use currently as 'good' (e.g. 6.0, 6.0.13, or 6.1-rc2). - In case you face a regression, it depends on the version range where the - regression was introduced: + * Do you face a regression, e.g. something broke or works worse after + switching to a newer kernel version? In that case it depends on the version + range during which the problem appeared: - * Something which used to work in Linux 6.0 broke when switching to Linux - 6.1-rc1? Then henceforth regard 'v6.0' as the last known 'good' version - and 'v6.1-rc1' as the first 'bad' one. + * Something regressed when updating from a stable/longterm release + (say 6.0.13) to a newer mainline series (like 6.1-rc7 or 6.1) or a + stable/longterm version based on one (say 6.1.5)? Then consider the + mainline release your working kernel is based on to be the 'good' + version (e.g. 6.0) and the first version to be broken as the 'bad' one + (e.g. 6.1-rc7, 6.1, or 6.1.5). Note, at this point it is merely assumed + that 6.0 is fine; this hypothesis will be checked in segment 2. - * Some function stopped working when updating from 6.0.11 to 6.1.4? Then for - the time being consider 'v6.0' as the last 'good' version and 'v6.1.4' as - the 'bad' one. Note, at this point it is merely assumed that 6.0 is fine; - this assumption will be checked in segment 2. + * Something regressed when switching from one mainline version (say 6.0) to + a later one (like 6.1-rc1) or a stable/longterm release based on it + (say 6.1.5)? Then regard the last working version (e.g. 6.0) as 'good' and + the first broken (e.g. 6.1-rc1 or 6.1.5) as 'bad'. - * A feature you used in 6.0.11 does not work at all or worse in 6.1.13? In - that case you want to bisect within a stable/longterm series: consider - 'v6.0.11' as the last known 'good' version and 'v6.0.13' as the first 'bad' - one. Note, in this case you still want to compile and test a mainline kernel - as explained in segment 1: the outcome will determine if you need to report - your issue to the regular developers or the stable team. + * Something regressed when updating within a stable/longterm series (say + from 6.0.13 to 6.0.15)? Then consider those versions as 'good' and 'bad' + (e.g. 6.0.13 and 6.0.15), as you need to bisect within that series. *Note, do not confuse 'good' version with 'working' kernel; the latter term throughout this guide will refer to the last kernel that has been working fine.* - [:ref:`details<rangecheck_bisref>`] + [:ref:`details <rangecheck_bisref>`] .. _bootworking_bissbs: * Boot into the 'working' kernel and briefly use the apparently broken feature. - [:ref:`details<bootworking_bisref>`] + [:ref:`details <bootworking_bisref>`] .. _diskspace_bissbs: @@ -324,7 +356,7 @@ Preparations: set up everything to build your own kernels debug symbols: both explain approaches reducing the amount of space, which should allow you to master these tasks with about 4 Gigabytes free space. - [:ref:`details<diskspace_bisref>`] + [:ref:`details <diskspace_bisref>`] .. _buildrequires_bissbs: @@ -334,7 +366,7 @@ Preparations: set up everything to build your own kernels reference section shows how to quickly install those on various popular Linux distributions. - [:ref:`details<buildrequires_bisref>`] + [:ref:`details <buildrequires_bisref>`] .. _sources_bissbs: @@ -349,7 +381,7 @@ Preparations: set up everything to build your own kernels internet connections.* Execute the following command to retrieve a fresh mainline codebase while - preparing things to add stable/longterm branches later:: + preparing things to add branches for stable/longterm series later:: git clone -o mainline --no-checkout \ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git ~/linux/ @@ -357,25 +389,39 @@ Preparations: set up everything to build your own kernels git remote add -t master stable \ https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git - [:ref:`details<sources_bisref>`] + [:ref:`details <sources_bisref>`] + +.. _stablesources_bissbs: + +* Is one of the versions you earlier established as 'good' or 'bad' a stable or + longterm release (say 6.1.5)? Then download the code for the series it belongs + to ('linux-6.1.y' in this example):: + + git remote set-branches --add stable linux-6.1.y + git fetch stable .. _oldconfig_bissbs: * Start preparing a kernel build configuration (the '.config' file). Before doing so, ensure you are still running the 'working' kernel an earlier - step told you to boot; if you are unsure, check the current kernel release + step told you to boot; if you are unsure, check the current kernelrelease identifier using ``uname -r``. Afterwards check out the source code for the version earlier established as - 'good' (in this example this is assumed to be 6.0) and create a .config file:: + 'good'. In the following example command this is assumed to be 6.0; note that + the version number in this and all later Git commands needs to be prefixed + with a 'v':: + + git switch --discard-changes --detach v6.0 + + Now create a build configuration file:: - git checkout --detach v6.0 make olddefconfig - The second command will try to locate the build configuration file for the - running kernel and then adjust it for the needs of the kernel sources you - checked out. While doing so, it will print a few lines you need to check. + The kernel build scripts then will try to locate the build configuration file + for the running kernel and then adjust it for the needs of the kernel sources + you checked out. While doing so, it will print a few lines you need to check. Look out for a line starting with '# using defaults found in'. It should be followed by a path to a file in '/boot/' that contains the release identifier @@ -390,7 +436,7 @@ Preparations: set up everything to build your own kernels 'make olddefconfig' again and check if it now picked up the right config file as base. - [:ref:`details<oldconfig_bisref>`] + [:ref:`details <oldconfig_bisref>`] .. _localmodconfig_bissbs: @@ -424,7 +470,7 @@ Preparations: set up everything to build your own kernels spending much effort on, as long as it boots and allows to properly test the feature that causes trouble. - [:ref:`details<localmodconfig_bisref>`] + [:ref:`details <localmodconfig_bisref>`] .. _tagging_bissbs: @@ -434,7 +480,7 @@ Preparations: set up everything to build your own kernels ./scripts/config --set-str CONFIG_LOCALVERSION '-local' ./scripts/config -e CONFIG_LOCALVERSION_AUTO - [:ref:`details<tagging_bisref>`] + [:ref:`details <tagging_bisref>`] .. _debugsymbols_bissbs: @@ -453,7 +499,7 @@ Preparations: set up everything to build your own kernels ./scripts/config -d DEBUG_INFO -d DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \ -d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF5 -e CONFIG_DEBUG_INFO_NONE - [:ref:`details<debugsymbols_bisref>`] + [:ref:`details <debugsymbols_bisref>`] .. _configmods_bissbs: @@ -463,14 +509,14 @@ Preparations: set up everything to build your own kernels * Are you running Debian? Then you want to avoid known problems by performing additional adjustments explained in the reference section. - [:ref:`details<configmods_distros_bisref>`]. + [:ref:`details <configmods_distros_bisref>`]. * If you want to influence other aspects of the configuration, do so now using your preferred tool. Note, to use make targets like 'menuconfig' or 'nconfig', you will need to install the development files of ncurses; for 'xconfig' you likewise need the Qt5 or Qt6 headers. - [:ref:`details<configmods_individual_bisref>`]. + [:ref:`details <configmods_individual_bisref>`]. .. _saveconfig_bissbs: @@ -480,7 +526,7 @@ Preparations: set up everything to build your own kernels make olddefconfig cp .config ~/kernel-config-working - [:ref:`details<saveconfig_bisref>`] + [:ref:`details <saveconfig_bisref>`] .. _introlatestcheck_bissbs: @@ -490,16 +536,30 @@ Segment 1: try to reproduce the problem with the latest codebase The following steps verify if the problem occurs with the code currently supported by developers. In case you face a regression, it also checks that the problem is not caused by some .config change, as reporting the issue then would -be a waste of time. [:ref:`details<introlatestcheck_bisref>`] +be a waste of time. [:ref:`details <introlatestcheck_bisref>`] .. _checkoutmaster_bissbs: -* Check out the latest Linux codebase:: +* Check out the latest Linux codebase. - cd ~/linux/ - git checkout --force --detach mainline/master + * Are your 'good' and 'bad' versions from the same stable or longterm series? + Then check the `front page of kernel.org <https://kernel.org/>`_: if it + lists a release from that series without an '[EOL]' tag, checkout the series + latest version ('linux-6.1.y' in the following example):: + + cd ~/linux/ + git switch --discard-changes --detach stable/linux-6.1.y + + Your series is unsupported, if is not listed or carrying a 'end of life' + tag. In that case you might want to check if a successor series (say + linux-6.2.y) or mainline (see next point) fix the bug. + + * In all other cases, run:: + + cd ~/linux/ + git switch --discard-changes --detach mainline/master - [:ref:`details<checkoutmaster_bisref>`] + [:ref:`details <checkoutmaster_bisref>`] .. _build_bissbs: @@ -514,52 +574,40 @@ be a waste of time. [:ref:`details<introlatestcheck_bisref>`] reference section for alternatives, which obviously will require other steps to install as well. - [:ref:`details<build_bisref>`] + [:ref:`details <build_bisref>`] .. _install_bissbs: * Install your newly built kernel. - Before doing so, consider checking if there is still enough room for it:: + Before doing so, consider checking if there is still enough space for it:: df -h /boot/ /lib/modules/ - 150 MByte in /boot/ and 200 in /lib/modules/ usually suffice. Those are rough - estimates assuming the worst case. How much your kernels actually require will - be determined later. + For now assume 150 MByte in /boot/ and 200 in /lib/modules/ will suffice; how + much your kernels actually require will be determined later during this guide. - Now install the kernel, which will be saved in parallel to the kernels from - your Linux distribution:: + Now install the kernel's modules and its image, which will be stored in + parallel to the your Linux distribution's kernels:: - command -v installkernel && sudo make modules_install install + sudo make modules_install + command -v installkernel && sudo make install - On many commodity Linux distributions this will take care of everything - required to boot your kernel. You might want to ensure that's the case by - checking if your boot loader's configuration was updated; furthermore ensure - an initramfs (also known as initrd) exists, which on many distributions can be - achieved by running ``ls -l /boot/init*$(make -s kernelrelease)*``. Those - steps are recommended, as there are quite a few Linux distribution where above - command is insufficient: + The second command ideally will take care of three steps required at this + point: copying the kernel's image to /boot/, generating an initramfs, and + adding an entry for both to the boot loader's configuration. - * On Arch Linux, its derivatives, many immutable Linux distributions, and a - few others the above command does nothing at, as they lack 'installkernel' - executable. + Sadly some distributions (among them Arch Linux, its derivatives, and many + immutable Linux distributions) will perform none or only some of those tasks. + You therefore want to check if all of them were taken care of and manually + perform those that were not. The reference section provides further details on + that; your distribution's documentation might help, too. - * Some distributions install the kernel, but don't add an entry for your - kernel in your boot loader's configuration -- the kernel thus won't show up - in the boot menu. + Once you figured out the steps needed at this point, consider writing them + down: if you will build more kernels as described in segment 2 and 3, you will + have to perform those again after executing ``command -v installkernel [...]``. - * Some distributions add a boot loader menu entry, but don't create an - initramfs on installation -- in that case your kernel most likely will be - unable to mount the root partition during bootup. - - If any of that applies to you, see the reference section for further guidance. - Once you figured out what to do, consider writing down the necessary - installation steps: if you will build more kernels as described in - segment 2 and 3, you will have to execute these commands every time that - ``command -v installkernel [...]`` comes up again. - - [:ref:`details<install_bisref>`] + [:ref:`details <install_bisref>`] .. _storagespace_bissbs: @@ -572,7 +620,7 @@ be a waste of time. [:ref:`details<introlatestcheck_bisref>`] Write down or remember those two values for later: they enable you to prevent running out of disk space accidentally during a bisection. - [:ref:`details<storagespace_bisref>`] + [:ref:`details <storagespace_bisref>`] .. _kernelrelease_bissbs: @@ -583,36 +631,43 @@ be a waste of time. [:ref:`details<introlatestcheck_bisref>`] Remember the identifier momentarily, as it will help you pick the right kernel from the boot menu upon restarting. -.. _recheckbroken_bissbs: - -* Reboot into the kernel you just built and check if the feature that is - expected to be broken really is. - - Start by making sure the kernel you booted is the one you just built. When - unsure, check if the output of these commands show the exact same release - identifier:: +* Reboot into your newly built kernel. To ensure your actually started the one + you just built, you might want to verify if the output of these commands + matches:: tail -n 1 ~/kernels-built uname -r - Now verify if the feature that causes trouble works with your newly built - kernel. If things work while investigating a regression, check the reference - section for further details. +.. _tainted_bissbs: + +* Check if the kernel marked itself as 'tainted':: + + cat /proc/sys/kernel/tainted - [:ref:`details<recheckbroken_bisref>`] + If that command does not return '0', check the reference section, as the cause + for this might interfere with your testing. + + [:ref:`details <tainted_bisref>`] + +.. _recheckbroken_bissbs: + +* Verify if your bug occurs with the newly built kernel. If it does not, check + out the instructions in the reference section to ensure nothing went sideways + during your tests. + + [:ref:`details <recheckbroken_bisref>`] .. _recheckstablebroken_bissbs: -* Are you facing a problem within a stable/longterm release, but failed to - reproduce it with the mainline kernel you just built? Then check if the latest - codebase for the particular series might already fix the problem. To do so, - add the stable series Git branch for your 'good' kernel (again, this here is - assumed to be 6.0) and check out the latest version:: +* Did you just built a stable or longterm kernel? And were you able to reproduce + the regression with it? Then you should test the latest mainline codebase as + well, because the result determines which developers the bug must be submitted + to. + + To prepare that test, check out current mainline:: cd ~/linux/ - git remote set-branches --add stable linux-6.0.y - git fetch stable - git checkout --force --detach linux-6.0.y + git switch --discard-changes --detach mainline/master Now use the checked out code to build and install another kernel using the commands the earlier steps already described in more detail:: @@ -622,22 +677,28 @@ be a waste of time. [:ref:`details<introlatestcheck_bisref>`] make -j $(nproc --all) # * Check if the free space suffices holding another kernel: df -h /boot/ /lib/modules/ - command -v installkernel && sudo make modules_install install + sudo make modules_install + command -v installkernel && sudo make install make -s kernelrelease | tee -a ~/kernels-built reboot - Now verify if you booted the kernel you intended to start, to then check if - everything works fine with this kernel:: + Confirm you booted the kernel you intended to start and check its tainted + status:: tail -n 1 ~/kernels-built uname -r + cat /proc/sys/kernel/tainted + + Now verify if this kernel is showing the problem. If it does, then you need + to report the bug to the primary developers; if it does not, report it to the + stable team. See Documentation/admin-guide/reporting-issues.rst for details. - [:ref:`details<recheckstablebroken_bisref>`] + [:ref:`details <recheckstablebroken_bisref>`] Do you follow this guide to verify if a problem is present in the code currently supported by Linux kernel developers? Then you are done at this point. If you later want to remove the kernel you just built, check out -:ref:`Supplementary tasks: cleanup during and after following this guide.<introclosure_bissbs>`. +:ref:`Complementary tasks: cleanup during and after following this guide <introclosure_bissbs>`. In case you face a regression, move on and execute at least the next segment as well. @@ -649,7 +710,7 @@ Segment 2: check if the kernels you build work fine In case of a regression, you now want to ensure the trimmed configuration file you created earlier works as expected; a bisection with the .config file -otherwise would be a waste of time. [:ref:`details<introworkingcheck_bisref>`] +otherwise would be a waste of time. [:ref:`details <introworkingcheck_bisref>`] .. _recheckworking_bissbs: @@ -660,7 +721,7 @@ otherwise would be a waste of time. [:ref:`details<introworkingcheck_bisref>`] 'good' (once again assumed to be 6.0 here):: cd ~/linux/ - git checkout --detach v6.0 + git switch --discard-changes --detach v6.0 Now use the checked out code to configure, build, and install another kernel using the commands the previous subsection explained in more detail:: @@ -670,12 +731,13 @@ otherwise would be a waste of time. [:ref:`details<introworkingcheck_bisref>`] make -j $(nproc --all) # * Check if the free space suffices holding another kernel: df -h /boot/ /lib/modules/ - command -v installkernel && sudo make modules_install install + sudo make modules_install + command -v installkernel && sudo make install make -s kernelrelease | tee -a ~/kernels-built reboot When the system booted, you may want to verify once again that the - kernel you started is the one you just built: + kernel you started is the one you just built:: tail -n 1 ~/kernels-built uname -r @@ -683,7 +745,7 @@ otherwise would be a waste of time. [:ref:`details<introworkingcheck_bisref>`] Now check if this kernel works as expected; if not, consult the reference section for further instructions. - [:ref:`details<recheckworking_bisref>`] + [:ref:`details <recheckworking_bisref>`] .. _introbisect_bissbs: @@ -693,18 +755,11 @@ Segment 3: perform the bisection and validate the result With all the preparations and precaution builds taken care of, you are now ready to begin the bisection. This will make you build quite a few kernels -- usually about 15 in case you encountered a regression when updating to a newer series -(say from 6.0.11 to 6.1.3). But do not worry, due to the trimmed build +(say from 6.0.13 to 6.1.5). But do not worry, due to the trimmed build configuration created earlier this works a lot faster than many people assume: overall on average it will often just take about 10 to 15 minutes to compile each kernel on commodity x86 machines. -* In case your 'bad' version is a stable/longterm release (say v6.1.5), add its - stable branch, unless you already did so earlier:: - - cd ~/linux/ - git remote set-branches --add stable linux-6.1.y - git fetch stable - .. _bisectstart_bissbs: * Start the bisection and tell Git about the versions earlier established as @@ -715,7 +770,7 @@ each kernel on commodity x86 machines. git bisect good v6.0 git bisect bad v6.1.5 - [:ref:`details<bisectstart_bisref>`] + [:ref:`details <bisectstart_bisref>`] .. _bisectbuild_bissbs: @@ -727,14 +782,15 @@ each kernel on commodity x86 machines. make -j $(nproc --all) # * Check if the free space suffices holding another kernel: df -h /boot/ /lib/modules/ - command -v installkernel && sudo make modules_install install + sudo make modules_install + command -v installkernel && sudo make install make -s kernelrelease | tee -a ~/kernels-built reboot If compilation fails for some reason, run ``git bisect skip`` and restart executing the stack of commands from the beginning. - In case you skipped the "test latest codebase" step in the guide, check its + In case you skipped the 'test latest codebase' step in the guide, check its description as for why the 'df [...]' and 'make -s kernelrelease [...]' commands are here. @@ -743,7 +799,7 @@ each kernel on commodity x86 machines. totally normal to see release identifiers like '6.0-rc1-local-gcafec0cacaca0' if you bisect between versions 6.1 and 6.2 for example. - [:ref:`details<bisectbuild_bisref>`] + [:ref:`details <bisectbuild_bisref>`] .. _bisecttest_bissbs: @@ -783,7 +839,7 @@ each kernel on commodity x86 machines. might need to scroll up to see the message mentioning the culprit; alternatively, run ``git bisect log > ~/bisection-log``. - [:ref:`details<bisecttest_bisref>`] + [:ref:`details <bisecttest_bisref>`] .. _bisectlog_bissbs: @@ -795,7 +851,7 @@ each kernel on commodity x86 machines. cp .config ~/bisection-config-culprit git bisect reset - [:ref:`details<bisectlog_bisref>`] + [:ref:`details <bisectlog_bisref>`] .. _revert_bissbs: @@ -812,16 +868,16 @@ each kernel on commodity x86 machines. Begin by checking out the latest codebase depending on the range you bisected: * Did you face a regression within a stable/longterm series (say between - 6.0.11 and 6.0.13) that does not happen in mainline? Then check out the + 6.0.13 and 6.0.15) that does not happen in mainline? Then check out the latest codebase for the affected series like this:: git fetch stable - git checkout --force --detach linux-6.0.y + git switch --discard-changes --detach linux-6.0.y * In all other cases check out latest mainline:: git fetch mainline - git checkout --force --detach mainline/master + git switch --discard-changes --detach mainline/master If you bisected a regression within a stable/longterm series that also happens in mainline, there is one more thing to do: look up the mainline @@ -835,26 +891,33 @@ each kernel on commodity x86 machines. git revert --no-edit cafec0cacaca0 - If that fails, give up trying and move on to the next step. But if it works, - build a kernel again using the familiar command sequence:: + If that fails, give up trying and move on to the next step; if it works, + adjust the tag to facilitate the identification and prevent accidentally + overwriting another kernel:: cp ~/kernel-config-working .config + ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted' + + Build a kernel using the familiar command sequence, just without copying the + the base .config over:: + make olddefconfig && - make -j $(nproc --all) && + make -j $(nproc --all) # * Check if the free space suffices holding another kernel: df -h /boot/ /lib/modules/ - command -v installkernel && sudo make modules_install install - Make -s kernelrelease | tee -a ~/kernels-built + sudo make modules_install + command -v installkernel && sudo make install + make -s kernelrelease | tee -a ~/kernels-built reboot - Now check one last time if the feature that made you perform a bisection work - with that kernel. + Now check one last time if the feature that made you perform a bisection works + with that kernel: if everything went well, it should not show the regression. - [:ref:`details<revert_bisref>`] + [:ref:`details <revert_bisref>`] .. _introclosure_bissbs: -Supplementary tasks: cleanup during and after the bisection +Complementary tasks: cleanup during and after the bisection ----------------------------------------------------------- During and after following this guide you might want or need to remove some of @@ -891,7 +954,7 @@ space might run out. kernel image and related files behind; in that case remove them as described in the reference section. - [:ref:`details<makeroom_bisref>`] + [:ref:`details <makeroom_bisref>`] .. _finishingtouch_bissbs: @@ -914,18 +977,99 @@ space might run out. the version considered 'good', and the last three or four you compiled during the actual bisection process. - [:ref:`details<finishingtouch_bisref>`] + [:ref:`details <finishingtouch_bisref>`] + +.. _introoptional_bissbs: + +Optional: test reverts, patches, or later versions +-------------------------------------------------- + +While or after reporting a bug, you might want or potentially will be asked to +test reverts, debug patches, proposed fixes, or other versions. In that case +follow these instructions. + +* Update your Git clone and check out the latest code. + + * In case you want to test mainline, fetch its latest changes before checking + its code out:: + + git fetch mainline + git switch --discard-changes --detach mainline/master + + * In case you want to test a stable or longterm kernel, first add the branch + holding the series you are interested in (6.2 in the example), unless you + already did so earlier:: + + git remote set-branches --add stable linux-6.2.y + + Then fetch the latest changes and check out the latest version from the + series:: + + git fetch stable + git switch --discard-changes --detach stable/linux-6.2.y + +* Copy your kernel build configuration over:: + + cp ~/kernel-config-working .config + +* Your next step depends on what you want to do: + + * In case you just want to test the latest codebase, head to the next step, + you are already all set. + + * In case you want to test if a revert fixes an issue, revert one or multiple + changes by specifying their commit ids:: + + git revert --no-edit cafec0cacaca0 + + Now give that kernel a special tag to facilitates its identification and + prevent accidentally overwriting another kernel:: + + ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-reverted' + + * In case you want to test a patch, store the patch in a file like + '/tmp/foobars-proposed-fix-v1.patch' and apply it like this:: + + git apply /tmp/foobars-proposed-fix-v1.patch + + In case of multiple patches, repeat this step with the others. + + Now give that kernel a special tag to facilitates its identification and + prevent accidentally overwriting another kernel:: + + ./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1' + +* Build a kernel using the familiar commands, just without copying the kernel + build configuration over, as that has been taken care of already:: + + make olddefconfig && + make -j $(nproc --all) + # * Check if the free space suffices holding another kernel: + df -h /boot/ /lib/modules/ + sudo make modules_install + command -v installkernel && sudo make install + make -s kernelrelease | tee -a ~/kernels-built + reboot + +* Now verify you booted the newly built kernel and check it. + +[:ref:`details <introoptional_bisref>`] .. _submit_improvements: -This concludes the step-by-step guide. +Conclusion +---------- + +You have reached the end of the step-by-step guide. Did you run into trouble following any of the above steps not cleared up by the reference section below? Did you spot errors? Or do you have ideas how to -improve the guide? Then please take a moment and let the maintainer of this +improve the guide? + +If any of that applies, please take a moment and let the maintainer of this document know by email (Thorsten Leemhuis <linux@leemhuis.info>), ideally while CCing the Linux docs mailing list (linux-doc@vger.kernel.org). Such feedback is -vital to improve this document further, which is in everybody's interest, as it +vital to improve this text further, which is in everybody's interest, as it will enable more people to master the task described here -- and hopefully also improve similar guides inspired by this one. @@ -936,10 +1080,20 @@ Reference section for the step-by-step guide This section holds additional information for almost all the items in the above step-by-step guide. +Preparations for building your own kernels +------------------------------------------ + + *The steps in this section lay the groundwork for all further tests.* + [:ref:`... <introprep_bissbs>`] + +The steps in all later sections of this guide depend on those described here. + +[:ref:`back to step-by-step guide <introprep_bissbs>`]. + .. _backup_bisref: Prepare for emergencies ------------------------ +~~~~~~~~~~~~~~~~~~~~~~~ *Create a fresh backup and put system repair and restore tools at hand.* [:ref:`... <backup_bissbs>`] @@ -954,7 +1108,7 @@ for something going sideways, even if that should not happen. .. _vanilla_bisref: Remove anything related to externally maintained kernel modules ---------------------------------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *Remove all software that depends on externally developed kernel drivers or builds them automatically.* [:ref:`...<vanilla_bissbs>`] @@ -972,7 +1126,7 @@ explains in more detail. .. _secureboot_bisref: Deal with techniques like Secure Boot -------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *On platforms with 'Secure Boot' or similar techniques, prepare everything to ensure the system will permit your self-compiled kernel to boot later.* @@ -1009,7 +1163,7 @@ Afterwards, permit MokManager to reboot the machine. .. _bootworking_bisref: Boot the last kernel that was working -------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *Boot into the last working kernel and briefly recheck if the feature that regressed really works.* [:ref:`...<bootworking_bissbs>`] @@ -1022,7 +1176,7 @@ the right thing. .. _diskspace_bisref: Space requirements ------------------- +~~~~~~~~~~~~~~~~~~ *Ensure to have enough free space for building Linux.* [:ref:`... <diskspace_bissbs>`] @@ -1040,32 +1194,32 @@ space by quite a few gigabytes. .. _rangecheck_bisref: Bisection range ---------------- +~~~~~~~~~~~~~~~ *Determine the kernel versions considered 'good' and 'bad' throughout this guide.* [:ref:`...<rangecheck_bissbs>`] Establishing the range of commits to be checked is mostly straightforward, except when a regression occurred when switching from a release of one stable -series to a release of a later series (e.g. from 6.0.11 to 6.1.4). In that case +series to a release of a later series (e.g. from 6.0.13 to 6.1.5). In that case Git will need some hand holding, as there is no straight line of descent. That's because with the release of 6.0 mainline carried on to 6.1 while the stable series 6.0.y branched to the side. It's therefore theoretically possible -that the issue you face with 6.1.4 only worked in 6.0.11, as it was fixed by a +that the issue you face with 6.1.5 only worked in 6.0.13, as it was fixed by a commit that went into one of the 6.0.y releases, but never hit mainline or the 6.1.y series. Thankfully that normally should not happen due to the way the stable/longterm maintainers maintain the code. It's thus pretty safe to assume 6.0 as a 'good' kernel. That assumption will be tested anyway, as that kernel will be built and tested in the segment '2' of this guide; Git would force you -to do this as well, if you tried bisecting between 6.0.11 and 6.1.13. +to do this as well, if you tried bisecting between 6.0.13 and 6.1.15. [:ref:`back to step-by-step guide <rangecheck_bissbs>`] .. _buildrequires_bisref: Install build requirements --------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~ *Install all software required to build a Linux kernel.* [:ref:`...<buildrequires_bissbs>`] @@ -1105,7 +1259,7 @@ These commands install a few packages that are often, but not always needed. You for example might want to skip installing the development headers for ncurses, which you will only need in case you later might want to adjust the kernel build configuration using make the targets 'menuconfig' or 'nconfig'; likewise omit -the headers of Qt6 is you do not plan to adjust the .config using 'xconfig'. +the headers of Qt6 if you do not plan to adjust the .config using 'xconfig'. You furthermore might need additional libraries and their development headers for tasks not covered in this guide -- for example when building utilities from @@ -1116,7 +1270,7 @@ the kernel's tools/ directory. .. _sources_bisref: Download the sources using Git ------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *Retrieve the Linux mainline sources.* [:ref:`...<sources_bissbs>`] @@ -1126,17 +1280,17 @@ Git clone of Linus' mainline repository. There is nothing more to say about that -- but there are two alternatives ways to retrieve the sources that might work better for you: - * If you have an unreliable internet connection, consider - :ref:`using a 'Git bundle'<sources_bundle_bisref>`. +* If you have an unreliable internet connection, consider + :ref:`using a 'Git bundle'<sources_bundle_bisref>`. - * If downloading the complete repository would take too long or requires too - much storage space, consider :ref:`using a 'shallow - clone'<sources_shallow_bisref>`. +* If downloading the complete repository would take too long or requires too + much storage space, consider :ref:`using a 'shallow + clone'<sources_shallow_bisref>`. .. _sources_bundle_bisref: Downloading Linux mainline sources using a bundle -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +""""""""""""""""""""""""""""""""""""""""""""""""" Use the following commands to retrieve the Linux mainline sources using a bundle:: @@ -1172,7 +1326,7 @@ First, execute the following command to retrieve the latest mainline codebase:: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git Now deepen your clone's history to the second predecessor of the mainline -release of your 'good' version. In case the latter are 6.0 or 6.0.11, 5.19 would +release of your 'good' version. In case the latter are 6.0 or 6.0.13, 5.19 would be the first predecessor and 5.18 the second -- hence deepen the history up to that version:: @@ -1183,23 +1337,23 @@ branches as explained in the step-by-step guide. Note, shallow clones have a few peculiar characteristics: - * For bisections the history needs to be deepened a few mainline versions - farther than it seems necessary, as explained above already. That's because - Git otherwise will be unable to revert or describe most of the commits within - a range (say v6.1..v6.2), as they are internally based on earlier kernels - releases (like v6.0-rc2 or 5.19-rc3). +* For bisections the history needs to be deepened a few mainline versions + farther than it seems necessary, as explained above already. That's because + Git otherwise will be unable to revert or describe most of the commits within + a range (say 6.1..6.2), as they are internally based on earlier kernels + releases (like 6.0-rc2 or 5.19-rc3). - * This document in most places uses ``git fetch`` with ``--shallow-exclude=`` - to specify the earliest version you care about (or to be precise: its git - tag). You alternatively can use the parameter ``--shallow-since=`` to specify - an absolute (say ``'2023-07-15'``) or relative (``'12 months'``) date to - define the depth of the history you want to download. When using them while - bisecting mainline, ensure to deepen the history to at least 7 months before - the release of the mainline release your 'good' kernel is based on. +* This document in most places uses ``git fetch`` with ``--shallow-exclude=`` + to specify the earliest version you care about (or to be precise: its git + tag). You alternatively can use the parameter ``--shallow-since=`` to specify + an absolute (say ``'2023-07-15'``) or relative (``'12 months'``) date to + define the depth of the history you want to download. When using them while + bisecting mainline, ensure to deepen the history to at least 7 months before + the release of the mainline release your 'good' kernel is based on. - * Be warned, when deepening your clone you might encounter an error like - 'fatal: error in object: unshallow cafecaca0c0dacafecaca0c0dacafecaca0c0da'. - In that case run ``git repack -d`` and try again. +* Be warned, when deepening your clone you might encounter an error like + 'fatal: error in object: unshallow cafecaca0c0dacafecaca0c0dacafecaca0c0da'. + In that case run ``git repack -d`` and try again. [:ref:`back to step-by-step guide <sources_bissbs>`] [:ref:`back to section intro <sources_bisref>`] @@ -1207,7 +1361,7 @@ Note, shallow clones have a few peculiar characteristics: .. _oldconfig_bisref: Start defining the build configuration for your kernel ------------------------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *Start preparing a kernel build configuration (the '.config' file).* [:ref:`... <oldconfig_bissbs>`] @@ -1223,23 +1377,23 @@ locate the right build configuration.* Two things can easily go wrong when creating a .config file as advised: - * The oldconfig target will use a .config file from your build directory, if - one is already present there (e.g. '~/linux/.config'). That's totally fine if - that's what you intend (see next step), but in all other cases you want to - delete it. This for example is important in case you followed this guide - further, but due to problems come back here to redo the configuration from - scratch. +* The oldconfig target will use a .config file from your build directory, if + one is already present there (e.g. '~/linux/.config'). That's totally fine if + that's what you intend (see next step), but in all other cases you want to + delete it. This for example is important in case you followed this guide + further, but due to problems come back here to redo the configuration from + scratch. - * Sometimes olddefconfig is unable to locate the .config file for your running - kernel and will use defaults, as briefly outlined in the guide. In that case - check if your distribution ships the configuration somewhere and manually put - it in the right place (e.g. '~/linux/.config') if it does. On distributions - where /proc/config.gz exists this can be achieved using this command:: +* Sometimes olddefconfig is unable to locate the .config file for your running + kernel and will use defaults, as briefly outlined in the guide. In that case + check if your distribution ships the configuration somewhere and manually put + it in the right place (e.g. '~/linux/.config') if it does. On distributions + where /proc/config.gz exists this can be achieved using this command:: - zcat /proc/config.gz > .config + zcat /proc/config.gz > .config - Once you put it there, run ``make olddefconfig`` again to adjust it to the - needs of the kernel about to be built. + Once you put it there, run ``make olddefconfig`` again to adjust it to the + needs of the kernel about to be built. Note, the olddefconfig target will set any undefined build options to their default value. If you prefer to set such configuration options manually, use @@ -1252,7 +1406,7 @@ restrictions). Occasionally odd things happen when trying to use a config file prepared for one kernel (say 6.1) on an older mainline release -- especially if it is much older -(say v5.15). That's one of the reasons why the previous step in the guide told +(say 5.15). That's one of the reasons why the previous step in the guide told you to boot the kernel where everything works. If you manually add a .config file you thus want to ensure it's from the working kernel and not from a one that shows the regression. @@ -1267,7 +1421,7 @@ that file to the build machine and store it as ~/linux/.config; afterwards run .. _localmodconfig_bisref: Trim the build configuration for your kernel --------------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *Disable any kernel modules apparently superfluous for your setup.* [:ref:`... <localmodconfig_bissbs>`] @@ -1316,7 +1470,7 @@ step-by-step guide mentions:: .. _tagging_bisref: Tag the kernels about to be build ---------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *Ensure all the kernels you will build are clearly identifiable using a special tag and a unique version identifier.* [:ref:`... <tagging_bissbs>`] @@ -1332,7 +1486,7 @@ confusing during the bisection. .. _debugsymbols_bisref: Decide to enable or disable debug symbols ------------------------------------------ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *Decide how to handle debug symbols.* [:ref:`... <debugsymbols_bissbs>`] @@ -1361,7 +1515,7 @@ explains this process in more detail. .. _configmods_bisref: Adjust build configuration --------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~ *Check if you may want or need to adjust some other kernel configuration options:* @@ -1372,7 +1526,7 @@ kernel configuration options. .. _configmods_distros_bisref: Distro specific adjustments -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +""""""""""""""""""""""""""" *Are you running* [:ref:`... <configmods_bissbs>`] @@ -1381,46 +1535,46 @@ when following this guide on a few commodity distributions. **Debian:** - * Remove a stale reference to a certificate file that would cause your build to - fail:: +* Remove a stale reference to a certificate file that would cause your build to + fail:: - ./scripts/config --set-str SYSTEM_TRUSTED_KEYS '' + ./scripts/config --set-str SYSTEM_TRUSTED_KEYS '' - Alternatively, download the needed certificate and make that configuration - option point to it, as `the Debian handbook explains in more detail - <https://debian-handbook.info/browse/stable/sect.kernel-compilation.html>`_ - -- or generate your own, as explained in - Documentation/admin-guide/module-signing.rst. + Alternatively, download the needed certificate and make that configuration + option point to it, as `the Debian handbook explains in more detail + <https://debian-handbook.info/browse/stable/sect.kernel-compilation.html>`_ + -- or generate your own, as explained in + Documentation/admin-guide/module-signing.rst. [:ref:`back to step-by-step guide <configmods_bissbs>`] .. _configmods_individual_bisref: Individual adjustments -~~~~~~~~~~~~~~~~~~~~~~ +"""""""""""""""""""""" *If you want to influence the other aspects of the configuration, do so now.* [:ref:`... <configmods_bissbs>`] -You at this point can use a command like ``make menuconfig`` to enable or -disable certain features using a text-based user interface; to use a graphical -configuration utility, call the make target ``xconfig`` or ``gconfig`` instead. -All of them require development libraries from toolkits they are based on -(ncurses, Qt5, Gtk2); an error message will tell you if something required is -missing. +At this point you can use a command like ``make menuconfig`` or ``make nconfig`` +to enable or disable certain features using a text-based user interface; to use +a graphical configuration utility, run ``make xconfig`` instead. Both of them +require development libraries from toolkits they are rely on (ncurses +respectively Qt5 or Qt6); an error message will tell you if something required +is missing. [:ref:`back to step-by-step guide <configmods_bissbs>`] .. _saveconfig_bisref: Put the .config file aside --------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~ *Reprocess the .config after the latest changes and store it in a safe place.* [:ref:`... <saveconfig_bissbs>`] Put the .config you prepared aside, as you want to copy it back to the build -directory every time during this guide before you start building another +directory every time during this guide before you start building another kernel. That's because going back and forth between different versions can alter .config files in odd ways; those occasionally cause side effects that could confuse testing or in some cases render the result of your bisection @@ -1430,8 +1584,8 @@ meaningless. .. _introlatestcheck_bisref: -Try to reproduce the regression ------------------------------------------ +Try to reproduce the problem with the latest codebase +----------------------------------------------------- *Verify the regression is not caused by some .config change and check if it still occurs with the latest codebase.* [:ref:`... <introlatestcheck_bissbs>`] @@ -1478,28 +1632,28 @@ highly recommended for these reasons: Your report might be ignored if you send it to the wrong party -- and even when you get a reply there is a decent chance that developers tell you to - evaluate which of the two cases it is before they take a closer look. + evaluate which of the two cases it is before they take a closer look. [:ref:`back to step-by-step guide <introlatestcheck_bissbs>`] .. _checkoutmaster_bisref: -Checkout the latest Linux codebase ----------------------------------- +Check out the latest Linux codebase +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - *Checkout the latest Linux codebase.* - [:ref:`... <introlatestcheck_bissbs>`] + *Check out the latest Linux codebase.* + [:ref:`... <checkoutmaster_bissbs>`] In case you later want to recheck if an ever newer codebase might fix the problem, remember to run that ``git fetch --shallow-exclude [...]`` command again mentioned earlier to update your local Git repository. -[:ref:`back to step-by-step guide <introlatestcheck_bissbs>`] +[:ref:`back to step-by-step guide <checkoutmaster_bissbs>`] .. _build_bisref: Build your kernel ------------------ +~~~~~~~~~~~~~~~~~ *Build the image and the modules of your first kernel using the config file you prepared.* [:ref:`... <build_bissbs>`] @@ -1509,13 +1663,13 @@ yourself. Another subsection explains how to directly package your kernel up as deb, rpm or tar file. Dealing with build errors -~~~~~~~~~~~~~~~~~~~~~~~~~ +""""""""""""""""""""""""" When a build error occurs, it might be caused by some aspect of your machine's setup that often can be fixed quickly; other times though the problem lies in the code and can only be fixed by a developer. A close examination of the failure messages coupled with some research on the internet will often tell you -which of the two it is. To perform such a investigation, restart the build +which of the two it is. To perform such investigation, restart the build process like this:: make V=1 @@ -1538,24 +1692,24 @@ often one of the hits will provide a solution for your problem, too. If you do not find anything that matches your problem, try again from a different angle by modifying your search terms or using another line from the error messages. -In the end, most trouble you are to run into has likely been encountered and +In the end, most issues you run into have likely been encountered and reported by others already. That includes issues where the cause is not your -system, but lies the code. If you run into one of those, you might thus find a -solution (e.g. a patch) or workaround for your problem, too. +system, but lies in the code. If you run into one of those, you might thus find +a solution (e.g. a patch) or workaround for your issue, too. Package your kernel up -~~~~~~~~~~~~~~~~~~~~~~ +"""""""""""""""""""""" The step-by-step guide uses the default make targets (e.g. 'bzImage' and 'modules' on x86) to build the image and the modules of your kernel, which later steps of the guide then install. You instead can also directly build everything and directly package it up by using one of the following targets: - * ``make -j $(nproc --all) bindeb-pkg`` to generate a deb package +* ``make -j $(nproc --all) bindeb-pkg`` to generate a deb package - * ``make -j $(nproc --all) binrpm-pkg`` to generate a rpm package +* ``make -j $(nproc --all) binrpm-pkg`` to generate a rpm package - * ``make -j $(nproc --all) tarbz2-pkg`` to generate a bz2 compressed tarball +* ``make -j $(nproc --all) tarbz2-pkg`` to generate a bz2 compressed tarball This is just a selection of available make targets for this purpose, see ``make help`` for others. You can also use these targets after running @@ -1575,51 +1729,50 @@ distribution's kernel packages. .. _install_bisref: Put the kernel in place ------------------------ +~~~~~~~~~~~~~~~~~~~~~~~ *Install the kernel you just built.* [:ref:`... <install_bissbs>`] What you need to do after executing the command in the step-by-step guide -depends on the existence and the implementation of an ``installkernel`` -executable. Many commodity Linux distributions ship such a kernel installer in -'/sbin/' that does everything needed, hence there is nothing left for you -except rebooting. But some distributions contain an installkernel that does -only part of the job -- and a few lack it completely and leave all the work to -you. - -If ``installkernel`` is found, the kernel's build system will delegate the -actual installation of your kernel's image and related files to this executable. -On almost all Linux distributions it will store the image as '/boot/vmlinuz- -<kernelrelease identifier>' and put a 'System.map-<kernelrelease -identifier>' alongside it. Your kernel will thus be installed in parallel to any -existing ones, unless you already have one with exactly the same release name. - -Installkernel on many distributions will afterwards generate an 'initramfs' -(often also called 'initrd'), which commodity distributions rely on for booting; -hence be sure to keep the order of the two make targets used in the step-by-step -guide, as things will go sideways if you install your kernel's image before its -modules. Often installkernel will then add your kernel to the bootloader -configuration, too. You have to take care of one or both of these tasks -yourself, if your distributions installkernel doesn't handle them. - -A few distributions like Arch Linux and its derivatives totally lack an -installkernel executable. On those just install the modules using the kernel's -build system and then install the image and the System.map file manually:: - - sudo make modules_install +depends on the existence and the implementation of ``/sbin/installkernel`` +executable on your distribution. + +If installkernel is found, the kernel's build system will delegate the actual +installation of your kernel image to this executable, which then performs some +or all of these tasks: + +* On almost all Linux distributions installkernel will store your kernel's + image in /boot/, usually as '/boot/vmlinuz-<kernelrelease_id>'; often it will + put a 'System.map-<kernelrelease_id>' alongside it. + +* On most distributions installkernel will then generate an 'initramfs' + (sometimes also called 'initrd'), which usually are stored as + '/boot/initramfs-<kernelrelease_id>.img' or + '/boot/initrd-<kernelrelease_id>'. Commodity distributions rely on this file + for booting, hence ensure to execute the make target 'modules_install' first, + as your distribution's initramfs generator otherwise will be unable to find + the modules that go into the image. + +* On some distributions installkernel will then add an entry for your kernel + to your bootloader's configuration. + +You have to take care of some or all of the tasks yourself, if your +distribution lacks a installkernel script or does only handle part of them. +Consult the distribution's documentation for details. If in doubt, install the +kernel manually:: + sudo install -m 0600 $(make -s image_name) /boot/vmlinuz-$(make -s kernelrelease) sudo install -m 0600 System.map /boot/System.map-$(make -s kernelrelease) -If your distribution boots with the help of an initramfs, now generate one for -your kernel using the tools your distribution provides for this process. -Afterwards add your kernel to your bootloader configuration and reboot. +Now generate your initramfs using the tools your distribution provides for this +process. Afterwards add your kernel to your bootloader configuration and reboot. [:ref:`back to step-by-step guide <install_bissbs>`] .. _storagespace_bisref: Storage requirements per kernel -------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *Check how much storage space the kernel, its modules, and other related files like the initramfs consume.* [:ref:`... <storagespace_bissbs>`] @@ -1637,20 +1790,39 @@ need to look in different places. [:ref:`back to step-by-step guide <storagespace_bissbs>`] +.. _tainted_bisref: + +Check if your newly built kernel considers itself 'tainted' +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + *Check if the kernel marked itself as 'tainted'.* + [:ref:`... <tainted_bissbs>`] + +Linux marks itself as tainted when something happens that potentially leads to +follow-up errors that look totally unrelated. That is why developers might +ignore or react scantly to reports from tainted kernels -- unless of course the +kernel set the flag right when the reported bug occurred. + +That's why you want check why a kernel is tainted as explained in +Documentation/admin-guide/tainted-kernels.rst; doing so is also in your own +interest, as your testing might be flawed otherwise. + +[:ref:`back to step-by-step guide <tainted_bissbs>`] + .. _recheckbroken_bisref: -Check the kernel built from the latest codebase ------------------------------------------------ +Check the kernel built from a recent mainline codebase +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - *Reboot into the kernel you just built and check if the feature that regressed - is really broken there.* [:ref:`... <recheckbroken_bissbs>`] + *Verify if your bug occurs with the newly built kernel.* + [:ref:`... <recheckbroken_bissbs>`] -There are a couple of reasons why the regression you face might not show up with -your own kernel built from the latest codebase. These are the most frequent: +There are a couple of reasons why your bug or regression might not show up with +the kernel you built from the latest codebase. These are the most frequent: -* The cause for the regression was fixed meanwhile. +* The bug was fixed meanwhile. -* The regression with the broken kernel was caused by a change in the build +* What you suspected to be a regression was caused by a change in the build configuration the provider of your kernel carried out. * Your problem might be a race condition that does not show up with your kernel; @@ -1666,7 +1838,7 @@ your own kernel built from the latest codebase. These are the most frequent: .. _recheckstablebroken_bisref: Check the kernel built from the latest stable/longterm codebase ---------------------------------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *Are you facing a regression within a stable/longterm release, but failed to reproduce it with the kernel you just built using the latest mainline sources? @@ -1702,16 +1874,16 @@ it's possible that the thing that regressed might never have worked in vanilla builds of the 'good' version in the first place. There is a third reason for those that noticed a regression between -stable/longterm kernels of different series (e.g. v6.0.13..v6.1.5): it will +stable/longterm kernels of different series (e.g. 6.0.13..6.1.5): it will ensure the kernel version you assumed to be 'good' earlier in the process (e.g. -v6.0) actually is working. +6.0) actually is working. [:ref:`back to step-by-step guide <introworkingcheck_bissbs>`] .. _recheckworking_bisref: Build your own version of the 'good' kernel -------------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *Build your own variant of the working kernel and check if the feature that regressed works as expected with it.* [:ref:`... <recheckworking_bissbs>`] @@ -1720,6 +1892,9 @@ In case the feature that broke with newer kernels does not work with your first self-built kernel, find and resolve the cause before moving on. There are a multitude of reasons why this might happen. Some ideas where to look: +* Check the taint status and the output of ``dmesg``, maybe something unrelated + went wrong. + * Maybe localmodconfig did something odd and disabled the module required to test the feature? Then you might want to recreate a .config file based on the one from the last working kernel and skip trimming it down; manually disabling @@ -1734,29 +1909,39 @@ multitude of reasons why this might happen. Some ideas where to look: Note, if you found and fixed problems with the .config file, you want to use it to build another kernel from the latest codebase, as your earlier tests with -mainline and the latest version from an affected stable/longterm series most -likely has been flawed. +mainline and the latest version from an affected stable/longterm series were +most likely flawed. [:ref:`back to step-by-step guide <recheckworking_bissbs>`] +Perform a bisection and validate the result +------------------------------------------- + + *With all the preparations and precaution builds taken care of, you are now + ready to begin the bisection.* [:ref:`... <introbisect_bissbs>`] + +The steps in this segment perform and validate the bisection. + +[:ref:`back to step-by-step guide <introbisect_bissbs>`]. + .. _bisectstart_bisref: Start the bisection -------------------- +~~~~~~~~~~~~~~~~~~~ *Start the bisection and tell Git about the versions earlier established as 'good' and 'bad'.* [:ref:`... <bisectstart_bissbs>`] This will start the bisection process; the last of the commands will make Git -checkout a commit round about half-way between the 'good' and the 'bad' changes -for your to test. +check out a commit round about half-way between the 'good' and the 'bad' changes +for you to test. [:ref:`back to step-by-step guide <bisectstart_bissbs>`] .. _bisectbuild_bisref: Build a kernel from the bisection point ---------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *Build, install, and boot a kernel from the code Git checked out using the same commands you used earlier.* [:ref:`... <bisectbuild_bissbs>`] @@ -1774,7 +1959,7 @@ There are two things worth of note here: * Those slightly odd looking version identifiers can happen during bisections, because the Linux kernel subsystems prepare their changes for a new mainline release (say 6.2) before its predecessor (e.g. 6.1) is finished. They thus - base them on a somewhat earlier point like v6.1-rc1 or even v6.0 -- and then + base them on a somewhat earlier point like 6.1-rc1 or even 6.0 -- and then get merged for 6.2 without rebasing nor squashing them once 6.1 is out. This leads to those slightly odd looking version identifiers coming up during bisections. @@ -1784,13 +1969,13 @@ There are two things worth of note here: .. _bisecttest_bisref: Bisection checkpoint --------------------- +~~~~~~~~~~~~~~~~~~~~ *Check if the feature that regressed works in the kernel you just built.* [:ref:`... <bisecttest_bissbs>`] Ensure what you tell Git is accurate: getting it wrong just one time will bring -the rest of the bisection totally of course, hence all testing after that point +the rest of the bisection totally off course, hence all testing after that point will be for nothing. [:ref:`back to step-by-step guide <bisecttest_bissbs>`] @@ -1798,7 +1983,7 @@ will be for nothing. .. _bisectlog_bisref: Put the bisection log away --------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~ *Store Git's bisection log and the current .config file in a safe place.* [:ref:`... <bisectlog_bissbs>`] @@ -1811,14 +1996,14 @@ instead of testing ten or more kernels you might only have to build a few to resolve things. The .config file is put aside, as there is a decent chance that developers might -ask for it after you reported the regression. +ask for it after you report the regression. [:ref:`back to step-by-step guide <bisectlog_bissbs>`] .. _revert_bisref: Try reverting the culprit -------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~ *Try reverting the culprit on top of the latest codebase to see if this fixes your regression.* [:ref:`... <revert_bissbs>`] @@ -1836,14 +2021,20 @@ succeeds, test that kernel version instead. [:ref:`back to step-by-step guide <revert_bissbs>`] +Cleanup steps during and after following this guide +--------------------------------------------------- -Supplementary tasks: cleanup during and after the bisection ------------------------------------------------------------ + *During and after following this guide you might want or need to remove some + of the kernels you installed.* [:ref:`... <introclosure_bissbs>`] + +The steps in this section describe clean-up procedures. + +[:ref:`back to step-by-step guide <introclosure_bissbs>`]. .. _makeroom_bisref: Cleaning up during the bisection --------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *To remove one of the kernels you installed, look up its 'kernelrelease' identifier.* [:ref:`... <makeroom_bissbs>`] @@ -1861,7 +2052,7 @@ simply remove its modules directory in /lib/modules/. The other place is /boot/, where typically two up to five files will be placed during installation of a kernel. All of them usually contain the release name in -their file name, but how many files and their exact name depends somewhat on +their file name, but how many files and their exact names depend somewhat on your distribution's installkernel executable and its initramfs generator. On some distributions the ``kernel-install remove...`` command mentioned in the step-by-step guide will delete all of these files for you while also removing @@ -1878,13 +2069,13 @@ Now remove the boot entry for the kernel from your bootloader's configuration; the steps to do that vary quite a bit between Linux distributions. Note, be careful with wildcards like '*' when deleting files or directories -for kernels manually: you might accidentally remove files of a 6.0.11 kernel +for kernels manually: you might accidentally remove files of a 6.0.13 kernel when all you want is to remove 6.0 or 6.0.1. [:ref:`back to step-by-step guide <makeroom_bissbs>`] Cleaning up after the bisection -------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. _finishingtouch_bisref: @@ -1899,26 +2090,105 @@ build artifacts and the Linux sources, but will leave the Git repository (~/linux/.git/) behind -- a simple ``git reset --hard`` thus will bring the sources back. -Removing the repository as well would likely be unwise at this point: there is a -decent chance developers will ask you to build another kernel to perform -additional tests. This is often required to debug an issue or check proposed -fixes. Before doing so you want to run the ``git fetch mainline`` command again -followed by ``git checkout mainline/master`` to bring your clone up to date and -checkout the latest codebase. Then apply the patch using ``git apply -<filename>`` or ``git am <filename>`` and build yet another kernel using the -familiar commands. +Removing the repository as well would likely be unwise at this point: there +is a decent chance developers will ask you to build another kernel to +perform additional tests -- like testing a debug patch or a proposed fix. +Details on how to perform those can be found in the section :ref:`Optional +tasks: test reverts, patches, or later versions <introoptional_bissbs>`. Additional tests are also the reason why you want to keep the ~/kernel-config-working file around for a few weeks. [:ref:`back to step-by-step guide <finishingtouch_bissbs>`] +.. _introoptional_bisref: -Additional reading material -=========================== +Test reverts, patches, or later versions +---------------------------------------- + + *While or after reporting a bug, you might want or potentially will be asked + to test reverts, patches, proposed fixes, or other versions.* + [:ref:`... <introoptional_bissbs>`] + +All the commands used in this section should be pretty straight forward, so +there is not much to add except one thing: when setting a kernel tag as +instructed, ensure it is not much longer than the one used in the example, as +problems will arise if the kernelrelease identifier exceeds 63 characters. + +[:ref:`back to step-by-step guide <introoptional_bissbs>`]. + + +Additional information +====================== + +.. _buildhost_bis: + +Build kernels on a different machine +------------------------------------ + +To compile kernels on another system, slightly alter the step-by-step guide's +instructions: + +* Start following the guide on the machine where you want to install and test + the kernels later. + +* After executing ':ref:`Boot into the working kernel and briefly use the + apparently broken feature <bootworking_bissbs>`', save the list of loaded + modules to a file using ``lsmod > ~/test-machine-lsmod``. Then locate the + build configuration for the running kernel (see ':ref:`Start defining the + build configuration for your kernel <oldconfig_bisref>`' for hints on where + to find it) and store it as '~/test-machine-config-working'. Transfer both + files to the home directory of your build host. -Further sources ---------------- +* Continue the guide on the build host (e.g. with ':ref:`Ensure to have enough + free space for building [...] <diskspace_bissbs>`'). + +* When you reach ':ref:`Start preparing a kernel build configuration[...] + <oldconfig_bissbs>`': before running ``make olddefconfig`` for the first time, + execute the following command to base your configuration on the one from the + test machine's 'working' kernel:: + + cp ~/test-machine-config-working ~/linux/.config + +* During the next step to ':ref:`disable any apparently superfluous kernel + modules <localmodconfig_bissbs>`' use the following command instead:: + + yes '' | make localmodconfig LSMOD=~/lsmod_foo-machine localmodconfig + +* Continue the guide, but ignore the instructions outlining how to compile, + install, and reboot into a kernel every time they come up. Instead build + like this:: + + cp ~/kernel-config-working .config + make olddefconfig && + make -j $(nproc --all) targz-pkg + + This will generate a gzipped tar file whose name is printed in the last + line shown; for example, a kernel with the kernelrelease identifier + '6.0.0-rc1-local-g928a87efa423' built for x86 machines usually will + be stored as '~/linux/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz'. + + Copy that file to your test machine's home directory. + +* Switch to the test machine to check if you have enough space to hold another + kernel. Then extract the file you transferred:: + + sudo tar -xvzf ~/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz -C / + + Afterwards :ref:`generate the initramfs and add the kernel to your boot + loader's configuration <install_bisref>`; on some distributions the following + command will take care of both these tasks:: + + sudo /sbin/installkernel 6.0.0-rc1-local-g928a87efa423 /boot/vmlinuz-6.0.0-rc1-local-g928a87efa423 + + Now reboot and ensure you started the intended kernel. + +This approach even works when building for another architecture: just install +cross-compilers and add the appropriate parameters to every invocation of make +(e.g. ``make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- [...]``). + +Additional reading material +--------------------------- * The `man page for 'git bisect' <https://git-scm.com/docs/git-bisect>`_ and `fighting regressions with 'git bisect' <https://git-scm.com/docs/git-bisect-lk2009.html>`_ |