diff options
Diffstat (limited to 'Documentation/dev-tools')
| -rw-r--r-- | Documentation/dev-tools/checkpatch.rst | 7 | ||||
| -rw-r--r-- | Documentation/dev-tools/index.rst | 1 | ||||
| -rw-r--r-- | Documentation/dev-tools/kasan.rst | 17 | ||||
| -rw-r--r-- | Documentation/dev-tools/kmsan.rst | 427 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/architecture.rst | 4 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/faq.rst | 8 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/index.rst | 18 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/kunit-tool.rst | 232 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/run_wrapper.rst | 38 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/start.rst | 138 | ||||
| -rw-r--r-- | Documentation/dev-tools/kunit/usage.rst | 4 |
11 files changed, 576 insertions, 318 deletions
diff --git a/Documentation/dev-tools/checkpatch.rst b/Documentation/dev-tools/checkpatch.rst index b52452bc2963..c3389c6f3838 100644 --- a/Documentation/dev-tools/checkpatch.rst +++ b/Documentation/dev-tools/checkpatch.rst @@ -612,6 +612,13 @@ Commit message See: https://www.kernel.org/doc/html/latest/process/submitting-patches.html#describe-your-changes + **BAD_FIXES_TAG** + The Fixes: tag is malformed or does not follow the community conventions. + This can occur if the tag have been split into multiple lines (e.g., when + pasted in an email program with word wrapping enabled). + + See: https://www.kernel.org/doc/html/latest/process/submitting-patches.html#describe-your-changes + Comparison style ---------------- diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/index.rst index 4621eac290f4..6b0663075dc0 100644 --- a/Documentation/dev-tools/index.rst +++ b/Documentation/dev-tools/index.rst @@ -24,6 +24,7 @@ Documentation/dev-tools/testing-overview.rst kcov gcov kasan + kmsan ubsan kmemleak kcsan diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst index 1772fd457fed..5c93ab915049 100644 --- a/Documentation/dev-tools/kasan.rst +++ b/Documentation/dev-tools/kasan.rst @@ -111,9 +111,17 @@ parameter can be used to control panic and reporting behaviour: report or also panic the kernel (default: ``report``). The panic happens even if ``kasan_multi_shot`` is enabled. -Hardware Tag-Based KASAN mode (see the section about various modes below) is -intended for use in production as a security mitigation. Therefore, it supports -additional boot parameters that allow disabling KASAN or controlling features: +Software and Hardware Tag-Based KASAN modes (see the section about various +modes below) support altering stack trace collection behavior: + +- ``kasan.stacktrace=off`` or ``=on`` disables or enables alloc and free stack + traces collection (default: ``on``). +- ``kasan.stack_ring_size=<number of entries>`` specifies the number of entries + in the stack ring (default: ``32768``). + +Hardware Tag-Based KASAN mode is intended for use in production as a security +mitigation. Therefore, it supports additional boot parameters that allow +disabling KASAN altogether or controlling its features: - ``kasan=off`` or ``=on`` controls whether KASAN is enabled (default: ``on``). @@ -132,9 +140,6 @@ additional boot parameters that allow disabling KASAN or controlling features: - ``kasan.vmalloc=off`` or ``=on`` disables or enables tagging of vmalloc allocations (default: ``on``). -- ``kasan.stacktrace=off`` or ``=on`` disables or enables alloc and free stack - traces collection (default: ``on``). - Error reports ~~~~~~~~~~~~~ diff --git a/Documentation/dev-tools/kmsan.rst b/Documentation/dev-tools/kmsan.rst new file mode 100644 index 000000000000..2a53a801198c --- /dev/null +++ b/Documentation/dev-tools/kmsan.rst @@ -0,0 +1,427 @@ +.. SPDX-License-Identifier: GPL-2.0 +.. Copyright (C) 2022, Google LLC. + +=================================== +The Kernel Memory Sanitizer (KMSAN) +=================================== + +KMSAN is a dynamic error detector aimed at finding uses of uninitialized +values. It is based on compiler instrumentation, and is quite similar to the +userspace `MemorySanitizer tool`_. + +An important note is that KMSAN is not intended for production use, because it +drastically increases kernel memory footprint and slows the whole system down. + +Usage +===== + +Building the kernel +------------------- + +In order to build a kernel with KMSAN you will need a fresh Clang (14.0.6+). +Please refer to `LLVM documentation`_ for the instructions on how to build Clang. + +Now configure and build the kernel with CONFIG_KMSAN enabled. + +Example report +-------------- + +Here is an example of a KMSAN report:: + + ===================================================== + BUG: KMSAN: uninit-value in test_uninit_kmsan_check_memory+0x1be/0x380 [kmsan_test] + test_uninit_kmsan_check_memory+0x1be/0x380 mm/kmsan/kmsan_test.c:273 + kunit_run_case_internal lib/kunit/test.c:333 + kunit_try_run_case+0x206/0x420 lib/kunit/test.c:374 + kunit_generic_run_threadfn_adapter+0x6d/0xc0 lib/kunit/try-catch.c:28 + kthread+0x721/0x850 kernel/kthread.c:327 + ret_from_fork+0x1f/0x30 ??:? + + Uninit was stored to memory at: + do_uninit_local_array+0xfa/0x110 mm/kmsan/kmsan_test.c:260 + test_uninit_kmsan_check_memory+0x1a2/0x380 mm/kmsan/kmsan_test.c:271 + kunit_run_case_internal lib/kunit/test.c:333 + kunit_try_run_case+0x206/0x420 lib/kunit/test.c:374 + kunit_generic_run_threadfn_adapter+0x6d/0xc0 lib/kunit/try-catch.c:28 + kthread+0x721/0x850 kernel/kthread.c:327 + ret_from_fork+0x1f/0x30 ??:? + + Local variable uninit created at: + do_uninit_local_array+0x4a/0x110 mm/kmsan/kmsan_test.c:256 + test_uninit_kmsan_check_memory+0x1a2/0x380 mm/kmsan/kmsan_test.c:271 + + Bytes 4-7 of 8 are uninitialized + Memory access of size 8 starts at ffff888083fe3da0 + + CPU: 0 PID: 6731 Comm: kunit_try_catch Tainted: G B E 5.16.0-rc3+ #104 + Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-2 04/01/2014 + ===================================================== + +The report says that the local variable ``uninit`` was created uninitialized in +``do_uninit_local_array()``. The third stack trace corresponds to the place +where this variable was created. + +The first stack trace shows where the uninit value was used (in +``test_uninit_kmsan_check_memory()``). The tool shows the bytes which were left +uninitialized in the local variable, as well as the stack where the value was +copied to another memory location before use. + +A use of uninitialized value ``v`` is reported by KMSAN in the following cases: + - in a condition, e.g. ``if (v) { ... }``; + - in an indexing or pointer dereferencing, e.g. ``array[v]`` or ``*v``; + - when it is copied to userspace or hardware, e.g. ``copy_to_user(..., &v, ...)``; + - when it is passed as an argument to a function, and + ``CONFIG_KMSAN_CHECK_PARAM_RETVAL`` is enabled (see below). + +The mentioned cases (apart from copying data to userspace or hardware, which is +a security issue) are considered undefined behavior from the C11 Standard point +of view. + +Disabling the instrumentation +----------------------------- + +A function can be marked with ``__no_kmsan_checks``. Doing so makes KMSAN +ignore uninitialized values in that function and mark its output as initialized. +As a result, the user will not get KMSAN reports related to that function. + +Another function attribute supported by KMSAN is ``__no_sanitize_memory``. +Applying this attribute to a function will result in KMSAN not instrumenting +it, which can be helpful if we do not want the compiler to interfere with some +low-level code (e.g. that marked with ``noinstr`` which implicitly adds +``__no_sanitize_memory``). + +This however comes at a cost: stack allocations from such functions will have +incorrect shadow/origin values, likely leading to false positives. Functions +called from non-instrumented code may also receive incorrect metadata for their +parameters. + +As a rule of thumb, avoid using ``__no_sanitize_memory`` explicitly. + +It is also possible to disable KMSAN for a single file (e.g. main.o):: + + KMSAN_SANITIZE_main.o := n + +or for the whole directory:: + + KMSAN_SANITIZE := n + +in the Makefile. Think of this as applying ``__no_sanitize_memory`` to every +function in the file or directory. Most users won't need KMSAN_SANITIZE, unless +their code gets broken by KMSAN (e.g. runs at early boot time). + +Support +======= + +In order for KMSAN to work the kernel must be built with Clang, which so far is +the only compiler that has KMSAN support. The kernel instrumentation pass is +based on the userspace `MemorySanitizer tool`_. + +The runtime library only supports x86_64 at the moment. + +How KMSAN works +=============== + +KMSAN shadow memory +------------------- + +KMSAN associates a metadata byte (also called shadow byte) with every byte of +kernel memory. A bit in the shadow byte is set iff the corresponding bit of the +kernel memory byte is uninitialized. Marking the memory uninitialized (i.e. +setting its shadow bytes to ``0xff``) is called poisoning, marking it +initialized (setting the shadow bytes to ``0x00``) is called unpoisoning. + +When a new variable is allocated on the stack, it is poisoned by default by +instrumentation code inserted by the compiler (unless it is a stack variable +that is immediately initialized). Any new heap allocation done without +``__GFP_ZERO`` is also poisoned. + +Compiler instrumentation also tracks the shadow values as they are used along +the code. When needed, instrumentation code invokes the runtime library in +``mm/kmsan/`` to persist shadow values. + +The shadow value of a basic or compound type is an array of bytes of the same +length. When a constant value is written into memory, that memory is unpoisoned. +When a value is read from memory, its shadow memory is also obtained and +propagated into all the operations which use that value. For every instruction +that takes one or more values the compiler generates code that calculates the +shadow of the result depending on those values and their shadows. + +Example:: + + int a = 0xff; // i.e. 0x000000ff + int b; + int c = a | b; + +In this case the shadow of ``a`` is ``0``, shadow of ``b`` is ``0xffffffff``, +shadow of ``c`` is ``0xffffff00``. This means that the upper three bytes of +``c`` are uninitialized, while the lower byte is initialized. + +Origin tracking +--------------- + +Every four bytes of kernel memory also have a so-called origin mapped to them. +This origin describes the point in program execution at which the uninitialized +value was created. Every origin is associated with either the full allocation +stack (for heap-allocated memory), or the function containing the uninitialized +variable (for locals). + +When an uninitialized variable is allocated on stack or heap, a new origin +value is created, and that variable's origin is filled with that value. When a +value is read from memory, its origin is also read and kept together with the +shadow. For every instruction that takes one or more values, the origin of the +result is one of the origins corresponding to any of the uninitialized inputs. +If a poisoned value is written into memory, its origin is written to the +corresponding storage as well. + +Example 1:: + + int a = 42; + int b; + int c = a + b; + +In this case the origin of ``b`` is generated upon function entry, and is +stored to the origin of ``c`` right before the addition result is written into +memory. + +Several variables may share the same origin address, if they are stored in the +same four-byte chunk. In this case every write to either variable updates the +origin for all of them. We have to sacrifice precision in this case, because +storing origins for individual bits (and even bytes) would be too costly. + +Example 2:: + + int combine(short a, short b) { + union ret_t { + int i; + short s[2]; + } ret; + ret.s[0] = a; + ret.s[1] = b; + return ret.i; + } + +If ``a`` is initialized and ``b`` is not, the shadow of the result would be +0xffff0000, and the origin of the result would be the origin of ``b``. +``ret.s[0]`` would have the same origin, but it will never be used, because +that variable is initialized. + +If both function arguments are uninitialized, only the origin of the second +argument is preserved. + +Origin chaining +~~~~~~~~~~~~~~~ + +To ease debugging, KMSAN creates a new origin for every store of an +uninitialized value to memory. The new origin references both its creation stack +and the previous origin the value had. This may cause increased memory +consumption, so we limit the length of origin chains in the runtime. + +Clang instrumentation API +------------------------- + +Clang instrumentation pass inserts calls to functions defined in +``mm/kmsan/nstrumentation.c`` into the kernel code. + +Shadow manipulation +~~~~~~~~~~~~~~~~~~~ + +For every memory access the compiler emits a call to a function that returns a +pair of pointers to the shadow and origin addresses of the given memory:: + + typedef struct { + void *shadow, *origin; + } shadow_origin_ptr_t + + shadow_origin_ptr_t __msan_metadata_ptr_for_load_{1,2,4,8}(void *addr) + shadow_origin_ptr_t __msan_metadata_ptr_for_store_{1,2,4,8}(void *addr) + shadow_origin_ptr_t __msan_metadata_ptr_for_load_n(void *addr, uintptr_t size) + shadow_origin_ptr_t __msan_metadata_ptr_for_store_n(void *addr, uintptr_t size) + +The function name depends on the memory access size. + +The compiler makes sure that for every loaded value its shadow and origin +values are read from memory. When a value is stored to memory, its shadow and +origin are also stored using the metadata pointers. + +Handling locals +~~~~~~~~~~~~~~~ + +A special function is used to create a new origin value for a local variable and +set the origin of that variable to that value:: + + void __msan_poison_alloca(void *addr, uintptr_t size, char *descr) + +Access to per-task data +~~~~~~~~~~~~~~~~~~~~~~~ + +At the beginning of every instrumented function KMSAN inserts a call to +``__msan_get_context_state()``:: + + kmsan_context_state *__msan_get_context_state(void) + +``kmsan_context_state`` is declared in ``include/linux/kmsan.h``:: + + struct kmsan_context_state { + char param_tls[KMSAN_PARAM_SIZE]; + char retval_tls[KMSAN_RETVAL_SIZE]; + char va_arg_tls[KMSAN_PARAM_SIZE]; + char va_arg_origin_tls[KMSAN_PARAM_SIZE]; + u64 va_arg_overflow_size_tls; + char param_origin_tls[KMSAN_PARAM_SIZE]; + depot_stack_handle_t retval_origin_tls; + }; + +This structure is used by KMSAN to pass parameter shadows and origins between +instrumented functions (unless the parameters are checked immediately by +``CONFIG_KMSAN_CHECK_PARAM_RETVAL``). + +Passing uninitialized values to functions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Clang's MemorySanitizer instrumentation has an option, +``-fsanitize-memory-param-retval``, which makes the compiler check function +parameters passed by value, as well as function return values. + +The option is controlled by ``CONFIG_KMSAN_CHECK_PARAM_RETVAL``, which is +enabled by default to let KMSAN report uninitialized values earlier. +Please refer to the `LKML discussion`_ for more details. + +Because of the way the checks are implemented in LLVM (they are only applied to +parameters marked as ``noundef``), not all parameters are guaranteed to be +checked, so we cannot give up the metadata storage in ``kmsan_context_state``. + +String functions +~~~~~~~~~~~~~~~~ + +The compiler replaces calls to ``memcpy()``/``memmove()``/``memset()`` with the +following functions. These functions are also called when data structures are +initialized or copied, making sure shadow and origin values are copied alongside +with the data:: + + void *__msan_memcpy(void *dst, void *src, uintptr_t n) + void *__msan_memmove(void *dst, void *src, uintptr_t n) + void *__msan_memset(void *dst, int c, uintptr_t n) + +Error reporting +~~~~~~~~~~~~~~~ + +For each use of a value the compiler emits a shadow check that calls +``__msan_warning()`` in the case that value is poisoned:: + + void __msan_warning(u32 origin) + +``__msan_warning()`` causes KMSAN runtime to print an error report. + +Inline assembly instrumentation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +KMSAN instruments every inline assembly output with a call to:: + + void __msan_instrument_asm_store(void *addr, uintptr_t size) + +, which unpoisons the memory region. + +This approach may mask certain errors, but it also helps to avoid a lot of +false positives in bitwise operations, atomics etc. + +Sometimes the pointers passed into inline assembly do not point to valid memory. +In such cases they are ignored at runtime. + + +Runtime library +--------------- + +The code is located in ``mm/kmsan/``. + +Per-task KMSAN state +~~~~~~~~~~~~~~~~~~~~ + +Every task_struct has an associated KMSAN task state that holds the KMSAN +context (see above) and a per-task flag disallowing KMSAN reports:: + + struct kmsan_context { + ... + bool allow_reporting; + struct kmsan_context_state cstate; + ... + } + + struct task_struct { + ... + struct kmsan_context kmsan; + ... + } + +KMSAN contexts +~~~~~~~~~~~~~~ + +When running in a kernel task context, KMSAN uses ``current->kmsan.cstate`` to +hold the metadata for function parameters and return values. + +But in the case the kernel is running in the interrupt, softirq or NMI context, +where ``current`` is unavailable, KMSAN switches to per-cpu interrupt state:: + + DEFINE_PER_CPU(struct kmsan_ctx, kmsan_percpu_ctx); + +Metadata allocation +~~~~~~~~~~~~~~~~~~~ + +There are several places in the kernel for which the metadata is stored. + +1. Each ``struct page`` instance contains two pointers to its shadow and +origin pages:: + + struct page { + ... + struct page *shadow, *origin; + ... + }; + +At boot-time, the kernel allocates shadow and origin pages for every available +kernel page. This is done quite late, when the kernel address space is already +fragmented, so normal data pages may arbitrarily interleave with the metadata +pages. + +This means that in general for two contiguous memory pages their shadow/origin +pages may not be contiguous. Consequently, if a memory access crosses the +boundary of a memory block, accesses to shadow/origin memory may potentially +corrupt other pages or read incorrect values from them. + +In practice, contiguous memory pages returned by the same ``alloc_pages()`` +call will have contiguous metadata, whereas if these pages belong to two +different allocations their metadata pages can be fragmented. + +For the kernel data (``.data``, ``.bss`` etc.) and percpu memory regions +there also are no guarantees on metadata contiguity. + +In the case ``__msan_metadata_ptr_for_XXX_YYY()`` hits the border between two +pages with non-contiguous metadata, it returns pointers to fake shadow/origin regions:: + + char dummy_load_page[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE))); + char dummy_store_page[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE))); + +``dummy_load_page`` is zero-initialized, so reads from it always yield zeroes. +All stores to ``dummy_store_page`` are ignored. + +2. For vmalloc memory and modules, there is a direct mapping between the memory +range, its shadow and origin. KMSAN reduces the vmalloc area by 3/4, making only +the first quarter available to ``vmalloc()``. The second quarter of the vmalloc +area contains shadow memory for the first quarter, the third one holds the +origins. A small part of the fourth quarter contains shadow and origins for the +kernel modules. Please refer to ``arch/x86/include/asm/pgtable_64_types.h`` for +more details. + +When an array of pages is mapped into a contiguous virtual memory space, their +shadow and origin pages are similarly mapped into contiguous regions. + +References +========== + +E. Stepanov, K. Serebryany. `MemorySanitizer: fast detector of uninitialized +memory use in C++ +<https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43308.pdf>`_. +In Proceedings of CGO 2015. + +.. _MemorySanitizer tool: https://clang.llvm.org/docs/MemorySanitizer.html +.. _LLVM documentation: https://llvm.org/docs/GettingStarted.html +.. _LKML discussion: https://lore.kernel.org/all/20220614144853.3693273-1-glider@google.com/ diff --git a/Documentation/dev-tools/kunit/architecture.rst b/Documentation/dev-tools/kunit/architecture.rst index cf9e6e3eeae4..8efe792bdcb9 100644 --- a/Documentation/dev-tools/kunit/architecture.rst +++ b/Documentation/dev-tools/kunit/architecture.rst @@ -6,8 +6,8 @@ KUnit Architecture The KUnit architecture can be divided into two parts: -- Kernel testing library -- kunit_tool (Command line test harness) +- `In-Kernel Testing Framework`_ +- `kunit_tool (Command Line Test Harness)`_ In-Kernel Testing Framework =========================== diff --git a/Documentation/dev-tools/kunit/faq.rst b/Documentation/dev-tools/kunit/faq.rst index 172e239791a8..fae426f2634a 100644 --- a/Documentation/dev-tools/kunit/faq.rst +++ b/Documentation/dev-tools/kunit/faq.rst @@ -31,13 +31,16 @@ For the most part, the KUnit core framework (what we use to write the tests) can compile to any architecture. It compiles like just another part of the kernel and runs when the kernel boots, or when built as a module, when the module is loaded. However, there is infrastructure, like the KUnit Wrapper -(``tools/testing/kunit/kunit.py``) that does not support other architectures. +(``tools/testing/kunit/kunit.py``) that might not support some architectures +(see :ref:`kunit-on-qemu`). In short, yes, you can run KUnit on other architectures, but it might require more work than using KUnit on UML. For more information, see :ref:`kunit-on-non-uml`. +.. _kinds-of-tests: + What is the difference between a unit test and other kinds of tests? ==================================================================== Most existing tests for the Linux kernel would be categorized as an integration @@ -95,8 +98,7 @@ things to try. seeing. When tests are built-in, they will execute when the kernel boots, and modules will automatically execute associated tests when loaded. Test results can be collected from ``/sys/kernel/debug/kunit/<test suite>/results``, and - can be parsed with ``kunit.py parse``. For more details, see "KUnit on - non-UML architectures" in Documentation/dev-tools/kunit/usage.rst. + can be parsed with ``kunit.py parse``. For more details, see :ref:`kunit-on-qemu`. If none of the above tricks help, you are always welcome to email any issues to kunit-dev@googlegroups.com. diff --git a/Documentation/dev-tools/kunit/index.rst b/Documentation/dev-tools/kunit/index.rst index 595205348d2d..f5d13f1d37be 100644 --- a/Documentation/dev-tools/kunit/index.rst +++ b/Documentation/dev-tools/kunit/index.rst @@ -13,7 +13,6 @@ KUnit - Linux Kernel Unit Testing run_wrapper run_manual usage - kunit-tool api/index style faq @@ -29,10 +28,10 @@ KUnit (Kernel unit testing framework) provides a common framework for unit tests within the Linux kernel. Using KUnit, you can define groups of test cases called test suites. The tests either run on kernel boot if built-in, or load as a module. KUnit automatically flags and reports -failed test cases in the kernel log. The test results appear in `TAP -(Test Anything Protocol) format <https://testanything.org/>`_. It is inspired by -JUnit, Python’s unittest.mock, and GoogleTest/GoogleMock (C++ unit testing -framework). +failed test cases in the kernel log. The test results appear in +:doc:`KTAP (Kernel - Test Anything Protocol) format</dev-tools/ktap>`. +It is inspired by JUnit, Python’s unittest.mock, and GoogleTest/GoogleMock +(C++ unit testing framework). KUnit tests are part of the kernel, written in the C (programming) language, and test parts of the Kernel implementation (example: a C @@ -46,8 +45,9 @@ internal system functionality. KUnit runs in kernel space and is not restricted to things exposed to user-space. In addition, KUnit has kunit_tool, a script (``tools/testing/kunit/kunit.py``) -that configures the Linux kernel, runs KUnit tests under QEMU or UML (`User Mode -Linux <http://user-mode-linux.sourceforge.net/>`_), parses the test results and +that configures the Linux kernel, runs KUnit tests under QEMU or UML +(:doc:`User Mode Linux </virt/uml/user_mode_linux_howto_v2>`), +parses the test results and displays them in a user friendly manner. Features @@ -95,6 +95,8 @@ Unit Testing Advantages - Improves code quality. - Encourages writing testable code. +Read also :ref:`kinds-of-tests`. + How do I use it? ================ @@ -107,7 +109,5 @@ How do I use it? examples. * Documentation/dev-tools/kunit/api/index.rst - KUnit APIs used for testing. -* Documentation/dev-tools/kunit/kunit-tool.rst - kunit_tool helper - script. * Documentation/dev-tools/kunit/faq.rst - KUnit common questions and answers. diff --git a/Documentation/dev-tools/kunit/kunit-tool.rst b/Documentation/dev-tools/kunit/kunit-tool.rst deleted file mode 100644 index ae52e0f489f9..000000000000 --- a/Documentation/dev-tools/kunit/kunit-tool.rst +++ /dev/null @@ -1,232 +0,0 @@ -.. SPDX-License-Identifier: GPL-2.0 - -================= -kunit_tool How-To -================= - -What is kunit_tool? -=================== - -kunit_tool is a script (``tools/testing/kunit/kunit.py``) that aids in building -the Linux kernel as UML (`User Mode Linux -<http://user-mode-linux.sourceforge.net/>`_), running KUnit tests, parsing -the test results and displaying them in a user friendly manner. - -kunit_tool addresses the problem of being able to run tests without needing a -virtual machine or actual hardware with User Mode Linux. User Mode Linux is a -Linux architecture, like ARM or x86; however, unlike other architectures it -compiles the kernel as a standalone Linux executable that can be run like any -other program directly inside of a host operating system. To be clear, it does -not require any virtualization support: it is just a regular program. - -What is a .kunitconfig? -======================= - -It's just a defconfig that kunit_tool looks for in the build directory -(``.kunit`` by default). kunit_tool uses it to generate a .config as you might -expect. In addition, it verifies that the generated .config contains the CONFIG -options in the .kunitconfig; the reason it does this is so that it is easy to -be sure that a CONFIG that enables a test actually ends up in the .config. - -It's also possible to pass a separate .kunitconfig fragment to kunit_tool, -which is useful if you have several different groups of tests you wish -to run independently, or if you want to use pre-defined test configs for -certain subsystems. - -Getting Started with kunit_tool -=============================== - -If a kunitconfig is present at the root directory, all you have to do is: - -.. code-block:: bash - - ./tools/testing/kunit/kunit.py run - -However, you most likely want to use it with the following options: - -.. code-block:: bash - - ./tools/testing/kunit/kunit.py run --timeout=30 --jobs=`nproc --all` - -- ``--timeout`` sets a maximum amount of time to allow tests to run. -- ``--jobs`` sets the number of threads to use to build the kernel. - -.. note:: - This command will work even without a .kunitconfig file: if no - .kunitconfig is present, a default one will be used instead. - -If you wish to use a different .kunitconfig file (such as one provided for -testing a particular subsystem), you can pass it as an option. - -.. code-block:: bash - - ./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4/.kunitconfig - -For a list of all the flags supported by kunit_tool, you can run: - -.. code-block:: bash - - ./tools/testing/kunit/kunit.py run --help - -Configuring, Building, and Running Tests -======================================== - -It's also possible to run just parts of the KUnit build process independently, -which is useful if you want to make manual changes to part of the process. - -A .config can be generated from a .kunitconfig by using the ``config`` argument -when running kunit_tool: - -.. code-block:: bash - - ./tools/testing/kunit/kunit.py config - -Similarly, if you just want to build a KUnit kernel from the current .config, -you can use the ``build`` argument: - -.. code-block:: bash - - ./tools/testing/kunit/kunit.py build - -And, if you already have a built UML kernel with built-in KUnit tests, you can -run the kernel and display the test results with the ``exec`` argument: - -.. code-block:: bash - - ./tools/testing/kunit/kunit.py exec - -The ``run`` command which is discussed above is equivalent to running all three -of these in sequence. - -All of these commands accept a number of optional command-line arguments. The -``--help`` flag will give a complete list of these, or keep reading this page -for a guide to some of the more useful ones. - -Parsing Test Results -==================== - -KUnit tests output their results in TAP (Test Anything Protocol) format. -kunit_tool will, when running tests, parse this output and print a summary -which is much more pleasant to read. If you wish to look at the raw test -results in TAP format, you can pass the ``--raw_output`` argument. - -.. code-block:: bash - - ./tools/testing/kunit/kunit.py run --raw_output - -The raw output from test runs may contain other, non-KUnit kernel log -lines. You can see just KUnit output with ``--raw_output=kunit``: - -.. code-block:: bash - - ./tools/testing/kunit/kunit.py run --raw_output=kunit - -If you have KUnit results in their raw TAP format, you can parse them and print -the human-readable summary with the ``parse`` command for kunit_tool. This -accepts a filename for an argument, or will read from standard input. - -.. code-block:: bash - - # Reading from a file - ./tools/testing/kunit/kunit.py parse /var/log/dmesg - # Reading from stdin - dmesg | ./tools/testing/kunit/kunit.py parse - -This is very useful if you wish to run tests in a configuration not supported -by kunit_tool (such as on real hardware, or an unsupported architecture). - -Filtering Tests -=============== - -It's possible to run only a subset of the tests built into a kernel by passing -a filter to the ``exec`` or ``run`` commands. For example, if you only wanted -to run KUnit resource tests, you could use: - -.. code-block:: bash - - ./tools/testing/kunit/kunit.py run 'kunit-resource*' - -This uses the standard glob format for wildcards. - -Running Tests on QEMU -===================== - -kunit_tool supports running tests on QEMU as well as via UML (as mentioned -elsewhere). The default way of running tests on QEMU requires two flags: - -``--arch`` - Selects a collection of configs (Kconfig as well as QEMU configs - options, etc) that allow KUnit tests to be run on the specified - architecture in a minimal way; this is usually not much slower than - using UML. The architecture argument is the same as the name of the - option passed to the ``ARCH`` variable used by Kbuild. Not all - architectures are currently supported by this flag, but can be handled - by the ``--qemu_config`` discussed later. If ``um`` is passed (or this - this flag is ignored) the tests will run via UML. Non-UML architectures, - e.g. i386, x86_64, arm, um, etc. Non-UML run on QEMU. - -``--cross_compile`` - Specifies the use of a toolchain by Kbuild. The argument passed here is - the same passed to the ``CROSS_COMPILE`` variable used by Kbuild. As a - reminder this will be the prefix for the toolchain binaries such as gcc - for example ``sparc64-linux-gnu-`` if you have the sparc toolchain - installed on your system, or - ``$HOME/toolchains/microblaze/gcc-9.2.0-nolibc/microblaze-linux/bin/microblaze-linux-`` - if you have downloaded the microblaze toolchain from the 0-day website - to a directory in your home directory called ``toolchains``. - -In many cases it is likely that you may want to run an architecture which is -not supported by the ``--arch`` flag, or you may want to just run KUnit tests -on QEMU using a non-default configuration. For this use case, you can write -your own QemuConfig. These QemuConfigs are written in Python. They must have an -import line ``from ..qemu_config import QemuArchParams`` at the top of the file -and the file must contain a variable called ``QEMU_ARCH`` that has an instance -of ``QemuArchParams`` assigned to it. An example can be seen in -``tools/testing/kunit/qemu_configs/x86_64.py``. - -Once you have a QemuConfig you can pass it into kunit_tool using the -``--qemu_config`` flag; when used this flag replaces the ``--arch`` flag. If we -were to do this with the ``x86_64.py`` example from above, the invocation would -look something like this: - -.. code-block:: bash - - ./tools/testing/kunit/kunit.py run \ - --timeout=60 \ - --jobs=12 \ - --qemu_config=./tools/testing/kunit/qemu_configs/x86_64.py - -Other Useful Options -==================== - -kunit_tool has a number of other command-line arguments which can be useful -when adapting it to fit your environment or needs. - -Some of the more useful ones are: - -``--help`` - Lists all of the available options. Note that different commands - (``config``, ``build``, ``run``, etc) will have different supported - options. Place ``--help`` before the command to list common options, - and after the command for options specific to that command. - -``--build_dir`` - Specifies the build directory that kunit_tool will use. This is where - the .kunitconfig file is located, as well as where the .config and - compiled kernel will be placed. Defaults to ``.kunit``. - -``--make_options`` - Specifies additional options to pass to ``make`` when compiling a - kernel (with the ``build`` or ``run`` commands). For example, to enable - compiler warnings, you can pass ``--make_options W=1``. - -``--alltests`` - Builds a UML kernel with all config options enabled using ``make - allyesconfig``. This allows you to run as many tests as is possible, - but is very slow and prone to breakage as new options are added or - modified. In most cases, enabling all tests which have satisfied - dependencies by adding ``CONFIG_KUNIT_ALL_TESTS=1`` to your - .kunitconfig is preferable. - -There are several other options (and new ones are often added), so do check -``--help`` if you're looking for something not mentioned here. diff --git a/Documentation/dev-tools/kunit/run_wrapper.rst b/Documentation/dev-tools/kunit/run_wrapper.rst index cce203138fb7..6b33caf6c8ab 100644 --- a/Documentation/dev-tools/kunit/run_wrapper.rst +++ b/Documentation/dev-tools/kunit/run_wrapper.rst @@ -1,8 +1,8 @@ .. SPDX-License-Identifier: GPL-2.0 -========================= -Run Tests with kunit_tool -========================= +============================= +Running tests with kunit_tool +============================= We can either run KUnit tests using kunit_tool or can run tests manually, and then use kunit_tool to parse the results. To run tests @@ -22,7 +22,7 @@ We should see the following: .. code-block:: - Generating .config... + Configuring KUnit Kernel ... Building KUnit kernel... Starting KUnit kernel... @@ -30,7 +30,7 @@ We may want to use the following options: .. code-block:: - ./tools/testing/kunit/kunit.py run --timeout=30 --jobs=`nproc --all + ./tools/testing/kunit/kunit.py run --timeout=30 --jobs=`nproc --all` - ``--timeout`` sets a maximum amount of time for tests to run. - ``--jobs`` sets the number of threads to build the kernel. @@ -58,8 +58,8 @@ To view kunit_tool flags (optional command-line arguments), run: ./tools/testing/kunit/kunit.py run --help -Create a ``.kunitconfig`` File -=============================== +Creating a ``.kunitconfig`` file +================================ If we want to run a specific set of tests (rather than those listed in the KUnit ``defconfig``), we can provide Kconfig options in the @@ -98,8 +98,8 @@ have not included the options dependencies. The build dir needs to be set for ``make menuconfig`` to work, therefore by default use ``make O=.kunit menuconfig``. -Configure, Build, and Run Tests -=============================== +Configuring, building, and running tests +======================================== If we want to make manual changes to the KUnit build process, we can run part of the KUnit build process independently. @@ -125,11 +125,11 @@ argument: ./tools/testing/kunit/kunit.py exec -The ``run`` command discussed in section: **Run Tests with kunit_tool**, +The ``run`` command discussed in section: **Running tests with kunit_tool**, is equivalent to running the above three commands in sequence. -Parse Test Results -================== +Parsing test results +==================== KUnit tests output displays results in TAP (Test Anything Protocol) format. When running tests, kunit_tool parses this output and prints @@ -152,8 +152,8 @@ standard input. # Reading from stdin dmesg | ./tools/testing/kunit/kunit.py parse -Run Selected Test Suites -======================== +Filtering tests +=============== By passing a bash style glob filter to the ``exec`` or ``run`` commands, we can run a subset of the tests built into a kernel . For @@ -165,8 +165,10 @@ example: if we only want to run KUnit resource tests, use: This uses the standard glob format with wildcard characters. -Run Tests on qemu -================= +.. _kunit-on-qemu: + +Running tests on QEMU +===================== kunit_tool supports running tests on qemu as well as via UML. To run tests on qemu, by default it requires two flags: @@ -229,8 +231,8 @@ as --jobs=12 \ --qemu_config=./tools/testing/kunit/qemu_configs/x86_64.py -Command-Line Arguments -====================== +Running command-line arguments +============================== kunit_tool has a number of other command-line arguments which can be useful for our test environment. Below are the most commonly used diff --git a/Documentation/dev-tools/kunit/start.rst b/Documentation/dev-tools/kunit/start.rst index 867a4bba6bf6..f4f504f1fb15 100644 --- a/Documentation/dev-tools/kunit/start.rst +++ b/Documentation/dev-tools/kunit/start.rst @@ -4,6 +4,10 @@ Getting Started =============== +This page contains an overview of the kunit_tool and KUnit framework, +teaching how to run existing tests and then how to write a simple test case, +and covers common problems users face when using KUnit for the first time. + Installing Dependencies ======================= KUnit has the same dependencies as the Linux kernel. As long as you can @@ -19,30 +23,53 @@ can run kunit_tool: ./tools/testing/kunit/kunit.py run -For more information on this wrapper, see: -Documentation/dev-tools/kunit/run_wrapper.rst. +.. note :: + You may see the following error: + "The source tree is not clean, please run 'make ARCH=um mrproper'" -Creating a ``.kunitconfig`` ---------------------------- + This happens because internally kunit.py specifies ``.kunit`` + (default option) as the build directory in the command ``make O=output/dir`` + through the argument ``--build_dir``. Hence, before starting an + out-of-tree build, the source tree must be clean. -By default, kunit_tool runs a selection of tests. However, you can specify which -unit tests to run by creating a ``.kunitconfig`` file with kernel config options -that enable only a specific set of tests and their dependencies. -The ``.kunitconfig`` file contains a list of kconfig options which are required -to run the desired targets. The ``.kunitconfig`` also contains any other test -specific config options, such as test dependencies. For example: the -``FAT_FS`` tests - ``FAT_KUNIT_TEST``, depends on -``FAT_FS``. ``FAT_FS`` can be enabled by selecting either ``MSDOS_FS`` -or ``VFAT_FS``. To run ``FAT_KUNIT_TEST``, the ``.kunitconfig`` has: + There is also the same caveat mentioned in the "Build directory for + the kernel" section of the :doc:`admin-guide </admin-guide/README>`, + that is, its use, it must be used for all invocations of ``make``. + The good news is that it can indeed be solved by running + ``make ARCH=um mrproper``, just be aware that this will delete the + current configuration and all generated files. -.. code-block:: none +If everything worked correctly, you should see the following: - CONFIG_KUNIT=y - CONFIG_MSDOS_FS=y - CONFIG_FAT_KUNIT_TEST=y +.. code-block:: -1. A good starting point for the ``.kunitconfig`` is the KUnit default config. - You can generate it by running: + Configuring KUnit Kernel ... + Building KUnit Kernel ... + Starting KUnit Kernel ... + +The tests will pass or fail. + +.. note :: + Because it is building a lot of sources for the first time, + the ``Building KUnit Kernel`` step may take a while. + +For detailed information on this wrapper, see: +Documentation/dev-tools/kunit/run_wrapper.rst. + +Selecting which tests to run +---------------------------- + +By default, kunit_tool runs all tests reachable with minimal configuration, +that is, using default values for most of the kconfig options. However, +you can select which tests to run by: + +- `Customizing Kconfig`_ used to compile the kernel, or +- `Filtering tests by name`_ to select specifically which compiled tests to run. + +Customizing Kconfig +~~~~~~~~~~~~~~~~~~~ +A good starting point for the ``.kunitconfig`` is the KUnit default config. +If you didn't run ``kunit.py run`` yet, you can generate it by running: .. code-block:: bash @@ -54,48 +81,69 @@ or ``VFAT_FS``. To run ``FAT_KUNIT_TEST``, the ``.kunitconfig`` has: ``.kunitconfig`` lives in the ``--build_dir`` used by kunit.py, which is ``.kunit`` by default. -.. note :: +Before running the tests, kunit_tool ensures that all config options +set in ``.kunitconfig`` are set in the kernel ``.config``. It will warn +you if you have not included dependencies for the options used. + +There are many ways to customize the configurations: + +a. Edit ``.kunit/.kunitconfig``. The file should contain the list of kconfig + options required to run the desired tests, including their dependencies. You may want to remove CONFIG_KUNIT_ALL_TESTS from the ``.kunitconfig`` as it will enable a number of additional tests that you may not want. + If you need to run on an architecture other than UML see :ref:`kunit-on-qemu`. -2. You can then add any other Kconfig options, for example: +b. Enable additional kconfig options on top of ``.kunit/.kunitconfig``. + For example, to include the kernel's linked-list test you can run:: -.. code-block:: none + ./tools/testing/kunit/kunit.py run \ + --kconfig_add CONFIG_LIST_KUNIT_TEST=y - CONFIG_LIST_KUNIT_TEST=y +c. Provide the path of one or more .kunitconfig files from the tree. + For example, to run only ``FAT_FS`` and ``EXT4`` tests you can run:: -Before running the tests, kunit_tool ensures that all config options -set in ``.kunitconfig`` are set in the kernel ``.config``. It will warn -you if you have not included dependencies for the options used. + ./tools/testing/kunit/kunit.py run \ + --kunitconfig ./fs/fat/.kunitconfig \ + --kunitconfig ./fs/ext4/.kunitconfig -.. note :: - If you change the ``.kunitconfig``, kunit.py will trigger a rebuild of the +d. If you change the ``.kunitconfig``, kunit.py will trigger a rebuild of the ``.config`` file. But you can edit the ``.config`` file directly or with tools like ``make menuconfig O=.kunit``. As long as its a superset of ``.kunitconfig``, kunit.py won't overwrite your changes. -Running Tests (KUnit Wrapper) ------------------------------ -1. To make sure that everything is set up correctly, invoke the Python - wrapper from your kernel repository: -.. code-block:: bash +.. note :: - ./tools/testing/kunit/kunit.py run + To save a .kunitconfig after finding a satisfactory configuration:: -If everything worked correctly, you should see the following: + make savedefconfig O=.kunit + cp .kunit/defconfig .kunit/.kunitconfig -.. code-block:: +Filtering tests by name +~~~~~~~~~~~~~~~~~~~~~~~ +If you want to be more specific than Kconfig can provide, it is also possible +to select which tests to execute at boot-time by passing a glob filter +(read instructions regarding the pattern in the manpage :manpage:`glob(7)`). +If there is a ``"."`` (period) in the filter, it will be interpreted as a +separator between the name of the test suite and the test case, +otherwise, it will be interpreted as the name of the test suite. +For example, let's assume we are using the default config: - Generating .config ... - Building KUnit Kernel ... - Starting KUnit Kernel ... +a. inform the name of a test suite, like ``"kunit_executor_test"``, + to run every test case it contains:: -The tests will pass or fail. + ./tools/testing/kunit/kunit.py run "kunit_executor_test" -.. note :: - Because it is building a lot of sources for the first time, the - ``Building KUnit kernel`` may take a while. +b. inform the name of a test case prefixed by its test suite, + like ``"example.example_simple_test"``, to run specifically that test case:: + + ./tools/testing/kunit/kunit.py run "example.example_simple_test" + +c. use wildcard characters (``*?[``) to run any test case that matches the pattern, + like ``"*.*64*"`` to run test cases containing ``"64"`` in the name inside + any test suite:: + + ./tools/testing/kunit/kunit.py run "*.*64*" Running Tests without the KUnit Wrapper ======================================= @@ -217,7 +265,7 @@ Now we are ready to write the test cases. obj-$(CONFIG_MISC_EXAMPLE_TEST) += example_test.o -4. Add the following lines to ``.kunitconfig``: +4. Add the following lines to ``.kunit/.kunitconfig``: .. code-block:: none @@ -254,7 +302,5 @@ Next Steps examples. * Documentation/dev-tools/kunit/api/index.rst - KUnit APIs used for testing. -* Documentation/dev-tools/kunit/kunit-tool.rst - kunit_tool helper - script. * Documentation/dev-tools/kunit/faq.rst - KUnit common questions and answers. diff --git a/Documentation/dev-tools/kunit/usage.rst b/Documentation/dev-tools/kunit/usage.rst index 44158eecb51e..2737863ef365 100644 --- a/Documentation/dev-tools/kunit/usage.rst +++ b/Documentation/dev-tools/kunit/usage.rst @@ -165,6 +165,8 @@ built as a module). For more information, see Documentation/dev-tools/kunit/api/test.rst. +.. _kunit-on-non-uml: + Writing Tests For Other Architectures ------------------------------------- @@ -544,8 +546,6 @@ By reusing the same ``cases`` array from above, we can write the test as a {} }; -.. _kunit-on-non-uml: - Exiting Early on Failed Expectations ------------------------------------ |