summaryrefslogtreecommitdiff
path: root/Documentation/dev-tools/kunit/run_wrapper.rst
blob: db1e867820e7444b23b7de0125b7a5376d308260 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
.. SPDX-License-Identifier: GPL-2.0

=========================
Run Tests with kunit_tool
=========================

We can either run KUnit tests using kunit_tool or can run tests
manually, and then use kunit_tool to parse the results. To run tests
manually, see: Documentation/dev-tools/kunit/run_manual.rst.
As long as we can build the kernel, we can run KUnit.

kunit_tool is a Python script which configures and builds a kernel, runs
tests, and formats the test results.

Run command:

.. code-block::

	./tools/testing/kunit/kunit.py run

We should see the following:

.. code-block::

	Generating .config...
	Building KUnit kernel...
	Starting KUnit kernel...

We may want to use the following options:

.. code-block::

	./tools/testing/kunit/kunit.py run --timeout=30 --jobs=`nproc --all`

- ``--timeout`` sets a maximum amount of time for tests to run.
- ``--jobs`` sets the number of threads to build the kernel.

kunit_tool will generate a ``.kunitconfig`` with a default
configuration, if no other ``.kunitconfig`` file exists
(in the build directory). In addition, it verifies that the
generated ``.config`` file contains the ``CONFIG`` options in the
``.kunitconfig``.
It is also possible to pass a separate ``.kunitconfig`` fragment to
kunit_tool. This is useful if we have several different groups of
tests we want to run independently, or if we want to use pre-defined
test configs for certain subsystems.

To use a different ``.kunitconfig`` file (such as one
provided to test a particular subsystem), pass it as an option:

.. code-block::

	./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4/.kunitconfig

To view kunit_tool flags (optional command-line arguments), run:

.. code-block::

	./tools/testing/kunit/kunit.py run --help

Create a  ``.kunitconfig`` File
===============================

If we want to run a specific set of tests (rather than those listed
in the KUnit ``defconfig``), we can provide Kconfig options in the
``.kunitconfig`` file. For default .kunitconfig, see:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/kunit/configs/default.config.
A ``.kunitconfig`` is a ``minconfig`` (a .config
generated by running ``make savedefconfig``), used for running a
specific set of tests. This file contains the regular Kernel configs
with specific test targets. The ``.kunitconfig`` also
contains any other config options required by the tests (For example:
dependencies for features under tests, configs that enable/disable
certain code blocks, arch configs and so on).

To create a ``.kunitconfig``, using the KUnit ``defconfig``:

.. code-block::

	cd $PATH_TO_LINUX_REPO
	cp tools/testing/kunit/configs/default.config .kunit/.kunitconfig

We can then add any other Kconfig options. For example:

.. code-block::

	CONFIG_LIST_KUNIT_TEST=y

kunit_tool ensures that all config options in ``.kunitconfig`` are
set in the kernel ``.config`` before running the tests. It warns if we
have not included the options dependencies.

.. note:: Removing something from the ``.kunitconfig`` will
   not rebuild the ``.config file``. The configuration is only
   updated if the ``.kunitconfig`` is not a subset of ``.config``.
   This means that we can use other tools
   (For example: ``make menuconfig``) to adjust other config options.
   The build dir needs to be set for ``make menuconfig`` to
   work, therefore  by default use ``make O=.kunit menuconfig``.

Configure, Build, and Run Tests
===============================

If we want to make manual changes to the KUnit build process, we
can run part of the KUnit build process independently.
When running kunit_tool, from a ``.kunitconfig``, we can generate a
``.config`` by using the ``config`` argument:

.. code-block::

	./tools/testing/kunit/kunit.py config

To build a KUnit kernel from the current ``.config``, we can use the
``build`` argument:

.. code-block::

	./tools/testing/kunit/kunit.py build

If we already have built UML kernel with built-in KUnit tests, we
can run the kernel, and display the test results with the ``exec``
argument:

.. code-block::

	./tools/testing/kunit/kunit.py exec

The ``run`` command discussed in section: **Run Tests with kunit_tool**,
is equivalent to running the above three commands in sequence.

Parse Test Results
==================

KUnit tests output displays results in TAP (Test Anything Protocol)
format. When running tests, kunit_tool parses this output and prints
a summary. To see the raw test results in TAP format, we can pass the
``--raw_output`` argument:

.. code-block::

	./tools/testing/kunit/kunit.py run --raw_output

If we have KUnit results in the raw TAP format, we can parse them and
print the human-readable summary with the ``parse`` command for
kunit_tool. This accepts a filename for an argument, or will read from
standard input.

.. code-block:: bash

	# Reading from a file
	./tools/testing/kunit/kunit.py parse /var/log/dmesg
	# Reading from stdin
	dmesg | ./tools/testing/kunit/kunit.py parse

Run Selected Test Suites
========================

By passing a bash style glob filter to the ``exec`` or ``run``
commands, we can run a subset of the tests built into a kernel . For
example: if we only want to run KUnit resource tests, use:

.. code-block::

	./tools/testing/kunit/kunit.py run 'kunit-resource*'

This uses the standard glob format with wildcard characters.

Run Tests on qemu
=================

kunit_tool supports running tests on  qemu as well as
via UML. To run tests on qemu, by default it requires two flags:

- ``--arch``: Selects a configs collection (Kconfig, qemu config options
  and so on), that allow KUnit tests to be run on the specified
  architecture in a minimal way. The architecture argument is same as
  the option name passed to the ``ARCH`` variable used by Kbuild.
  Not all architectures currently support this flag, but we can use
  ``--qemu_config`` to handle it. If ``um`` is passed (or this flag
  is ignored), the tests will run via UML. Non-UML architectures,
  for example: i386, x86_64, arm and so on; run on qemu.

- ``--cross_compile``: Specifies the Kbuild toolchain. It passes the
  same argument as passed to the ``CROSS_COMPILE`` variable used by
  Kbuild. As a reminder, this will be the prefix for the toolchain
  binaries such as GCC. For example:

  - ``sparc64-linux-gnu`` if we have the sparc toolchain installed on
    our system.

  - ``$HOME/toolchains/microblaze/gcc-9.2.0-nolibc/microblaze-linux/bin/microblaze-linux``
    if we have downloaded the microblaze toolchain from the 0-day
    website to a directory in our home directory called toolchains.

This means that for most architectures, running under qemu is as simple as:

.. code-block:: bash

	./tools/testing/kunit/kunit.py run --arch=x86_64

When cross-compiling, we'll likely need to specify a different toolchain, for
example:

.. code-block:: bash

	./tools/testing/kunit/kunit.py run \
		--arch=s390 \
		--cross_compile=s390x-linux-gnu-

If we want to run KUnit tests on an architecture not supported by
the ``--arch`` flag, or want to run KUnit tests on qemu using a
non-default configuration; then we can write our own``QemuConfig``.
These ``QemuConfigs`` are written in Python. They have an import line
``from..qemu_config import QemuArchParams`` at the top of the file.
The file must contain a variable called ``QEMU_ARCH`` that has an
instance of ``QemuArchParams`` assigned to it. See example in:
``tools/testing/kunit/qemu_configs/x86_64.py``.

Once we have a ``QemuConfig``, we can pass it into kunit_tool,
using the ``--qemu_config`` flag. When used, this flag replaces the
``--arch`` flag. For example: using
``tools/testing/kunit/qemu_configs/x86_64.py``, the invocation appear
as

.. code-block:: bash

	./tools/testing/kunit/kunit.py run \
		--timeout=60 \
		--jobs=12 \
		--qemu_config=./tools/testing/kunit/qemu_configs/x86_64.py

Command-Line Arguments
======================

kunit_tool has a number of other command-line arguments which can
be useful for our test environment. Below are the most commonly used
command line arguments:

- ``--help``: Lists all available options. To list common options,
  place ``--help`` before the command. To list options specific to that
  command, place ``--help`` after the command.

  .. note:: Different commands (``config``, ``build``, ``run``, etc)
            have different supported options.
- ``--build_dir``: Specifies kunit_tool build directory. It includes
  the ``.kunitconfig``, ``.config`` files and compiled kernel.

- ``--make_options``: Specifies additional options to pass to make, when
  compiling a kernel (using ``build`` or ``run`` commands). For example:
  to enable compiler warnings, we can pass ``--make_options W=1``.

- ``--alltests``: Builds a UML kernel with all config options enabled
  using ``make allyesconfig``. This allows us to run as many tests as
  possible.

  .. note:: It is slow and prone to breakage as new options are
            added or modified. Instead, enable all tests
            which have satisfied dependencies by adding
            ``CONFIG_KUNIT_ALL_TESTS=y`` to your ``.kunitconfig``.

- ``--kunitconfig``: Specifies the path or the directory of the ``.kunitconfig``
  file. For example:

  - ``lib/kunit/.kunitconfig`` can be the path of the file.

  - ``lib/kunit`` can be the directory in which the file is located.

  This file is used to build and run with a predefined set of tests
  and their dependencies. For example, to run tests for a given subsystem.

- ``--kconfig_add``: Specifies additional configuration options to be
  appended to the ``.kunitconfig`` file. For example:

  .. code-block::

	./tools/testing/kunit/kunit.py run --kconfig_add CONFIG_KASAN=y

- ``--arch``: Runs tests on the specified architecture. The architecture
  argument is same as the Kbuild ARCH environment variable.
  For example, i386, x86_64, arm, um, etc. Non-UML architectures run on qemu.
  Default is `um`.

- ``--cross_compile``: Specifies the Kbuild toolchain. It passes the
  same argument as passed to the ``CROSS_COMPILE`` variable used by
  Kbuild. This will be the prefix for the toolchain
  binaries such as GCC. For example:

  - ``sparc64-linux-gnu-`` if we have the sparc toolchain installed on
    our system.

  - ``$HOME/toolchains/microblaze/gcc-9.2.0-nolibc/microblaze-linux/bin/microblaze-linux``
    if we have downloaded the microblaze toolchain from the 0-day
    website to a specified path in our home directory called toolchains.

- ``--qemu_config``: Specifies the path to a file containing a
  custom qemu architecture definition. This should be a python file
  containing a `QemuArchParams` object.

- ``--qemu_args``: Specifies additional qemu arguments, for example, ``-smp 8``.

- ``--jobs``: Specifies the number of jobs (commands) to run simultaneously.
  By default, this is set to the number of cores on your system.

- ``--timeout``: Specifies the maximum number of seconds allowed for all tests to run.
  This does not include the time taken to build the tests.

- ``--kernel_args``: Specifies additional kernel command-line arguments. May be repeated.

- ``--run_isolated``: If set, boots the kernel for each individual suite/test.
  This is useful for debugging a non-hermetic test, one that
  might pass/fail based on what ran before it.

- ``--raw_output``: If set, generates unformatted output from kernel. Possible options are:

   - ``all``: To view the full kernel output, use ``--raw_output=all``.

   - ``kunit``: This is the default option and filters to KUnit output. Use ``--raw_output`` or ``--raw_output=kunit``.

- ``--json``: If set, stores the test results in a JSON format and prints to `stdout` or
  saves to a file if a filename is specified.