diff options
Diffstat (limited to 'Documentation/admin-guide')
45 files changed, 2499 insertions, 832 deletions
diff --git a/Documentation/admin-guide/acpi/ssdt-overlays.rst b/Documentation/admin-guide/acpi/ssdt-overlays.rst index 5d7e25988085..b5fbf54dca19 100644 --- a/Documentation/admin-guide/acpi/ssdt-overlays.rst +++ b/Documentation/admin-guide/acpi/ssdt-overlays.rst @@ -30,22 +30,21 @@ following ASL code can be used:: { Device (STAC) { - Name (_ADR, Zero) Name (_HID, "BMA222E") + Name (RBUF, ResourceTemplate () + { + I2cSerialBus (0x0018, ControllerInitiated, 0x00061A80, + AddressingMode7Bit, "\\_SB.I2C6", 0x00, + ResourceConsumer, ,) + GpioInt (Edge, ActiveHigh, Exclusive, PullDown, 0x0000, + "\\_SB.GPO2", 0x00, ResourceConsumer, , ) + { // Pin list + 0 + } + }) Method (_CRS, 0, Serialized) { - Name (RBUF, ResourceTemplate () - { - I2cSerialBus (0x0018, ControllerInitiated, 0x00061A80, - AddressingMode7Bit, "\\_SB.I2C6", 0x00, - ResourceConsumer, ,) - GpioInt (Edge, ActiveHigh, Exclusive, PullDown, 0x0000, - "\\_SB.GPO2", 0x00, ResourceConsumer, , ) - { // Pin list - 0 - } - }) Return (RBUF) } } @@ -75,7 +74,7 @@ This option allows loading of user defined SSDTs from initrd and it is useful when the system does not support EFI or when there is not enough EFI storage. It works in a similar way with initrd based ACPI tables override/upgrade: SSDT -aml code must be placed in the first, uncompressed, initrd under the +AML code must be placed in the first, uncompressed, initrd under the "kernel/firmware/acpi" path. Multiple files can be used and this will translate in loading multiple tables. Only SSDT and OEM tables are allowed. See initrd_table_override.txt for more details. @@ -103,12 +102,14 @@ This is the preferred method, when EFI is supported on the platform, because it allows a persistent, OS independent way of storing the user defined SSDTs. There is also work underway to implement EFI support for loading user defined SSDTs and using this method will make it easier to convert to the EFI loading -mechanism when that will arrive. +mechanism when that will arrive. To enable it, the +CONFIG_EFI_CUSTOM_SSDT_OVERLAYS shoyld be chosen to y. -In order to load SSDTs from an EFI variable the efivar_ssdt kernel command line -parameter can be used. The argument for the option is the variable name to -use. If there are multiple variables with the same name but with different -vendor GUIDs, all of them will be loaded. +In order to load SSDTs from an EFI variable the ``"efivar_ssdt=..."`` kernel +command line parameter can be used (the name has a limitation of 16 characters). +The argument for the option is the variable name to use. If there are multiple +variables with the same name but with different vendor GUIDs, all of them will +be loaded. In order to store the AML code in an EFI variable the efivarfs filesystem can be used. It is enabled and mounted by default in /sys/firmware/efi/efivars in all @@ -127,7 +128,7 @@ variable with the content from a given file:: #!/bin/sh -e - while ! [ -z "$1" ]; do + while [ -n "$1" ]; do case "$1" in "-f") filename="$2"; shift;; "-g") guid="$2"; shift;; @@ -167,14 +168,14 @@ variable with the content from a given file:: Loading ACPI SSDTs from configfs ================================ -This option allows loading of user defined SSDTs from userspace via the configfs +This option allows loading of user defined SSDTs from user space via the configfs interface. The CONFIG_ACPI_CONFIGFS option must be select and configfs must be mounted. In the following examples, we assume that configfs has been mounted in -/config. +/sys/kernel/config. -New tables can be loading by creating new directories in /config/acpi/table/ and -writing the SSDT aml code in the aml attribute:: +New tables can be loading by creating new directories in /sys/kernel/config/acpi/table +and writing the SSDT AML code in the aml attribute:: - cd /config/acpi/table + cd /sys/kernel/config/acpi/table mkdir my_ssdt cat ~/ssdt.aml > my_ssdt/aml diff --git a/Documentation/admin-guide/binderfs.rst b/Documentation/admin-guide/binderfs.rst index 199d84314a14..41a4db00df8d 100644 --- a/Documentation/admin-guide/binderfs.rst +++ b/Documentation/admin-guide/binderfs.rst @@ -72,3 +72,16 @@ that the `rm() <rm_>`_ tool can be used to delete them. Note that the ``binder-control`` device cannot be deleted since this would make the binderfs instance unusable. The ``binder-control`` device will be deleted when the binderfs instance is unmounted and all references to it have been dropped. + +Binder features +--------------- + +Assuming an instance of binderfs has been mounted at ``/dev/binderfs``, the +features supported by the binder driver can be located under +``/dev/binderfs/features/``. The presence of individual files can be tested +to determine whether a particular feature is supported by the driver. + +Example:: + + cat /dev/binderfs/features/oneway_spam_detection + 1 diff --git a/Documentation/admin-guide/bootconfig.rst b/Documentation/admin-guide/bootconfig.rst index 452b7dcd7f6b..a1860fc0ca88 100644 --- a/Documentation/admin-guide/bootconfig.rst +++ b/Documentation/admin-guide/bootconfig.rst @@ -89,13 +89,35 @@ you can use ``+=`` operator. For example:: In this case, the key ``foo`` has ``bar``, ``baz`` and ``qux``. -However, a sub-key and a value can not co-exist under a parent key. -For example, following config is NOT allowed.:: +Moreover, sub-keys and a value can coexist under a parent key. +For example, following config is allowed.:: foo = value1 - foo.bar = value2 # !ERROR! subkey "bar" and value "value1" can NOT co-exist - foo.bar := value2 # !ERROR! even with the override operator, this is NOT allowed. + foo.bar = value2 + foo := value3 # This will update foo's value. + +Note, since there is no syntax to put a raw value directly under a +structured key, you have to define it outside of the brace. For example:: + + foo { + bar = value1 + bar { + baz = value2 + qux = value3 + } + } + +Also, the order of the value node under a key is fixed. If there +are a value and subkeys, the value is always the first child node +of the key. Thus if user specifies subkeys first, e.g.:: + + foo.bar = value1 + foo = value2 +In the program (and /proc/bootconfig), it will be shown as below:: + + foo = value2 + foo.bar = value1 Comments -------- @@ -156,7 +178,7 @@ update the boot loader and the kernel image itself as long as the boot loader passes the correct initrd file size. If by any chance, the boot loader passes a longer size, the kernel fails to find the bootconfig data. -To do this operation, Linux kernel provides "bootconfig" command under +To do this operation, Linux kernel provides ``bootconfig`` command under tools/bootconfig, which allows admin to apply or delete the config file to/from initrd image. You can build it by the following command:: @@ -174,6 +196,43 @@ To remove the config from the image, you can use -d option as below:: Then add "bootconfig" on the normal kernel command line to tell the kernel to look for the bootconfig at the end of the initrd file. + +Kernel parameters via Boot Config +================================= + +In addition to the kernel command line, the boot config can be used for +passing the kernel parameters. All the key-value pairs under ``kernel`` +key will be passed to kernel cmdline directly. Moreover, the key-value +pairs under ``init`` will be passed to init process via the cmdline. +The parameters are concatinated with user-given kernel cmdline string +as the following order, so that the command line parameter can override +bootconfig parameters (this depends on how the subsystem handles parameters +but in general, earlier parameter will be overwritten by later one.):: + + [bootconfig params][cmdline params] -- [bootconfig init params][cmdline init params] + +Here is an example of the bootconfig file for kernel/init parameters.:: + + kernel { + root = 01234567-89ab-cdef-0123-456789abcd + } + init { + splash + } + +This will be copied into the kernel cmdline string as the following:: + + root="01234567-89ab-cdef-0123-456789abcd" -- splash + +If user gives some other command line like,:: + + ro bootconfig -- quiet + +The final kernel cmdline will be the following:: + + root="01234567-89ab-cdef-0123-456789abcd" ro bootconfig -- splash quiet + + Config File Limitation ====================== diff --git a/Documentation/admin-guide/cgroup-v1/blkio-controller.rst b/Documentation/admin-guide/cgroup-v1/blkio-controller.rst index 36d43ae7dc13..16253eda192e 100644 --- a/Documentation/admin-guide/cgroup-v1/blkio-controller.rst +++ b/Documentation/admin-guide/cgroup-v1/blkio-controller.rst @@ -17,36 +17,37 @@ level logical devices like device mapper. HOWTO ===== + Throttling/Upper Limit policy ----------------------------- -- Enable Block IO controller:: +Enable Block IO controller:: CONFIG_BLK_CGROUP=y -- Enable throttling in block layer:: +Enable throttling in block layer:: CONFIG_BLK_DEV_THROTTLING=y -- Mount blkio controller (see cgroups.txt, Why are cgroups needed?):: +Mount blkio controller (see cgroups.txt, Why are cgroups needed?):: mount -t cgroup -o blkio none /sys/fs/cgroup/blkio -- Specify a bandwidth rate on particular device for root group. The format - for policy is "<major>:<minor> <bytes_per_second>":: +Specify a bandwidth rate on particular device for root group. The format +for policy is "<major>:<minor> <bytes_per_second>":: echo "8:16 1048576" > /sys/fs/cgroup/blkio/blkio.throttle.read_bps_device - Above will put a limit of 1MB/second on reads happening for root group - on device having major/minor number 8:16. +This will put a limit of 1MB/second on reads happening for root group +on device having major/minor number 8:16. -- Run dd to read a file and see if rate is throttled to 1MB/s or not:: +Run dd to read a file and see if rate is throttled to 1MB/s or not:: # dd iflag=direct if=/mnt/common/zerofile of=/dev/null bs=4K count=1024 1024+0 records in 1024+0 records out 4194304 bytes (4.2 MB) copied, 4.0001 s, 1.0 MB/s - Limits for writes can be put using blkio.throttle.write_bps_device file. +Limits for writes can be put using blkio.throttle.write_bps_device file. Hierarchical Cgroups ==================== @@ -79,85 +80,89 @@ following:: Various user visible config options =================================== -CONFIG_BLK_CGROUP - - Block IO controller. -CONFIG_BFQ_CGROUP_DEBUG - - Debug help. Right now some additional stats file show up in cgroup + CONFIG_BLK_CGROUP + Block IO controller. + + CONFIG_BFQ_CGROUP_DEBUG + Debug help. Right now some additional stats file show up in cgroup if this option is enabled. -CONFIG_BLK_DEV_THROTTLING - - Enable block device throttling support in block layer. + CONFIG_BLK_DEV_THROTTLING + Enable block device throttling support in block layer. Details of cgroup files ======================= + Proportional weight policy files -------------------------------- -- blkio.weight - - Specifies per cgroup weight. This is default weight of the group - on all the devices until and unless overridden by per device rule. - (See blkio.weight_device). - Currently allowed range of weights is from 10 to 1000. -- blkio.weight_device - - One can specify per cgroup per device rules using this interface. - These rules override the default value of group weight as specified - by blkio.weight. + blkio.bfq.weight + Specifies per cgroup weight. This is default weight of the group + on all the devices until and unless overridden by per device rule + (see `blkio.bfq.weight_device` below). + + Currently allowed range of weights is from 1 to 1000. For more details, + see Documentation/block/bfq-iosched.rst. + + blkio.bfq.weight_device + Specifes per cgroup per device weights, overriding the default group + weight. For more details, see Documentation/block/bfq-iosched.rst. Following is the format:: - # echo dev_maj:dev_minor weight > blkio.weight_device + # echo dev_maj:dev_minor weight > blkio.bfq.weight_device Configure weight=300 on /dev/sdb (8:16) in this cgroup:: - # echo 8:16 300 > blkio.weight_device - # cat blkio.weight_device + # echo 8:16 300 > blkio.bfq.weight_device + # cat blkio.bfq.weight_device dev weight 8:16 300 Configure weight=500 on /dev/sda (8:0) in this cgroup:: - # echo 8:0 500 > blkio.weight_device - # cat blkio.weight_device + # echo 8:0 500 > blkio.bfq.weight_device + # cat blkio.bfq.weight_device dev weight 8:0 500 8:16 300 Remove specific weight for /dev/sda in this cgroup:: - # echo 8:0 0 > blkio.weight_device - # cat blkio.weight_device + # echo 8:0 0 > blkio.bfq.weight_device + # cat blkio.bfq.weight_device dev weight 8:16 300 -- blkio.time - - disk time allocated to cgroup per device in milliseconds. First + blkio.time + Disk time allocated to cgroup per device in milliseconds. First two fields specify the major and minor number of the device and third field specifies the disk time allocated to group in milliseconds. -- blkio.sectors - - number of sectors transferred to/from disk by the group. First + blkio.sectors + Number of sectors transferred to/from disk by the group. First two fields specify the major and minor number of the device and third field specifies the number of sectors transferred by the group to/from the device. -- blkio.io_service_bytes - - Number of bytes transferred to/from the disk by the group. These + blkio.io_service_bytes + Number of bytes transferred to/from the disk by the group. These are further divided by the type of operation - read or write, sync or async. First two fields specify the major and minor number of the device, third field specifies the operation type and the fourth field specifies the number of bytes. -- blkio.io_serviced - - Number of IOs (bio) issued to the disk by the group. These + blkio.io_serviced + Number of IOs (bio) issued to the disk by the group. These are further divided by the type of operation - read or write, sync or async. First two fields specify the major and minor number of the device, third field specifies the operation type and the fourth field specifies the number of IOs. -- blkio.io_service_time - - Total amount of time between request dispatch and request completion + blkio.io_service_time + Total amount of time between request dispatch and request completion for the IOs done by this cgroup. This is in nanoseconds to make it meaningful for flash devices too. For devices with queue depth of 1, this time represents the actual service time. When queue_depth > 1, @@ -170,8 +175,8 @@ Proportional weight policy files specifies the operation type and the fourth field specifies the io_service_time in ns. -- blkio.io_wait_time - - Total amount of time the IOs for this cgroup spent waiting in the + blkio.io_wait_time + Total amount of time the IOs for this cgroup spent waiting in the scheduler queues for service. This can be greater than the total time elapsed since it is cumulative io_wait_time for all IOs. It is not a measure of total time the cgroup spent waiting but rather a measure of @@ -185,24 +190,24 @@ Proportional weight policy files minor number of the device, third field specifies the operation type and the fourth field specifies the io_wait_time in ns. -- blkio.io_merged - - Total number of bios/requests merged into requests belonging to this + blkio.io_merged + Total number of bios/requests merged into requests belonging to this cgroup. This is further divided by the type of operation - read or write, sync or async. -- blkio.io_queued - - Total number of requests queued up at any given instant for this + blkio.io_queued + Total number of requests queued up at any given instant for this cgroup. This is further divided by the type of operation - read or write, sync or async. -- blkio.avg_queue_size - - Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y. + blkio.avg_queue_size + Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y. The average queue size for this cgroup over the entire time of this cgroup's existence. Queue size samples are taken each time one of the queues of this cgroup gets a timeslice. -- blkio.group_wait_time - - Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y. + blkio.group_wait_time + Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y. This is the amount of time the cgroup had to wait since it became busy (i.e., went from 0 to 1 request queued) to get a timeslice for one of its queues. This is different from the io_wait_time which is the @@ -212,8 +217,8 @@ Proportional weight policy files will only report the group_wait_time accumulated till the last time it got a timeslice and will not include the current delta. -- blkio.empty_time - - Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y. + blkio.empty_time + Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y. This is the amount of time a cgroup spends without any pending requests when not being served, i.e., it does not include any time spent idling for one of the queues of the cgroup. This is in @@ -221,8 +226,8 @@ Proportional weight policy files the stat will only report the empty_time accumulated till the last time it had a pending request and will not include the current delta. -- blkio.idle_time - - Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y. + blkio.idle_time + Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y. This is the amount of time spent by the IO scheduler idling for a given cgroup in anticipation of a better request than the existing ones from other queues/cgroups. This is in nanoseconds. If this is read @@ -230,60 +235,60 @@ Proportional weight policy files idle_time accumulated till the last idle period and will not include the current delta. -- blkio.dequeue - - Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y. This + blkio.dequeue + Debugging aid only enabled if CONFIG_BFQ_CGROUP_DEBUG=y. This gives the statistics about how many a times a group was dequeued from service tree of the device. First two fields specify the major and minor number of the device and third field specifies the number of times a group was dequeued from a particular device. -- blkio.*_recursive - - Recursive version of various stats. These files show the + blkio.*_recursive + Recursive version of various stats. These files show the same information as their non-recursive counterparts but include stats from all the descendant cgroups. Throttling/Upper limit policy files ----------------------------------- -- blkio.throttle.read_bps_device - - Specifies upper limit on READ rate from the device. IO rate is + blkio.throttle.read_bps_device + Specifies upper limit on READ rate from the device. IO rate is specified in bytes per second. Rules are per device. Following is the format:: echo "<major>:<minor> <rate_bytes_per_second>" > /cgrp/blkio.throttle.read_bps_device -- blkio.throttle.write_bps_device - - Specifies upper limit on WRITE rate to the device. IO rate is + blkio.throttle.write_bps_device + Specifies upper limit on WRITE rate to the device. IO rate is specified in bytes per second. Rules are per device. Following is the format:: echo "<major>:<minor> <rate_bytes_per_second>" > /cgrp/blkio.throttle.write_bps_device -- blkio.throttle.read_iops_device - - Specifies upper limit on READ rate from the device. IO rate is + blkio.throttle.read_iops_device + Specifies upper limit on READ rate from the device. IO rate is specified in IO per second. Rules are per device. Following is the format:: echo "<major>:<minor> <rate_io_per_second>" > /cgrp/blkio.throttle.read_iops_device -- blkio.throttle.write_iops_device - - Specifies upper limit on WRITE rate to the device. IO rate is + blkio.throttle.write_iops_device + Specifies upper limit on WRITE rate to the device. IO rate is specified in io per second. Rules are per device. Following is the format:: echo "<major>:<minor> <rate_io_per_second>" > /cgrp/blkio.throttle.write_iops_device -Note: If both BW and IOPS rules are specified for a device, then IO is - subjected to both the constraints. + Note: If both BW and IOPS rules are specified for a device, then IO is + subjected to both the constraints. -- blkio.throttle.io_serviced - - Number of IOs (bio) issued to the disk by the group. These + blkio.throttle.io_serviced + Number of IOs (bio) issued to the disk by the group. These are further divided by the type of operation - read or write, sync or async. First two fields specify the major and minor number of the device, third field specifies the operation type and the fourth field specifies the number of IOs. -- blkio.throttle.io_service_bytes - - Number of bytes transferred to/from the disk by the group. These + blkio.throttle.io_service_bytes + Number of bytes transferred to/from the disk by the group. These are further divided by the type of operation - read or write, sync or async. First two fields specify the major and minor number of the device, third field specifies the operation type and the fourth field @@ -291,6 +296,6 @@ Note: If both BW and IOPS rules are specified for a device, then IO is Common files among various policies ----------------------------------- -- blkio.reset_stats - - Writing an int to this file will result in resetting all the stats + blkio.reset_stats + Writing an int to this file will result in resetting all the stats for that cgroup. diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst index b1e81aa8598a..babbe04c8d37 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -56,6 +56,7 @@ v1 is available under :ref:`Documentation/admin-guide/cgroup-v1/index.rst <cgrou 5-3-3. IO Latency 5-3-3-1. How IO Latency Throttling Works 5-3-3-2. IO Latency Interface Files + 5-3-4. IO Priority 5-4. PID 5-4-1. PID Interface Files 5-5. Cpuset @@ -952,6 +953,21 @@ All cgroup core files are prefixed with "cgroup." it's possible to delete a frozen (and empty) cgroup, as well as create new sub-cgroups. + cgroup.kill + A write-only single value file which exists in non-root cgroups. + The only allowed value is "1". + + Writing "1" to the file causes the cgroup and all descendant cgroups to + be killed. This means that all processes located in the affected cgroup + tree will be killed via SIGKILL. + + Killing a cgroup tree will deal with concurrent forks appropriately and + is protected against migrations. + + In a threaded cgroup, writing this file fails with EOPNOTSUPP as + killing cgroups is a process directed operation, i.e. it affects + the whole thread-group. + Controllers =========== @@ -1866,6 +1882,60 @@ IO Latency Interface Files duration of time between evaluation events. Windows only elapse with IO activity. Idle periods extend the most recent window. +IO Priority +~~~~~~~~~~~ + +A single attribute controls the behavior of the I/O priority cgroup policy, +namely the blkio.prio.class attribute. The following values are accepted for +that attribute: + + no-change + Do not modify the I/O priority class. + + none-to-rt + For requests that do not have an I/O priority class (NONE), + change the I/O priority class into RT. Do not modify + the I/O priority class of other requests. + + restrict-to-be + For requests that do not have an I/O priority class or that have I/O + priority class RT, change it into BE. Do not modify the I/O priority + class of requests that have priority class IDLE. + + idle + Change the I/O priority class of all requests into IDLE, the lowest + I/O priority class. + +The following numerical values are associated with the I/O priority policies: + ++-------------+---+ +| no-change | 0 | ++-------------+---+ +| none-to-rt | 1 | ++-------------+---+ +| rt-to-be | 2 | ++-------------+---+ +| all-to-idle | 3 | ++-------------+---+ + +The numerical value that corresponds to each I/O priority class is as follows: + ++-------------------------------+---+ +| IOPRIO_CLASS_NONE | 0 | ++-------------------------------+---+ +| IOPRIO_CLASS_RT (real-time) | 1 | ++-------------------------------+---+ +| IOPRIO_CLASS_BE (best effort) | 2 | ++-------------------------------+---+ +| IOPRIO_CLASS_IDLE | 3 | ++-------------------------------+---+ + +The algorithm to set the I/O priority class for a request is as follows: + +- Translate the I/O priority class policy into a number. +- Change the request I/O priority class into the maximum of the I/O priority + class policy number and the numerical I/O priority class. + PID --- @@ -1986,6 +2056,17 @@ Cpuset Interface Files The value of "cpuset.mems" stays constant until the next update and won't be affected by any memory nodes hotplug events. + Setting a non-empty value to "cpuset.mems" causes memory of + tasks within the cgroup to be migrated to the designated nodes if + they are currently using memory outside of the designated nodes. + + There is a cost for this memory migration. The migration + may not be complete and some memory pages may be left behind. + So it is recommended that "cpuset.mems" should be set properly + before spawning new tasks into the cpuset. Even if there is + a need to change "cpuset.mems" with active tasks, it shouldn't + be done frequently. + cpuset.mems.effective A read-only multiple values file which exists on all cpuset-enabled cgroups. diff --git a/Documentation/admin-guide/cputopology.rst b/Documentation/admin-guide/cputopology.rst index b90dafcc8237..b085dbac60a5 100644 --- a/Documentation/admin-guide/cputopology.rst +++ b/Documentation/admin-guide/cputopology.rst @@ -2,87 +2,10 @@ How CPU topology info is exported via sysfs =========================================== -Export CPU topology info via sysfs. Items (attributes) are similar -to /proc/cpuinfo output of some architectures. They reside in -/sys/devices/system/cpu/cpuX/topology/: - -physical_package_id: - - physical package id of cpuX. Typically corresponds to a physical - socket number, but the actual value is architecture and platform - dependent. - -die_id: - - the CPU die ID of cpuX. Typically it is the hardware platform's - identifier (rather than the kernel's). The actual value is - architecture and platform dependent. - -core_id: - - the CPU core ID of cpuX. Typically it is the hardware platform's - identifier (rather than the kernel's). The actual value is - architecture and platform dependent. - -book_id: - - the book ID of cpuX. Typically it is the hardware platform's - identifier (rather than the kernel's). The actual value is - architecture and platform dependent. - -drawer_id: - - the drawer ID of cpuX. Typically it is the hardware platform's - identifier (rather than the kernel's). The actual value is - architecture and platform dependent. - -core_cpus: - - internal kernel map of CPUs within the same core. - (deprecated name: "thread_siblings") - -core_cpus_list: - - human-readable list of CPUs within the same core. - (deprecated name: "thread_siblings_list"); - -package_cpus: - - internal kernel map of the CPUs sharing the same physical_package_id. - (deprecated name: "core_siblings") - -package_cpus_list: - - human-readable list of CPUs sharing the same physical_package_id. - (deprecated name: "core_siblings_list") - -die_cpus: - - internal kernel map of CPUs within the same die. - -die_cpus_list: - - human-readable list of CPUs within the same die. - -book_siblings: - - internal kernel map of cpuX's hardware threads within the same - book_id. - -book_siblings_list: - - human-readable list of cpuX's hardware threads within the same - book_id. - -drawer_siblings: - - internal kernel map of cpuX's hardware threads within the same - drawer_id. - -drawer_siblings_list: - - human-readable list of cpuX's hardware threads within the same - drawer_id. +CPU topology info is exported via sysfs. Items (attributes) are similar +to /proc/cpuinfo output of some architectures. They reside in +/sys/devices/system/cpu/cpuX/topology/. Please refer to the ABI file: +Documentation/ABI/stable/sysfs-devices-system-cpu. Architecture-neutral, drivers/base/topology.c, exports these attributes. However, the book and drawer related sysfs files will only be created if @@ -135,9 +58,9 @@ source for the output is in brackets ("[]"). [NR_CPUS-1] offline: CPUs that are not online because they have been - HOTPLUGGED off (see cpu-hotplug.txt) or exceed the limit - of CPUs allowed by the kernel configuration (kernel_max - above). [~cpu_online_mask + cpus >= NR_CPUS] + HOTPLUGGED off or exceed the limit of CPUs allowed by the + kernel configuration (kernel_max above). + [~cpu_online_mask + cpus >= NR_CPUS] online: CPUs that are online and being scheduled [cpu_online_mask] @@ -173,5 +96,5 @@ online.):: possible: 0-127 present: 0-3 -See cpu-hotplug.txt for the possible_cpus=NUM kernel start parameter -as well as more information on the various cpumasks. +See Documentation/core-api/cpu_hotplug.rst for the possible_cpus=NUM +kernel start parameter as well as more information on the various cpumasks. diff --git a/Documentation/admin-guide/device-mapper/dm-ima.rst b/Documentation/admin-guide/device-mapper/dm-ima.rst new file mode 100644 index 000000000000..a4aa50a828e0 --- /dev/null +++ b/Documentation/admin-guide/device-mapper/dm-ima.rst @@ -0,0 +1,715 @@ +====== +dm-ima +====== + +For a given system, various external services/infrastructure tools +(including the attestation service) interact with it - both during the +setup and during rest of the system run-time. They share sensitive data +and/or execute critical workload on that system. The external services +may want to verify the current run-time state of the relevant kernel +subsystems before fully trusting the system with business-critical +data/workload. + +Device mapper plays a critical role on a given system by providing +various important functionalities to the block devices using various +target types like crypt, verity, integrity etc. Each of these target +types’ functionalities can be configured with various attributes. +The attributes chosen to configure these target types can significantly +impact the security profile of the block device, and in-turn, of the +system itself. For instance, the type of encryption algorithm and the +key size determines the strength of encryption for a given block device. + +Therefore, verifying the current state of various block devices as well +as their various target attributes is crucial for external services before +fully trusting the system with business-critical data/workload. + +IMA kernel subsystem provides the necessary functionality for +device mapper to measure the state and configuration of +various block devices - + +- by device mapper itself, from within the kernel, +- in a tamper resistant way, +- and re-measured - triggered on state/configuration change. + +Setting the IMA Policy: +======================= +For IMA to measure the data on a given system, the IMA policy on the +system needs to be updated to have following line, and the system needs +to be restarted for the measurements to take effect. + +:: + + /etc/ima/ima-policy + measure func=CRITICAL_DATA label=device-mapper template=ima-buf + +The measurements will be reflected in the IMA logs, which are located at: + +:: + + /sys/kernel/security/integrity/ima/ascii_runtime_measurements + /sys/kernel/security/integrity/ima/binary_runtime_measurements + +Then IMA ASCII measurement log has the following format: + +:: + + <PCR> <TEMPLATE_DATA_DIGEST> <TEMPLATE_NAME> <TEMPLATE_DATA> + + PCR := Platform Configuration Register, in which the values are registered. + This is applicable if TPM chip is in use. + + TEMPLATE_DATA_DIGEST := Template data digest of the IMA record. + TEMPLATE_NAME := Template name that registered the integrity value (e.g. ima-buf). + + TEMPLATE_DATA := <ALG> ":" <EVENT_DIGEST> <EVENT_NAME> <EVENT_DATA> + It contains data for the specific event to be measured, + in a given template data format. + + ALG := Algorithm to compute event digest + EVENT_DIGEST := Digest of the event data + EVENT_NAME := Description of the event (e.g. 'dm_table_load'). + EVENT_DATA := The event data to be measured. + +| + +| *NOTE #1:* +| The DM target data measured by IMA subsystem can alternatively + be queried from userspace by setting DM_IMA_MEASUREMENT_FLAG with + DM_TABLE_STATUS_CMD. + +| + +| *NOTE #2:* +| The Kernel configuration CONFIG_IMA_DISABLE_HTABLE allows measurement of duplicate records. +| To support recording duplicate IMA events in the IMA log, the Kernel needs to be configured with + CONFIG_IMA_DISABLE_HTABLE=y. + +Supported Device States: +======================== +Following device state changes will trigger IMA measurements: + + 1. Table load + #. Device resume + #. Device remove + #. Table clear + #. Device rename + +1. Table load: +--------------- +When a new table is loaded in a device's inactive table slot, +the device information and target specific details from the +targets in the table are measured. + +The IMA measurement log has the following format for 'dm_table_load': + +:: + + EVENT_NAME := "dm_table_load" + EVENT_DATA := <dm_version_str> ";" <device_metadata> ";" <table_load_data> + + dm_version_str := "dm_version=" <N> "." <N> "." <N> + Same as Device Mapper driver version. + device_metadata := <device_name> "," <device_uuid> "," <device_major> "," <device_minor> "," + <minor_count> "," <num_device_targets> ";" + + device_name := "name=" <dm-device-name> + device_uuid := "uuid=" <dm-device-uuid> + device_major := "major=" <N> + device_minor := "minor=" <N> + minor_count := "minor_count=" <N> + num_device_targets := "num_targets=" <N> + dm-device-name := Name of the device. If it contains special characters like '\', ',', ';', + they are prefixed with '\'. + dm-device-uuid := UUID of the device. If it contains special characters like '\', ',', ';', + they are prefixed with '\'. + + table_load_data := <target_data> + Represents the data (as name=value pairs) from various targets in the table, + which is being loaded into the DM device's inactive table slot. + target_data := <target_data_row> | <target_data><target_data_row> + + target_data_row := <target_index> "," <target_begin> "," <target_len> "," <target_name> "," + <target_version> "," <target_attributes> ";" + target_index := "target_index=" <N> + Represents nth target in the table (from 0 to N-1 targets specified in <num_device_targets>) + If all the data for N targets doesn't fit in the given buffer - then the data that fits + in the buffer (say from target 0 to x) is measured in a given IMA event. + The remaining data from targets x+1 to N-1 is measured in the subsequent IMA events, + with the same format as that of 'dm_table_load' + i.e. <dm_version_str> ";" <device_metadata> ";" <table_load_data>. + + target_begin := "target_begin=" <N> + target_len := "target_len=" <N> + target_name := Name of the target. 'linear', 'crypt', 'integrity' etc. + The targets that are supported for IMA measurements are documented below in the + 'Supported targets' section. + target_version := "target_version=" <N> "." <N> "." <N> + target_attributes := Data containing comma separated list of name=value pairs of target specific attributes. + + For instance, if a linear device is created with the following table entries, + # dmsetup create linear1 + 0 2 linear /dev/loop0 512 + 2 2 linear /dev/loop0 512 + 4 2 linear /dev/loop0 512 + 6 2 linear /dev/loop0 512 + + Then IMA ASCII measurement log will have the following entry: + (converted from ASCII to text for readability) + + 10 a8c5ff755561c7a28146389d1514c318592af49a ima-buf sha256:4d73481ecce5eadba8ab084640d85bb9ca899af4d0a122989252a76efadc5b72 + dm_table_load + dm_version=4.45.0; + name=linear1,uuid=,major=253,minor=0,minor_count=1,num_targets=4; + target_index=0,target_begin=0,target_len=2,target_name=linear,target_version=1.4.0,device_name=7:0,start=512; + target_index=1,target_begin=2,target_len=2,target_name=linear,target_version=1.4.0,device_name=7:0,start=512; + target_index=2,target_begin=4,target_len=2,target_name=linear,target_version=1.4.0,device_name=7:0,start=512; + target_index=3,target_begin=6,target_len=2,target_name=linear,target_version=1.4.0,device_name=7:0,start=512; + +2. Device resume: +------------------ +When a suspended device is resumed, the device information and the hash of the +data from previous load of an active table are measured. + +The IMA measurement log has the following format for 'dm_device_resume': + +:: + + EVENT_NAME := "dm_device_resume" + EVENT_DATA := <dm_version_str> ";" <device_metadata> ";" <active_table_hash> ";" <current_device_capacity> ";" + + dm_version_str := As described in the 'Table load' section above. + device_metadata := As described in the 'Table load' section above. + active_table_hash := "active_table_hash=" <table_hash_alg> ":" <table_hash> + Rerpresents the hash of the IMA data being measured for the + active table for the device. + table_hash_alg := Algorithm used to compute the hash. + table_hash := Hash of the (<dm_version_str> ";" <device_metadata> ";" <table_load_data> ";") + as described in the 'dm_table_load' above. + Note: If the table_load data spans across multiple IMA 'dm_table_load' + events for a given device, the hash is computed combining all the event data + i.e. (<dm_version_str> ";" <device_metadata> ";" <table_load_data> ";") + across all those events. + current_device_capacity := "current_device_capacity=" <N> + + For instance, if a linear device is resumed with the following command, + #dmsetup resume linear1 + + then IMA ASCII measurement log will have an entry with: + (converted from ASCII to text for readability) + + 10 56c00cc062ffc24ccd9ac2d67d194af3282b934e ima-buf sha256:e7d12c03b958b4e0e53e7363a06376be88d98a1ac191fdbd3baf5e4b77f329b6 + dm_device_resume + dm_version=4.45.0; + name=linear1,uuid=,major=253,minor=0,minor_count=1,num_targets=4; + active_table_hash=sha256:4d73481ecce5eadba8ab084640d85bb9ca899af4d0a122989252a76efadc5b72;current_device_capacity=8; + +3. Device remove: +------------------ +When a device is removed, the device information and a sha256 hash of the +data from an active and inactive table are measured. + +The IMA measurement log has the following format for 'dm_device_remove': + +:: + + EVENT_NAME := "dm_device_remove" + EVENT_DATA := <dm_version_str> ";" <device_active_metadata> ";" <device_inactive_metadata> ";" + <active_table_hash> "," <inactive_table_hash> "," <remove_all> ";" <current_device_capacity> ";" + + dm_version_str := As described in the 'Table load' section above. + device_active_metadata := Device metadata that reflects the currently loaded active table. + The format is same as 'device_metadata' described in the 'Table load' section above. + device_inactive_metadata := Device metadata that reflects the inactive table. + The format is same as 'device_metadata' described in the 'Table load' section above. + active_table_hash := Hash of the currently loaded active table. + The format is same as 'active_table_hash' described in the 'Device resume' section above. + inactive_table_hash := Hash of the inactive table. + The format is same as 'active_table_hash' described in the 'Device resume' section above. + remove_all := "remove_all=" <yes_no> + yes_no := "y" | "n" + current_device_capacity := "current_device_capacity=" <N> + + For instance, if a linear device is removed with the following command, + #dmsetup remove l1 + + then IMA ASCII measurement log will have the following entry: + (converted from ASCII to text for readability) + + 10 790e830a3a7a31590824ac0642b3b31c2d0e8b38 ima-buf sha256:ab9f3c959367a8f5d4403d6ce9c3627dadfa8f9f0e7ec7899299782388de3840 + dm_device_remove + dm_version=4.45.0; + device_active_metadata=name=l1,uuid=,major=253,minor=2,minor_count=1,num_targets=2; + device_inactive_metadata=name=l1,uuid=,major=253,minor=2,minor_count=1,num_targets=1; + active_table_hash=sha256:4a7e62efaebfc86af755831998b7db6f59b60d23c9534fb16a4455907957953a, + inactive_table_hash=sha256:9d79c175bc2302d55a183e8f50ad4bafd60f7692fd6249e5fd213e2464384b86,remove_all=n; + current_device_capacity=2048; + +4. Table clear: +---------------- +When an inactive table is cleared from the device, the device information and a sha256 hash of the +data from an inactive table are measured. + +The IMA measurement log has the following format for 'dm_table_clear': + +:: + + EVENT_NAME := "dm_table_clear" + EVENT_DATA := <dm_version_str> ";" <device_inactive_metadata> ";" <inactive_table_hash> ";" <current_device_capacity> ";" + + dm_version_str := As described in the 'Table load' section above. + device_inactive_metadata := Device metadata that was captured during the load time inactive table being cleared. + The format is same as 'device_metadata' described in the 'Table load' section above. + inactive_table_hash := Hash of the inactive table being cleared from the device. + The format is same as 'active_table_hash' described in the 'Device resume' section above. + current_device_capacity := "current_device_capacity=" <N> + + For instance, if a linear device's inactive table is cleared, + #dmsetup clear l1 + + then IMA ASCII measurement log will have an entry with: + (converted from ASCII to text for readability) + + 10 77d347408f557f68f0041acb0072946bb2367fe5 ima-buf sha256:42f9ca22163fdfa548e6229dece2959bc5ce295c681644240035827ada0e1db5 + dm_table_clear + dm_version=4.45.0; + name=l1,uuid=,major=253,minor=2,minor_count=1,num_targets=1; + inactive_table_hash=sha256:75c0dc347063bf474d28a9907037eba060bfe39d8847fc0646d75e149045d545;current_device_capacity=1024; + +5. Device rename: +------------------ +When an device's NAME or UUID is changed, the device information and the new NAME and UUID +are measured. + +The IMA measurement log has the following format for 'dm_device_rename': + +:: + + EVENT_NAME := "dm_device_rename" + EVENT_DATA := <dm_version_str> ";" <device_active_metadata> ";" <new_device_name> "," <new_device_uuid> ";" <current_device_capacity> ";" + + dm_version_str := As described in the 'Table load' section above. + device_active_metadata := Device metadata that reflects the currently loaded active table. + The format is same as 'device_metadata' described in the 'Table load' section above. + new_device_name := "new_name=" <dm-device-name> + dm-device-name := Same as <dm-device-name> described in 'Table load' section above + new_device_uuid := "new_uuid=" <dm-device-uuid> + dm-device-uuid := Same as <dm-device-uuid> described in 'Table load' section above + current_device_capacity := "current_device_capacity=" <N> + + E.g 1: if a linear device's name is changed with the following command, + #dmsetup rename linear1 --setuuid 1234-5678 + + then IMA ASCII measurement log will have an entry with: + (converted from ASCII to text for readability) + + 10 8b0423209b4c66ac1523f4c9848c9b51ee332f48 ima-buf sha256:6847b7258134189531db593e9230b257c84f04038b5a18fd2e1473860e0569ac + dm_device_rename + dm_version=4.45.0; + name=linear1,uuid=,major=253,minor=2,minor_count=1,num_targets=1;new_name=linear1,new_uuid=1234-5678; + current_device_capacity=1024; + + E.g 2: if a linear device's name is changed with the following command, + # dmsetup rename linear1 linear=2 + + then IMA ASCII measurement log will have an entry with: + (converted from ASCII to text for readability) + + 10 bef70476b99c2bdf7136fae033aa8627da1bf76f ima-buf sha256:8c6f9f53b9ef9dc8f92a2f2cca8910e622543d0f0d37d484870cb16b95111402 + dm_device_rename + dm_version=4.45.0; + name=linear1,uuid=1234-5678,major=253,minor=2,minor_count=1,num_targets=1; + new_name=linear\=2,new_uuid=1234-5678; + current_device_capacity=1024; + +Supported targets: +================== + +Following targets are supported to measure their data using IMA: + + 1. cache + #. crypt + #. integrity + #. linear + #. mirror + #. multipath + #. raid + #. snapshot + #. striped + #. verity + +1. cache +--------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'cache' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <metadata_mode> "," <cache_metadata_device> "," + <cache_device> "," <cache_origin_device> "," <writethrough> "," <writeback> "," + <passthrough> "," <no_discard_passdown> ";" + + target_name := "target_name=cache" + target_version := "target_version=" <N> "." <N> "." <N> + metadata_mode := "metadata_mode=" <cache_metadata_mode> + cache_metadata_mode := "fail" | "ro" | "rw" + cache_device := "cache_device=" <cache_device_name_string> + cache_origin_device := "cache_origin_device=" <cache_origin_device_string> + writethrough := "writethrough=" <yes_no> + writeback := "writeback=" <yes_no> + passthrough := "passthrough=" <yes_no> + no_discard_passdown := "no_discard_passdown=" <yes_no> + yes_no := "y" | "n" + + E.g. + When a 'cache' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'cache' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0;name=cache1,uuid=cache_uuid,major=253,minor=2,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=28672,target_name=cache,target_version=2.2.0,metadata_mode=rw, + cache_metadata_device=253:4,cache_device=253:3,cache_origin_device=253:5,writethrough=y,writeback=n, + passthrough=n,metadata2=y,no_discard_passdown=n; + + +2. crypt +--------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'crypt' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <allow_discards> "," <same_cpu_crypt> "," + <submit_from_crypt_cpus> "," <no_read_workqueue> "," <no_write_workqueue> "," + <iv_large_sectors> "," <iv_large_sectors> "," [<integrity_tag_size> ","] [<cipher_auth> ","] + [<sector_size> ","] [<cipher_string> ","] <key_size> "," <key_parts> "," + <key_extra_size> "," <key_mac_size> ";" + + target_name := "target_name=crypt" + target_version := "target_version=" <N> "." <N> "." <N> + allow_discards := "allow_discards=" <yes_no> + same_cpu_crypt := "same_cpu_crypt=" <yes_no> + submit_from_crypt_cpus := "submit_from_crypt_cpus=" <yes_no> + no_read_workqueue := "no_read_workqueue=" <yes_no> + no_write_workqueue := "no_write_workqueue=" <yes_no> + iv_large_sectors := "iv_large_sectors=" <yes_no> + integrity_tag_size := "integrity_tag_size=" <N> + cipher_auth := "cipher_auth=" <string> + sector_size := "sector_size=" <N> + cipher_string := "cipher_string=" + key_size := "key_size=" <N> + key_parts := "key_parts=" <N> + key_extra_size := "key_extra_size=" <N> + key_mac_size := "key_mac_size=" <N> + yes_no := "y" | "n" + + E.g. + When a 'crypt' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'crypt' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=crypt1,uuid=crypt_uuid1,major=253,minor=0,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=1953125,target_name=crypt,target_version=1.23.0, + allow_discards=y,same_cpu=n,submit_from_crypt_cpus=n,no_read_workqueue=n,no_write_workqueue=n, + iv_large_sectors=n,cipher_string=aes-xts-plain64,key_size=32,key_parts=1,key_extra_size=0,key_mac_size=0; + +3. integrity +------------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'integrity' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <dev_name> "," <start> + <tag_size> "," <mode> "," [<meta_device> ","] [<block_size> ","] <recalculate> "," + <allow_discards> "," <fix_padding> "," <fix_hmac> "," <legacy_recalculate> "," + <journal_sectors> "," <interleave_sectors> "," <buffer_sectors> ";" + + target_name := "target_name=integrity" + target_version := "target_version=" <N> "." <N> "." <N> + dev_name := "dev_name=" <device_name_str> + start := "start=" <N> + tag_size := "tag_size=" <N> + mode := "mode=" <integrity_mode_str> + integrity_mode_str := "J" | "B" | "D" | "R" + meta_device := "meta_device=" <meta_device_str> + block_size := "block_size=" <N> + recalculate := "recalculate=" <yes_no> + allow_discards := "allow_discards=" <yes_no> + fix_padding := "fix_padding=" <yes_no> + fix_hmac := "fix_hmac=" <yes_no> + legacy_recalculate := "legacy_recalculate=" <yes_no> + journal_sectors := "journal_sectors=" <N> + interleave_sectors := "interleave_sectors=" <N> + buffer_sectors := "buffer_sectors=" <N> + yes_no := "y" | "n" + + E.g. + When a 'integrity' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'integrity' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=integrity1,uuid=,major=253,minor=1,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=7856,target_name=integrity,target_version=1.10.0, + dev_name=253:0,start=0,tag_size=32,mode=J,recalculate=n,allow_discards=n,fix_padding=n, + fix_hmac=n,legacy_recalculate=n,journal_sectors=88,interleave_sectors=32768,buffer_sectors=128; + + +4. linear +---------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'linear' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <device_name> <,> <start> ";" + + target_name := "target_name=linear" + target_version := "target_version=" <N> "." <N> "." <N> + device_name := "device_name=" <linear_device_name_str> + start := "start=" <N> + + E.g. + When a 'linear' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'linear' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=linear1,uuid=linear_uuid1,major=253,minor=2,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=28672,target_name=linear,target_version=1.4.0, + device_name=253:1,start=2048; + +5. mirror +---------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'mirror' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <nr_mirrors> "," + <mirror_device_data> "," <handle_errors> "," <keep_log> "," <log_type_status> ";" + + target_name := "target_name=mirror" + target_version := "target_version=" <N> "." <N> "." <N> + nr_mirrors := "nr_mirrors=" <NR> + mirror_device_data := <mirror_device_row> | <mirror_device_data><mirror_device_row> + mirror_device_row is repeated <NR> times - for <NR> described in <nr_mirrors>. + mirror_device_row := <mirror_device_name> "," <mirror_device_status> + mirror_device_name := "mirror_device_" <X> "=" <mirror_device_name_str> + where <X> ranges from 0 to (<NR> -1) - for <NR> described in <nr_mirrors>. + mirror_device_status := "mirror_device_" <X> "_status=" <mirror_device_status_char> + where <X> ranges from 0 to (<NR> -1) - for <NR> described in <nr_mirrors>. + mirror_device_status_char := "A" | "F" | "D" | "S" | "R" | "U" + handle_errors := "handle_errors=" <yes_no> + keep_log := "keep_log=" <yes_no> + log_type_status := "log_type_status=" <log_type_status_str> + yes_no := "y" | "n" + + E.g. + When a 'mirror' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'mirror' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=mirror1,uuid=mirror_uuid1,major=253,minor=6,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=2048,target_name=mirror,target_version=1.14.0,nr_mirrors=2, + mirror_device_0=253:4,mirror_device_0_status=A, + mirror_device_1=253:5,mirror_device_1_status=A, + handle_errors=y,keep_log=n,log_type_status=; + +6. multipath +------------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'multipath' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <nr_priority_groups> + ["," <pg_state> "," <priority_groups> "," <priority_group_paths>] ";" + + target_name := "target_name=multipath" + target_version := "target_version=" <N> "." <N> "." <N> + nr_priority_groups := "nr_priority_groups=" <NPG> + priority_groups := <priority_groups_row>|<priority_groups_row><priority_groups> + priority_groups_row := "pg_state_" <X> "=" <pg_state_str> "," "nr_pgpaths_" <X> "=" <NPGP> "," + "path_selector_name_" <X> "=" <string> "," <priority_group_paths> + where <X> ranges from 0 to (<NPG> -1) - for <NPG> described in <nr_priority_groups>. + pg_state_str := "E" | "A" | "D" + <priority_group_paths> := <priority_group_paths_row> | <priority_group_paths_row><priority_group_paths> + priority_group_paths_row := "path_name_" <X> "_" <Y> "=" <string> "," "is_active_" <X> "_" <Y> "=" <is_active_str> + "fail_count_" <X> "_" <Y> "=" <N> "," "path_selector_status_" <X> "_" <Y> "=" <path_selector_status_str> + where <X> ranges from 0 to (<NPG> -1) - for <NPG> described in <nr_priority_groups>, + and <Y> ranges from 0 to (<NPGP> -1) - for <NPGP> described in <priority_groups_row>. + is_active_str := "A" | "F" + + E.g. + When a 'multipath' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'multipath' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=mp,uuid=,major=253,minor=0,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=2097152,target_name=multipath,target_version=1.14.0,nr_priority_groups=2, + pg_state_0=E,nr_pgpaths_0=2,path_selector_name_0=queue-length, + path_name_0_0=8:16,is_active_0_0=A,fail_count_0_0=0,path_selector_status_0_0=, + path_name_0_1=8:32,is_active_0_1=A,fail_count_0_1=0,path_selector_status_0_1=, + pg_state_1=E,nr_pgpaths_1=2,path_selector_name_1=queue-length, + path_name_1_0=8:48,is_active_1_0=A,fail_count_1_0=0,path_selector_status_1_0=, + path_name_1_1=8:64,is_active_1_1=A,fail_count_1_1=0,path_selector_status_1_1=; + +7. raid +-------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'raid' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <raid_type> "," <raid_disks> "," <raid_state> + <raid_device_status> ["," journal_dev_mode] ";" + + target_name := "target_name=raid" + target_version := "target_version=" <N> "." <N> "." <N> + raid_type := "raid_type=" <raid_type_str> + raid_disks := "raid_disks=" <NRD> + raid_state := "raid_state=" <raid_state_str> + raid_state_str := "frozen" | "reshape" |"resync" | "check" | "repair" | "recover" | "idle" |"undef" + raid_device_status := <raid_device_status_row> | <raid_device_status_row><raid_device_status> + <raid_device_status_row> is repeated <NRD> times - for <NRD> described in <raid_disks>. + raid_device_status_row := "raid_device_" <X> "_status=" <raid_device_status_str> + where <X> ranges from 0 to (<NRD> -1) - for <NRD> described in <raid_disks>. + raid_device_status_str := "A" | "D" | "a" | "-" + journal_dev_mode := "journal_dev_mode=" <journal_dev_mode_str> + journal_dev_mode_str := "writethrough" | "writeback" | "invalid" + + E.g. + When a 'raid' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'raid' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=raid_LV1,uuid=uuid_raid_LV1,major=253,minor=12,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=2048,target_name=raid,target_version=1.15.1, + raid_type=raid10,raid_disks=4,raid_state=idle, + raid_device_0_status=A, + raid_device_1_status=A, + raid_device_2_status=A, + raid_device_3_status=A; + + +8. snapshot +------------ +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'snapshot' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <snap_origin_name> "," + <snap_cow_name> "," <snap_valid> "," <snap_merge_failed> "," <snapshot_overflowed> ";" + + target_name := "target_name=snapshot" + target_version := "target_version=" <N> "." <N> "." <N> + snap_origin_name := "snap_origin_name=" <string> + snap_cow_name := "snap_cow_name=" <string> + snap_valid := "snap_valid=" <yes_no> + snap_merge_failed := "snap_merge_failed=" <yes_no> + snapshot_overflowed := "snapshot_overflowed=" <yes_no> + yes_no := "y" | "n" + + E.g. + When a 'snapshot' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'snapshot' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=snap1,uuid=snap_uuid1,major=253,minor=13,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=4096,target_name=snapshot,target_version=1.16.0, + snap_origin_name=253:11,snap_cow_name=253:12,snap_valid=y,snap_merge_failed=n,snapshot_overflowed=n; + +9. striped +----------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'striped' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <stripes> "," <chunk_size> "," + <stripe_data> ";" + + target_name := "target_name=striped" + target_version := "target_version=" <N> "." <N> "." <N> + stripes := "stripes=" <NS> + chunk_size := "chunk_size=" <N> + stripe_data := <stripe_data_row>|<stripe_data><stripe_data_row> + stripe_data_row := <stripe_device_name> "," <stripe_physical_start> "," <stripe_status> + stripe_device_name := "stripe_" <X> "_device_name=" <stripe_device_name_str> + where <X> ranges from 0 to (<NS> -1) - for <NS> described in <stripes>. + stripe_physical_start := "stripe_" <X> "_physical_start=" <N> + where <X> ranges from 0 to (<NS> -1) - for <NS> described in <stripes>. + stripe_status := "stripe_" <X> "_status=" <stripe_status_str> + where <X> ranges from 0 to (<NS> -1) - for <NS> described in <stripes>. + stripe_status_str := "D" | "A" + + E.g. + When a 'striped' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'striped' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=striped1,uuid=striped_uuid1,major=253,minor=5,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=640,target_name=striped,target_version=1.6.0,stripes=2,chunk_size=64, + stripe_0_device_name=253:0,stripe_0_physical_start=2048,stripe_0_status=A, + stripe_1_device_name=253:3,stripe_1_physical_start=2048,stripe_1_status=A; + +10. verity +---------- +The 'target_attributes' (described as part of EVENT_DATA in 'Table load' +section above) has the following data format for 'verity' target. + +:: + + target_attributes := <target_name> "," <target_version> "," <hash_failed> "," <verity_version> "," + <data_device_name> "," <hash_device_name> "," <verity_algorithm> "," <root_digest> "," + <salt> "," <ignore_zero_blocks> "," <check_at_most_once> ["," <root_hash_sig_key_desc>] + ["," <verity_mode>] ";" + + target_name := "target_name=verity" + target_version := "target_version=" <N> "." <N> "." <N> + hash_failed := "hash_failed=" <hash_failed_str> + hash_failed_str := "C" | "V" + verity_version := "verity_version=" <verity_version_str> + data_device_name := "data_device_name=" <data_device_name_str> + hash_device_name := "hash_device_name=" <hash_device_name_str> + verity_algorithm := "verity_algorithm=" <verity_algorithm_str> + root_digest := "root_digest=" <root_digest_str> + salt := "salt=" <salt_str> + salt_str := "-" <verity_salt_str> + ignore_zero_blocks := "ignore_zero_blocks=" <yes_no> + check_at_most_once := "check_at_most_once=" <yes_no> + root_hash_sig_key_desc := "root_hash_sig_key_desc=" + verity_mode := "verity_mode=" <verity_mode_str> + verity_mode_str := "ignore_corruption" | "restart_on_corruption" | "panic_on_corruption" | "invalid" + yes_no := "y" | "n" + + E.g. + When a 'verity' target is loaded, then IMA ASCII measurement log will have an entry + similar to the following, depicting what 'verity' attributes are measured in EVENT_DATA + for 'dm_table_load' event. + (converted from ASCII to text for readability) + + dm_version=4.45.0; + name=test-verity,uuid=,major=253,minor=2,minor_count=1,num_targets=1; + target_index=0,target_begin=0,target_len=1953120,target_name=verity,target_version=1.8.0,hash_failed=V, + verity_version=1,data_device_name=253:1,hash_device_name=253:0,verity_algorithm=sha256, + root_digest=29cb87e60ce7b12b443ba6008266f3e41e93e403d7f298f8e3f316b29ff89c5e, + salt=e48da609055204e89ae53b655ca2216dd983cf3cb829f34f63a297d106d53e2d, + ignore_zero_blocks=n,check_at_most_once=n; diff --git a/Documentation/admin-guide/device-mapper/index.rst b/Documentation/admin-guide/device-mapper/index.rst index 6cf8adc86fa8..cde52cc09645 100644 --- a/Documentation/admin-guide/device-mapper/index.rst +++ b/Documentation/admin-guide/device-mapper/index.rst @@ -13,6 +13,7 @@ Device Mapper dm-dust dm-ebs dm-flakey + dm-ima dm-init dm-integrity dm-io diff --git a/Documentation/admin-guide/device-mapper/writecache.rst b/Documentation/admin-guide/device-mapper/writecache.rst index dce0184e07ca..10429779a91a 100644 --- a/Documentation/admin-guide/device-mapper/writecache.rst +++ b/Documentation/admin-guide/device-mapper/writecache.rst @@ -12,7 +12,6 @@ first sector should contain valid superblock from previous invocation. Constructor parameters: 1. type of the cache device - "p" or "s" - - p - persistent memory - s - SSD 2. the underlying device that will be cached @@ -21,7 +20,6 @@ Constructor parameters: size) 5. the number of optional parameters (the parameters with an argument count as two) - start_sector n (default: 0) offset from the start of cache device in 512-byte sectors high_watermark n (default: 50) @@ -53,19 +51,50 @@ Constructor parameters: - some underlying devices perform better with fua, some with nofua. The user should test it + cleaner + when this option is activated (either in the constructor + arguments or by a message), the cache will not promote + new writes (however, writes to already cached blocks are + promoted, to avoid data corruption due to misordered + writes) and it will gradually writeback any cached + data. The userspace can then monitor the cleaning + process with "dmsetup status". When the number of cached + blocks drops to zero, userspace can unload the + dm-writecache target and replace it with dm-linear or + other targets. + max_age n + specifies the maximum age of a block in milliseconds. If + a block is stored in the cache for too long, it will be + written to the underlying device and cleaned up. + metadata_only + only metadata is promoted to the cache. This option + improves performance for heavier REQ_META workloads. + pause_writeback n (default: 3000) + pause writeback if there was some write I/O redirected to + the origin volume in the last n milliseconds Status: 1. error indicator - 0 if there was no error, otherwise error number 2. the number of blocks 3. the number of free blocks 4. the number of blocks under writeback +5. the number of read requests +6. the number of read requests that hit the cache +7. the number of write requests +8. the number of write requests that hit uncommitted block +9. the number of write requests that hit committed block +10. the number of write requests that bypass the cache +11. the number of write requests that are allocated in the cache +12. the number of write requests that are blocked on the freelist +13. the number of flush requests +14. the number of discard requests Messages: flush - flush the cache device. The message returns successfully + Flush the cache device. The message returns successfully if the cache device was flushed without an error flush_on_suspend - flush the cache device on next suspend. Use this message + Flush the cache device on next suspend. Use this message when you are going to remove the cache device. The proper sequence for removing the cache device is: @@ -77,3 +106,7 @@ Messages: 5. resume the device, so that it will use the linear target 6. the cache device is now inactive and it can be deleted + cleaner + See above "cleaner" constructor documentation. + clear_stats + Clear the statistics that are reported on the status line diff --git a/Documentation/admin-guide/devices.txt b/Documentation/admin-guide/devices.txt index 9c2be821c225..922c23bb4372 100644 --- a/Documentation/admin-guide/devices.txt +++ b/Documentation/admin-guide/devices.txt @@ -2993,10 +2993,10 @@ 65 = /dev/infiniband/issm1 Second InfiniBand IsSM device ... 127 = /dev/infiniband/issm63 63rd InfiniBand IsSM device - 128 = /dev/infiniband/uverbs0 First InfiniBand verbs device - 129 = /dev/infiniband/uverbs1 Second InfiniBand verbs device + 192 = /dev/infiniband/uverbs0 First InfiniBand verbs device + 193 = /dev/infiniband/uverbs1 Second InfiniBand verbs device ... - 159 = /dev/infiniband/uverbs31 31st InfiniBand verbs device + 223 = /dev/infiniband/uverbs31 31st InfiniBand verbs device 232 char Biometric Devices 0 = /dev/biometric/sensor0/fingerprint first fingerprint sensor on first device diff --git a/Documentation/admin-guide/ext4.rst b/Documentation/admin-guide/ext4.rst index d2795ca6821e..4c559e08d11e 100644 --- a/Documentation/admin-guide/ext4.rst +++ b/Documentation/admin-guide/ext4.rst @@ -392,7 +392,7 @@ When mounting an ext4 filesystem, the following option are accepted: dax Use direct access (no page cache). See - Documentation/filesystems/dax.txt. Note that this option is + Documentation/filesystems/dax.rst. Note that this option is incompatible with data=journal. inlinecrypt diff --git a/Documentation/admin-guide/hw-vuln/core-scheduling.rst b/Documentation/admin-guide/hw-vuln/core-scheduling.rst new file mode 100644 index 000000000000..0febe458597c --- /dev/null +++ b/Documentation/admin-guide/hw-vuln/core-scheduling.rst @@ -0,0 +1,225 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============== +Core Scheduling +=============== +Core scheduling support allows userspace to define groups of tasks that can +share a core. These groups can be specified either for security usecases (one +group of tasks don't trust another), or for performance usecases (some +workloads may benefit from running on the same core as they don't need the same +hardware resources of the shared core, or may prefer different cores if they +do share hardware resource needs). This document only describes the security +usecase. + +Security usecase +---------------- +A cross-HT attack involves the attacker and victim running on different Hyper +Threads of the same core. MDS and L1TF are examples of such attacks. The only +full mitigation of cross-HT attacks is to disable Hyper Threading (HT). Core +scheduling is a scheduler feature that can mitigate some (not all) cross-HT +attacks. It allows HT to be turned on safely by ensuring that only tasks in a +user-designated trusted group can share a core. This increase in core sharing +can also improve performance, however it is not guaranteed that performance +will always improve, though that is seen to be the case with a number of real +world workloads. In theory, core scheduling aims to perform at least as good as +when Hyper Threading is disabled. In practice, this is mostly the case though +not always: as synchronizing scheduling decisions across 2 or more CPUs in a +core involves additional overhead - especially when the system is lightly +loaded. When ``total_threads <= N_CPUS/2``, the extra overhead may cause core +scheduling to perform more poorly compared to SMT-disabled, where N_CPUS is the +total number of CPUs. Please measure the performance of your workloads always. + +Usage +----- +Core scheduling support is enabled via the ``CONFIG_SCHED_CORE`` config option. +Using this feature, userspace defines groups of tasks that can be co-scheduled +on the same core. The core scheduler uses this information to make sure that +tasks that are not in the same group never run simultaneously on a core, while +doing its best to satisfy the system's scheduling requirements. + +Core scheduling can be enabled via the ``PR_SCHED_CORE`` prctl interface. +This interface provides support for the creation of core scheduling groups, as +well as admission and removal of tasks from created groups:: + + #include <sys/prctl.h> + + int prctl(int option, unsigned long arg2, unsigned long arg3, + unsigned long arg4, unsigned long arg5); + +option: + ``PR_SCHED_CORE`` + +arg2: + Command for operation, must be one off: + + - ``PR_SCHED_CORE_GET`` -- get core_sched cookie of ``pid``. + - ``PR_SCHED_CORE_CREATE`` -- create a new unique cookie for ``pid``. + - ``PR_SCHED_CORE_SHARE_TO`` -- push core_sched cookie to ``pid``. + - ``PR_SCHED_CORE_SHARE_FROM`` -- pull core_sched cookie from ``pid``. + +arg3: + ``pid`` of the task for which the operation applies. + +arg4: + ``pid_type`` for which the operation applies. It is of type ``enum pid_type``. + For example, if arg4 is ``PIDTYPE_TGID``, then the operation of this command + will be performed for all tasks in the task group of ``pid``. + +arg5: + userspace pointer to an unsigned long for storing the cookie returned by + ``PR_SCHED_CORE_GET`` command. Should be 0 for all other commands. + +In order for a process to push a cookie to, or pull a cookie from a process, it +is required to have the ptrace access mode: `PTRACE_MODE_READ_REALCREDS` to the +process. + +Building hierarchies of tasks +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The simplest way to build hierarchies of threads/processes which share a +cookie and thus a core is to rely on the fact that the core-sched cookie is +inherited across forks/clones and execs, thus setting a cookie for the +'initial' script/executable/daemon will place every spawned child in the +same core-sched group. + +Cookie Transferral +~~~~~~~~~~~~~~~~~~ +Transferring a cookie between the current and other tasks is possible using +PR_SCHED_CORE_SHARE_FROM and PR_SCHED_CORE_SHARE_TO to inherit a cookie from a +specified task or a share a cookie with a task. In combination this allows a +simple helper program to pull a cookie from a task in an existing core +scheduling group and share it with already running tasks. + +Design/Implementation +--------------------- +Each task that is tagged is assigned a cookie internally in the kernel. As +mentioned in `Usage`_, tasks with the same cookie value are assumed to trust +each other and share a core. + +The basic idea is that, every schedule event tries to select tasks for all the +siblings of a core such that all the selected tasks running on a core are +trusted (same cookie) at any point in time. Kernel threads are assumed trusted. +The idle task is considered special, as it trusts everything and everything +trusts it. + +During a schedule() event on any sibling of a core, the highest priority task on +the sibling's core is picked and assigned to the sibling calling schedule(), if +the sibling has the task enqueued. For rest of the siblings in the core, +highest priority task with the same cookie is selected if there is one runnable +in their individual run queues. If a task with same cookie is not available, +the idle task is selected. Idle task is globally trusted. + +Once a task has been selected for all the siblings in the core, an IPI is sent to +siblings for whom a new task was selected. Siblings on receiving the IPI will +switch to the new task immediately. If an idle task is selected for a sibling, +then the sibling is considered to be in a `forced idle` state. I.e., it may +have tasks on its on runqueue to run, however it will still have to run idle. +More on this in the next section. + +Forced-idling of hyperthreads +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The scheduler tries its best to find tasks that trust each other such that all +tasks selected to be scheduled are of the highest priority in a core. However, +it is possible that some runqueues had tasks that were incompatible with the +highest priority ones in the core. Favoring security over fairness, one or more +siblings could be forced to select a lower priority task if the highest +priority task is not trusted with respect to the core wide highest priority +task. If a sibling does not have a trusted task to run, it will be forced idle +by the scheduler (idle thread is scheduled to run). + +When the highest priority task is selected to run, a reschedule-IPI is sent to +the sibling to force it into idle. This results in 4 cases which need to be +considered depending on whether a VM or a regular usermode process was running +on either HT:: + + HT1 (attack) HT2 (victim) + A idle -> user space user space -> idle + B idle -> user space guest -> idle + C idle -> guest user space -> idle + D idle -> guest guest -> idle + +Note that for better performance, we do not wait for the destination CPU +(victim) to enter idle mode. This is because the sending of the IPI would bring +the destination CPU immediately into kernel mode from user space, or VMEXIT +in the case of guests. At best, this would only leak some scheduler metadata +which may not be worth protecting. It is also possible that the IPI is received +too late on some architectures, but this has not been observed in the case of +x86. + +Trust model +~~~~~~~~~~~ +Core scheduling maintains trust relationships amongst groups of tasks by +assigning them a tag that is the same cookie value. +When a system with core scheduling boots, all tasks are considered to trust +each other. This is because the core scheduler does not have information about +trust relationships until userspace uses the above mentioned interfaces, to +communicate them. In other words, all tasks have a default cookie value of 0. +and are considered system-wide trusted. The forced-idling of siblings running +cookie-0 tasks is also avoided. + +Once userspace uses the above mentioned interfaces to group sets of tasks, tasks +within such groups are considered to trust each other, but do not trust those +outside. Tasks outside the group also don't trust tasks within. + +Limitations of core-scheduling +------------------------------ +Core scheduling tries to guarantee that only trusted tasks run concurrently on a +core. But there could be small window of time during which untrusted tasks run +concurrently or kernel could be running concurrently with a task not trusted by +kernel. + +IPI processing delays +~~~~~~~~~~~~~~~~~~~~~ +Core scheduling selects only trusted tasks to run together. IPI is used to notify +the siblings to switch to the new task. But there could be hardware delays in +receiving of the IPI on some arch (on x86, this has not been observed). This may +cause an attacker task to start running on a CPU before its siblings receive the +IPI. Even though cache is flushed on entry to user mode, victim tasks on siblings +may populate data in the cache and micro architectural buffers after the attacker +starts to run and this is a possibility for data leak. + +Open cross-HT issues that core scheduling does not solve +-------------------------------------------------------- +1. For MDS +~~~~~~~~~~ +Core scheduling cannot protect against MDS attacks between the siblings +running in user mode and the others running in kernel mode. Even though all +siblings run tasks which trust each other, when the kernel is executing +code on behalf of a task, it cannot trust the code running in the +sibling. Such attacks are possible for any combination of sibling CPU modes +(host or guest mode). + +2. For L1TF +~~~~~~~~~~~ +Core scheduling cannot protect against an L1TF guest attacker exploiting a +guest or host victim. This is because the guest attacker can craft invalid +PTEs which are not inverted due to a vulnerable guest kernel. The only +solution is to disable EPT (Extended Page Tables). + +For both MDS and L1TF, if the guest vCPU is configured to not trust each +other (by tagging separately), then the guest to guest attacks would go away. +Or it could be a system admin policy which considers guest to guest attacks as +a guest problem. + +Another approach to resolve these would be to make every untrusted task on the +system to not trust every other untrusted task. While this could reduce +parallelism of the untrusted tasks, it would still solve the above issues while +allowing system processes (trusted tasks) to share a core. + +3. Protecting the kernel (IRQ, syscall, VMEXIT) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Unfortunately, core scheduling does not protect kernel contexts running on +sibling hyperthreads from one another. Prototypes of mitigations have been posted +to LKML to solve this, but it is debatable whether such windows are practically +exploitable, and whether the performance overhead of the prototypes are worth +it (not to mention, the added code complexity). + +Other Use cases +--------------- +The main use case for Core scheduling is mitigating the cross-HT vulnerabilities +with SMT enabled. There are other use cases where this feature could be used: + +- Isolating tasks that needs a whole core: Examples include realtime tasks, tasks + that uses SIMD instructions etc. +- Gang scheduling: Requirements for a group of tasks that needs to be scheduled + together could also be realized using core scheduling. One example is vCPUs of + a VM. diff --git a/Documentation/admin-guide/hw-vuln/index.rst b/Documentation/admin-guide/hw-vuln/index.rst index ca4dbdd9016d..8cbc711cda93 100644 --- a/Documentation/admin-guide/hw-vuln/index.rst +++ b/Documentation/admin-guide/hw-vuln/index.rst @@ -15,3 +15,5 @@ are configurable at compile, boot or run time. tsx_async_abort multihit.rst special-register-buffer-data-sampling.rst + core-scheduling.rst + l1d_flush.rst diff --git a/Documentation/admin-guide/hw-vuln/l1d_flush.rst b/Documentation/admin-guide/hw-vuln/l1d_flush.rst new file mode 100644 index 000000000000..210020bc3f56 --- /dev/null +++ b/Documentation/admin-guide/hw-vuln/l1d_flush.rst @@ -0,0 +1,69 @@ +L1D Flushing +============ + +With an increasing number of vulnerabilities being reported around data +leaks from the Level 1 Data cache (L1D) the kernel provides an opt-in +mechanism to flush the L1D cache on context switch. + +This mechanism can be used to address e.g. CVE-2020-0550. For applications +the mechanism keeps them safe from vulnerabilities, related to leaks +(snooping of) from the L1D cache. + + +Related CVEs +------------ +The following CVEs can be addressed by this +mechanism + + ============= ======================== ================== + CVE-2020-0550 Improper Data Forwarding OS related aspects + ============= ======================== ================== + +Usage Guidelines +---------------- + +Please see document: :ref:`Documentation/userspace-api/spec_ctrl.rst +<set_spec_ctrl>` for details. + +**NOTE**: The feature is disabled by default, applications need to +specifically opt into the feature to enable it. + +Mitigation +---------- + +When PR_SET_L1D_FLUSH is enabled for a task a flush of the L1D cache is +performed when the task is scheduled out and the incoming task belongs to a +different process and therefore to a different address space. + +If the underlying CPU supports L1D flushing in hardware, the hardware +mechanism is used, software fallback for the mitigation, is not supported. + +Mitigation control on the kernel command line +--------------------------------------------- + +The kernel command line allows to control the L1D flush mitigations at boot +time with the option "l1d_flush=". The valid arguments for this option are: + + ============ ============================================================= + on Enables the prctl interface, applications trying to use + the prctl() will fail with an error if l1d_flush is not + enabled + ============ ============================================================= + +By default the mechanism is disabled. + +Limitations +----------- + +The mechanism does not mitigate L1D data leaks between tasks belonging to +different processes which are concurrently executing on sibling threads of +a physical CPU core when SMT is enabled on the system. + +This can be addressed by controlled placement of processes on physical CPU +cores or by disabling SMT. See the relevant chapter in the L1TF mitigation +document: :ref:`Documentation/admin-guide/hw-vuln/l1tf.rst <smt_control>`. + +**NOTE** : The opt-in of a task for L1D flushing works only when the task's +affinity is limited to cores running in non-SMT mode. If a task which +requested L1D flushing is scheduled on a SMT-enabled core the kernel sends +a SIGBUS to the task. diff --git a/Documentation/admin-guide/hw-vuln/special-register-buffer-data-sampling.rst b/Documentation/admin-guide/hw-vuln/special-register-buffer-data-sampling.rst index 3b1ce68d2456..966c9b3296ea 100644 --- a/Documentation/admin-guide/hw-vuln/special-register-buffer-data-sampling.rst +++ b/Documentation/admin-guide/hw-vuln/special-register-buffer-data-sampling.rst @@ -3,7 +3,8 @@ SRBDS - Special Register Buffer Data Sampling ============================================= -SRBDS is a hardware vulnerability that allows MDS :doc:`mds` techniques to +SRBDS is a hardware vulnerability that allows MDS +Documentation/admin-guide/hw-vuln/mds.rst techniques to infer values returned from special register accesses. Special register accesses are accesses to off core registers. According to Intel's evaluation, the special register reads that have a security expectation of privacy are diff --git a/Documentation/admin-guide/kdump/kdump.rst b/Documentation/admin-guide/kdump/kdump.rst index 75a9dd98e76e..cb30ca3df27c 100644 --- a/Documentation/admin-guide/kdump/kdump.rst +++ b/Documentation/admin-guide/kdump/kdump.rst @@ -2,7 +2,7 @@ Documentation for Kdump - The kexec-based Crash Dumping Solution ================================================================ -This document includes overview, setup and installation, and analysis +This document includes overview, setup, installation, and analysis information. Overview @@ -13,9 +13,9 @@ dump of the system kernel's memory needs to be taken (for example, when the system panics). The system kernel's memory image is preserved across the reboot and is accessible to the dump-capture kernel. -You can use common commands, such as cp and scp, to copy the -memory image to a dump file on the local disk, or across the network to -a remote system. +You can use common commands, such as cp, scp or makedumpfile to copy +the memory image to a dump file on the local disk, or across the network +to a remote system. Kdump and kexec are currently supported on the x86, x86_64, ppc64, ia64, s390x, arm and arm64 architectures. @@ -26,13 +26,15 @@ the dump-capture kernel. This ensures that ongoing Direct Memory Access The kexec -p command loads the dump-capture kernel into this reserved memory. -On x86 machines, the first 640 KB of physical memory is needed to boot, -regardless of where the kernel loads. Therefore, kexec backs up this -region just before rebooting into the dump-capture kernel. +On x86 machines, the first 640 KB of physical memory is needed for boot, +regardless of where the kernel loads. For simpler handling, the whole +low 1M is reserved to avoid any later kernel or device driver writing +data into this area. Like this, the low 1M can be reused as system RAM +by kdump kernel without extra handling. -Similarly on PPC64 machines first 32KB of physical memory is needed for -booting regardless of where the kernel is loaded and to support 64K page -size kexec backs up the first 64KB memory. +On PPC64 machines first 32KB of physical memory is needed for booting +regardless of where the kernel is loaded and to support 64K page size +kexec backs up the first 64KB memory. For s390x, when kdump is triggered, the crashkernel region is exchanged with the region [0, crashkernel region size] and then the kdump kernel @@ -46,14 +48,14 @@ passed to the dump-capture kernel through the elfcorehdr= boot parameter. Optionally the size of the ELF header can also be passed when using the elfcorehdr=[size[KMG]@]offset[KMG] syntax. - With the dump-capture kernel, you can access the memory image through /proc/vmcore. This exports the dump as an ELF-format file that you can -write out using file copy commands such as cp or scp. Further, you can -use analysis tools such as the GNU Debugger (GDB) and the Crash tool to -debug the dump file. This method ensures that the dump pages are correctly -ordered. - +write out using file copy commands such as cp or scp. You can also use +makedumpfile utility to analyze and write out filtered contents with +options, e.g with '-d 31' it will only write out kernel data. Further, +you can use analysis tools such as the GNU Debugger (GDB) and the Crash +tool to debug the dump file. This method ensures that the dump pages are +correctly ordered. Setup and Installation ====================== @@ -125,9 +127,18 @@ dump-capture kernels for enabling kdump support. System kernel config options ---------------------------- -1) Enable "kexec system call" in "Processor type and features.":: +1) Enable "kexec system call" or "kexec file based system call" in + "Processor type and features.":: + + CONFIG_KEXEC=y or CONFIG_KEXEC_FILE=y + + And both of them will select KEXEC_CORE:: - CONFIG_KEXEC=y + CONFIG_KEXEC_CORE=y + + Subsequently, CRASH_CORE is selected by KEXEC_CORE:: + + CONFIG_CRASH_CORE=y 2) Enable "sysfs file system support" in "Filesystem" -> "Pseudo filesystems." This is usually enabled by default:: @@ -175,17 +186,19 @@ Dump-capture kernel config options (Arch Dependent, i386 and x86_64) CONFIG_HIGHMEM4G -2) On i386 and x86_64, disable symmetric multi-processing support - under "Processor type and features":: +2) With CONFIG_SMP=y, usually nr_cpus=1 need specified on the kernel + command line when loading the dump-capture kernel because one + CPU is enough for kdump kernel to dump vmcore on most of systems. - CONFIG_SMP=n + However, you can also specify nr_cpus=X to enable multiple processors + in kdump kernel. In this case, "disable_cpu_apicid=" is needed to + tell kdump kernel which cpu is 1st kernel's BSP. Please refer to + admin-guide/kernel-parameters.txt for more details. - (If CONFIG_SMP=y, then specify maxcpus=1 on the kernel command line - when loading the dump-capture kernel, see section "Load the Dump-capture - Kernel".) + With CONFIG_SMP=n, the above things are not related. -3) If one wants to build and use a relocatable kernel, - Enable "Build a relocatable kernel" support under "Processor type and +3) A relocatable kernel is suggested to be built by default. If not yet, + enable "Build a relocatable kernel" support under "Processor type and features":: CONFIG_RELOCATABLE=y @@ -232,7 +245,7 @@ Dump-capture kernel config options (Arch Dependent, ia64) as a dump-capture kernel if desired. The crashkernel region can be automatically placed by the system - kernel at run time. This is done by specifying the base address as 0, + kernel at runtime. This is done by specifying the base address as 0, or omitting it all together:: crashkernel=256M@0 @@ -241,10 +254,6 @@ Dump-capture kernel config options (Arch Dependent, ia64) crashkernel=256M - If the start address is specified, note that the start address of the - kernel will be aligned to 64Mb, so if the start address is not then - any space below the alignment point will be wasted. - Dump-capture kernel config options (Arch Dependent, arm) ---------------------------------------------------------- @@ -260,46 +269,82 @@ Dump-capture kernel config options (Arch Dependent, arm64) on non-VHE systems even if it is configured. This is because the CPU will not be reset to EL2 on panic. -Extended crashkernel syntax +crashkernel syntax =========================== +1) crashkernel=size@offset -While the "crashkernel=size[@offset]" syntax is sufficient for most -configurations, sometimes it's handy to have the reserved memory dependent -on the value of System RAM -- that's mostly for distributors that pre-setup -the kernel command line to avoid a unbootable system after some memory has -been removed from the machine. + Here 'size' specifies how much memory to reserve for the dump-capture kernel + and 'offset' specifies the beginning of this reserved memory. For example, + "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory + starting at physical address 0x01000000 (16MB) for the dump-capture kernel. -The syntax is:: + The crashkernel region can be automatically placed by the system + kernel at run time. This is done by specifying the base address as 0, + or omitting it all together:: - crashkernel=<range1>:<size1>[,<range2>:<size2>,...][@offset] - range=start-[end] + crashkernel=256M@0 -For example:: + or:: - crashkernel=512M-2G:64M,2G-:128M + crashkernel=256M -This would mean: + If the start address is specified, note that the start address of the + kernel will be aligned to a value (which is Arch dependent), so if the + start address is not then any space below the alignment point will be + wasted. - 1) if the RAM is smaller than 512M, then don't reserve anything - (this is the "rescue" case) - 2) if the RAM size is between 512M and 2G (exclusive), then reserve 64M - 3) if the RAM size is larger than 2G, then reserve 128M +2) range1:size1[,range2:size2,...][@offset] + While the "crashkernel=size[@offset]" syntax is sufficient for most + configurations, sometimes it's handy to have the reserved memory dependent + on the value of System RAM -- that's mostly for distributors that pre-setup + the kernel command line to avoid a unbootable system after some memory has + been removed from the machine. + The syntax is:: -Boot into System Kernel -======================= + crashkernel=<range1>:<size1>[,<range2>:<size2>,...][@offset] + range=start-[end] + + For example:: + + crashkernel=512M-2G:64M,2G-:128M + This would mean: + + 1) if the RAM is smaller than 512M, then don't reserve anything + (this is the "rescue" case) + 2) if the RAM size is between 512M and 2G (exclusive), then reserve 64M + 3) if the RAM size is larger than 2G, then reserve 128M + +3) crashkernel=size,high and crashkernel=size,low + + If memory above 4G is preferred, crashkernel=size,high can be used to + fulfill that. With it, physical memory is allowed to be allocated from top, + so could be above 4G if system has more than 4G RAM installed. Otherwise, + memory region will be allocated below 4G if available. + + When crashkernel=X,high is passed, kernel could allocate physical memory + region above 4G, low memory under 4G is needed in this case. There are + three ways to get low memory: + + 1) Kernel will allocate at least 256M memory below 4G automatically + if crashkernel=Y,low is not specified. + 2) Let user specify low memory size instead. + 3) Specified value 0 will disable low memory allocation:: + + crashkernel=0,low + +Boot into System Kernel +----------------------- 1) Update the boot loader (such as grub, yaboot, or lilo) configuration files as necessary. -2) Boot the system kernel with the boot parameter "crashkernel=Y@X", - where Y specifies how much memory to reserve for the dump-capture kernel - and X specifies the beginning of this reserved memory. For example, - "crashkernel=64M@16M" tells the system kernel to reserve 64 MB of memory - starting at physical address 0x01000000 (16MB) for the dump-capture kernel. +2) Boot the system kernel with the boot parameter "crashkernel=Y@X". - On x86 and x86_64, use "crashkernel=64M@16M". + On x86 and x86_64, use "crashkernel=Y[@X]". Most of the time, the + start address 'X' is not necessary, kernel will search a suitable + area. Unless an explicit start address is expected. On ppc64, use "crashkernel=128M@32M". @@ -331,8 +376,8 @@ of dump-capture kernel. Following is the summary. For i386 and x86_64: - - Use vmlinux if kernel is not relocatable. - Use bzImage/vmlinuz if kernel is relocatable. + - Use vmlinux if kernel is not relocatable. For ppc64: @@ -392,7 +437,7 @@ loading dump-capture kernel. For i386, x86_64 and ia64: - "1 irqpoll maxcpus=1 reset_devices" + "1 irqpoll nr_cpus=1 reset_devices" For ppc64: @@ -400,7 +445,7 @@ For ppc64: For s390x: - "1 maxcpus=1 cgroup_disable=memory" + "1 nr_cpus=1 cgroup_disable=memory" For arm: @@ -408,7 +453,7 @@ For arm: For arm64: - "1 maxcpus=1 reset_devices" + "1 nr_cpus=1 reset_devices" Notes on loading the dump-capture kernel: @@ -488,6 +533,10 @@ the following command:: cp /proc/vmcore <dump-file> +You can also use makedumpfile utility to write out the dump file +with specified options to filter out unwanted contents, e.g:: + + makedumpfile -l --message-level 1 -d 31 /proc/vmcore <dump-file> Analysis ======== @@ -535,8 +584,7 @@ This will cause a kdump to occur at the add_taint()->panic() call. Contact ======= -- Vivek Goyal (vgoyal@redhat.com) -- Maneesh Soni (maneesh@in.ibm.com) +- kexec@lists.infradead.org GDB macros ========== diff --git a/Documentation/admin-guide/kernel-parameters.rst b/Documentation/admin-guide/kernel-parameters.rst index 3996b54158bf..01ba293a2d70 100644 --- a/Documentation/admin-guide/kernel-parameters.rst +++ b/Documentation/admin-guide/kernel-parameters.rst @@ -76,6 +76,11 @@ to change, such as less cores in the CPU list, then N and any ranges using N will also change. Use the same on a small 4 core system, and "16-N" becomes "16-3" and now the same boot input will be flagged as invalid (start > end). +The special case-tolerant group name "all" has a meaning of selecting all CPUs, +so that "nohz_full=all" is the equivalent of "nohz_full=0-N". + +The semantics of "N" and "all" is supported on a level of bitmaps and holds for +all users of bitmap_parse(). This document may not be entirely up to date and comprehensive. The command "modinfo -p ${modulename}" shows a current list of all parameters of a loadable diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index cb89dbdedc46..91ba391f9b32 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -113,7 +113,7 @@ the GPE dispatcher. This facility can be used to prevent such uncontrolled GPE floodings. - Format: <byte> + Format: <byte> or <bitmap-list> acpi_no_auto_serialize [HW,ACPI] Disable auto-serialization of AML methods @@ -287,13 +287,21 @@ do not want to use tracing_snapshot_alloc() as it needs to be done where GFP_KERNEL allocations are allowed. + allow_mismatched_32bit_el0 [ARM64] + Allow execve() of 32-bit applications and setting of the + PER_LINUX32 personality on systems where only a strict + subset of the CPUs support 32-bit EL0. When this + parameter is present, the set of CPUs supporting 32-bit + EL0 is indicated by /sys/devices/system/cpu/aarch32_el0 + and hot-unplug operations may be restricted. + + See Documentation/arm64/asymmetric-32bit.rst for more + information. + amd_iommu= [HW,X86-64] Pass parameters to the AMD IOMMU driver in the system. Possible values are: - fullflush - enable flushing of IO/TLB entries when - they are unmapped. Otherwise they are - flushed before they will be reused, which - is a lot of faster + fullflush - Deprecated, equivalent to iommu.strict=1 off - do not initialize any AMD IOMMU found in the system force_isolation - Force device isolation for all @@ -301,6 +309,9 @@ allowed anymore to lift isolation requirements as needed. This option does not override iommu=pt + force_enable - Force enable the IOMMU on platforms known + to be buggy with IOMMU enabled. Use this + option with care. amd_iommu_dump= [HW,X86-64] Enable AMD IOMMU driver option to dump the ACPI table @@ -377,6 +388,9 @@ arm64.nopauth [ARM64] Unconditionally disable Pointer Authentication support + arm64.nomte [ARM64] Unconditionally disable Memory Tagging Extension + support + ataflop= [HW,M68k] atarimouse= [HW,MOUSE] Atari Mouse @@ -497,16 +511,21 @@ ccw_timeout_log [S390] See Documentation/s390/common_io.rst for details. - cgroup_disable= [KNL] Disable a particular controller - Format: {name of the controller(s) to disable} + cgroup_disable= [KNL] Disable a particular controller or optional feature + Format: {name of the controller(s) or feature(s) to disable} The effects of cgroup_disable=foo are: - foo isn't auto-mounted if you mount all cgroups in a single hierarchy - foo isn't visible as an individually mountable subsystem + - if foo is an optional feature then the feature is + disabled and corresponding cgroup files are not + created {Currently only "memory" controller deal with this and cut the overhead, others just disable the usage. So only cgroup_disable=memory is actually worthy} + Specifying "pressure" disables per-cgroup pressure + stall information accounting feature cgroup_no_v1= [KNL] Disable cgroup controllers and named hierarchies in v1 Format: { { controller | "all" | "named" } @@ -581,6 +600,28 @@ loops can be debugged more effectively on production systems. + clocksource.max_cswd_read_retries= [KNL] + Number of clocksource_watchdog() retries due to + external delays before the clock will be marked + unstable. Defaults to three retries, that is, + four attempts to read the clock under test. + + clocksource.verify_n_cpus= [KNL] + Limit the number of CPUs checked for clocksources + marked with CLOCK_SOURCE_VERIFY_PERCPU that + are marked unstable due to excessive skew. + A negative value says to check all CPUs, while + zero says not to check any. Values larger than + nr_cpu_ids are silently truncated to nr_cpu_ids. + The actual CPUs are chosen randomly, with + no replacement if the same CPU is chosen twice. + + clocksource-wdtest.holdoff= [KNL] + Set the time in seconds that the clocksource + watchdog test waits before commencing its tests. + Defaults to zero when built as a module and to + 10 seconds when built into the kernel. + clearcpuid=BITNUM[,BITNUM...] [X86] Disable CPUID feature X for the kernel. See arch/x86/include/asm/cpufeatures.h for the valid bit @@ -1092,6 +1133,11 @@ the driver will use only 32-bit accessors to read/write the device registers. + liteuart,<addr> + Start an early console on a litex serial port at the + specified address. The serial port must already be + setup and configured. Options are not yet supported. + meson,<addr> Start an early, polled-mode console on a meson serial port at the specified address. The serial port must @@ -1567,6 +1613,23 @@ Documentation/admin-guide/mm/hugetlbpage.rst. Format: size[KMG] + hugetlb_free_vmemmap= + [KNL] Reguires CONFIG_HUGETLB_PAGE_FREE_VMEMMAP + enabled. + Allows heavy hugetlb users to free up some more + memory (6 * PAGE_SIZE for each 2MB hugetlb page). + Format: { on | off (default) } + + on: enable the feature + off: disable the feature + + Built with CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON=y, + the default is on. + + This is not compatible with memory_hotplug.memmap_on_memory. + If both parameters are enabled, hugetlb_free_vmemmap takes + precedence over memory_hotplug.memmap_on_memory. + hung_task_panic= [KNL] Should the hung task detector generate panics. Format: 0 | 1 @@ -1695,6 +1758,11 @@ support for the idxd driver. By default it is set to true (1). + idxd.tc_override= [HW] + Format: <bool> + Allow override of default traffic class configuration + for the device. By default it is set to false (0). + ieee754= [MIPS] Select IEEE Std 754 conformance mode Format: { strict | legacy | 2008 | relaxed } Default: strict @@ -1892,18 +1960,17 @@ this case, gfx device will use physical address for DMA. strict [Default Off] - With this option on every unmap_single operation will - result in a hardware IOTLB flush operation as opposed - to batching them for performance. + Deprecated, equivalent to iommu.strict=1. sp_off [Default Off] By default, super page will be supported if Intel IOMMU has the capability. With this option, super page will not be supported. - sm_on [Default Off] - By default, scalable mode will be disabled even if the - hardware advertises that it has support for the scalable - mode translation. With this option set, scalable mode - will be used on hardware which claims to support it. + sm_on + Enable the Intel IOMMU scalable mode if the hardware + advertises that it has support for the scalable mode + translation. + sm_off + Disallow use of the Intel IOMMU scalable mode. tboot_noforce [Default Off] Do not force the Intel IOMMU enabled under tboot. By default, tboot will force Intel IOMMU on, which @@ -1987,7 +2054,7 @@ forcing Dual Address Cycle for PCI cards supporting greater than 32-bit addressing. - iommu.strict= [ARM64] Configure TLB invalidation behaviour + iommu.strict= [ARM64, X86] Configure TLB invalidation behaviour Format: { "0" | "1" } 0 - Lazy mode. Request that DMA unmap operations use deferred @@ -1995,9 +2062,12 @@ throughput at the cost of reduced device isolation. Will fall back to strict mode if not supported by the relevant IOMMU driver. - 1 - Strict mode (default). + 1 - Strict mode. DMA unmap operations invalidate IOMMU hardware TLBs synchronously. + unset - Use value of CONFIG_IOMMU_DEFAULT_DMA_{LAZY,STRICT}. + Note: on x86, strict mode specified via one of the + legacy driver-specific options takes precedence. iommu.passthrough= [ARM64, X86] Configure DMA to bypass the IOMMU by default. @@ -2365,6 +2435,23 @@ feature (tagged TLBs) on capable Intel chips. Default is 1 (enabled) + l1d_flush= [X86,INTEL] + Control mitigation for L1D based snooping vulnerability. + + Certain CPUs are vulnerable to an exploit against CPU + internal buffers which can forward information to a + disclosure gadget under certain conditions. + + In vulnerable processors, the speculatively + forwarded data can be used in a cache side channel + attack, to access data to which the attacker does + not have direct access. + + This parameter controls the mitigation. The + options are: + + on - enable the interface for the mitigation + l1tf= [X86] Control mitigation of the L1TF vulnerability on affected CPUs @@ -2833,6 +2920,10 @@ Note that even when enabled, there are a few cases where the feature is not effective. + This is not compatible with hugetlb_free_vmemmap. If + both parameters are enabled, hugetlb_free_vmemmap takes + precedence over memory_hotplug.memmap_on_memory. + memtest= [KNL,X86,ARM,PPC,RISCV] Enable memtest Format: <integer> default : 0 <disable> @@ -3244,7 +3335,7 @@ noclflush [BUGS=X86] Don't use the CLFLUSH instruction - nodelayacct [KNL] Disable per-task delay accounting + delayacct [KNL] Enable per-task delay accounting nodsp [SH] Disable hardware DSP at boot time. @@ -3513,6 +3604,9 @@ nr_uarts= [SERIAL] maximum number of UARTs to be registered. + numa=off [KNL, ARM64, PPC, RISCV, SPARC, X86] Disable NUMA, Only + set up a single NUMA node spanning all memory. + numa_balancing= [KNL,ARM64,PPC,RISCV,S390,X86] Enable or disable automatic NUMA balancing. Allowed values are enable and disable @@ -3566,6 +3660,12 @@ off: turn off poisoning (default) on: turn on poisoning + page_reporting.page_reporting_order= + [KNL] Minimal page reporting order + Format: <integer> + Adjust the minimal page reporting order. The page + reporting is disabled when it exceeds (MAX_ORDER-1). + panic= [KNL] Kernel behaviour on panic: delay <timeout> timeout > 0: seconds before rebooting timeout = 0: wait forever @@ -4098,6 +4198,15 @@ Format: <bool> (1/Y/y=enable, 0/N/n=disable) default: disabled + printk.console_no_auto_verbose= + Disable console loglevel raise on oops, panic + or lockdep-detected issues (only if lock debug is on). + With an exception to setups with low baudrate on + serial console, keeping this 0 is a good choice + in order to provide more debug information. + Format: <bool> + default: 0 (auto_verbose is enabled) + printk.devkmsg={on,off,ratelimit} Control writing to /dev/kmsg. on - unlimited logging to /dev/kmsg from userspace @@ -4290,6 +4399,11 @@ whole algorithm to behave better in low memory condition. + rcutree.rcu_delay_page_cache_fill_msec= [KNL] + Set the page-cache refill delay (in milliseconds) + in response to low-memory conditions. The range + of permitted values is in the range 0:100000. + rcutree.jiffies_till_first_fqs= [KNL] Set delay from grace-period initialization to first attempt to force quiescent states. @@ -4703,7 +4817,7 @@ reboot= [KNL] Format (x86 or x86_64): - [w[arm] | c[old] | h[ard] | s[oft] | g[pio]] \ + [w[arm] | c[old] | h[ard] | s[oft] | g[pio]] | d[efault] \ [[,]s[mp]#### \ [[,]b[ios] | a[cpi] | k[bd] | t[riple] | e[fi] | p[ci]] \ [[,]f[orce] @@ -4775,11 +4889,6 @@ Reserves a hole at the top of the kernel virtual address space. - reservelow= [X86] - Format: nn[K] - Set the amount of memory to reserve for BIOS at - the bottom of the address space. - reset_devices [KNL] Force drivers to reset the underlying device during initialization. @@ -4876,8 +4985,6 @@ sa1100ir [NET] See drivers/net/irda/sa1100_ir.c. - sbni= [NET] Granch SBNI12 leased line adapter - sched_verbose [KNL] Enables verbose scheduler debug messages. schedstats= [KNL,X86] Enable or disable scheduled statistics. @@ -5283,6 +5390,14 @@ exception. Default behavior is by #AC if both features are enabled in hardware. + ratelimit:N - + Set system wide rate limit to N bus locks + per second for bus lock detection. + 0 < N <= 1000. + + N/A for split lock detection. + + If an #AC exception is hit in the kernel or in firmware (i.e. not while executing in user mode) the kernel will oops in either "warn" or "fatal" @@ -5605,12 +5720,25 @@ Note, echoing 1 into this file without the tracepoint_printk kernel cmdline option has no effect. + The tp_printk_stop_on_boot (see below) can also be used + to stop the printing of events to console at + late_initcall_sync. + ** CAUTION ** Having tracepoints sent to printk() and activating high frequency tracepoints such as irq or sched, can cause the system to live lock. + tp_printk_stop_on_boot[FTRACE] + When tp_printk (above) is set, it can cause a lot of noise + on the console. It may be useful to only include the + printing of events during boot up, as user space may + make the system inoperable. + + This command line option will stop the printing of events + to console at the late_initcall_sync() time frame. + traceoff_on_warning [FTRACE] enable this option to disable tracing when a warning is hit. This turns off "tracing_on". Tracing can diff --git a/Documentation/admin-guide/laptops/laptop-mode.rst b/Documentation/admin-guide/laptops/laptop-mode.rst index c984c4262f2e..b61cc601d298 100644 --- a/Documentation/admin-guide/laptops/laptop-mode.rst +++ b/Documentation/admin-guide/laptops/laptop-mode.rst @@ -101,17 +101,6 @@ this results in concentration of disk activity in a small time interval which occurs only once every 10 minutes, or whenever the disk is forced to spin up by a cache miss. The disk can then be spun down in the periods of inactivity. -If you want to find out which process caused the disk to spin up, you can -gather information by setting the flag /proc/sys/vm/block_dump. When this flag -is set, Linux reports all disk read and write operations that take place, and -all block dirtyings done to files. This makes it possible to debug why a disk -needs to spin up, and to increase battery life even more. The output of -block_dump is written to the kernel output, and it can be retrieved using -"dmesg". When you use block_dump and your kernel logging level also includes -kernel debugging messages, you probably want to turn off klogd, otherwise -the output of block_dump will be logged, causing disk activity that is not -normally there. - Configuration ------------- diff --git a/Documentation/admin-guide/laptops/lg-laptop.rst b/Documentation/admin-guide/laptops/lg-laptop.rst index ce9b14671cb9..6fbe165dcd27 100644 --- a/Documentation/admin-guide/laptops/lg-laptop.rst +++ b/Documentation/admin-guide/laptops/lg-laptop.rst @@ -13,10 +13,8 @@ Hotkeys The following FN keys are ignored by the kernel without this driver: - FN-F1 (LG control panel) - Generates F15 -- FN-F5 (Touchpad toggle) - Generates F13 +- FN-F5 (Touchpad toggle) - Generates F21 - FN-F6 (Airplane mode) - Generates RFKILL -- FN-F8 (Keyboard backlight) - Generates F16. - This key also changes keyboard backlight mode. - FN-F9 (Reader mode) - Generates F14 The rest of the FN keys work without a need for a special driver. diff --git a/Documentation/admin-guide/lockup-watchdogs.rst b/Documentation/admin-guide/lockup-watchdogs.rst index 290840c160af..3e09284a8b9b 100644 --- a/Documentation/admin-guide/lockup-watchdogs.rst +++ b/Documentation/admin-guide/lockup-watchdogs.rst @@ -39,7 +39,7 @@ in principle, they should work in any architecture where these subsystems are present. A periodic hrtimer runs to generate interrupts and kick the watchdog -task. An NMI perf event is generated every "watchdog_thresh" +job. An NMI perf event is generated every "watchdog_thresh" (compile-time initialized to 10 and configurable through sysctl of the same name) seconds to check for hardlockups. If any CPU in the system does not receive any hrtimer interrupt during that time the @@ -47,7 +47,7 @@ does not receive any hrtimer interrupt during that time the generate a kernel warning or call panic, depending on the configuration. -The watchdog task is a high priority kernel thread that updates a +The watchdog job runs in a stop scheduling thread that updates a timestamp every time it is scheduled. If that timestamp is not updated for 2*watchdog_thresh seconds (the softlockup threshold) the 'softlockup detector' (coded inside the hrtimer callback function) diff --git a/Documentation/admin-guide/media/bt8xx.rst b/Documentation/admin-guide/media/bt8xx.rst index 1382ada1e38e..3589f6ab7e46 100644 --- a/Documentation/admin-guide/media/bt8xx.rst +++ b/Documentation/admin-guide/media/bt8xx.rst @@ -15,11 +15,12 @@ Authors: General information ------------------- -This class of cards has a bt878a as the PCI interface, and require the bttv driver -for accessing the i2c bus and the gpio pins of the bt8xx chipset. +This class of cards has a bt878a as the PCI interface, and require the bttv +driver for accessing the i2c bus and the gpio pins of the bt8xx chipset. -Please see :doc:`bttv-cardlist` for a complete list of Cards based on the -Conexant Bt8xx PCI bridge supported by the Linux Kernel. +Please see Documentation/admin-guide/media/bttv-cardlist.rst for a complete +list of Cards based on the Conexant Bt8xx PCI bridge supported by the +Linux Kernel. In order to be able to compile the kernel, some config options should be enabled:: @@ -80,7 +81,7 @@ for dvb-bt8xx drivers by passing modprobe parameters may be necessary. Running TwinHan and Clones ~~~~~~~~~~~~~~~~~~~~~~~~~~ -As shown at :doc:`bttv-cardlist`, TwinHan and +As shown at Documentation/admin-guide/media/bttv-cardlist.rst, TwinHan and clones use ``card=113`` modprobe parameter. So, in order to properly detect it for devices without EEPROM, you should use:: @@ -105,12 +106,12 @@ The autodetected values are determined by the cards' "response string". In your logs see f. ex.: dst_get_device_id: Recognize [DSTMCI]. For bug reports please send in a complete log with verbose=4 activated. -Please also see :doc:`ci`. +Please also see Documentation/admin-guide/media/ci.rst. Running multiple cards ~~~~~~~~~~~~~~~~~~~~~~ -See :doc:`bttv-cardlist` for a complete list of +See Documentation/admin-guide/media/bttv-cardlist.rst for a complete list of Card ID. Some examples: =========================== === diff --git a/Documentation/admin-guide/media/bttv.rst b/Documentation/admin-guide/media/bttv.rst index 0ef1f203104d..125f6f47123d 100644 --- a/Documentation/admin-guide/media/bttv.rst +++ b/Documentation/admin-guide/media/bttv.rst @@ -24,7 +24,8 @@ If your board has digital TV, you'll also need:: ./scripts/config -m DVB_BT8XX -In this case, please see :doc:`bt8xx` for additional notes. +In this case, please see Documentation/admin-guide/media/bt8xx.rst +for additional notes. Make bttv work with your card ----------------------------- @@ -39,7 +40,7 @@ If it doesn't bttv likely could not autodetect your card and needs some insmod options. The most important insmod option for bttv is "card=n" to select the correct card type. If you get video but no sound you've very likely specified the wrong (or no) card type. A list of supported -cards is in :doc:`bttv-cardlist`. +cards is in Documentation/admin-guide/media/bttv-cardlist.rst. If bttv takes very long to load (happens sometimes with the cheap cards which have no tuner), try adding this to your modules configuration @@ -57,8 +58,8 @@ directory should be enough for it to be autoload during the driver's probing mode (e. g. when the Kernel boots or when the driver is manually loaded via ``modprobe`` command). -If your card isn't listed in :doc:`bttv-cardlist` or if you have -trouble making audio work, please read :ref:`still_doesnt_work`. +If your card isn't listed in Documentation/admin-guide/media/bttv-cardlist.rst +or if you have trouble making audio work, please read :ref:`still_doesnt_work`. Autodetecting cards @@ -77,8 +78,8 @@ the Subsystem ID in the second line, looks like this: only bt878-based cards can have a subsystem ID (which does not mean that every card really has one). bt848 cards can't have a Subsystem ID and therefore can't be autodetected. There is a list with the ID's -at :doc:`bttv-cardlist` (in case you are interested or want to mail -patches with updates). +at Documentation/admin-guide/media/bttv-cardlist.rst +(in case you are interested or want to mail patches with updates). .. _still_doesnt_work: @@ -259,15 +260,15 @@ bug. It is very helpful if you can tell where exactly it broke With a hard freeze you probably doesn't find anything in the logfiles. The only way to capture any kernel messages is to hook up a serial console and let some terminal application log the messages. /me uses -screen. See :doc:`/admin-guide/serial-console` for details on setting -up a serial console. +screen. See Documentation/admin-guide/serial-console.rst for details on +setting up a serial console. -Read :doc:`/admin-guide/bug-hunting` to learn how to get any useful +Read Documentation/admin-guide/bug-hunting.rst to learn how to get any useful information out of a register+stack dump printed by the kernel on protection faults (so-called "kernel oops"). If you run into some kind of deadlock, you can try to dump a call trace -for each process using sysrq-t (see :doc:`/admin-guide/sysrq`). +for each process using sysrq-t (see Documentation/admin-guide/sysrq.rst). This way it is possible to figure where *exactly* some process in "D" state is stuck. diff --git a/Documentation/admin-guide/media/index.rst b/Documentation/admin-guide/media/index.rst index 6e0d2bae7154..c676af665111 100644 --- a/Documentation/admin-guide/media/index.rst +++ b/Documentation/admin-guide/media/index.rst @@ -11,12 +11,14 @@ its supported drivers. Please see: -- :doc:`/userspace-api/media/index` - for the userspace APIs used on media devices. +Documentation/userspace-api/media/index.rst -- :doc:`/driver-api/media/index` - for driver development information and Kernel APIs used by - media devices; + - for the userspace APIs used on media devices. + +Documentation/driver-api/media/index.rst + + - for driver development information and Kernel APIs used by + media devices; The media subsystem =================== diff --git a/Documentation/admin-guide/media/ipu3.rst b/Documentation/admin-guide/media/ipu3.rst index f59697c7b374..52c1c04173da 100644 --- a/Documentation/admin-guide/media/ipu3.rst +++ b/Documentation/admin-guide/media/ipu3.rst @@ -234,22 +234,23 @@ The IPU3 ImgU pipelines can be configured using the Media Controller, defined at Running mode and firmware binary selection ------------------------------------------ -ImgU works based on firmware, currently the ImgU firmware support run 2 pipes in -time-sharing with single input frame data. Each pipe can run at certain mode - -"VIDEO" or "STILL", "VIDEO" mode is commonly used for video frames capture, and -"STILL" is used for still frame capture. However, you can also select "VIDEO" to -capture still frames if you want to capture images with less system load and -power. For "STILL" mode, ImgU will try to use smaller BDS factor and output -larger bayer frame for further YUV processing than "VIDEO" mode to get high -quality images. Besides, "STILL" mode need XNR3 to do noise reduction, hence -"STILL" mode will need more power and memory bandwidth than "VIDEO" mode. TNR -will be enabled in "VIDEO" mode and bypassed by "STILL" mode. ImgU is running at -“VIDEO” mode by default, the user can use v4l2 control V4L2_CID_INTEL_IPU3_MODE -(currently defined in drivers/staging/media/ipu3/include/intel-ipu3.h) to query -and set the running mode. For user, there is no difference for buffer queueing -between the "VIDEO" and "STILL" mode, mandatory input and main output node -should be enabled and buffers need be queued, the statistics and the view-finder -queues are optional. +ImgU works based on firmware, currently the ImgU firmware support run 2 pipes +in time-sharing with single input frame data. Each pipe can run at certain mode +- "VIDEO" or "STILL", "VIDEO" mode is commonly used for video frames capture, +and "STILL" is used for still frame capture. However, you can also select +"VIDEO" to capture still frames if you want to capture images with less system +load and power. For "STILL" mode, ImgU will try to use smaller BDS factor and +output larger bayer frame for further YUV processing than "VIDEO" mode to get +high quality images. Besides, "STILL" mode need XNR3 to do noise reduction, +hence "STILL" mode will need more power and memory bandwidth than "VIDEO" mode. +TNR will be enabled in "VIDEO" mode and bypassed by "STILL" mode. ImgU is +running at "VIDEO" mode by default, the user can use v4l2 control +V4L2_CID_INTEL_IPU3_MODE (currently defined in +drivers/staging/media/ipu3/include/uapi/intel-ipu3.h) to query and set the +running mode. For user, there is no difference for buffer queueing between the +"VIDEO" and "STILL" mode, mandatory input and main output node should be +enabled and buffers need be queued, the statistics and the view-finder queues +are optional. The firmware binary will be selected according to current running mode, such log "using binary if_to_osys_striped " or "using binary if_to_osys_primary_striped" @@ -586,7 +587,7 @@ preserved. References ========== -.. [#f5] drivers/staging/media/ipu3/include/intel-ipu3.h +.. [#f5] drivers/staging/media/ipu3/include/uapi/intel-ipu3.h .. [#f1] https://github.com/intel/nvt diff --git a/Documentation/admin-guide/media/saa7134.rst b/Documentation/admin-guide/media/saa7134.rst index 7ab9c70b9abe..51eae7eb5ab7 100644 --- a/Documentation/admin-guide/media/saa7134.rst +++ b/Documentation/admin-guide/media/saa7134.rst @@ -50,7 +50,8 @@ To build and install, you should run:: Once the new Kernel is booted, saa7134 driver should be loaded automatically. Depending on the card you might have to pass ``card=<nr>`` as insmod option. -If so, please check :doc:`saa7134-cardlist` for valid choices. +If so, please check Documentation/admin-guide/media/saa7134-cardlist.rst +for valid choices. Once you have your card type number, you can pass a modules configuration via a file (usually, it is either ``/etc/modules.conf`` or some file at diff --git a/Documentation/admin-guide/mm/damon/index.rst b/Documentation/admin-guide/mm/damon/index.rst new file mode 100644 index 000000000000..8c5dde3a5754 --- /dev/null +++ b/Documentation/admin-guide/mm/damon/index.rst @@ -0,0 +1,15 @@ +.. SPDX-License-Identifier: GPL-2.0 + +======================== +Monitoring Data Accesses +======================== + +:doc:`DAMON </vm/damon/index>` allows light-weight data access monitoring. +Using DAMON, users can analyze the memory access patterns of their systems and +optimize those. + +.. toctree:: + :maxdepth: 2 + + start + usage diff --git a/Documentation/admin-guide/mm/damon/start.rst b/Documentation/admin-guide/mm/damon/start.rst new file mode 100644 index 000000000000..d5eb89a8fc38 --- /dev/null +++ b/Documentation/admin-guide/mm/damon/start.rst @@ -0,0 +1,114 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============== +Getting Started +=============== + +This document briefly describes how you can use DAMON by demonstrating its +default user space tool. Please note that this document describes only a part +of its features for brevity. Please refer to :doc:`usage` for more details. + + +TL; DR +====== + +Follow the commands below to monitor and visualize the memory access pattern of +your workload. :: + + # # build the kernel with CONFIG_DAMON_*=y, install it, and reboot + # mount -t debugfs none /sys/kernel/debug/ + # git clone https://github.com/awslabs/damo + # ./damo/damo record $(pidof <your workload>) + # ./damo/damo report heat --plot_ascii + +The final command draws the access heatmap of ``<your workload>``. The heatmap +shows which memory region (x-axis) is accessed when (y-axis) and how frequently +(number; the higher the more accesses have been observed). :: + + 111111111111111111111111111111111111111111111111111111110000 + 111121111111111111111111111111211111111111111111111111110000 + 000000000000000000000000000000000000000000000000001555552000 + 000000000000000000000000000000000000000000000222223555552000 + 000000000000000000000000000000000000000011111677775000000000 + 000000000000000000000000000000000000000488888000000000000000 + 000000000000000000000000000000000177888400000000000000000000 + 000000000000000000000000000046666522222100000000000000000000 + 000000000000000000000014444344444300000000000000000000000000 + 000000000000000002222245555510000000000000000000000000000000 + # access_frequency: 0 1 2 3 4 5 6 7 8 9 + # x-axis: space (140286319947776-140286426374096: 101.496 MiB) + # y-axis: time (605442256436361-605479951866441: 37.695430s) + # resolution: 60x10 (1.692 MiB and 3.770s for each character) + + +Prerequisites +============= + +Kernel +------ + +You should first ensure your system is running on a kernel built with +``CONFIG_DAMON_*=y``. + + +User Space Tool +--------------- + +For the demonstration, we will use the default user space tool for DAMON, +called DAMON Operator (DAMO). It is available at +https://github.com/awslabs/damo. The examples below assume that ``damo`` is on +your ``$PATH``. It's not mandatory, though. + +Because DAMO is using the debugfs interface (refer to :doc:`usage` for the +detail) of DAMON, you should ensure debugfs is mounted. Mount it manually as +below:: + + # mount -t debugfs none /sys/kernel/debug/ + +or append the following line to your ``/etc/fstab`` file so that your system +can automatically mount debugfs upon booting:: + + debugfs /sys/kernel/debug debugfs defaults 0 0 + + +Recording Data Access Patterns +============================== + +The commands below record the memory access patterns of a program and save the +monitoring results to a file. :: + + $ git clone https://github.com/sjp38/masim + $ cd masim; make; ./masim ./configs/zigzag.cfg & + $ sudo damo record -o damon.data $(pidof masim) + +The first two lines of the commands download an artificial memory access +generator program and run it in the background. The generator will repeatedly +access two 100 MiB sized memory regions one by one. You can substitute this +with your real workload. The last line asks ``damo`` to record the access +pattern in the ``damon.data`` file. + + +Visualizing Recorded Patterns +============================= + +The following three commands visualize the recorded access patterns and save +the results as separate image files. :: + + $ damo report heats --heatmap access_pattern_heatmap.png + $ damo report wss --range 0 101 1 --plot wss_dist.png + $ damo report wss --range 0 101 1 --sortby time --plot wss_chron_change.png + +- ``access_pattern_heatmap.png`` will visualize the data access pattern in a + heatmap, showing which memory region (y-axis) got accessed when (x-axis) + and how frequently (color). +- ``wss_dist.png`` will show the distribution of the working set size. +- ``wss_chron_change.png`` will show how the working set size has + chronologically changed. + +You can view the visualizations of this example workload at [1]_. +Visualizations of other realistic workloads are available at [2]_ [3]_ [4]_. + +.. [1] https://damonitor.github.io/doc/html/v17/admin-guide/mm/damon/start.html#visualizing-recorded-patterns +.. [2] https://damonitor.github.io/test/result/visual/latest/rec.heatmap.1.png.html +.. [3] https://damonitor.github.io/test/result/visual/latest/rec.wss_sz.png.html +.. [4] https://damonitor.github.io/test/result/visual/latest/rec.wss_time.png.html diff --git a/Documentation/admin-guide/mm/damon/usage.rst b/Documentation/admin-guide/mm/damon/usage.rst new file mode 100644 index 000000000000..a72cda374aba --- /dev/null +++ b/Documentation/admin-guide/mm/damon/usage.rst @@ -0,0 +1,112 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============== +Detailed Usages +=============== + +DAMON provides below three interfaces for different users. + +- *DAMON user space tool.* + This is for privileged people such as system administrators who want a + just-working human-friendly interface. Using this, users can use the DAMON’s + major features in a human-friendly way. It may not be highly tuned for + special cases, though. It supports only virtual address spaces monitoring. +- *debugfs interface.* + This is for privileged user space programmers who want more optimized use of + DAMON. Using this, users can use DAMON’s major features by reading + from and writing to special debugfs files. Therefore, you can write and use + your personalized DAMON debugfs wrapper programs that reads/writes the + debugfs files instead of you. The DAMON user space tool is also a reference + implementation of such programs. It supports only virtual address spaces + monitoring. +- *Kernel Space Programming Interface.* + This is for kernel space programmers. Using this, users can utilize every + feature of DAMON most flexibly and efficiently by writing kernel space + DAMON application programs for you. You can even extend DAMON for various + address spaces. + +Nevertheless, you could write your own user space tool using the debugfs +interface. A reference implementation is available at +https://github.com/awslabs/damo. If you are a kernel programmer, you could +refer to :doc:`/vm/damon/api` for the kernel space programming interface. For +the reason, this document describes only the debugfs interface + +debugfs Interface +================= + +DAMON exports three files, ``attrs``, ``target_ids``, and ``monitor_on`` under +its debugfs directory, ``<debugfs>/damon/``. + + +Attributes +---------- + +Users can get and set the ``sampling interval``, ``aggregation interval``, +``regions update interval``, and min/max number of monitoring target regions by +reading from and writing to the ``attrs`` file. To know about the monitoring +attributes in detail, please refer to the :doc:`/vm/damon/design`. For +example, below commands set those values to 5 ms, 100 ms, 1,000 ms, 10 and +1000, and then check it again:: + + # cd <debugfs>/damon + # echo 5000 100000 1000000 10 1000 > attrs + # cat attrs + 5000 100000 1000000 10 1000 + + +Target IDs +---------- + +Some types of address spaces supports multiple monitoring target. For example, +the virtual memory address spaces monitoring can have multiple processes as the +monitoring targets. Users can set the targets by writing relevant id values of +the targets to, and get the ids of the current targets by reading from the +``target_ids`` file. In case of the virtual address spaces monitoring, the +values should be pids of the monitoring target processes. For example, below +commands set processes having pids 42 and 4242 as the monitoring targets and +check it again:: + + # cd <debugfs>/damon + # echo 42 4242 > target_ids + # cat target_ids + 42 4242 + +Note that setting the target ids doesn't start the monitoring. + + +Turning On/Off +-------------- + +Setting the files as described above doesn't incur effect unless you explicitly +start the monitoring. You can start, stop, and check the current status of the +monitoring by writing to and reading from the ``monitor_on`` file. Writing +``on`` to the file starts the monitoring of the targets with the attributes. +Writing ``off`` to the file stops those. DAMON also stops if every target +process is terminated. Below example commands turn on, off, and check the +status of DAMON:: + + # cd <debugfs>/damon + # echo on > monitor_on + # echo off > monitor_on + # cat monitor_on + off + +Please note that you cannot write to the above-mentioned debugfs files while +the monitoring is turned on. If you write to the files while DAMON is running, +an error code such as ``-EBUSY`` will be returned. + + +Tracepoint for Monitoring Results +================================= + +DAMON provides the monitoring results via a tracepoint, +``damon:damon_aggregated``. While the monitoring is turned on, you could +record the tracepoint events and show results using tracepoint supporting tools +like ``perf``. For example:: + + # echo on > monitor_on + # perf record -e damon:damon_aggregated & + # sleep 5 + # kill 9 $(pidof perf) + # echo off > monitor_on + # perf script diff --git a/Documentation/admin-guide/mm/hugetlbpage.rst b/Documentation/admin-guide/mm/hugetlbpage.rst index f7b1c7462991..8abaeb144e44 100644 --- a/Documentation/admin-guide/mm/hugetlbpage.rst +++ b/Documentation/admin-guide/mm/hugetlbpage.rst @@ -60,6 +60,10 @@ HugePages_Surp the pool above the value in ``/proc/sys/vm/nr_hugepages``. The maximum number of surplus huge pages is controlled by ``/proc/sys/vm/nr_overcommit_hugepages``. + Note: When the feature of freeing unused vmemmap pages associated + with each hugetlb page is enabled, the number of surplus huge pages + may be temporarily larger than the maximum number of surplus huge + pages when the system is under memory pressure. Hugepagesize is the default hugepage size (in Kb). Hugetlb @@ -80,6 +84,10 @@ returned to the huge page pool when freed by a task. A user with root privileges can dynamically allocate more or free some persistent huge pages by increasing or decreasing the value of ``nr_hugepages``. +Note: When the feature of freeing unused vmemmap pages associated with each +hugetlb page is enabled, we can fail to free the huge pages triggered by +the user when ths system is under memory pressure. Please try again later. + Pages that are used as huge pages are reserved inside the kernel and cannot be used for other purposes. Huge pages cannot be swapped out under memory pressure. @@ -145,6 +153,9 @@ default_hugepagesz will all result in 256 2M huge pages being allocated. Valid default huge page size is architecture dependent. +hugetlb_free_vmemmap + When CONFIG_HUGETLB_PAGE_FREE_VMEMMAP is set, this enables freeing + unused vmemmap pages associated with each HugeTLB page. When multiple huge page sizes are supported, ``/proc/sys/vm/nr_hugepages`` indicates the current number of pre-allocated huge pages of the default size. diff --git a/Documentation/admin-guide/mm/index.rst b/Documentation/admin-guide/mm/index.rst index 4b14d8b50e9e..cbd19d5e625f 100644 --- a/Documentation/admin-guide/mm/index.rst +++ b/Documentation/admin-guide/mm/index.rst @@ -27,6 +27,7 @@ the Linux memory management. concepts cma_debugfs + damon/index hugetlbpage idle_page_tracking ksm diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst index 05d51d2d8beb..03dfbc925252 100644 --- a/Documentation/admin-guide/mm/memory-hotplug.rst +++ b/Documentation/admin-guide/mm/memory-hotplug.rst @@ -1,453 +1,576 @@ .. _admin_guide_memory_hotplug: -============== -Memory Hotplug -============== +================== +Memory Hot(Un)Plug +================== -:Created: Jul 28 2007 -:Updated: Add some details about locking internals: Aug 20 2018 - -This document is about memory hotplug including how-to-use and current status. -Because Memory Hotplug is still under development, contents of this text will -be changed often. +This document describes generic Linux support for memory hot(un)plug with +a focus on System RAM, including ZONE_MOVABLE support. .. contents:: :local: -.. note:: +Introduction +============ - (1) x86_64's has special implementation for memory hotplug. - This text does not describe it. - (2) This text assumes that sysfs is mounted at ``/sys``. +Memory hot(un)plug allows for increasing and decreasing the size of physical +memory available to a machine at runtime. In the simplest case, it consists of +physically plugging or unplugging a DIMM at runtime, coordinated with the +operating system. +Memory hot(un)plug is used for various purposes: -Introduction -============ +- The physical memory available to a machine can be adjusted at runtime, up- or + downgrading the memory capacity. This dynamic memory resizing, sometimes + referred to as "capacity on demand", is frequently used with virtual machines + and logical partitions. -Purpose of memory hotplug -------------------------- +- Replacing hardware, such as DIMMs or whole NUMA nodes, without downtime. One + example is replacing failing memory modules. -Memory Hotplug allows users to increase/decrease the amount of memory. -Generally, there are two purposes. +- Reducing energy consumption either by physically unplugging memory modules or + by logically unplugging (parts of) memory modules from Linux. -(A) For changing the amount of memory. - This is to allow a feature like capacity on demand. -(B) For installing/removing DIMMs or NUMA-nodes physically. - This is to exchange DIMMs/NUMA-nodes, reduce power consumption, etc. +Further, the basic memory hot(un)plug infrastructure in Linux is nowadays also +used to expose persistent memory, other performance-differentiated memory and +reserved memory regions as ordinary system RAM to Linux. -(A) is required by highly virtualized environments and (B) is required by -hardware which supports memory power management. +Linux only supports memory hot(un)plug on selected 64 bit architectures, such as +x86_64, arm64, ppc64, s390x and ia64. -Linux memory hotplug is designed for both purpose. +Memory Hot(Un)Plug Granularity +------------------------------ -Phases of memory hotplug +Memory hot(un)plug in Linux uses the SPARSEMEM memory model, which divides the +physical memory address space into chunks of the same size: memory sections. The +size of a memory section is architecture dependent. For example, x86_64 uses +128 MiB and ppc64 uses 16 MiB. + +Memory sections are combined into chunks referred to as "memory blocks". The +size of a memory block is architecture dependent and corresponds to the smallest +granularity that can be hot(un)plugged. The default size of a memory block is +the same as memory section size, unless an architecture specifies otherwise. + +All memory blocks have the same size. + +Phases of Memory Hotplug ------------------------ -There are 2 phases in Memory Hotplug: +Memory hotplug consists of two phases: - 1) Physical Memory Hotplug phase - 2) Logical Memory Hotplug phase. +(1) Adding the memory to Linux +(2) Onlining memory blocks -The First phase is to communicate hardware/firmware and make/erase -environment for hotplugged memory. Basically, this phase is necessary -for the purpose (B), but this is good phase for communication between -highly virtualized environments too. +In the first phase, metadata, such as the memory map ("memmap") and page tables +for the direct mapping, is allocated and initialized, and memory blocks are +created; the latter also creates sysfs files for managing newly created memory +blocks. -When memory is hotplugged, the kernel recognizes new memory, makes new memory -management tables, and makes sysfs files for new memory's operation. +In the second phase, added memory is exposed to the page allocator. After this +phase, the memory is visible in memory statistics, such as free and total +memory, of the system. -If firmware supports notification of connection of new memory to OS, -this phase is triggered automatically. ACPI can notify this event. If not, -"probe" operation by system administration is used instead. -(see :ref:`memory_hotplug_physical_mem`). +Phases of Memory Hotunplug +-------------------------- -Logical Memory Hotplug phase is to change memory state into -available/unavailable for users. Amount of memory from user's view is -changed by this phase. The kernel makes all memory in it as free pages -when a memory range is available. +Memory hotunplug consists of two phases: -In this document, this phase is described as online/offline. +(1) Offlining memory blocks +(2) Removing the memory from Linux -Logical Memory Hotplug phase is triggered by write of sysfs file by system -administrator. For the hot-add case, it must be executed after Physical Hotplug -phase by hand. -(However, if you writes udev's hotplug scripts for memory hotplug, these -phases can be execute in seamless way.) +In the fist phase, memory is "hidden" from the page allocator again, for +example, by migrating busy memory to other memory locations and removing all +relevant free pages from the page allocator After this phase, the memory is no +longer visible in memory statistics of the system. -Unit of Memory online/offline operation ---------------------------------------- +In the second phase, the memory blocks are removed and metadata is freed. -Memory hotplug uses SPARSEMEM memory model which allows memory to be divided -into chunks of the same size. These chunks are called "sections". The size of -a memory section is architecture dependent. For example, power uses 16MiB, ia64 -uses 1GiB. +Memory Hotplug Notifications +============================ -Memory sections are combined into chunks referred to as "memory blocks". The -size of a memory block is architecture dependent and represents the logical -unit upon which memory online/offline operations are to be performed. The -default size of a memory block is the same as memory section size unless an -architecture specifies otherwise. (see :ref:`memory_hotplug_sysfs_files`.) +There are various ways how Linux is notified about memory hotplug events such +that it can start adding hotplugged memory. This description is limited to +systems that support ACPI; mechanisms specific to other firmware interfaces or +virtual machines are not described. -To determine the size (in bytes) of a memory block please read this file:: +ACPI Notifications +------------------ - /sys/devices/system/memory/block_size_bytes +Platforms that support ACPI, such as x86_64, can support memory hotplug +notifications via ACPI. -Kernel Configuration -==================== +In general, a firmware supporting memory hotplug defines a memory class object +HID "PNP0C80". When notified about hotplug of a new memory device, the ACPI +driver will hotplug the memory to Linux. -To use memory hotplug feature, kernel must be compiled with following -config options. +If the firmware supports hotplug of NUMA nodes, it defines an object _HID +"ACPI0004", "PNP0A05", or "PNP0A06". When notified about an hotplug event, all +assigned memory devices are added to Linux by the ACPI driver. -- For all memory hotplug: - - Memory model -> Sparse Memory (``CONFIG_SPARSEMEM``) - - Allow for memory hot-add (``CONFIG_MEMORY_HOTPLUG``) +Similarly, Linux can be notified about requests to hotunplug a memory device or +a NUMA node via ACPI. The ACPI driver will try offlining all relevant memory +blocks, and, if successful, hotunplug the memory from Linux. -- To enable memory removal, the following are also necessary: - - Allow for memory hot remove (``CONFIG_MEMORY_HOTREMOVE``) - - Page Migration (``CONFIG_MIGRATION``) +Manual Probing +-------------- -- For ACPI memory hotplug, the following are also necessary: - - Memory hotplug (under ACPI Support menu) (``CONFIG_ACPI_HOTPLUG_MEMORY``) - - This option can be kernel module. +On some architectures, the firmware may not be able to notify the operating +system about a memory hotplug event. Instead, the memory has to be manually +probed from user space. -- As a related configuration, if your box has a feature of NUMA-node hotplug - via ACPI, then this option is necessary too. +The probe interface is located at:: - - ACPI0004,PNP0A05 and PNP0A06 Container Driver (under ACPI Support menu) - (``CONFIG_ACPI_CONTAINER``). + /sys/devices/system/memory/probe - This option can be kernel module too. +Only complete memory blocks can be probed. Individual memory blocks are probed +by providing the physical start address of the memory block:: + % echo addr > /sys/devices/system/memory/probe -.. _memory_hotplug_sysfs_files: +Which results in a memory block for the range [addr, addr + memory_block_size) +being created. -sysfs files for memory hotplug -============================== +.. note:: -All memory blocks have their device information in sysfs. Each memory block -is described under ``/sys/devices/system/memory`` as:: + Using the probe interface is discouraged as it is easy to crash the kernel, + because Linux cannot validate user input; this interface might be removed in + the future. - /sys/devices/system/memory/memoryXXX +Onlining and Offlining Memory Blocks +==================================== -where XXX is the memory block id. +After a memory block has been created, Linux has to be instructed to actually +make use of that memory: the memory block has to be "online". -For the memory block covered by the sysfs directory. It is expected that all -memory sections in this range are present and no memory holes exist in the -range. Currently there is no way to determine if there is a memory hole, but -the existence of one should not affect the hotplug capabilities of the memory -block. +Before a memory block can be removed, Linux has to stop using any memory part of +the memory block: the memory block has to be "offlined". -For example, assume 1GiB memory block size. A device for a memory starting at -0x100000000 is ``/sys/device/system/memory/memory4``:: +The Linux kernel can be configured to automatically online added memory blocks +and drivers automatically trigger offlining of memory blocks when trying +hotunplug of memory. Memory blocks can only be removed once offlining succeeded +and drivers may trigger offlining of memory blocks when attempting hotunplug of +memory. - (0x100000000 / 1Gib = 4) +Onlining Memory Blocks Manually +------------------------------- -This device covers address range [0x100000000 ... 0x140000000) +If auto-onlining of memory blocks isn't enabled, user-space has to manually +trigger onlining of memory blocks. Often, udev rules are used to automate this +task in user space. -Under each memory block, you can see 5 files: +Onlining of a memory block can be triggered via:: -- ``/sys/devices/system/memory/memoryXXX/phys_index`` -- ``/sys/devices/system/memory/memoryXXX/phys_device`` -- ``/sys/devices/system/memory/memoryXXX/state`` -- ``/sys/devices/system/memory/memoryXXX/removable`` -- ``/sys/devices/system/memory/memoryXXX/valid_zones`` + % echo online > /sys/devices/system/memory/memoryXXX/state -=================== ============================================================ -``phys_index`` read-only and contains memory block id, same as XXX. -``state`` read-write +Or alternatively:: - - at read: contains online/offline state of memory. - - at write: user can specify "online_kernel", + % echo 1 > /sys/devices/system/memory/memoryXXX/online - "online_movable", "online", "offline" command - which will be performed on all sections in the block. -``phys_device`` read-only: legacy interface only ever used on s390x to - expose the covered storage increment. -``removable`` read-only: legacy interface that indicated whether a memory - block was likely to be offlineable or not. Newer kernel - versions return "1" if and only if the kernel supports - memory offlining. -``valid_zones`` read-only: designed to show by which zone memory provided by - a memory block is managed, and to show by which zone memory - provided by an offline memory block could be managed when - onlining. - - The first column shows it`s default zone. - - "memory6/valid_zones: Normal Movable" shows this memoryblock - can be onlined to ZONE_NORMAL by default and to ZONE_MOVABLE - by online_movable. - - "memory7/valid_zones: Movable Normal" shows this memoryblock - can be onlined to ZONE_MOVABLE by default and to ZONE_NORMAL - by online_kernel. -=================== ============================================================ +The kernel will select the target zone automatically, usually defaulting to +``ZONE_NORMAL`` unless ``movablecore=1`` has been specified on the kernel +command line or if the memory block would intersect the ZONE_MOVABLE already. -.. note:: +One can explicitly request to associate an offline memory block with +ZONE_MOVABLE by:: - These directories/files appear after physical memory hotplug phase. + % echo online_movable > /sys/devices/system/memory/memoryXXX/state -If CONFIG_NUMA is enabled the memoryXXX/ directories can also be accessed -via symbolic links located in the ``/sys/devices/system/node/node*`` directories. +Or one can explicitly request a kernel zone (usually ZONE_NORMAL) by:: -For example:: + % echo online_kernel > /sys/devices/system/memory/memoryXXX/state - /sys/devices/system/node/node0/memory9 -> ../../memory/memory9 +In any case, if onlining succeeds, the state of the memory block is changed to +be "online". If it fails, the state of the memory block will remain unchanged +and the above commands will fail. -A backlink will also be created:: +Onlining Memory Blocks Automatically +------------------------------------ - /sys/devices/system/memory/memory9/node0 -> ../../node/node0 +The kernel can be configured to try auto-onlining of newly added memory blocks. +If this feature is disabled, the memory blocks will stay offline until +explicitly onlined from user space. -.. _memory_hotplug_physical_mem: +The configured auto-online behavior can be observed via:: -Physical memory hot-add phase -============================= + % cat /sys/devices/system/memory/auto_online_blocks -Hardware(Firmware) Support --------------------------- +Auto-onlining can be enabled by writing ``online``, ``online_kernel`` or +``online_movable`` to that file, like:: -On x86_64/ia64 platform, memory hotplug by ACPI is supported. + % echo online > /sys/devices/system/memory/auto_online_blocks -In general, the firmware (ACPI) which supports memory hotplug defines -memory class object of _HID "PNP0C80". When a notify is asserted to PNP0C80, -Linux's ACPI handler does hot-add memory to the system and calls a hotplug udev -script. This will be done automatically. +Modifying the auto-online behavior will only affect all subsequently added +memory blocks only. -But scripts for memory hotplug are not contained in generic udev package(now). -You may have to write it by yourself or online/offline memory by hand. -Please see :ref:`memory_hotplug_how_to_online_memory` and -:ref:`memory_hotplug_how_to_offline_memory`. +.. note:: -If firmware supports NUMA-node hotplug, and defines an object _HID "ACPI0004", -"PNP0A05", or "PNP0A06", notification is asserted to it, and ACPI handler -calls hotplug code for all of objects which are defined in it. -If memory device is found, memory hotplug code will be called. + In corner cases, auto-onlining can fail. The kernel won't retry. Note that + auto-onlining is not expected to fail in default configurations. -Notify memory hot-add event by hand ------------------------------------ +.. note:: -On some architectures, the firmware may not notify the kernel of a memory -hotplug event. Therefore, the memory "probe" interface is supported to -explicitly notify the kernel. This interface depends on -CONFIG_ARCH_MEMORY_PROBE and can be configured on powerpc, sh, and x86 -if hotplug is supported, although for x86 this should be handled by ACPI -notification. + DLPAR on ppc64 ignores the ``offline`` setting and will still online added + memory blocks; if onlining fails, memory blocks are removed again. -Probe interface is located at:: +Offlining Memory Blocks +----------------------- - /sys/devices/system/memory/probe +In the current implementation, Linux's memory offlining will try migrating all +movable pages off the affected memory block. As most kernel allocations, such as +page tables, are unmovable, page migration can fail and, therefore, inhibit +memory offlining from succeeding. -You can tell the physical address of new memory to the kernel by:: +Having the memory provided by memory block managed by ZONE_MOVABLE significantly +increases memory offlining reliability; still, memory offlining can fail in +some corner cases. - % echo start_address_of_new_memory > /sys/devices/system/memory/probe +Further, memory offlining might retry for a long time (or even forever), until +aborted by the user. -Then, [start_address_of_new_memory, start_address_of_new_memory + -memory_block_size] memory range is hot-added. In this case, hotplug script is -not called (in current implementation). You'll have to online memory by -yourself. Please see :ref:`memory_hotplug_how_to_online_memory`. +Offlining of a memory block can be triggered via:: -Logical Memory hot-add phase -============================ + % echo offline > /sys/devices/system/memory/memoryXXX/state -State of memory ---------------- +Or alternatively:: -To see (online/offline) state of a memory block, read 'state' file:: + % echo 0 > /sys/devices/system/memory/memoryXXX/online - % cat /sys/device/system/memory/memoryXXX/state +If offlining succeeds, the state of the memory block is changed to be "offline". +If it fails, the state of the memory block will remain unchanged and the above +commands will fail, for example, via:: + bash: echo: write error: Device or resource busy -- If the memory block is online, you'll read "online". -- If the memory block is offline, you'll read "offline". +or via:: + bash: echo: write error: Invalid argument -.. _memory_hotplug_how_to_online_memory: +Observing the State of Memory Blocks +------------------------------------ -How to online memory --------------------- +The state (online/offline/going-offline) of a memory block can be observed +either via:: -When the memory is hot-added, the kernel decides whether or not to "online" -it according to the policy which can be read from "auto_online_blocks" file:: + % cat /sys/device/system/memory/memoryXXX/state - % cat /sys/devices/system/memory/auto_online_blocks +Or alternatively (1/0) via:: -The default depends on the CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE kernel config -option. If it is disabled the default is "offline" which means the newly added -memory is not in a ready-to-use state and you have to "online" the newly added -memory blocks manually. Automatic onlining can be requested by writing "online" -to "auto_online_blocks" file:: + % cat /sys/device/system/memory/memoryXXX/online - % echo online > /sys/devices/system/memory/auto_online_blocks +For an online memory block, the managing zone can be observed via:: -This sets a global policy and impacts all memory blocks that will subsequently -be hotplugged. Currently offline blocks keep their state. It is possible, under -certain circumstances, that some memory blocks will be added but will fail to -online. User space tools can check their "state" files -(``/sys/devices/system/memory/memoryXXX/state``) and try to online them manually. + % cat /sys/device/system/memory/memoryXXX/valid_zones -If the automatic onlining wasn't requested, failed, or some memory block was -offlined it is possible to change the individual block's state by writing to the -"state" file:: +Configuring Memory Hot(Un)Plug +============================== - % echo online > /sys/devices/system/memory/memoryXXX/state +There are various ways how system administrators can configure memory +hot(un)plug and interact with memory blocks, especially, to online them. -This onlining will not change the ZONE type of the target memory block, -If the memory block doesn't belong to any zone an appropriate kernel zone -(usually ZONE_NORMAL) will be used unless movable_node kernel command line -option is specified when ZONE_MOVABLE will be used. +Memory Hot(Un)Plug Configuration via Sysfs +------------------------------------------ -You can explicitly request to associate it with ZONE_MOVABLE by:: +Some memory hot(un)plug properties can be configured or inspected via sysfs in:: - % echo online_movable > /sys/devices/system/memory/memoryXXX/state + /sys/devices/system/memory/ -.. note:: current limit: this memory block must be adjacent to ZONE_MOVABLE +The following files are currently defined: -Or you can explicitly request a kernel zone (usually ZONE_NORMAL) by:: +====================== ========================================================= +``auto_online_blocks`` read-write: set or get the default state of new memory + blocks; configure auto-onlining. - % echo online_kernel > /sys/devices/system/memory/memoryXXX/state + The default value depends on the + CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE kernel configuration + option. -.. note:: current limit: this memory block must be adjacent to ZONE_NORMAL + See the ``state`` property of memory blocks for details. +``block_size_bytes`` read-only: the size in bytes of a memory block. +``probe`` write-only: add (probe) selected memory blocks manually + from user space by supplying the physical start address. -An explicit zone onlining can fail (e.g. when the range is already within -and existing and incompatible zone already). + Availability depends on the CONFIG_ARCH_MEMORY_PROBE + kernel configuration option. +``uevent`` read-write: generic udev file for device subsystems. +====================== ========================================================= -After this, memory block XXX's state will be 'online' and the amount of -available memory will be increased. +.. note:: -This may be changed in future. + When the CONFIG_MEMORY_FAILURE kernel configuration option is enabled, two + additional files ``hard_offline_page`` and ``soft_offline_page`` are available + to trigger hwpoisoning of pages, for example, for testing purposes. Note that + this functionality is not really related to memory hot(un)plug or actual + offlining of memory blocks. -Logical memory remove -===================== +Memory Block Configuration via Sysfs +------------------------------------ -Memory offline and ZONE_MOVABLE -------------------------------- +Each memory block is represented as a memory block device that can be +onlined or offlined. All memory blocks have their device information located in +sysfs. Each present memory block is listed under +``/sys/devices/system/memory`` as:: -Memory offlining is more complicated than memory online. Because memory offline -has to make the whole memory block be unused, memory offline can fail if -the memory block includes memory which cannot be freed. + /sys/devices/system/memory/memoryXXX -In general, memory offline can use 2 techniques. +where XXX is the memory block id; the number of digits is variable. -(1) reclaim and free all memory in the memory block. -(2) migrate all pages in the memory block. +A present memory block indicates that some memory in the range is present; +however, a memory block might span memory holes. A memory block spanning memory +holes cannot be offlined. -In the current implementation, Linux's memory offline uses method (2), freeing -all pages in the memory block by page migration. But not all pages are -migratable. Under current Linux, migratable pages are anonymous pages and -page caches. For offlining a memory block by migration, the kernel has to -guarantee that the memory block contains only migratable pages. +For example, assume 1 GiB memory block size. A device for a memory starting at +0x100000000 is ``/sys/device/system/memory/memory4``:: -Now, a boot option for making a memory block which consists of migratable pages -is supported. By specifying "kernelcore=" or "movablecore=" boot option, you can -create ZONE_MOVABLE...a zone which is just used for movable pages. -(See also Documentation/admin-guide/kernel-parameters.rst) + (0x100000000 / 1Gib = 4) -Assume the system has "TOTAL" amount of memory at boot time, this boot option -creates ZONE_MOVABLE as following. +This device covers address range [0x100000000 ... 0x140000000) -1) When kernelcore=YYYY boot option is used, - Size of memory not for movable pages (not for offline) is YYYY. - Size of memory for movable pages (for offline) is TOTAL-YYYY. +The following files are currently defined: -2) When movablecore=ZZZZ boot option is used, - Size of memory not for movable pages (not for offline) is TOTAL - ZZZZ. - Size of memory for movable pages (for offline) is ZZZZ. +=================== ============================================================ +``online`` read-write: simplified interface to trigger onlining / + offlining and to observe the state of a memory block. + When onlining, the zone is selected automatically. +``phys_device`` read-only: legacy interface only ever used on s390x to + expose the covered storage increment. +``phys_index`` read-only: the memory block id (XXX). +``removable`` read-only: legacy interface that indicated whether a memory + block was likely to be offlineable or not. Nowadays, the + kernel return ``1`` if and only if it supports memory + offlining. +``state`` read-write: advanced interface to trigger onlining / + offlining and to observe the state of a memory block. + + When writing, ``online``, ``offline``, ``online_kernel`` and + ``online_movable`` are supported. + + ``online_movable`` specifies onlining to ZONE_MOVABLE. + ``online_kernel`` specifies onlining to the default kernel + zone for the memory block, such as ZONE_NORMAL. + ``online`` let's the kernel select the zone automatically. + + When reading, ``online``, ``offline`` and ``going-offline`` + may be returned. +``uevent`` read-write: generic uevent file for devices. +``valid_zones`` read-only: when a block is online, shows the zone it + belongs to; when a block is offline, shows what zone will + manage it when the block will be onlined. + + For online memory blocks, ``DMA``, ``DMA32``, ``Normal``, + ``Movable`` and ``none`` may be returned. ``none`` indicates + that memory provided by a memory block is managed by + multiple zones or spans multiple nodes; such memory blocks + cannot be offlined. ``Movable`` indicates ZONE_MOVABLE. + Other values indicate a kernel zone. + + For offline memory blocks, the first column shows the + zone the kernel would select when onlining the memory block + right now without further specifying a zone. + + Availability depends on the CONFIG_MEMORY_HOTREMOVE + kernel configuration option. +=================== ============================================================ .. note:: - Unfortunately, there is no information to show which memory block belongs - to ZONE_MOVABLE. This is TBD. + If the CONFIG_NUMA kernel configuration option is enabled, the memoryXXX/ + directories can also be accessed via symbolic links located in the + ``/sys/devices/system/node/node*`` directories. + + For example:: + + /sys/devices/system/node/node0/memory9 -> ../../memory/memory9 + + A backlink will also be created:: + + /sys/devices/system/memory/memory9/node0 -> ../../node/node0 + +Command Line Parameters +----------------------- + +Some command line parameters affect memory hot(un)plug handling. The following +command line parameters are relevant: + +======================== ======================================================= +``memhp_default_state`` configure auto-onlining by essentially setting + ``/sys/devices/system/memory/auto_online_blocks``. +``movablecore`` configure automatic zone selection of the kernel. When + set, the kernel will default to ZONE_MOVABLE, unless + other zones can be kept contiguous. +======================== ======================================================= + +Module Parameters +------------------ + +Instead of additional command line parameters or sysfs files, the +``memory_hotplug`` subsystem now provides a dedicated namespace for module +parameters. Module parameters can be set via the command line by predicating +them with ``memory_hotplug.`` such as:: + + memory_hotplug.memmap_on_memory=1 + +and they can be observed (and some even modified at runtime) via:: + + /sys/modules/memory_hotplug/parameters/ + +The following module parameters are currently defined: + +======================== ======================================================= +``memmap_on_memory`` read-write: Allocate memory for the memmap from the + added memory block itself. Even if enabled, actual + support depends on various other system properties and + should only be regarded as a hint whether the behavior + would be desired. + + While allocating the memmap from the memory block + itself makes memory hotplug less likely to fail and + keeps the memmap on the same NUMA node in any case, it + can fragment physical memory in a way that huge pages + in bigger granularity cannot be formed on hotplugged + memory. +======================== ======================================================= + +ZONE_MOVABLE +============ + +ZONE_MOVABLE is an important mechanism for more reliable memory offlining. +Further, having system RAM managed by ZONE_MOVABLE instead of one of the +kernel zones can increase the number of possible transparent huge pages and +dynamically allocated huge pages. + +Most kernel allocations are unmovable. Important examples include the memory +map (usually 1/64ths of memory), page tables, and kmalloc(). Such allocations +can only be served from the kernel zones. + +Most user space pages, such as anonymous memory, and page cache pages are +movable. Such allocations can be served from ZONE_MOVABLE and the kernel zones. + +Only movable allocations are served from ZONE_MOVABLE, resulting in unmovable +allocations being limited to the kernel zones. Without ZONE_MOVABLE, there is +absolutely no guarantee whether a memory block can be offlined successfully. + +Zone Imbalances +--------------- + +Having too much system RAM managed by ZONE_MOVABLE is called a zone imbalance, +which can harm the system or degrade performance. As one example, the kernel +might crash because it runs out of free memory for unmovable allocations, +although there is still plenty of free memory left in ZONE_MOVABLE. + +Usually, MOVABLE:KERNEL ratios of up to 3:1 or even 4:1 are fine. Ratios of 63:1 +are definitely impossible due to the overhead for the memory map. + +Actual safe zone ratios depend on the workload. Extreme cases, like excessive +long-term pinning of pages, might not be able to deal with ZONE_MOVABLE at all. .. note:: - Techniques that rely on long-term pinnings of memory (especially, RDMA and - vfio) are fundamentally problematic with ZONE_MOVABLE and, therefore, memory - hot remove. Pinned pages cannot reside on ZONE_MOVABLE, to guarantee that - memory can still get hot removed - be aware that pinning can fail even if - there is plenty of free memory in ZONE_MOVABLE. In addition, using - ZONE_MOVABLE might make page pinning more expensive, because pages have to be - migrated off that zone first. -.. _memory_hotplug_how_to_offline_memory: + CMA memory part of a kernel zone essentially behaves like memory in + ZONE_MOVABLE and similar considerations apply, especially when combining + CMA with ZONE_MOVABLE. -How to offline memory ---------------------- +ZONE_MOVABLE Sizing Considerations +---------------------------------- -You can offline a memory block by using the same sysfs interface that was used -in memory onlining:: +We usually expect that a large portion of available system RAM will actually +be consumed by user space, either directly or indirectly via the page cache. In +the normal case, ZONE_MOVABLE can be used when allocating such pages just fine. - % echo offline > /sys/devices/system/memory/memoryXXX/state +With that in mind, it makes sense that we can have a big portion of system RAM +managed by ZONE_MOVABLE. However, there are some things to consider when using +ZONE_MOVABLE, especially when fine-tuning zone ratios: + +- Having a lot of offline memory blocks. Even offline memory blocks consume + memory for metadata and page tables in the direct map; having a lot of offline + memory blocks is not a typical case, though. + +- Memory ballooning without balloon compaction is incompatible with + ZONE_MOVABLE. Only some implementations, such as virtio-balloon and + pseries CMM, fully support balloon compaction. + + Further, the CONFIG_BALLOON_COMPACTION kernel configuration option might be + disabled. In that case, balloon inflation will only perform unmovable + allocations and silently create a zone imbalance, usually triggered by + inflation requests from the hypervisor. + +- Gigantic pages are unmovable, resulting in user space consuming a + lot of unmovable memory. + +- Huge pages are unmovable when an architectures does not support huge + page migration, resulting in a similar issue as with gigantic pages. + +- Page tables are unmovable. Excessive swapping, mapping extremely large + files or ZONE_DEVICE memory can be problematic, although only really relevant + in corner cases. When we manage a lot of user space memory that has been + swapped out or is served from a file/persistent memory/... we still need a lot + of page tables to manage that memory once user space accessed that memory. + +- In certain DAX configurations the memory map for the device memory will be + allocated from the kernel zones. + +- KASAN can have a significant memory overhead, for example, consuming 1/8th of + the total system memory size as (unmovable) tracking metadata. + +- Long-term pinning of pages. Techniques that rely on long-term pinnings + (especially, RDMA and vfio/mdev) are fundamentally problematic with + ZONE_MOVABLE, and therefore, memory offlining. Pinned pages cannot reside + on ZONE_MOVABLE as that would turn these pages unmovable. Therefore, they + have to be migrated off that zone while pinning. Pinning a page can fail + even if there is plenty of free memory in ZONE_MOVABLE. + + In addition, using ZONE_MOVABLE might make page pinning more expensive, + because of the page migration overhead. + +By default, all the memory configured at boot time is managed by the kernel +zones and ZONE_MOVABLE is not used. + +To enable ZONE_MOVABLE to include the memory present at boot and to control the +ratio between movable and kernel zones there are two command line options: +``kernelcore=`` and ``movablecore=``. See +Documentation/admin-guide/kernel-parameters.rst for their description. + +Memory Offlining and ZONE_MOVABLE +--------------------------------- + +Even with ZONE_MOVABLE, there are some corner cases where offlining a memory +block might fail: + +- Memory blocks with memory holes; this applies to memory blocks present during + boot and can apply to memory blocks hotplugged via the XEN balloon and the + Hyper-V balloon. + +- Mixed NUMA nodes and mixed zones within a single memory block prevent memory + offlining; this applies to memory blocks present during boot only. + +- Special memory blocks prevented by the system from getting offlined. Examples + include any memory available during boot on arm64 or memory blocks spanning + the crashkernel area on s390x; this usually applies to memory blocks present + during boot only. + +- Memory blocks overlapping with CMA areas cannot be offlined, this applies to + memory blocks present during boot only. + +- Concurrent activity that operates on the same physical memory area, such as + allocating gigantic pages, can result in temporary offlining failures. + +- Out of memory when dissolving huge pages, especially when freeing unused + vmemmap pages associated with each hugetlb page is enabled. + + Offlining code may be able to migrate huge page contents, but may not be able + to dissolve the source huge page because it fails allocating (unmovable) pages + for the vmemmap, because the system might not have free memory in the kernel + zones left. + + Users that depend on memory offlining to succeed for movable zones should + carefully consider whether the memory savings gained from this feature are + worth the risk of possibly not being able to offline memory in certain + situations. + +Further, when running into out of memory situations while migrating pages, or +when still encountering permanently unmovable pages within ZONE_MOVABLE +(-> BUG), memory offlining will keep retrying until it eventually succeeds. + +When offlining is triggered from user space, the offlining context can be +terminated by sending a fatal signal. A timeout based offlining can easily be +implemented via:: -If offline succeeds, the state of the memory block is changed to be "offline". -If it fails, some error core (like -EBUSY) will be returned by the kernel. -Even if a memory block does not belong to ZONE_MOVABLE, you can try to offline -it. If it doesn't contain 'unmovable' memory, you'll get success. - -A memory block under ZONE_MOVABLE is considered to be able to be offlined -easily. But under some busy state, it may return -EBUSY. Even if a memory -block cannot be offlined due to -EBUSY, you can retry offlining it and may be -able to offline it (or not). (For example, a page is referred to by some kernel -internal call and released soon.) - -Consideration: - Memory hotplug's design direction is to make the possibility of memory - offlining higher and to guarantee unplugging memory under any situation. But - it needs more work. Returning -EBUSY under some situation may be good because - the user can decide to retry more or not by himself. Currently, memory - offlining code does some amount of retry with 120 seconds timeout. - -Physical memory remove -====================== - -Need more implementation yet.... - - Notification completion of remove works by OS to firmware. - - Guard from remove if not yet. - - -Locking Internals -================= - -When adding/removing memory that uses memory block devices (i.e. ordinary RAM), -the device_hotplug_lock should be held to: - -- synchronize against online/offline requests (e.g. via sysfs). This way, memory - block devices can only be accessed (.online/.state attributes) by user - space once memory has been fully added. And when removing memory, we - know nobody is in critical sections. -- synchronize against CPU hotplug and similar (e.g. relevant for ACPI and PPC) - -Especially, there is a possible lock inversion that is avoided using -device_hotplug_lock when adding memory and user space tries to online that -memory faster than expected: - -- device_online() will first take the device_lock(), followed by - mem_hotplug_lock -- add_memory_resource() will first take the mem_hotplug_lock, followed by - the device_lock() (while creating the devices, during bus_add_device()). - -As the device is visible to user space before taking the device_lock(), this -can result in a lock inversion. - -onlining/offlining of memory should be done via device_online()/ -device_offline() - to make sure it is properly synchronized to actions -via sysfs. Holding device_hotplug_lock is advised (to e.g. protect online_type) - -When adding/removing/onlining/offlining memory or adding/removing -heterogeneous/device memory, we should always hold the mem_hotplug_lock in -write mode to serialise memory hotplug (e.g. access to global/zone -variables). - -In addition, mem_hotplug_lock (in contrast to device_hotplug_lock) in read -mode allows for a quite efficient get_online_mems/put_online_mems -implementation, so code accessing memory can protect from that memory -vanishing. - - -Future Work -=========== - - - allowing memory hot-add to ZONE_MOVABLE. maybe we need some switch like - sysctl or new control file. - - showing memory block and physical device relationship. - - test and make it better memory offlining. - - support HugeTLB page migration and offlining. - - memmap removing at memory offline. - - physical remove memory. + % timeout $TIMEOUT offline_block | failure_handling diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Documentation/admin-guide/mm/numa_memory_policy.rst index 067a90a1499c..64fd0ba0d057 100644 --- a/Documentation/admin-guide/mm/numa_memory_policy.rst +++ b/Documentation/admin-guide/mm/numa_memory_policy.rst @@ -245,6 +245,13 @@ MPOL_INTERLEAVED address range or file. During system boot up, the temporary interleaved system default policy works in this mode. +MPOL_PREFERRED_MANY + This mode specifices that the allocation should be preferrably + satisfied from the nodemask specified in the policy. If there is + a memory pressure on all nodes in the nodemask, the allocation + can fall back to all existing numa nodes. This is effectively + MPOL_PREFERRED allowed for a mask rather than a single node. + NUMA memory policy supports the following optional mode flags: MPOL_F_STATIC_NODES @@ -253,10 +260,10 @@ MPOL_F_STATIC_NODES nodes changes after the memory policy has been defined. Without this flag, any time a mempolicy is rebound because of a - change in the set of allowed nodes, the node (Preferred) or - nodemask (Bind, Interleave) is remapped to the new set of - allowed nodes. This may result in nodes being used that were - previously undesired. + change in the set of allowed nodes, the preferred nodemask (Preferred + Many), preferred node (Preferred) or nodemask (Bind, Interleave) is + remapped to the new set of allowed nodes. This may result in nodes + being used that were previously undesired. With this flag, if the user-specified nodes overlap with the nodes allowed by the task's cpuset, then the memory policy is diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst index 340a5aee9b80..fb578fbbb76c 100644 --- a/Documentation/admin-guide/mm/pagemap.rst +++ b/Documentation/admin-guide/mm/pagemap.rst @@ -21,6 +21,8 @@ There are four components to pagemap: * Bit 55 pte is soft-dirty (see :ref:`Documentation/admin-guide/mm/soft-dirty.rst <soft_dirty>`) * Bit 56 page exclusively mapped (since 4.2) + * Bit 57 pte is uffd-wp write-protected (since 5.13) (see + :ref:`Documentation/admin-guide/mm/userfaultfd.rst <userfaultfd>`) * Bits 57-60 zero * Bit 61 page is file-page or shared-anon (since 3.5) * Bit 62 page swapped diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst index 3aa38e8b8361..6528036093e1 100644 --- a/Documentation/admin-guide/mm/userfaultfd.rst +++ b/Documentation/admin-guide/mm/userfaultfd.rst @@ -77,7 +77,8 @@ events, except page fault notifications, may be generated: - ``UFFD_FEATURE_MINOR_HUGETLBFS`` indicates that the kernel supports ``UFFDIO_REGISTER_MODE_MINOR`` registration for hugetlbfs virtual memory - areas. + areas. ``UFFD_FEATURE_MINOR_SHMEM`` is the analogous feature indicating + support for shmem virtual memory areas. The userland application should set the feature flags it intends to use when invoking the ``UFFDIO_API`` ioctl, to request that those features be diff --git a/Documentation/admin-guide/pm/cpuidle.rst b/Documentation/admin-guide/pm/cpuidle.rst index 10fde58d0869..aec2cd2aaea7 100644 --- a/Documentation/admin-guide/pm/cpuidle.rst +++ b/Documentation/admin-guide/pm/cpuidle.rst @@ -347,81 +347,8 @@ for tickless systems. It follows the same basic strategy as the ``menu`` `one <menu-gov_>`_: it always tries to find the deepest idle state suitable for the given conditions. However, it applies a different approach to that problem. -First, it does not use sleep length correction factors, but instead it attempts -to correlate the observed idle duration values with the available idle states -and use that information to pick up the idle state that is most likely to -"match" the upcoming CPU idle interval. Second, it does not take the tasks -that were running on the given CPU in the past and are waiting on some I/O -operations to complete now at all (there is no guarantee that they will run on -the same CPU when they become runnable again) and the pattern detection code in -it avoids taking timer wakeups into account. It also only uses idle duration -values less than the current time till the closest timer (with the scheduler -tick excluded) for that purpose. - -Like in the ``menu`` governor `case <menu-gov_>`_, the first step is to obtain -the *sleep length*, which is the time until the closest timer event with the -assumption that the scheduler tick will be stopped (that also is the upper bound -on the time until the next CPU wakeup). That value is then used to preselect an -idle state on the basis of three metrics maintained for each idle state provided -by the ``CPUIdle`` driver: ``hits``, ``misses`` and ``early_hits``. - -The ``hits`` and ``misses`` metrics measure the likelihood that a given idle -state will "match" the observed (post-wakeup) idle duration if it "matches" the -sleep length. They both are subject to decay (after a CPU wakeup) every time -the target residency of the idle state corresponding to them is less than or -equal to the sleep length and the target residency of the next idle state is -greater than the sleep length (that is, when the idle state corresponding to -them "matches" the sleep length). The ``hits`` metric is increased if the -former condition is satisfied and the target residency of the given idle state -is less than or equal to the observed idle duration and the target residency of -the next idle state is greater than the observed idle duration at the same time -(that is, it is increased when the given idle state "matches" both the sleep -length and the observed idle duration). In turn, the ``misses`` metric is -increased when the given idle state "matches" the sleep length only and the -observed idle duration is too short for its target residency. - -The ``early_hits`` metric measures the likelihood that a given idle state will -"match" the observed (post-wakeup) idle duration if it does not "match" the -sleep length. It is subject to decay on every CPU wakeup and it is increased -when the idle state corresponding to it "matches" the observed (post-wakeup) -idle duration and the target residency of the next idle state is less than or -equal to the sleep length (i.e. the idle state "matching" the sleep length is -deeper than the given one). - -The governor walks the list of idle states provided by the ``CPUIdle`` driver -and finds the last (deepest) one with the target residency less than or equal -to the sleep length. Then, the ``hits`` and ``misses`` metrics of that idle -state are compared with each other and it is preselected if the ``hits`` one is -greater (which means that that idle state is likely to "match" the observed idle -duration after CPU wakeup). If the ``misses`` one is greater, the governor -preselects the shallower idle state with the maximum ``early_hits`` metric -(or if there are multiple shallower idle states with equal ``early_hits`` -metric which also is the maximum, the shallowest of them will be preselected). -[If there is a wakeup latency constraint coming from the `PM QoS framework -<cpu-pm-qos_>`_ which is hit before reaching the deepest idle state with the -target residency within the sleep length, the deepest idle state with the exit -latency within the constraint is preselected without consulting the ``hits``, -``misses`` and ``early_hits`` metrics.] - -Next, the governor takes several idle duration values observed most recently -into consideration and if at least a half of them are greater than or equal to -the target residency of the preselected idle state, that idle state becomes the -final candidate to ask for. Otherwise, the average of the most recent idle -duration values below the target residency of the preselected idle state is -computed and the governor walks the idle states shallower than the preselected -one and finds the deepest of them with the target residency within that average. -That idle state is then taken as the final candidate to ask for. - -Still, at this point the governor may need to refine the idle state selection if -it has not decided to `stop the scheduler tick <idle-cpus-and-tick_>`_. That -generally happens if the target residency of the idle state selected so far is -less than the tick period and the tick has not been stopped already (in a -previous iteration of the idle loop). Then, like in the ``menu`` governor -`case <menu-gov_>`_, the sleep length used in the previous computations may not -reflect the real time until the closest timer event and if it really is greater -than that time, a shallower state with a suitable target residency may need to -be selected. - +.. kernel-doc:: drivers/cpuidle/governors/teo.c + :doc: teo-description .. _idle-states-representation: diff --git a/Documentation/admin-guide/pm/intel_idle.rst b/Documentation/admin-guide/pm/intel_idle.rst index 89309e1b0e48..b799a43da62e 100644 --- a/Documentation/admin-guide/pm/intel_idle.rst +++ b/Documentation/admin-guide/pm/intel_idle.rst @@ -20,8 +20,8 @@ Nehalem and later generations of Intel processors, but the level of support for a particular processor model in it depends on whether or not it recognizes that processor model and may also depend on information coming from the platform firmware. [To understand ``intel_idle`` it is necessary to know how ``CPUIdle`` -works in general, so this is the time to get familiar with :doc:`cpuidle` if you -have not done that yet.] +works in general, so this is the time to get familiar with +Documentation/admin-guide/pm/cpuidle.rst if you have not done that yet.] ``intel_idle`` uses the ``MWAIT`` instruction to inform the processor that the logical CPU executing it is idle and so it may be possible to put some of the @@ -53,7 +53,8 @@ processor) corresponding to them depends on the processor model and it may also depend on the configuration of the platform. In order to create a list of available idle states required by the ``CPUIdle`` -subsystem (see :ref:`idle-states-representation` in :doc:`cpuidle`), +subsystem (see :ref:`idle-states-representation` in +Documentation/admin-guide/pm/cpuidle.rst), ``intel_idle`` can use two sources of information: static tables of idle states for different processor models included in the driver itself and the ACPI tables of the system. The former are always used if the processor model at hand is @@ -98,7 +99,8 @@ states may not be enabled by default if there are no matching entries in the preliminary list of idle states coming from the ACPI tables. In that case user space still can enable them later (on a per-CPU basis) with the help of the ``disable`` idle state attribute in ``sysfs`` (see -:ref:`idle-states-representation` in :doc:`cpuidle`). This basically means that +:ref:`idle-states-representation` in +Documentation/admin-guide/pm/cpuidle.rst). This basically means that the idle states "known" to the driver may not be enabled by default if they have not been exposed by the platform firmware (through the ACPI tables). @@ -186,7 +188,8 @@ be desirable. In practice, it is only really necessary to do that if the idle states in question cannot be enabled during system startup, because in the working state of the system the CPU power management quality of service (PM QoS) feature can be used to prevent ``CPUIdle`` from touching those idle states -even if they have been enumerated (see :ref:`cpu-pm-qos` in :doc:`cpuidle`). +even if they have been enumerated (see :ref:`cpu-pm-qos` in +Documentation/admin-guide/pm/cpuidle.rst). Setting ``max_cstate`` to 0 causes the ``intel_idle`` initialization to fail. The ``no_acpi`` and ``use_acpi`` module parameters (recognized by ``intel_idle`` @@ -202,7 +205,8 @@ Namely, the positions of the bits that are set in the ``states_off`` value are the indices of idle states to be disabled by default (as reflected by the names of the corresponding idle state directories in ``sysfs``, :file:`state0`, :file:`state1` ... :file:`state<i>` ..., where ``<i>`` is the index of the given -idle state; see :ref:`idle-states-representation` in :doc:`cpuidle`). +idle state; see :ref:`idle-states-representation` in +Documentation/admin-guide/pm/cpuidle.rst). For example, if ``states_off`` is equal to 3, the driver will disable idle states 0 and 1 by default, and if it is equal to 8, idle state 3 will be diff --git a/Documentation/admin-guide/pm/intel_pstate.rst b/Documentation/admin-guide/pm/intel_pstate.rst index df29b4f1f219..d5043cd8d2f5 100644 --- a/Documentation/admin-guide/pm/intel_pstate.rst +++ b/Documentation/admin-guide/pm/intel_pstate.rst @@ -18,8 +18,8 @@ General Information (``CPUFreq``). It is a scaling driver for the Sandy Bridge and later generations of Intel processors. Note, however, that some of those processors may not be supported. [To understand ``intel_pstate`` it is necessary to know -how ``CPUFreq`` works in general, so this is the time to read :doc:`cpufreq` if -you have not done that yet.] +how ``CPUFreq`` works in general, so this is the time to read +Documentation/admin-guide/pm/cpufreq.rst if you have not done that yet.] For the processors supported by ``intel_pstate``, the P-state concept is broader than just an operating frequency or an operating performance point (see the @@ -365,6 +365,9 @@ argument is passed to the kernel in the command line. inclusive) including both turbo and non-turbo P-states (see `Turbo P-states Support`_). + This attribute is present only if the value exposed by it is the same + for all of the CPUs in the system. + The value of this attribute is not affected by the ``no_turbo`` setting described `below <no_turbo_attr_>`_. @@ -374,6 +377,9 @@ argument is passed to the kernel in the command line. Ratio of the `turbo range <turbo_>`_ size to the size of the entire range of supported P-states, in percent. + This attribute is present only if the value exposed by it is the same + for all of the CPUs in the system. + This attribute is read-only. .. _no_turbo_attr: @@ -445,8 +451,9 @@ Interpretation of Policy Attributes ----------------------------------- The interpretation of some ``CPUFreq`` policy attributes described in -:doc:`cpufreq` is special with ``intel_pstate`` as the current scaling driver -and it generally depends on the driver's `operation mode <Operation Modes_>`_. +Documentation/admin-guide/pm/cpufreq.rst is special with ``intel_pstate`` +as the current scaling driver and it generally depends on the driver's +`operation mode <Operation Modes_>`_. First of all, the values of the ``cpuinfo_max_freq``, ``cpuinfo_min_freq`` and ``scaling_cur_freq`` attributes are produced by applying a processor-specific diff --git a/Documentation/admin-guide/pstore-blk.rst b/Documentation/admin-guide/pstore-blk.rst index 49d8149f8d32..2d22ead9520e 100644 --- a/Documentation/admin-guide/pstore-blk.rst +++ b/Documentation/admin-guide/pstore-blk.rst @@ -45,15 +45,18 @@ blkdev The block device to use. Most of the time, it is a partition of block device. It's required for pstore/blk. It is also used for MTD device. -It accepts the following variants for block device: +When pstore/blk is built as a module, "blkdev" accepts the following variants: -1. <hex_major><hex_minor> device number in hexadecimal represents itself; no - leading 0x, for example b302. -#. /dev/<disk_name> represents the device number of disk +1. /dev/<disk_name> represents the device number of disk #. /dev/<disk_name><decimal> represents the device number of partition - device number of disk plus the partition number #. /dev/<disk_name>p<decimal> - same as the above; this form is used when disk name of partitioned disk ends with a digit. + +When pstore/blk is built into the kernel, "blkdev" accepts the following variants: + +#. <hex_major><hex_minor> device number in hexadecimal representation, + with no leading 0x, for example b302. #. PARTUUID=00112233-4455-6677-8899-AABBCCDDEEFF represents the unique id of a partition if the partition table provides it. The UUID may be either an EFI/GPT UUID, or refer to an MSDOS partition using the format SSSSSSSS-PP, @@ -227,8 +230,5 @@ For developer reference, here are all the important structures and APIs: .. kernel-doc:: include/linux/pstore_zone.h :internal: -.. kernel-doc:: fs/pstore/blk.c - :internal: - .. kernel-doc:: include/linux/pstore_blk.h :internal: diff --git a/Documentation/admin-guide/reporting-issues.rst b/Documentation/admin-guide/reporting-issues.rst index 18d8e25ba9df..d7ac13f789cc 100644 --- a/Documentation/admin-guide/reporting-issues.rst +++ b/Documentation/admin-guide/reporting-issues.rst @@ -1248,7 +1248,7 @@ paragraph makes the severeness obvious. In case you performed a successful bisection, use the title of the change that introduced the regression as the second part of your subject. Make the report -also mention the commit id of the culprit. In case of an unsuccessful bisection, +also mention the commit id of the culprit. In case of an unsuccessful bisection, make your report mention the latest tested version that's working fine (say 5.7) and the oldest where the issue occurs (say 5.8-rc1). diff --git a/Documentation/admin-guide/sysctl/abi.rst b/Documentation/admin-guide/sysctl/abi.rst index 77b1d1b2ad42..4e6db0a2a4c0 100644 --- a/Documentation/admin-guide/sysctl/abi.rst +++ b/Documentation/admin-guide/sysctl/abi.rst @@ -11,7 +11,7 @@ Documentation for /proc/sys/abi/ Copyright (c) 2020, Stephen Kitt -For general info, see :doc:`index`. +For general info, see Documentation/admin-guide/sysctl/index.rst. ------------------------------------------------------------------------------ diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst index 68b21395a743..426162009ce9 100644 --- a/Documentation/admin-guide/sysctl/kernel.rst +++ b/Documentation/admin-guide/sysctl/kernel.rst @@ -9,7 +9,8 @@ Copyright (c) 1998, 1999, Rik van Riel <riel@nl.linux.org> Copyright (c) 2009, Shen Feng<shen@cn.fujitsu.com> -For general info and legal blurb, please look in :doc:`index`. +For general info and legal blurb, please look in +Documentation/admin-guide/sysctl/index.rst. ------------------------------------------------------------------------------ @@ -54,7 +55,7 @@ free space valid for 30 seconds. acpi_video_flags ================ -See :doc:`/power/video`. This allows the video resume mode to be set, +See Documentation/power/video.rst. This allows the video resume mode to be set, in a similar fashion to the ``acpi_sleep`` kernel parameter, by combining the following values: @@ -89,7 +90,7 @@ is 0x15 and the full version number is 0x234, this file will contain the value 340 = 0x154. See the ``type_of_loader`` and ``ext_loader_type`` fields in -:doc:`/x86/boot` for additional information. +Documentation/x86/boot.rst for additional information. bootloader_version (x86 only) @@ -99,7 +100,7 @@ The complete bootloader version number. In the example above, this file will contain the value 564 = 0x234. See the ``type_of_loader`` and ``ext_loader_ver`` fields in -:doc:`/x86/boot` for additional information. +Documentation/x86/boot.rst for additional information. bpf_stats_enabled @@ -269,7 +270,7 @@ see the ``hostname(1)`` man page. firmware_config =============== -See :doc:`/driver-api/firmware/fallback-mechanisms`. +See Documentation/driver-api/firmware/fallback-mechanisms.rst. The entries in this directory allow the firmware loader helper fallback to be controlled: @@ -297,7 +298,7 @@ crashes and outputting them to a serial console. ftrace_enabled, stack_tracer_enabled ==================================== -See :doc:`/trace/ftrace`. +See Documentation/trace/ftrace.rst. hardlockup_all_cpu_backtrace @@ -325,7 +326,7 @@ when a hard lockup is detected. 1 Panic on hard lockup. = =========================== -See :doc:`/admin-guide/lockup-watchdogs` for more information. +See Documentation/admin-guide/lockup-watchdogs.rst for more information. This can also be set using the nmi_watchdog kernel parameter. @@ -333,7 +334,12 @@ hotplug ======= Path for the hotplug policy agent. -Default value is "``/sbin/hotplug``". +Default value is ``CONFIG_UEVENT_HELPER_PATH``, which in turn defaults +to the empty string. + +This file only exists when ``CONFIG_UEVENT_HELPER`` is enabled. Most +modern systems rely exclusively on the netlink-based uevent source and +don't need this. hung_task_all_cpu_backtrace @@ -582,7 +588,8 @@ in a KVM virtual machine. This default can be overridden by adding:: nmi_watchdog=1 -to the guest kernel command line (see :doc:`/admin-guide/kernel-parameters`). +to the guest kernel command line (see +Documentation/admin-guide/kernel-parameters.rst). numa_balancing @@ -1067,7 +1074,7 @@ that support this feature. real-root-dev ============= -See :doc:`/admin-guide/initrd`. +See Documentation/admin-guide/initrd.rst. reboot-cmd (SPARC only) @@ -1088,6 +1095,13 @@ Model available). If your platform happens to meet the requirements for EAS but you do not want to use it, change this value to 0. +task_delayacct +=============== + +Enables/disables task delay accounting (see +:doc:`accounting/delay-accounting.rst`). Enabling this feature incurs +a small amount of overhead in the scheduler but is useful for debugging +and performance tuning. It is required by some tools such as iotop. sched_schedstats ================ @@ -1154,7 +1168,7 @@ will take effect. seccomp ======= -See :doc:`/userspace-api/seccomp_filter`. +See Documentation/userspace-api/seccomp_filter.rst. sg-big-buff @@ -1283,11 +1297,11 @@ This parameter can be used to control the soft lockup detector. = ================================= The soft lockup detector monitors CPUs for threads that are hogging the CPUs -without rescheduling voluntarily, and thus prevent the 'watchdog/N' threads -from running. The mechanism depends on the CPUs ability to respond to timer -interrupts which are needed for the 'watchdog/N' threads to be woken up by -the watchdog timer function, otherwise the NMI watchdog — if enabled — can -detect a hard lockup condition. +without rescheduling voluntarily, and thus prevent the 'migration/N' threads +from running, causing the watchdog work fail to execute. The mechanism depends +on the CPUs ability to respond to timer interrupts which are needed for the +watchdog work to be queued by the watchdog timer function, otherwise the NMI +watchdog — if enabled — can detect a hard lockup condition. stack_erasing @@ -1325,7 +1339,7 @@ the boot PROM. sysrq ===== -See :doc:`/admin-guide/sysrq`. +See Documentation/admin-guide/sysrq.rst. tainted @@ -1355,15 +1369,16 @@ ORed together. The letters are seen in "Tainted" line of Oops reports. 131072 `(T)` The kernel was built with the struct randomization plugin ====== ===== ============================================================== -See :doc:`/admin-guide/tainted-kernels` for more information. +See Documentation/admin-guide/tainted-kernels.rst for more information. Note: writes to this sysctl interface will fail with ``EINVAL`` if the kernel is booted with the command line option ``panic_on_taint=<bitmask>,nousertaint`` and any of the ORed together values being written to ``tainted`` match with the bitmask declared on panic_on_taint. - See :doc:`/admin-guide/kernel-parameters` for more details on that particular - kernel command line option and its optional ``nousertaint`` switch. + See Documentation/admin-guide/kernel-parameters.rst for more details on + that particular kernel command line option and its optional + ``nousertaint`` switch. threads-max =========== @@ -1387,7 +1402,7 @@ If a value outside of this range is written to ``threads-max`` an traceoff_on_warning =================== -When set, disables tracing (see :doc:`/trace/ftrace`) when a +When set, disables tracing (see Documentation/trace/ftrace.rst) when a ``WARN()`` is hit. @@ -1407,8 +1422,8 @@ will send them to printk() again. This only works if the kernel was booted with ``tp_printk`` enabled. -See :doc:`/admin-guide/kernel-parameters` and -:doc:`/trace/boottime-trace`. +See Documentation/admin-guide/kernel-parameters.rst and +Documentation/trace/boottime-trace.rst. .. _unaligned-dump-stack: diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index 586cd4b86428..5e795202111f 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -25,7 +25,6 @@ files can be found in mm/swap.c. Currently, these files are in /proc/sys/vm: - admin_reserve_kbytes -- block_dump - compact_memory - compaction_proactiveness - compact_unevictable_allowed @@ -64,7 +63,7 @@ Currently, these files are in /proc/sys/vm: - overcommit_ratio - page-cluster - panic_on_oom -- percpu_pagelist_fraction +- percpu_pagelist_high_fraction - stat_interval - stat_refresh - numa_stat @@ -106,13 +105,6 @@ On x86_64 this is about 128MB. Changing this takes effect whenever an application requests memory. -block_dump -========== - -block_dump enables block I/O debugging when set to a nonzero value. More -information on block I/O debugging is in Documentation/admin-guide/laptops/laptop-mode.rst. - - compact_memory ============== @@ -126,7 +118,8 @@ compaction_proactiveness This tunable takes a value in the range [0, 100] with a default value of 20. This tunable determines how aggressively compaction is done in the -background. Setting it to 0 disables proactive compaction. +background. Write of a non zero value to this tunable will immediately +trigger the proactive compaction. Setting it to 0 disables proactive compaction. Note that compaction has a non-trivial system-wide impact as pages belonging to different processes are moved around, which could also lead @@ -790,22 +783,24 @@ panic_on_oom=2+kdump gives you very strong tool to investigate why oom happens. You can get snapshot. -percpu_pagelist_fraction -======================== +percpu_pagelist_high_fraction +============================= -This is the fraction of pages at most (high mark pcp->high) in each zone that -are allocated for each per cpu page list. The min value for this is 8. It -means that we don't allow more than 1/8th of pages in each zone to be -allocated in any single per_cpu_pagelist. This entry only changes the value -of hot per cpu pagelists. User can specify a number like 100 to allocate -1/100th of each zone to each per cpu page list. +This is the fraction of pages in each zone that are can be stored to +per-cpu page lists. It is an upper boundary that is divided depending +on the number of online CPUs. The min value for this is 8 which means +that we do not allow more than 1/8th of pages in each zone to be stored +on per-cpu page lists. This entry only changes the value of hot per-cpu +page lists. A user can specify a number like 100 to allocate 1/100th of +each zone between per-cpu lists. -The batch value of each per cpu pagelist is also updated as a result. It is -set to pcp->high/4. The upper limit of batch is (PAGE_SHIFT * 8) +The batch value of each per-cpu page list remains the same regardless of +the value of the high fraction so allocation latencies are unaffected. -The initial value is zero. Kernel does not use this value at boot time to set -the high water marks for each per cpu page list. If the user writes '0' to this -sysctl, it will revert to this default behavior. +The initial value is zero. Kernel uses this value to set the high pcp->high +mark based on the low watermark for the zone and the number of local +online CPUs. If the user writes '0' to this sysctl, it will revert to +this default behavior. stat_interval @@ -936,12 +931,12 @@ allocations, THP and hugetlbfs pages. To make it sensible with respect to the watermark_scale_factor parameter, the unit is in fractions of 10,000. The default value of -15,000 on !DISCONTIGMEM configurations means that up to 150% of the high -watermark will be reclaimed in the event of a pageblock being mixed due -to fragmentation. The level of reclaim is determined by the number of -fragmentation events that occurred in the recent past. If this value is -smaller than a pageblock then a pageblocks worth of pages will be reclaimed -(e.g. 2MB on 64-bit x86). A boost factor of 0 will disable the feature. +15,000 means that up to 150% of the high watermark will be reclaimed in the +event of a pageblock being mixed due to fragmentation. The level of reclaim +is determined by the number of fragmentation events that occurred in the +recent past. If this value is smaller than a pageblock then a pageblocks +worth of pages will be reclaimed (e.g. 2MB on 64-bit x86). A boost factor +of 0 will disable the feature. watermark_scale_factor diff --git a/Documentation/admin-guide/sysrq.rst b/Documentation/admin-guide/sysrq.rst index 60ce5f5ebab6..0a178ef0111d 100644 --- a/Documentation/admin-guide/sysrq.rst +++ b/Documentation/admin-guide/sysrq.rst @@ -72,7 +72,7 @@ On PowerPC On other If you know of the key combos for other architectures, please - let me know so I can add them to this section. + submit a patch to be included in this section. On all Write a character to /proc/sysrq-trigger. e.g.:: @@ -205,10 +205,12 @@ frozen (probably root) filesystem via the FIFREEZE ioctl. Sometimes SysRq seems to get 'stuck' after using it, what can I do? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -That happens to me, also. I've found that tapping shift, alt, and control -on both sides of the keyboard, and hitting an invalid sysrq sequence again -will fix the problem. (i.e., something like :kbd:`alt-sysrq-z`). Switching to -another virtual console (:kbd:`ALT+Fn`) and then back again should also help. +When this happens, try tapping shift, alt and control on both sides of the +keyboard, and hitting an invalid sysrq sequence again. (i.e., something like +:kbd:`alt-sysrq-z`). + +Switching to another virtual console (:kbd:`ALT+Fn`) and then back again +should also help. I hit SysRq, but nothing seems to happen, what's wrong? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/Documentation/admin-guide/thunderbolt.rst b/Documentation/admin-guide/thunderbolt.rst index f18e881373c4..2ed79f41a411 100644 --- a/Documentation/admin-guide/thunderbolt.rst +++ b/Documentation/admin-guide/thunderbolt.rst @@ -256,6 +256,35 @@ Note names of the NVMem devices ``nvm_activeN`` and ``nvm_non_activeN`` depend on the order they are registered in the NVMem subsystem. N in the name is the identifier added by the NVMem subsystem. +Upgrading on-board retimer NVM when there is no cable connected +--------------------------------------------------------------- +If the platform supports, it may be possible to upgrade the retimer NVM +firmware even when there is nothing connected to the USB4 +ports. When this is the case the ``usb4_portX`` devices have two special +attributes: ``offline`` and ``rescan``. The way to upgrade the firmware +is to first put the USB4 port into offline mode:: + + # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline + +This step makes sure the port does not respond to any hotplug events, +and also ensures the retimers are powered on. The next step is to scan +for the retimers:: + + # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan + +This enumerates and adds the on-board retimers. Now retimer NVM can be +upgraded in the same way than with cable connected (see previous +section). However, the retimer is not disconnected as we are offline +mode) so after writing ``1`` to ``nvm_authenticate`` one should wait for +5 or more seconds before running rescan again:: + + # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan + +This point if everything went fine, the port can be put back to +functional state again:: + + # echo 0 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline + Upgrading NVM when host controller is in safe mode -------------------------------------------------- If the existing NVM is not properly authenticated (or is missing) the |