summaryrefslogtreecommitdiff
path: root/Documentation/xz.txt
diff options
context:
space:
mode:
authorLinus Torvalds <torvalds@linux-foundation.org>2017-07-15 22:58:58 +0300
committerLinus Torvalds <torvalds@linux-foundation.org>2017-07-15 22:58:58 +0300
commit486088bc4689f826b80aa317b45ac9e42e8b25ee (patch)
treeadf5847a6119d24da990d9e336f005c4a316e6be /Documentation/xz.txt
parent52f6c588c77b76d548201470c2a28263a41b462b (diff)
parent43e5f7e1fa66531777c49791014c3124ea9208d8 (diff)
downloadlinux-486088bc4689f826b80aa317b45ac9e42e8b25ee.tar.xz
Merge tag 'standardize-docs' of git://git.lwn.net/linux
Pull documentation format standardization from Jonathan Corbet: "This series converts a number of top-level documents to the RST format without incorporating them into the Sphinx tree. The hope is to bring some uniformity to kernel documentation and, perhaps more importantly, have our existing docs serve as an example of the desired formatting for those that will be added later. Mauro has gone through and fixed up a lot of top-level documentation files to make them conform to the RST format, but without moving or renaming them in any way. This will help when we incorporate the ones we want to keep into the Sphinx doctree, but the real purpose is to bring a bit of uniformity to our documentation and let the top-level docs serve as examples for those writing new ones" * tag 'standardize-docs' of git://git.lwn.net/linux: (84 commits) docs: kprobes.txt: Fix whitespacing tee.txt: standardize document format cgroup-v2.txt: standardize document format dell_rbu.txt: standardize document format zorro.txt: standardize document format xz.txt: standardize document format xillybus.txt: standardize document format vfio.txt: standardize document format vfio-mediated-device.txt: standardize document format unaligned-memory-access.txt: standardize document format this_cpu_ops.txt: standardize document format svga.txt: standardize document format static-keys.txt: standardize document format smsc_ece1099.txt: standardize document format SM501.txt: standardize document format siphash.txt: standardize document format sgi-ioc4.txt: standardize document format SAK.txt: standardize document format rpmsg.txt: standardize document format robust-futexes.txt: standardize document format ...
Diffstat (limited to 'Documentation/xz.txt')
-rw-r--r--Documentation/xz.txt200
1 files changed, 103 insertions, 97 deletions
diff --git a/Documentation/xz.txt b/Documentation/xz.txt
index 2cf3e2608de3..b2220d03aa50 100644
--- a/Documentation/xz.txt
+++ b/Documentation/xz.txt
@@ -1,121 +1,127 @@
-
+============================
XZ data compression in Linux
============================
Introduction
+============
- XZ is a general purpose data compression format with high compression
- ratio and relatively fast decompression. The primary compression
- algorithm (filter) is LZMA2. Additional filters can be used to improve
- compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters
- improve compression ratio of executable data.
+XZ is a general purpose data compression format with high compression
+ratio and relatively fast decompression. The primary compression
+algorithm (filter) is LZMA2. Additional filters can be used to improve
+compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters
+improve compression ratio of executable data.
- The XZ decompressor in Linux is called XZ Embedded. It supports
- the LZMA2 filter and optionally also BCJ filters. CRC32 is supported
- for integrity checking. The home page of XZ Embedded is at
- <http://tukaani.org/xz/embedded.html>, where you can find the
- latest version and also information about using the code outside
- the Linux kernel.
+The XZ decompressor in Linux is called XZ Embedded. It supports
+the LZMA2 filter and optionally also BCJ filters. CRC32 is supported
+for integrity checking. The home page of XZ Embedded is at
+<http://tukaani.org/xz/embedded.html>, where you can find the
+latest version and also information about using the code outside
+the Linux kernel.
- For userspace, XZ Utils provide a zlib-like compression library
- and a gzip-like command line tool. XZ Utils can be downloaded from
- <http://tukaani.org/xz/>.
+For userspace, XZ Utils provide a zlib-like compression library
+and a gzip-like command line tool. XZ Utils can be downloaded from
+<http://tukaani.org/xz/>.
XZ related components in the kernel
-
- The xz_dec module provides XZ decompressor with single-call (buffer
- to buffer) and multi-call (stateful) APIs. The usage of the xz_dec
- module is documented in include/linux/xz.h.
-
- The xz_dec_test module is for testing xz_dec. xz_dec_test is not
- useful unless you are hacking the XZ decompressor. xz_dec_test
- allocates a char device major dynamically to which one can write
- .xz files from userspace. The decompressed output is thrown away.
- Keep an eye on dmesg to see diagnostics printed by xz_dec_test.
- See the xz_dec_test source code for the details.
-
- For decompressing the kernel image, initramfs, and initrd, there
- is a wrapper function in lib/decompress_unxz.c. Its API is the
- same as in other decompress_*.c files, which is defined in
- include/linux/decompress/generic.h.
-
- scripts/xz_wrap.sh is a wrapper for the xz command line tool found
- from XZ Utils. The wrapper sets compression options to values suitable
- for compressing the kernel image.
-
- For kernel makefiles, two commands are provided for use with
- $(call if_needed). The kernel image should be compressed with
- $(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2
- dictionary. It will also append a four-byte trailer containing the
- uncompressed size of the file, which is needed by the boot code.
- Other things should be compressed with $(call if_needed,xzmisc)
- which will use no BCJ filter and 1 MiB LZMA2 dictionary.
+===================================
+
+The xz_dec module provides XZ decompressor with single-call (buffer
+to buffer) and multi-call (stateful) APIs. The usage of the xz_dec
+module is documented in include/linux/xz.h.
+
+The xz_dec_test module is for testing xz_dec. xz_dec_test is not
+useful unless you are hacking the XZ decompressor. xz_dec_test
+allocates a char device major dynamically to which one can write
+.xz files from userspace. The decompressed output is thrown away.
+Keep an eye on dmesg to see diagnostics printed by xz_dec_test.
+See the xz_dec_test source code for the details.
+
+For decompressing the kernel image, initramfs, and initrd, there
+is a wrapper function in lib/decompress_unxz.c. Its API is the
+same as in other decompress_*.c files, which is defined in
+include/linux/decompress/generic.h.
+
+scripts/xz_wrap.sh is a wrapper for the xz command line tool found
+from XZ Utils. The wrapper sets compression options to values suitable
+for compressing the kernel image.
+
+For kernel makefiles, two commands are provided for use with
+$(call if_needed). The kernel image should be compressed with
+$(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2
+dictionary. It will also append a four-byte trailer containing the
+uncompressed size of the file, which is needed by the boot code.
+Other things should be compressed with $(call if_needed,xzmisc)
+which will use no BCJ filter and 1 MiB LZMA2 dictionary.
Notes on compression options
+============================
- Since the XZ Embedded supports only streams with no integrity check or
- CRC32, make sure that you don't use some other integrity check type
- when encoding files that are supposed to be decoded by the kernel. With
- liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32
- when encoding. With the xz command line tool, use --check=none or
- --check=crc32.
-
- Using CRC32 is strongly recommended unless there is some other layer
- which will verify the integrity of the uncompressed data anyway.
- Double checking the integrity would probably be waste of CPU cycles.
- Note that the headers will always have a CRC32 which will be validated
- by the decoder; you can only change the integrity check type (or
- disable it) for the actual uncompressed data.
-
- In userspace, LZMA2 is typically used with dictionary sizes of several
- megabytes. The decoder needs to have the dictionary in RAM, thus big
- dictionaries cannot be used for files that are intended to be decoded
- by the kernel. 1 MiB is probably the maximum reasonable dictionary
- size for in-kernel use (maybe more is OK for initramfs). The presets
- in XZ Utils may not be optimal when creating files for the kernel,
- so don't hesitate to use custom settings. Example:
-
- xz --check=crc32 --lzma2=dict=512KiB inputfile
-
- An exception to above dictionary size limitation is when the decoder
- is used in single-call mode. Decompressing the kernel itself is an
- example of this situation. In single-call mode, the memory usage
- doesn't depend on the dictionary size, and it is perfectly fine to
- use a big dictionary: for maximum compression, the dictionary should
- be at least as big as the uncompressed data itself.
+Since the XZ Embedded supports only streams with no integrity check or
+CRC32, make sure that you don't use some other integrity check type
+when encoding files that are supposed to be decoded by the kernel. With
+liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32
+when encoding. With the xz command line tool, use --check=none or
+--check=crc32.
+
+Using CRC32 is strongly recommended unless there is some other layer
+which will verify the integrity of the uncompressed data anyway.
+Double checking the integrity would probably be waste of CPU cycles.
+Note that the headers will always have a CRC32 which will be validated
+by the decoder; you can only change the integrity check type (or
+disable it) for the actual uncompressed data.
+
+In userspace, LZMA2 is typically used with dictionary sizes of several
+megabytes. The decoder needs to have the dictionary in RAM, thus big
+dictionaries cannot be used for files that are intended to be decoded
+by the kernel. 1 MiB is probably the maximum reasonable dictionary
+size for in-kernel use (maybe more is OK for initramfs). The presets
+in XZ Utils may not be optimal when creating files for the kernel,
+so don't hesitate to use custom settings. Example::
+
+ xz --check=crc32 --lzma2=dict=512KiB inputfile
+
+An exception to above dictionary size limitation is when the decoder
+is used in single-call mode. Decompressing the kernel itself is an
+example of this situation. In single-call mode, the memory usage
+doesn't depend on the dictionary size, and it is perfectly fine to
+use a big dictionary: for maximum compression, the dictionary should
+be at least as big as the uncompressed data itself.
Future plans
+============
- Creating a limited XZ encoder may be considered if people think it is
- useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at
- the fastest settings, so it isn't clear if LZMA2 encoder is wanted
- into the kernel.
+Creating a limited XZ encoder may be considered if people think it is
+useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at
+the fastest settings, so it isn't clear if LZMA2 encoder is wanted
+into the kernel.
- Support for limited random-access reading is planned for the
- decompression code. I don't know if it could have any use in the
- kernel, but I know that it would be useful in some embedded projects
- outside the Linux kernel.
+Support for limited random-access reading is planned for the
+decompression code. I don't know if it could have any use in the
+kernel, but I know that it would be useful in some embedded projects
+outside the Linux kernel.
Conformance to the .xz file format specification
+================================================
- There are a couple of corner cases where things have been simplified
- at expense of detecting errors as early as possible. These should not
- matter in practice all, since they don't cause security issues. But
- it is good to know this if testing the code e.g. with the test files
- from XZ Utils.
+There are a couple of corner cases where things have been simplified
+at expense of detecting errors as early as possible. These should not
+matter in practice all, since they don't cause security issues. But
+it is good to know this if testing the code e.g. with the test files
+from XZ Utils.
Reporting bugs
+==============
- Before reporting a bug, please check that it's not fixed already
- at upstream. See <http://tukaani.org/xz/embedded.html> to get the
- latest code.
+Before reporting a bug, please check that it's not fixed already
+at upstream. See <http://tukaani.org/xz/embedded.html> to get the
+latest code.
- Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on
- Freenode and talk to Larhzu. I don't actively read LKML or other
- kernel-related mailing lists, so if there's something I should know,
- you should email to me personally or use IRC.
+Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on
+Freenode and talk to Larhzu. I don't actively read LKML or other
+kernel-related mailing lists, so if there's something I should know,
+you should email to me personally or use IRC.
- Don't bother Igor Pavlov with questions about the XZ implementation
- in the kernel or about XZ Utils. While these two implementations
- include essential code that is directly based on Igor Pavlov's code,
- these implementations aren't maintained nor supported by him.
+Don't bother Igor Pavlov with questions about the XZ implementation
+in the kernel or about XZ Utils. While these two implementations
+include essential code that is directly based on Igor Pavlov's code,
+these implementations aren't maintained nor supported by him.