diff options
Diffstat (limited to 'import-layers/yocto-poky/documentation/dev-manual/dev-manual-common-tasks.xml')
| -rw-r--r-- | import-layers/yocto-poky/documentation/dev-manual/dev-manual-common-tasks.xml | 8508 |
1 files changed, 6533 insertions, 1975 deletions
diff --git a/import-layers/yocto-poky/documentation/dev-manual/dev-manual-common-tasks.xml b/import-layers/yocto-poky/documentation/dev-manual/dev-manual-common-tasks.xml index 0081738087..fe1bfba6cf 100644 --- a/import-layers/yocto-poky/documentation/dev-manual/dev-manual-common-tasks.xml +++ b/import-layers/yocto-poky/documentation/dev-manual/dev-manual-common-tasks.xml @@ -22,90 +22,24 @@ multiple layers. Layers allow you to isolate different types of customizations from each other. - You might find it tempting to keep everything in one layer when - working on a single project. - However, the more modular your Metadata, the easier - it is to cope with future changes. - </para> - - <para> - To illustrate how layers are used to keep things modular, consider - machine customizations. - These types of customizations typically reside in a special layer, - rather than a general layer, called a Board Support Package (BSP) - Layer. - Furthermore, the machine customizations should be isolated from - recipes and Metadata that support a new GUI environment, - for example. - This situation gives you a couple of layers: one for the machine - configurations, and one for the GUI environment. - It is important to understand, however, that the BSP layer can - still make machine-specific additions to recipes within the GUI - environment layer without polluting the GUI layer itself - with those machine-specific changes. - You can accomplish this through a recipe that is a BitBake append - (<filename>.bbappend</filename>) file, which is described later - in this section. - <note> - For general information on BSP layer structure, see the - <ulink url='&YOCTO_DOCS_BSP_URL;#bsp'>Board Support Packages (BSP) - Developer's Guide</ulink>. - </note> - </para> - - <para> + For introductory information on the Yocto Project Layer Model, + see the + "<ulink url='&YOCTO_DOCS_OM_URL;#the-yocto-project-layer-model'>The Yocto Project Layer Model</ulink>" + section in the Yocto Project Overview and Concepts Manual. </para> - <section id='yocto-project-layers'> - <title>Layers</title> - - <para> - The <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> - contains both general layers and BSP - layers right out of the box. - You can easily identify layers that ship with a - Yocto Project release in the Source Directory by their - folder names. - Folders that represent layers typically have names that begin with - the string <filename>meta-</filename>. - <note> - It is not a requirement that a layer name begin with the - prefix <filename>meta-</filename>, but it is a commonly - accepted standard in the Yocto Project community. - </note> - For example, when you set up the Source Directory structure, - you will see several layers: - <filename>meta</filename>, - <filename>meta-skeleton</filename>, - <filename>meta-selftest</filename>, - <filename>meta-poky</filename>, and - <filename>meta-yocto-bsp</filename>. - Each of these folders represents a distinct layer. - </para> - - <para> - As another example, if you set up a local copy of the - <filename>meta-intel</filename> Git repository - and then explore the folder of that general layer, - you will discover many Intel-specific BSP layers inside. - For more information on BSP layers, see the - "<ulink url='&YOCTO_DOCS_BSP_URL;#bsp-layers'>BSP Layers</ulink>" - section in the Yocto Project Board Support Package (BSP) - Developer's Guide. - </para> - </section> - <section id='creating-your-own-layer'> <title>Creating Your Own Layer</title> <para> It is very easy to create your own layers to use with the OpenEmbedded build system. - The Yocto Project ships with scripts that speed up creating - general layers and BSP layers. + The Yocto Project ships with tools that speed up creating + layers. This section describes the steps you perform by hand to create - a layer so that you can better understand them. - For information about the layer-creation scripts, see the - "<ulink url='&YOCTO_DOCS_BSP_URL;#creating-a-new-bsp-layer-using-the-yocto-bsp-script'>Creating a New BSP Layer Using the yocto-bsp Script</ulink>" + layers so that you can better understand them. + For information about the layer-creation tools, see the + "<ulink url='&YOCTO_DOCS_BSP_URL;#creating-a-new-bsp-layer-using-the-bitbake-layers-script'>Creating a New BSP Layer Using the <filename>bitbake-layers</filename> Script</ulink>" section in the Yocto Project Board Support Package (BSP) Developer's Guide and the "<link linkend='creating-a-general-layer-using-the-bitbake-layers-script'>Creating a General Layer Using the <filename>bitbake-layers</filename> Script</link>" @@ -113,40 +47,70 @@ </para> <para> - Follow these general steps to create your layer without the aid of a script: + Follow these general steps to create your layer without using + tools: <orderedlist> - <listitem><para><emphasis>Check Existing Layers:</emphasis> + <listitem><para> + <emphasis>Check Existing Layers:</emphasis> Before creating a new layer, you should be sure someone has not already created a layer containing the Metadata you need. You can see the - <ulink url='http://layers.openembedded.org/layerindex/layers/'><filename>OpenEmbedded Metadata Index</filename></ulink> + <ulink url='http://layers.openembedded.org/layerindex/layers/'>OpenEmbedded Metadata Index</ulink> for a list of layers from the OpenEmbedded community that can be used in the Yocto Project. + You could find a layer that is identical or close to + what you need. </para></listitem> - <listitem><para><emphasis>Create a Directory:</emphasis> + <listitem><para> + <emphasis>Create a Directory:</emphasis> Create the directory for your layer. - While not strictly required, prepend the name of the - folder with the string <filename>meta-</filename>. + When you create the layer, be sure to create the + directory in an area not associated with the + Yocto Project + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + (e.g. the cloned <filename>poky</filename> repository). + </para> + + <para>While not strictly required, prepend the name of + the directory with the string "meta-". For example: <literallayout class='monospaced'> meta-mylayer meta-GUI_xyz meta-mymachine </literallayout> + With rare exceptions, a layer's name follows this + form: + <literallayout class='monospaced'> + meta-<replaceable>root_name</replaceable> + </literallayout> + Following this layer naming convention can + save you trouble later when tools, components, or + variables "assume" your layer name begins with "meta-". + A notable example is in configuration files as + shown in the following step where layer names without + the "meta-" string are appended + to several variables used in the configuration. </para></listitem> - <listitem><para><emphasis>Create a Layer Configuration - File:</emphasis> - Inside your new layer folder, you need to create a - <filename>conf/layer.conf</filename> file. - It is easiest to take an existing layer configuration - file and copy that to your layer's - <filename>conf</filename> directory and then modify the - file as needed.</para> - <para>The - <filename>meta-yocto-bsp/conf/layer.conf</filename> file - demonstrates the required syntax: - <literallayout class='monospaced'> + <listitem><para id='dev-layer-config-file-description'> + <emphasis>Create a Layer Configuration File:</emphasis> + Inside your new layer folder, you need to create a + <filename>conf/layer.conf</filename> file. + It is easiest to take an existing layer configuration + file and copy that to your layer's + <filename>conf</filename> directory and then modify the + file as needed.</para> + + <para>The + <filename>meta-yocto-bsp/conf/layer.conf</filename> file + in the Yocto Project + <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-yocto-bsp/conf'>Source Repositories</ulink> + demonstrates the required syntax. + For your layer, you need to replace "yoctobsp" with + a unique identifier for your layer (e.g. "machinexyz" + for a layer named "meta-machinexyz"): + <literallayout class='monospaced'> # We have a conf and classes directory, add to BBPATH BBPATH .= ":${LAYERDIR}" @@ -157,75 +121,82 @@ BBFILE_COLLECTIONS += "yoctobsp" BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/" BBFILE_PRIORITY_yoctobsp = "5" - LAYERVERSION_yoctobsp = "3" - </literallayout></para> - <para>Here is an explanation of the example: + LAYERVERSION_yoctobsp = "4" + LAYERSERIES_COMPAT_yoctobsp = "&DISTRO_NAME_NO_CAP;" + </literallayout> + Following is an explanation of the layer configuration + file: <itemizedlist> - <listitem><para>The configuration and - classes directory is appended to - <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>. - <note> - All non-distro layers, which include all BSP - layers, are expected to append the layer - directory to the - <filename>BBPATH</filename>. - On the other hand, distro layers, such as - <filename>meta-poky</filename>, can choose - to enforce their own precedence over - <filename>BBPATH</filename>. - For an example of that syntax, see the - <filename>layer.conf</filename> file for - the <filename>meta-poky</filename> layer. - </note></para></listitem> - <listitem><para>The recipes for the layers are - appended to - <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILES'>BBFILES</ulink></filename>. - </para></listitem> - <listitem><para>The - <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</ulink></filename> - variable is then appended with the layer name. + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>: + Adds the layer's root directory to BitBake's + search path. + Through the use of the + <filename>BBPATH</filename> variable, BitBake + locates class files + (<filename>.bbclass</filename>), + configuration files, and files that are + included with <filename>include</filename> and + <filename>require</filename> statements. + For these cases, BitBake uses the first file + that matches the name found in + <filename>BBPATH</filename>. + This is similar to the way the + <filename>PATH</filename> variable is used for + binaries. + It is recommended, therefore, that you use + unique class and configuration filenames in + your custom layer. </para></listitem> - <listitem><para>The - <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_PATTERN'>BBFILE_PATTERN</ulink></filename> - variable is set to a regular expression and is - used to match files from - <filename>BBFILES</filename> into a particular + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILES'><filename>BBFILES</filename></ulink>: + Defines the location for all recipes in the layer. - In this case, - <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-LAYERDIR'>LAYERDIR</ulink></filename> - is used to make <filename>BBFILE_PATTERN</filename> match within the - layer's path.</para></listitem> - <listitem><para>The - <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_PRIORITY'>BBFILE_PRIORITY</ulink></filename> - variable then assigns a priority to the layer. - Applying priorities is useful in situations - where the same recipe might appear in multiple - layers and allows you to choose the layer - that takes precedence.</para></listitem> - <listitem><para>The - <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-LAYERVERSION'>LAYERVERSION</ulink></filename> - variable optionally specifies the version of a - layer as a single number.</para></listitem> - </itemizedlist></para> - <para>Note the use of the - <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-LAYERDIR'>LAYERDIR</ulink></filename> - variable, which expands to the directory of the current - layer.</para> - <para>Through the use of the <filename>BBPATH</filename> - variable, BitBake locates class files - (<filename>.bbclass</filename>), - configuration files, and files that are included - with <filename>include</filename> and - <filename>require</filename> statements. - For these cases, BitBake uses the first file that - matches the name found in <filename>BBPATH</filename>. - This is similar to the way the <filename>PATH</filename> - variable is used for binaries. - It is recommended, therefore, that you use unique - class and configuration - filenames in your custom layer.</para></listitem> - <listitem><para><emphasis>Add Content:</emphasis> Depending - on the type of layer, add the content. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_COLLECTIONS'><filename>BBFILE_COLLECTIONS</filename></ulink>: + Establishes the current layer through a + unique identifier that is used throughout the + OpenEmbedded build system to refer to the layer. + In this example, the identifier "yoctobsp" is + the representation for the container layer + named "meta-yocto-bsp". + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_PATTERN'><filename>BBFILE_PATTERN</filename></ulink>: + Expands immediately during parsing to + provide the directory of the layer. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_PRIORITY'><filename>BBFILE_PRIORITY</filename></ulink>: + Establishes a priority to use for + recipes in the layer when the OpenEmbedded build + finds recipes of the same name in different + layers. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-LAYERVERSION'><filename>LAYERVERSION</filename></ulink>: + Establishes a version number for the layer. + You can use this version number to specify this + exact version of the layer as a dependency when + using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-LAYERDEPENDS'><filename>LAYERDEPENDS</filename></ulink> + variable. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-LAYERSERIES_COMPAT'><filename>LAYERSERIES_COMPAT</filename></ulink>: + Lists the + <ulink url='&YOCTO_WIKI_URL;/wiki/Releases'>Yocto Project</ulink> + releases for which the current version is + compatible. + This variable is a good way to indicate how + up-to-date your particular layer is. + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <emphasis>Add Content:</emphasis> + Depending on the type of layer, add the content. If the layer adds support for a machine, add the machine configuration in a <filename>conf/machine/</filename> file within the layer. @@ -235,9 +206,13 @@ If the layer introduces new recipes, put the recipes you need in <filename>recipes-*</filename> subdirectories within the layer. - <note>In order to be compliant with the Yocto Project, - a layer must contain a - <ulink url='&YOCTO_DOCS_BSP_URL;#bsp-filelayout-readme'>README file.</ulink> + <note> + For an explanation of layer hierarchy that + is compliant with the Yocto Project, see + the + "<ulink url='&YOCTO_DOCS_BSP_URL;#bsp-filelayout'>Example Filesystem Layout</ulink>" + section in the Yocto Project Board + Support Package (BSP) Developer's Guide. </note> </para></listitem> <listitem><para> @@ -492,7 +467,7 @@ acceptable explanation for any questions answered "No". </para></listitem> <listitem><para> - You need to be a Yocto Project Member Organization. + Be a Yocto Project Member Organization. </para></listitem> </itemizedlist> </para> @@ -562,7 +537,7 @@ </para> <para> - The script divides tests into three areas: COMMON, BSD, + The script divides tests into three areas: COMMON, BSP, and DISTRO. For example, given a distribution layer (DISTRO), the layer must pass both the COMMON and DISTRO related tests. @@ -646,23 +621,26 @@ The following example shows how to enable a layer named <filename>meta-mylayer</filename>: <literallayout class='monospaced'> - LCONF_VERSION = "6" + # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf + # changes incompatibly + POKY_BBLAYERS_CONF_VERSION = "2" BBPATH = "${TOPDIR}" BBFILES ?= "" BBLAYERS ?= " \ - $HOME/poky/meta \ - $HOME/poky/meta-poky \ - $HOME/poky/meta-yocto-bsp \ - $HOME/poky/meta-mylayer \ + /home/<replaceable>user</replaceable>/poky/meta \ + /home/<replaceable>user</replaceable>/poky/meta-poky \ + /home/<replaceable>user</replaceable>/poky/meta-yocto-bsp \ + /home/<replaceable>user</replaceable>/poky/meta-mylayer \ " </literallayout> </para> <para> BitBake parses each <filename>conf/layer.conf</filename> file - as specified in the <filename>BBLAYERS</filename> variable + from the top down as specified in the + <filename>BBLAYERS</filename> variable within the <filename>conf/bblayers.conf</filename> file. During the processing of each <filename>conf/layer.conf</filename> file, BitBake adds the @@ -840,8 +818,7 @@ <para> To specify the layer's priority manually, use the <ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_PRIORITY'><filename>BBFILE_PRIORITY</filename></ulink> - variable. - For example: + variable and append the layer's root name: <literallayout class='monospaced'> BBFILE_PRIORITY_mylayer = "1" </literallayout> @@ -862,7 +839,8 @@ <title>Managing Layers</title> <para> - You can use the BitBake layer management tool to provide a view + You can use the BitBake layer management tool + <filename>bitbake-layers</filename> to provide a view into the structure of recipes across a multi-layer project. Being able to generate output that reports on configured layers with their paths and priorities and on @@ -871,42 +849,88 @@ </para> <para> - Use the following form when running the layer management tool. + For help on the BitBake layer management tool, use the + following command: <literallayout class='monospaced'> - $ bitbake-layers <replaceable>command</replaceable> [<replaceable>arguments</replaceable>] + $ bitbake-layers --help + NOTE: Starting bitbake server... + usage: bitbake-layers [-d] [-q] [-F] [--color COLOR] [-h] <subcommand> ... + + BitBake layers utility + + optional arguments: + -d, --debug Enable debug output + -q, --quiet Print only errors + -F, --force Force add without recipe parse verification + --color COLOR Colorize output (where COLOR is auto, always, never) + -h, --help show this help message and exit + + subcommands: + <subcommand> + show-layers show current configured layers. + show-overlayed list overlayed recipes (where the same recipe exists + in another layer) + show-recipes list available recipes, showing the layer they are + provided by + show-appends list bbappend files and recipe files they apply to + show-cross-depends Show dependencies between recipes that cross layer + boundaries. + add-layer Add one or more layers to bblayers.conf. + remove-layer Remove one or more layers from bblayers.conf. + flatten flatten layer configuration into a separate output + directory. + layerindex-fetch Fetches a layer from a layer index along with its + dependent layers, and adds them to conf/bblayers.conf. + layerindex-show-depends + Find layer dependencies from layer index. + create-layer Create a basic layer + + Use bitbake-layers <subcommand> --help to get help on a specific command </literallayout> + </para> + + <para> The following list describes the available commands: <itemizedlist> - <listitem><para><filename><emphasis>help:</emphasis></filename> + <listitem><para> + <emphasis><filename>help:</filename></emphasis> Displays general help or help on a specified command. </para></listitem> - <listitem><para><filename><emphasis>show-layers:</emphasis></filename> + <listitem><para> + <emphasis><filename>show-layers:</filename></emphasis> Shows the current configured layers. </para></listitem> - <listitem><para><filename><emphasis>show-recipes:</emphasis></filename> - Lists available recipes and the layers that provide them. - </para></listitem> - <listitem><para><filename><emphasis>show-overlayed:</emphasis></filename> + <listitem><para> + <emphasis><filename>show-overlayed:</filename></emphasis> Lists overlayed recipes. A recipe is overlayed when a recipe with the same name exists in another layer that has a higher layer priority. </para></listitem> - <listitem><para><filename><emphasis>show-appends:</emphasis></filename> + <listitem><para> + <emphasis><filename>show-recipes:</filename></emphasis> + Lists available recipes and the layers that provide them. + </para></listitem> + <listitem><para> + <emphasis><filename>show-appends:</filename></emphasis> Lists <filename>.bbappend</filename> files and the recipe files to which they apply. </para></listitem> - <listitem><para><filename><emphasis>show-cross-depends:</emphasis></filename> + <listitem><para> + <emphasis><filename>show-cross-depends:</filename></emphasis> Lists dependency relationships between recipes that cross layer boundaries. </para></listitem> - <listitem><para><filename><emphasis>add-layer:</emphasis></filename> + <listitem><para> + <emphasis><filename>add-layer:</filename></emphasis> Adds a layer to <filename>bblayers.conf</filename>. </para></listitem> - <listitem><para><filename><emphasis>remove-layer:</emphasis></filename> + <listitem><para> + <emphasis><filename>remove-layer:</filename></emphasis> Removes a layer from <filename>bblayers.conf</filename> </para></listitem> - <listitem><para><filename><emphasis>flatten:</emphasis></filename> + <listitem><para> + <emphasis><filename>flatten:</filename></emphasis> Flattens the layer configuration into a separate output directory. Flattening your layer configuration builds a "flattened" @@ -917,18 +941,21 @@ You might have to perform some manual cleanup of the flattened layer as follows: <itemizedlist> - <listitem><para>Non-recipe files (such as patches) + <listitem><para> + Non-recipe files (such as patches) are overwritten. The flatten command shows a warning for these files. </para></listitem> - <listitem><para>Anything beyond the normal layer + <listitem><para> + Anything beyond the normal layer setup has been added to the <filename>layer.conf</filename> file. Only the lowest priority layer's <filename>layer.conf</filename> is used. </para></listitem> - <listitem><para>Overridden and appended items from + <listitem><para> + Overridden and appended items from <filename>.bbappend</filename> files need to be cleaned up. The contents of each @@ -1000,13 +1027,13 @@ Developer's Guide. </para></listitem> <listitem><para> - The <filename>bitbake-layers</filename> script - replaces the <filename>yocto-layer</filename> - script, which is deprecated in the Yocto Project - 2.4 release. - The <filename>yocto-layer</filename> script - continues to function as part of the 2.4 release - but will be removed post 2.4. + In order to use a layer with the OpenEmbedded + build system, you need to add the layer to your + <filename>bblayers.conf</filename> configuration + file. + See the + "<link linkend='adding-a-layer-using-the-bitbake-layers-script'>Adding a Layer Using the <filename>bitbake-layers</filename> Script</link>" + section for more information. </para></listitem> </itemizedlist> </note> @@ -1047,6 +1074,14 @@ <literallayout class='monospaced'> $ bitbake-layers create-layer <replaceable>your_layer_name</replaceable> </literallayout> + As an example, the following command creates a layer named + <filename>meta-scottrif</filename> in your home directory: + <literallayout class='monospaced'> + $ cd /usr/home + $ bitbake-layers create-layer meta-scottrif + NOTE: Starting bitbake server... + Add your new layer with 'bitbake-layers add-layer meta-scottrif' + </literallayout> </para> <para> @@ -1089,26 +1124,36 @@ Filename of the example recipe </literallayout> </para> + </section> + + <section id='adding-a-layer-using-the-bitbake-layers-script'> + <title>Adding a Layer Using the <filename>bitbake-layers</filename> Script</title> <para> Once you create your general layer, you must add it to your <filename>bblayers.conf</filename> file. - You can add your layer by using the + Adding the layer to this configuration file makes the + OpenEmbedded build system aware of your layer so that it can + search it for metadata. + </para> + + <para> + Add your layer by using the <filename>bitbake-layers add-layer</filename> command: <literallayout class='monospaced'> $ bitbake-layers add-layer <replaceable>your_layer_name</replaceable> </literallayout> - Here is an example where a layer named - <filename>meta-scottrif</filename> is added and then the - layers are shown using the - <filename>bitbake-layers show-layers</filename> command: + Here is an example that adds a layer named + <filename>meta-scottrif</filename> to the configuration file. + Following the command that adds the layer is another + <filename>bitbake-layers</filename> command that shows the + layers that are in your <filename>bblayers.conf</filename> + file: <literallayout class='monospaced'> $ bitbake-layers add-layer meta-scottrif NOTE: Starting bitbake server... - Loading cache: 100% |############################################| Time: 0:00:00 - Loaded 1275 entries from dependency cache. - Parsing recipes: 100% |##########################################| Time: 0:00:00 - Parsing of 819 .bb files complete (817 cached, 2 parsed). 1276 targets, 44 skipped, 0 masked, 0 errors. + Parsing recipes: 100% |##########################################################| Time: 0:00:49 + Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors. $ bitbake-layers show-layers NOTE: Starting bitbake server... layer path priority @@ -1116,12 +1161,16 @@ meta /home/scottrif/poky/meta 5 meta-poky /home/scottrif/poky/meta-poky 5 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5 - meta-mylayer /home/scottrif/meta-mylayer 6 workspace /home/scottrif/poky/build/workspace 99 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6 </literallayout> Adding the layer to this file enables the build system to locate the layer during the build. + <note> + During a build, the OpenEmbedded build system looks in + the layers from the top of the list down to the bottom + in that order. + </note> </para> </section> </section> @@ -1222,8 +1271,8 @@ To understand how these features work, the best reference is <filename>meta/classes/core-image.bbclass</filename>. This class lists out the available - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink> - of which most map to package groups while some, such as + <filename>IMAGE_FEATURES</filename> of which most map to + package groups while some, such as <filename>debug-tweaks</filename> and <filename>read-only-rootfs</filename>, resolve as general configuration settings. @@ -1520,8 +1569,8 @@ </itemizedlist> <note> For information on recipe syntax, see the - "<ulink url='&YOCTO_DOCS_REF_URL;#recipe-syntax'>Recipe Syntax</ulink>" - section in the Yocto Project Reference Manual. + "<link linkend='recipe-syntax'>Recipe Syntax</link>" + section. </note> </para> @@ -1572,47 +1621,43 @@ and have sourced the build environment setup script (i.e. <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>oe-init-build-env</filename></ulink>). - Here is the basic <filename>recipetool</filename> syntax: - <note> - Running <filename>recipetool -h</filename> or - <filename>recipetool create -h</filename> produces the - Python-generated help, which presented differently - than what follows here. - </note> + To get help on the tool, use the following command: <literallayout class='monospaced'> - recipetool -h - recipetool create [-h] - recipetool [-d] [-q] [--color auto | always | never ] create -o <replaceable>OUTFILE</replaceable> [-m] [-x <replaceable>EXTERNALSRC</replaceable>] <replaceable>source</replaceable> - - -d Enables debug output. - -q Outputs only errors (quiet mode). - --color Colorizes the output automatically, always, or never. - -h Displays Python generated syntax for recipetool. - create Causes recipetool to create a base recipe. The create - command is further defined with these options: - - -o <replaceable>OUTFILE</replaceable> Specifies the full path and filename for the generated - recipe. - -m Causes the recipe to be machine-specific rather than - architecture-specific (default). - -x <replaceable>EXTERNALSRC</replaceable> Fetches and extracts source files from <replaceable>source</replaceable> - and places them in <replaceable>EXTERNALSRC</replaceable>. - <replaceable>source</replaceable> must be a URL. - -h Displays Python-generated syntax for create. - <replaceable>source</replaceable> Specifies the source code on which to base the - recipe. + $ recipetool -h + NOTE: Starting bitbake server... + usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ... + + OpenEmbedded recipe tool + + options: + -d, --debug Enable debug output + -q, --quiet Print only errors + --color COLOR Colorize output (where COLOR is auto, always, never) + -h, --help show this help message and exit + + subcommands: + create Create a new recipe + newappend Create a bbappend for the specified target in the specified + layer + setvar Set a variable within a recipe + appendfile Create/update a bbappend to replace a target file + appendsrcfiles Create/update a bbappend to add or replace source files + appendsrcfile Create/update a bbappend to add or replace a source file + Use recipetool <subcommand> --help to get help on a specific command </literallayout> </para> <para> - Running <filename>recipetool create -o</filename> <replaceable>OUTFILE</replaceable> + Running + <filename>recipetool create -o</filename> <replaceable>OUTFILE</replaceable> creates the base recipe and locates it properly in the layer that contains your source files. Following are some syntax examples: </para> <para> - Use this syntax to generate a recipe based on <replaceable>source</replaceable>. + Use this syntax to generate a recipe based on + <replaceable>source</replaceable>. Once generated, the recipe resides in the existing source code layer: <literallayout class='monospaced'> @@ -1625,7 +1670,8 @@ <literallayout class='monospaced'> recipetool create -o <replaceable>OUTFILE</replaceable> -x <replaceable>EXTERNALSRC</replaceable> <replaceable>source</replaceable> </literallayout> - Use this syntax to generate a recipe based on <replaceable>source</replaceable>. + Use this syntax to generate a recipe based on + <replaceable>source</replaceable>. The options direct <filename>recipetool</filename> to generate debugging information. Once generated, the recipe resides in the existing source @@ -1646,7 +1692,7 @@ The Yocto Project and OpenEmbedded communities maintain many recipes that might be candidates for what you are doing. You can find a good central index of these recipes in the - <ulink url='http://layers.openembedded.org'>OpenEmbedded metadata index</ulink>. + <ulink url='http://layers.openembedded.org'>OpenEmbedded Layer Index</ulink>. </para> <para> @@ -1811,8 +1857,8 @@ <para> You can find more information about the build process in - "<ulink url='&YOCTO_DOCS_REF_URL;#ref-development-environment'>The Yocto Project Development Environment</ulink>" - chapter of the Yocto Project Reference Manual. + "<ulink url='&YOCTO_DOCS_OM_URL;#overview-development-environment'>The Yocto Project Development Environment</ulink>" + chapter of the Yocto Project Overview and Concepts Manual. </para> </section> @@ -1828,8 +1874,8 @@ Your recipe must have a <filename>SRC_URI</filename> variable that points to where the source is located. For a graphical representation of source locations, see the - "<ulink url='&YOCTO_DOCS_REF_URL;#sources-dev-environment'>Sources</ulink>" - section in the Yocto Project Reference Manual. + "<ulink url='&YOCTO_DOCS_OM_URL;#sources-dev-environment'>Sources</ulink>" + section in the Yocto Project Overview and Concepts Manual. </para> <para> @@ -2141,8 +2187,9 @@ containing the current checksum. For more explanation and examples of how to set the <filename>LIC_FILES_CHKSUM</filename> variable, see the - "<ulink url='&YOCTO_DOCS_REF_URL;#usingpoky-configuring-LIC_FILES_CHKSUM'>Tracking License Changes</ulink>" - section in the Yocto Project Reference Manual.</para> + "<link link='usingpoky-configuring-LIC_FILES_CHKSUM'>Tracking License Changes</link>" + section.</para> + <para>To determine the correct checksum string, you can list the appropriate files in the <filename>LIC_FILES_CHKSUM</filename> variable with @@ -2288,8 +2335,9 @@ automatically add a runtime dependency to "mypackage" on "example"). See the - "<ulink url='&YOCTO_DOCS_REF_URL;#automatically-added-runtime-dependencies'>Automatically Added Runtime Dependencies</ulink>" - in the Yocto Project Reference Manual for further details. + "<ulink url='&YOCTO_DOCS_OM_URL;#automatically-added-runtime-dependencies'>Automatically Added Runtime Dependencies</ulink>" + section in the Yocto Project Overview and Concepts Manual for + further details. </para> </section> @@ -2977,6 +3025,133 @@ </para> </section> + <section id='metadata-virtual-providers'> + <title>Using Virtual Providers</title> + + <para> + Prior to a build, if you know that several different recipes + provide the same functionality, you can use a virtual provider + (i.e. <filename>virtual/*</filename>) as a placeholder for the + actual provider. + The actual provider is determined at build-time. + </para> + + <para> + A common scenario where a virtual provider is used would be + for the kernel recipe. + Suppose you have three kernel recipes whose + <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink> + values map to <filename>kernel-big</filename>, + <filename>kernel-mid</filename>, and + <filename>kernel-small</filename>. + Furthermore, each of these recipes in some way uses a + <ulink url='&YOCTO_DOCS_REF_URL;#var-PROVIDES'><filename>PROVIDES</filename></ulink> + statement that essentially identifies itself as being able + to provide <filename>virtual/kernel</filename>. + Here is one way through the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-kernel'><filename>kernel</filename></ulink> + class: + <literallayout class='monospaced'> + PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel") else "" }" + </literallayout> + Any recipe that inherits the <filename>kernel</filename> class + is going to utilize a <filename>PROVIDES</filename> statement + that identifies that recipe as being able to provide the + <filename>virtual/kernel</filename> item. + </para> + + <para> + Now comes the time to actually build an image and you need a + kernel recipe, but which one? + You can configure your build to call out the kernel recipe + you want by using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></ulink> + variable. + As an example, consider the + <ulink url='https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/conf/machine/include/x86-base.inc'><filename>x86-base.inc</filename></ulink> + include file, which is a machine + (i.e. <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>) + configuration file. + This include file is the reason all x86-based machines use the + <filename>linux-yocto</filename> kernel. + Here are the relevant lines from the include file: + <literallayout class='monospaced'> + PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto" + PREFERRED_VERSION_linux-yocto ??= "4.15%" + </literallayout> + </para> + + <para> + When you use a virtual provider, you do not have to + "hard code" a recipe name as a build dependency. + You can use the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> + variable to state the build is dependent on + <filename>virtual/kernel</filename> for example: + <literallayout class='monospaced'> + DEPENDS = "virtual/kernel" + </literallayout> + During the build, the OpenEmbedded build system picks + the correct recipe needed for the + <filename>virtual/kernel</filename> dependency based on the + <filename>PREFERRED_PROVIDER</filename> variable. + If you want to use the small kernel mentioned at the beginning + of this section, configure your build as follows: + <literallayout class='monospaced'> + PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small" + </literallayout> + <note> + Any recipe that + <ulink url='&YOCTO_DOCS_REF_URL;#var-PROVIDES'><filename>PROVIDES</filename></ulink> + a <filename>virtual/*</filename> item that is ultimately + not selected through + <filename>PREFERRED_PROVIDER</filename> does not get built. + Preventing these recipes from building is usually the + desired behavior since this mechanism's purpose is to + select between mutually exclusive alternative providers. + </note> + </para> + + <para> + The following lists specific examples of virtual providers: + <itemizedlist> + <listitem><para> + <filename>virtual/kernel</filename>: + Provides the name of the kernel recipe to use when + building a kernel image. + </para></listitem> + <listitem><para> + <filename>virtual/bootloader</filename>: + Provides the name of the bootloader to use when + building an image. + </para></listitem> + <listitem><para> + <filename>virtual/mesa</filename>: + Provides <filename>gbm.pc</filename>. + </para></listitem> + <listitem><para> + <filename>virtual/egl</filename>: + Provides <filename>egl.pc</filename> and possibly + <filename>wayland-egl.pc</filename>. + </para></listitem> + <listitem><para> + <filename>virtual/libgl</filename>: + Provides <filename>gl.pc</filename> (i.e. libGL). + </para></listitem> + <listitem><para> + <filename>virtual/libgles1</filename>: + Provides <filename>glesv1_cm.pc</filename> + (i.e. libGLESv1_CM). + </para></listitem> + <listitem><para> + <filename>virtual/libgles2</filename>: + Provides <filename>glesv2.pc</filename> + (i.e. libGLESv2). + </para></listitem> + </itemizedlist> + </para> + </section> + <section id='properly-versioning-pre-release-recipes'> <title>Properly Versioning Pre-Release Recipes</title> @@ -3224,8 +3399,10 @@ The variable <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-LIC_FILES_CHKSUM'>LIC_FILES_CHKSUM</ulink></filename> is used to track source license changes as described in the - "<ulink url='&YOCTO_DOCS_REF_URL;#usingpoky-configuring-LIC_FILES_CHKSUM'>Tracking License Changes</ulink>" section. - You can quickly create Autotool-based recipes in a manner similar to the previous example. + "<link linkend='usingpoky-configuring-LIC_FILES_CHKSUM'>Tracking License Changes</link>" + section in the Yocto Project Overview and Concepts Manual. + You can quickly create Autotool-based recipes in a manner + similar to the previous example. </para> </section> @@ -3430,9 +3607,9 @@ allows runtime dependencies between packages to be added automatically. See the - "<ulink url='&YOCTO_DOCS_REF_URL;#automatically-added-runtime-dependencies'>Automatically Added Runtime Dependencies</ulink>" - section in the Yocto Project Reference Manual - for more information. + "<ulink url='&YOCTO_DOCS_OM_URL;#automatically-added-runtime-dependencies'>Automatically Added Runtime Dependencies</ulink>" + section in the Yocto Project Overview and + Concepts Manual for more information. </para></listitem> </itemizedlist> </note> @@ -3517,6 +3694,330 @@ style. </para> </section> + + <section id='recipe-syntax'> + <title>Recipe Syntax</title> + + <para> + Understanding recipe file syntax is important for writing + recipes. + The following list overviews the basic items that make up a + BitBake recipe file. + For more complete BitBake syntax descriptions, see the + "<ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink>" + chapter of the BitBake User Manual. + <itemizedlist> + <listitem><para> + <emphasis>Variable Assignments and Manipulations:</emphasis> + Variable assignments allow a value to be assigned to a + variable. + The assignment can be static text or might include + the contents of other variables. + In addition to the assignment, appending and prepending + operations are also supported.</para> + + <para>The following example shows some of the ways + you can use variables in recipes: + <literallayout class='monospaced'> + S = "${WORKDIR}/postfix-${PV}" + CFLAGS += "-DNO_ASM" + SRC_URI_append = " file://fixup.patch" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Functions:</emphasis> + Functions provide a series of actions to be performed. + You usually use functions to override the default + implementation of a task function or to complement + a default function (i.e. append or prepend to an + existing function). + Standard functions use <filename>sh</filename> shell + syntax, although access to OpenEmbedded variables and + internal methods are also available.</para> + + <para>The following is an example function from the + <filename>sed</filename> recipe: + <literallayout class='monospaced'> + do_install () { + autotools_do_install + install -d ${D}${base_bindir} + mv ${D}${bindir}/sed ${D}${base_bindir}/sed + rmdir ${D}${bindir}/ + } + </literallayout> + It is also possible to implement new functions that + are called between existing tasks as long as the + new functions are not replacing or complementing the + default functions. + You can implement functions in Python + instead of shell. + Both of these options are not seen in the majority of + recipes. + </para></listitem> + <listitem><para><emphasis>Keywords:</emphasis> + BitBake recipes use only a few keywords. + You use keywords to include common + functions (<filename>inherit</filename>), load parts + of a recipe from other files + (<filename>include</filename> and + <filename>require</filename>) and export variables + to the environment (<filename>export</filename>). + </para> + + <para>The following example shows the use of some of + these keywords: + <literallayout class='monospaced'> + export POSTCONF = "${STAGING_BINDIR}/postconf" + inherit autoconf + require otherfile.inc + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Comments (#):</emphasis> + Any lines that begin with the hash character + (<filename>#</filename>) are treated as comment lines + and are ignored: + <literallayout class='monospaced'> + # This is a comment + </literallayout> + </para></listitem> + </itemizedlist> + </para> + + <para> + This next list summarizes the most important and most commonly + used parts of the recipe syntax. + For more information on these parts of the syntax, you can + reference the + <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink> + chapter in the BitBake User Manual. + <itemizedlist> + <listitem><para> + <emphasis>Line Continuation (\):</emphasis> + Use the backward slash (<filename>\</filename>) + character to split a statement over multiple lines. + Place the slash character at the end of the line that + is to be continued on the next line: + <literallayout class='monospaced'> + VAR = "A really long \ + line" + </literallayout> + <note> + You cannot have any characters including spaces + or tabs after the slash character. + </note> + </para></listitem> + <listitem><para> + <emphasis>Using Variables (${<replaceable>VARNAME</replaceable>}):</emphasis> + Use the <filename>${<replaceable>VARNAME</replaceable>}</filename> + syntax to access the contents of a variable: + <literallayout class='monospaced'> + SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz" + </literallayout> + <note> + It is important to understand that the value of a + variable expressed in this form does not get + substituted automatically. + The expansion of these expressions happens + on-demand later (e.g. usually when a function that + makes reference to the variable executes). + This behavior ensures that the values are most + appropriate for the context in which they are + finally used. + On the rare occasion that you do need the variable + expression to be expanded immediately, you can use + the <filename>:=</filename> operator instead of + <filename>=</filename> when you make the + assignment, but this is not generally needed. + </note> + </para></listitem> + <listitem><para> + <emphasis>Quote All Assignments ("<replaceable>value</replaceable>"):</emphasis> + Use double quotes around values in all variable + assignments (e.g. + <filename>"<replaceable>value</replaceable>"</filename>). + Following is an example: + <literallayout class='monospaced'> + VAR1 = "${OTHERVAR}" + VAR2 = "The version is ${PV}" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Conditional Assignment (?=):</emphasis> + Conditional assignment is used to assign a + value to a variable, but only when the variable is + currently unset. + Use the question mark followed by the equal sign + (<filename>?=</filename>) to make a "soft" assignment + used for conditional assignment. + Typically, "soft" assignments are used in the + <filename>local.conf</filename> file for variables + that are allowed to come through from the external + environment. + </para> + + <para>Here is an example where + <filename>VAR1</filename> is set to "New value" if + it is currently empty. + However, if <filename>VAR1</filename> has already been + set, it remains unchanged: + <literallayout class='monospaced'> + VAR1 ?= "New value" + </literallayout> + In this next example, <filename>VAR1</filename> + is left with the value "Original value": + <literallayout class='monospaced'> + VAR1 = "Original value" + VAR1 ?= "New value" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Appending (+=):</emphasis> + Use the plus character followed by the equals sign + (<filename>+=</filename>) to append values to existing + variables. + <note> + This operator adds a space between the existing + content of the variable and the new content. + </note></para> + + <para>Here is an example: + <literallayout class='monospaced'> + SRC_URI += "file://fix-makefile.patch" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Prepending (=+):</emphasis> + Use the equals sign followed by the plus character + (<filename>=+</filename>) to prepend values to existing + variables. + <note> + This operator adds a space between the new content + and the existing content of the variable. + </note></para> + + <para>Here is an example: + <literallayout class='monospaced'> + VAR =+ "Starts" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Appending (_append):</emphasis> + Use the <filename>_append</filename> operator to + append values to existing variables. + This operator does not add any additional space. + Also, the operator is applied after all the + <filename>+=</filename>, and + <filename>=+</filename> operators have been applied and + after all <filename>=</filename> assignments have + occurred. + </para> + + <para>The following example shows the space being + explicitly added to the start to ensure the appended + value is not merged with the existing value: + <literallayout class='monospaced'> + SRC_URI_append = " file://fix-makefile.patch" + </literallayout> + You can also use the <filename>_append</filename> + operator with overrides, which results in the actions + only being performed for the specified target or + machine: + <literallayout class='monospaced'> + SRC_URI_append_sh4 = " file://fix-makefile.patch" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Prepending (_prepend):</emphasis> + Use the <filename>_prepend</filename> operator to + prepend values to existing variables. + This operator does not add any additional space. + Also, the operator is applied after all the + <filename>+=</filename>, and + <filename>=+</filename> operators have been applied and + after all <filename>=</filename> assignments have + occurred. + </para> + + <para>The following example shows the space being + explicitly added to the end to ensure the prepended + value is not merged with the existing value: + <literallayout class='monospaced'> + CFLAGS_prepend = "-I${S}/myincludes " + </literallayout> + You can also use the <filename>_prepend</filename> + operator with overrides, which results in the actions + only being performed for the specified target or + machine: + <literallayout class='monospaced'> + CFLAGS_prepend_sh4 = "-I${S}/myincludes " + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Overrides:</emphasis> + You can use overrides to set a value conditionally, + typically based on how the recipe is being built. + For example, to set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-KBRANCH'><filename>KBRANCH</filename></ulink> + variable's value to "standard/base" for any target + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>, + except for qemuarm where it should be set to + "standard/arm-versatile-926ejs", you would do the + following: + <literallayout class='monospaced'> + KBRANCH = "standard/base" + KBRANCH_qemuarm = "standard/arm-versatile-926ejs" + </literallayout> + Overrides are also used to separate alternate values + of a variable in other situations. + For example, when setting variables such as + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink> + that are specific to individual packages produced by + a recipe, you should always use an override that + specifies the name of the package. + </para></listitem> + <listitem><para> + <emphasis>Indentation:</emphasis> + Use spaces for indentation rather than than tabs. + For shell functions, both currently work. + However, it is a policy decision of the Yocto Project + to use tabs in shell functions. + Realize that some layers have a policy to use spaces + for all indentation. + </para></listitem> + <listitem><para> + <emphasis>Using Python for Complex Operations:</emphasis> + For more advanced processing, it is possible to use + Python code during variable assignments (e.g. + search and replacement on a variable).</para> + + <para>You indicate Python code using the + <filename>${@<replaceable>python_code</replaceable>}</filename> + syntax for the variable assignment: + <literallayout class='monospaced'> + SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Shell Function Syntax:</emphasis> + Write shell functions as if you were writing a shell + script when you describe a list of actions to take. + You should ensure that your script works with a generic + <filename>sh</filename> and that it does not require + any <filename>bash</filename> or other shell-specific + functionality. + The same considerations apply to various system + utilities (e.g. <filename>sed</filename>, + <filename>grep</filename>, <filename>awk</filename>, + and so forth) that you might wish to use. + If in doubt, you should check with multiple + implementations - including those from BusyBox. + </para></listitem> + </itemizedlist> + </para> + </section> </section> <section id="platdev-newmachine"> @@ -3538,8 +4039,9 @@ <para> For a complete example that shows how to add a new machine, see the - "<ulink url='&YOCTO_DOCS_BSP_URL;#creating-a-new-bsp-layer-using-the-yocto-bsp-script'>Creating a New BSP Layer Using the yocto-bsp Script</ulink>" - section in the Yocto Project Board Support Package (BSP) Developer's Guide. + "<ulink url='&YOCTO_DOCS_BSP_URL;#creating-a-new-bsp-layer-using-the-bitbake-layers-script'>Creating a New BSP Layer Using the <filename>bitbake-layers</filename> Script</ulink>" + section in the Yocto Project Board Support Package (BSP) + Developer's Guide. </para> <section id="platdev-newmachine-conffile"> @@ -3694,6 +4196,562 @@ </section> </section> + <section id='gs-upgrading-recipes'> + <title>Upgrading Recipes</title> + + <para> + Over time, upstream developers publish new versions for software + built by layer recipes. + It is recommended to keep recipes up-to-date with upstream + version releases. + You can use the Automated Upgrade Helper (AUH) to set up + automatic version upgrades. + Alternatively, you can use <filename>devtool upgrade</filename> + to set up semi-automatic version upgrades. + Finally, you can even manually upgrade a recipe by editing the + recipe itself. + </para> + + <section id='gs-using-the-auto-upgrade-helper'> + <title>Using the Auto Upgrade Helper (AUH)</title> + + <para> + The AUH utility works in conjunction with the + OpenEmbedded build system in order to automatically generate + upgrades for recipes based on new versions being + published upstream. + Use AUH when you want to create a service that performs the + upgrades automatically and optionally sends you an email with + the results. + </para> + + <para> + AUH allows you to update several recipes with a single use. + You can also optionally perform build and integration tests + using images with the results saved to your hard drive and + emails of results optionally sent to recipe maintainers. + Finally, AUH creates Git commits with appropriate commit + messages in the layer's tree for the changes made to recipes. + <note> + Conditions do exist when you should not use AUH to upgrade + recipes and you should instead use either + <filename>devtool upgrade</filename> or upgrade your + recipes manually: + <itemizedlist> + <listitem><para> + When AUH cannot complete the upgrade sequence. + This situation usually results because custom + patches carried by the recipe cannot be + automatically rebased to the new version. + In this case, <filename>devtool upgrade</filename> + allows you to manually resolve conflicts. + </para></listitem> + <listitem><para> + When for any reason you want fuller control over + the upgrade process. + For example, when you want special arrangements + for testing. + </para></listitem> + </itemizedlist> + </note> + </para> + + <para> + The following steps describe how to set up the AUH utility: + <orderedlist> + <listitem><para> + <emphasis>Be Sure the Development Host is Set Up:</emphasis> + You need to be sure that your development host is + set up to use the Yocto Project. + For information on how to set up your host, see the + "<link linkend='setting-up-the-development-host-to-use-the-yocto-project'>Preparing the Build Host</link>" + section. + </para></listitem> + <listitem><para> + <emphasis>Make Sure Git is Configured:</emphasis> + The AUH utility requires Git to be configured because + AUH uses Git to save upgrades. + Thus, you must have Git user and email configured. + The following command shows your configurations: + <literallayout class='monospaced'> + $ git config --list + </literallayout> + If you do not have the user and email configured, you + can use the following commands to do so: + <literallayout class='monospaced'> + $ git config --global user.name <replaceable>some_name</replaceable> + $ git config --global user.email <replaceable>username</replaceable>@<replaceable>domain</replaceable>.com + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Clone the AUH Repository:</emphasis> + To use AUH, you must clone the repository onto your + development host. + The following command uses Git to create a local + copy of the repository on your system: + <literallayout class='monospaced'> + $ git clone git://git.yoctoproject.org/auto-upgrade-helper + Cloning into 'auto-upgrade-helper'... + remote: Counting objects: 768, done. + remote: Compressing objects: 100% (300/300), done. + remote: Total 768 (delta 499), reused 703 (delta 434) + Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done. + Resolving deltas: 100% (499/499), done. + Checking connectivity... done. + </literallayout> + AUH is not part of the + <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core (OE-Core)</ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#poky'>Poky</ulink> + repositories. + </para></listitem> + <listitem><para> + <emphasis>Create a Dedicated Build Directory:</emphasis> + Run the + <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>oe-init-build-env</filename></ulink> + script to create a fresh build directory that you + use exclusively for running the AUH utility: + <literallayout class='monospaced'> + $ cd ~/poky + $ source oe-init-build-env <replaceable>your_AUH_build_directory</replaceable> + </literallayout> + Re-using an existing build directory and its + configurations is not recommended as existing settings + could cause AUH to fail or behave undesirably. + </para></listitem> + <listitem><para> + <emphasis>Make Configurations in Your Local Configuration File:</emphasis> + Several settings need to exist in the + <filename>local.conf</filename> file in the build + directory you just created for AUH. + Make these following configurations: + <itemizedlist> + <listitem><para> + Enable "distrodata" as follows: + <literallayout class='monospaced'> + INHERIT =+ "distrodata" + </literallayout> + </para></listitem> + <listitem><para> + If you want to enable + <ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-build-output-quality'>Build History</ulink>, + which is optional, you need the following + lines in the + <filename>conf/local.conf</filename> file: + <literallayout class='monospaced'> + INHERIT =+ "buildhistory" + BUILDHISTORY_COMMIT = "1" + </literallayout> + With this configuration and a successful + upgrade, a build history "diff" file appears in + the + <filename>upgrade-helper/work/recipe/buildhistory-diff.txt</filename> + file found in your build directory. + </para></listitem> + <listitem><para> + If you want to enable testing through the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage*'><filename>testimage</filename></ulink> + class, which is optional, you need to have the + following set in your + <filename>conf/local.conf</filename> file: + <literallayout class='monospaced'> + INHERIT += "testimage" + </literallayout> + <note> + If your distro does not enable by default + ptest, which Poky does, you need the + following in your + <filename>local.conf</filename> file: + <literallayout class='monospaced'> + DISTRO_FEATURES_append = " ptest" + </literallayout> + </note> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <emphasis>Optionally Start a vncserver:</emphasis> + If you are running in a server without an X11 session, + you need to start a vncserver: + <literallayout class='monospaced'> + $ vncserver :1 + $ export DISPLAY=:1 + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Create and Edit an AUH Configuration File:</emphasis> + You need to have the + <filename>upgrade-helper/upgrade-helper.conf</filename> + configuration file in your build directory. + You can find a sample configuration file in the + <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/auto-upgrade-helper/tree/'>AUH source repository</ulink>. + </para> + + <para>Read through the sample file and make + configurations as needed. + For example, if you enabled build history in your + <filename>local.conf</filename> as described earlier, + you must enable it in + <filename>upgrade-helper.conf</filename>.</para> + + <para>Also, if you are using the default + <filename>maintainers.inc</filename> file supplied + with Poky and located in + <filename>meta-yocto</filename> and you do not set a + "maintainers_whitelist" or "global_maintainer_override" + in the <filename>upgrade-helper.conf</filename> + configuration, and you specify "-e all" on the + AUH command-line, the utility automatically sends out + emails to all the default maintainers. + Please avoid this. + </para></listitem> + </orderedlist> + </para> + + <para> + This next set of examples describes how to use the AUH: + <itemizedlist> + <listitem><para> + <emphasis>Upgrading a Specific Recipe:</emphasis> + To upgrade a specific recipe, use the following + form: + <literallayout class='monospaced'> + $ upgrade-helper.py <replaceable>recipe_name</replaceable> + </literallayout> + For example, this command upgrades the + <filename>xmodmap</filename> recipe: + <literallayout class='monospaced'> + $ upgrade-helper.py xmodmap + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Upgrading a Specific Recipe to a Particular Version:</emphasis> + To upgrade a specific recipe to a particular version, + use the following form: + <literallayout class='monospaced'> + $ upgrade-helper.py <replaceable>recipe_name</replaceable> -t <replaceable>version</replaceable> + </literallayout> + For example, this command upgrades the + <filename>xmodmap</filename> recipe to version + 1.2.3: + <literallayout class='monospaced'> + $ upgrade-helper.py xmodmap -t 1.2.3 + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Upgrading all Recipes to the Latest Versions and Suppressing Email Notifications:</emphasis> + To upgrade all recipes to their most recent versions + and suppress the email notifications, use the following + command: + <literallayout class='monospaced'> + $ upgrade-helper.py all + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Upgrading all Recipes to the Latest Versions and Send Email Notifications:</emphasis> + To upgrade all recipes to their most recent versions + and send email messages to maintainers for each + attempted recipe as well as a status email, use the + following command: + <literallayout class='monospaced'> + $ upgrade-helper.py -e all + </literallayout> + </para></listitem> + </itemizedlist> + </para> + + <para> + Once you have run the AUH utility, you can find the results + in the AUH build directory: + <literallayout class='monospaced'> + ${BUILDDIR}/upgrade-helper/<replaceable>timestamp</replaceable> + </literallayout> + The AUH utility also creates recipe update commits from + successful upgrade attempts in the layer tree. + </para> + + <para> + You can easily set up to run the AUH utility on a regular + basis by using a cron job. + See the + <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/auto-upgrade-helper/tree/weeklyjob.sh'><filename>weeklyjob.sh</filename></ulink> + file distributed with the utility for an example. + </para> + </section> + + <section id='gs-using-devtool-upgrade'> + <title>Using <filename>devtool upgrade</filename></title> + + <para> + As mentioned earlier, an alternative method for upgrading + recipes to newer versions is to use + <ulink url='&YOCTO_DOCS_REF_URL;#ref-devtool-reference'><filename>devtool upgrade</filename></ulink>. + You can read about <filename>devtool upgrade</filename> in + general in the + "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-devtool-use-devtool-upgrade-to-create-a-version-of-the-recipe-that-supports-a-newer-version-of-the-software'>Use <filename>devtool upgrade</filename> to Create a Version of the Recipe that Supports a Newer Version of the Software</ulink>" + section in the Yocto Project Application Development and the + Extensible Software Development Kit (eSDK) Manual. + </para> + + <para> + To see all the command-line options available with + <filename>devtool upgrade</filename>, use the following help + command: + <literallayout class='monospaced'> + $ devtool upgrade -h + </literallayout> + </para> + + <para> + If you want to find out what version a recipe is currently at + upstream without any attempt to upgrade your local version of + the recipe, you can use the following command: + <literallayout class='monospaced'> + $ devtool latest-version <replaceable>recipe_name</replaceable> + </literallayout> + </para> + + <para> + As mentioned in the previous section describing AUH, + <filename>devtool upgrade</filename> works in a + less-automated manner than AUH. + Specifically, <filename>devtool upgrade</filename> only + works on a single recipe that you name on the command line, + cannot perform build and integration testing using images, + and does not automatically generate commits for changes in + the source tree. + Despite all these "limitations", + <filename>devtool upgrade</filename> updates the recipe file + to the new upstream version and attempts to rebase custom + patches contained by the recipe as needed. + <note> + AUH uses much of <filename>devtool upgrade</filename> + behind the scenes making AUH somewhat of a "wrapper" + application for <filename>devtool upgrade</filename>. + </note> + </para> + + <para> + A typical scenario involves having used Git to clone an + upstream repository that you use during build operations. + Because you are (or have) built the recipe in the past, the + layer is likely added to your configuration already. + If for some reason, the layer is not added, you could add + it easily using the + <ulink url='&YOCTO_DOCS_BSP_URL;#creating-a-new-bsp-layer-using-the-bitbake-layers-script'><filename>bitbake-layers</filename></ulink> + script. + For example, suppose you use the <filename>nano.bb</filename> + recipe from the <filename>meta-oe</filename> layer in the + <filename>meta-openembedded</filename> repository. + For this example, assume that the layer has been cloned into + following area: + <literallayout class='monospaced'> + /home/scottrif/meta-openembedded + </literallayout> + The following command from your + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> + adds the layer to your build configuration (i.e. + <filename>${BUILDDIR}/conf/bblayers.conf</filename>): + <literallayout class='monospaced'> + $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe + NOTE: Starting bitbake server... + Parsing recipes: 100% |##########################################| Time: 0:00:55 + Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors. + Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00 + Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00 + Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00 + Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00 + </literallayout> + For this example, assume that the <filename>nano.bb</filename> + recipe that is upstream has a 2.9.3 version number. + However, the version in the local repository is 2.7.4. + The following command from your build directory automatically + upgrades the recipe for you: + <note> + Using the <filename>-V</filename> option is not necessary. + Omitting the version number causes + <filename>devtool upgrade</filename> to upgrade the recipe + to the most recent version. + </note> + <literallayout class='monospaced'> + $ devtool upgrade nano -V 2.9.3 + NOTE: Starting bitbake server... + NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace + Parsing recipes: 100% |##########################################| Time: 0:00:46 + Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors. + NOTE: Extracting current version source... + NOTE: Resolving any missing task queue dependencies + . + . + . + NOTE: Executing SetScene Tasks + NOTE: Executing RunQueue Tasks + NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded. + Adding changed files: 100% |#####################################| Time: 0:00:00 + NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano + NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb + </literallayout> + Continuing with this example, you can use + <filename>devtool build</filename> to build the newly upgraded + recipe: + <literallayout class='monospaced'> + $ devtool build nano + NOTE: Starting bitbake server... + Loading cache: 100% |################################################################################################| Time: 0:00:01 + Loaded 2040 entries from dependency cache. + Parsing recipes: 100% |##############################################################################################| Time: 0:00:00 + Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors. + NOTE: Resolving any missing task queue dependencies + . + . + . + NOTE: Executing SetScene Tasks + NOTE: Executing RunQueue Tasks + NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano + NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded. + </literallayout> + Within the <filename>devtool upgrade</filename> workflow, + opportunity exists to deploy and test your rebuilt software. + For this example, however, running + <filename>devtool finish</filename> cleans up the workspace + once the source in your workspace is clean. + This usually means using Git to stage and submit commits + for the changes generated by the upgrade process. + </para> + + <para> + Once the tree is clean, you can clean things up in this + example with the following command from the + <filename>${BUILDDIR}/workspace/sources/nano</filename> + directory: + <literallayout class='monospaced'> + $ devtool finish nano meta-oe + NOTE: Starting bitbake server... + Loading cache: 100% |################################################################################################| Time: 0:00:00 + Loaded 2040 entries from dependency cache. + Parsing recipes: 100% |##############################################################################################| Time: 0:00:01 + Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors. + NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch + NOTE: Updating recipe nano_2.9.3.bb + NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb + NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano + NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually + </literallayout> + Using the <filename>devtool finish</filename> command cleans + up the workspace and creates a patch file based on your + commits. + The tool puts all patch files back into the source directory + in a sub-directory named <filename>nano</filename> in this + case. + </para> + </section> + + <section id='dev-manually-upgrading-a-recipe'> + <title>Manually Upgrading a Recipe</title> + + <para> + If for some reason you choose not to upgrade recipes using the + <link linkend='gs-using-the-auto-upgrade-helper'>Auto Upgrade Helper (AUH)</link> + or by using + <link linkend='gs-using-devtool-upgrade'><filename>devtool upgrade</filename></link>, + you can manually edit the recipe files to upgrade the versions. + <note><title>Caution</title> + Manually updating multiple recipes scales poorly and + involves many steps. + The recommendation to upgrade recipe versions is through + AUH or <filename>devtool upgrade</filename>, both of which + automate some steps and provide guidance for others needed + for the manual process. + </note> + </para> + + <para> + To manually upgrade recipe versions, follow these general steps: + <orderedlist> + <listitem><para> + <emphasis>Change the Version:</emphasis> + Rename the recipe such that the version (i.e. the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink> + part of the recipe name) changes appropriately. + If the version is not part of the recipe name, change + the value as it is set for <filename>PV</filename> + within the recipe itself. + </para></listitem> + <listitem><para> + <emphasis>Update <filename>SRCREV</filename> if Needed:</emphasis> + If the source code your recipe builds is fetched from + Git or some other version control system, update + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink> + to point to the commit hash that matches the new + version. + </para></listitem> + <listitem><para> + <emphasis>Build the Software:</emphasis> + Try to build the recipe using BitBake. + Typical build failures include the following: + <itemizedlist> + <listitem><para> + License statements were updated for the new + version. + For this case, you need to review any changes + to the license and update the values of + <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE'><filename>LICENSE</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-LIC_FILES_CHKSUM'><filename>LIC_FILES_CHKSUM</filename></ulink> + as needed. + <note> + License changes are often inconsequential. + For example, the license text's copyright + year might have changed. + </note> + </para></listitem> + <listitem><para> + Custom patches carried by the older version of + the recipe might fail to apply to the new + version. + For these cases, you need to review the + failures. + Patches might not be necessary for the new + version of the software if the upgraded version + has fixed those issues. + If a patch is necessary and failing, you need + to rebase it into the new version. + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <emphasis>Optionally Attempt to Build for Several Architectures:</emphasis> + Once you successfully build the new software for a + given architecture, you could test the build for + other architectures by changing the + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + variable and rebuilding the software. + This optional step is especially important if the + recipe is to be released publicly. + </para></listitem> + <listitem><para> + <emphasis>Check the Upstream Change Log or Release Notes:</emphasis> + Checking both these reveals if new features exist that + could break backwards-compatibility. + If so, you need to take steps to mitigate or eliminate + that situation. + </para></listitem> + <listitem><para> + <emphasis>Optionally Create a Bootable Image and Test:</emphasis> + If you want, you can test the new software by booting + it onto actual hardware. + </para></listitem> + <listitem><para> + <emphasis>Create a Commit with the Change in the Layer Repository:</emphasis> + After all builds work and any testing is successful, + you can create commits for any changes in the layer + holding your upgraded recipe. + </para></listitem> + </orderedlist> + </para> + </section> + </section> + <section id='finding-the-temporary-source-code'> <title>Finding Temporary Source Code</title> @@ -3888,8 +4946,8 @@ Modifications will also disappear if you use the <filename>rm_work</filename> feature as described in the - "<ulink url='&YOCTO_DOCS_QS_URL;#qs-building-images'>Building Images</ulink>" - section of the Yocto Project Quick Start. + "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-saving-memory-during-a-build'>Conserving Disk Space During Builds</ulink>" + section. </note> </para></listitem> <listitem><para> @@ -4105,103 +5163,1225 @@ </para> </section> - <section id='platdev-building-targets-with-multiple-configurations'> - <title>Building Targets with Multiple Configurations</title> + <section id='dev-building'> + <title>Building</title> <para> - Bitbake also has functionality that allows you to build - multiple targets at the same time, where each target uses - a different configuration. + This section describes various build procedures. + For example, the steps needed for a simple build, a target that + uses multiple configurations, building an image for more than + one machine, and so forth. </para> - <para> - In order to accomplish this, you setup each of the configurations - you need to use in parallel by placing the configuration files in - your current build directory alongside the usual - <filename>local.conf</filename> file. - </para> + <section id='dev-building-a-simple-image'> + <title>Building a Simple Image</title> - <para> - Follow these guidelines to create an environment that supports - multiple configurations: - <itemizedlist> - <listitem><para> - <emphasis>Create Configuration Files</emphasis>: - You need to create a single configuration file for each - configuration for which you want to add support. - These files would contain lines such as the following: - <literallayout class='monospaced'> + <para> + In the development environment, you need to build an image + whenever you change hardware support, add or change system + libraries, or add or change services that have dependencies. + Several methods exist that allow you to build an image within + the Yocto Project. + This section presents the basic steps you need to build a + simple image using BitBake from a build host running Linux. + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + For information on how to build an image using + <ulink url='&YOCTO_DOCS_REF_URL;#toaster-term'>Toaster</ulink>, + see the + <ulink url='&YOCTO_DOCS_TOAST_URL;'>Toaster User Manual</ulink>. + </para></listitem> + <listitem><para> + For information on how to use + <filename>devtool</filename> to build images, see + the + "<ulink url='&YOCTO_DOCS_SDK_URL;#using-devtool-in-your-sdk-workflow'>Using <filename>devtool</filename> in Your SDK Workflow</ulink>" + section in the Yocto Project Application + Development and the Extensible Software Development + Kit (eSDK) manual. + </para></listitem> + <listitem><para> + For a quick example on how to build an image using + the OpenEmbedded build system, see the + <ulink url='&YOCTO_DOCS_BRIEF_URL;'>Yocto Project Quick Build</ulink> + document. + </para></listitem> + </itemizedlist> + </note> + </para> + + <para> + The build process creates an entire Linux distribution from + source and places it in your + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> + under <filename>tmp/deploy/images</filename>. + For detailed information on the build process using BitBake, + see the + "<ulink url='&YOCTO_DOCS_OM_URL;#images-dev-environment'>Images</ulink>" + section in the Yocto Project Overview and Concepts Manual. + </para> + + <para> + The following figure and list overviews the build process: + <imagedata fileref="figures/bitbake-build-flow.png" width="7in" depth="4in" align="center" scalefit="1" /> + <orderedlist> + <listitem><para> + <emphasis>Set up Your Host Development System to Support + Development Using the Yocto Project</emphasis>: + See the + "<link linkend='dev-manual-start'>Setting Up to Use the Yocto Project</link>" + section for options on how to get a build host ready to + use the Yocto Project. + </para></listitem> + <listitem><para> + <emphasis>Initialize the Build Environment:</emphasis> + Initialize the build environment by sourcing the build + environment script (i.e. + <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>): + <literallayout class='monospaced'> + $ source &OE_INIT_FILE; [<replaceable>build_dir</replaceable>] + </literallayout></para> + + <para>When you use the initialization script, the + OpenEmbedded build system uses + <filename>build</filename> as the default Build + Directory in your current work directory. + You can use a <replaceable>build_dir</replaceable> + argument with the script to specify a different build + directory. + <note><title>Tip</title> + A common practice is to use a different Build + Directory for different targets. + For example, <filename>~/build/x86</filename> for a + <filename>qemux86</filename> target, and + <filename>~/build/arm</filename> for a + <filename>qemuarm</filename> target. + </note> + </para></listitem> + <listitem><para> + <emphasis>Make Sure Your <filename>local.conf</filename> + File is Correct:</emphasis> + Ensure the <filename>conf/local.conf</filename> + configuration file, which is found in the Build + Directory, is set up how you want it. + This file defines many aspects of the build environment + including the target machine architecture through the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'>MACHINE</ulink></filename> variable, + the packaging format used during the build + (<ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>), + and a centralized tarball download directory through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> variable. + </para></listitem> + <listitem><para> + <emphasis>Build the Image:</emphasis> + Build the image using the <filename>bitbake</filename> + command: + <literallayout class='monospaced'> + $ bitbake <replaceable>target</replaceable> + </literallayout> + <note> + For information on BitBake, see the + <ulink url='&YOCTO_DOCS_BB_URL;'>BitBake User Manual</ulink>. + </note> + The <replaceable>target</replaceable> is the name of the + recipe you want to build. + Common targets are the images in + <filename>meta/recipes-core/images</filename>, + <filename>meta/recipes-sato/images</filename>, and so + forth all found in the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>. + Or, the target can be the name of a recipe for a + specific piece of software such as BusyBox. + For more details about the images the OpenEmbedded build + system supports, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>" + chapter in the Yocto Project Reference Manual.</para> + + <para>As an example, the following command builds the + <filename>core-image-minimal</filename> image: + <literallayout class='monospaced'> + $ bitbake core-image-minimal + </literallayout> + Once an image has been built, it often needs to be + installed. + The images and kernels built by the OpenEmbedded + build system are placed in the Build Directory in + <filename class="directory">tmp/deploy/images</filename>. + For information on how to run pre-built images such as + <filename>qemux86</filename> and <filename>qemuarm</filename>, + see the + <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink> + manual. + For information about how to install these images, + see the documentation for your particular board or + machine. + </para></listitem> + </orderedlist> + </para> + </section> + + <section id='platdev-building-targets-with-multiple-configurations'> + <title>Building Targets with Multiple Configurations</title> + + <para> + Bitbake also has functionality that allows you to build + multiple targets at the same time, where each target uses + a different configuration. + </para> + + <para> + In order to accomplish this, you setup each of the configurations + you need to use in parallel by placing the configuration files in + your current build directory alongside the usual + <filename>local.conf</filename> file. + </para> + + <para> + Follow these guidelines to create an environment that supports + multiple configurations: + <itemizedlist> + <listitem><para> + <emphasis>Create Configuration Files</emphasis>: + You need to create a single configuration file for each + configuration for which you want to add support. + These files would contain lines such as the following: + <literallayout class='monospaced'> MACHINE = "A" - </literallayout> - The files would contain any other variables that can - be set and built in the same directory. + </literallayout> + The files would contain any other variables that can + be set and built in the same directory. + <note> + You can change the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> + to not conflict. + </note></para> + + <para> + Furthermore, the configuration file must be located in the + current build directory in a directory named + <filename>multiconfig</filename> under the build's + <filename>conf</filename> directory where + <filename>local.conf</filename> resides. + The reason for this restriction is because the + <filename>BBPATH</filename> variable is not constructed + until the layers are parsed. + Consequently, using the configuration file as a + pre-configuration file is not possible unless it is + located in the current working directory. + </para></listitem> + <listitem><para> + <emphasis>Add the BitBake Multi-Config Variable to you Local Configuration File</emphasis>: + Use the + <filename>BBMULTICONFIG</filename> + variable in your <filename>conf/local.conf</filename> + configuration file to specify each separate configuration. + For example, the following line tells BitBake it should load + <filename>conf/multiconfig/configA.conf</filename>, + <filename>conf/multiconfig/configB.conf</filename>, and + <filename>conf/multiconfig/configC.conf</filename>. + <literallayout class='monospaced'> + BBMULTICONFIG = "configA configB configC" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Launch BitBake</emphasis>: + Use the following BitBake command form to launch the + build: + <literallayout class='monospaced'> + $ bitbake [multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable> [[[multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>] ... ] + </literallayout> + Following is an example that supports building a minimal + image for configuration A alongside a standard + <filename>core-image-sato</filename>, which takes its + configuration from <filename>local.conf</filename>: + <literallayout class='monospaced'> + $ bitbake multiconfig:configA:core-image-minimal core-image-sato + </literallayout> + </para></listitem> + </itemizedlist> + </para> + + <para> + Support for multiple configurations in this current release of + the Yocto Project (&DISTRO_NAME; &DISTRO;) has some known issues: + <itemizedlist> + <listitem><para> + No inter-multi-configuration dependencies exist. + </para></listitem> + <listitem><para> + Shared State (sstate) optimizations do not exist. + Consequently, if the build uses the same object twice + in, for example, two different + <filename>TMPDIR</filename> directories, the build + will either load from an existing sstate cache at the + start or build the object twice. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='building-an-initramfs-image'> + <title>Building an Initial RAM Filesystem (initramfs) Image</title> + + <para> + An initial RAM filesystem (initramfs) image provides a temporary + root filesystem used for early system initialization (e.g. + loading of modules needed to locate and mount the "real" root + filesystem). + <note> + The initramfs image is the successor of initial RAM disk + (initrd). + It is a "copy in and out" (cpio) archive of the initial + filesystem that gets loaded into memory during the Linux + startup process. + Because Linux uses the contents of the archive during + initialization, the initramfs image needs to contain all of the + device drivers and tools needed to mount the final root + filesystem. + </note> + </para> + + <para> + Follow these steps to create an initramfs image: + <orderedlist> + <listitem><para> + <emphasis>Create the initramfs Image Recipe:</emphasis> + You can reference the + <filename>core-image-minimal-initramfs.bb</filename> + recipe found in the <filename>meta/recipes-core</filename> + directory of the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + as an example from which to work. + </para></listitem> + <listitem><para> + <emphasis>Decide if You Need to Bundle the initramfs Image + Into the Kernel Image:</emphasis> + If you want the initramfs image that is built to be + bundled in with the kernel image, set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-INITRAMFS_IMAGE_BUNDLE'><filename>INITRAMFS_IMAGE_BUNDLE</filename></ulink> + variable to "1" in your <filename>local.conf</filename> + configuration file and set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-INITRAMFS_IMAGE'><filename>INITRAMFS_IMAGE</filename></ulink> + variable in the recipe that builds the kernel image. + <note><title>Tip</title> + It is recommended that you do bundle the initramfs + image with the kernel image to avoid circular + dependencies between the kernel recipe and the + initramfs recipe should the initramfs image + include kernel modules. + </note> + Setting the <filename>INITRAMFS_IMAGE_BUNDLE</filename> + flag causes the initramfs image to be unpacked + into the <filename>${B}/usr/</filename> directory. + The unpacked initramfs image is then passed to the kernel's + <filename>Makefile</filename> using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-CONFIG_INITRAMFS_SOURCE'><filename>CONFIG_INITRAMFS_SOURCE</filename></ulink> + variable, allowing the initramfs image to be built into + the kernel normally. + <note> + If you choose to not bundle the initramfs image with + the kernel image, you are essentially using an + <ulink url='https://en.wikipedia.org/wiki/Initrd'>Initial RAM Disk (initrd)</ulink>. + Creating an initrd is handled primarily through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-INITRD_IMAGE'><filename>INITRD_IMAGE</filename></ulink>, + <filename>INITRD_LIVE</filename>, and + <filename>INITRD_IMAGE_LIVE</filename> variables. + For more information, see the + <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/image-live.bbclass'><filename>image-live.bbclass</filename></ulink> + file. + </note> + </para></listitem> + <listitem><para> + <emphasis>Optionally Add Items to the initramfs Image + Through the initramfs Image Recipe:</emphasis> + If you add items to the initramfs image by way of its + recipe, you should use + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_INSTALL'><filename>PACKAGE_INSTALL</filename></ulink> + rather than + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink>. + <filename>PACKAGE_INSTALL</filename> gives more direct + control of what is added to the image as compared to + the defaults you might not necessarily want that are + set by the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-image'><filename>image</filename></ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-core-image'><filename>core-image</filename></ulink> + classes. + </para></listitem> + <listitem><para> + <emphasis>Build the Kernel Image and the initramfs + Image:</emphasis> + Build your kernel image using BitBake. + Because the initramfs image recipe is a dependency of the + kernel image, the initramfs image is built as well and + bundled with the kernel image if you used the + <ulink url='&YOCTO_DOCS_REF_URL;#var-INITRAMFS_IMAGE_BUNDLE'><filename>INITRAMFS_IMAGE_BUNDLE</filename></ulink> + variable described earlier. + </para></listitem> + </orderedlist> + </para> + </section> + + <section id='building-a-tiny-system'> + <title>Building a Tiny System</title> + + <para> + Very small distributions have some significant advantages such + as requiring less on-die or in-package memory (cheaper), better + performance through efficient cache usage, lower power requirements + due to less memory, faster boot times, and reduced development + overhead. + Some real-world examples where a very small distribution gives + you distinct advantages are digital cameras, medical devices, + and small headless systems. + </para> + + <para> + This section presents information that shows you how you can + trim your distribution to even smaller sizes than the + <filename>poky-tiny</filename> distribution, which is around + 5 Mbytes, that can be built out-of-the-box using the Yocto Project. + </para> + + <section id='tiny-system-overview'> + <title>Overview</title> + + <para> + The following list presents the overall steps you need to + consider and perform to create distributions with smaller + root filesystems, achieve faster boot times, maintain your critical + functionality, and avoid initial RAM disks: + <itemizedlist> + <listitem><para> + <link linkend='goals-and-guiding-principles'>Determine your goals and guiding principles.</link> + </para></listitem> + <listitem><para> + <link linkend='understand-what-gives-your-image-size'>Understand what contributes to your image size.</link> + </para></listitem> + <listitem><para> + <link linkend='trim-the-root-filesystem'>Reduce the size of the root filesystem.</link> + </para></listitem> + <listitem><para> + <link linkend='trim-the-kernel'>Reduce the size of the kernel.</link> + </para></listitem> + <listitem><para> + <link linkend='remove-package-management-requirements'>Eliminate packaging requirements.</link> + </para></listitem> + <listitem><para> + <link linkend='look-for-other-ways-to-minimize-size'>Look for other ways to minimize size.</link> + </para></listitem> + <listitem><para> + <link linkend='iterate-on-the-process'>Iterate on the process.</link> + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='goals-and-guiding-principles'> + <title>Goals and Guiding Principles</title> + + <para> + Before you can reach your destination, you need to know + where you are going. + Here is an example list that you can use as a guide when + creating very small distributions: + <itemizedlist> + <listitem><para>Determine how much space you need + (e.g. a kernel that is 1 Mbyte or less and + a root filesystem that is 3 Mbytes or less). + </para></listitem> + <listitem><para>Find the areas that are currently + taking 90% of the space and concentrate on reducing + those areas. + </para></listitem> + <listitem><para>Do not create any difficult "hacks" + to achieve your goals.</para></listitem> + <listitem><para>Leverage the device-specific + options.</para></listitem> + <listitem><para>Work in a separate layer so that you + keep changes isolated. + For information on how to create layers, see + the "<link linkend='understanding-and-creating-layers'>Understanding and Creating Layers</link>" section. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='understand-what-gives-your-image-size'> + <title>Understand What Contributes to Your Image Size</title> + + <para> + It is easiest to have something to start with when creating + your own distribution. + You can use the Yocto Project out-of-the-box to create the + <filename>poky-tiny</filename> distribution. + Ultimately, you will want to make changes in your own + distribution that are likely modeled after + <filename>poky-tiny</filename>. <note> - You can change the - <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> - to not conflict. - </note></para> + To use <filename>poky-tiny</filename> in your build, + set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink> + variable in your + <filename>local.conf</filename> file to "poky-tiny" + as described in the + "<link linkend='creating-your-own-distribution'>Creating Your Own Distribution</link>" + section. + </note> + </para> - <para> - Furthermore, the configuration file must be located in the - current build directory in a directory named - <filename>multiconfig</filename> under the build's - <filename>conf</filename> directory where - <filename>local.conf</filename> resides. - The reason for this restriction is because the - <filename>BBPATH</filename> variable is not constructed - until the layers are parsed. - Consequently, using the configuration file as a - pre-configuration file is not possible unless it is - located in the current working directory. - </para></listitem> - <listitem><para> - <emphasis>Add the BitBake Multi-Config Variable to you Local Configuration File</emphasis>: - Use the - <filename>BBMULTICONFIG</filename> - variable in your <filename>conf/local.conf</filename> - configuration file to specify each separate configuration. - For example, the following line tells BitBake it should load - <filename>conf/multiconfig/configA.conf</filename>, - <filename>conf/multiconfig/configB.conf</filename>, and - <filename>conf/multiconfig/configC.conf</filename>. + <para> + Understanding some memory concepts will help you reduce the + system size. + Memory consists of static, dynamic, and temporary memory. + Static memory is the TEXT (code), DATA (initialized data + in the code), and BSS (uninitialized data) sections. + Dynamic memory represents memory that is allocated at runtime: + stacks, hash tables, and so forth. + Temporary memory is recovered after the boot process. + This memory consists of memory used for decompressing + the kernel and for the <filename>__init__</filename> + functions. + </para> + + <para> + To help you see where you currently are with kernel and root + filesystem sizes, you can use two tools found in the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> in + the <filename>scripts/tiny/</filename> directory: + <itemizedlist> + <listitem><para><filename>ksize.py</filename>: Reports + component sizes for the kernel build objects. + </para></listitem> + <listitem><para><filename>dirsize.py</filename>: Reports + component sizes for the root filesystem.</para></listitem> + </itemizedlist> + This next tool and command help you organize configuration + fragments and view file dependencies in a human-readable form: + <itemizedlist> + <listitem><para><filename>merge_config.sh</filename>: + Helps you manage configuration files and fragments + within the kernel. + With this tool, you can merge individual configuration + fragments together. + The tool allows you to make overrides and warns you + of any missing configuration options. + The tool is ideal for allowing you to iterate on + configurations, create minimal configurations, and + create configuration files for different machines + without having to duplicate your process.</para> + <para>The <filename>merge_config.sh</filename> script is + part of the Linux Yocto kernel Git repositories + (i.e. <filename>linux-yocto-3.14</filename>, + <filename>linux-yocto-3.10</filename>, + <filename>linux-yocto-3.8</filename>, and so forth) + in the + <filename>scripts/kconfig</filename> directory.</para> + <para>For more information on configuration fragments, + see the + "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#creating-config-fragments'>Creating Configuration Fragments</ulink>" + section in the Yocto Project Linux Kernel Development + Manual. + </para></listitem> + <listitem><para><filename>bitbake -u taskexp -g <replaceable>bitbake_target</replaceable></filename>: + Using the BitBake command with these options brings up + a Dependency Explorer from which you can view file + dependencies. + Understanding these dependencies allows you to make + informed decisions when cutting out various pieces of the + kernel and root filesystem.</para></listitem> + </itemizedlist> + </para> + </section> + + <section id='trim-the-root-filesystem'> + <title>Trim the Root Filesystem</title> + + <para> + The root filesystem is made up of packages for booting, + libraries, and applications. + To change things, you can configure how the packaging happens, + which changes the way you build them. + You can also modify the filesystem itself or select a different + filesystem. + </para> + + <para> + First, find out what is hogging your root filesystem by running the + <filename>dirsize.py</filename> script from your root directory: <literallayout class='monospaced'> - BBMULTICONFIG = "configA configB configC" + $ cd <replaceable>root-directory-of-image</replaceable> + $ dirsize.py 100000 > dirsize-100k.log + $ cat dirsize-100k.log </literallayout> - </para></listitem> - <listitem><para> - <emphasis>Launch BitBake</emphasis>: - Use the following BitBake command form to launch the - build: + You can apply a filter to the script to ignore files under + a certain size. + The previous example filters out any files below 100 Kbytes. + The sizes reported by the tool are uncompressed, and thus + will be smaller by a relatively constant factor in a + compressed root filesystem. + When you examine your log file, you can focus on areas of the + root filesystem that take up large amounts of memory. + </para> + + <para> + You need to be sure that what you eliminate does not cripple + the functionality you need. + One way to see how packages relate to each other is by using + the Dependency Explorer UI with the BitBake command: <literallayout class='monospaced'> - $ bitbake [multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable> [[[multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>] ... ] + $ cd <replaceable>image-directory</replaceable> + $ bitbake -u taskexp -g <replaceable>image</replaceable> </literallayout> - Following is an example that supports building a minimal - image for configuration A alongside a standard - <filename>core-image-sato</filename>, which takes its - configuration from <filename>local.conf</filename>: + Use the interface to select potential packages you wish to + eliminate and see their dependency relationships. + </para> + + <para> + When deciding how to reduce the size, get rid of packages that + result in minimal impact on the feature set. + For example, you might not need a VGA display. + Or, you might be able to get by with <filename>devtmpfs</filename> + and <filename>mdev</filename> instead of + <filename>udev</filename>. + </para> + + <para> + Use your <filename>local.conf</filename> file to make changes. + For example, to eliminate <filename>udev</filename> and + <filename>glib</filename>, set the following in the + local configuration file: <literallayout class='monospaced'> - $ bitbake multiconfig:configA:core-image-minimal core-image-sato + VIRTUAL-RUNTIME_dev_manager = "" + </literallayout> + </para> + + <para> + Finally, you should consider exactly the type of root + filesystem you need to meet your needs while also reducing + its size. + For example, consider <filename>cramfs</filename>, + <filename>squashfs</filename>, <filename>ubifs</filename>, + <filename>ext2</filename>, or an <filename>initramfs</filename> + using <filename>initramfs</filename>. + Be aware that <filename>ext3</filename> requires a 1 Mbyte + journal. + If you are okay with running read-only, you do not need this + journal. + </para> + + <note> + After each round of elimination, you need to rebuild your + system and then use the tools to see the effects of your + reductions. + </note> + </section> + + <section id='trim-the-kernel'> + <title>Trim the Kernel</title> + + <para> + The kernel is built by including policies for hardware-independent + aspects. + What subsystems do you enable? + For what architecture are you building? + Which drivers do you build by default? + <note>You can modify the kernel source if you want to help + with boot time. + </note> + </para> + + <para> + Run the <filename>ksize.py</filename> script from the top-level + Linux build directory to get an idea of what is making up + the kernel: + <literallayout class='monospaced'> + $ cd <replaceable>top-level-linux-build-directory</replaceable> + $ ksize.py > ksize.log + $ cat ksize.log </literallayout> + When you examine the log, you will see how much space is + taken up with the built-in <filename>.o</filename> files for + drivers, networking, core kernel files, filesystem, sound, + and so forth. + The sizes reported by the tool are uncompressed, and thus + will be smaller by a relatively constant factor in a compressed + kernel image. + Look to reduce the areas that are large and taking up around + the "90% rule." + </para> + + <para> + To examine, or drill down, into any particular area, use the + <filename>-d</filename> option with the script: + <literallayout class='monospaced'> + $ ksize.py -d > ksize.log + </literallayout> + Using this option breaks out the individual file information + for each area of the kernel (e.g. drivers, networking, and + so forth). + </para> + + <para> + Use your log file to see what you can eliminate from the kernel + based on features you can let go. + For example, if you are not going to need sound, you do not + need any drivers that support sound. + </para> + + <para> + After figuring out what to eliminate, you need to reconfigure + the kernel to reflect those changes during the next build. + You could run <filename>menuconfig</filename> and make all your + changes at once. + However, that makes it difficult to see the effects of your + individual eliminations and also makes it difficult to replicate + the changes for perhaps another target device. + A better method is to start with no configurations using + <filename>allnoconfig</filename>, create configuration + fragments for individual changes, and then manage the + fragments into a single configuration file using + <filename>merge_config.sh</filename>. + The tool makes it easy for you to iterate using the + configuration change and build cycle. + </para> + + <para> + Each time you make configuration changes, you need to rebuild + the kernel and check to see what impact your changes had on + the overall size. + </para> + </section> + + <section id='remove-package-management-requirements'> + <title>Remove Package Management Requirements</title> + + <para> + Packaging requirements add size to the image. + One way to reduce the size of the image is to remove all the + packaging requirements from the image. + This reduction includes both removing the package manager + and its unique dependencies as well as removing the package + management data itself. + </para> + + <para> + To eliminate all the packaging requirements for an image, + be sure that "package-management" is not part of your + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink> + statement for the image. + When you remove this feature, you are removing the package + manager as well as its dependencies from the root filesystem. + </para> + </section> + + <section id='look-for-other-ways-to-minimize-size'> + <title>Look for Other Ways to Minimize Size</title> + + <para> + Depending on your particular circumstances, other areas that you + can trim likely exist. + The key to finding these areas is through tools and methods + described here combined with experimentation and iteration. + Here are a couple of areas to experiment with: + <itemizedlist> + <listitem><para><filename>glibc</filename>: + In general, follow this process: + <orderedlist> + <listitem><para>Remove <filename>glibc</filename> + features from + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></ulink> + that you think you do not need.</para></listitem> + <listitem><para>Build your distribution. + </para></listitem> + <listitem><para>If the build fails due to missing + symbols in a package, determine if you can + reconfigure the package to not need those + features. + For example, change the configuration to not + support wide character support as is done for + <filename>ncurses</filename>. + Or, if support for those characters is needed, + determine what <filename>glibc</filename> + features provide the support and restore the + configuration. + </para></listitem> + <listitem><para>Rebuild and repeat the process. + </para></listitem> + </orderedlist></para></listitem> + <listitem><para><filename>busybox</filename>: + For BusyBox, use a process similar as described for + <filename>glibc</filename>. + A difference is you will need to boot the resulting + system to see if you are able to do everything you + expect from the running system. + You need to be sure to integrate configuration fragments + into Busybox because BusyBox handles its own core + features and then allows you to add configuration + fragments on top. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='iterate-on-the-process'> + <title>Iterate on the Process</title> + + <para> + If you have not reached your goals on system size, you need + to iterate on the process. + The process is the same. + Use the tools and see just what is taking up 90% of the root + filesystem and the kernel. + Decide what you can eliminate without limiting your device + beyond what you need. + </para> + + <para> + Depending on your system, a good place to look might be + Busybox, which provides a stripped down + version of Unix tools in a single, executable file. + You might be able to drop virtual terminal services or perhaps + ipv6. + </para> + </section> + </section> + + <section id='building-images-for-more-than-one-machine'> + <title>Building Images for More than One Machine</title> + + <para> + A common scenario developers face is creating images for several + different machines that use the same software environment. + In this situation, it is tempting to set the + tunings and optimization flags for each build specifically for + the targeted hardware (i.e. "maxing out" the tunings). + Doing so can considerably add to build times and package feed + maintenance collectively for the machines. + For example, selecting tunes that are extremely specific to a + CPU core used in a system might enable some micro optimizations + in GCC for that particular system but would otherwise not gain + you much of a performance difference across the other systems + as compared to using a more general tuning across all the builds + (e.g. setting + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEFAULTTUNE'><filename>DEFAULTTUNE</filename></ulink> + specifically for each machine's build). + Rather than "max out" each build's tunings, you can take steps that + cause the OpenEmbedded build system to reuse software across the + various machines where it makes sense. + </para> + + <para> + If build speed and package feed maintenance are considerations, + you should consider the points in this section that can help you + optimize your tunings to best consider build times and package + feed maintenance. + <itemizedlist> + <listitem><para> + <emphasis>Share the Build Directory:</emphasis> + If at all possible, share the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> + across builds. + The Yocto Project supports switching between different + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + values in the same <filename>TMPDIR</filename>. + This practice is well supported and regularly used by + developers when building for multiple machines. + When you use the same <filename>TMPDIR</filename> for + multiple machine builds, the OpenEmbedded build system can + reuse the existing native and often cross-recipes for + multiple machines. + Thus, build time decreases. + <note> + If + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink> + settings change or fundamental configuration settings + such as the filesystem layout, you need to work with + a clean <filename>TMPDIR</filename>. + Sharing <filename>TMPDIR</filename> under these + circumstances might work but since it is not + guaranteed, you should use a clean + <filename>TMPDIR</filename>. + </note> + </para></listitem> + <listitem><para> + <emphasis>Enable the Appropriate Package Architecture:</emphasis> + By default, the OpenEmbedded build system enables three + levels of package architectures: "all", "tune" or "package", + and "machine". + Any given recipe usually selects one of these package + architectures (types) for its output. + Depending for what a given recipe creates packages, making + sure you enable the appropriate package architecture can + directly impact the build time.</para> + + <para>A recipe that just generates scripts can enable + "all" architecture because there are no binaries to build. + To specifically enable "all" architecture, be sure your + recipe inherits the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-allarch'><filename>allarch</filename></ulink> + class. + This class is useful for "all" architectures because it + configures many variables so packages can be used across + multiple architectures.</para> + + <para>If your recipe needs to generate packages that are + machine-specific or when one of the build or runtime + dependencies is already machine-architecture dependent, + which makes your recipe also machine-architecture dependent, + make sure your recipe enables the "machine" package + architecture through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE_ARCH'><filename>MACHINE_ARCH</filename></ulink> + variable: + <literallayout class='monospaced'> + PACKAGE_ARCH = "${MACHINE_ARCH}" + </literallayout> + When you do not specifically enable a package + architecture through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>, + The OpenEmbedded build system defaults to the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TUNE_PKGARCH'><filename>TUNE_PKGARCH</filename></ulink> + setting: + <literallayout class='monospaced'> + PACKAGE_ARCH = "${TUNE_PKGARCH}" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Choose a Generic Tuning File if Possible:</emphasis> + Some tunes are more generic and can run on multiple targets + (e.g. an <filename>armv5</filename> set of packages could + run on <filename>armv6</filename> and + <filename>armv7</filename> processors in most cases). + Similarly, <filename>i486</filename> binaries could work + on <filename>i586</filename> and higher processors. + You should realize, however, that advances on newer + processor versions would not be used.</para> + + <para>If you select the same tune for several different + machines, the OpenEmbedded build system reuses software + previously built, thus speeding up the overall build time. + Realize that even though a new sysroot for each machine is + generated, the software is not recompiled and only one + package feed exists. + </para></listitem> + <listitem><para> + <emphasis>Manage Granular Level Packaging:</emphasis> + Sometimes cases exist where injecting another level of + package architecture beyond the three higher levels noted + earlier can be useful. + For example, consider how NXP (formerly Freescale) allows + for the easy reuse of binary packages in their layer + <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/meta-freescale/'><filename>meta-freescale</filename></ulink>. + In this example, the + <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass'><filename>fsl-dynamic-packagearch</filename></ulink> + class shares GPU packages for i.MX53 boards because + all boards share the AMD GPU. + The i.MX6-based boards can do the same because all boards + share the Vivante GPU. + This class inspects the BitBake datastore to identify if + the package provides or depends on one of the + sub-architecture values. + If so, the class sets the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink> + value based on the <filename>MACHINE_SUBARCH</filename> + value. + If the package does not provide or depend on one of the + sub-architecture values but it matches a value in the + machine-specific filter, it sets + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE_ARCH'><filename>MACHINE_ARCH</filename></ulink>. + This behavior reduces the number of packages built and + saves build time by reusing binaries. + </para></listitem> + <listitem><para> + <emphasis>Use Tools to Debug Issues:</emphasis> + Sometimes you can run into situations where software is + being rebuilt when you think it should not be. + For example, the OpenEmbedded build system might not be + using shared state between machines when you think it + should be. + These types of situations are usually due to references + to machine-specific variables such as + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-SERIAL_CONSOLES'><filename>SERIAL_CONSOLES</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-XSERVER'><filename>XSERVER</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE_FEATURES'><filename>MACHINE_FEATURES</filename></ulink>, + and so forth in code that is supposed to only be + tune-specific or when the recipe depends + (<ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-RSUGGESTS'><filename>RSUGGESTS</filename></ulink>, + and so forth) on some other recipe that already has + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink> + defined as "${MACHINE_ARCH}". + <note> + Patches to fix any issues identified are most welcome + as these issues occasionally do occur. + </note></para> + + <para>For such cases, you can use some tools to help you + sort out the situation: + <itemizedlist> + <listitem><para> + <emphasis><filename>sstate-diff-machines.sh</filename>:</emphasis> + You can find this tool in the + <filename>scripts</filename> directory of the + Source Repositories. + See the comments in the script for information on + how to use the tool. + </para></listitem> + <listitem><para> + <emphasis>BitBake's "-S printdiff" Option:</emphasis> + Using this option causes BitBake to try to + establish the closest signature match it can + (e.g. in the shared state cache) and then run + <filename>bitbake-diffsigs</filename> over the + matches to determine the stamps and delta where + these two stamp trees diverge. + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id="building-software-from-an-external-source"> + <title>Building Software from an External Source</title> + + <para> + By default, the OpenEmbedded build system uses the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> + when building source code. + The build process involves fetching the source files, unpacking + them, and then patching them if necessary before the build takes + place. + </para> + + <para> + Situations exist where you might want to build software from source + files that are external to and thus outside of the + OpenEmbedded build system. + For example, suppose you have a project that includes a new BSP with + a heavily customized kernel. + And, you want to minimize exposing the build system to the + development team so that they can focus on their project and + maintain everyone's workflow as much as possible. + In this case, you want a kernel source directory on the development + machine where the development occurs. + You want the recipe's + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + variable to point to the external directory and use it as is, not + copy it. + </para> + + <para> + To build from software that comes from an external source, all you + need to do is inherit the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink> + class and then set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTERNALSRC'><filename>EXTERNALSRC</filename></ulink> + variable to point to your external source code. + Here are the statements to put in your + <filename>local.conf</filename> file: + <literallayout class='monospaced'> + INHERIT += "externalsrc" + EXTERNALSRC_pn-<replaceable>myrecipe</replaceable> = "<replaceable>path-to-your-source-tree</replaceable>" + </literallayout> + </para> + + <para> + This next example shows how to accomplish the same thing by setting + <filename>EXTERNALSRC</filename> in the recipe itself or in the + recipe's append file: + <literallayout class='monospaced'> + EXTERNALSRC = "<replaceable>path</replaceable>" + EXTERNALSRC_BUILD = "<replaceable>path</replaceable>" + </literallayout> + <note> + In order for these settings to take effect, you must globally + or locally inherit the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink> + class. + </note> + </para> + + <para> + By default, <filename>externalsrc.bbclass</filename> builds + the source code in a directory separate from the external source + directory as specified by + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTERNALSRC'><filename>EXTERNALSRC</filename></ulink>. + If you need to have the source built in the same directory in + which it resides, or some other nominated directory, you can set + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTERNALSRC_BUILD'><filename>EXTERNALSRC_BUILD</filename></ulink> + to point to that directory: + <literallayout class='monospaced'> + EXTERNALSRC_BUILD_pn-<replaceable>myrecipe</replaceable> = "<replaceable>path-to-your-source-tree</replaceable>" + </literallayout> + </para> + </section> + </section> + + + + <section id='speeding-up-a-build'> + <title>Speeding Up a Build</title> + + <para> + Build time can be an issue. + By default, the build system uses simple controls to try and maximize + build efficiency. + In general, the default settings for all the following variables + result in the most efficient build times when dealing with single + socket systems (i.e. a single CPU). + If you have multiple CPUs, you might try increasing the default + values to gain more speed. + See the descriptions in the glossary for each variable for more + information: + <itemizedlist> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename>:</ulink> + The maximum number of threads BitBake simultaneously executes. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_NUMBER_PARSE_THREADS'><filename>BB_NUMBER_PARSE_THREADS</filename>:</ulink> + The number of threads BitBake uses during parsing. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename>:</ulink> + Extra options passed to the <filename>make</filename> command + during the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink> + task in order to specify parallel compilation on the + local build host. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKEINST'><filename>PARALLEL_MAKEINST</filename>:</ulink> + Extra options passed to the <filename>make</filename> command + during the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> + task in order to specify parallel installation on the + local build host. </para></listitem> </itemizedlist> + As mentioned, these variables all scale to the number of processor + cores available on the build system. + For single socket systems, this auto-scaling ensures that the build + system fundamentally takes advantage of potential parallel operations + during the build based on the build machine's capabilities. </para> <para> - Support for multiple configurations in this current release of - the Yocto Project (&DISTRO_NAME; &DISTRO;) has some known issues: + Following are additional factors that can affect build speed: <itemizedlist> <listitem><para> - No inter-multi-configuration dependencies exist. + File system type: + The file system type that the build is being performed on can + also influence performance. + Using <filename>ext4</filename> is recommended as compared + to <filename>ext2</filename> and <filename>ext3</filename> + due to <filename>ext4</filename> improved features + such as extents. + </para></listitem> + <listitem><para> + Disabling the updating of access time using + <filename>noatime</filename>: + The <filename>noatime</filename> mount option prevents the + build system from updating file and directory access times. + </para></listitem> + <listitem><para> + Setting a longer commit: + Using the "commit=" mount option increases the interval + in seconds between disk cache writes. + Changing this interval from the five second default to + something longer increases the risk of data loss but decreases + the need to write to the disk, thus increasing the build + performance. + </para></listitem> + <listitem><para> + Choosing the packaging backend: + Of the available packaging backends, IPK is the fastest. + Additionally, selecting a singular packaging backend also + helps. + </para></listitem> + <listitem><para> + Using <filename>tmpfs</filename> for + <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> + as a temporary file system: + While this can help speed up the build, the benefits are + limited due to the compiler using + <filename>-pipe</filename>. + The build system goes to some lengths to avoid + <filename>sync()</filename> calls into the + file system on the principle that if there was a significant + failure, the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> + contents could easily be rebuilt. </para></listitem> <listitem><para> - Shared State (sstate) optimizations do not exist. - Consequently, if the build uses the same object twice - in, for example, two different - <filename>TMPDIR</filename> directories, the build - will either load from an existing sstate cache at the - start or build the object twice. + Inheriting the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-rm-work'><filename>rm_work</filename></ulink> + class: + Inheriting this class has shown to speed up builds due to + significantly lower amounts of data stored in the data + cache as well as on disk. + Inheriting this class also makes cleanup of + <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> + faster, at the expense of being easily able to dive into the + source code. + File system maintainers have recommended that the fastest way + to clean up large numbers of files is to reformat partitions + rather than delete files due to the linear nature of + partitions. + This, of course, assumes you structure the disk partitions and + file systems in a way that this is practical. </para></listitem> </itemizedlist> + Aside from the previous list, you should keep some trade offs in + mind that can help you speed up the build: + <itemizedlist> + <listitem><para> + Remove items from + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></ulink> + that you might not need. + </para></listitem> + <listitem><para> + Exclude debug symbols and other debug information: + If you do not need these symbols and other debug information, + disabling the <filename>*-dbg</filename> package generation + can speed up the build. + You can disable this generation by setting the + <ulink url='&YOCTO_DOCS_REF_URL;#var-INHIBIT_PACKAGE_DEBUG_SPLIT'><filename>INHIBIT_PACKAGE_DEBUG_SPLIT</filename></ulink> + variable to "1". + </para></listitem> + <listitem><para> + Disable static library generation for recipes derived from + <filename>autoconf</filename> or <filename>libtool</filename>: + Following is an example showing how to disable static + libraries and still provide an override to handle exceptions: + <literallayout class='monospaced'> + STATICLIBCONF = "--disable-static" + STATICLIBCONF_sqlite3-native = "" + EXTRA_OECONF += "${STATICLIBCONF}" + </literallayout> + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + Some recipes need static libraries in order to work + correctly (e.g. <filename>pseudo-native</filename> + needs <filename>sqlite3-native</filename>). + Overrides, as in the previous example, account for + these kinds of exceptions. + </para></listitem> + <listitem><para> + Some packages have packaging code that assumes the + presence of the static libraries. + If so, you might need to exclude them as well. + </para></listitem> + </itemizedlist> + </note> + </para></listitem> + </itemizedlist> </para> </section> @@ -4543,6 +6723,84 @@ </section> </section> + <section id='using-x32-psabi'> + <title>Using x32 psABI</title> + + <para> + x32 processor-specific Application Binary Interface + (<ulink url='https://software.intel.com/en-us/node/628948'>x32 psABI</ulink>) + is a native 32-bit processor-specific ABI for + <trademark class='registered'>Intel</trademark> 64 (x86-64) + architectures. + An ABI defines the calling conventions between functions in a + processing environment. + The interface determines what registers are used and what the + sizes are for various C data types. + </para> + + <para> + Some processing environments prefer using 32-bit applications even + when running on Intel 64-bit platforms. + Consider the i386 psABI, which is a very old 32-bit ABI for Intel + 64-bit platforms. + The i386 psABI does not provide efficient use and access of the + Intel 64-bit processor resources, leaving the system underutilized. + Now consider the x86_64 psABI. + This ABI is newer and uses 64-bits for data sizes and program + pointers. + The extra bits increase the footprint size of the programs, + libraries, and also increases the memory and file system size + requirements. + Executing under the x32 psABI enables user programs to utilize CPU + and system resources more efficiently while keeping the memory + footprint of the applications low. + Extra bits are used for registers but not for addressing mechanisms. + </para> + + <para> + The Yocto Project supports the final specifications of x32 psABI + as follows: + <itemizedlist> + <listitem><para> + You can create packages and images in x32 psABI format on + x86_64 architecture targets. + </para></listitem> + <listitem><para> + You can successfully build recipes with the x32 toolchain. + </para></listitem> + <listitem><para> + You can create and boot + <filename>core-image-minimal</filename> and + <filename>core-image-sato</filename> images. + </para></listitem> + <listitem><para> + RPM Package Manager (RPM) support exists for x32 binaries. + </para></listitem> + <listitem><para> + Support for large images exists. + </para></listitem> + </itemizedlist> + </para> + + <para> + To use the x32 psABI, you need to edit your + <filename>conf/local.conf</filename> configuration file as + follows: + <literallayout class='monospaced'> + MACHINE = "qemux86-64" + DEFAULTTUNE = "x86-64-x32" + baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE', True) \ + or 'INVALID'), True) or 'lib'}" + </literallayout> + Once you have set up your configuration file, use BitBake to + build an image that supports the x32 psABI. + Here is an example: + <literallayout class='monospaced'> + $ bitbake core-image-sato + </literallayout> + </para> + </section> + <section id='enabling-gobject-introspection-support'> <title>Enabling GObject Introspection Support</title> @@ -4795,8 +7053,7 @@ variable. </para></listitem> <listitem><para> - Set the - <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTERNAL_TOOLCHAIN'><filename>EXTERNAL_TOOLCHAIN</filename></ulink> + Set the <filename>EXTERNAL_TOOLCHAIN</filename> variable in your <filename>local.conf</filename> file to the location in which you installed the toolchain. </para></listitem> @@ -4846,7 +7103,7 @@ system. <note> For a kickstart file reference, see the - "<ulink url='&YOCTO_DOCS_REF_URL;#openembedded-kickstart-wks-reference'>OpenEmbedded Kickstart (<filename>.wks</filename>) Reference</ulink>" + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-kickstart'>OpenEmbedded Kickstart (<filename>.wks</filename>) Reference</ulink>" Chapter in the Yocto Project Reference Manual. </note> </para> @@ -4858,16 +7115,17 @@ customized images, and as such, was designed to be completely extensible through a plug-in interface. See the - "<ulink url='&YOCTO_DOCS_REF_URL;#wic-plug-ins-interface'>Wic Plug-Ins Interface</ulink>" - section in the Yocto Project Reference Manual for information - on these plug-ins. + "<link linkend='wic-using-the-wic-plug-ins-interface'>Using the Wic Plug-Ins Interface</link>" + section for information on these plug-ins. </para> <para> This section provides some background information on Wic, describes what you need to have in place to run the tool, provides instruction on how to use - the Wic utility, and provides several examples. + the Wic utility, provides information on using the Wic plug-ins + interface, and provides several examples that show how to use + Wic. </para> <section id='wic-background'> @@ -4899,7 +7157,7 @@ Wic is a completely independent standalone utility that initially provides easier-to-use and more flexible replacements for an - existing functionality in OE Core's + existing functionality in OE-Core's <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-image-live'><filename>image-live</filename></ulink> class and <filename>mkefidisk.sh</filename> script. The difference between @@ -4988,6 +7246,7 @@ <literallayout class='monospaced'> $ wic -h $ wic --help + $ wic help </literallayout> </para> @@ -4997,51 +7256,66 @@ <filename>help</filename>, <filename>list</filename>, <filename>ls</filename>, <filename>rm</filename>, and <filename>write</filename>. - You can get help for these commands as follows with - <replaceable>command</replaceable> being one of the - supported commands: + You can get help for all these commands except "help" by + using the following form: <literallayout class='monospaced'> $ wic help <replaceable>command</replaceable> </literallayout> + For example, the following command returns help for the + <filename>write</filename> command: + <literallayout class='monospaced'> + $ wic help write + </literallayout> </para> <para> - You can also get detailed help on a number of topics - from the help system. - The output of <filename>wic --help</filename> - displays a list of available help - topics under a "Help topics" heading. - You can have the help system display the help text for - a given topic by prefacing the topic with - <filename>wic help</filename>: + Wic supports help for three topics: + <filename>overview</filename>, + <filename>plugins</filename>, and + <filename>kickstart</filename>. + You can get help for any topic using the following form: + <literallayout class='monospaced'> + $ wic help <replaceable>topic</replaceable> + </literallayout> + For example, the following returns overview help for Wic: <literallayout class='monospaced'> - $ wic help <replaceable>help_topic</replaceable> + $ wic help overview </literallayout> </para> <para> - You can find out more about the images Wic creates using - the existing kickstart files with the following form of - the command: + One additional level of help exists for Wic. + You can get help on individual images through the + <filename>list</filename> command. + You can use the <filename>list</filename> command to return the + available Wic images as follows: <literallayout class='monospaced'> - $ wic list <replaceable>image</replaceable> help + $ wic list images + mpc8315e-rdb Create SD card image for MPC8315E-RDB + genericx86 Create an EFI disk image for genericx86* + beaglebone-yocto Create SD card image for Beaglebone + edgerouter Create SD card image for Edgerouter + qemux86-directdisk Create a qemu machine 'pcbios' direct disk image + directdisk-gpt Create a 'pcbios' direct disk image + mkefidisk Create an EFI disk image + directdisk Create a 'pcbios' direct disk image + systemd-bootdisk Create an EFI disk image with systemd-boot + mkhybridiso Create a hybrid ISO image + sdimage-bootpart Create SD card image with a boot partition + directdisk-multi-rootfs Create multi rootfs image using rootfs plugin + directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config </literallayout> - For <replaceable>image</replaceable>, you can provide - any of the following: + Once you know the list of available Wic images, you can use + <filename>help</filename> with the command to get help on a + particular image. + For example, the following command returns help on the + "beaglebone-yocto" image: <literallayout class='monospaced'> - beaglebone - mpc8315e-rdb - genericx86 - edgerouter - qemux86-directdisk - directdisk-gpt - mkefidisk - directdisk - systemd-bootdisk - mkhybridiso - sdimage-bootpart - directdisk-multi-rootfs - directdisk-bootloader-config + $ wic list beaglebone-yocto help + + + Creates a partitioned SD card image for Beaglebone. + Boot files are located in the first vfat partition. </literallayout> </para> </section> @@ -5058,7 +7332,7 @@ <listitem><para> <emphasis>Raw Mode:</emphasis> You explicitly specify build artifacts through - <filename>wic</filename> command-line arguments. + Wic command-line arguments. </para></listitem> <listitem><para> <emphasis>Cooked Mode:</emphasis> @@ -5153,7 +7427,7 @@ <para> Running Wic in cooked mode leverages off artifacts in - Build Directory. + the Build Directory. In other words, you do not have to specify kernel or root filesystem locations as part of the command. All you need to provide is a kickstart file and the @@ -5195,7 +7469,7 @@ can use an existing file provided by the Wic installation. As shipped, kickstart files can be found in the Yocto Project - <ulink url='&YOCTO_DOCS_REF_URL;#source-repositories'>Source Repositories</ulink> + <ulink url='&YOCTO_DOCS_OM_URL;#source-repositories'>Source Repositories</ulink> in the following two locations: <literallayout class='monospaced'> poky/meta-yocto-bsp/wic @@ -5205,9 +7479,9 @@ files: <literallayout class='monospaced'> $ wic list images - beaglebone Create SD card image for Beaglebone mpc8315e-rdb Create SD card image for MPC8315E-RDB genericx86 Create an EFI disk image for genericx86* + beaglebone-yocto Create SD card image for Beaglebone edgerouter Create SD card image for Edgerouter qemux86-directdisk Create a qemu machine 'pcbios' direct disk image directdisk-gpt Create a 'pcbios' direct disk image @@ -5245,6 +7519,209 @@ </para> </section> + <section id='wic-using-the-wic-plug-ins-interface'> + <title>Using the Wic Plug-Ins Interface</title> + + <para> + You can extend and specialize Wic functionality by using + Wic plug-ins. + This section explains the Wic plug-in interface. + <note> + Wic plug-ins consist of "source" and "imager" plug-ins. + Imager plug-ins are beyond the scope of this section. + </note> + </para> + + <para> + Source plug-ins provide a mechanism to customize partition + content during the Wic image generation process. + You can use source plug-ins to map values that you specify + using <filename>--source</filename> commands in kickstart + files (i.e. <filename>*.wks</filename>) to a plug-in + implementation used to populate a given partition. + <note> + If you use plug-ins that have build-time dependencies + (e.g. native tools, bootloaders, and so forth) + when building a Wic image, you need to specify those + dependencies using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-WKS_FILE_DEPENDS'><filename>WKS_FILE_DEPENDS</filename></ulink> + variable. + </note> + </para> + + <para> + Source plug-ins are subclasses defined in plug-in files. + As shipped, the Yocto Project provides several plug-in + files. + You can see the source plug-in files that ship with the + Yocto Project + <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/scripts/lib/wic/plugins/source'>here</ulink>. + Each of these plug-in files contains source plug-ins that + are designed to populate a specific Wic image partition. + </para> + + <para> + Source plug-ins are subclasses of the + <filename>SourcePlugin</filename> class, which is + defined in the + <filename>poky/scripts/lib/wic/pluginbase.py</filename> + file. + For example, the <filename>BootimgEFIPlugin</filename> + source plug-in found in the + <filename>bootimg-efi.py</filename> file is a subclass of + the <filename>SourcePlugin</filename> class, which is found + in the <filename>pluginbase.py</filename> file. + </para> + + <para> + You can also implement source plug-ins in a layer outside + of the Source Repositories (external layer). + To do so, be sure that your plug-in files are located in + a directory whose path is + <filename>scripts/lib/wic/plugins/source/</filename> + within your external layer. + When the plug-in files are located there, the source + plug-ins they contain are made available to Wic. + </para> + + <para> + When the Wic implementation needs to invoke a + partition-specific implementation, it looks for the plug-in + with the same name as the <filename>--source</filename> + parameter used in the kickstart file given to that + partition. + For example, if the partition is set up using the following + command in a kickstart file: + <literallayout class='monospaced'> + part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024 + </literallayout> + The methods defined as class members of the matching + source plug-in (i.e. <filename>bootimg-pcbios</filename>) + in the <filename>bootimg-pcbios.py</filename> plug-in file + are used. + </para> + + <para> + To be more concrete, here is the corresponding plug-in + definition from the <filename>bootimg-pcbios.py</filename> + file for the previous command along with an example + method called by the Wic implementation when it needs to + prepare a partition using an implementation-specific + function: + <literallayout class='monospaced'> + . + . + . + class BootimgPcbiosPlugin(SourcePlugin): + """ + Create MBR boot partition and install syslinux on it. + """ + + name = 'bootimg-pcbios' + . + . + . + @classmethod + def do_prepare_partition(cls, part, source_params, creator, cr_workdir, + oe_builddir, bootimg_dir, kernel_dir, + rootfs_dir, native_sysroot): + """ + Called to do the actual content population for a partition i.e. it + 'prepares' the partition to be incorporated into the image. + In this case, prepare content for legacy bios boot partition. + """ + . + . + . + </literallayout> + If a subclass (plug-in) itself does not implement a + particular function, Wic locates and uses the default + version in the superclass. + It is for this reason that all source plug-ins are derived + from the <filename>SourcePlugin</filename> class. + </para> + + <para> + The <filename>SourcePlugin</filename> class defined in + the <filename>pluginbase.py</filename> file defines + a set of methods that source plug-ins can implement or + override. + Any plug-ins (subclass of + <filename>SourcePlugin</filename>) that do not implement + a particular method inherit the implementation of the + method from the <filename>SourcePlugin</filename> class. + For more information, see the + <filename>SourcePlugin</filename> class in the + <filename>pluginbase.py</filename> file for details: + </para> + + <para> + The following list describes the methods implemented in the + <filename>SourcePlugin</filename> class: + <itemizedlist> + <listitem><para> + <emphasis><filename>do_prepare_partition()</filename>:</emphasis> + Called to populate a partition with actual content. + In other words, the method prepares the final + partition image that is incorporated into the + disk image. + </para></listitem> + <listitem><para> + <emphasis><filename>do_configure_partition()</filename>:</emphasis> + Called before + <filename>do_prepare_partition()</filename> to + create custom configuration files for a partition + (e.g. syslinux or grub configuration files). + </para></listitem> + <listitem><para> + <emphasis><filename>do_install_disk()</filename>:</emphasis> + Called after all partitions have been prepared and + assembled into a disk image. + This method provides a hook to allow finalization + of a disk image (e.g. writing an MBR). + </para></listitem> + <listitem><para> + <emphasis><filename>do_stage_partition()</filename>:</emphasis> + Special content-staging hook called before + <filename>do_prepare_partition()</filename>. + This method is normally empty.</para> + + <para>Typically, a partition just uses the passed-in + parameters (e.g. the unmodified value of + <filename>bootimg_dir</filename>). + However, in some cases, things might need to be + more tailored. + As an example, certain files might additionally + need to be taken from + <filename>bootimg_dir + /boot</filename>. + This hook allows those files to be staged in a + customized fashion. + <note> + <filename>get_bitbake_var()</filename> + allows you to access non-standard variables + that you might want to use for this + behavior. + </note> + </para></listitem> + </itemizedlist> + </para> + + <para> + You can extend the source plug-in mechanism. + To add more hooks, create more source plug-in methods + within <filename>SourcePlugin</filename> and the + corresponding derived subclasses. + The code that calls the plug-in methods uses the + <filename>plugin.get_source_plugin_methods()</filename> + function to find the method or methods needed by the call. + Retrieval of those methods is accomplished by filling up + a dict with keys that contain the method names of interest. + On success, these will be filled in with the actual + methods. + See the Wic implementation for examples and details. + </para> + </section> + <section id='wic-usage-examples'> <title>Examples</title> @@ -5271,16 +7748,16 @@ . . INFO: The new image(s) can be found here: - ./mkefidisk-201710061409-sda.direct + ./mkefidisk-201804191017-sda.direct The following build artifacts were used to create the image(s): - ROOTFS_DIR: /home/scottrif/poky/build/tmp.wic.r4hkds0b/rootfs_copy - BOOTIMG_DIR: /home/scottrif/poky/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share - KERNEL_DIR: /home/scottrif/poky/build/tmp/deploy/images/qemux86 - NATIVE_SYSROOT: /home/scottrif/poky/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native + ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs + BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share + KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86 + NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native INFO: The image(s) were created using OE kickstart file: - /home/scottrif/poky/scripts/lib/wic/canned-wks/mkefidisk.wks + /home/stephano/build/master/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks </literallayout> The previous example shows the easiest way to create an image by running in cooked mode and supplying @@ -5305,17 +7782,18 @@ <para> Continuing with the example, you can now write the - image to a USB stick, or whatever media for which you - built your image, and boot from the media. + image from the Build Directory onto a USB stick, or + whatever media for which you built your image, and boot + from the media. You can write the image by using <filename>bmaptool</filename> or <filename>dd</filename>: <literallayout class='monospaced'> - $ oe-run-native bmaptool copy build/mkefidisk-201710061409-sda.direct /dev/sd<replaceable>X</replaceable> + $ oe-run-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sd<replaceable>X</replaceable> </literallayout> or <literallayout class='monospaced'> - $ sudo dd if=build/mkefidisk-201710061409-sda.direct of=/dev/sd<replaceable>X</replaceable> + $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sd<replaceable>X</replaceable> </literallayout> <note> For more information on how to use the @@ -5375,8 +7853,8 @@ directory and then by changing the lines that specify the target disk from which to boot. <literallayout class='monospaced'> - $ cp /home/scottrif/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \ - /home/scottrif/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks + $ cp /home/stephano/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \ + /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks </literallayout> Next, the example modifies the <filename>directdisksdb-gpt.wks</filename> file and @@ -5412,13 +7890,13 @@ ./directdisksdb-gpt-201710090938-sdb.direct The following build artifacts were used to create the image(s): - ROOTFS_DIR: /home/scottrif/poky/build/tmp.wic.hk3wl6zn/rootfs_copy - BOOTIMG_DIR: /home/scottrif/poky/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share - KERNEL_DIR: /home/scottrif/poky/build/tmp/deploy/images/qemux86 - NATIVE_SYSROOT: /home/scottrif/poky/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native + ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs + BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share + KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86 + NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native INFO: The image(s) were created using OE kickstart file: - /home/scottrif/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks + /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks </literallayout> Continuing with the example, you can now directly <filename>dd</filename> the image to a USB stick, or @@ -5445,25 +7923,25 @@ somewhere other than the default output directory, which is the current directory: <literallayout class='monospaced'> - $ wic create /home/scottrif/my_yocto/test.wks -o /home/scottrif/testwic \ - --rootfs-dir /home/scottrif/poky/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \ - --bootimg-dir /home/scottrif/poky/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \ - --kernel-dir /home/scottrif/poky/build/tmp/deploy/images/qemux86 \ - --native-sysroot /home/scottrif/poky/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native + $ wic create /home/stephano/my_yocto/test.wks -o /home/stephano/testwic \ + --rootfs-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \ + --bootimg-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \ + --kernel-dir /home/stephano/build/master/build/tmp/deploy/images/qemux86 \ + --native-sysroot /home/stephano/build/master/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native INFO: Creating image(s)... INFO: The new image(s) can be found here: - /home/scottrif/testwic/test-201710091445-sdb.direct + /home/stephano/testwic/test-201710091445-sdb.direct The following build artifacts were used to create the image(s): - ROOTFS_DIR: /home/scottrif/testwic/tmp.wic.x4wipbmb/rootfs_copy - BOOTIMG_DIR: /home/scottrif/poky/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share - KERNEL_DIR: /home/scottrif/poky/build/tmp/deploy/images/qemux86 - NATIVE_SYSROOT: /home/scottrif/poky/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native + ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs + BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share + KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86 + NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native INFO: The image(s) were created using OE kickstart file: - /home/scottrif/my_yocto/test.wks + /home/stephano/my_yocto/test.wks </literallayout> For this example, <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> @@ -5608,222 +8086,91 @@ </section> </section> - <section id='building-an-initramfs-image'> - <title>Building an Initial RAM Filesystem (initramfs) Image</title> - - <para> - An initial RAM filesystem (initramfs) image provides a temporary - root filesystem used for early system initialization (e.g. - loading of modules needed to locate and mount the "real" root - filesystem). - <note> - The initramfs image is the successor of initial RAM disk - (initrd). - It is a "copy in and out" (cpio) archive of the initial - filesystem that gets loaded into memory during the Linux - startup process. - Because Linux uses the contents of the archive during - initialization, the initramfs image needs to contain all of the - device drivers and tools needed to mount the final root - filesystem. - </note> - </para> - - <para> - Follow these steps to create an initramfs image: - <orderedlist> - <listitem><para> - <emphasis>Create the initramfs Image Recipe:</emphasis> - You can reference the - <filename>core-image-minimal-initramfs.bb</filename> - recipe found in the <filename>meta/recipes-core</filename> - directory of the - <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> - as an example from which to work. - </para></listitem> - <listitem><para> - <emphasis>Decide if You Need to Bundle the initramfs Image - Into the Kernel Image:</emphasis> - If you want the initramfs image that is built to be - bundled in with the kernel image, set the - <ulink url='&YOCTO_DOCS_REF_URL;#var-INITRAMFS_IMAGE_BUNDLE'><filename>INITRAMFS_IMAGE_BUNDLE</filename></ulink> - variable to "1" in your <filename>local.conf</filename> - configuration file and set the - <ulink url='&YOCTO_DOCS_REF_URL;#var-INITRAMFS_IMAGE'><filename>INITRAMFS_IMAGE</filename></ulink> - variable in the recipe that builds the kernel image. - <note><title>Tip</title> - It is recommended that you do bundle the initramfs - image with the kernel image to avoid circular - dependencies between the kernel recipe and the - initramfs recipe should the initramfs image - include kernel modules. - </note> - Setting the <filename>INITRAMFS_IMAGE_BUNDLE</filename> - flag causes the initramfs image to be unpacked - into the <filename>${B}/usr/</filename> directory. - The unpacked initramfs image is then passed to the kernel's - <filename>Makefile</filename> using the - <ulink url='&YOCTO_DOCS_REF_URL;#var-CONFIG_INITRAMFS_SOURCE'><filename>CONFIG_INITRAMFS_SOURCE</filename></ulink> - variable, allowing the initramfs image to be built into - the kernel normally. - <note> - If you choose to not bundle the initramfs image with - the kernel image, you are essentially using an - <ulink url='https://en.wikipedia.org/wiki/Initrd'>Initial RAM Disk (initrd)</ulink>. - Creating an initrd is handled primarily through the - <ulink url='&YOCTO_DOCS_REF_URL;#var-INITRD_IMAGE'><filename>INITRD_IMAGE</filename></ulink>, - <filename>INITRD_LIVE</filename>, and - <filename>INITRD_IMAGE_LIVE</filename> variables. - For more information, see the - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/image-live.bbclass'><filename>image-live.bbclass</filename></ulink> - file. - </note> - </para></listitem> -<!-- -Some notes from Cal: - - A non-bundled initramfs is essentially an initrd, which I am discovering - to be rather confusingly supported in OE at the moment. - - Its primarily handled through INITRD_IMAGE(_LIVE/_VM) and INITRD(_LIVE/_VM) - variables. INITRD_IMAGE* is the primary image target, which gets added to - INITRD*, which is a list of cpio filesystems. You can add more cpio - filesystems to the INITRD variable to add more to the initrd. For - instance, meta-intel adds intel-microcode via the following: - - INITRD_LIVE_prepend = "${@bb.utils.contains('MACHINE_FEATURES', 'intel-ucode', '${DEPLOY_DIR_IMAGE}/microcode.cpio ', '', d)}" - - If 'intel-ucode' is in MACHINE_FEATURES, this resolves to: - - INITRD_LIVE_prepend = "${DEPLOY_DIR_IMAGE}/microcode.cpio " - - Unfortunately you need the full path, and its up to you to sort out - dependencies as well. For instance, we have the following: - - MACHINE_ESSENTIAL_EXTRA_RDEPENDS_append = "${@bb.utils.contains('MACHINE_FEATURES', 'intel-ucode', ' intel-microcode', '', d)}" - - which resolves to: - - MACHINE_ESSENTIAL_EXTRA_RDEPENDS_append = "intel-microcode" - - However, the above is only true with the "live" IMAGE_FSTYPE. Wic is - another beast entirely, with current wic kickstart files not supporting - initrds, and only partial support in the source plugins. That being said, - I know the generic bootfs work Ed is working on will help immensely in this - aspect. He or Saul can provide more details here. - - Anyhow, its rather fractured and confusing and could probably use a - rework honestly. I don't know how feasible it is to document all the - details and corner cases of this area. ---> - <listitem><para> - <emphasis>Optionally Add Items to the initramfs Image - Through the initramfs Image Recipe:</emphasis> - If you add items to the initramfs image by way of its - recipe, you should use - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_INSTALL'><filename>PACKAGE_INSTALL</filename></ulink> - rather than - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink>. - <filename>PACKAGE_INSTALL</filename> gives more direct - control of what is added to the image as compared to - the defaults you might not necessarily want that are - set by the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-image'><filename>image</filename></ulink> - or - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-core-image'><filename>core-image</filename></ulink> - classes. - </para></listitem> - <listitem><para> - <emphasis>Build the Kernel Image and the initramfs - Image:</emphasis> - Build your kernel image using BitBake. - Because the initramfs image recipe is a dependency of the - kernel image, the initramfs image is built as well and - bundled with the kernel image if you used the - <ulink url='&YOCTO_DOCS_REF_URL;#var-INITRAMFS_IMAGE_BUNDLE'><filename>INITRAMFS_IMAGE_BUNDLE</filename></ulink> - variable described earlier. - </para></listitem> - </orderedlist> - </para> - </section> - <section id='flashing-images-using-bmaptool'> <title>Flashing Images Using <filename>bmaptool</filename></title> <para> - An easy way to flash an image to a bootable device is to use - <filename>bmaptool</filename>, which is integrated into the - OpenEmbedded build system. + A fast and easy way to flash an image to a bootable device + is to use Bmaptool, which is integrated into the OpenEmbedded + build system. + Bmaptool is a generic tool that creates a file's block map (bmap) + and then uses that map to copy the file. + As compared to traditional tools such as dd or cp, Bmaptool + can copy (or flash) large files like raw system image files + much faster. + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + If you are using Ubuntu or Debian distributions, you + can install the <filename>bmap-tools</filename> package + using the following command and then use the tool + without specifying <filename>PATH</filename> even from + the root account: + <literallayout class='monospaced'> + $ sudo apt-get install bmap-tools + </literallayout> + </para></listitem> + <listitem><para> + If you are unable to install the + <filename>bmap-tools</filename> package, you will + need to build Bmaptool before using it. + Use the following command: + <literallayout class='monospaced'> + $ bitbake bmap-tools-native + </literallayout> + </para></listitem> + </itemizedlist> + </note> </para> <para> Following, is an example that shows how to flash a Wic image. - <note> - You can use <filename>bmaptool</filename> to flash any - type of image. - </note> - Use these steps to flash an image using - <filename>bmaptool</filename>: - <note> - Unless you are able to install the - <filename>bmap-tools</filename> package as mentioned in the note - in the second bullet of step 3 further down, you will need to build - <filename>bmaptool</filename> before using it. - Build the tool using the following command: - <literallayout class='monospaced'> - $ bitbake bmap-tools-native - </literallayout> - </note> + Realize that while this example uses a Wic image, you can use + Bmaptool to flash any type of image. + Use these steps to flash an image using Bmaptool: <orderedlist> <listitem><para> - <emphasis>Update the <filename>local.conf</filename> File:</emphasis> - Add the following to your <filename>local.conf</filename> - file: + <emphasis>Update your <filename>local.conf</filename> File:</emphasis> + You need to have the following set in your + <filename>local.conf</filename> file before building + your image: <literallayout class='monospaced'> IMAGE_FSTYPES += "wic wic.bmap" </literallayout> </para></listitem> <listitem><para> <emphasis>Get Your Image:</emphasis> - Either have your image ready (pre-built) or take the step - build the image: + Either have your image ready (pre-built with the + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink> + setting previously mentioned) or take the step to build + the image: <literallayout class='monospaced'> $ bitbake <replaceable>image</replaceable> </literallayout> </para></listitem> <listitem><para> <emphasis>Flash the Device:</emphasis> - Flash the device with the image by using - <filename>bmaptool</filename> depending on your particular - setup: + Flash the device with the image by using Bmaptool + depending on your particular setup. + The following commands assume the image resides in the + Build Directory's <filename>deploy/images/</filename> + area: <itemizedlist> <listitem><para> - If you have write access to the media, - use this command form: + If you have write access to the media, use this + command form: <literallayout class='monospaced'> - $ oe-run-native bmap-tools-native bmaptool copy ./tmp/deploy/images/qemux86-64-core-image-minimal-<replaceable>machine</replaceable>.wic /dev/sd<replaceable>X</replaceable> + $ oe-run-native bmap-tools-native bmaptool copy <replaceable>build-directory</replaceable>/tmp/deploy/images/<replaceable>machine</replaceable>/<replaceable>image</replaceable>.wic /dev/sd<replaceable>X</replaceable> </literallayout> </para></listitem> <listitem><para> - If you do not have write access to - the media, use the following - commands: + If you do not have write access to the media, set + your permissions first and then use the same + command form: <literallayout class='monospaced'> $ sudo chmod 666 /dev/sd<replaceable>X</replaceable> - $ oe-run-native bmap-tools-native bmaptool copy ./tmp/deploy/images/qemux86-64-core-image-minimal-<replaceable>machine</replaceable>.wic /dev/sd<replaceable>X</replaceable> + $ oe-run-native bmap-tools-native bmaptool copy <replaceable>build-directory</replaceable>/tmp/deploy/images/<replaceable>machine</replaceable>/<replaceable>image</replaceable>.wic /dev/sd<replaceable>X</replaceable> </literallayout> - <note> - If you are using Ubuntu or Debian distributions, - you can install the - <filename>bmap-tools</filename> package using - the following command and then use the tool - without specifying - <filename>PATH</filename> even from the - root account: - <literallayout class='monospaced'> - $ sudo apt-get install bmap-tools - </literallayout> - </note> </para></listitem> </itemizedlist> </para></listitem> @@ -5852,7 +8199,7 @@ Some notes from Cal: by Bruce Schneier </para></listitem> <listitem><para><emphasis> - "<ulink url='http://internetcensus2012.bitbucket.org/paper.html'>Internet Census 2012</ulink>"</emphasis> + "<ulink url='http://census2012.sourceforge.net/paper.html'>Internet Census 2012</ulink>"</emphasis> by Carna Botnet</para></listitem> <listitem><para><emphasis> "<ulink url='http://elinux.org/images/6/6f/Security-issues.pdf'>Security Issues for Embedded Devices</ulink>"</emphasis> @@ -6060,7 +8407,7 @@ Some notes from Cal: more secure. You can find these tools in the <filename>meta-security</filename> layer of the - <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi'>Yocto Project Source Repositories</ulink>. + <ulink url='&YOCTO_GIT_URL;'>Yocto Project Source Repositories</ulink>. </para> </section> </section> @@ -6308,616 +8655,22 @@ Some notes from Cal: </para> </section> - <section id='building-a-tiny-system'> - <title>Building a Tiny System</title> + <section id='dev-saving-memory-during-a-build'> + <title>Conserving Disk Space During Builds</title> <para> - Very small distributions have some significant advantages such - as requiring less on-die or in-package memory (cheaper), better - performance through efficient cache usage, lower power requirements - due to less memory, faster boot times, and reduced development - overhead. - Some real-world examples where a very small distribution gives - you distinct advantages are digital cameras, medical devices, - and small headless systems. - </para> - - <para> - This section presents information that shows you how you can - trim your distribution to even smaller sizes than the - <filename>poky-tiny</filename> distribution, which is around - 5 Mbytes, that can be built out-of-the-box using the Yocto Project. - </para> - - <section id='tiny-system-overview'> - <title>Overview</title> - - <para> - The following list presents the overall steps you need to - consider and perform to create distributions with smaller - root filesystems, achieve faster boot times, maintain your critical - functionality, and avoid initial RAM disks: - <itemizedlist> - <listitem><para> - <link linkend='goals-and-guiding-principles'>Determine your goals and guiding principles.</link> - </para></listitem> - <listitem><para> - <link linkend='understand-what-gives-your-image-size'>Understand what contributes to your image size.</link> - </para></listitem> - <listitem><para> - <link linkend='trim-the-root-filesystem'>Reduce the size of the root filesystem.</link> - </para></listitem> - <listitem><para> - <link linkend='trim-the-kernel'>Reduce the size of the kernel.</link> - </para></listitem> - <listitem><para> - <link linkend='remove-package-management-requirements'>Eliminate packaging requirements.</link> - </para></listitem> - <listitem><para> - <link linkend='look-for-other-ways-to-minimize-size'>Look for other ways to minimize size.</link> - </para></listitem> - <listitem><para> - <link linkend='iterate-on-the-process'>Iterate on the process.</link> - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='goals-and-guiding-principles'> - <title>Goals and Guiding Principles</title> - - <para> - Before you can reach your destination, you need to know - where you are going. - Here is an example list that you can use as a guide when - creating very small distributions: - <itemizedlist> - <listitem><para>Determine how much space you need - (e.g. a kernel that is 1 Mbyte or less and - a root filesystem that is 3 Mbytes or less). - </para></listitem> - <listitem><para>Find the areas that are currently - taking 90% of the space and concentrate on reducing - those areas. - </para></listitem> - <listitem><para>Do not create any difficult "hacks" - to achieve your goals.</para></listitem> - <listitem><para>Leverage the device-specific - options.</para></listitem> - <listitem><para>Work in a separate layer so that you - keep changes isolated. - For information on how to create layers, see - the "<link linkend='understanding-and-creating-layers'>Understanding and Creating Layers</link>" section. - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='understand-what-gives-your-image-size'> - <title>Understand What Contributes to Your Image Size</title> - - <para> - It is easiest to have something to start with when creating - your own distribution. - You can use the Yocto Project out-of-the-box to create the - <filename>poky-tiny</filename> distribution. - Ultimately, you will want to make changes in your own - distribution that are likely modeled after - <filename>poky-tiny</filename>. - <note> - To use <filename>poky-tiny</filename> in your build, - set the - <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink> - variable in your - <filename>local.conf</filename> file to "poky-tiny" - as described in the - "<link linkend='creating-your-own-distribution'>Creating Your Own Distribution</link>" - section. - </note> - </para> - - <para> - Understanding some memory concepts will help you reduce the - system size. - Memory consists of static, dynamic, and temporary memory. - Static memory is the TEXT (code), DATA (initialized data - in the code), and BSS (uninitialized data) sections. - Dynamic memory represents memory that is allocated at runtime: - stacks, hash tables, and so forth. - Temporary memory is recovered after the boot process. - This memory consists of memory used for decompressing - the kernel and for the <filename>__init__</filename> - functions. - </para> - - <para> - To help you see where you currently are with kernel and root - filesystem sizes, you can use two tools found in the - <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> in - the <filename>scripts/tiny/</filename> directory: - <itemizedlist> - <listitem><para><filename>ksize.py</filename>: Reports - component sizes for the kernel build objects. - </para></listitem> - <listitem><para><filename>dirsize.py</filename>: Reports - component sizes for the root filesystem.</para></listitem> - </itemizedlist> - This next tool and command help you organize configuration - fragments and view file dependencies in a human-readable form: - <itemizedlist> - <listitem><para><filename>merge_config.sh</filename>: - Helps you manage configuration files and fragments - within the kernel. - With this tool, you can merge individual configuration - fragments together. - The tool allows you to make overrides and warns you - of any missing configuration options. - The tool is ideal for allowing you to iterate on - configurations, create minimal configurations, and - create configuration files for different machines - without having to duplicate your process.</para> - <para>The <filename>merge_config.sh</filename> script is - part of the Linux Yocto kernel Git repositories - (i.e. <filename>linux-yocto-3.14</filename>, - <filename>linux-yocto-3.10</filename>, - <filename>linux-yocto-3.8</filename>, and so forth) - in the - <filename>scripts/kconfig</filename> directory.</para> - <para>For more information on configuration fragments, - see the - "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#creating-config-fragments'>Creating Configuration Fragments</ulink>" - section in the Yocto Project Linux Kernel Development - Manual. - </para></listitem> - <listitem><para><filename>bitbake -u taskexp -g <replaceable>bitbake_target</replaceable></filename>: - Using the BitBake command with these options brings up - a Dependency Explorer from which you can view file - dependencies. - Understanding these dependencies allows you to make - informed decisions when cutting out various pieces of the - kernel and root filesystem.</para></listitem> - </itemizedlist> - </para> - </section> - - <section id='trim-the-root-filesystem'> - <title>Trim the Root Filesystem</title> - - <para> - The root filesystem is made up of packages for booting, - libraries, and applications. - To change things, you can configure how the packaging happens, - which changes the way you build them. - You can also modify the filesystem itself or select a different - filesystem. - </para> - - <para> - First, find out what is hogging your root filesystem by running the - <filename>dirsize.py</filename> script from your root directory: - <literallayout class='monospaced'> - $ cd <replaceable>root-directory-of-image</replaceable> - $ dirsize.py 100000 > dirsize-100k.log - $ cat dirsize-100k.log - </literallayout> - You can apply a filter to the script to ignore files under - a certain size. - The previous example filters out any files below 100 Kbytes. - The sizes reported by the tool are uncompressed, and thus - will be smaller by a relatively constant factor in a - compressed root filesystem. - When you examine your log file, you can focus on areas of the - root filesystem that take up large amounts of memory. - </para> - - <para> - You need to be sure that what you eliminate does not cripple - the functionality you need. - One way to see how packages relate to each other is by using - the Dependency Explorer UI with the BitBake command: - <literallayout class='monospaced'> - $ cd <replaceable>image-directory</replaceable> - $ bitbake -u taskexp -g <replaceable>image</replaceable> - </literallayout> - Use the interface to select potential packages you wish to - eliminate and see their dependency relationships. - </para> - - <para> - When deciding how to reduce the size, get rid of packages that - result in minimal impact on the feature set. - For example, you might not need a VGA display. - Or, you might be able to get by with <filename>devtmpfs</filename> - and <filename>mdev</filename> instead of - <filename>udev</filename>. - </para> - - <para> - Use your <filename>local.conf</filename> file to make changes. - For example, to eliminate <filename>udev</filename> and - <filename>glib</filename>, set the following in the - local configuration file: - <literallayout class='monospaced'> - VIRTUAL-RUNTIME_dev_manager = "" - </literallayout> - </para> - - <para> - Finally, you should consider exactly the type of root - filesystem you need to meet your needs while also reducing - its size. - For example, consider <filename>cramfs</filename>, - <filename>squashfs</filename>, <filename>ubifs</filename>, - <filename>ext2</filename>, or an <filename>initramfs</filename> - using <filename>initramfs</filename>. - Be aware that <filename>ext3</filename> requires a 1 Mbyte - journal. - If you are okay with running read-only, you do not need this - journal. - </para> - - <note> - After each round of elimination, you need to rebuild your - system and then use the tools to see the effects of your - reductions. - </note> - - - </section> - - <section id='trim-the-kernel'> - <title>Trim the Kernel</title> - - <para> - The kernel is built by including policies for hardware-independent - aspects. - What subsystems do you enable? - For what architecture are you building? - Which drivers do you build by default? - <note>You can modify the kernel source if you want to help - with boot time. - </note> - </para> - - <para> - Run the <filename>ksize.py</filename> script from the top-level - Linux build directory to get an idea of what is making up - the kernel: - <literallayout class='monospaced'> - $ cd <replaceable>top-level-linux-build-directory</replaceable> - $ ksize.py > ksize.log - $ cat ksize.log - </literallayout> - When you examine the log, you will see how much space is - taken up with the built-in <filename>.o</filename> files for - drivers, networking, core kernel files, filesystem, sound, - and so forth. - The sizes reported by the tool are uncompressed, and thus - will be smaller by a relatively constant factor in a compressed - kernel image. - Look to reduce the areas that are large and taking up around - the "90% rule." - </para> - - <para> - To examine, or drill down, into any particular area, use the - <filename>-d</filename> option with the script: - <literallayout class='monospaced'> - $ ksize.py -d > ksize.log - </literallayout> - Using this option breaks out the individual file information - for each area of the kernel (e.g. drivers, networking, and - so forth). - </para> - - <para> - Use your log file to see what you can eliminate from the kernel - based on features you can let go. - For example, if you are not going to need sound, you do not - need any drivers that support sound. - </para> - - <para> - After figuring out what to eliminate, you need to reconfigure - the kernel to reflect those changes during the next build. - You could run <filename>menuconfig</filename> and make all your - changes at once. - However, that makes it difficult to see the effects of your - individual eliminations and also makes it difficult to replicate - the changes for perhaps another target device. - A better method is to start with no configurations using - <filename>allnoconfig</filename>, create configuration - fragments for individual changes, and then manage the - fragments into a single configuration file using - <filename>merge_config.sh</filename>. - The tool makes it easy for you to iterate using the - configuration change and build cycle. - </para> - - <para> - Each time you make configuration changes, you need to rebuild - the kernel and check to see what impact your changes had on - the overall size. - </para> - </section> - - <section id='remove-package-management-requirements'> - <title>Remove Package Management Requirements</title> - - <para> - Packaging requirements add size to the image. - One way to reduce the size of the image is to remove all the - packaging requirements from the image. - This reduction includes both removing the package manager - and its unique dependencies as well as removing the package - management data itself. - </para> - - <para> - To eliminate all the packaging requirements for an image, - be sure that "package-management" is not part of your - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink> - statement for the image. - When you remove this feature, you are removing the package - manager as well as its dependencies from the root filesystem. - </para> - </section> - - <section id='look-for-other-ways-to-minimize-size'> - <title>Look for Other Ways to Minimize Size</title> - - <para> - Depending on your particular circumstances, other areas that you - can trim likely exist. - The key to finding these areas is through tools and methods - described here combined with experimentation and iteration. - Here are a couple of areas to experiment with: - <itemizedlist> - <listitem><para><filename>glibc</filename>: - In general, follow this process: - <orderedlist> - <listitem><para>Remove <filename>glibc</filename> - features from - <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></ulink> - that you think you do not need.</para></listitem> - <listitem><para>Build your distribution. - </para></listitem> - <listitem><para>If the build fails due to missing - symbols in a package, determine if you can - reconfigure the package to not need those - features. - For example, change the configuration to not - support wide character support as is done for - <filename>ncurses</filename>. - Or, if support for those characters is needed, - determine what <filename>glibc</filename> - features provide the support and restore the - configuration. - </para></listitem> - <listitem><para>Rebuild and repeat the process. - </para></listitem> - </orderedlist></para></listitem> - <listitem><para><filename>busybox</filename>: - For BusyBox, use a process similar as described for - <filename>glibc</filename>. - A difference is you will need to boot the resulting - system to see if you are able to do everything you - expect from the running system. - You need to be sure to integrate configuration fragments - into Busybox because BusyBox handles its own core - features and then allows you to add configuration - fragments on top. - </para></listitem> - </itemizedlist> - </para> - </section> - - <section id='iterate-on-the-process'> - <title>Iterate on the Process</title> - - <para> - If you have not reached your goals on system size, you need - to iterate on the process. - The process is the same. - Use the tools and see just what is taking up 90% of the root - filesystem and the kernel. - Decide what you can eliminate without limiting your device - beyond what you need. - </para> - - <para> - Depending on your system, a good place to look might be - Busybox, which provides a stripped down - version of Unix tools in a single, executable file. - You might be able to drop virtual terminal services or perhaps - ipv6. - </para> - </section> - </section> - - <section id='building-images-for-more-than-one-machine'> - <title>Building Images for More than One Machine</title> - - <para> - A common scenario developers face is creating images for several - different machines that use the same software environment. - In this situation, it is tempting to set the - tunings and optimization flags for each build specifically for - the targeted hardware (i.e. "maxing out" the tunings). - Doing so can considerably add to build times and package feed - maintenance collectively for the machines. - For example, selecting tunes that are extremely specific to a - CPU core used in a system might enable some micro optimizations - in GCC for that particular system but would otherwise not gain - you much of a performance difference across the other systems - as compared to using a more general tuning across all the builds - (e.g. setting - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEFAULTTUNE'><filename>DEFAULTTUNE</filename></ulink> - specifically for each machine's build). - Rather than "max out" each build's tunings, you can take steps that - cause the OpenEmbedded build system to reuse software across the - various machines where it makes sense. - </para> - <para> - If build speed and package feed maintenance are considerations, - you should consider the points in this section that can help you - optimize your tunings to best consider build times and package - feed maintenance. - <itemizedlist> - <listitem><para><emphasis>Share the Build Directory:</emphasis> - If at all possible, share the - <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> - across builds. - The Yocto Project supports switching between different - <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> - values in the same <filename>TMPDIR</filename>. - This practice is well supported and regularly used by - developers when building for multiple machines. - When you use the same <filename>TMPDIR</filename> for - multiple machine builds, the OpenEmbedded build system can - reuse the existing native and often cross-recipes for - multiple machines. - Thus, build time decreases. - <note> - If - <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink> - settings change or fundamental configuration settings - such as the filesystem layout, you need to work with - a clean <filename>TMPDIR</filename>. - Sharing <filename>TMPDIR</filename> under these - circumstances might work but since it is not - guaranteed, you should use a clean - <filename>TMPDIR</filename>. - </note> - </para></listitem> - <listitem><para><emphasis>Enable the Appropriate Package Architecture:</emphasis> - By default, the OpenEmbedded build system enables three - levels of package architectures: "all", "tune" or "package", - and "machine". - Any given recipe usually selects one of these package - architectures (types) for its output. - Depending for what a given recipe creates packages, making - sure you enable the appropriate package architecture can - directly impact the build time.</para> - <para>A recipe that just generates scripts can enable - "all" architecture because there are no binaries to build. - To specifically enable "all" architecture, be sure your - recipe inherits the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-allarch'><filename>allarch</filename></ulink> - class. - This class is useful for "all" architectures because it - configures many variables so packages can be used across - multiple architectures.</para> - <para>If your recipe needs to generate packages that are - machine-specific or when one of the build or runtime - dependencies is already machine-architecture dependent, - which makes your recipe also machine-architecture dependent, - make sure your recipe enables the "machine" package - architecture through the - <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE_ARCH'><filename>MACHINE_ARCH</filename></ulink> - variable: - <literallayout class='monospaced'> - PACKAGE_ARCH = "${MACHINE_ARCH}" - </literallayout> - When you do not specifically enable a package - architecture through the - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>, - The OpenEmbedded build system defaults to the - <ulink url='&YOCTO_DOCS_REF_URL;#var-TUNE_PKGARCH'><filename>TUNE_PKGARCH</filename></ulink> - setting: - <literallayout class='monospaced'> - PACKAGE_ARCH = "${TUNE_PKGARCH}" - </literallayout> - </para></listitem> - <listitem><para><emphasis>Choose a Generic Tuning File if Possible:</emphasis> - Some tunes are more generic and can run on multiple targets - (e.g. an <filename>armv5</filename> set of packages could - run on <filename>armv6</filename> and - <filename>armv7</filename> processors in most cases). - Similarly, <filename>i486</filename> binaries could work - on <filename>i586</filename> and higher processors. - You should realize, however, that advances on newer - processor versions would not be used.</para> - <para>If you select the same tune for several different - machines, the OpenEmbedded build system reuses software - previously built, thus speeding up the overall build time. - Realize that even though a new sysroot for each machine is - generated, the software is not recompiled and only one - package feed exists. - </para></listitem> - <listitem><para><emphasis>Manage Granular Level Packaging:</emphasis> - Sometimes cases exist where injecting another level - of package architecture beyond the three higher levels - noted earlier can be useful. - For example, consider the <filename>emgd</filename> - graphics stack in the - <filename>meta-intel</filename> layer. - In this layer, a subset of software exists that is - compiled against something different from the rest of the - generic packages. - You can examine the key code in the - <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi'>Source Repositories</ulink> - "daisy" branch in - <filename>classes/emgd-gl.bbclass</filename>. - For a specific set of packages, the code redefines - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>. - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_EXTRA_ARCHS'><filename>PACKAGE_EXTRA_ARCHS</filename></ulink> - is then appended with this extra tune name in - <filename>meta-intel-emgd.inc</filename>. - The result is that when searching for packages, the - build system uses a four-level search and the packages - in this new level are preferred as compared to the standard - tune. - The overall result is that the build system reuses most - software from the common tune except for specific cases - as needed. - </para></listitem> - <listitem><para><emphasis>Use Tools to Debug Issues:</emphasis> - Sometimes you can run into situations where software is - being rebuilt when you think it should not be. - For example, the OpenEmbedded build system might not be - using shared state between machines when you think it - should be. - These types of situations are usually due to references - to machine-specific variables such as - <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#var-SERIAL_CONSOLE'><filename>SERIAL_CONSOLE</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#var-XSERVER'><filename>XSERVER</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE_FEATURES'><filename>MACHINE_FEATURES</filename></ulink>, - and so forth in code that is supposed to only be - tune-specific or when the recipe depends - (<ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#var-RSUGGESTS'><filename>RSUGGESTS</filename></ulink>, - and so forth) on some other recipe that already has - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink> - defined as "${MACHINE_ARCH}". - <note> - Patches to fix any issues identified are most welcome - as these issues occasionally do occur. - </note></para> - <para>For such cases, you can use some tools to help you - sort out the situation: - <itemizedlist> - <listitem><para><emphasis><filename>sstate-diff-machines.sh</filename>:</emphasis> - You can find this tool in the - <filename>scripts</filename> directory of the - Source Repositories. - See the comments in the script for information on - how to use the tool. - </para></listitem> - <listitem><para><emphasis>BitBake's "-S printdiff" Option:</emphasis> - Using this option causes BitBake to try to - establish the closest signature match it can - (e.g. in the shared state cache) and then run - <filename>bitbake-diffsigs</filename> over the - matches to determine the stamps and delta where - these two stamp trees diverge. - </para></listitem> - </itemizedlist> - </para></listitem> - </itemizedlist> + To help conserve disk space during builds, you can add the + following statement to your project's + <filename>local.conf</filename> configuration file found in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>: + <literallayout class='monospaced'> + INHERIT += "rm_work" + </literallayout> + Adding this statement deletes the work directory used for building + a recipe once the recipe is built. + For more information on "rm_work", see the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-rm-work'><filename>rm_work</filename></ulink> + class in the Yocto Project Reference Manual. </para> </section> @@ -7121,7 +8874,7 @@ Some notes from Cal: <para> Because the OpenEmbedded build system uses - "<ulink url='&YOCTO_DOCS_REF_URL;#checksums'>signatures</ulink>", + "<ulink url='&YOCTO_DOCS_OM_URL;#overview-checksums'>signatures</ulink>", which are unique to a given build, the build system knows when to rebuild packages. All the inputs into a given task are represented by a @@ -7206,8 +8959,8 @@ Some notes from Cal: BUILDHISTORY_COMMIT = "1" </literallayout> For information on build history, see the - "<ulink url='&YOCTO_DOCS_REF_URL;#maintaining-build-output-quality'>Maintaining Build Output Quality</ulink>" - section in the Yocto Project Reference Manual. + "<link linkend='maintaining-build-output-quality'>Maintaining Build Output Quality</link>" + section. </para> <note> @@ -7225,8 +8978,9 @@ Some notes from Cal: <para> For more information on shared state, see the - "<ulink url='&YOCTO_DOCS_REF_URL;#shared-state-cache'>Shared State Cache</ulink>" - section in the Yocto Project Reference Manual. + "<ulink url='&YOCTO_DOCS_OM_URL;#shared-state-cache'>Shared State Cache</ulink>" + section in the Yocto Project Overview and Concepts + Manual. </para> </note> </section> @@ -7470,7 +9224,7 @@ Some notes from Cal: <filename>connman.inc</filename> file in the <filename>meta/recipes-connectivity/connman/</filename> directory of the <filename>poky</filename> - <ulink url='&YOCTO_DOCS_REF_URL;#yocto-project-repositories'>source repository</ulink>. + <ulink url='&YOCTO_DOCS_OM_URL;#yocto-project-repositories'>source repository</ulink>. You can also find examples in <filename>meta/classes/kernel.bbclass</filename>. </para> @@ -7599,11 +9353,13 @@ Some notes from Cal: During a build, BitBake always transforms a recipe into one or more packages. For example, BitBake takes the <filename>bash</filename> recipe - and currently produces the <filename>bash-dbg</filename>, - <filename>bash-staticdev</filename>, - <filename>bash-dev</filename>, <filename>bash-doc</filename>, - <filename>bash-locale</filename>, and - <filename>bash</filename> packages. + and produces a number of packages (e.g. + <filename>bash</filename>, <filename>bash-bashbug</filename>, + <filename>bash-completion</filename>, + <filename>bash-completion-dbg</filename>, + <filename>bash-completion-dev</filename>, + <filename>bash-completion-extra</filename>, + <filename>bash-dbg</filename>, and so forth). Not all generated packages are included in an image. </para> @@ -7647,7 +9403,7 @@ Some notes from Cal: <para> In order to use runtime package management, you - need a host/server machine that serves up the pre-compiled + need a host or server machine that serves up the pre-compiled packages plus the required metadata. You also need package manipulation tools on the target. The build machine is a likely candidate to act as the server. @@ -7655,6 +9411,10 @@ Some notes from Cal: package server. The build machine could push its artifacts to another machine that acts as the server (e.g. Internet-facing). + In fact, doing so is advantageous for a production + environment as getting the packages away from the + development system's build directory prevents accidental + overwrites. </para> <para> @@ -7664,11 +9424,11 @@ Some notes from Cal: out into a couple of different package groupings based on criteria such as the target's CPU architecture, the target board, or the C library used on the target. - For example, a build targeting the <filename>qemuarm</filename> + For example, a build targeting the <filename>qemux86</filename> device produces the following three package databases: - <filename>all</filename>, <filename>armv5te</filename>, and - <filename>qemuarm</filename>. - If you wanted your <filename>qemuarm</filename> device to be + <filename>noarch</filename>, <filename>i586</filename>, and + <filename>qemux86</filename>. + If you wanted your <filename>qemux86</filename> device to be aware of all the packages that were available to it, you would need to point it to each of these databases individually. @@ -7714,10 +9474,10 @@ Some notes from Cal: PACKAGE_CLASSES ?= “package_<replaceable>packageformat</replaceable>” </literallayout> where <replaceable>packageformat</replaceable> - can be "ipk", "rpm", and "deb", which are the + can be "ipk", "rpm", "deb", or "tar" which are the supported package formats. <note> - Because the Yocto Project supports three + Because the Yocto Project supports four different package formats, you can set the variable with more than one argument. However, the OpenEmbedded build system only @@ -7736,12 +9496,12 @@ Some notes from Cal: "package-management" in the <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink> variable. - Including "package-management" in this - configuration variable ensures that when the image - is assembled for your target, the image includes - the currently-known package databases as well as - the target-specific tools required for runtime - package management to be performed on the target. + Including "package-management" in this configuration + variable ensures that when the image is assembled for your + target, the image includes the currently-known package + databases as well as the target-specific tools required + for runtime package management to be performed on the + target. However, this is not strictly necessary. You could start your image off without any databases but only include the required on-target package @@ -7755,21 +9515,26 @@ Some notes from Cal: <para> Whenever you perform any sort of build step that can - potentially generate a package or modify an existing + potentially generate a package or modify existing package, it is always a good idea to re-generate the - package index with: + package index after the build by using the following + command: <literallayout class='monospaced'> $ bitbake package-index </literallayout> - Realize that it is not sufficient to simply do the - following: + It might be tempting to build the package and the + package index at the same time with a command such as + the following: <literallayout class='monospaced'> $ bitbake <replaceable>some-package</replaceable> package-index </literallayout> - The reason for this restriction is because BitBake does not - properly schedule the <filename>package-index</filename> - target fully after any other target has completed. - Thus, be sure to run the package update step separately. + Do not do this as BitBake does not schedule the package + index for after the completion of the package you are + building. + Consequently, you cannot be sure of the package index + including information for the package you just built. + Thus, be sure to run the package update step separately + after building any packages. </para> <para> @@ -7794,8 +9559,8 @@ Some notes from Cal: For example, if <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink><filename>}</filename> is <filename>tmp</filename> and your selected package type - is IPK, then your IPK packages are available in - <filename>tmp/deploy/ipk</filename>. + is RPM, then your RPM packages are available in + <filename>tmp/deploy/rpm</filename>. </para> </section> @@ -7850,17 +9615,39 @@ Some notes from Cal: <title>Using RPM</title> <para> - The <filename>dnf</filename> application performs - runtime package management of RPM packages. - You must perform an initial setup for - <filename>dnf</filename> on the target machine - if the - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_ARCHS'><filename>PACKAGE_FEED_ARCHS</filename></ulink>, - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_BASE_PATHS'><filename>PACKAGE_FEED_BASE_PATHS</filename></ulink>, - and - <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_URIS'><filename>PACKAGE_FEED_URIS</filename></ulink> - variables have not been set or the target image was - built before the variables were set. + The + <ulink url='https://en.wikipedia.org/wiki/DNF_(software)'>Dandified Packaging Tool</ulink> + (DNF) performs runtime package management of RPM + packages. + In order to use DNF for runtime package management, + you must perform an initial setup on the target + machine for cases where the + <filename>PACKAGE_FEED_*</filename> variables were not + set as part of the image that is running on the + target. + This means if you built your image and did not not use + these variables as part of the build and your image is + now running on the target, you need to perform the + steps in this section if you want to use runtime + package management. + <note> + For information on the + <filename>PACKAGE_FEED_*</filename> variables, see + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_ARCHS'><filename>PACKAGE_FEED_ARCHS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_BASE_PATHS'><filename>PACKAGE_FEED_BASE_PATHS</filename></ulink>, + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_URIS'><filename>PACKAGE_FEED_URIS</filename></ulink> + in the Yocto Project Reference Manual variables + glossary. + </note> + </para> + + <para> + On the target, you must inform DNF that package + databases are available. + You do this by creating a file named + <filename>/etc/yum.repos.d/oe-packages.repo</filename> + and defining the <filename>oe-packages</filename>. </para> <para> @@ -7869,21 +9656,65 @@ Some notes from Cal: <filename>all</filename>, <filename>i586</filename>, and <filename>qemux86</filename> from a server named <filename>my.server</filename>. - You must inform <filename>dnf</filename> of the - availability of these databases by creating a - <filename>/etc/yum.repos.d/oe-packages.repo</filename> - file with the following content: - <literallayout class='monospaced'> + The specifics for setting up the web server are up to + you. + The critical requirement is that the URIs in the + target repository configuration point to the + correct remote location for the feeds. + <note><title>Tip</title> + For development purposes, you can point the web + server to the build system's + <filename>deploy</filename> directory. + However, for production use, it is better to copy + the package directories to a location outside of + the build area and use that location. + Doing so avoids situations where the build system + overwrites or changes the + <filename>deploy</filename> directory. + </note> + </para> + + <para> + When telling DNF where to look for the package + databases, you must declare individual locations + per architecture or a single location used for all + architectures. + You cannot do both: + <itemizedlist> + <listitem><para> + <emphasis>Create an Explicit List of Architectures:</emphasis> + Define individual base URLs to identify where + each package database is located: + <literallayout class='monospaced'> [oe-packages] baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all - </literallayout> - From the target machine, fetch the repository: + </literallayout> + This example informs DNF about individual + package databases for all three architectures. + </para></listitem> + <listitem><para> + <emphasis>Create a Single (Full) Package Index:</emphasis> + Define a single base URL that identifies where + a full package database is located: + <literallayout class='monospaced'> + [oe-packages] + baseurl=http://my.server/rpm + </literallayout> + This example informs DNF about a single package + database that contains all the package index + information for all supported architectures. + </para></listitem> + </itemizedlist> + </para> + + <para> + Once you have informed DNF where to find the package + databases, you need to fetch them: <literallayout class='monospaced'> # dnf makecache </literallayout> - After everything is set up, <filename>dnf</filename> - is able to find, install, and upgrade packages from - the specified repository. + DNF is now able to find, install, and upgrade packages + from the specified repository or repositories. <note> See the <ulink url='http://dnf.readthedocs.io/en/latest/'>DNF documentation</ulink> @@ -8294,8 +10125,8 @@ Some notes from Cal: </section> </section> - <section id='working-with-source-files'> - <title>Working with Source Files</title> + <section id='efficiently-fetching-source-files-during-a-build'> + <title>Efficiently Fetching Source Files During a Build</title> <para> The OpenEmbedded build system works with source files located @@ -8309,16 +10140,16 @@ Some notes from Cal: </para> <para> - This section presents information for working with source - files that can lead to more efficient use of resources and - time. + This section shows you how you can use mirrors to speed up + fetching source files and how you can pre-fetch files all of which + leads to more efficient use of resources and time. </para> <section id='setting-up-effective-mirrors'> <title>Setting up Effective Mirrors</title> <para> - As mentioned, a good deal that goes into a Yocto Project + A good deal that goes into a Yocto Project build is simply downloading all of the source tarballs. Maybe you have been working with another build system (OpenEmbedded or Angstrom) for which you have built up a @@ -8381,7 +10212,7 @@ Some notes from Cal: Use the following BitBake command form to fetch all the necessary sources without starting the build: <literallayout class='monospaced'> - $ bitbake -c fetchall <replaceable>target</replaceable> + $ bitbake -c <replaceable>target</replaceable> runall="fetch" </literallayout> This variation of the BitBake command guarantees that you have all the sources for that BitBake target should you @@ -8391,81 +10222,6 @@ Some notes from Cal: </section> </section> - <section id="building-software-from-an-external-source"> - <title>Building Software from an External Source</title> - - <para> - By default, the OpenEmbedded build system uses the - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> - when building source code. - The build process involves fetching the source files, unpacking - them, and then patching them if necessary before the build takes - place. - </para> - - <para> - Situations exist where you might want to build software from source - files that are external to and thus outside of the - OpenEmbedded build system. - For example, suppose you have a project that includes a new BSP with - a heavily customized kernel. - And, you want to minimize exposing the build system to the - development team so that they can focus on their project and - maintain everyone's workflow as much as possible. - In this case, you want a kernel source directory on the development - machine where the development occurs. - You want the recipe's - <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> - variable to point to the external directory and use it as is, not - copy it. - </para> - - <para> - To build from software that comes from an external source, all you - need to do is inherit the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink> - class and then set the - <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTERNALSRC'><filename>EXTERNALSRC</filename></ulink> - variable to point to your external source code. - Here are the statements to put in your - <filename>local.conf</filename> file: - <literallayout class='monospaced'> - INHERIT += "externalsrc" - EXTERNALSRC_pn-<replaceable>myrecipe</replaceable> = "<replaceable>path-to-your-source-tree</replaceable>" - </literallayout> - </para> - - <para> - This next example shows how to accomplish the same thing by setting - <filename>EXTERNALSRC</filename> in the recipe itself or in the - recipe's append file: - <literallayout class='monospaced'> - EXTERNALSRC = "<replaceable>path</replaceable>" - EXTERNALSRC_BUILD = "<replaceable>path</replaceable>" - </literallayout> - <note> - In order for these settings to take effect, you must globally - or locally inherit the - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink> - class. - </note> - </para> - - <para> - By default, <filename>externalsrc.bbclass</filename> builds - the source code in a directory separate from the external source - directory as specified by - <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTERNALSRC'><filename>EXTERNALSRC</filename></ulink>. - If you need to have the source built in the same directory in - which it resides, or some other nominated directory, you can set - <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTERNALSRC_BUILD'><filename>EXTERNALSRC_BUILD</filename></ulink> - to point to that directory: - <literallayout class='monospaced'> - EXTERNALSRC_BUILD_pn-<replaceable>myrecipe</replaceable> = "<replaceable>path-to-your-source-tree</replaceable>" - </literallayout> - </para> - </section> - <section id="selecting-an-initialization-manager"> <title>Selecting an Initialization Manager</title> @@ -8863,6 +10619,566 @@ Some notes from Cal: </section> </section> + + + + <section id='maintaining-build-output-quality'> + <title>Maintaining Build Output Quality</title> + + <para> + Many factors can influence the quality of a build. + For example, if you upgrade a recipe to use a new version of an + upstream software package or you experiment with some new + configuration options, subtle changes can occur that you might + not detect until later. + Consider the case where your recipe is using a newer version of + an upstream package. + In this case, a new version of a piece of software might + introduce an optional dependency on another library, which is + auto-detected. + If that library has already been built when the software is + building, the software will link to the built library and that + library will be pulled into your image along with the new + software even if you did not want the library. + </para> + + <para> + The + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-buildhistory'><filename>buildhistory</filename></ulink> + class exists to help you maintain the quality of your build + output. + You can use the class to highlight unexpected and possibly + unwanted changes in the build output. + When you enable build history, it records information about the + contents of each package and image and then commits that + information to a local Git repository where you can examine + the information. + </para> + + <para> + The remainder of this section describes the following: + <itemizedlist> + <listitem><para> + How you can enable and disable build history + </para></listitem> + <listitem><para> + How to understand what the build history contains + </para></listitem> + <listitem><para> + How to limit the information used for build history + </para></listitem> + <listitem><para> + How to examine the build history from both a + command-line and web interface + </para></listitem> + </itemizedlist> + </para> + + <section id='enabling-and-disabling-build-history'> + <title>Enabling and Disabling Build History</title> + + <para> + Build history is disabled by default. + To enable it, add the following <filename>INHERIT</filename> + statement and set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT</filename></ulink> + variable to "1" at the end of your + <filename>conf/local.conf</filename> file found in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>: + <literallayout class='monospaced'> + INHERIT += "buildhistory" + BUILDHISTORY_COMMIT = "1" + </literallayout> + Enabling build history as previously described causes the + OpenEmbedded build system to collect build output information + and commit it as a single commit to a local + <ulink url='&YOCTO_DOCS_OM_URL;#git'>Git</ulink> + repository. + <note> + Enabling build history increases your build times slightly, + particularly for images, and increases the amount of disk + space used during the build. + </note> + </para> + + <para> + You can disable build history by removing the previous + statements from your <filename>conf/local.conf</filename> + file. + </para> + </section> + + <section id='understanding-what-the-build-history-contains'> + <title>Understanding What the Build History Contains</title> + + <para> + Build history information is kept in + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-TOPDIR'><filename>TOPDIR</filename></ulink><filename>}/buildhistory</filename> + in the Build Directory as defined by the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_DIR'><filename>BUILDHISTORY_DIR</filename></ulink> + variable. + The following is an example abbreviated listing: + <imagedata fileref="figures/buildhistory.png" align="center" width="6in" depth="4in" /> + </para> + + <para> + At the top level, a <filename>metadata-revs</filename> + file exists that lists the revisions of the repositories for + the enabled layers when the build was produced. + The rest of the data splits into separate + <filename>packages</filename>, <filename>images</filename> + and <filename>sdk</filename> directories, the contents of + which are described as follows. + </para> + + <section id='build-history-package-information'> + <title>Build History Package Information</title> + + <para> + The history for each package contains a text file that has + name-value pairs with information about the package. + For example, + <filename>buildhistory/packages/i586-poky-linux/busybox/busybox/latest</filename> + contains the following: + <literallayout class='monospaced'> + PV = 1.22.1 + PR = r32 + RPROVIDES = + RDEPENDS = glibc (>= 2.20) update-alternatives-opkg + RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d + PKGSIZE = 540168 + FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \ + /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \ + /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \ + /usr/share/pixmaps /usr/share/applications /usr/share/idl \ + /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers + FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \ + /etc/busybox.links.nosuid /etc/busybox.links.suid + </literallayout> + Most of these name-value pairs correspond to variables + used to produce the package. + The exceptions are <filename>FILELIST</filename>, which + is the actual list of files in the package, and + <filename>PKGSIZE</filename>, which is the total size of + files in the package in bytes. + </para> + + <para> + A file also exists that corresponds to the recipe from + which the package came (e.g. + <filename>buildhistory/packages/i586-poky-linux/busybox/latest</filename>): + <literallayout class='monospaced'> + PV = 1.22.1 + PR = r32 + DEPENDS = initscripts kern-tools-native update-rc.d-native \ + virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \ + virtual/libc virtual/update-alternatives + PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \ + busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \ + busybox-staticdev busybox-dev busybox-doc busybox-locale busybox + </literallayout> + </para> + + <para> + Finally, for those recipes fetched from a version control + system (e.g., Git), a file exists that lists source + revisions that are specified in the recipe and lists + the actual revisions used during the build. + Listed and actual revisions might differ when + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink> + is set to + ${<ulink url='&YOCTO_DOCS_REF_URL;#var-AUTOREV'><filename>AUTOREV</filename></ulink>}. + Here is an example assuming + <filename>buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev</filename>): + <literallayout class='monospaced'> + # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" + SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" + # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f" + SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f" + </literallayout> + You can use the + <filename>buildhistory-collect-srcrevs</filename> + command with the <filename>-a</filename> option to + collect the stored <filename>SRCREV</filename> values + from build history and report them in a format suitable for + use in global configuration (e.g., + <filename>local.conf</filename> or a distro include file) + to override floating <filename>AUTOREV</filename> values + to a fixed set of revisions. + Here is some example output from this command: + <literallayout class='monospaced'> + $ buildhistory-collect-srcrevs -a + # i586-poky-linux + SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072" + SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072" + SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a" + SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4" + # x86_64-linux + SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa" + SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf" + SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11" + SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072" + SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3" + SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca" + SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a" + SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff" + SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4" + # qemux86-poky-linux + SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" + SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f" + # all-poky-linux + SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11" + </literallayout> + <note> + Here are some notes on using the + <filename>buildhistory-collect-srcrevs</filename> + command: + <itemizedlist> + <listitem><para> + By default, only values where the + <filename>SRCREV</filename> was not hardcoded + (usually when <filename>AUTOREV</filename> + is used) are reported. + Use the <filename>-a</filename> option to + see all <filename>SRCREV</filename> values. + </para></listitem> + <listitem><para> + The output statements might not have any effect + if overrides are applied elsewhere in the + build system configuration. + Use the <filename>-f</filename> option to add + the <filename>forcevariable</filename> override + to each output line if you need to work around + this restriction. + </para></listitem> + <listitem><para> + The script does apply special handling when + building for multiple machines. + However, the script does place a comment before + each set of values that specifies which + triplet to which they belong as previously + shown (e.g., + <filename>i586-poky-linux</filename>). + </para></listitem> + </itemizedlist> + </note> + </para> + </section> + + <section id='build-history-image-information'> + <title>Build History Image Information</title> + + <para> + The files produced for each image are as follows: + <itemizedlist> + <listitem><para> + <filename>image-files:</filename> + A directory containing selected files from the root + filesystem. + The files are defined by + <ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_IMAGE_FILES'><filename>BUILDHISTORY_IMAGE_FILES</filename></ulink>. + </para></listitem> + <listitem><para> + <filename>build-id.txt:</filename> + Human-readable information about the build + configuration and metadata source revisions. + This file contains the full build header as printed + by BitBake. + </para></listitem> + <listitem><para> + <filename>*.dot:</filename> + Dependency graphs for the image that are + compatible with <filename>graphviz</filename>. + </para></listitem> + <listitem><para> + <filename>files-in-image.txt:</filename> + A list of files in the image with permissions, + owner, group, size, and symlink information. + </para></listitem> + <listitem><para> + <filename>image-info.txt:</filename> + A text file containing name-value pairs with + information about the image. + See the following listing example for more + information. + </para></listitem> + <listitem><para> + <filename>installed-package-names.txt:</filename> + A list of installed packages by name only. + </para></listitem> + <listitem><para> + <filename>installed-package-sizes.txt:</filename> + A list of installed packages ordered by size. + </para></listitem> + <listitem><para> + <filename>installed-packages.txt:</filename> + A list of installed packages with full package + filenames. + </para></listitem> + </itemizedlist> + <note> + Installed package information is able to be gathered + and produced even if package management is disabled + for the final image. + </note> + </para> + + <para> + Here is an example of <filename>image-info.txt</filename>: + <literallayout class='monospaced'> + DISTRO = poky + DISTRO_VERSION = 1.7 + USER_CLASSES = buildstats image-mklibs image-prelink + IMAGE_CLASSES = image_types + IMAGE_FEATURES = debug-tweaks + IMAGE_LINGUAS = + IMAGE_INSTALL = packagegroup-core-boot run-postinsts + BAD_RECOMMENDATIONS = + NO_RECOMMENDATIONS = + PACKAGE_EXCLUDE = + ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \ + write_image_manifest ; buildhistory_list_installed_image ; \ + buildhistory_get_image_installed ; ssh_allow_empty_password; \ + postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ; + IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ; + IMAGESIZE = 6900 + </literallayout> + Other than <filename>IMAGESIZE</filename>, which is the + total size of the files in the image in Kbytes, the + name-value pairs are variables that may have influenced the + content of the image. + This information is often useful when you are trying to + determine why a change in the package or file + listings has occurred. + </para> + </section> + + <section id='using-build-history-to-gather-image-information-only'> + <title>Using Build History to Gather Image Information Only</title> + + <para> + As you can see, build history produces image information, + including dependency graphs, so you can see why something + was pulled into the image. + If you are just interested in this information and not + interested in collecting specific package or SDK + information, you can enable writing only image information + without any history by adding the following to your + <filename>conf/local.conf</filename> file found in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>: + <literallayout class='monospaced'> + INHERIT += "buildhistory" + BUILDHISTORY_COMMIT = "0" + BUILDHISTORY_FEATURES = "image" + </literallayout> + Here, you set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_FEATURES'><filename>BUILDHISTORY_FEATURES</filename></ulink> + variable to use the image feature only. + </para> + </section> + + <section id='build-history-sdk-information'> + <title>Build History SDK Information</title> + + <para> + Build history collects similar information on the contents + of SDKs + (e.g. <filename>bitbake -c populate_sdk imagename</filename>) + as compared to information it collects for images. + Furthermore, this information differs depending on whether + an extensible or standard SDK is being produced. + </para> + + <para> + The following list shows the files produced for SDKs: + <itemizedlist> + <listitem><para> + <filename>files-in-sdk.txt:</filename> + A list of files in the SDK with permissions, + owner, group, size, and symlink information. + This list includes both the host and target parts + of the SDK. + </para></listitem> + <listitem><para> + <filename>sdk-info.txt:</filename> + A text file containing name-value pairs with + information about the SDK. + See the following listing example for more + information. + </para></listitem> + <listitem><para> + <filename>sstate-task-sizes.txt:</filename> + A text file containing name-value pairs with + information about task group sizes + (e.g. <filename>do_populate_sysroot</filename> + tasks have a total size). + The <filename>sstate-task-sizes.txt</filename> file + exists only when an extensible SDK is created. + </para></listitem> + <listitem><para> + <filename>sstate-package-sizes.txt:</filename> + A text file containing name-value pairs with + information for the shared-state packages and + sizes in the SDK. + The <filename>sstate-package-sizes.txt</filename> + file exists only when an extensible SDK is created. + </para></listitem> + <listitem><para> + <filename>sdk-files:</filename> + A folder that contains copies of the files + mentioned in + <filename>BUILDHISTORY_SDK_FILES</filename> if the + files are present in the output. + Additionally, the default value of + <filename>BUILDHISTORY_SDK_FILES</filename> is + specific to the extensible SDK although you can + set it differently if you would like to pull in + specific files from the standard SDK.</para> + + <para>The default files are + <filename>conf/local.conf</filename>, + <filename>conf/bblayers.conf</filename>, + <filename>conf/auto.conf</filename>, + <filename>conf/locked-sigs.inc</filename>, and + <filename>conf/devtool.conf</filename>. + Thus, for an extensible SDK, these files get + copied into the <filename>sdk-files</filename> + directory. + </para></listitem> + <listitem><para> + The following information appears under + each of the <filename>host</filename> + and <filename>target</filename> directories + for the portions of the SDK that run on the host + and on the target, respectively: + <note> + The following files for the most part are empty + when producing an extensible SDK because this + type of SDK is not constructed from packages + as is the standard SDK. + </note> + <itemizedlist> + <listitem><para> + <filename>depends.dot:</filename> + Dependency graph for the SDK that is + compatible with + <filename>graphviz</filename>. + </para></listitem> + <listitem><para> + <filename>installed-package-names.txt:</filename> + A list of installed packages by name only. + </para></listitem> + <listitem><para> + <filename>installed-package-sizes.txt:</filename> + A list of installed packages ordered by size. + </para></listitem> + <listitem><para> + <filename>installed-packages.txt:</filename> + A list of installed packages with full + package filenames. + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + </para> + + <para> + Here is an example of <filename>sdk-info.txt</filename>: + <literallayout class='monospaced'> + DISTRO = poky + DISTRO_VERSION = 1.3+snapshot-20130327 + SDK_NAME = poky-glibc-i686-arm + SDK_VERSION = 1.3+snapshot + SDKMACHINE = + SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs + BAD_RECOMMENDATIONS = + SDKSIZE = 352712 + </literallayout> + Other than <filename>SDKSIZE</filename>, which is the + total size of the files in the SDK in Kbytes, the + name-value pairs are variables that might have influenced + the content of the SDK. + This information is often useful when you are trying to + determine why a change in the package or file listings + has occurred. + </para> + </section> + + <section id='examining-build-history-information'> + <title>Examining Build History Information</title> + + <para> + You can examine build history output from the command + line or from a web interface. + </para> + + <para> + To see any changes that have occurred (assuming you have + <ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT</filename></ulink><filename> = "1"</filename>), + you can simply use any Git command that allows you to + view the history of a repository. + Here is one method: + <literallayout class='monospaced'> + $ git log -p + </literallayout> + You need to realize, however, that this method does show + changes that are not significant (e.g. a package's size + changing by a few bytes). + </para> + + <para> + A command-line tool called + <filename>buildhistory-diff</filename> does exist, though, + that queries the Git repository and prints just the + differences that might be significant in human-readable + form. + Here is an example: + <literallayout class='monospaced'> + $ ~/poky/poky/scripts/buildhistory-diff . HEAD^ + Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt): + /etc/anotherpkg.conf was added + /sbin/anotherpkg was added + * (installed-package-names.txt): + * anotherpkg was added + Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt): + anotherpkg was added + packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras" + * PR changed from "r0" to "r1" + * PV changed from "0.1.10" to "0.1.12" + packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%) + * PR changed from "r0" to "r1" + * PV changed from "0.1.10" to "0.1.12" + </literallayout> + <note> + The <filename>buildhistory-diff</filename> tool + requires the <filename>GitPython</filename> package. + Be sure to install it using Pip3 as follows: + <literallayout class='monospaced'> + $ pip3 install GitPython --user + </literallayout> + Alternatively, you can install + <filename>python3-git</filename> using the appropriate + distribution package manager (e.g. + <filename>apt-get</filename>, <filename>dnf</filename>, + or <filename>zipper</filename>). + </note> + </para> + + <para> + To see changes to the build history using a web interface, + follow the instruction in the <filename>README</filename> + file here. + <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/buildhistory-web/'></ulink>. + </para> + + <para> + Here is a sample screenshot of the interface: + <imagedata fileref="figures/buildhistory-web.png" align="center" scalefit="1" width="130%" contentdepth="130%" /> + </para> + </section> + </section> + </section> + <section id="performing-automated-runtime-testing"> <title>Performing Automated Runtime Testing</title> @@ -8926,6 +11242,25 @@ Some notes from Cal: which should generate a list of tap devices. This is the option typically chosen for Autobuilder-type environments. + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + Be sure to use an absolute path + when calling this script + with sudo. + </para></listitem> + <listitem><para> + The package recipe + <filename>qemu-helper-native</filename> + is required to run this script. + Build the package using the + following command: + <literallayout class='monospaced'> + $ bitbake qemu-helper-native + </literallayout> + </para></listitem> + </itemizedlist> + </note> </para></listitem> </itemizedlist></para></listitem> <listitem><para><emphasis>Set the @@ -9768,368 +12103,1009 @@ Some notes from Cal: </section> </section> - <section id="platdev-gdb-remotedebug"> - <title>Debugging With the GNU Project Debugger (GDB) Remotely</title> + <section id='usingpoky-debugging-tools-and-techniques'> + <title>Debugging Tools and Techniques</title> <para> - GDB allows you to examine running programs, which in turn helps you to understand and fix problems. - It also allows you to perform post-mortem style analysis of program crashes. - GDB is available as a package within the Yocto Project and is - installed in SDK images by default. - See the "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>" chapter - in the Yocto Project Reference Manual for a description of these images. - You can find information on GDB at <ulink url="http://sourceware.org/gdb/"/>. + The exact method for debugging build failures depends on the nature + of the problem and on the system's area from which the bug + originates. + Standard debugging practices such as comparison against the last + known working version with examination of the changes and the + re-application of steps to identify the one causing the problem are + valid for the Yocto Project just as they are for any other system. + Even though it is impossible to detail every possible potential + failure, this section provides some general tips to aid in + debugging given a variety of situations. + <note><title>Tip</title> + A useful feature for debugging is the error reporting tool. + Configuring the Yocto Project to use this tool causes the + OpenEmbedded build system to produce error reporting commands as + part of the console output. + You can enter the commands after the build completes to log + error information into a common database, that can help you + figure out what might be going wrong. + For information on how to enable and use this feature, see the + "<link linkend='using-the-error-reporting-tool'>Using the Error Reporting Tool</link>" + section. + </note> </para> - <tip> - For best results, install debug (<filename>-dbg</filename>) packages - for the applications you are going to debug. - Doing so makes extra debug symbols available that give you more - meaningful output. - </tip> - <para> - Sometimes, due to memory or disk space constraints, it is not possible - to use GDB directly on the remote target to debug applications. - These constraints arise because GDB needs to load the debugging information and the - binaries of the process being debugged. - Additionally, GDB needs to perform many computations to locate information such as function - names, variable names and values, stack traces and so forth - even before starting the - debugging process. - These extra computations place more load on the target system and can alter the - characteristics of the program being debugged. + The following list shows the debugging topics in the remainder of + this section: + <itemizedlist> + <listitem><para> + "<link linkend='dev-debugging-viewing-logs-from-failed-tasks'>Viewing Logs from Failed Tasks</link>" + describes how to find and view logs from tasks that + failed during the build process. + </para></listitem> + <listitem><para> + "<link linkend='dev-debugging-viewing-variable-values'>Viewing Variable Values</link>" + describes how to use the BitBake <filename>-e</filename> + option to examine variable values after a recipe has been + parsed. + </para></listitem> + <listitem><para> + "<link linkend='viewing-package-information-with-oe-pkgdata-util'>Viewing Package Information with <filename>oe-pkgdata-util</filename></link>" + describes how to use the + <filename>oe-pkgdata-util</filename> utility to query + <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink> + and display package-related information for built + packages. + </para></listitem> + <listitem><para> + "<link linkend='dev-viewing-dependencies-between-recipes-and-tasks'>Viewing Dependencies Between Recipes and Tasks</link>" + describes how to use the BitBake <filename>-g</filename> + option to display recipe dependency information used + during the build. + </para></listitem> + <listitem><para> + "<link linkend='dev-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</link>" + describes how to use the + <filename>bitbake-dumpsig</filename> command in + conjunction with key subdirectories in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> + to determine variable dependencies. + </para></listitem> + <listitem><para> + "<link linkend='dev-debugging-taskrunning'>Running Specific Tasks</link>" + describes how to use several BitBake options (e.g. + <filename>-c</filename>, <filename>-C</filename>, and + <filename>-f</filename>) to run specific tasks in the + build chain. + It can be useful to run tasks "out-of-order" when trying + isolate build issues. + </para></listitem> + <listitem><para> + "<link linkend='dev-debugging-bitbake'>General BitBake Problems</link>" + describes how to use BitBake's <filename>-D</filename> + debug output option to reveal more about what BitBake is + doing during the build. + </para></listitem> + <listitem><para> + "<link linkend='dev-debugging-buildfile'>Building with No Dependencies</link>" + describes how to use the BitBake <filename>-b</filename> + option to build a recipe while ignoring dependencies. + </para></listitem> + <listitem><para> + "<link linkend='recipe-logging-mechanisms'>Recipe Logging Mechanisms</link>" + describes how to use the many recipe logging functions + to produce debugging output and report errors and warnings. + </para></listitem> + <listitem><para> + "<link linkend='debugging-parallel-make-races'>Debugging Parallel Make Races</link>" + describes how to debug situations where the build consists + of several parts that are run simultaneously and when the + output or result of one part is not ready for use with a + different part of the build that depends on that output. + </para></listitem> + <listitem><para> + "<link linkend='platdev-gdb-remotedebug'>Debugging With the GNU Project Debugger (GDB) Remotely</link>" + describes how to use GDB to allow you to examine running + programs, which can help you fix problems. + </para></listitem> + <listitem><para> + "<link linkend='debugging-with-the-gnu-project-debugger-gdb-on-the-target'>Debugging with the GNU Project Debugger (GDB) on the Target</link>" + describes how to use GDB directly on target hardware for + debugging. + </para></listitem> + <listitem><para> + "<link linkend='dev-other-debugging-others'>Other Debugging Tips</link>" + describes miscellaneous debugging tips that can be useful. + </para></listitem> + </itemizedlist> </para> <para> - To help get past the previously mentioned constraints, you can use - gdbserver, which runs on the remote target and does not load any - debugging information from the debugged process. - Instead, a GDB instance processes the debugging information that is run on a - remote computer - the host GDB. - The host GDB then sends control commands to gdbserver to make it stop or start the debugged - program, as well as read or write memory regions of that debugged program. - All the debugging information loaded and processed as well - as all the heavy debugging is done by the host GDB. - Offloading these processes gives the gdbserver running on the target a chance to remain - small and fast. + For debugging information within the popular + <trademark class='trade'>Eclipse</trademark> IDE, see the + "<ulink url='&YOCTO_DOCS_SDK_URL;#adt-eclipse'>Working within Eclipse</ulink>" + section in the Yocto Project Application Development and the + Extensible Software Development Kit (eSDK) manual. </para> - <para> - Because the host GDB is responsible for loading the debugging information and - for doing the necessary processing to make actual debugging happen, - you have to make sure the host can access the unstripped binaries complete - with their debugging information and also be sure the target is compiled with no optimizations. - The host GDB must also have local access to all the libraries used by the - debugged program. - Because gdbserver does not need any local debugging information, the binaries on - the remote target can remain stripped. - However, the binaries must also be compiled without optimization - so they match the host's binaries. - </para> + <section id='dev-debugging-viewing-logs-from-failed-tasks'> + <title>Viewing Logs from Failed Tasks</title> - <para> - To remain consistent with GDB documentation and terminology, the binary being debugged - on the remote target machine is referred to as the "inferior" binary. - For documentation on GDB see the - <ulink url="http://sourceware.org/gdb/documentation/">GDB site</ulink>. - </para> + <para> + You can find the log for a task in the file + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}/temp/log.do_</filename><replaceable>taskname</replaceable>. + For example, the log for the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink> + task of the QEMU minimal image for the x86 machine + (<filename>qemux86</filename>) might be in + <filename>tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile</filename>. + To see the commands + <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> + ran to generate a log, look at the corresponding + <filename>run.do_</filename><replaceable>taskname</replaceable> + file in the same directory. + </para> - <para> - The following steps show you how to debug using the GNU project - debugger. - <orderedlist> - <listitem><para> - <emphasis>Configure your build system to construct the - companion debug filesystem:</emphasis></para> + <para> + <filename>log.do_</filename><replaceable>taskname</replaceable> + and + <filename>run.do_</filename><replaceable>taskname</replaceable> + are actually symbolic links to + <filename>log.do_</filename><replaceable>taskname</replaceable><filename>.</filename><replaceable>pid</replaceable> + and + <filename>log.run_</filename><replaceable>taskname</replaceable><filename>.</filename><replaceable>pid</replaceable>, + where <replaceable>pid</replaceable> is the PID the task had + when it ran. + The symlinks always point to the files corresponding to the most + recent run. + </para> + </section> - <para>In your <filename>local.conf</filename> file, set - the following: - <literallayout class='monospaced'> - IMAGE_GEN_DEBUGFS = "1" - IMAGE_FSTYPES_DEBUGFS = "tar.bz2" - </literallayout> - These options cause the OpenEmbedded build system - to generate a special companion filesystem fragment, - which contains the matching source and debug symbols to - your deployable filesystem. - The build system does this by looking at what is in the - deployed filesystem, and pulling the corresponding - <filename>-dbg</filename> packages.</para> - - <para>The companion debug filesystem is not a complete - filesystem, but only contains the debug fragments. - This filesystem must be combined with the full filesystem - for debugging. - Subsequent steps in this procedure show how to combine - the partial filesystem with the full filesystem. - </para></listitem> - <listitem><para> - <emphasis>Configure the system to include gdbserver in - the target filesystem:</emphasis></para> + <section id='dev-debugging-viewing-variable-values'> + <title>Viewing Variable Values</title> - <para>Make the following addition in either your - <filename>local.conf</filename> file or in an image - recipe: - <literallayout class='monospaced'> - IMAGE_INSTALL_append = “ gdbserver" - </literallayout> - The change makes sure the <filename>gdbserver</filename> - package is included. - </para></listitem> - <listitem><para> - <emphasis>Build the environment:</emphasis></para> + <para> + BitBake's <filename>-e</filename> option is used to display + variable values after parsing. + The following command displays the variable values after the + configuration files (i.e. <filename>local.conf</filename>, + <filename>bblayers.conf</filename>, + <filename>bitbake.conf</filename> and so forth) have been + parsed: + <literallayout class='monospaced'> + $ bitbake -e + </literallayout> + The following command displays variable values after a specific + recipe has been parsed. + The variables include those from the configuration as well: + <literallayout class='monospaced'> + $ bitbake -e recipename + </literallayout> + <note><para> + Each recipe has its own private set of variables + (datastore). + Internally, after parsing the configuration, a copy of the + resulting datastore is made prior to parsing each recipe. + This copying implies that variables set in one recipe will + not be visible to other recipes.</para> + + <para>Likewise, each task within a recipe gets a private + datastore based on the recipe datastore, which means that + variables set within one task will not be visible to + other tasks.</para> + </note> + </para> - <para>Use the following command to construct the image and - the companion Debug Filesystem: - <literallayout class='monospaced'> - $ bitbake <replaceable>image</replaceable> - </literallayout> - Build the cross GDB component and make it available - for debugging. - Build the SDK that matches the image. - Building the SDK is best for a production build - that can be used later for debugging, especially - during long term maintenance: - <literallayout class='monospaced'> - $ bitbake -c populate_sdk <replaceable>image</replaceable> - </literallayout></para> - - <para>Alternatively, you can build the minimal - toolchain components that match the target. - Doing so creates a smaller than typical SDK and only - contains a minimal set of components with which to - build simple test applications, as well as run the - debugger: - <literallayout class='monospaced'> - $ bitbake meta-toolchain - </literallayout></para> + <para> + In the output of <filename>bitbake -e</filename>, each + variable is preceded by a description of how the variable + got its value, including temporary values that were later + overriden. + This description also includes variable flags (varflags) set on + the variable. + The output can be very helpful during debugging. + </para> - <para>A final method is to build Gdb itself within - the build system: - <literallayout class='monospaced'> - $ bitbake gdb-cross-<replaceable>architecture</replaceable> - </literallayout> - Doing so produces a temporary copy of - <filename>cross-gdb</filename> you can use for - debugging during development. - While this is the quickest approach, the two previous - methods in this step are better when considering - long-term maintenance strategies. - <note> - If you run - <filename>bitbake gdb-cross</filename>, the - OpenEmbedded build system suggests the actual - image (e.g. <filename>gdb-cross-i586</filename>). - The suggestion is usually the actual name you want - to use. - </note> - </para></listitem> - <listitem><para> - <emphasis>Set up the</emphasis> <filename>debugfs</filename></para> + <para> + Variables that are exported to the environment are preceded by + <filename>export</filename> in the output of + <filename>bitbake -e</filename>. + See the following example: + <literallayout class='monospaced'> + export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86" + </literallayout> + </para> - <para>Run the following commands to set up the - <filename>debugfs</filename>: - <literallayout class='monospaced'> - $ mkdir debugfs - $ cd debugfs - $ tar xvfj <replaceable>build-dir</replaceable>/tmp-glibc/deploy/images/<replaceable>machine</replaceable>/<replaceable>image</replaceable>.rootfs.tar.bz2 - $ tar xvfj <replaceable>build-dir</replaceable>/tmp-glibc/deploy/images/<replaceable>machine</replaceable>/<replaceable>image</replaceable>-dbg.rootfs.tar.bz2 - </literallayout> - </para></listitem> - <listitem><para> - <emphasis>Set up GDB</emphasis></para> + <para> + In addition to variable values, the output of the + <filename>bitbake -e</filename> and + <filename>bitbake -e</filename> <replaceable>recipe</replaceable> + commands includes the following information: + <itemizedlist> + <listitem><para> + The output starts with a tree listing all configuration + files and classes included globally, recursively listing + the files they include or inherit in turn. + Much of the behavior of the OpenEmbedded build system + (including the behavior of the + <ulink url='&YOCTO_DOCS_REF_URL;#normal-recipe-build-tasks'>normal recipe build tasks</ulink>) + is implemented in the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-base'><filename>base</filename></ulink> + class and the classes it inherits, rather than being + built into BitBake itself. + </para></listitem> + <listitem><para> + After the variable values, all functions appear in the + output. + For shell functions, variables referenced within the + function body are expanded. + If a function has been modified using overrides or + using override-style operators like + <filename>_append</filename> and + <filename>_prepend</filename>, then the final assembled + function body appears in the output. + </para></listitem> + </itemizedlist> + </para> + </section> - <para>Install the SDK (if you built one) and then - source the correct environment file. - Sourcing the environment file puts the SDK in your - <filename>PATH</filename> environment variable.</para> + <section id='viewing-package-information-with-oe-pkgdata-util'> + <title>Viewing Package Information with <filename>oe-pkgdata-util</filename></title> - <para>If you are using the build system, Gdb is - located in - <replaceable>build-dir</replaceable>/tmp/sysroots/<replaceable>host</replaceable>/usr/bin/<replaceable>architecture</replaceable>/<replaceable>architecture</replaceable>-gdb - </para></listitem> - <listitem><para> - <emphasis>Boot the target:</emphasis></para> + <para> + You can use the <filename>oe-pkgdata-util</filename> + command-line utility to query + <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink> + and display various package-related information. + When you use the utility, you must use it to view information + on packages that have already been built. + </para> - <para>For information on how to run QEMU, see the - <ulink url='http://wiki.qemu.org/Documentation/GettingStartedDevelopers'>QEMU Documentation</ulink>. - <note> - Be sure to verify that your host can access the - target via TCP. - </note> - </para></listitem> - <listitem><para> - <emphasis>Debug a program:</emphasis></para> + <para> + Following are a few of the available + <filename>oe-pkgdata-util</filename> subcommands. + <note> + You can use the standard * and ? globbing wildcards as part + of package names and paths. + </note> + <itemizedlist> + <listitem><para> + <filename>oe-pkgdata-util list-pkgs [</filename><replaceable>pattern</replaceable><filename>]</filename>: + Lists all packages that have been built, optionally + limiting the match to packages that match + <replaceable>pattern</replaceable>. + </para></listitem> + <listitem><para> + <filename>oe-pkgdata-util list-pkg-files </filename><replaceable>package</replaceable><filename> ...</filename>: + Lists the files and directories contained in the given + packages. + <note> + <para> + A different way to view the contents of a package is + to look at the + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}/packages-split</filename> + directory of the recipe that generates the + package. + This directory is created by the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> + task and has one subdirectory for each package the + recipe generates, which contains the files stored in + that package.</para> + <para> + If you want to inspect the + <filename>${WORKDIR}/packages-split</filename> + directory, make sure that + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-rm-work'><filename>rm_work</filename></ulink> + is not enabled when you build the recipe. + </para> + </note> + </para></listitem> + <listitem><para> + <filename>oe-pkgdata-util find-path </filename><replaceable>path</replaceable><filename> ...</filename>: + Lists the names of the packages that contain the given + paths. + For example, the following tells us that + <filename>/usr/share/man/man1/make.1</filename> + is contained in the <filename>make-doc</filename> + package: + <literallayout class='monospaced'> + $ oe-pkgdata-util find-path /usr/share/man/man1/make.1 + make-doc: /usr/share/man/man1/make.1 + </literallayout> + </para></listitem> + <listitem><para> + <filename>oe-pkgdata-util lookup-recipe </filename><replaceable>package</replaceable><filename> ...</filename>: + Lists the name of the recipes that + produce the given packages. + </para></listitem> + </itemizedlist> + </para> - <para>Debugging a program involves running gdbserver - on the target and then running Gdb on the host. - The example in this step debugs - <filename>gzip</filename>: - <literallayout class='monospaced'> - root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help - </literallayout> - For additional gdbserver options, see the - <ulink url='https://www.gnu.org/software/gdb/documentation/'>GDB Server Documentation</ulink>. - </para> + <para> + For more information on the <filename>oe-pkgdata-util</filename> + command, use the help facility: + <literallayout class='monospaced'> + $ oe-pkgdata-util ‐‐help + $ oe-pkgdata-util <replaceable>subcommand</replaceable> --help + </literallayout> + </para> + </section> - <para>After running gdbserver on the target, you need - to run Gdb on the host and configure it and connect to - the target. - Use these commands: - <literallayout class='monospaced'> - $ cd <replaceable>directory-holding-the-debugfs-directory</replaceable> - $ <replaceable>arch</replaceable>-gdb + <section id='dev-viewing-dependencies-between-recipes-and-tasks'> + <title>Viewing Dependencies Between Recipes and Tasks</title> - (gdb) set sysroot debugfs - (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug - (gdb) target remote <replaceable>IP-of-target</replaceable>:1234 - </literallayout> - At this point, everything should automatically load - (i.e. matching binaries, symbols and headers). - <note> - The Gdb <filename>set</filename> commands in the - previous example can be placed into the users - <filename>~/.gdbinit</filename> file. - Upon starting, Gdb automatically runs whatever - commands are in that file. - </note> - </para></listitem> - <listitem><para> - <emphasis>Deploying without a full image - rebuild:</emphasis></para> + <para> + Sometimes it can be hard to see why BitBake wants to build other + recipes before the one you have specified. + Dependency information can help you understand why a recipe is + built. + </para> - <para>In many cases, during development you want a - quick method to deploy a new binary to the target and - debug it, without waiting for a full image build. - </para> + <para> + To generate dependency information for a recipe, run the + following command: + <literallayout class='monospaced'> + $ bitbake -g <replaceable>recipename</replaceable> + </literallayout> + This command writes the following files in the current + directory: + <itemizedlist> + <listitem><para> + <filename>pn-buildlist</filename>: A list of + recipes/targets involved in building + <replaceable>recipename</replaceable>. + "Involved" here means that at least one task from the + recipe needs to run when building + <replaceable>recipename</replaceable> from scratch. + Targets that are in + <ulink url='&YOCTO_DOCS_REF_URL;#var-ASSUME_PROVIDED'><filename>ASSUME_PROVIDED</filename></ulink> + are not listed. + </para></listitem> + <listitem><para> + <filename>task-depends.dot</filename>: A graph showing + dependencies between tasks. + </para></listitem> + </itemizedlist> + </para> - <para>One approach to solving this situation is to - just build the component you want to debug. - Once you have built the component, copy the - executable directly to both the target and the - host <filename>debugfs</filename>.</para> - - <para>If the binary is processed through the debug - splitting in OpenEmbedded, you should also - copy the debug items (i.e. <filename>.debug</filename> - contents and corresponding - <filename>/usr/src/debug</filename> files) - from the work directory. - Here is an example: - <literallayout class='monospaced'> - $ bitbake bash - $ bitbake -c devshell bash - $ cd .. - $ scp packages-split/bash/bin/bash <replaceable>target</replaceable>:/bin/bash - $ cp -a packages-split/bash-dbg/* <replaceable>path</replaceable>/debugfs - </literallayout> - </para></listitem> - </orderedlist> - </para> - </section> + <para> + The graphs are in + <ulink url='https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29'>DOT</ulink> + format and can be converted to images (e.g. using the + <filename>dot</filename> tool from + <ulink url='http://www.graphviz.org/'>Graphviz</ulink>). + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + DOT files use a plain text format. + The graphs generated using the + <filename>bitbake -g</filename> command are often so + large as to be difficult to read without special + pruning (e.g. with Bitbake's + <filename>-I</filename> option) and processing. + Despite the form and size of the graphs, the + corresponding <filename>.dot</filename> files can + still be possible to read and provide useful + information. + </para> - <section id='debugging-with-the-gnu-project-debugger-gdb-on-the-target'> - <title>Debugging with the GNU Project Debugger (GDB) on the Target</title> + <para>As an example, the + <filename>task-depends.dot</filename> file contains + lines such as the following: + <literallayout class='monospaced'> + "libxslt.do_configure" -> "libxml2.do_populate_sysroot" + </literallayout> + The above example line reveals that the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink> + task in <filename>libxslt</filename> depends on the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink> + task in <filename>libxml2</filename>, which is a + normal + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> + dependency between the two recipes. + </para></listitem> + <listitem><para> + For an example of how <filename>.dot</filename> + files can be processed, see the + <filename>scripts/contrib/graph-tool</filename> + Python script, which finds and displays paths + between graph nodes. + </para></listitem> + </itemizedlist> + </note> + </para> - <para> - The previous section addressed using GDB remotely for debugging - purposes, which is the most usual case due to the inherent - hardware limitations on many embedded devices. - However, debugging in the target hardware itself is also possible - with more powerful devices. - This section describes what you need to do in order to support - using GDB to debug on the target hardware. - </para> + <para> + You can use a different method to view dependency information + by using the following command: + <literallayout class='monospaced'> + $ bitbake -g -u taskexp <replaceable>recipename</replaceable> + </literallayout> + This command displays a GUI window from which you can view + build-time and runtime dependencies for the recipes involved in + building <replaceable>recipename</replaceable>. + </para> + </section> - <para> - To support this kind of debugging, you need do the following: - <itemizedlist> - <listitem><para> - Ensure that GDB is on the target. - You can do this by adding "gdb" to - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink>: - <literallayout class='monospaced'> - IMAGE_INSTALL_append = " gdb" - </literallayout> - Alternatively, you can add "tools-debug" to - <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink>: - <literallayout class='monospaced'> - IMAGE_FEATURES_append = " tools-debug" - </literallayout> - </para></listitem> - <listitem><para> - Ensure that debug symbols are present. - You can make sure these symbols are present by installing - <filename>-dbg</filename>: + <section id='dev-viewing-task-variable-dependencies'> + <title>Viewing Task Variable Dependencies</title> + + <para> + As mentioned in the + "<ulink url='&YOCTO_DOCS_BB_URL;#checksums'>Checksums (Signatures)</ulink>" + section of the BitBake User Manual, BitBake tries to + automatically determine what variables a task depends on so + that it can rerun the task if any values of the variables + change. + This determination is usually reliable. + However, if you do things like construct variable names at + runtime, then you might have to manually declare dependencies + on those variables using <filename>vardeps</filename> as + described in the + "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>" + section of the BitBake User Manual. + </para> + + <para> + If you are unsure whether a variable dependency is being + picked up automatically for a given task, you can list the + variable dependencies BitBake has determined by doing the + following: + <orderedlist> + <listitem><para> + Build the recipe containing the task: + <literallayout class='monospaced'> + $ bitbake <replaceable>recipename</replaceable> + </literallayout> + </para></listitem> + <listitem><para> + Inside the + <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMPS_DIR'><filename>STAMPS_DIR</filename></ulink> + directory, find the signature data + (<filename>sigdata</filename>) file that corresponds + to the task. + The <filename>sigdata</filename> files contain a pickled + Python database of all the metadata that went into + creating the input checksum for the task. + As an example, for the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink> + task of the <filename>db</filename> recipe, the + <filename>sigdata</filename> file might be found in the + following location: + <literallayout class='monospaced'> + ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1 + </literallayout> + For tasks that are accelerated through the shared state + (<ulink url='&YOCTO_DOCS_OM_URL;#shared-state-cache'>sstate</ulink>) + cache, an additional <filename>siginfo</filename> file + is written into + <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink> + along with the cached task output. + The <filename>siginfo</filename> files contain exactly + the same information as <filename>sigdata</filename> + files. + </para></listitem> + <listitem><para> + Run <filename>bitbake-dumpsig</filename> on the + <filename>sigdata</filename> or + <filename>siginfo</filename> file. + Here is an example: + <literallayout class='monospaced'> + $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1 + </literallayout> + In the output of the above command, you will find a + line like the following, which lists all the (inferred) + variable dependencies for the task. + This list also includes indirect dependencies from + variables depending on other variables, recursively. + <literallayout class='monospaced'> + Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch'] + </literallayout> + <note> + Functions (e.g. <filename>base_do_fetch</filename>) + also count as variable dependencies. + These functions in turn depend on the variables they + reference. + </note> + The output of <filename>bitbake-dumpsig</filename> also + includes the value each variable had, a list of + dependencies for each variable, and + <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></ulink> + information. + </para></listitem> + </orderedlist> + </para> + + <para> + There is also a <filename>bitbake-diffsigs</filename> command + for comparing two <filename>siginfo</filename> or + <filename>sigdata</filename> files. + This command can be helpful when trying to figure out what + changed between two versions of a task. + If you call <filename>bitbake-diffsigs</filename> with just one + file, the command behaves like + <filename>bitbake-dumpsig</filename>. + </para> + + <para> + You can also use BitBake to dump out the signature construction + information without executing tasks by using either of the + following BitBake command-line options: + <literallayout class='monospaced'> + ‐‐dump-signatures=<replaceable>SIGNATURE_HANDLER</replaceable> + -S <replaceable>SIGNATURE_HANDLER</replaceable> + </literallayout> + <note> + Two common values for + <replaceable>SIGNATURE_HANDLER</replaceable> are "none" and + "printdiff", which dump only the signature or compare the + dumped signature with the cached one, respectively. + </note> + Using BitBake with either of these options causes BitBake to + dump out <filename>sigdata</filename> files in the + <filename>stamps</filename> directory for every task it would + have executed instead of building the specified target package. + </para> + </section> + + <section id='dev-viewing-metadata-used-to-create-the-input-signature-of-a-shared-state-task'> + <title>Viewing Metadata Used to Create the Input Signature of a Shared State Task</title> + + <para> + Seeing what metadata went into creating the input signature + of a shared state (sstate) task can be a useful debugging + aid. + This information is available in signature information + (<filename>siginfo</filename>) files in + <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>. + For information on how to view and interpret information in + <filename>siginfo</filename> files, see the + "<link linkend='dev-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</link>" + section. + </para> + + <para> + For conceptual information on shared state, see the + "<ulink url='&YOCTO_DOCS_OM_URL;#shared-state'>Shared State</ulink>" + section in the Yocto Project Overview and Concepts Manual. + </para> + </section> + + <section id='dev-invalidating-shared-state-to-force-a-task-to-run'> + <title>Invalidating Shared State to Force a Task to Run</title> + + <para> + The OpenEmbedded build system uses + <ulink url='&YOCTO_DOCS_OM_URL;#overview-checksums'>checksums</ulink> + and + <ulink url='&YOCTO_DOCS_OM_URL;#shared-state'>shared state</ulink> + cache to avoid unnecessarily rebuilding tasks. + Collectively, this scheme is known as "shared state code." + </para> + + <para> + As with all schemes, this one has some drawbacks. + It is possible that you could make implicit changes to your + code that the checksum calculations do not take into + account. + These implicit changes affect a task's output but do not + trigger the shared state code into rebuilding a recipe. + Consider an example during which a tool changes its output. + Assume that the output of <filename>rpmdeps</filename> + changes. + The result of the change should be that all the + <filename>package</filename> and + <filename>package_write_rpm</filename> shared state cache + items become invalid. + However, because the change to the output is + external to the code and therefore implicit, + the associated shared state cache items do not become + invalidated. + In this case, the build process uses the cached items + rather than running the task again. + Obviously, these types of implicit changes can cause + problems. + </para> + + <para> + To avoid these problems during the build, you need to + understand the effects of any changes you make. + Realize that changes you make directly to a function + are automatically factored into the checksum calculation. + Thus, these explicit changes invalidate the associated + area of shared state cache. + However, you need to be aware of any implicit changes that + are not obvious changes to the code and could affect + the output of a given task. + </para> + + <para> + When you identify an implicit change, you can easily + take steps to invalidate the cache and force the tasks + to run. + The steps you can take are as simple as changing a + function's comments in the source code. + For example, to invalidate package shared state files, + change the comment statements of + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> + or the comments of one of the functions it calls. + Even though the change is purely cosmetic, it causes the + checksum to be recalculated and forces the build system to + run the task again. + <note> + For an example of a commit that makes a cosmetic + change to invalidate shared state, see this + <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>. + </note> + </para> + </section> + + <section id='dev-debugging-taskrunning'> + <title>Running Specific Tasks</title> + + <para> + Any given recipe consists of a set of tasks. + The standard BitBake behavior in most cases is: + <filename>do_fetch</filename>, + <filename>do_unpack</filename>, + <filename>do_patch</filename>, + <filename>do_configure</filename>, + <filename>do_compile</filename>, + <filename>do_install</filename>, + <filename>do_package</filename>, + <filename>do_package_write_*</filename>, and + <filename>do_build</filename>. + The default task is <filename>do_build</filename> and any tasks + on which it depends build first. + Some tasks, such as <filename>do_devshell</filename>, are not + part of the default build chain. + If you wish to run a task that is not part of the default build + chain, you can use the <filename>-c</filename> option in + BitBake. + Here is an example: + <literallayout class='monospaced'> + $ bitbake matchbox-desktop -c devshell + </literallayout> + </para> + + <para> + The <filename>-c</filename> option respects task dependencies, + which means that all other tasks (including tasks from other + recipes) that the specified task depends on will be run before + the task. + Even when you manually specify a task to run with + <filename>-c</filename>, BitBake will only run the task if it + considers it "out of date". + See the + "<ulink url='&YOCTO_DOCS_OM_URL;#stamp-files-and-the-rerunning-of-tasks'>Stamp Files and the Rerunning of Tasks</ulink>" + section in the Yocto Project Overview and Concepts Manual for + how BitBake determines whether a task is "out of date". + </para> + + <para> + If you want to force an up-to-date task to be rerun (e.g. + because you made manual modifications to the recipe's + <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink> + that you want to try out), then you can use the + <filename>-f</filename> option. + <note> + The reason <filename>-f</filename> is never required when + running the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-devshell'><filename>do_devshell</filename></ulink> + task is because the + <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink><filename>]</filename> + variable flag is already set for the task. + </note> + The following example shows one way you can use the + <filename>-f</filename> option: + <literallayout class='monospaced'> + $ bitbake matchbox-desktop + . + . + make some changes to the source code in the work directory + . + . + $ bitbake matchbox-desktop -c compile -f + $ bitbake matchbox-desktop + </literallayout> + </para> + + <para> + This sequence first builds and then recompiles + <filename>matchbox-desktop</filename>. + The last command reruns all tasks (basically the packaging + tasks) after the compile. + BitBake recognizes that the <filename>do_compile</filename> + task was rerun and therefore understands that the other tasks + also need to be run again. + </para> + + <para> + Another, shorter way to rerun a task and all + <ulink url='&YOCTO_DOCS_REF_URL;#normal-recipe-build-tasks'>normal recipe build tasks</ulink> + that depend on it is to use the <filename>-C</filename> + option. + <note> + This option is upper-cased and is separate from the + <filename>-c</filename> option, which is lower-cased. + </note> + Using this option invalidates the given task and then runs the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-build'><filename>do_build</filename></ulink> + task, which is the default task if no task is given, and the + tasks on which it depends. + You could replace the final two commands in the previous example + with the following single command: + <literallayout class='monospaced'> + $ bitbake matchbox-desktop -C compile + </literallayout> + Internally, the <filename>-f</filename> and + <filename>-C</filename> options work by tainting (modifying) the + input checksum of the specified task. + This tainting indirectly causes the task and its + dependent tasks to be rerun through the normal task dependency + mechanisms. + <note> + BitBake explicitly keeps track of which tasks have been + tainted in this fashion, and will print warnings such as the + following for builds involving such tasks: <literallayout class='monospaced'> - IMAGE_INSTALL_append = " <replaceable>packagename</replaceable>-dbg" + WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run </literallayout> - Alternatively, you can do the following to include all the - debug symbols: + The purpose of the warning is to let you know that the work + directory and build output might not be in the clean state + they would be in for a "normal" build, depending on what + actions you took. + To get rid of such warnings, you can remove the work + directory and rebuild the recipe, as follows: <literallayout class='monospaced'> - IMAGE_FEATURES_append = " dbg-pkgs" + $ bitbake matchbox-desktop -c clean + $ bitbake matchbox-desktop </literallayout> - </para></listitem> - </itemizedlist> - <note> - To improve the debug information accuracy, you can reduce the - level of optimization used by the compiler. - For example, when adding the following line to your - <filename>local.conf</filename> file, you will reduce - optimization from - <ulink url='&YOCTO_DOCS_REF_URL;#var-FULL_OPTIMIZATION'><filename>FULL_OPTIMIZATION</filename></ulink> - of "-O2" to - <ulink url='&YOCTO_DOCS_REF_URL;#var-DEBUG_OPTIMIZATION'><filename>DEBUG_OPTIMIZATION</filename></ulink> - of "-O -fno-omit-frame-pointer": + </note> + </para> + + <para> + You can view a list of tasks in a given package by running the + <filename>do_listtasks</filename> task as follows: <literallayout class='monospaced'> - DEBUG_BUILD = "1" + $ bitbake matchbox-desktop -c listtasks </literallayout> - Consider that this will reduce the application's performance - and is recommended only for debugging purposes. - </note> - </para> - </section> + The results appear as output to the console and are also in the + file <filename>${WORKDIR}/temp/log.do_listtasks</filename>. + </para> + </section> - <section id='debugging-parallel-make-races'> - <title>Debugging Parallel Make Races</title> + <section id='dev-debugging-bitbake'> + <title>General BitBake Problems</title> - <para> - A parallel <filename>make</filename> race occurs when the build - consists of several parts that are run simultaneously and - a situation occurs when the output or result of one - part is not ready for use with a different part of the build that - depends on that output. - Parallel make races are annoying and can sometimes be difficult - to reproduce and fix. - However, some simple tips and tricks exist that can help - you debug and fix them. - This section presents a real-world example of an error encountered - on the Yocto Project autobuilder and the process used to fix it. - <note> - If you cannot properly fix a <filename>make</filename> race - condition, you can work around it by clearing either the - <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></ulink> - or - <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKEINST'><filename>PARALLEL_MAKEINST</filename></ulink> - variables. - </note> - </para> + <para> + You can see debug output from BitBake by using the + <filename>-D</filename> option. + The debug output gives more information about what BitBake + is doing and the reason behind it. + Each <filename>-D</filename> option you use increases the + logging level. + The most common usage is <filename>-DDD</filename>. + </para> - <section id='the-failure'> - <title>The Failure</title> + <para> + The output from + <filename>bitbake -DDD -v</filename> <replaceable>targetname</replaceable> + can reveal why BitBake chose a certain version of a package or + why BitBake picked a certain provider. + This command could also help you in a situation where you think + BitBake did something unexpected. + </para> + </section> + + <section id='dev-debugging-buildfile'> + <title>Building with No Dependencies</title> <para> - For this example, assume that you are building an image that - depends on the "neard" package. - And, during the build, BitBake runs into problems and - creates the following output. + To build a specific recipe (<filename>.bb</filename> file), + you can use the following command form: + <literallayout class='monospaced'> + $ bitbake -b <replaceable>somepath</replaceable>/<replaceable>somerecipe</replaceable>.bb + </literallayout> + This command form does not check for dependencies. + Consequently, you should use it only when you know existing + dependencies have been met. <note> - This example log file has longer lines artificially - broken to make the listing easier to read. + You can also specify fragments of the filename. + In this case, BitBake checks for a unique match. </note> - If you examine the output or the log file, you see the - failure during <filename>make</filename>: - <literallayout class='monospaced'> + </para> + </section> + + <section id='recipe-logging-mechanisms'> + <title>Recipe Logging Mechanisms</title> + + <para> + The Yocto Project provides several logging functions for + producing debugging output and reporting errors and warnings. + For Python functions, the following logging functions exist. + All of these functions log to + <filename>${T}/log.do_</filename><replaceable>task</replaceable>, + and can also log to standard output (stdout) with the right + settings: + <itemizedlist> + <listitem><para> + <filename>bb.plain(</filename><replaceable>msg</replaceable><filename>)</filename>: + Writes <replaceable>msg</replaceable> as is to the + log while also logging to stdout. + </para></listitem> + <listitem><para> + <filename>bb.note(</filename><replaceable>msg</replaceable><filename>)</filename>: + Writes "NOTE: <replaceable>msg</replaceable>" to the + log. + Also logs to stdout if BitBake is called with "-v". + </para></listitem> + <listitem><para> + <filename>bb.debug(</filename><replaceable>level</replaceable><filename>, </filename><replaceable>msg</replaceable><filename>)</filename>: + Writes "DEBUG: <replaceable>msg</replaceable>" to the + log. + Also logs to stdout if the log level is greater than or + equal to <replaceable>level</replaceable>. + See the + "<ulink url='&YOCTO_DOCS_BB_URL;#usage-and-syntax'>-D</ulink>" + option in the BitBake User Manual for more information. + </para></listitem> + <listitem><para> + <filename>bb.warn(</filename><replaceable>msg</replaceable><filename>)</filename>: + Writes "WARNING: <replaceable>msg</replaceable>" to the + log while also logging to stdout. + </para></listitem> + <listitem><para> + <filename>bb.error(</filename><replaceable>msg</replaceable><filename>)</filename>: + Writes "ERROR: <replaceable>msg</replaceable>" to the + log while also logging to standard out (stdout). + <note> + Calling this function does not cause the task to fail. + </note> + </para></listitem> + <listitem><para> + <filename>bb.fatal(</filename><replaceable>msg</replaceable><filename>)</filename>: + This logging function is similar to + <filename>bb.error(</filename><replaceable>msg</replaceable><filename>)</filename> + but also causes the calling task to fail. + <note> + <filename>bb.fatal()</filename> raises an exception, + which means you do not need to put a "return" + statement after the function. + </note> + </para></listitem> + </itemizedlist> + </para> + + <para> + The same logging functions are also available in shell + functions, under the names + <filename>bbplain</filename>, <filename>bbnote</filename>, + <filename>bbdebug</filename>, <filename>bbwarn</filename>, + <filename>bberror</filename>, and <filename>bbfatal</filename>. + The + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-logging'><filename>logging</filename></ulink> + class implements these functions. + See that class in the + <filename>meta/classes</filename> folder of the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + for information. + </para> + + <section id='logging-with-python'> + <title>Logging With Python</title> + + <para> + When creating recipes using Python and inserting code that + handles build logs, keep in mind the goal is to have + informative logs while keeping the console as "silent" as + possible. + Also, if you want status messages in the log, use the + "debug" loglevel. + </para> + + <para> + Following is an example written in Python. + The code handles logging for a function that determines the + number of tasks needed to be run. + See the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-listtasks'><filename>do_listtasks</filename></ulink>" + section for additional information: + <literallayout class='monospaced'> + python do_listtasks() { + bb.debug(2, "Starting to figure out the task list") + if noteworthy_condition: + bb.note("There are 47 tasks to run") + bb.debug(2, "Got to point xyz") + if warning_trigger: + bb.warn("Detected warning_trigger, this might be a problem later.") + if recoverable_error: + bb.error("Hit recoverable_error, you really need to fix this!") + if fatal_error: + bb.fatal("fatal_error detected, unable to print the task list") + bb.plain("The tasks present are abc") + bb.debug(2, "Finished figuring out the tasklist") + } + </literallayout> + </para> + </section> + + <section id='logging-with-bash'> + <title>Logging With Bash</title> + + <para> + When creating recipes using Bash and inserting code that + handles build logs, you have the same goals - informative + with minimal console output. + The syntax you use for recipes written in Bash is similar + to that of recipes written in Python described in the + previous section. + </para> + + <para> + Following is an example written in Bash. + The code logs the progress of the <filename>do_my_function</filename> function. + <literallayout class='monospaced'> + do_my_function() { + bbdebug 2 "Running do_my_function" + if [ exceptional_condition ]; then + bbnote "Hit exceptional_condition" + fi + bbdebug 2 "Got to point xyz" + if [ warning_trigger ]; then + bbwarn "Detected warning_trigger, this might cause a problem later." + fi + if [ recoverable_error ]; then + bberror "Hit recoverable_error, correcting" + fi + if [ fatal_error ]; then + bbfatal "fatal_error detected" + fi + bbdebug 2 "Completed do_my_function" + } + </literallayout> + </para> + </section> + </section> + + <section id='debugging-parallel-make-races'> + <title>Debugging Parallel Make Races</title> + + <para> + A parallel <filename>make</filename> race occurs when the build + consists of several parts that are run simultaneously and + a situation occurs when the output or result of one + part is not ready for use with a different part of the build + that depends on that output. + Parallel make races are annoying and can sometimes be difficult + to reproduce and fix. + However, some simple tips and tricks exist that can help + you debug and fix them. + This section presents a real-world example of an error + encountered on the Yocto Project autobuilder and the process + used to fix it. + <note> + If you cannot properly fix a <filename>make</filename> race + condition, you can work around it by clearing either the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKEINST'><filename>PARALLEL_MAKEINST</filename></ulink> + variables. + </note> + </para> + + <section id='the-failure'> + <title>The Failure</title> + + <para> + For this example, assume that you are building an image that + depends on the "neard" package. + And, during the build, BitBake runs into problems and + creates the following output. + <note> + This example log file has longer lines artificially + broken to make the listing easier to read. + </note> + If you examine the output or the log file, you see the + failure during <filename>make</filename>: + <literallayout class='monospaced'> | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common'] | DEBUG: Executing shell function do_compile | NOTE: make -j 16 @@ -10194,61 +13170,62 @@ Some notes from Cal: | make[1]: *** Waiting for unfinished jobs.... | make: *** [all] Error 2 | ERROR: oe_runmake failed - </literallayout> - </para> - </section> + </literallayout> + </para> + </section> - <section id='reproducing-the-error'> - <title>Reproducing the Error</title> + <section id='reproducing-the-error'> + <title>Reproducing the Error</title> - <para> - Because race conditions are intermittent, they do not - manifest themselves every time you do the build. - In fact, most times the build will complete without problems - even though the potential race condition exists. - Thus, once the error surfaces, you need a way to reproduce it. - </para> + <para> + Because race conditions are intermittent, they do not + manifest themselves every time you do the build. + In fact, most times the build will complete without problems + even though the potential race condition exists. + Thus, once the error surfaces, you need a way to reproduce + it. + </para> - <para> - In this example, compiling the "neard" package is causing the - problem. - So the first thing to do is build "neard" locally. - Before you start the build, set the - <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></ulink> - variable in your <filename>local.conf</filename> file to - a high number (e.g. "-j 20"). - Using a high value for <filename>PARALLEL_MAKE</filename> - increases the chances of the race condition showing up: - <literallayout class='monospaced'> + <para> + In this example, compiling the "neard" package is causing + the problem. + So the first thing to do is build "neard" locally. + Before you start the build, set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></ulink> + variable in your <filename>local.conf</filename> file to + a high number (e.g. "-j 20"). + Using a high value for <filename>PARALLEL_MAKE</filename> + increases the chances of the race condition showing up: + <literallayout class='monospaced'> $ bitbake neard - </literallayout> - </para> + </literallayout> + </para> - <para> - Once the local build for "neard" completes, start a - <filename>devshell</filename> build: - <literallayout class='monospaced'> + <para> + Once the local build for "neard" completes, start a + <filename>devshell</filename> build: + <literallayout class='monospaced'> $ bitbake neard -c devshell - </literallayout> - For information on how to use a - <filename>devshell</filename>, see the - "<link linkend='platdev-appdev-devshell'>Using a Development Shell</link>" - section. - </para> + </literallayout> + For information on how to use a + <filename>devshell</filename>, see the + "<link linkend='platdev-appdev-devshell'>Using a Development Shell</link>" + section. + </para> - <para> - In the <filename>devshell</filename>, do the following: - <literallayout class='monospaced'> + <para> + In the <filename>devshell</filename>, do the following: + <literallayout class='monospaced'> $ make clean $ make tools/snep-send.o - </literallayout> - The <filename>devshell</filename> commands cause the failure - to clearly be visible. - In this case, a missing dependency exists for the "neard" - Makefile target. - Here is some abbreviated, sample output with the - missing dependency clearly visible at the end: - <literallayout class='monospaced'> + </literallayout> + The <filename>devshell</filename> commands cause the failure + to clearly be visible. + In this case, a missing dependency exists for the "neard" + Makefile target. + Here is some abbreviated, sample output with the + missing dependency clearly visible at the end: + <literallayout class='monospaced'> i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/...... . . @@ -10261,238 +13238,1687 @@ Some notes from Cal: compilation terminated. make: *** [tools/snep-send.o] Error 1 $ - </literallayout> - </para> - </section> + </literallayout> + </para> + </section> - <section id='creating-a-patch-for-the-fix'> - <title>Creating a Patch for the Fix</title> + <section id='creating-a-patch-for-the-fix'> + <title>Creating a Patch for the Fix</title> - <para> - Because there is a missing dependency for the Makefile - target, you need to patch the - <filename>Makefile.am</filename> file, which is generated - from <filename>Makefile.in</filename>. - You can use Quilt to create the patch: - <literallayout class='monospaced'> + <para> + Because there is a missing dependency for the Makefile + target, you need to patch the + <filename>Makefile.am</filename> file, which is generated + from <filename>Makefile.in</filename>. + You can use Quilt to create the patch: + <literallayout class='monospaced'> $ quilt new parallelmake.patch Patch patches/parallelmake.patch is now on top $ quilt add Makefile.am File Makefile.am added to patch patches/parallelmake.patch - </literallayout> - For more information on using Quilt, see the - "<link linkend='using-a-quilt-workflow'>Using Quilt in Your Workflow</link>" - section. - </para> + </literallayout> + For more information on using Quilt, see the + "<link linkend='using-a-quilt-workflow'>Using Quilt in Your Workflow</link>" + section. + </para> - <para> - At this point you need to make the edits to - <filename>Makefile.am</filename> to add the missing - dependency. - For our example, you have to add the following line - to the file: - <literallayout class='monospaced'> + <para> + At this point you need to make the edits to + <filename>Makefile.am</filename> to add the missing + dependency. + For our example, you have to add the following line + to the file: + <literallayout class='monospaced'> tools/snep-send.$(OBJEXT): include/near/dbus.h - </literallayout> - </para> + </literallayout> + </para> - <para> - Once you have edited the file, use the - <filename>refresh</filename> command to create the patch: - <literallayout class='monospaced'> + <para> + Once you have edited the file, use the + <filename>refresh</filename> command to create the patch: + <literallayout class='monospaced'> $ quilt refresh Refreshed patch patches/parallelmake.patch - </literallayout> - Once the patch file exists, you need to add it back to the - originating recipe folder. - Here is an example assuming a top-level - <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> - named <filename>poky</filename>: - <literallayout class='monospaced'> + </literallayout> + Once the patch file exists, you need to add it back to the + originating recipe folder. + Here is an example assuming a top-level + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + named <filename>poky</filename>: + <literallayout class='monospaced'> $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard - </literallayout> - The final thing you need to do to implement the fix in the - build is to update the "neard" recipe (i.e. - <filename>neard-0.14.bb</filename>) so that the - <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> - statement includes the patch file. - The recipe file is in the folder above the patch. - Here is what the edited <filename>SRC_URI</filename> - statement would look like: - <literallayout class='monospaced'> + </literallayout> + The final thing you need to do to implement the fix in the + build is to update the "neard" recipe (i.e. + <filename>neard-0.14.bb</filename>) so that the + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + statement includes the patch file. + The recipe file is in the folder above the patch. + Here is what the edited <filename>SRC_URI</filename> + statement would look like: + <literallayout class='monospaced'> SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \ file://neard.in \ file://neard.service.in \ file://parallelmake.patch \ " - </literallayout> + </literallayout> + </para> + + <para> + With the patch complete and moved to the correct folder and + the <filename>SRC_URI</filename> statement updated, you can + exit the <filename>devshell</filename>: + <literallayout class='monospaced'> + $ exit + </literallayout> + </para> + </section> + + <section id='testing-the-build'> + <title>Testing the Build</title> + + <para> + With everything in place, you can get back to trying the + build again locally: + <literallayout class='monospaced'> + $ bitbake neard + </literallayout> + This build should succeed. + </para> + + <para> + Now you can open up a <filename>devshell</filename> again + and repeat the clean and make operations as follows: + <literallayout class='monospaced'> + $ bitbake neard -c devshell + $ make clean + $ make tools/snep-send.o + </literallayout> + The build should work without issue. + </para> + + <para> + As with all solved problems, if they originated upstream, + you need to submit the fix for the recipe in OE-Core and + upstream so that the problem is taken care of at its + source. + See the + "<link linkend='how-to-submit-a-change'>Submitting a Change to the Yocto Project</link>" + section for more information. + </para> + </section> + </section> + + <section id="platdev-gdb-remotedebug"> + <title>Debugging With the GNU Project Debugger (GDB) Remotely</title> + + <para> + GDB allows you to examine running programs, which in turn helps + you to understand and fix problems. + It also allows you to perform post-mortem style analysis of + program crashes. + GDB is available as a package within the Yocto Project and is + installed in SDK images by default. + See the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>" + chapter in the Yocto Project Reference Manual for a description of + these images. + You can find information on GDB at + <ulink url="http://sourceware.org/gdb/"/>. + <note><title>Tip</title> + For best results, install debug (<filename>-dbg</filename>) + packages for the applications you are going to debug. + Doing so makes extra debug symbols available that give you + more meaningful output. + </note> </para> <para> - With the patch complete and moved to the correct folder and - the <filename>SRC_URI</filename> statement updated, you can - exit the <filename>devshell</filename>: - <literallayout class='monospaced'> - $ exit - </literallayout> + Sometimes, due to memory or disk space constraints, it is not + possible to use GDB directly on the remote target to debug + applications. + These constraints arise because GDB needs to load the debugging + information and the binaries of the process being debugged. + Additionally, GDB needs to perform many computations to locate + information such as function names, variable names and values, + stack traces and so forth - even before starting the debugging + process. + These extra computations place more load on the target system + and can alter the characteristics of the program being debugged. + </para> + + <para> + To help get past the previously mentioned constraints, you can + use gdbserver, which runs on the remote target and does not + load any debugging information from the debugged process. + Instead, a GDB instance processes the debugging information that + is run on a remote computer - the host GDB. + The host GDB then sends control commands to gdbserver to make + it stop or start the debugged program, as well as read or write + memory regions of that debugged program. + All the debugging information loaded and processed as well + as all the heavy debugging is done by the host GDB. + Offloading these processes gives the gdbserver running on the + target a chance to remain small and fast. + </para> + + <para> + Because the host GDB is responsible for loading the debugging + information and for doing the necessary processing to make + actual debugging happen, you have to make sure the host can + access the unstripped binaries complete with their debugging + information and also be sure the target is compiled with no + optimizations. + The host GDB must also have local access to all the libraries + used by the debugged program. + Because gdbserver does not need any local debugging information, + the binaries on the remote target can remain stripped. + However, the binaries must also be compiled without optimization + so they match the host's binaries. + </para> + + <para> + To remain consistent with GDB documentation and terminology, + the binary being debugged on the remote target machine is + referred to as the "inferior" binary. + For documentation on GDB see the + <ulink url="http://sourceware.org/gdb/documentation/">GDB site</ulink>. + </para> + + <para> + The following steps show you how to debug using the GNU project + debugger. + <orderedlist> + <listitem><para> + <emphasis>Configure your build system to construct the + companion debug filesystem:</emphasis></para> + + <para>In your <filename>local.conf</filename> file, set + the following: + <literallayout class='monospaced'> + IMAGE_GEN_DEBUGFS = "1" + IMAGE_FSTYPES_DEBUGFS = "tar.bz2" + </literallayout> + These options cause the OpenEmbedded build system + to generate a special companion filesystem fragment, + which contains the matching source and debug symbols to + your deployable filesystem. + The build system does this by looking at what is in the + deployed filesystem, and pulling the corresponding + <filename>-dbg</filename> packages.</para> + + <para>The companion debug filesystem is not a complete + filesystem, but only contains the debug fragments. + This filesystem must be combined with the full filesystem + for debugging. + Subsequent steps in this procedure show how to combine + the partial filesystem with the full filesystem. + </para></listitem> + <listitem><para> + <emphasis>Configure the system to include gdbserver in + the target filesystem:</emphasis></para> + + <para>Make the following addition in either your + <filename>local.conf</filename> file or in an image + recipe: + <literallayout class='monospaced'> + IMAGE_INSTALL_append = “ gdbserver" + </literallayout> + The change makes sure the <filename>gdbserver</filename> + package is included. + </para></listitem> + <listitem><para> + <emphasis>Build the environment:</emphasis></para> + + <para>Use the following command to construct the image + and the companion Debug Filesystem: + <literallayout class='monospaced'> + $ bitbake <replaceable>image</replaceable> + </literallayout> + Build the cross GDB component and make it available + for debugging. + Build the SDK that matches the image. + Building the SDK is best for a production build + that can be used later for debugging, especially + during long term maintenance: + <literallayout class='monospaced'> + $ bitbake -c populate_sdk <replaceable>image</replaceable> + </literallayout></para> + + <para>Alternatively, you can build the minimal + toolchain components that match the target. + Doing so creates a smaller than typical SDK and only + contains a minimal set of components with which to + build simple test applications, as well as run the + debugger: + <literallayout class='monospaced'> + $ bitbake meta-toolchain + </literallayout></para> + + <para>A final method is to build Gdb itself within + the build system: + <literallayout class='monospaced'> + $ bitbake gdb-cross-<replaceable>architecture</replaceable> + </literallayout> + Doing so produces a temporary copy of + <filename>cross-gdb</filename> you can use for + debugging during development. + While this is the quickest approach, the two previous + methods in this step are better when considering + long-term maintenance strategies. + <note> + If you run + <filename>bitbake gdb-cross</filename>, the + OpenEmbedded build system suggests the actual + image (e.g. <filename>gdb-cross-i586</filename>). + The suggestion is usually the actual name you want + to use. + </note> + </para></listitem> + <listitem><para> + <emphasis>Set up the</emphasis> <filename>debugfs</filename></para> + + <para>Run the following commands to set up the + <filename>debugfs</filename>: + <literallayout class='monospaced'> + $ mkdir debugfs + $ cd debugfs + $ tar xvfj <replaceable>build-dir</replaceable>/tmp-glibc/deploy/images/<replaceable>machine</replaceable>/<replaceable>image</replaceable>.rootfs.tar.bz2 + $ tar xvfj <replaceable>build-dir</replaceable>/tmp-glibc/deploy/images/<replaceable>machine</replaceable>/<replaceable>image</replaceable>-dbg.rootfs.tar.bz2 + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Set up GDB</emphasis></para> + + <para>Install the SDK (if you built one) and then + source the correct environment file. + Sourcing the environment file puts the SDK in your + <filename>PATH</filename> environment variable.</para> + + <para>If you are using the build system, Gdb is + located in + <replaceable>build-dir</replaceable>/tmp/sysroots/<replaceable>host</replaceable>/usr/bin/<replaceable>architecture</replaceable>/<replaceable>architecture</replaceable>-gdb + </para></listitem> + <listitem><para> + <emphasis>Boot the target:</emphasis></para> + + <para>For information on how to run QEMU, see the + <ulink url='http://wiki.qemu.org/Documentation/GettingStartedDevelopers'>QEMU Documentation</ulink>. + <note> + Be sure to verify that your host can access the + target via TCP. + </note> + </para></listitem> + <listitem><para> + <emphasis>Debug a program:</emphasis></para> + + <para>Debugging a program involves running gdbserver + on the target and then running Gdb on the host. + The example in this step debugs + <filename>gzip</filename>: + <literallayout class='monospaced'> + root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help + </literallayout> + For additional gdbserver options, see the + <ulink url='https://www.gnu.org/software/gdb/documentation/'>GDB Server Documentation</ulink>. + </para> + + <para>After running gdbserver on the target, you need + to run Gdb on the host and configure it and connect to + the target. + Use these commands: + <literallayout class='monospaced'> + $ cd <replaceable>directory-holding-the-debugfs-directory</replaceable> + $ <replaceable>arch</replaceable>-gdb + + (gdb) set sysroot debugfs + (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug + (gdb) target remote <replaceable>IP-of-target</replaceable>:1234 + </literallayout> + At this point, everything should automatically load + (i.e. matching binaries, symbols and headers). + <note> + The Gdb <filename>set</filename> commands in the + previous example can be placed into the users + <filename>~/.gdbinit</filename> file. + Upon starting, Gdb automatically runs whatever + commands are in that file. + </note> + </para></listitem> + <listitem><para> + <emphasis>Deploying without a full image + rebuild:</emphasis></para> + + <para>In many cases, during development you want a + quick method to deploy a new binary to the target and + debug it, without waiting for a full image build. + </para> + + <para>One approach to solving this situation is to + just build the component you want to debug. + Once you have built the component, copy the + executable directly to both the target and the + host <filename>debugfs</filename>.</para> + + <para>If the binary is processed through the debug + splitting in OpenEmbedded, you should also + copy the debug items (i.e. <filename>.debug</filename> + contents and corresponding + <filename>/usr/src/debug</filename> files) + from the work directory. + Here is an example: + <literallayout class='monospaced'> + $ bitbake bash + $ bitbake -c devshell bash + $ cd .. + $ scp packages-split/bash/bin/bash <replaceable>target</replaceable>:/bin/bash + $ cp -a packages-split/bash-dbg/* <replaceable>path</replaceable>/debugfs + </literallayout> + </para></listitem> + </orderedlist> </para> </section> - <section id='testing-the-build'> - <title>Testing the Build</title> + <section id='debugging-with-the-gnu-project-debugger-gdb-on-the-target'> + <title>Debugging with the GNU Project Debugger (GDB) on the Target</title> <para> - With everything in place, you can get back to trying the - build again locally: - <literallayout class='monospaced'> - $ bitbake neard - </literallayout> - This build should succeed. + The previous section addressed using GDB remotely for debugging + purposes, which is the most usual case due to the inherent + hardware limitations on many embedded devices. + However, debugging in the target hardware itself is also + possible with more powerful devices. + This section describes what you need to do in order to support + using GDB to debug on the target hardware. </para> <para> - Now you can open up a <filename>devshell</filename> again - and repeat the clean and make operations as follows: - <literallayout class='monospaced'> - $ bitbake neard -c devshell - $ make clean - $ make tools/snep-send.o - </literallayout> - The build should work without issue. + To support this kind of debugging, you need do the following: + <itemizedlist> + <listitem><para> + Ensure that GDB is on the target. + You can do this by adding "gdb" to + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink>: + <literallayout class='monospaced'> + IMAGE_INSTALL_append = " gdb" + </literallayout> + Alternatively, you can add "tools-debug" to + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink>: + <literallayout class='monospaced'> + IMAGE_FEATURES_append = " tools-debug" + </literallayout> + </para></listitem> + <listitem><para> + Ensure that debug symbols are present. + You can make sure these symbols are present by + installing <filename>-dbg</filename>: + <literallayout class='monospaced'> + IMAGE_INSTALL_append = " <replaceable>packagename</replaceable>-dbg" + </literallayout> + Alternatively, you can do the following to include all + the debug symbols: + <literallayout class='monospaced'> + IMAGE_FEATURES_append = " dbg-pkgs" + </literallayout> + </para></listitem> + </itemizedlist> + <note> + To improve the debug information accuracy, you can reduce + the level of optimization used by the compiler. + For example, when adding the following line to your + <filename>local.conf</filename> file, you will reduce + optimization from + <ulink url='&YOCTO_DOCS_REF_URL;#var-FULL_OPTIMIZATION'><filename>FULL_OPTIMIZATION</filename></ulink> + of "-O2" to + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEBUG_OPTIMIZATION'><filename>DEBUG_OPTIMIZATION</filename></ulink> + of "-O -fno-omit-frame-pointer": + <literallayout class='monospaced'> + DEBUG_BUILD = "1" + </literallayout> + Consider that this will reduce the application's performance + and is recommended only for debugging purposes. + </note> </para> + </section> + + <section id='dev-other-debugging-others'> + <title>Other Debugging Tips</title> <para> - As with all solved problems, if they originated upstream, you - need to submit the fix for the recipe in OE-Core and upstream - so that the problem is taken care of at its source. - See the - "<link linkend='how-to-submit-a-change'>Submitting a Change to the Yocto Project</link>" - section for more information. + Here are some other tips that you might find useful: + <itemizedlist> + <listitem><para> + When adding new packages, it is worth watching for + undesirable items making their way into compiler command + lines. + For example, you do not want references to local system + files like + <filename>/usr/lib/</filename> or + <filename>/usr/include/</filename>. + </para></listitem> + <listitem><para> + If you want to remove the <filename>psplash</filename> + boot splashscreen, + add <filename>psplash=false</filename> to the kernel + command line. + Doing so prevents <filename>psplash</filename> from + loading and thus allows you to see the console. + It is also possible to switch out of the splashscreen by + switching the virtual console (e.g. Fn+Left or Fn+Right + on a Zaurus). + </para></listitem> + <listitem><para> + Removing + <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> + (usually <filename>tmp/</filename>, within the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>) + can often fix temporary build issues. + Removing <filename>TMPDIR</filename> is usually a + relatively cheap operation, because task output will be + cached in + <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink> + (usually <filename>sstate-cache/</filename>, which is + also in the Build Directory). + <note> + Removing <filename>TMPDIR</filename> might be a + workaround rather than a fix. + Consequently, trying to determine the underlying + cause of an issue before removing the directory is + a good idea. + </note> + </para></listitem> + <listitem><para> + Understanding how a feature is used in practice within + existing recipes can be very helpful. + It is recommended that you configure some method that + allows you to quickly search through files.</para> + + <para>Using GNU Grep, you can use the following shell + function to recursively search through common + recipe-related files, skipping binary files, + <filename>.git</filename> directories, and the + Build Directory (assuming its name starts with + "build"): + <literallayout class='monospaced'> + g() { + grep -Ir \ + --exclude-dir=.git \ + --exclude-dir='build*' \ + --include='*.bb*' \ + --include='*.inc*' \ + --include='*.conf*' \ + --include='*.py*' \ + "$@" + } + </literallayout> + Following are some usage examples: + <literallayout class='monospaced'> + $ g FOO # Search recursively for "FOO" + $ g -i foo # Search recursively for "foo", ignoring case + $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR" + </literallayout> + If figuring out how some feature works requires a lot of + searching, it might indicate that the documentation + should be extended or improved. + In such cases, consider filing a documentation bug using + the Yocto Project implementation of + <ulink url='https://bugzilla.yoctoproject.org/'>Bugzilla</ulink>. + For information on how to submit a bug against + the Yocto Project, see the Yocto Project Bugzilla + <ulink url='&YOCTO_WIKI_URL;/wiki/Bugzilla_Configuration_and_Bug_Tracking'>wiki page</ulink> + and the + "<link linkend='submitting-a-defect-against-the-yocto-project'>Submitting a Defect Against the Yocto Project</link>" + section. + <note> + The manuals might not be the right place to document + variables that are purely internal and have a + limited scope (e.g. internal variables used to + implement a single <filename>.bbclass</filename> + file). + </note> + </para></listitem> + </itemizedlist> </para> </section> </section> - <section id='maintaining-open-source-license-compliance-during-your-products-lifecycle'> - <title>Maintaining Open Source License Compliance During Your Product's Lifecycle</title> + <section id='making-changes-to-the-yocto-project'> + <title>Making Changes to the Yocto Project</title> <para> - One of the concerns for a development organization using open source - software is how to maintain compliance with various open source - licensing during the lifecycle of the product. - While this section does not provide legal advice or - comprehensively cover all scenarios, it does - present methods that you can use to - assist you in meeting the compliance requirements during a software - release. + Because the Yocto Project is an open-source, community-based + project, you can effect changes to the project. + This section presents procedures that show you how to submit + a defect against the project and how to submit a change. </para> - <para> - With hundreds of different open source licenses that the Yocto - Project tracks, it is difficult to know the requirements of each - and every license. - However, the requirements of the major FLOSS licenses can begin - to be covered by - assuming that three main areas of concern exist: - <itemizedlist> - <listitem><para>Source code must be provided.</para></listitem> - <listitem><para>License text for the software must be - provided.</para></listitem> - <listitem><para>Compilation scripts and modifications to the - source code must be provided. - </para></listitem> - </itemizedlist> - There are other requirements beyond the scope of these - three and the methods described in this section - (e.g. the mechanism through which source code is distributed). - </para> + <section id='submitting-a-defect-against-the-yocto-project'> + <title>Submitting a Defect Against the Yocto Project</title> - <para> - As different organizations have different methods of complying with - open source licensing, this section is not meant to imply that - there is only one single way to meet your compliance obligations, - but rather to describe one method of achieving compliance. - The remainder of this section describes methods supported to meet the - previously mentioned three requirements. - Once you take steps to meet these requirements, - and prior to releasing images, sources, and the build system, - you should audit all artifacts to ensure completeness. - <note> - The Yocto Project generates a license manifest during - image creation that is located - in <filename>${DEPLOY_DIR}/licenses/<replaceable>image_name-datestamp</replaceable></filename> - to assist with any audits. - </note> - </para> + <para> + Use the Yocto Project implementation of + <ulink url='http://www.bugzilla.org/about/'>Bugzilla</ulink> + to submit a defect (bug) against the Yocto Project. + For additional information on this implementation of Bugzilla see the + "<ulink url='&YOCTO_DOCS_REF_URL;#resources-bugtracker'>Yocto Project Bugzilla</ulink>" + section in the Yocto Project Reference Manual. + For more detail on any of the following steps, see the Yocto Project + <ulink url='&YOCTO_WIKI_URL;/wiki/Bugzilla_Configuration_and_Bug_Tracking'>Bugzilla wiki page</ulink>. + </para> + + <para> + Use the following general steps to submit a bug" + + <orderedlist> + <listitem><para> + Open the Yocto Project implementation of + <ulink url='&YOCTO_BUGZILLA_URL;'>Bugzilla</ulink>. + </para></listitem> + <listitem><para> + Click "File a Bug" to enter a new bug. + </para></listitem> + <listitem><para> + Choose the appropriate "Classification", "Product", and + "Component" for which the bug was found. + Bugs for the Yocto Project fall into one of several + classifications, which in turn break down into several + products and components. + For example, for a bug against the + <filename>meta-intel</filename> layer, you would choose + "Build System, Metadata & Runtime", "BSPs", and + "bsps-meta-intel", respectively. + </para></listitem> + <listitem><para> + Choose the "Version" of the Yocto Project for which you found + the bug (e.g. &DISTRO;). + </para></listitem> + <listitem><para> + Determine and select the "Severity" of the bug. + The severity indicates how the bug impacted your work. + </para></listitem> + <listitem><para> + Choose the "Hardware" that the bug impacts. + </para></listitem> + <listitem><para> + Choose the "Architecture" that the bug impacts. + </para></listitem> + <listitem><para> + Choose a "Documentation change" item for the bug. + Fixing a bug might or might not affect the Yocto Project + documentation. + If you are unsure of the impact to the documentation, select + "Don't Know". + </para></listitem> + <listitem><para> + Provide a brief "Summary" of the bug. + Try to limit your summary to just a line or two and be sure + to capture the essence of the bug. + </para></listitem> + <listitem><para> + Provide a detailed "Description" of the bug. + You should provide as much detail as you can about the context, + behavior, output, and so forth that surrounds the bug. + You can even attach supporting files for output from logs by + using the "Add an attachment" button. + </para></listitem> + <listitem><para> + Click the "Submit Bug" button submit the bug. + A new Bugzilla number is assigned to the bug and the defect + is logged in the bug tracking system. + </para></listitem> + </orderedlist> + Once you file a bug, the bug is processed by the Yocto Project Bug + Triage Team and further details concerning the bug are assigned + (e.g. priority and owner). + You are the "Submitter" of the bug and any further categorization, + progress, or comments on the bug result in Bugzilla sending you an + automated email concerning the particular change or progress to the + bug. + </para> + </section> - <section id='providing-the-source-code'> - <title>Providing the Source Code</title> + <section id='how-to-submit-a-change'> + <title>Submitting a Change to the Yocto Project</title> <para> - Compliance activities should begin before you generate the - final image. - The first thing you should look at is the requirement that - tops the list for most compliance groups - providing - the source. - The Yocto Project has a few ways of meeting this - requirement. + Contributions to the Yocto Project and OpenEmbedded are very welcome. + Because the system is extremely configurable and flexible, we recognize + that developers will want to extend, configure or optimize it for + their specific uses. </para> <para> - One of the easiest ways to meet this requirement is - to provide the entire - <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> - used by the build. - This method, however, has a few issues. - The most obvious is the size of the directory since it includes - all sources used in the build and not just the source used in - the released image. - It will include toolchain source, and other artifacts, which - you would not generally release. - However, the more serious issue for most companies is accidental - release of proprietary software. - The Yocto Project provides an - <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-archiver'><filename>archiver</filename></ulink> - class to help avoid some of these concerns. + The Yocto Project uses a mailing list and a patch-based workflow + that is similar to the Linux kernel but contains important + differences. + In general, a mailing list exists through which you can submit + patches. + You should send patches to the appropriate mailing list so that they + can be reviewed and merged by the appropriate maintainer. + The specific mailing list you need to use depends on the + location of the code you are changing. + Each component (e.g. layer) should have a + <filename>README</filename> file that indicates where to send + the changes and which process to follow. + </para> + + <para> + You can send the patch to the mailing list using whichever approach + you feel comfortable with to generate the patch. + Once sent, the patch is usually reviewed by the community at large. + If somebody has concerns with the patch, they will usually voice + their concern over the mailing list. + If a patch does not receive any negative reviews, the maintainer of + the affected layer typically takes the patch, tests it, and then + based on successful testing, merges the patch. + </para> + + <para id='figuring-out-the-mailing-list-to-use'> + The "poky" repository, which is the Yocto Project's reference build + environment, is a hybrid repository that contains several + individual pieces (e.g. BitBake, Metadata, documentation, + and so forth) built using the combo-layer tool. + The upstream location used for submitting changes varies by + component: + <itemizedlist> + <listitem><para> + <emphasis>Core Metadata:</emphasis> + Send your patch to the + <ulink url='http://lists.openembedded.org/mailman/listinfo/openembedded-core'>openembedded-core</ulink> + mailing list. For example, a change to anything under + the <filename>meta</filename> or + <filename>scripts</filename> directories should be sent + to this mailing list. + </para></listitem> + <listitem><para> + <emphasis>BitBake:</emphasis> + For changes to BitBake (i.e. anything under the + <filename>bitbake</filename> directory), send your patch + to the + <ulink url='http://lists.openembedded.org/mailman/listinfo/bitbake-devel'>bitbake-devel</ulink> + mailing list. + </para></listitem> + <listitem><para> + <emphasis>"meta-*" trees:</emphasis> + These trees contain Metadata. + Use the + <ulink url='https://lists.yoctoproject.org/listinfo/poky'>poky</ulink> + mailing list. + </para></listitem> + </itemizedlist> </para> <para> - Before you employ <filename>DL_DIR</filename> or the - <filename>archiver</filename> class, you need to decide how - you choose to provide source. - The source <filename>archiver</filename> class can generate - tarballs and SRPMs and can create them with various levels of - compliance in mind. + For changes to other layers hosted in the Yocto Project source + repositories (i.e. <filename>yoctoproject.org</filename>), tools, + and the Yocto Project documentation, use the + <ulink url='https://lists.yoctoproject.org/listinfo/yocto'>Yocto Project</ulink> + general mailing list. + <note> + Sometimes a layer's documentation specifies to use a + particular mailing list. + If so, use that list. + </note> + For additional recipes that do not fit into the core Metadata, you + should determine which layer the recipe should go into and submit + the change in the manner recommended by the documentation (e.g. + the <filename>README</filename> file) supplied with the layer. + If in doubt, please ask on the Yocto general mailing list or on + the openembedded-devel mailing list. </para> <para> - One way of doing this (but certainly not the only way) is to - release just the source as a tarball. - You can do this by adding the following to the - <filename>local.conf</filename> file found in the - <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>: + You can also push a change upstream and request a maintainer to + pull the change into the component's upstream repository. + You do this by pushing to a contribution repository that is upstream. + See the + "<ulink url='&YOCTO_DOCS_OM_URL;#gs-git-workflows-and-the-yocto-project'>Git Workflows and the Yocto Project</ulink>" + section in the Yocto Project Overview and Concepts Manual for additional + concepts on working in the Yocto Project development environment. + </para> + + <para> + Two commonly used testing repositories exist for + OpenEmbedded-Core: + <itemizedlist> + <listitem><para> + <emphasis>"ross/mut" branch:</emphasis> + The "mut" (master-under-test) tree + exists in the <filename>poky-contrib</filename> repository + in the + <ulink url='&YOCTO_GIT_URL;'>Yocto Project source repositories</ulink>. + </para></listitem> + <listitem><para> + <emphasis>"master-next" branch:</emphasis> + This branch is part of the main + "poky" repository in the Yocto Project source repositories. + </para></listitem> + </itemizedlist> + Maintainers use these branches to test submissions prior to merging + patches. + Thus, you can get an idea of the status of a patch based on + whether the patch has been merged into one of these branches. + <note> + This system is imperfect and changes can sometimes get lost in the + flow. + Asking about the status of a patch or change is reasonable if the + change has been idle for a while with no feedback. + The Yocto Project does have plans to use + <ulink url='https://en.wikipedia.org/wiki/Patchwork_(software)'>Patchwork</ulink> + to track the status of patches and also to automatically preview + patches. + </note> + </para> + + <para> + The following sections provide procedures for submitting a change. + </para> + + <section id='pushing-a-change-upstream'> + <title>Using Scripts to Push a Change Upstream and Request a Pull</title> + + <para> + Follow this procedure to push a change to an upstream "contrib" + Git repository: + <note> + You can find general Git information on how to push a change + upstream in the + <ulink url='http://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows'>Git Community Book</ulink>. + </note> + <orderedlist> + <listitem><para> + <emphasis>Make Your Changes Locally:</emphasis> + Make your changes in your local Git repository. + You should make small, controlled, isolated changes. + Keeping changes small and isolated aids review, + makes merging/rebasing easier and keeps the change + history clean should anyone need to refer to it in + future. + </para></listitem> + <listitem><para> + <emphasis>Stage Your Changes:</emphasis> + Stage your changes by using the <filename>git add</filename> + command on each file you changed. + </para></listitem> + <listitem><para id='making-sure-you-have-correct-commit-information'> + <emphasis>Commit Your Changes:</emphasis> + Commit the change by using the + <filename>git commit</filename> command. + Make sure your commit information follows standards by + following these accepted conventions: + <itemizedlist> + <listitem><para> + Be sure to include a "Signed-off-by:" line in the + same style as required by the Linux kernel. + Adding this line signifies that you, the submitter, + have agreed to the Developer's Certificate of + Origin 1.1 as follows: + <literallayout class='monospaced'> + Developer's Certificate of Origin 1.1 + + By making a contribution to this project, I certify that: + + (a) The contribution was created in whole or in part by me and I + have the right to submit it under the open source license + indicated in the file; or + + (b) The contribution is based upon previous work that, to the best + of my knowledge, is covered under an appropriate open source + license and I have the right under that license to submit that + work with modifications, whether created in whole or in part + by me, under the same open source license (unless I am + permitted to submit under a different license), as indicated + in the file; or + + (c) The contribution was provided directly to me by some other + person who certified (a), (b) or (c) and I have not modified + it. + + (d) I understand and agree that this project and the contribution + are public and that a record of the contribution (including all + personal information I submit with it, including my sign-off) is + maintained indefinitely and may be redistributed consistent with + this project or the open source license(s) involved. + </literallayout> + </para></listitem> + <listitem><para> + Provide a single-line summary of the change. + and, + if more explanation is needed, provide more + detail in the body of the commit. + This summary is typically viewable in the + "shortlist" of changes. + Thus, providing something short and descriptive + that gives the reader a summary of the change is + useful when viewing a list of many commits. + You should prefix this short description with the + recipe name (if changing a recipe), or else with + the short form path to the file being changed. + </para></listitem> + <listitem><para> + For the body of the commit message, provide + detailed information that describes what you + changed, why you made the change, and the approach + you used. + It might also be helpful if you mention how you + tested the change. + Provide as much detail as you can in the body of + the commit message. + <note> + You do not need to provide a more detailed + explanation of a change if the change is + minor to the point of the single line + summary providing all the information. + </note> + </para></listitem> + <listitem><para> + If the change addresses a specific bug or issue + that is associated with a bug-tracking ID, + include a reference to that ID in your detailed + description. + For example, the Yocto Project uses a specific + convention for bug references - any commit that + addresses a specific bug should use the following + form for the detailed description. + Be sure to use the actual bug-tracking ID from + Bugzilla for + <replaceable>bug-id</replaceable>: + <literallayout class='monospaced'> + Fixes [YOCTO #<replaceable>bug-id</replaceable>] + + <replaceable>detailed description of change</replaceable> + </literallayout> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <emphasis>Push Your Commits to a "Contrib" Upstream:</emphasis> + If you have arranged for permissions to push to an + upstream contrib repository, push the change to that + repository: + <literallayout class='monospaced'> + $ git push <replaceable>upstream_remote_repo</replaceable> <replaceable>local_branch_name</replaceable> + </literallayout> + For example, suppose you have permissions to push into the + upstream <filename>meta-intel-contrib</filename> + repository and you are working in a local branch named + <replaceable>your_name</replaceable><filename>/README</filename>. + The following command pushes your local commits to the + <filename>meta-intel-contrib</filename> upstream + repository and puts the commit in a branch named + <replaceable>your_name</replaceable><filename>/README</filename>: + <literallayout class='monospaced'> + $ git push meta-intel-contrib <replaceable>your_name</replaceable>/README + </literallayout> + </para></listitem> + <listitem><para id='push-determine-who-to-notify'> + <emphasis>Determine Who to Notify:</emphasis> + Determine the maintainer or the mailing list + that you need to notify for the change.</para> + + <para>Before submitting any change, you need to be sure + who the maintainer is or what mailing list that you need + to notify. + Use either these methods to find out: + <itemizedlist> + <listitem><para> + <emphasis>Maintenance File:</emphasis> + Examine the <filename>maintainers.inc</filename> + file, which is located in the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + at + <filename>meta/conf/distro/include</filename>, + to see who is responsible for code. + </para></listitem> + <listitem><para> + <emphasis>Search by File:</emphasis> + Using <ulink url='&YOCTO_DOCS_OM_URL;#git'>Git</ulink>, + you can enter the following command to bring up a + short list of all commits against a specific file: + <literallayout class='monospaced'> + git shortlog -- <replaceable>filename</replaceable> + </literallayout> + Just provide the name of the file for which you + are interested. + The information returned is not ordered by history + but does include a list of everyone who has + committed grouped by name. + From the list, you can see who is responsible for + the bulk of the changes against the file. + </para></listitem> + <listitem><para> + <emphasis>Examine the List of Mailing Lists:</emphasis> + For a list of the Yocto Project and related mailing + lists, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#resources-mailinglist'>Mailing lists</ulink>" + section in the Yocto Project Reference Manual. + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <emphasis>Make a Pull Request:</emphasis> + Notify the maintainer or the mailing list that you have + pushed a change by making a pull request.</para> + + <para>The Yocto Project provides two scripts that + conveniently let you generate and send pull requests to the + Yocto Project. + These scripts are <filename>create-pull-request</filename> + and <filename>send-pull-request</filename>. + You can find these scripts in the + <filename>scripts</filename> directory within the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + (e.g. <filename>~/poky/scripts</filename>). + </para> + + <para>Using these scripts correctly formats the requests + without introducing any whitespace or HTML formatting. + The maintainer that receives your patches either directly + or through the mailing list needs to be able to save and + apply them directly from your emails. + Using these scripts is the preferred method for sending + patches.</para> + + <para>First, create the pull request. + For example, the following command runs the script, + specifies the upstream repository in the contrib directory + into which you pushed the change, and provides a subject + line in the created patch files: + <literallayout class='monospaced'> + $ ~/poky/scripts/create-pull-request -u meta-intel-contrib -s "Updated Manual Section Reference in README" + </literallayout> + Running this script forms + <filename>*.patch</filename> files in a folder named + <filename>pull-</filename><replaceable>PID</replaceable> + in the current directory. + One of the patch files is a cover letter.</para> + + <para>Before running the + <filename>send-pull-request</filename> script, you must + edit the cover letter patch to insert information about + your change. + After editing the cover letter, send the pull request. + For example, the following command runs the script and + specifies the patch directory and email address. + In this example, the email address is a mailing list: + <literallayout class='monospaced'> + $ ~/poky/scripts/send-pull-request -p ~/meta-intel/pull-10565 -t meta-intel@yoctoproject.org + </literallayout> + You need to follow the prompts as the script is + interactive. + <note> + For help on using these scripts, simply provide the + <filename>-h</filename> argument as follows: + <literallayout class='monospaced'> + $ poky/scripts/create-pull-request -h + $ poky/scripts/send-pull-request -h + </literallayout> + </note> + </para></listitem> + </orderedlist> + </para> + </section> + + <section id='submitting-a-patch'> + <title>Using Email to Submit a Patch</title> + + <para> + You can submit patches without using the + <filename>create-pull-request</filename> and + <filename>send-pull-request</filename> scripts described in the + previous section. + However, keep in mind, the preferred method is to use the scripts. + </para> + + <para> + Depending on the components changed, you need to submit the email + to a specific mailing list. + For some guidance on which mailing list to use, see the + <link linkend='figuring-out-the-mailing-list-to-use'>list</link> + at the beginning of this section. + For a description of all the available mailing lists, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#resources-mailinglist'>Mailing Lists</ulink>" + section in the Yocto Project Reference Manual. + </para> + + <para> + Here is the general procedure on how to submit a patch through + email without using the scripts: + <orderedlist> + <listitem><para> + <emphasis>Make Your Changes Locally:</emphasis> + Make your changes in your local Git repository. + You should make small, controlled, isolated changes. + Keeping changes small and isolated aids review, + makes merging/rebasing easier and keeps the change + history clean should anyone need to refer to it in + future. + </para></listitem> + <listitem><para> + <emphasis>Stage Your Changes:</emphasis> + Stage your changes by using the <filename>git add</filename> + command on each file you changed. + </para></listitem> + <listitem><para> + <emphasis>Commit Your Changes:</emphasis> + Commit the change by using the + <filename>git commit --signoff</filename> command. + Using the <filename>--signoff</filename> option identifies + you as the person making the change and also satisfies + the Developer's Certificate of Origin (DCO) shown earlier. + </para> + + <para>When you form a commit, you must follow certain + standards established by the Yocto Project development + team. + See + <link linkend='making-sure-you-have-correct-commit-information'>Step 3</link> + in the previous section for information on how to + provide commit information that meets Yocto Project + commit message standards. + </para></listitem> + <listitem><para> + <emphasis>Format the Commit:</emphasis> + Format the commit into an email message. + To format commits, use the + <filename>git format-patch</filename> command. + When you provide the command, you must include a revision + list or a number of patches as part of the command. + For example, either of these two commands takes your most + recent single commit and formats it as an email message in + the current directory: + <literallayout class='monospaced'> + $ git format-patch -1 + </literallayout> + or + <literallayout class='monospaced'> + $ git format-patch HEAD~ + </literallayout></para> + + <para>After the command is run, the current directory + contains a numbered <filename>.patch</filename> file for + the commit.</para> + + <para>If you provide several commits as part of the + command, the <filename>git format-patch</filename> command + produces a series of numbered files in the current + directory – one for each commit. + If you have more than one patch, you should also use the + <filename>--cover</filename> option with the command, + which generates a cover letter as the first "patch" in + the series. + You can then edit the cover letter to provide a + description for the series of patches. + For information on the + <filename>git format-patch</filename> command, + see <filename>GIT_FORMAT_PATCH(1)</filename> displayed + using the <filename>man git-format-patch</filename> + command. + <note> + If you are or will be a frequent contributor to the + Yocto Project or to OpenEmbedded, you might consider + requesting a contrib area and the necessary associated + rights. + </note> + </para></listitem> + <listitem><para> + <emphasis>Import the Files Into Your Mail Client:</emphasis> + Import the files into your mail client by using the + <filename>git send-email</filename> command. + <note> + In order to use <filename>git send-email</filename>, + you must have the proper Git packages installed on + your host. + For Ubuntu, Debian, and Fedora the package is + <filename>git-email</filename>. + </note></para> + + <para>The <filename>git send-email</filename> command + sends email by using a local or remote Mail Transport Agent + (MTA) such as <filename>msmtp</filename>, + <filename>sendmail</filename>, or through a direct + <filename>smtp</filename> configuration in your Git + <filename>~/.gitconfig</filename> file. + If you are submitting patches through email only, it is + very important that you submit them without any whitespace + or HTML formatting that either you or your mailer + introduces. + The maintainer that receives your patches needs to be able + to save and apply them directly from your emails. + A good way to verify that what you are sending will be + applicable by the maintainer is to do a dry run and send + them to yourself and then save and apply them as the + maintainer would.</para> + + <para>The <filename>git send-email</filename> command is + the preferred method for sending your patches using + email since there is no risk of compromising whitespace + in the body of the message, which can occur when you use + your own mail client. + The command also has several options that let you + specify recipients and perform further editing of the + email message. + For information on how to use the + <filename>git send-email</filename> command, + see <filename>GIT-SEND-EMAIL(1)</filename> displayed using + the <filename>man git-send-email</filename> command. + </para></listitem> + </orderedlist> + </para> + </section> + </section> + </section> + + <section id='working-with-licenses'> + <title>Working With Licenses</title> + + <para> + As mentioned in the + "<ulink url='&YOCTO_DOCS_OM_URL;#licensing'>Licensing</ulink>" + section in the Yocto Project Overview and Concepts Manual, + open source projects are open to the public and they + consequently have different licensing structures in place. + This section describes the mechanism by which the + <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink> + tracks changes to licensing text and covers how to maintain open + source license compliance during your project's lifecycle. + The section also describes how to enable commercially licensed + recipes, which by default are disabled. + </para> + + <section id="usingpoky-configuring-LIC_FILES_CHKSUM"> + <title>Tracking License Changes</title> + + <para> + The license of an upstream project might change in the future. + In order to prevent these changes going unnoticed, the + <ulink url='&YOCTO_DOCS_REF_URL;#var-LIC_FILES_CHKSUM'><filename>LIC_FILES_CHKSUM</filename></ulink> + variable tracks changes to the license text. The checksums are + validated at the end of the configure step, and if the + checksums do not match, the build will fail. + </para> + + <section id="usingpoky-specifying-LIC_FILES_CHKSUM"> + <title>Specifying the <filename>LIC_FILES_CHKSUM</filename> Variable</title> + + <para> + The <filename>LIC_FILES_CHKSUM</filename> + variable contains checksums of the license text in the + source code for the recipe. + Following is an example of how to specify + <filename>LIC_FILES_CHKSUM</filename>: + <literallayout class='monospaced'> + LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \ + file://licfile1.txt;beginline=5;endline=29;md5=yyyy \ + file://licfile2.txt;endline=50;md5=zzzz \ + ..." + </literallayout> + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + When using "beginline" and "endline", realize + that line numbering begins with one and not + zero. + Also, the included lines are inclusive (i.e. + lines five through and including 29 in the + previous example for + <filename>licfile1.txt</filename>). + </para></listitem> + <listitem><para> + When a license check fails, the selected license + text is included as part of the QA message. + Using this output, you can determine the exact + start and finish for the needed license text. + </para></listitem> + </itemizedlist> + </note> + </para> + + <para> + The build system uses the + <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink> + variable as the default directory when searching files + listed in <filename>LIC_FILES_CHKSUM</filename>. + The previous example employs the default directory. + </para> + + <para> + Consider this next example: + <literallayout class='monospaced'> + LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\ + md5=bb14ed3c4cda583abc85401304b5cd4e" + LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6" + </literallayout> + </para> + + <para> + The first line locates a file in + <filename>${S}/src/ls.c</filename> and isolates lines five + through 16 as license text. + The second line refers to a file in + <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>. + </para> + + <para> + Note that <filename>LIC_FILES_CHKSUM</filename> variable is + mandatory for all recipes, unless the + <filename>LICENSE</filename> variable is set to "CLOSED". + </para> + </section> + + <section id="usingpoky-LIC_FILES_CHKSUM-explanation-of-syntax"> + <title>Explanation of Syntax</title> + + <para> + As mentioned in the previous section, the + <filename>LIC_FILES_CHKSUM</filename> variable lists all + the important files that contain the license text for the + source code. + It is possible to specify a checksum for an entire file, + or a specific section of a file (specified by beginning and + ending line numbers with the "beginline" and "endline" + parameters, respectively). + The latter is useful for source files with a license + notice header, README documents, and so forth. + If you do not use the "beginline" parameter, then it is + assumed that the text begins on the first line of the file. + Similarly, if you do not use the "endline" parameter, + it is assumed that the license text ends with the last + line of the file. + </para> + + <para> + The "md5" parameter stores the md5 checksum of the license + text. + If the license text changes in any way as compared to + this parameter then a mismatch occurs. + This mismatch triggers a build failure and notifies + the developer. + Notification allows the developer to review and address + the license text changes. + Also note that if a mismatch occurs during the build, + the correct md5 checksum is placed in the build log and + can be easily copied to the recipe. + </para> + + <para> + There is no limit to how many files you can specify using + the <filename>LIC_FILES_CHKSUM</filename> variable. + Generally, however, every project requires a few + specifications for license tracking. + Many projects have a "COPYING" file that stores the + license information for all the source code files. + This practice allows you to just track the "COPYING" + file as long as it is kept up to date. + <note><title>Tips</title> + <itemizedlist> + <listitem><para> + If you specify an empty or invalid "md5" + parameter, + <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> + returns an md5 mis-match + error and displays the correct "md5" parameter + value during the build. + The correct parameter is also captured in + the build log. + </para></listitem> + <listitem><para> + If the whole file contains only license text, + you do not need to use the "beginline" and + "endline" parameters. + </para></listitem> + </itemizedlist> + </note> + </para> + </section> + </section> + + <section id="enabling-commercially-licensed-recipes"> + <title>Enabling Commercially Licensed Recipes</title> + + <para> + By default, the OpenEmbedded build system disables + components that have commercial or other special licensing + requirements. + Such requirements are defined on a + recipe-by-recipe basis through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></ulink> + variable definition in the affected recipe. + For instance, the + <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename> + recipe contains the following statement: <literallayout class='monospaced'> - INHERIT += "archiver" - ARCHIVER_MODE[src] = "original" + LICENSE_FLAGS = "commercial" + </literallayout> + Here is a slightly more complicated example that contains both + an explicit recipe name and version (after variable expansion): + <literallayout class='monospaced'> + LICENSE_FLAGS = "license_${PN}_${PV}" + </literallayout> + In order for a component restricted by a + <filename>LICENSE_FLAGS</filename> definition to be enabled and + included in an image, it needs to have a matching entry in the + global + <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE_FLAGS_WHITELIST'><filename>LICENSE_FLAGS_WHITELIST</filename></ulink> + variable, which is a variable typically defined in your + <filename>local.conf</filename> file. + For example, to enable the + <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename> + package, you could add either the string + "commercial_gst-plugins-ugly" or the more general string + "commercial" to <filename>LICENSE_FLAGS_WHITELIST</filename>. + See the + "<link linkend='license-flag-matching'>License Flag Matching</link>" + section for a full + explanation of how <filename>LICENSE_FLAGS</filename> matching + works. + Here is the example: + <literallayout class='monospaced'> + LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly" + </literallayout> + Likewise, to additionally enable the package built from the + recipe containing + <filename>LICENSE_FLAGS = "license_${PN}_${PV}"</filename>, + and assuming that the actual recipe name was + <filename>emgd_1.10.bb</filename>, the following string would + enable that package as well as the original + <filename>gst-plugins-ugly</filename> package: + <literallayout class='monospaced'> + LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10" + </literallayout> + As a convenience, you do not need to specify the complete + license string in the whitelist for every package. + You can use an abbreviated form, which consists + of just the first portion or portions of the license + string before the initial underscore character or characters. + A partial string will match any license that contains the + given string as the first portion of its license. + For example, the following whitelist string will also match + both of the packages previously mentioned as well as any other + packages that have licenses starting with "commercial" or + "license". + <literallayout class='monospaced'> + LICENSE_FLAGS_WHITELIST = "commercial license" </literallayout> - During the creation of your image, the source from all - recipes that deploy packages to the image is placed within - subdirectories of - <filename>DEPLOY_DIR/sources</filename> based on the - <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE'><filename>LICENSE</filename></ulink> - for each recipe. - Releasing the entire directory enables you to comply with - requirements concerning providing the unmodified source. - It is important to note that the size of the directory can - get large. </para> + <section id="license-flag-matching"> + <title>License Flag Matching</title> + + <para> + License flag matching allows you to control what recipes + the OpenEmbedded build system includes in the build. + Fundamentally, the build system attempts to match + <filename>LICENSE_FLAGS</filename> strings found in recipes + against <filename>LICENSE_FLAGS_WHITELIST</filename> + strings found in the whitelist. + A match causes the build system to include a recipe in the + build, while failure to find a match causes the build + system to exclude a recipe. + </para> + + <para> + In general, license flag matching is simple. + However, understanding some concepts will help you + correctly and effectively use matching. + </para> + + <para> + Before a flag + defined by a particular recipe is tested against the + contents of the whitelist, the expanded string + <filename>_${PN}</filename> is appended to the flag. + This expansion makes each + <filename>LICENSE_FLAGS</filename> value recipe-specific. + After expansion, the string is then matched against the + whitelist. + Thus, specifying + <filename>LICENSE_FLAGS = "commercial"</filename> + in recipe "foo", for example, results in the string + <filename>"commercial_foo"</filename>. + And, to create a match, that string must appear in the + whitelist. + </para> + + <para> + Judicious use of the <filename>LICENSE_FLAGS</filename> + strings and the contents of the + <filename>LICENSE_FLAGS_WHITELIST</filename> variable + allows you a lot of flexibility for including or excluding + recipes based on licensing. + For example, you can broaden the matching capabilities by + using license flags string subsets in the whitelist. + <note> + When using a string subset, be sure to use the part of + the expanded string that precedes the appended + underscore character (e.g. + <filename>usethispart_1.3</filename>, + <filename>usethispart_1.4</filename>, and so forth). + </note> + For example, simply specifying the string "commercial" in + the whitelist matches any expanded + <filename>LICENSE_FLAGS</filename> definition that starts + with the string "commercial" such as "commercial_foo" and + "commercial_bar", which are the strings the build system + automatically generates for hypothetical recipes named + "foo" and "bar" assuming those recipes simply specify the + following: + <literallayout class='monospaced'> + LICENSE_FLAGS = "commercial" + </literallayout> + Thus, you can choose to exhaustively + enumerate each license flag in the whitelist and + allow only specific recipes into the image, or + you can use a string subset that causes a broader range of + matches to allow a range of recipes into the image. + </para> + + <para> + This scheme works even if the + <filename>LICENSE_FLAGS</filename> string already + has <filename>_${PN}</filename> appended. + For example, the build system turns the license flag + "commercial_1.2_foo" into "commercial_1.2_foo_foo" and + would match both the general "commercial" and the specific + "commercial_1.2_foo" strings found in the whitelist, as + expected. + </para> + + <para> + Here are some other scenarios: + <itemizedlist> + <listitem><para> + You can specify a versioned string in the recipe + such as "commercial_foo_1.2" in a "foo" recipe. + The build system expands this string to + "commercial_foo_1.2_foo". + Combine this license flag with a whitelist that has + the string "commercial" and you match the flag + along with any other flag that starts with the + string "commercial". + </para></listitem> + <listitem><para> + Under the same circumstances, you can use + "commercial_foo" in the whitelist and the build + system not only matches "commercial_foo_1.2" but + also matches any license flag with the string + "commercial_foo", regardless of the version. + </para></listitem> + <listitem><para> + You can be very specific and use both the + package and version parts in the whitelist (e.g. + "commercial_foo_1.2") to specifically match a + versioned recipe. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id="other-variables-related-to-commercial-licenses"> + <title>Other Variables Related to Commercial Licenses</title> + + <para> + Other helpful variables related to commercial + license handling exist and are defined in the + <filename>poky/meta/conf/distro/include/default-distrovars.inc</filename> file: + <literallayout class='monospaced'> + COMMERCIAL_AUDIO_PLUGINS ?= "" + COMMERCIAL_VIDEO_PLUGINS ?= "" + </literallayout> + If you want to enable these components, you can do so by + making sure you have statements similar to the following + in your <filename>local.conf</filename> configuration file: + <literallayout class='monospaced'> + COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \ + gst-plugins-ugly-mpegaudioparse" + COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \ + gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse" + LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp" + </literallayout> + Of course, you could also create a matching whitelist + for those components using the more general "commercial" + in the whitelist, but that would also enable all the + other packages with <filename>LICENSE_FLAGS</filename> + containing "commercial", which you may or may not want: + <literallayout class='monospaced'> + LICENSE_FLAGS_WHITELIST = "commercial" + </literallayout> + </para> + + <para> + Specifying audio and video plug-ins as part of the + <filename>COMMERCIAL_AUDIO_PLUGINS</filename> and + <filename>COMMERCIAL_VIDEO_PLUGINS</filename> statements + (along with the enabling + <filename>LICENSE_FLAGS_WHITELIST</filename>) includes the + plug-ins or components into built images, thus adding + support for media formats or components. + </para> + </section> + </section> + + <section id='maintaining-open-source-license-compliance-during-your-products-lifecycle'> + <title>Maintaining Open Source License Compliance During Your Product's Lifecycle</title> + <para> - A way to help mitigate the size issue is to only release - tarballs for licenses that require the release of - source. - Let us assume you are only concerned with GPL code as - identified by running the following script: - <literallayout class='monospaced'> + One of the concerns for a development organization using open source + software is how to maintain compliance with various open source + licensing during the lifecycle of the product. + While this section does not provide legal advice or + comprehensively cover all scenarios, it does + present methods that you can use to + assist you in meeting the compliance requirements during a software + release. + </para> + + <para> + With hundreds of different open source licenses that the Yocto + Project tracks, it is difficult to know the requirements of each + and every license. + However, the requirements of the major FLOSS licenses can begin + to be covered by + assuming that three main areas of concern exist: + <itemizedlist> + <listitem><para>Source code must be provided.</para></listitem> + <listitem><para>License text for the software must be + provided.</para></listitem> + <listitem><para>Compilation scripts and modifications to the + source code must be provided. + </para></listitem> + </itemizedlist> + There are other requirements beyond the scope of these + three and the methods described in this section + (e.g. the mechanism through which source code is distributed). + </para> + + <para> + As different organizations have different methods of complying with + open source licensing, this section is not meant to imply that + there is only one single way to meet your compliance obligations, + but rather to describe one method of achieving compliance. + The remainder of this section describes methods supported to meet the + previously mentioned three requirements. + Once you take steps to meet these requirements, + and prior to releasing images, sources, and the build system, + you should audit all artifacts to ensure completeness. + <note> + The Yocto Project generates a license manifest during + image creation that is located + in <filename>${DEPLOY_DIR}/licenses/<replaceable>image_name-datestamp</replaceable></filename> + to assist with any audits. + </note> + </para> + + <section id='providing-the-source-code'> + <title>Providing the Source Code</title> + + <para> + Compliance activities should begin before you generate the + final image. + The first thing you should look at is the requirement that + tops the list for most compliance groups - providing + the source. + The Yocto Project has a few ways of meeting this + requirement. + </para> + + <para> + One of the easiest ways to meet this requirement is + to provide the entire + <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> + used by the build. + This method, however, has a few issues. + The most obvious is the size of the directory since it includes + all sources used in the build and not just the source used in + the released image. + It will include toolchain source, and other artifacts, which + you would not generally release. + However, the more serious issue for most companies is accidental + release of proprietary software. + The Yocto Project provides an + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-archiver'><filename>archiver</filename></ulink> + class to help avoid some of these concerns. + </para> + + <para> + Before you employ <filename>DL_DIR</filename> or the + <filename>archiver</filename> class, you need to decide how + you choose to provide source. + The source <filename>archiver</filename> class can generate + tarballs and SRPMs and can create them with various levels of + compliance in mind. + </para> + + <para> + One way of doing this (but certainly not the only way) is to + release just the source as a tarball. + You can do this by adding the following to the + <filename>local.conf</filename> file found in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>: + <literallayout class='monospaced'> + INHERIT += "archiver" + ARCHIVER_MODE[src] = "original" + </literallayout> + During the creation of your image, the source from all + recipes that deploy packages to the image is placed within + subdirectories of + <filename>DEPLOY_DIR/sources</filename> based on the + <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE'><filename>LICENSE</filename></ulink> + for each recipe. + Releasing the entire directory enables you to comply with + requirements concerning providing the unmodified source. + It is important to note that the size of the directory can + get large. + </para> + + <para> + A way to help mitigate the size issue is to only release + tarballs for licenses that require the release of + source. + Let us assume you are only concerned with GPL code as + identified by running the following script: + <literallayout class='monospaced'> # Script to archive a subset of packages matching specific license(s) # Source and license files are copied into sub folders of package folder # Must be run from build folder @@ -10515,96 +14941,97 @@ Some notes from Cal: cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null fi done - done </literallayout> - At this point, you could create a tarball from the - <filename>gpl_source_release</filename> directory and - provide that to the end user. - This method would be a step toward achieving compliance - with section 3a of GPLv2 and with section 6 of GPLv3. - </para> - </section> + done + </literallayout> + At this point, you could create a tarball from the + <filename>gpl_source_release</filename> directory and + provide that to the end user. + This method would be a step toward achieving compliance + with section 3a of GPLv2 and with section 6 of GPLv3. + </para> + </section> - <section id='providing-license-text'> - <title>Providing License Text</title> + <section id='providing-license-text'> + <title>Providing License Text</title> - <para> - One requirement that is often overlooked is inclusion - of license text. - This requirement also needs to be dealt with prior to - generating the final image. - Some licenses require the license text to accompany - the binary. - You can achieve this by adding the following to your - <filename>local.conf</filename> file: - <literallayout class='monospaced'> + <para> + One requirement that is often overlooked is inclusion + of license text. + This requirement also needs to be dealt with prior to + generating the final image. + Some licenses require the license text to accompany + the binary. + You can achieve this by adding the following to your + <filename>local.conf</filename> file: + <literallayout class='monospaced'> COPY_LIC_MANIFEST = "1" COPY_LIC_DIRS = "1" LICENSE_CREATE_PACKAGE = "1" - </literallayout> - Adding these statements to the configuration file ensures - that the licenses collected during package generation - are included on your image. - <note> - <para>Setting all three variables to "1" results in the - image having two copies of the same license file. - One copy resides in - <filename>/usr/share/common-licenses</filename> and - the other resides in - <filename>/usr/share/license</filename>.</para> - - <para>The reason for this behavior is because - <ulink url='&YOCTO_DOCS_REF_URL;#var-COPY_LIC_DIRS'><filename>COPY_LIC_DIRS</filename></ulink> - and - <ulink url='&YOCTO_DOCS_REF_URL;#var-COPY_LIC_MANIFEST'><filename>COPY_LIC_MANIFEST</filename></ulink> - add a copy of the license when the image is built but do - not offer a path for adding licenses for newly installed - packages to an image. - <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE_CREATE_PACKAGE'><filename>LICENSE_CREATE_PACKAGE</filename></ulink> - adds a separate package and an upgrade path for adding - licenses to an image.</para> - </note> - </para> + </literallayout> + Adding these statements to the configuration file ensures + that the licenses collected during package generation + are included on your image. + <note> + <para>Setting all three variables to "1" results in the + image having two copies of the same license file. + One copy resides in + <filename>/usr/share/common-licenses</filename> and + the other resides in + <filename>/usr/share/license</filename>.</para> + + <para>The reason for this behavior is because + <ulink url='&YOCTO_DOCS_REF_URL;#var-COPY_LIC_DIRS'><filename>COPY_LIC_DIRS</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-COPY_LIC_MANIFEST'><filename>COPY_LIC_MANIFEST</filename></ulink> + add a copy of the license when the image is built but do + not offer a path for adding licenses for newly installed + packages to an image. + <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE_CREATE_PACKAGE'><filename>LICENSE_CREATE_PACKAGE</filename></ulink> + adds a separate package and an upgrade path for adding + licenses to an image.</para> + </note> + </para> - <para> - As the source <filename>archiver</filename> class has already - archived the original - unmodified source that contains the license files, - you would have already met the requirements for inclusion - of the license information with source as defined by the GPL - and other open source licenses. - </para> - </section> + <para> + As the source <filename>archiver</filename> class has already + archived the original + unmodified source that contains the license files, + you would have already met the requirements for inclusion + of the license information with source as defined by the GPL + and other open source licenses. + </para> + </section> - <section id='providing-compilation-scripts-and-source-code-modifications'> - <title>Providing Compilation Scripts and Source Code Modifications</title> + <section id='providing-compilation-scripts-and-source-code-modifications'> + <title>Providing Compilation Scripts and Source Code Modifications</title> - <para> - At this point, we have addressed all we need to - prior to generating the image. - The next two requirements are addressed during the final - packaging of the release. - </para> + <para> + At this point, we have addressed all we need to + prior to generating the image. + The next two requirements are addressed during the final + packaging of the release. + </para> - <para> - By releasing the version of the OpenEmbedded build system - and the layers used during the build, you will be providing both - compilation scripts and the source code modifications in one - step. - </para> + <para> + By releasing the version of the OpenEmbedded build system + and the layers used during the build, you will be providing both + compilation scripts and the source code modifications in one + step. + </para> - <para> - If the deployment team has a - <ulink url='&YOCTO_DOCS_BSP_URL;#bsp-layers'>BSP layer</ulink> - and a distro layer, and those those layers are used to patch, - compile, package, or modify (in any way) any open source - software included in your released images, you - might be required to release those layers under section 3 of - GPLv2 or section 1 of GPLv3. - One way of doing that is with a clean - checkout of the version of the Yocto Project and layers used - during your build. - Here is an example: - <literallayout class='monospaced'> + <para> + If the deployment team has a + <ulink url='&YOCTO_DOCS_BSP_URL;#bsp-layers'>BSP layer</ulink> + and a distro layer, and those those layers are used to patch, + compile, package, or modify (in any way) any open source + software included in your released images, you + might be required to release those layers under section 3 of + GPLv2 or section 1 of GPLv3. + One way of doing that is with a clean + checkout of the version of the Yocto Project and layers used + during your build. + Here is an example: + <literallayout class='monospaced'> # We built using the &DISTRO_NAME_NO_CAP; branch of the poky repo $ git clone -b &DISTRO_NAME_NO_CAP; git://git.yoctoproject.org/poky $ cd poky @@ -10613,18 +15040,18 @@ Some notes from Cal: $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer # clean up the .git repos $ find . -name ".git" -type d -exec rm -rf {} \; - </literallayout> - One thing a development organization might want to consider - for end-user convenience is to modify - <filename>meta-poky/conf/bblayers.conf.sample</filename> to - ensure that when the end user utilizes the released build - system to build an image, the development organization's - layers are included in the <filename>bblayers.conf</filename> - file automatically: - <literallayout class='monospaced'> - # LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf + </literallayout> + One thing a development organization might want to consider + for end-user convenience is to modify + <filename>meta-poky/conf/bblayers.conf.sample</filename> to + ensure that when the end user utilizes the released build + system to build an image, the development organization's + layers are included in the <filename>bblayers.conf</filename> + file automatically: + <literallayout class='monospaced'> + # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf # changes incompatibly - LCONF_VERSION = "6" + POKY_BBLAYERS_CONF_VERSION = "2" BBPATH = "${TOPDIR}" BBFILES ?= "" @@ -10635,14 +15062,15 @@ Some notes from Cal: ##OEROOT##/meta-yocto-bsp \ ##OEROOT##/meta-mylayer \ " - </literallayout> - Creating and providing an archive of the - <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink> - layers (recipes, configuration files, and so forth) - enables you to meet your - requirements to include the scripts to control compilation - as well as any modifications to the original source. - </para> + </literallayout> + Creating and providing an archive of the + <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink> + layers (recipes, configuration files, and so forth) + enables you to meet your + requirements to include the scripts to control compilation + as well as any modifications to the original source. + </para> + </section> </section> </section> @@ -10761,6 +15189,136 @@ Some notes from Cal: </para> </section> </section> + + <section id="dev-using-wayland-and-weston"> + <title>Using Wayland and Weston</title> + + <para> + <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)'>Wayland</ulink> + is a computer display server protocol that + provides a method for compositing window managers to communicate + directly with applications and video hardware and expects them to + communicate with input hardware using other libraries. + Using Wayland with supporting targets can result in better control + over graphics frame rendering than an application might otherwise + achieve. + </para> + + <para> + The Yocto Project provides the Wayland protocol libraries and the + reference + <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston'>Weston</ulink> + compositor as part of its release. + You can find the integrated packages in the + <filename>meta</filename> layer of the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>. + Specifically, you can find the recipes that build both Wayland + and Weston at <filename>meta/recipes-graphics/wayland</filename>. + </para> + + <para> + You can build both the Wayland and Weston packages for use only + with targets that accept the + <ulink url='https://en.wikipedia.org/wiki/Mesa_(computer_graphics)'>Mesa 3D and Direct Rendering Infrastructure</ulink>, + which is also known as Mesa DRI. + This implies that you cannot build and use the packages if your + target uses, for example, the + <trademark class='registered'>Intel</trademark> Embedded Media + and Graphics Driver + (<trademark class='registered'>Intel</trademark> EMGD) that + overrides Mesa DRI. + <note> + Due to lack of EGL support, Weston 1.0.3 will not run + directly on the emulated QEMU hardware. + However, this version of Weston will run under X emulation + without issues. + </note> + </para> + + <para> + This section describes what you need to do to implement Wayland and + use the Weston compositor when building an image for a supporting + target. + </para> + + <section id="enabling-wayland-in-an-image"> + <title>Enabling Wayland in an Image</title> + + <para> + To enable Wayland, you need to enable it to be built and enable + it to be included (installed) in the image. + </para> + + <section id="enable-building"> + <title>Building</title> + + <para> + To cause Mesa to build the <filename>wayland-egl</filename> + platform and Weston to build Wayland with Kernel Mode + Setting + (<ulink url='https://wiki.archlinux.org/index.php/Kernel_Mode_Setting'>KMS</ulink>) + support, include the "wayland" flag in the + <ulink url="&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES"><filename>DISTRO_FEATURES</filename></ulink> + statement in your <filename>local.conf</filename> file: + <literallayout class='monospaced'> + DISTRO_FEATURES_append = " wayland" + </literallayout> + <note> + If X11 has been enabled elsewhere, Weston will build + Wayland with X11 support + </note> + </para> + </section> + + <section id="enable-installation-in-an-image"> + <title>Installing</title> + + <para> + To install the Wayland feature into an image, you must + include the following + <ulink url='&YOCTO_DOCS_REF_URL;#var-CORE_IMAGE_EXTRA_INSTALL'><filename>CORE_IMAGE_EXTRA_INSTALL</filename></ulink> + statement in your <filename>local.conf</filename> file: + <literallayout class='monospaced'> + CORE_IMAGE_EXTRA_INSTALL += "wayland weston" + </literallayout> + </para> + </section> + </section> + + <section id="running-weston"> + <title>Running Weston</title> + + <para> + To run Weston inside X11, enabling it as described earlier and + building a Sato image is sufficient. + If you are running your image under Sato, a Weston Launcher + appears in the "Utility" category. + </para> + + <para> + Alternatively, you can run Weston through the command-line + interpretor (CLI), which is better suited for development work. + To run Weston under the CLI, you need to do the following after + your image is built: + <orderedlist> + <listitem><para> + Run these commands to export + <filename>XDG_RUNTIME_DIR</filename>: + <literallayout class='monospaced'> + mkdir -p /tmp/$USER-weston + chmod 0700 /tmp/$USER-weston + export XDG_RUNTIME_DIR=/tmp/$USER-weston + </literallayout> + </para></listitem> + <listitem><para> + Launch Weston in the shell: + <literallayout class='monospaced'> + weston + </literallayout></para></listitem> + </orderedlist> + </para> + </section> + </section> </chapter> <!-- |