poky: subtree update:26ae42ded7..5951cbcabe

Alex Kiernan (1):
      recipetool: Fix list concatenation when using edit

Alexander Kanavin (4):
      apr-util: make gdbm optional
      gobject-introspection: add a patch to fix a build race
      icu: merge .inc into main recipe
      icu: make filtered data generation optional, serial and off by default

Alexandru N. Onea (3):
      bitbake: perforce: add basic progress handler for perforce
      bitbake: perforce: add local path handling SRC_URI options
      bitbake: bitbake-user-manual: update perforce fetcher docs

Andreas M?ller (1):
      meson.bbclass: avoid unexpected operating-system names

Andreas Müller (6):
      boost: Add upstream patch to fix build on depending projects
      libinput: upgrade 1.15.5 -> 1.15.6
      sqlite3: upgrade 3.32.2 -> 3.32.3
      desktop-file-utils: upgrade 0.24 -> 0.26
      file: upgrade 5.38 -> 5.39
      ffmpeg: upgrade 4.2.3 -> 4.3

Andrej Valek (1):
      oeqa/runtime/cases/ptest: Make output content path absolute

Andrew Geissler (1):
      meson: backport library ordering fix

Armin Kuster (1):
      libuv: move from meta-oe to core for bind update

Arthur She (1):
      igt-gpu-tools: add new package

Changqing Li (1):
      mime.bbclass: fix post install scriptlet error

Chen Qi (1):
      systemd-serialgetty: do not use BindsTo

Daniel McGregor (3):
      sign_rpm.bbclass: ignore thread count
      systemd-conf: Accept MTU from DHCP
      buildhistory-collect-srcrevs: sort directories

He Zhe (1):
      ltp: Fix copy_file_rang02 for 32-bit arches

Hongxu Jia (1):
      libmodulemd: switch branch master -> main

Jacob Kroon (5):
      bitbake: lib/bb/utils.py: Do not preserve TERM in the environment
      bitbake: bitbake-user-manual: Remove TERM from BB_HASHBASE_WHITELIST example
      bitbake.conf: Remove TERM from default BB_HASHBASE_WHITELIST
      grub: Remove native version of grub-efi
      distro_alias: Remove unused grub-efi distro aliases

Jens Rehsack (1):
      u-boot: avoid blind merging all *.cfg

Joe Slater (1):
      systemd: fix CVE-2020-13776

Joshua Watt (5):
      sstatesig: Account for all dataCaches being passed
      bitbake: bitbake: cache: Fix error message with bad multiconfig
      wic: Fix error message when reporting invalid offset
      classes/archiver: Create patched archive before configuring
      bitbake: cache: Bump cache version

Konrad Weihmann (3):
      oeqa/runtime: Add OERequirePackage decorator
      bitbake: cookerdata: Add BBFILES_DYNAMIC inverse mode
      bitbake: bitbake-user-manual: Add BBFILES_DYNAMIC

Mark Morton (2):
      New source files and Makefile update for Test Manual
      test-manual: Fixed codeblock formatting

Martin Jansa (1):
      net-tools: backport a patch from upstream to use the same ifconfig format as debian/ubuntu

Mingli Yu (3):
      python3: add the rdepends for python3-misc
      python3: add rdepends for python3-idle
      python3-dbusmock: add the missing rdepends

Otavio Salvador (2):
      systemd: Sync systemd-serialgetty@.service with upstream
      mtd-utils: Fix return value of ubiformat

Ovidiu Panait (2):
      dbus-test: Remove EXTRA_OECONF_X configs
      dbus,dbus-test: Move common parts to dbus.inc

Paul Barker (2):
      bitbake: fetch2/gitsm: Mark srcrev as fetched once all submodules are processed
      bitbake: fetch2/gitsm: Make need_update() process submodules

Paul Eggleton (5):
      graph-tool: switch to argparse
      graph-tool: add filter subcommand
      dpkg-native: rebase and reinstate fix for "tar: file changed as we read it"
      shadow-sysroot: drop unused SRC_URI checksums
      devtool: fix typo

Peter Kjellerstedt (1):
      relocatable.bbclass: Avoid an exception if an empty pkgconfig dir exist

Pierre-Jean Texier (3):
      diffoscope: upgrade 146 -> 147
      ell: upgrade 0.31 -> 0.32
      curl: upgrade 7.70.0 -> 7.71.0

Rasmus Villemoes (1):
      curl: add debug info

Richard Purdie (15):
      buildhistory: Add simplistic file move detection
      bitbake: bin/bitbake: Update to next series release version
      perl: Fix host specific modules problems
      sanity.conf: Require bitbake 1.47.0 as the minimum version
      patchelf: Upgrade 0.10 -> 0.11
      test-manual: Add SPDX license headers
      Makefile: Drop obsolete edison/denzil branch conditionals
      bitbake: tests/fetch: Switch from git.infradead.org to a YP mirror
      pseudo: Fix attr errors due to incorrect library resolution issues
      oeqa/selftest/runcmd: Add better debug for thread count mismatch failures
      oeqa/utils/command: Improve stdin handling in runCmd
      vulkan-headers: Fix upstream branch deletion issue
      recipes: Fix Upstream-Status Accepted -> Backport
      scripts/install-buildtools: Update to 3.2 M1 buildtools
      scripts/install-buildtools: Handle new format checksum files

Robert P. J. Day (1):
      python: use official "pypi.org" URLs for HOMEPAGE

Ross Burton (8):
      install-buildtools: fail if an error occurs
      install-buildtools: remove hardcoded x86-64 architecture
      install-buildtools: add option to disable checksum validation
      common-licenses: add BSD-2-Clause-Patent
      gstreamer1.0-plugins-bad: add support for vdpau
      go-binary-native: add binary Go to bootstrap
      tcmode-default: use go-binary-native by default
      go-native: merge bb/inc and add comment

Ryan Rowe (1):
      python3: fix PGO for non-reproducible biniaries

Sakib Sajal (1):
      qemu: uprev v4.2.0 -> v5.0.0

Samuli Piippo (2):
      cmake: allow chainloading of the toolchain file
      perl: use relative paths in the perl wrapper

Steve Sakoman (1):
      buildtools-tarball: export OPENSSL_CONF in environment setup

Tanu Kaskinen (1):
      pulseaudio: remove unnecessary libltdl copying

Trevor Gamblin (1):
      python3-setuptools: patch entrypoints for faster initialization

Tuomas Salokanto (1):
      recipetool: create: fix SRCBRANCH not being passed to params

Valentin Longchamp (2):
      tools-profile: disable valgrind for powerpc soft-float
      valgrind: disable it for powerpc soft-float

Wang Mingyu (5):
      powertop: upgrade 2.12 -> 2.13
      man-db: upgrade 2.9.2 -> 2.9.3
      valgrind: upgrade 3.16.0 -> 3.16.1
      man-pages: upgrade 5.06 -> 5.07
      harfbuzz: upgrade 2.6.7 -> 2.6.8

Yi Zhao (2):
      iptables: fix invalid symbolic link for ip6tables-apply
      iptables: split iptables-apply to its own package

Yongxin Liu (1):
      linux-firmware: add ice for Intel E800 series driver

Yuki Hoshino (1):
      sysvinit-inittab: Add support for tty devices with 10 or more number.

akuster (9):
      bind: update to 9.11.19
      adt-manual: Add SPDX license headers
      bsp-guide: Add SPDX license headers
      brief-yoctoprojectsqa: Add SPDX license headers
      dev-manual: Add SPDX License headers
      kernel-dev: Add SPDX license headers
      profile-manual: Add SPDX licence headers
      sdk-manual: Add SPDX license headers
      toaster-manaul: Add SPDX license headers

haiqing (1):
      libpam: Remove option 'obscure' from common-password

hongxu (1):
      kmod: add nativesdk support

zangrc (1):
      ethtool:upgrade 5.6 -> 5.7

Signed-off-by: Andrew Geissler <geissonator@yahoo.com>
Change-Id: I1190ca17297b1167286cfc06033e8485396c7cce

1 files changed, 314 insertions, 0 deletions

diff --git a/poky/documentation/test-manual/test-manual-understand-autobuilder.xml b/poky/documentation/test-manual/test-manual-understand-autobuilder.xml
new file mode 100644
index 000000000..a04006605
--- /dev/null
+++ b/poky/documentation/test-manual/test-manual-understand-autobuilder.xml
@@ -0,0 +1,314 @@
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
+[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
+<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
+
+<chapter id='test-manual-understand-autobuilder'>
+
+<title>Understanding the Yocto Project Autobuilder</title>
+    <section>
+        <title>Execution Flow within the Autobuilder</title>
+        <para>The “a-full” and “a-quick” targets are the usual entry points into the Autobuilder and
+            it makes sense to follow the process through the system starting there. This is best
+            visualised from the Autobuilder Console view (<link linkend=""
+                >https://autobuilder.yoctoproject.org/typhoon/#/console</link>). </para>
+        <para>Each item along the top of that view represents some “target build” and these targets
+            are all run in parallel. The ‘full’ build will trigger the majority of them, the “quick”
+            build will trigger some subset of them. The Autobuilder effectively runs whichever
+            configuration is defined for each of those targets on a seperate buildbot worker. To
+            understand the configuration, you need to look at the entry on
+                <filename>config.json</filename> file within the
+                <filename>yocto-autobuilder-helper</filename> repository. The targets are defined in
+            the ‘overrides’ section, a quick example could be qemux86-64 which looks
+            like:<literallayout class="monospaced">
+     "qemux86-64" : {
+         "MACHINE" : "qemux86-64",
+         "TEMPLATE" : "arch-qemu",
+         "step1" : {
+             "extravars" : [
+                  "IMAGE_FSTYPES_append = ' wic wic.bmap'"
+             ]
+         }
+     },
+                    </literallayout>And
+            to expand that, you need the “arch-qemu” entry from the “templates” section, which looks
+            like:<literallayout class="monospaced">
+     "arch-qemu" : {
+         "BUILDINFO" : true,
+         "BUILDHISTORY" : true,
+         "step1" : {
+             "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
+             "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"
+         },
+         "step2" : {
+             "SDKMACHINE" : "x86_64",
+             "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext",
+             "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
+         },
+         "step3" : {
+             "BUILDHISTORY" : false,
+             "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"],
+             "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
+         }
+     },
+                    </literallayout>Combining
+            these two entries you can see that “qemux86-64” is a three step build where the
+                <filename>bitbake BBTARGETS</filename> would be run, then <filename>bitbake
+                SANITYTARGETS</filename> for each step; all for
+                <filename>MACHINE=”qemx86-64”</filename> but with differing SDKMACHINE settings. In
+            step 1 an extra variable is added to the <filename>auto.conf</filename> file to enable
+            wic image generation.</para>
+        <para>While not every detail of this is covered here, you can see how the templating
+            mechanism allows quite complex configurations to be built up yet allows duplication and
+            repetition to be kept to a minimum.</para>
+        <para>The different build targets are designed to allow for parallelisation, so different
+            machines are usually built in parallel, operations using the same machine and metadata
+            are built sequentially, with the aim of trying to optimise build efficiency as much as
+            possible.</para>
+        <para>The <filename>config.json</filename> file is processed by the scripts in the Helper
+            repository in the <filename>scripts</filename> directory. The following  section details
+            how this works.</para>
+    </section>
+
+    <section id='test-autobuilder-target-exec-overview'>
+        <title>Autobuilder Target Execution Overview</title>
+
+        <para>For each given target in a build, the Autobuilder executes several steps. These are
+            configured in <filename>yocto-autobuilder2/builders.py</filename> and roughly consist
+            of: <orderedlist>
+                <listitem id='test-list-tgt-exec-clobberdir'>
+                    <para><emphasis>Run <filename>clobberdir</filename></emphasis></para>
+                    <para>This cleans out any previous build. Old builds are left around to allow
+                        easier debugging of failed builds. For additional information, see <link
+                            linkend="test-clobberdir"><filename>clobberdir</filename></link>.</para>
+                </listitem>
+                <listitem>
+                    <para><emphasis>Obtain yocto-autobuilder-helper</emphasis></para>
+                    <para>This step clones the <filename>yocto-autobuilder-helper</filename> git
+                        repository. This is necessary to prevent the requirement to maintain all the
+                        release or project-specific code within Buildbot. The branch chosen matches
+                        the release being built so we can support older releases and still make
+                        changes in newer ones.</para>
+                </listitem>
+                <listitem>
+                    <para><emphasis>Write layerinfo.json</emphasis></para>
+                    <para>This transfers data in the Buildbot UI when the build was configured to
+                        the Helper.</para>
+                </listitem>
+                <listitem>
+                    <para><emphasis>Call scripts/shared-repo-unpack</emphasis></para>
+                    <para>This is a call into the Helper scripts to set up a checkout of all the
+                        pieces this build might need. It might clone the BitBake repository and the
+                        OpenEmbedded-Core repository. It may clone the Poky repository, as well as
+                        additional layers. It will use the data from the
+                            <filename>layerinfo.json</filename> file to help understand the
+                        configuration. It will also use a local cache of repositories to speed up
+                        the clone checkouts. For additional information, see <link
+                            linkend="test-autobuilder-clone-cache">Autobuilder Clone
+                        Cache</link>.</para>
+                    <para>This step has two possible modes of operation. If the build is part of a
+                        parent build, its possible that all the repositories needed may already be
+                        available, ready in a pre-prepared directory. An "a-quick" or "a-full" build
+                        would prepare this before starting the other sub-target builds. This is done
+                        for two reasons:<itemizedlist>
+                            <listitem>
+                                <para>the upstream may change during a build, for example, from a
+                                    forced push and this ensures we have matching content for the
+                                    whole build</para>
+                            </listitem>
+                            <listitem>
+                                <para>if 15 Workers all tried to pull the same data from the same
+                                    repos, we can hit resource limits on upstream servers as they
+                                    can think they are under some kind of network attack</para>
+                            </listitem>
+                        </itemizedlist>This pre-prepared directory is shared among the Workers over
+                        NFS. If the build is an individual build and there is no "shared" directory
+                        available, it would clone from the cache and the upstreams as necessary.
+                        This is considered the fallback mode.</para>
+                </listitem>
+                <listitem>
+                    <para><emphasis>Call scripts/run-config</emphasis></para>
+                    <para>This is another call into the Helper scripts where its expected that the
+                        main functionality of this target will be executed.</para>
+                </listitem>
+            </orderedlist></para>
+    </section>
+    <section id='test-autobuilder-tech'>
+        <title>Autobuilder Technology</title>
+        <para>The Autobuilder has Yocto Project-specific functionality to allow builds to operate
+            with increased efficiency and speed.</para>
+        <section id='test-clobberdir'>
+            <title>clobberdir</title>
+            <para>When deleting files, the Autobuilder uses <filename>clobberdir</filename>, which
+                is a special script that moves files to a special location, rather than deleting
+                them. Files in this location are deleted by an <filename>rm</filename> command,
+                which is run under <filename>ionice -c 3</filename>. For example, the deletion only
+                happens when there is idle IO capacity on the Worker. The Autobuilder Worker Janitor
+                runs this deletion. See <link linkend="test-autobuilder-worker-janitor">Autobuilder
+                    Worker Janitor</link>.</para>
+        </section>
+        <section id='test-autobuilder-clone-cache'>
+            <title>Autobuilder Clone Cache</title>
+            <para>Cloning repositories from scratch each time they are required was slow on the
+                Autobuilder. We therefore have a stash of commonly used repositories pre-cloned on
+                the Workers. Data is fetched from these during clones first, then "topped up" with
+                later revisions from any upstream when necesary. The cache is maintained by the
+                Autobuilder Worker Janitor. See <link linkend="test-autobuilder-worker-janitor"
+                    >Autobuilder Worker Janitor</link>.</para>
+        </section>
+        <section id='test-autobuilder-worker-janitor'>
+            <title>Autobuilder Worker Janitor</title>
+            <para>This is a process running on each Worker that performs two basic operations,
+                including background file deletion at IO idle (see <link
+                    linkend="test-list-tgt-exec-clobberdir">Target Execution: clobberdir</link>) and
+                maintainenance of a cache of cloned repositories to improve the speed the system can
+                checkout repositories.</para>
+        </section>
+        <section id='test-shared-dl-dir'>
+            <title>Shared DL_DIR</title>
+            <para>The Workers are all connected over NFS which allows DL_DIR to be shared between
+                them. This reduces network accesses from the system and allows the build to be sped
+                up. Usage of the directory within the build system is designed to be able to be
+                shared over NFS.</para>
+        </section>
+        <section id='test-shared-sstate-cache'>
+            <title>Shared SSTATE_DIR</title>
+            <para>The Workers are all connected over NFS which allows the
+                    <filename>sstate</filename> directory to be shared between them. This means once
+                a Worker has built an artefact, all the others can benefit from it. Usage of the
+                directory within the directory is designed for sharing over NFS.</para>
+        </section>
+        <section id='test-resulttool'>
+            <title>Resulttool</title>
+            <para>All of the different tests run as part of the build generate output into
+                    <filename>testresults.json</filename> files. This allows us to determine which
+                tests ran in a given build and their status. Additional information, such as failure
+                logs or the time taken to run the tests, may also be included.</para>
+            <para>Resulttool is part of OpenEmbedded-Core and is used to manipulate these json
+                results files. It has the ability to merge files together, display reports of the
+                test results and compare different result files.</para>
+            <para>For details, see <link linkend=""
+                    >https://wiki.yoctoproject.org/wiki/Resulttool</link>.</para>
+        </section>
+    </section>
+    <section id='test-run-config-tgt-execution'>
+        <title>run-config Target Execution</title>
+        <para>The <filename>scripts/run-config</filename> execution is where most of the work within
+            the Autobuilder happens. It runs through a number of steps; the first are general setup
+            steps that are run once and include:<orderedlist>
+                <listitem>
+                    <para>Set up any <filename>buildtools-tarball</filename> if configured.</para>
+                </listitem>
+                <listitem>
+                    <para>Call "buildhistory-init" if buildhistory is configured.</para>
+                </listitem>
+            </orderedlist></para>
+        <para>For each step that is configured in <filename>config.json</filename>, it will perform
+            the following:</para>
+        <para>
+            <remark>## WRITER's question: What does "logging in as stepXa" and others refer to
+                below? ##</remark>
+            <orderedlist>
+                <listitem id="test-run-config-add-layers-step">
+                    <para dir="ltr">Add any layers that are specified using the
+                            <filename>bitbake-layers add-layer</filename> command (logging as
+                        stepXa)</para>
+                </listitem>
+                <listitem>
+                    <para dir="ltr">Call the <filename>scripts/setup-config</filename> script to
+                        generate the necessary <filename>auto.conf</filename> configuration file for
+                        the build</para>
+                </listitem>
+                <listitem>
+                    <para dir="ltr">Run the <filename>bitbake BBTARGETS</filename> command (logging
+                        as stepXb)</para>
+                </listitem>
+                <listitem>
+                    <para dir="ltr">Run the <filename>bitbake SANITYTARGETS</filename> command
+                        (logging as stepXc)</para>
+                </listitem>
+                <listitem>
+                    <para dir="ltr">Run the <filename>EXTRACMDS</filename> command, which are run
+                        within the BitBake build environment (logging as stepXd)</para>
+                </listitem>
+                <listitem>
+                    <para dir="ltr">Run the <filename>EXTRAPLAINCMDS</filename> command(s), which
+                        are run outside the BitBake build environment (logging as stepXd)</para>
+                </listitem>
+                <listitem>
+                    <para dir="ltr">Remove any layers added in <link
+                            linkend="test-run-config-add-layers-step">step 1</link> using the
+                            <filename>bitbake-layers remove-layer</filename> command (logging as
+                        stepXa)</para>
+                </listitem>
+            </orderedlist>
+        </para>
+        <para>Once the execution steps above complete, <filename>run-config</filename> executes a
+            set of post-build steps, including:<orderedlist>
+                <listitem>
+                    <para dir="ltr">Call <filename>scripts/publish-artifacts</filename> to collect
+                        any output which is to be saved from the build.</para>
+                </listitem>
+                <listitem>
+                    <para dir="ltr">Call <filename>scripts/collect-results</filename> to collect any
+                        test results to be saved from the build.</para>
+                </listitem>
+                <listitem>
+                    <para dir="ltr">Call <filename>scripts/upload-error-reports</filename> to send
+                        any error reports generated to the remote server.</para>
+                </listitem>
+                <listitem>
+                    <para dir="ltr">Cleanup the build directory using <link
+                            linkend="test-clobberdir"><filename>clobberdir</filename></link> if the
+                        build was successful, else rename it to “build-renamed” for potential future
+                        debugging.</para>
+                </listitem>
+            </orderedlist></para>
+    </section>
+    <section id='test-deploying-yp-autobuilder'>
+        <title>Deploying Yocto Autobuilder</title>
+        <para>The most up to date information about how to setup and deploy your own Autbuilder can
+            be found in README.md in the <filename>yocto-autobuilder2</filename> repository.</para>
+        <para>We hope that people can use the <filename>yocto-autobuilder2</filename> code directly
+            but it is inevitable that users will end up needing to heavily customise the
+                <filename>yocto-autobuilder-helper</filename> repository, particularly the
+                <filename>config.json</filename> file as they will want to define their own test
+            matrix.</para>
+        <para>The Autobuilder supports wo customization options: <itemizedlist>
+                <listitem>
+                    <para>variable substitution</para>
+                </listitem>
+                <listitem>
+                    <para>overlaying configuration files</para>
+                </listitem>
+            </itemizedlist>The standard <filename>config.json</filename> minimally attempts to allow
+            substitution of the paths. The Helper script repository includes a
+                <filename>local-example.json</filename> file to show how you could override these
+            from a separate configuration file. Pass the following into the environment of the
+            Autobuilder:<literallayout class="monospaced">
+     $ ABHELPER_JSON="config.json local-example.json"
+                    </literallayout>As
+            another example, you could also pass the following into the
+            environment:<literallayout class="monospaced">
+     $ ABHELPER_JSON="config.json <replaceable>/some/location/</replaceable>local.json"
+                    </literallayout>One
+            issue users often run into is validation of the <filename>config.json</filename> files.
+            A tip for minimizing issues from invalid json files is to use a Git
+                <filename>pre-commit-hook.sh</filename> script to verify the JSON file before
+            committing it. Create a symbolic link as
+            follows:<literallayout class="monospaced">
+     $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
+                    </literallayout></para>
+    </section>
+
+
+
+
+
+
+
+
+</chapter>
+<!--
+vim: expandtab tw=80 ts=4
+-->