poky: subtree update:0ac99625bf..796be0593a

Alexander Kanavin (31):
      netbase: upgrade 6.1 -> 6.2
      meson: upgrade 0.55.1 -> 0.56.0
      vulkan-samples: update to latest revision
      libcap: update 2.44 -> 2.45
      bind: upgrade 9.16.7 -> 9.16.9
      quota: upgrade 4.05 -> 4.06
      pango: upgrade 1.46.2 -> 1.48.0
      elfutils: upgrade 0.181 -> 0.182
      ifupdown: upgrade 0.8.35 -> 0.8.36
      createrepo-c: upgrade 0.16.1 -> 0.16.2
      acpica: upgrade 20200925 -> 20201113
      grep: upgrade 3.5 -> 3.6
      man-pages: upgrade 5.08 -> 5.09
      stress-ng: upgrade 0.11.23 -> 0.11.24
      libhandy: upgrade 1.0.1 -> 1.0.2
      piglit: upgrade to latest revision
      xkbcomp: upgrade 1.4.3 -> 1.4.4
      lz4: upgrade 1.9.2 -> 1.9.3
      bison: upgrade 3.7.3 -> 3.7.4
      python3-setuptools-scm: fix upstream version check
      cantarell-fonts: update 0.0.25 -> 0.201
      meta/lib/oe/reproducible.py: gitsm:// works just as fine as git:// for timestamps
      llvm: fix reproducibility
      ruby: fix reproducibility
      webkitgtk: fix reproducibility
      ffmpeg: fix reproducibility
      piglit: fix reproducibility
      serf: do not install the static library
      llvm: sort the lists in generated source reproducibibly
      kea: fix reproducibility
      poky.conf: do not write current date into distro version, use git hash instead

Andrej Valek (1):
      kernel-dummy: fix executing unexpected tasks

Anuj Mittal (1):
      releases.rst: add gatesgarth to current releases

Brett Warren (1):
      libffi: add patch to revert clang VFP workaround

Chandana kalluri (1):
      populate_sdk_ext: use SDK_CUSTOM_TEPLATECONF variable to enable custom templateconf.cfg

Changqing Li (1):
      buildtools-tarball: add wic dependency into extended buildtools

Diego Sueiro (2):
      modutils-initscripts: Fix modules.dep creation when USE_DEPMOD="0"
      initscripts: Change execution order between checkroot and modutils

Dmitry Baryshkov (2):
      linux-firmware: upgrade 20201022 -> 20201118
      linux-firmware: package ath11k firmware

Fabio Berton (1):
      mesa: Update 20.2.1 -> 20.2.4

Gratian Crisan (1):
      kernel-module-split.bbclass: fix kernel modules getting marked as CONFFILES

Jack Mitchell (3):
      Revert "connman: set service to conflict with systemd-networkd"
      systemd-conf: add PACKAGECONFIG to enable/disable auto ethernet DHCP
      systemd-conf: match ethernet interfaces by type rather than globbing

Joshua Watt (2):
      bitbake: hashserv: client: Fix AF_UNIX path length limits
      bitbake: hashserv: Fix broken AF_UNIX path length limit

Kai Kang (2):
      systemd-systemctl-native: capable to call without argument
      systemd.bbclass: update command to check systemctl available

Kevin Hao (1):
      tune-octeontx2.inc: Add tune for Marvell OCTEON TX2 core

Li Wang (2):
      qemu: CVE-2020-29129 CVE-2020-29130
      qemu: CVE-2020-25624

Luca Boccassi (1):
      dbus: move messagebus user to dbus-common package

Michael Halstead (1):
      releases: conf: add link to 3.1.4, update to include 3.1.4

Nicolas Dechesne (19):
      sphinx: add .vscode in .gitignore
      {dev,kernel,sdk}-manual: replace hardcoded release version with &DISTRO;
      sphinx: replace bitbake labels with references to corresponding title
      brief-yoctoprojectqs: replace labels with references to section title
      dev-manual: replace labels with references to section title
      ref-manual: replace labels with references to section title
      sdk-manual: replace labels with references to section title
      overview-manual: remove unused labels
      dev-manual: remove unused labels
      sphinx: rename top level document in each manual
      sphinx: use absolute paths for :doc: references
      test-manual: remove 'test-manual' from filenames
      toaster-manual: remove 'toaster-manual' from filenames
      dev-manual: remove 'dev-manual' from filenames
      kernel-dev: remove 'kernel-dev' from filenames
      profile-manual: remove 'profile-manual' from filenames
      overview-manual: remove 'overview-manual' from filenames
      sdk-manual: remove 'sdk' from filenames
      ref-manual: remove 'ref' from filenames

Paul Barker (5):
      documentation: Simplify yocto_wiki links
      documentation: Simplify yocto_git links
      ref-manual: Simplify oe_git links
      poky.conf: Add opensuseleap-15.2 and fedora-33 to tested distros
      poky.conf: Drop fedora-30 from tested distros

Peter Kjellerstedt (2):
      pseudo: Simplify pseudo_client_ignore_path_chroot()
      bitbake.conf: Add all layers (from BBLAYERS) to PSEUDO_IGNORE_PATHS

Richard Purdie (8):
      lz4: Use the new branch naming from upstream
      Revert "bitbake.conf: Add all layers (from BBLAYERS) to PSEUDO_IGNORE_PATHS"
      build-appliance-image: Update to master head revision
      bitbake: Revert "fetch2: use relative symlinks for anything pulled from PREMIRRORS"
      build-appliance-image: Update to master head revision
      metadata_scm: Fix signature handling of METADATA_REVISION and METADATA_BRANCH
      poky: Set SDK_VERSION explicitly
      build-appliance-image: Update to master head revision

Ross Burton (9):
      oeqa/devtool: use Yocto mirror for pv-1.5.3 tarball
      image_types: remove obsolete tar comment
      image_types: sort tarball file listings
      package_manager/ipk: neaten OPKGLIBDIR logic
      ldconfig-native: don't write auxiliary cache
      package_manager/ipk: improve remove_packaging_data
      oeqa/selftest/containerimage: update for improved cleanup
      coreutils: add SUSE-specific issues to CVE whitelist
      bitbake: msg: use safe YAML loader

Sinan Kaya (1):
      poky-tiny: enable section removal

Tomasz Dziendzielski (1):
      pseudo: Update to print PSEUDO_LOGFILE in abort message on path mismatches

sangeeta jain (1):
      meta/lib/oeqa/manual/oe-core.json: Update test_bitbake_devshell

zangrc (3):
      libinput: upgrade 1.16.3 -> 1.16.4
      lighttpd: upgrade 1.4.55 -> 1.4.56
      sysstat: upgrade 12.4.0 -> 12.4.1

Signed-off-by: Andrew Geissler <geissonator@yahoo.com>
Change-Id: I65f2f1c9d44433f3e62609240012c42256679b51

1 files changed, 285 insertions, 0 deletions

diff --git a/poky/documentation/test-manual/understand-autobuilder.rst b/poky/documentation/test-manual/understand-autobuilder.rst
new file mode 100644
index 0000000000..199cc97a85
--- /dev/null
+++ b/poky/documentation/test-manual/understand-autobuilder.rst
@@ -0,0 +1,285 @@
+.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
+
+*******************************************
+Understanding the Yocto Project Autobuilder
+*******************************************
+
+Execution Flow within the Autobuilder
+=====================================
+
+The "a-full" and "a-quick" targets are the usual entry points into the
+Autobuilder and it makes sense to follow the process through the system
+starting there. This is best visualised from the Autobuilder Console
+view (:yocto_ab:`/typhoon/#/console`).
+
+Each item along the top of that view represents some "target build" and
+these targets are all run in parallel. The 'full' build will trigger the
+majority of them, the "quick" build will trigger some subset of them.
+The Autobuilder effectively runs whichever configuration is defined for
+each of those targets on a seperate buildbot worker. To understand the
+configuration, you need to look at the entry on ``config.json`` file
+within the ``yocto-autobuilder-helper`` repository. The targets are
+defined in the ‘overrides' section, a quick example could be qemux86-64
+which looks like::
+
+   "qemux86-64" : {
+         "MACHINE" : "qemux86-64",
+         "TEMPLATE" : "arch-qemu",
+         "step1" : {
+               "extravars" : [
+                     "IMAGE_FSTYPES_append = ' wic wic.bmap'"
+                    ]
+        }
+   },
+
+And to expand that, you need the "arch-qemu" entry from
+the "templates" section, which looks like::
+
+   "arch-qemu" : {
+         "BUILDINFO" : true,
+         "BUILDHISTORY" : true,
+         "step1" : {
+               "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
+         "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"
+         },
+         "step2" : {
+               "SDKMACHINE" : "x86_64",
+               "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext",
+               "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
+         },
+         "step3" : {
+               "BUILDHISTORY" : false,
+               "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"],
+               "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
+         }
+   },
+
+Combining these two entries you can see that "qemux86-64" is a three step build where the
+``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for
+``MACHINE="qemx86-64"`` but with differing SDKMACHINE settings. In step
+1 an extra variable is added to the ``auto.conf`` file to enable wic
+image generation.
+
+While not every detail of this is covered here, you can see how the
+template mechanism allows quite complex configurations to be built up
+yet allows duplication and repetition to be kept to a minimum.
+
+The different build targets are designed to allow for parallelisation,
+so different machines are usually built in parallel, operations using
+the same machine and metadata are built sequentially, with the aim of
+trying to optimise build efficiency as much as possible.
+
+The ``config.json`` file is processed by the scripts in the Helper
+repository in the ``scripts`` directory. The following section details
+how this works.
+
+Autobuilder Target Execution Overview
+=====================================
+
+For each given target in a build, the Autobuilder executes several
+steps. These are configured in ``yocto-autobuilder2/builders.py`` and
+roughly consist of:
+
+#. *Run clobberdir*.
+
+   This cleans out any previous build. Old builds are left around to
+   allow easier debugging of failed builds. For additional information,
+   see :ref:`test-manual/understand-autobuilder:clobberdir`.
+
+#. *Obtain yocto-autobuilder-helper*
+
+   This step clones the ``yocto-autobuilder-helper`` git repository.
+   This is necessary to prevent the requirement to maintain all the
+   release or project-specific code within Buildbot. The branch chosen
+   matches the release being built so we can support older releases and
+   still make changes in newer ones.
+
+#. *Write layerinfo.json*
+
+   This transfers data in the Buildbot UI when the build was configured
+   to the Helper.
+
+#. *Call scripts/shared-repo-unpack*
+
+   This is a call into the Helper scripts to set up a checkout of all
+   the pieces this build might need. It might clone the BitBake
+   repository and the OpenEmbedded-Core repository. It may clone the
+   Poky repository, as well as additional layers. It will use the data
+   from the ``layerinfo.json`` file to help understand the
+   configuration. It will also use a local cache of repositories to
+   speed up the clone checkouts. For additional information, see
+   :ref:`test-manual/understand-autobuilder:Autobuilder Clone Cache`.
+
+   This step has two possible modes of operation. If the build is part
+   of a parent build, its possible that all the repositories needed may
+   already be available, ready in a pre-prepared directory. An "a-quick"
+   or "a-full" build would prepare this before starting the other
+   sub-target builds. This is done for two reasons:
+
+   -  the upstream may change during a build, for example, from a forced
+      push and this ensures we have matching content for the whole build
+
+   -  if 15 Workers all tried to pull the same data from the same repos,
+      we can hit resource limits on upstream servers as they can think
+      they are under some kind of network attack
+
+   This pre-prepared directory is shared among the Workers over NFS. If
+   the build is an individual build and there is no "shared" directory
+   available, it would clone from the cache and the upstreams as
+   necessary. This is considered the fallback mode.
+
+#. *Call scripts/run-config*
+
+   This is another call into the Helper scripts where its expected that
+   the main functionality of this target will be executed.
+
+Autobuilder Technology
+======================
+
+The Autobuilder has Yocto Project-specific functionality to allow builds
+to operate with increased efficiency and speed.
+
+clobberdir
+----------
+
+When deleting files, the Autobuilder uses ``clobberdir``, which is a
+special script that moves files to a special location, rather than
+deleting them. Files in this location are deleted by an ``rm`` command,
+which is run under ``ionice -c 3``. For example, the deletion only
+happens when there is idle IO capacity on the Worker. The Autobuilder
+Worker Janitor runs this deletion. See :ref:`test-manual/understand-autobuilder:Autobuilder Worker Janitor`.
+
+Autobuilder Clone Cache
+-----------------------
+
+Cloning repositories from scratch each time they are required was slow
+on the Autobuilder. We therefore have a stash of commonly used
+repositories pre-cloned on the Workers. Data is fetched from these
+during clones first, then "topped up" with later revisions from any
+upstream when necessary. The cache is maintained by the Autobuilder
+Worker Janitor. See :ref:`test-manual/understand-autobuilder:Autobuilder Worker Janitor`.
+
+Autobuilder Worker Janitor
+--------------------------
+
+This is a process running on each Worker that performs two basic
+operations, including background file deletion at IO idle (see :ref:`test-manual/understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and
+maintainenance of a cache of cloned repositories to improve the speed
+the system can checkout repositories.
+
+Shared DL_DIR
+-------------
+
+The Workers are all connected over NFS which allows DL_DIR to be shared
+between them. This reduces network accesses from the system and allows
+the build to be sped up. Usage of the directory within the build system
+is designed to be able to be shared over NFS.
+
+Shared SSTATE_DIR
+-----------------
+
+The Workers are all connected over NFS which allows the ``sstate``
+directory to be shared between them. This means once a Worker has built
+an artifact, all the others can benefit from it. Usage of the directory
+within the directory is designed for sharing over NFS.
+
+Resulttool
+----------
+
+All of the different tests run as part of the build generate output into
+``testresults.json`` files. This allows us to determine which tests ran
+in a given build and their status. Additional information, such as
+failure logs or the time taken to run the tests, may also be included.
+
+Resulttool is part of OpenEmbedded-Core and is used to manipulate these
+json results files. It has the ability to merge files together, display
+reports of the test results and compare different result files.
+
+For details, see :yocto_wiki:`/Resulttool`.
+
+run-config Target Execution
+===========================
+
+The ``scripts/run-config`` execution is where most of the work within
+the Autobuilder happens. It runs through a number of steps; the first
+are general setup steps that are run once and include:
+
+#. Set up any ``buildtools-tarball`` if configured.
+
+#. Call "buildhistory-init" if buildhistory is configured.
+
+For each step that is configured in ``config.json``, it will perform the
+following:
+
+#. Add any layers that are specified using the
+   ``bitbake-layers add-layer`` command (logging as stepXa)
+
+#. Call the ``scripts/setup-config`` script to generate the necessary
+   ``auto.conf`` configuration file for the build
+
+#. Run the ``bitbake BBTARGETS`` command (logging as stepXb)
+
+#. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc)
+
+#. Run the ``EXTRACMDS`` command, which are run within the BitBake build
+   environment (logging as stepXd)
+
+#. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the
+   BitBake build environment (logging as stepXd)
+
+#. Remove any layers added in step
+   1 using the ``bitbake-layers remove-layer`` command (logging as stepXa)
+
+Once the execution steps above complete, ``run-config`` executes a set
+of post-build steps, including:
+
+#. Call ``scripts/publish-artifacts`` to collect any output which is to
+   be saved from the build.
+
+#. Call ``scripts/collect-results`` to collect any test results to be
+   saved from the build.
+
+#. Call ``scripts/upload-error-reports`` to send any error reports
+   generated to the remote server.
+
+#. Cleanup the build directory using
+   :ref:`test-manual/understand-autobuilder:clobberdir` if the build was successful,
+   else rename it to "build-renamed" for potential future debugging.
+
+Deploying Yocto Autobuilder
+===========================
+
+The most up to date information about how to setup and deploy your own
+Autbuilder can be found in README.md in the ``yocto-autobuilder2``
+repository.
+
+We hope that people can use the ``yocto-autobuilder2`` code directly but
+it is inevitable that users will end up needing to heavily customise the
+``yocto-autobuilder-helper`` repository, particularly the
+``config.json`` file as they will want to define their own test matrix.
+
+The Autobuilder supports wo customization options:
+
+-  variable substitution
+
+-  overlaying configuration files
+
+The standard ``config.json`` minimally attempts to allow substitution of
+the paths. The Helper script repository includes a
+``local-example.json`` file to show how you could override these from a
+separate configuration file. Pass the following into the environment of
+the Autobuilder::
+
+   $ ABHELPER_JSON="config.json local-example.json"
+
+As another example, you could also pass the following into the
+environment::
+
+   $ ABHELPER_JSON="config.json /some/location/local.json"
+
+One issue users often run into is validation of the ``config.json`` files. A
+tip for minimizing issues from invalid json files is to use a Git
+``pre-commit-hook.sh`` script to verify the JSON file before committing
+it. Create a symbolic link as follows::
+
+   $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit