summaryrefslogtreecommitdiff
path: root/poky/documentation/test-manual/test-manual-understand-autobuilder.rst
diff options
context:
space:
mode:
authorAndrew Geissler <geissonator@yahoo.com>2020-12-13 17:44:15 +0300
committerAndrew Geissler <geissonator@yahoo.com>2020-12-15 21:53:47 +0300
commit09209eec235a35b7089db987561c12e9bd023237 (patch)
tree2d3580484ffacafe11b72e9abaab50a428dd617d /poky/documentation/test-manual/test-manual-understand-autobuilder.rst
parentf7ba29eda266e04f867e4338b6b8b10c1969419c (diff)
downloadopenbmc-09209eec235a35b7089db987561c12e9bd023237.tar.xz
poky: subtree update:0ac99625bf..796be0593a
Alexander Kanavin (31): netbase: upgrade 6.1 -> 6.2 meson: upgrade 0.55.1 -> 0.56.0 vulkan-samples: update to latest revision libcap: update 2.44 -> 2.45 bind: upgrade 9.16.7 -> 9.16.9 quota: upgrade 4.05 -> 4.06 pango: upgrade 1.46.2 -> 1.48.0 elfutils: upgrade 0.181 -> 0.182 ifupdown: upgrade 0.8.35 -> 0.8.36 createrepo-c: upgrade 0.16.1 -> 0.16.2 acpica: upgrade 20200925 -> 20201113 grep: upgrade 3.5 -> 3.6 man-pages: upgrade 5.08 -> 5.09 stress-ng: upgrade 0.11.23 -> 0.11.24 libhandy: upgrade 1.0.1 -> 1.0.2 piglit: upgrade to latest revision xkbcomp: upgrade 1.4.3 -> 1.4.4 lz4: upgrade 1.9.2 -> 1.9.3 bison: upgrade 3.7.3 -> 3.7.4 python3-setuptools-scm: fix upstream version check cantarell-fonts: update 0.0.25 -> 0.201 meta/lib/oe/reproducible.py: gitsm:// works just as fine as git:// for timestamps llvm: fix reproducibility ruby: fix reproducibility webkitgtk: fix reproducibility ffmpeg: fix reproducibility piglit: fix reproducibility serf: do not install the static library llvm: sort the lists in generated source reproducibibly kea: fix reproducibility poky.conf: do not write current date into distro version, use git hash instead Andrej Valek (1): kernel-dummy: fix executing unexpected tasks Anuj Mittal (1): releases.rst: add gatesgarth to current releases Brett Warren (1): libffi: add patch to revert clang VFP workaround Chandana kalluri (1): populate_sdk_ext: use SDK_CUSTOM_TEPLATECONF variable to enable custom templateconf.cfg Changqing Li (1): buildtools-tarball: add wic dependency into extended buildtools Diego Sueiro (2): modutils-initscripts: Fix modules.dep creation when USE_DEPMOD="0" initscripts: Change execution order between checkroot and modutils Dmitry Baryshkov (2): linux-firmware: upgrade 20201022 -> 20201118 linux-firmware: package ath11k firmware Fabio Berton (1): mesa: Update 20.2.1 -> 20.2.4 Gratian Crisan (1): kernel-module-split.bbclass: fix kernel modules getting marked as CONFFILES Jack Mitchell (3): Revert "connman: set service to conflict with systemd-networkd" systemd-conf: add PACKAGECONFIG to enable/disable auto ethernet DHCP systemd-conf: match ethernet interfaces by type rather than globbing Joshua Watt (2): bitbake: hashserv: client: Fix AF_UNIX path length limits bitbake: hashserv: Fix broken AF_UNIX path length limit Kai Kang (2): systemd-systemctl-native: capable to call without argument systemd.bbclass: update command to check systemctl available Kevin Hao (1): tune-octeontx2.inc: Add tune for Marvell OCTEON TX2 core Li Wang (2): qemu: CVE-2020-29129 CVE-2020-29130 qemu: CVE-2020-25624 Luca Boccassi (1): dbus: move messagebus user to dbus-common package Michael Halstead (1): releases: conf: add link to 3.1.4, update to include 3.1.4 Nicolas Dechesne (19): sphinx: add .vscode in .gitignore {dev,kernel,sdk}-manual: replace hardcoded release version with &DISTRO; sphinx: replace bitbake labels with references to corresponding title brief-yoctoprojectqs: replace labels with references to section title dev-manual: replace labels with references to section title ref-manual: replace labels with references to section title sdk-manual: replace labels with references to section title overview-manual: remove unused labels dev-manual: remove unused labels sphinx: rename top level document in each manual sphinx: use absolute paths for :doc: references test-manual: remove 'test-manual' from filenames toaster-manual: remove 'toaster-manual' from filenames dev-manual: remove 'dev-manual' from filenames kernel-dev: remove 'kernel-dev' from filenames profile-manual: remove 'profile-manual' from filenames overview-manual: remove 'overview-manual' from filenames sdk-manual: remove 'sdk' from filenames ref-manual: remove 'ref' from filenames Paul Barker (5): documentation: Simplify yocto_wiki links documentation: Simplify yocto_git links ref-manual: Simplify oe_git links poky.conf: Add opensuseleap-15.2 and fedora-33 to tested distros poky.conf: Drop fedora-30 from tested distros Peter Kjellerstedt (2): pseudo: Simplify pseudo_client_ignore_path_chroot() bitbake.conf: Add all layers (from BBLAYERS) to PSEUDO_IGNORE_PATHS Richard Purdie (8): lz4: Use the new branch naming from upstream Revert "bitbake.conf: Add all layers (from BBLAYERS) to PSEUDO_IGNORE_PATHS" build-appliance-image: Update to master head revision bitbake: Revert "fetch2: use relative symlinks for anything pulled from PREMIRRORS" build-appliance-image: Update to master head revision metadata_scm: Fix signature handling of METADATA_REVISION and METADATA_BRANCH poky: Set SDK_VERSION explicitly build-appliance-image: Update to master head revision Ross Burton (9): oeqa/devtool: use Yocto mirror for pv-1.5.3 tarball image_types: remove obsolete tar comment image_types: sort tarball file listings package_manager/ipk: neaten OPKGLIBDIR logic ldconfig-native: don't write auxiliary cache package_manager/ipk: improve remove_packaging_data oeqa/selftest/containerimage: update for improved cleanup coreutils: add SUSE-specific issues to CVE whitelist bitbake: msg: use safe YAML loader Sinan Kaya (1): poky-tiny: enable section removal Tomasz Dziendzielski (1): pseudo: Update to print PSEUDO_LOGFILE in abort message on path mismatches sangeeta jain (1): meta/lib/oeqa/manual/oe-core.json: Update test_bitbake_devshell zangrc (3): libinput: upgrade 1.16.3 -> 1.16.4 lighttpd: upgrade 1.4.55 -> 1.4.56 sysstat: upgrade 12.4.0 -> 12.4.1 Signed-off-by: Andrew Geissler <geissonator@yahoo.com> Change-Id: I65f2f1c9d44433f3e62609240012c42256679b51
Diffstat (limited to 'poky/documentation/test-manual/test-manual-understand-autobuilder.rst')
-rw-r--r--poky/documentation/test-manual/test-manual-understand-autobuilder.rst285
1 files changed, 0 insertions, 285 deletions
diff --git a/poky/documentation/test-manual/test-manual-understand-autobuilder.rst b/poky/documentation/test-manual/test-manual-understand-autobuilder.rst
deleted file mode 100644
index 698a266ee..000000000
--- a/poky/documentation/test-manual/test-manual-understand-autobuilder.rst
+++ /dev/null
@@ -1,285 +0,0 @@
-.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
-
-*******************************************
-Understanding the Yocto Project Autobuilder
-*******************************************
-
-Execution Flow within the Autobuilder
-=====================================
-
-The "a-full" and "a-quick" targets are the usual entry points into the
-Autobuilder and it makes sense to follow the process through the system
-starting there. This is best visualised from the Autobuilder Console
-view (:yocto_ab:`/typhoon/#/console`).
-
-Each item along the top of that view represents some "target build" and
-these targets are all run in parallel. The 'full' build will trigger the
-majority of them, the "quick" build will trigger some subset of them.
-The Autobuilder effectively runs whichever configuration is defined for
-each of those targets on a seperate buildbot worker. To understand the
-configuration, you need to look at the entry on ``config.json`` file
-within the ``yocto-autobuilder-helper`` repository. The targets are
-defined in the ‘overrides' section, a quick example could be qemux86-64
-which looks like::
-
- "qemux86-64" : {
- "MACHINE" : "qemux86-64",
- "TEMPLATE" : "arch-qemu",
- "step1" : {
- "extravars" : [
- "IMAGE_FSTYPES_append = ' wic wic.bmap'"
- ]
- }
- },
-
-And to expand that, you need the "arch-qemu" entry from
-the "templates" section, which looks like::
-
- "arch-qemu" : {
- "BUILDINFO" : true,
- "BUILDHISTORY" : true,
- "step1" : {
- "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk",
- "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk"
- },
- "step2" : {
- "SDKMACHINE" : "x86_64",
- "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext",
- "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext"
- },
- "step3" : {
- "BUILDHISTORY" : false,
- "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"],
- "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"]
- }
- },
-
-Combining these two entries you can see that "qemux86-64" is a three step build where the
-``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for
-``MACHINE="qemx86-64"`` but with differing SDKMACHINE settings. In step
-1 an extra variable is added to the ``auto.conf`` file to enable wic
-image generation.
-
-While not every detail of this is covered here, you can see how the
-template mechanism allows quite complex configurations to be built up
-yet allows duplication and repetition to be kept to a minimum.
-
-The different build targets are designed to allow for parallelisation,
-so different machines are usually built in parallel, operations using
-the same machine and metadata are built sequentially, with the aim of
-trying to optimise build efficiency as much as possible.
-
-The ``config.json`` file is processed by the scripts in the Helper
-repository in the ``scripts`` directory. The following section details
-how this works.
-
-Autobuilder Target Execution Overview
-=====================================
-
-For each given target in a build, the Autobuilder executes several
-steps. These are configured in ``yocto-autobuilder2/builders.py`` and
-roughly consist of:
-
-#. *Run clobberdir*.
-
- This cleans out any previous build. Old builds are left around to
- allow easier debugging of failed builds. For additional information,
- see :ref:`test-manual/test-manual-understand-autobuilder:clobberdir`.
-
-#. *Obtain yocto-autobuilder-helper*
-
- This step clones the ``yocto-autobuilder-helper`` git repository.
- This is necessary to prevent the requirement to maintain all the
- release or project-specific code within Buildbot. The branch chosen
- matches the release being built so we can support older releases and
- still make changes in newer ones.
-
-#. *Write layerinfo.json*
-
- This transfers data in the Buildbot UI when the build was configured
- to the Helper.
-
-#. *Call scripts/shared-repo-unpack*
-
- This is a call into the Helper scripts to set up a checkout of all
- the pieces this build might need. It might clone the BitBake
- repository and the OpenEmbedded-Core repository. It may clone the
- Poky repository, as well as additional layers. It will use the data
- from the ``layerinfo.json`` file to help understand the
- configuration. It will also use a local cache of repositories to
- speed up the clone checkouts. For additional information, see
- :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Clone Cache`.
-
- This step has two possible modes of operation. If the build is part
- of a parent build, its possible that all the repositories needed may
- already be available, ready in a pre-prepared directory. An "a-quick"
- or "a-full" build would prepare this before starting the other
- sub-target builds. This is done for two reasons:
-
- - the upstream may change during a build, for example, from a forced
- push and this ensures we have matching content for the whole build
-
- - if 15 Workers all tried to pull the same data from the same repos,
- we can hit resource limits on upstream servers as they can think
- they are under some kind of network attack
-
- This pre-prepared directory is shared among the Workers over NFS. If
- the build is an individual build and there is no "shared" directory
- available, it would clone from the cache and the upstreams as
- necessary. This is considered the fallback mode.
-
-#. *Call scripts/run-config*
-
- This is another call into the Helper scripts where its expected that
- the main functionality of this target will be executed.
-
-Autobuilder Technology
-======================
-
-The Autobuilder has Yocto Project-specific functionality to allow builds
-to operate with increased efficiency and speed.
-
-clobberdir
-----------
-
-When deleting files, the Autobuilder uses ``clobberdir``, which is a
-special script that moves files to a special location, rather than
-deleting them. Files in this location are deleted by an ``rm`` command,
-which is run under ``ionice -c 3``. For example, the deletion only
-happens when there is idle IO capacity on the Worker. The Autobuilder
-Worker Janitor runs this deletion. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`.
-
-Autobuilder Clone Cache
------------------------
-
-Cloning repositories from scratch each time they are required was slow
-on the Autobuilder. We therefore have a stash of commonly used
-repositories pre-cloned on the Workers. Data is fetched from these
-during clones first, then "topped up" with later revisions from any
-upstream when necessary. The cache is maintained by the Autobuilder
-Worker Janitor. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`.
-
-Autobuilder Worker Janitor
---------------------------
-
-This is a process running on each Worker that performs two basic
-operations, including background file deletion at IO idle (see :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and
-maintainenance of a cache of cloned repositories to improve the speed
-the system can checkout repositories.
-
-Shared DL_DIR
--------------
-
-The Workers are all connected over NFS which allows DL_DIR to be shared
-between them. This reduces network accesses from the system and allows
-the build to be sped up. Usage of the directory within the build system
-is designed to be able to be shared over NFS.
-
-Shared SSTATE_DIR
------------------
-
-The Workers are all connected over NFS which allows the ``sstate``
-directory to be shared between them. This means once a Worker has built
-an artifact, all the others can benefit from it. Usage of the directory
-within the directory is designed for sharing over NFS.
-
-Resulttool
-----------
-
-All of the different tests run as part of the build generate output into
-``testresults.json`` files. This allows us to determine which tests ran
-in a given build and their status. Additional information, such as
-failure logs or the time taken to run the tests, may also be included.
-
-Resulttool is part of OpenEmbedded-Core and is used to manipulate these
-json results files. It has the ability to merge files together, display
-reports of the test results and compare different result files.
-
-For details, see :yocto_wiki:`/wiki/Resulttool`.
-
-run-config Target Execution
-===========================
-
-The ``scripts/run-config`` execution is where most of the work within
-the Autobuilder happens. It runs through a number of steps; the first
-are general setup steps that are run once and include:
-
-#. Set up any ``buildtools-tarball`` if configured.
-
-#. Call "buildhistory-init" if buildhistory is configured.
-
-For each step that is configured in ``config.json``, it will perform the
-following:
-
-#. Add any layers that are specified using the
- ``bitbake-layers add-layer`` command (logging as stepXa)
-
-#. Call the ``scripts/setup-config`` script to generate the necessary
- ``auto.conf`` configuration file for the build
-
-#. Run the ``bitbake BBTARGETS`` command (logging as stepXb)
-
-#. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc)
-
-#. Run the ``EXTRACMDS`` command, which are run within the BitBake build
- environment (logging as stepXd)
-
-#. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the
- BitBake build environment (logging as stepXd)
-
-#. Remove any layers added in step
- 1 using the ``bitbake-layers remove-layer`` command (logging as stepXa)
-
-Once the execution steps above complete, ``run-config`` executes a set
-of post-build steps, including:
-
-#. Call ``scripts/publish-artifacts`` to collect any output which is to
- be saved from the build.
-
-#. Call ``scripts/collect-results`` to collect any test results to be
- saved from the build.
-
-#. Call ``scripts/upload-error-reports`` to send any error reports
- generated to the remote server.
-
-#. Cleanup the build directory using
- :ref:`test-manual/test-manual-understand-autobuilder:clobberdir` if the build was successful,
- else rename it to "build-renamed" for potential future debugging.
-
-Deploying Yocto Autobuilder
-===========================
-
-The most up to date information about how to setup and deploy your own
-Autbuilder can be found in README.md in the ``yocto-autobuilder2``
-repository.
-
-We hope that people can use the ``yocto-autobuilder2`` code directly but
-it is inevitable that users will end up needing to heavily customise the
-``yocto-autobuilder-helper`` repository, particularly the
-``config.json`` file as they will want to define their own test matrix.
-
-The Autobuilder supports wo customization options:
-
-- variable substitution
-
-- overlaying configuration files
-
-The standard ``config.json`` minimally attempts to allow substitution of
-the paths. The Helper script repository includes a
-``local-example.json`` file to show how you could override these from a
-separate configuration file. Pass the following into the environment of
-the Autobuilder::
-
- $ ABHELPER_JSON="config.json local-example.json"
-
-As another example, you could also pass the following into the
-environment::
-
- $ ABHELPER_JSON="config.json /some/location/local.json"
-
-One issue users often run into is validation of the ``config.json`` files. A
-tip for minimizing issues from invalid json files is to use a Git
-``pre-commit-hook.sh`` script to verify the JSON file before committing
-it. Create a symbolic link as follows::
-
- $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit