diff options
author | Andrew Geissler <geissonator@yahoo.com> | 2020-12-13 17:44:15 +0300 |
---|---|---|
committer | Andrew Geissler <geissonator@yahoo.com> | 2020-12-15 21:53:47 +0300 |
commit | 09209eec235a35b7089db987561c12e9bd023237 (patch) | |
tree | 2d3580484ffacafe11b72e9abaab50a428dd617d /poky/documentation/test-manual/test-manual-understand-autobuilder.rst | |
parent | f7ba29eda266e04f867e4338b6b8b10c1969419c (diff) | |
download | openbmc-09209eec235a35b7089db987561c12e9bd023237.tar.xz |
poky: subtree update:0ac99625bf..796be0593a
Alexander Kanavin (31):
netbase: upgrade 6.1 -> 6.2
meson: upgrade 0.55.1 -> 0.56.0
vulkan-samples: update to latest revision
libcap: update 2.44 -> 2.45
bind: upgrade 9.16.7 -> 9.16.9
quota: upgrade 4.05 -> 4.06
pango: upgrade 1.46.2 -> 1.48.0
elfutils: upgrade 0.181 -> 0.182
ifupdown: upgrade 0.8.35 -> 0.8.36
createrepo-c: upgrade 0.16.1 -> 0.16.2
acpica: upgrade 20200925 -> 20201113
grep: upgrade 3.5 -> 3.6
man-pages: upgrade 5.08 -> 5.09
stress-ng: upgrade 0.11.23 -> 0.11.24
libhandy: upgrade 1.0.1 -> 1.0.2
piglit: upgrade to latest revision
xkbcomp: upgrade 1.4.3 -> 1.4.4
lz4: upgrade 1.9.2 -> 1.9.3
bison: upgrade 3.7.3 -> 3.7.4
python3-setuptools-scm: fix upstream version check
cantarell-fonts: update 0.0.25 -> 0.201
meta/lib/oe/reproducible.py: gitsm:// works just as fine as git:// for timestamps
llvm: fix reproducibility
ruby: fix reproducibility
webkitgtk: fix reproducibility
ffmpeg: fix reproducibility
piglit: fix reproducibility
serf: do not install the static library
llvm: sort the lists in generated source reproducibibly
kea: fix reproducibility
poky.conf: do not write current date into distro version, use git hash instead
Andrej Valek (1):
kernel-dummy: fix executing unexpected tasks
Anuj Mittal (1):
releases.rst: add gatesgarth to current releases
Brett Warren (1):
libffi: add patch to revert clang VFP workaround
Chandana kalluri (1):
populate_sdk_ext: use SDK_CUSTOM_TEPLATECONF variable to enable custom templateconf.cfg
Changqing Li (1):
buildtools-tarball: add wic dependency into extended buildtools
Diego Sueiro (2):
modutils-initscripts: Fix modules.dep creation when USE_DEPMOD="0"
initscripts: Change execution order between checkroot and modutils
Dmitry Baryshkov (2):
linux-firmware: upgrade 20201022 -> 20201118
linux-firmware: package ath11k firmware
Fabio Berton (1):
mesa: Update 20.2.1 -> 20.2.4
Gratian Crisan (1):
kernel-module-split.bbclass: fix kernel modules getting marked as CONFFILES
Jack Mitchell (3):
Revert "connman: set service to conflict with systemd-networkd"
systemd-conf: add PACKAGECONFIG to enable/disable auto ethernet DHCP
systemd-conf: match ethernet interfaces by type rather than globbing
Joshua Watt (2):
bitbake: hashserv: client: Fix AF_UNIX path length limits
bitbake: hashserv: Fix broken AF_UNIX path length limit
Kai Kang (2):
systemd-systemctl-native: capable to call without argument
systemd.bbclass: update command to check systemctl available
Kevin Hao (1):
tune-octeontx2.inc: Add tune for Marvell OCTEON TX2 core
Li Wang (2):
qemu: CVE-2020-29129 CVE-2020-29130
qemu: CVE-2020-25624
Luca Boccassi (1):
dbus: move messagebus user to dbus-common package
Michael Halstead (1):
releases: conf: add link to 3.1.4, update to include 3.1.4
Nicolas Dechesne (19):
sphinx: add .vscode in .gitignore
{dev,kernel,sdk}-manual: replace hardcoded release version with &DISTRO;
sphinx: replace bitbake labels with references to corresponding title
brief-yoctoprojectqs: replace labels with references to section title
dev-manual: replace labels with references to section title
ref-manual: replace labels with references to section title
sdk-manual: replace labels with references to section title
overview-manual: remove unused labels
dev-manual: remove unused labels
sphinx: rename top level document in each manual
sphinx: use absolute paths for :doc: references
test-manual: remove 'test-manual' from filenames
toaster-manual: remove 'toaster-manual' from filenames
dev-manual: remove 'dev-manual' from filenames
kernel-dev: remove 'kernel-dev' from filenames
profile-manual: remove 'profile-manual' from filenames
overview-manual: remove 'overview-manual' from filenames
sdk-manual: remove 'sdk' from filenames
ref-manual: remove 'ref' from filenames
Paul Barker (5):
documentation: Simplify yocto_wiki links
documentation: Simplify yocto_git links
ref-manual: Simplify oe_git links
poky.conf: Add opensuseleap-15.2 and fedora-33 to tested distros
poky.conf: Drop fedora-30 from tested distros
Peter Kjellerstedt (2):
pseudo: Simplify pseudo_client_ignore_path_chroot()
bitbake.conf: Add all layers (from BBLAYERS) to PSEUDO_IGNORE_PATHS
Richard Purdie (8):
lz4: Use the new branch naming from upstream
Revert "bitbake.conf: Add all layers (from BBLAYERS) to PSEUDO_IGNORE_PATHS"
build-appliance-image: Update to master head revision
bitbake: Revert "fetch2: use relative symlinks for anything pulled from PREMIRRORS"
build-appliance-image: Update to master head revision
metadata_scm: Fix signature handling of METADATA_REVISION and METADATA_BRANCH
poky: Set SDK_VERSION explicitly
build-appliance-image: Update to master head revision
Ross Burton (9):
oeqa/devtool: use Yocto mirror for pv-1.5.3 tarball
image_types: remove obsolete tar comment
image_types: sort tarball file listings
package_manager/ipk: neaten OPKGLIBDIR logic
ldconfig-native: don't write auxiliary cache
package_manager/ipk: improve remove_packaging_data
oeqa/selftest/containerimage: update for improved cleanup
coreutils: add SUSE-specific issues to CVE whitelist
bitbake: msg: use safe YAML loader
Sinan Kaya (1):
poky-tiny: enable section removal
Tomasz Dziendzielski (1):
pseudo: Update to print PSEUDO_LOGFILE in abort message on path mismatches
sangeeta jain (1):
meta/lib/oeqa/manual/oe-core.json: Update test_bitbake_devshell
zangrc (3):
libinput: upgrade 1.16.3 -> 1.16.4
lighttpd: upgrade 1.4.55 -> 1.4.56
sysstat: upgrade 12.4.0 -> 12.4.1
Signed-off-by: Andrew Geissler <geissonator@yahoo.com>
Change-Id: I65f2f1c9d44433f3e62609240012c42256679b51
Diffstat (limited to 'poky/documentation/test-manual/test-manual-understand-autobuilder.rst')
-rw-r--r-- | poky/documentation/test-manual/test-manual-understand-autobuilder.rst | 285 |
1 files changed, 0 insertions, 285 deletions
diff --git a/poky/documentation/test-manual/test-manual-understand-autobuilder.rst b/poky/documentation/test-manual/test-manual-understand-autobuilder.rst deleted file mode 100644 index 698a266ee..000000000 --- a/poky/documentation/test-manual/test-manual-understand-autobuilder.rst +++ /dev/null @@ -1,285 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-SA-2.0-UK - -******************************************* -Understanding the Yocto Project Autobuilder -******************************************* - -Execution Flow within the Autobuilder -===================================== - -The "a-full" and "a-quick" targets are the usual entry points into the -Autobuilder and it makes sense to follow the process through the system -starting there. This is best visualised from the Autobuilder Console -view (:yocto_ab:`/typhoon/#/console`). - -Each item along the top of that view represents some "target build" and -these targets are all run in parallel. The 'full' build will trigger the -majority of them, the "quick" build will trigger some subset of them. -The Autobuilder effectively runs whichever configuration is defined for -each of those targets on a seperate buildbot worker. To understand the -configuration, you need to look at the entry on ``config.json`` file -within the ``yocto-autobuilder-helper`` repository. The targets are -defined in the ‘overrides' section, a quick example could be qemux86-64 -which looks like:: - - "qemux86-64" : { - "MACHINE" : "qemux86-64", - "TEMPLATE" : "arch-qemu", - "step1" : { - "extravars" : [ - "IMAGE_FSTYPES_append = ' wic wic.bmap'" - ] - } - }, - -And to expand that, you need the "arch-qemu" entry from -the "templates" section, which looks like:: - - "arch-qemu" : { - "BUILDINFO" : true, - "BUILDHISTORY" : true, - "step1" : { - "BBTARGETS" : "core-image-sato core-image-sato-dev core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-sato:do_populate_sdk", - "SANITYTARGETS" : "core-image-minimal:do_testimage core-image-sato:do_testimage core-image-sato-sdk:do_testimage core-image-sato:do_testsdk" - }, - "step2" : { - "SDKMACHINE" : "x86_64", - "BBTARGETS" : "core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext", - "SANITYTARGETS" : "core-image-sato:do_testsdk core-image-minimal:do_testsdkext core-image-sato:do_testsdkext" - }, - "step3" : { - "BUILDHISTORY" : false, - "EXTRACMDS" : ["${SCRIPTSDIR}/checkvnc; DISPLAY=:1 oe-selftest ${HELPERSTMACHTARGS} -j 15"], - "ADDLAYER" : ["${BUILDDIR}/../meta-selftest"] - } - }, - -Combining these two entries you can see that "qemux86-64" is a three step build where the -``bitbake BBTARGETS`` would be run, then ``bitbake SANITYTARGETS`` for each step; all for -``MACHINE="qemx86-64"`` but with differing SDKMACHINE settings. In step -1 an extra variable is added to the ``auto.conf`` file to enable wic -image generation. - -While not every detail of this is covered here, you can see how the -template mechanism allows quite complex configurations to be built up -yet allows duplication and repetition to be kept to a minimum. - -The different build targets are designed to allow for parallelisation, -so different machines are usually built in parallel, operations using -the same machine and metadata are built sequentially, with the aim of -trying to optimise build efficiency as much as possible. - -The ``config.json`` file is processed by the scripts in the Helper -repository in the ``scripts`` directory. The following section details -how this works. - -Autobuilder Target Execution Overview -===================================== - -For each given target in a build, the Autobuilder executes several -steps. These are configured in ``yocto-autobuilder2/builders.py`` and -roughly consist of: - -#. *Run clobberdir*. - - This cleans out any previous build. Old builds are left around to - allow easier debugging of failed builds. For additional information, - see :ref:`test-manual/test-manual-understand-autobuilder:clobberdir`. - -#. *Obtain yocto-autobuilder-helper* - - This step clones the ``yocto-autobuilder-helper`` git repository. - This is necessary to prevent the requirement to maintain all the - release or project-specific code within Buildbot. The branch chosen - matches the release being built so we can support older releases and - still make changes in newer ones. - -#. *Write layerinfo.json* - - This transfers data in the Buildbot UI when the build was configured - to the Helper. - -#. *Call scripts/shared-repo-unpack* - - This is a call into the Helper scripts to set up a checkout of all - the pieces this build might need. It might clone the BitBake - repository and the OpenEmbedded-Core repository. It may clone the - Poky repository, as well as additional layers. It will use the data - from the ``layerinfo.json`` file to help understand the - configuration. It will also use a local cache of repositories to - speed up the clone checkouts. For additional information, see - :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Clone Cache`. - - This step has two possible modes of operation. If the build is part - of a parent build, its possible that all the repositories needed may - already be available, ready in a pre-prepared directory. An "a-quick" - or "a-full" build would prepare this before starting the other - sub-target builds. This is done for two reasons: - - - the upstream may change during a build, for example, from a forced - push and this ensures we have matching content for the whole build - - - if 15 Workers all tried to pull the same data from the same repos, - we can hit resource limits on upstream servers as they can think - they are under some kind of network attack - - This pre-prepared directory is shared among the Workers over NFS. If - the build is an individual build and there is no "shared" directory - available, it would clone from the cache and the upstreams as - necessary. This is considered the fallback mode. - -#. *Call scripts/run-config* - - This is another call into the Helper scripts where its expected that - the main functionality of this target will be executed. - -Autobuilder Technology -====================== - -The Autobuilder has Yocto Project-specific functionality to allow builds -to operate with increased efficiency and speed. - -clobberdir ----------- - -When deleting files, the Autobuilder uses ``clobberdir``, which is a -special script that moves files to a special location, rather than -deleting them. Files in this location are deleted by an ``rm`` command, -which is run under ``ionice -c 3``. For example, the deletion only -happens when there is idle IO capacity on the Worker. The Autobuilder -Worker Janitor runs this deletion. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`. - -Autobuilder Clone Cache ------------------------ - -Cloning repositories from scratch each time they are required was slow -on the Autobuilder. We therefore have a stash of commonly used -repositories pre-cloned on the Workers. Data is fetched from these -during clones first, then "topped up" with later revisions from any -upstream when necessary. The cache is maintained by the Autobuilder -Worker Janitor. See :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Worker Janitor`. - -Autobuilder Worker Janitor --------------------------- - -This is a process running on each Worker that performs two basic -operations, including background file deletion at IO idle (see :ref:`test-manual/test-manual-understand-autobuilder:Autobuilder Target Execution Overview`: Run clobberdir) and -maintainenance of a cache of cloned repositories to improve the speed -the system can checkout repositories. - -Shared DL_DIR -------------- - -The Workers are all connected over NFS which allows DL_DIR to be shared -between them. This reduces network accesses from the system and allows -the build to be sped up. Usage of the directory within the build system -is designed to be able to be shared over NFS. - -Shared SSTATE_DIR ------------------ - -The Workers are all connected over NFS which allows the ``sstate`` -directory to be shared between them. This means once a Worker has built -an artifact, all the others can benefit from it. Usage of the directory -within the directory is designed for sharing over NFS. - -Resulttool ----------- - -All of the different tests run as part of the build generate output into -``testresults.json`` files. This allows us to determine which tests ran -in a given build and their status. Additional information, such as -failure logs or the time taken to run the tests, may also be included. - -Resulttool is part of OpenEmbedded-Core and is used to manipulate these -json results files. It has the ability to merge files together, display -reports of the test results and compare different result files. - -For details, see :yocto_wiki:`/wiki/Resulttool`. - -run-config Target Execution -=========================== - -The ``scripts/run-config`` execution is where most of the work within -the Autobuilder happens. It runs through a number of steps; the first -are general setup steps that are run once and include: - -#. Set up any ``buildtools-tarball`` if configured. - -#. Call "buildhistory-init" if buildhistory is configured. - -For each step that is configured in ``config.json``, it will perform the -following: - -#. Add any layers that are specified using the - ``bitbake-layers add-layer`` command (logging as stepXa) - -#. Call the ``scripts/setup-config`` script to generate the necessary - ``auto.conf`` configuration file for the build - -#. Run the ``bitbake BBTARGETS`` command (logging as stepXb) - -#. Run the ``bitbake SANITYTARGETS`` command (logging as stepXc) - -#. Run the ``EXTRACMDS`` command, which are run within the BitBake build - environment (logging as stepXd) - -#. Run the ``EXTRAPLAINCMDS`` command(s), which are run outside the - BitBake build environment (logging as stepXd) - -#. Remove any layers added in step - 1 using the ``bitbake-layers remove-layer`` command (logging as stepXa) - -Once the execution steps above complete, ``run-config`` executes a set -of post-build steps, including: - -#. Call ``scripts/publish-artifacts`` to collect any output which is to - be saved from the build. - -#. Call ``scripts/collect-results`` to collect any test results to be - saved from the build. - -#. Call ``scripts/upload-error-reports`` to send any error reports - generated to the remote server. - -#. Cleanup the build directory using - :ref:`test-manual/test-manual-understand-autobuilder:clobberdir` if the build was successful, - else rename it to "build-renamed" for potential future debugging. - -Deploying Yocto Autobuilder -=========================== - -The most up to date information about how to setup and deploy your own -Autbuilder can be found in README.md in the ``yocto-autobuilder2`` -repository. - -We hope that people can use the ``yocto-autobuilder2`` code directly but -it is inevitable that users will end up needing to heavily customise the -``yocto-autobuilder-helper`` repository, particularly the -``config.json`` file as they will want to define their own test matrix. - -The Autobuilder supports wo customization options: - -- variable substitution - -- overlaying configuration files - -The standard ``config.json`` minimally attempts to allow substitution of -the paths. The Helper script repository includes a -``local-example.json`` file to show how you could override these from a -separate configuration file. Pass the following into the environment of -the Autobuilder:: - - $ ABHELPER_JSON="config.json local-example.json" - -As another example, you could also pass the following into the -environment:: - - $ ABHELPER_JSON="config.json /some/location/local.json" - -One issue users often run into is validation of the ``config.json`` files. A -tip for minimizing issues from invalid json files is to use a Git -``pre-commit-hook.sh`` script to verify the JSON file before committing -it. Create a symbolic link as follows:: - - $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit |