From 79641f25e882b55af6a647cfc0bf6bcc025661a5 Mon Sep 17 00:00:00 2001 From: Brad Bishop Date: Tue, 10 Sep 2019 07:20:22 -0400 Subject: poky: subtree update:c17113f1e2..3b87508a9a MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adrian Bunk (18): help2man: Upgrade 1.47.10 -> 1.47.11 glib-networking: Upgrade 2.60.2 -> 2.60.3 flac: Upgrade 1.3.2 -> 1.3.3 subversion: Upgrade 1.12.0 -> 1.12.2 world-broken.inc: Remove meta/recipes.txt: Stop documenting the long removed recipes-lsb4 python: Remove PYTHONLSBOPTS that hasn't been set for some time systemd: Print a buildtime warning about the status with musl linux-dtb.inc: Remove systemd: Update to the latest 242 branch Remove LSB support Remove LSB support bits libxcrypt: Fix the build with -Os systemd: Update to the latest 242 branch Remove leftovers from the eglinfo removal Update to KERNEL_DEVICETREE variable. ref-manual: Remove documentation for the removed bluez5 distro feature ref-manual: Remove documentation for the removed gnome class Alex Kiernan (1): iputils: Set prefix correctly for usrmerge Alexander Kanavin (23): python3-numpy: update to 1.17.0 images: use 512M of RAM in qemu if 'opengl" is in DISTRO_FEATURES boost: update to 1.71.0 beaglebone: use the default of 256M rather than 512M rpm: resolve a host contamination issue for mono packaging qemu: set default RAM to 256M for all machines mesa: disable gallium swrast driver on x86 x32 ltp: move from python 2.x to python 3.x lsb-release: fix upstream version check libffi: fix upstream version check qemu: switch to '-vga std' emulated hardware from vmware/cirrus for x86/mips ca-certificates: use python 3 for builds libpsl: use python 3 during builds pseudo: use python 3 during builds llvm: use python 3 during builds virglrenderer: update 0.7.0 -> 0.8.0 webkitgtk: use Python 3 for builds webkitgtk: update 2.24.3 -> 2.24.4 default-providers.inc: set mesa as the provider for native(sdk) virtual/libgl libepoxy: enable x11 PACKAGECONFIG for native/nativesdk eglinfo: remove the recipe local.conf.sample: add a commented-out option to enable the gtk UI qemu frontend package.bbclass: allow shell-style wildcards in PRIVATE_LIBS Alistair Francis (2): linux-yocto-dev: Mark qemuriscv64 as compatible qemuriscv: Generate a wic rootFS with a larger filesystem Andre McCurdy (11): bitbake.conf: drop FETCHCMD, GITDIR, etc fetcher over-rides tune-arm1136jf-s.inc: add hardfloat support tune-arm1176jz-s.inc: add support for arm1176jz-s CPU specific tuning tune-ep9312.inc: fix TUNEVALID description packagegroup-core-boot: drop duplicate MACHINE_ESSENTIAL_EXTRA_RDEPENDS, etc rootfs-postcommands.bbclass: add inittab tweak to read_only_rootfs_hook() base-files: nsswitch.conf is glibc specific sqlite3: make CFLAGS consistent across native, nativesdk and target cases sqlite3: explicitly set target endian-ness feature-arm-thumb.inc: fix ARM_THUMB_SUFFIX dropbear: don't create invalid symlinks if ssh, scp, etc are disabled Anuj Mittal (1): libjpeg-turbo: upgrade 2.0.2 -> 2.0.3 Armin Kuster (4): bitbake: tests/fetch: Resolve fetch error in bitbake-selftest gcc-9.2: Security fix for CVE-2019-14250 gcc-8.3: Security fix for CVE-2019-14250 gcc-8.3: remove 8.3 from zeus Bruce Ashfield (11): bluez5: fix build against libc-headers 5.2+ kernel-devsrc: remove python2 dependency perf: change dependencies on python to python3 perf: v5.3+ build fixes devsrc: update arm64 scripts/prepare for kernels 5.3+ linux-libc-headers: update to v5.2 headers linux-yocto: introduce 5.2 recipes qemu: bump linux-yocto preferred version to 5.2 linux-yocto/5.2: update qemuriscv64 SRCREV linux-yocto/5.2: config tweaks and kallsyms fix linux-yocto: drop 5.0 recipes Changhyeok Bae (1): iputils: upgrade to s20190709 Changqing Li (8): libedit: fix failed to get history avahi-ui: support usrmerge bootchart2: support usrmerge systemd-bootchart: support usrmerge bootchart2: switch to add patch from change source in do_install mdadm: skip timeout testcase 11spare-migration dnf.py: installroot support usrmerge man-db: support usrmerge Chen Qi (7): bbtests.py: remove recipe-depends.dot commands.py: fix typo bitbake: cooker.py: remove generation of recipe-depends.dot bind: fix build with python3 PACKAGECONFIG enabled python-testtools: rdepend on doctest module python-numpy: rdepend on multiprocessing module python3: make misc package rdepend on pydoc and pickle modules Christophe PRIOUZEAU (1): common-licenses: update BSD-2-CLAUSE license text Dan Callaghan (1): autotools: always include config_site.d files in CONFIG_SITE Daniel Klauer (1): image.bbclass: Fix debug output for rootfs size Denys Zagorui (1): gzip: add nativesdk support Drew Moseley (1): runqemu: Fix typo in help text. He Zhe (4): ltp: cve/meltdown.c: Fix kernel symbol finding ltp: syscalls: rt_sigwaitinfo01: Fix failure for MIPS arches qemu: Add pkg-config handling for libgcrypt qemu: Fix dependency of PACKAGECONFIG libcurl Jacob Kroon (2): btrfs-tools: Add PACKAGECONFIG knob for controlling python support rm_work: Remove redundant 'after' in addtask statement Jason Wessel (2): serial-getty@.service: Allow device to fast fail if it does not exist Revert "kernel.bbclass: adjust a condition checking" Joshua Watt (9): bitbake: bitbake: respect force flag in runall and runonly oeqa: reproducible: Cleanup reproducible build oeqa: reproducible: Use subTest for packages oeqa: Set LD_LIBRARY_PATH when executing native commands oeqa: reproducible: Record packages in test results oeqa: Enable reproducible build test oeqa: reproducible: Do two clean builds oeqa: reproducible: Preserve sstate mirror for first build resulttool: Add reproducible log extraction Kai Kang (4): avahi: launch avahi-daemon after connman cups: redefine CUPS_SERVERBIN texinfo: fix multilib file confliction sysstat: 12.1.3 -> 12.1.6 Kevin Hao (1): meta-yocto-bsp: Bump to the v5.2 kernel Khem Raj (12): go-1.12: update to 1.12.9 minor release powerpc-tunes: Introduce bigendian tune tune-riscv: Drop littleendian and introduce bigendian tune gcc: Move STANDARD_STARTFILE_PREFIX_1 and STANDARD_STARTFILE_PREFIX_2 back to gcc.c gcc: Fix ldbl-128 support for musl libgpg-error: Fix build with gawk 5.x libgpg-error: Fix build with gawk 5.x musl: Update to latest master gcc-9.2: Fix risc-v dynamic linker relocation python3native, pythonnative: export PYTHON_LIBRARY and PYTHON_INCLUDE_DIR oeqa: Recognise svlogd as another logger python3-manifest.json: Fix typo in summary Mark Asselstine (1): sdkext: use simpler kernel module for devtool test Mark Hatle (1): yocto-check-layer: Allow any case for README file detection Martin Hundebøll (1): initscripts: remove warnings on read-only-rootfs (again) Martin Jansa (3): bitbake: utils: Fix movefile() exception handling with python3 bitbake: fetch2: show warning when renaming the archive with bad checksum failed bitbake: bitbake-user-manual: Improve the example for Removal (Override Style Syntax) Mazliana (1): oeqa/kerneldevelopment: Able to apply a single patch to the Linux kernel source Ming Liu (1): weston: uprev to 7.0.0 Nathan Rossi (30): binutils: Fix mips patch which changes default emulation oeqa/utils/nfs: Add unfs_server function to setup a userspace NFS server scripts/lib/resulttool/report.py: Add more result types gcc-common.inc: Process staging fixme with correct target/native sysroot oeqa/core/tests: Skip test_fail_duplicated_module oeqa/core/tests: Fix test_data module tests Clean up remnants of glibc-initial oe-selftest: Implement console 'keepalive' output dejagnu: Add dejagnu for binutils/gcc test suites binutils-cross-testsuite: Create recipe for test suite execution gcc-runtime: Add do_check task for executing gcc test suites glibc-testsuite: Create a recipe to implement glibc test suite gcc-testsuite.inc: Fix ssh.exp, ensure multilib_flags are populated security_flags.inc: Add overrides for glibc-testsuite to match glibc archiver.bbclass: Do not archive srpm's if there are no package tasks oeqa/core: Implement proper extra result collection and serialization oeqa/core: Rework OETestTag and remove unused OETestFilter oeqa/selftest: Add test run filtering based on test tags oeqa/core/runner.py: Fix OETestTag listing oeqa/selftest/binutils: Create selftest case for binutils test suite oeqa/selftest/gcc: Create selftest case for gcc test suite oeqa/selftest/glibc: Create selftest case for glibc test suite oeqa/selftest/cases/glibc.py: Don't install python3 and pexpect oeqa/selftest: For toolchain tests do not fail if a test failed oeqa/selftest: Toolchain tests suffix "-user" for qemu usermode results oeqa/selftest: Use extraresults on self instead of self.tc oeqa/selftest/context.py: Change -t/-T args to be optional oeqa/core/decorator: Fix super class modifying subclass tags oeqa/selftest/cases/gcc.py: Split into classes for parallelism oeqa/selftest/cases/glibc.py: Rework and tag with toolchain-user/system Neil Armstrong (1): mesa: Add support for the panfrost PACKAGECONFIG Oleksandr Kravchuk (8): libsolv: update to 0.7.6 libevdev: update to 1.8.0 ell: update to 0.22 go-dep: update to 0.5.4 libsecret: 0.19.0 python3-pbr: update to 5.4.3 python3-git: update to 3.0.2 libxcrypt: update to 4.4.8 Peiran Hong (1): diffutils: Added perl to support ptest & Skipped one test case Peter Kjellerstedt (2): libffi: Make it build for MIPS o32 devtool: Avoid failure for recipes with S == WORKDIR and no local files Richard Purdie (11): poky-altcfg: Replace poky-lsb with poky-altcfg ca-certificates: Fix reproducibilty and multilib issue gcc-cross: Clean up fixed-includes systemtap: Use latest and greatest git version for 5.2 kernel support gcc-cross: Fix header file corruption problems poky: Use 5.2 kernel by default binutils/glibc-testsuite: inherit nopackages to fix world builds systemtap: Drop patches merged upstream poky-tiny/poky-altcfg: Use busybox-mdev and systemd oeqa/selftest: Markup 'machine' specific test cases oeqa/selftest: Tweak binutils tests tags as toolchain/user and system Robert Yang (3): ccache: 3.7.1 -> 3.7.3 nfs-utils: 2.3.3 -> 2.4.1 build-compare: 2015.02.10 -> 2019.08.14 Ross Burton (11): libedit: remove FETCHCMD append libnl: upgrade to 3.5.0 libnl: don't patch libnl-3.0.pc systemd: explicitly set the path to nologin core-image-sato: don't use 512MB in qemumips patch: add CVE tags to patches pango: 1.44.6 upgrade pango: add PACKAGECONFIG for libthai support harfbuzz: upgrade to 2.6.1 systemd: ensure reproducible builds by clearly exposing the time epoch support ref-system-requirements: add Debian 10 to supported distribution list Scott Rifenbark (5): ref-manual: Updated the "AS" variable ref-manual: Updates to "Building Your Recipe" dev-manual: Updated "Creating a Team Development Environment" poky-end: Added "pylint3" package to Ubuntu essentials. ref-manual: sync list of image types with source Tim Blechmann (1): icecc: blacklist systemtap Trevor Gamblin (2): lighttpd: drop hard-coded lua version in PACKAGECONFIG libevent: add granularity to ptest log Vijai Kumar K (1): lib/oe/gpg_sign.py: Remove unwanted import Wes Lindauer (1): bitbake: bitbake: cooker: Ensure bbappends are found in stable order Yi Zhao (1): libbsd: upgrade 0.9.1 -> 0.10.0 Yuan Chao (1): man-db: upgrade 2.8.6.1 -> 2.8.7 Zang Ruochen (7): libarchive:upgrade 3.3.3 -> 3.4.0 python3-pip:upgrade 19.2.2 -> 19.2.3 logrotate:upgrade 3.15.0 -> 3.15.1 libogg:upgrade 1.3.3 -> 1.3.4 dtc:upgrade 1.5.0 -> 1.5.1 python3-dbus:upgrade 1.2.8 -> 1.2.10 meson:upgrade 0.51.1 -> 0.51.2 Change-Id: I341e8df65e8196114088a2d6d6eb3a33296188b4 Signed-off-by: Brad Bishop --- poky/scripts/lib/checklayer/cases/common.py | 2 +- poky/scripts/lib/devtool/standard.py | 6 ++-- poky/scripts/lib/resulttool/log.py | 33 ++++++++++++++++++-- poky/scripts/lib/resulttool/report.py | 6 ++-- poky/scripts/lib/scriptutils.py | 43 +++++++++++++++++++++++++-- poky/scripts/lib/wic/canned-wks/qemuriscv.wks | 3 ++ poky/scripts/oe-selftest | 2 +- poky/scripts/runqemu | 12 ++------ 8 files changed, 85 insertions(+), 22 deletions(-) create mode 100644 poky/scripts/lib/wic/canned-wks/qemuriscv.wks (limited to 'poky/scripts') diff --git a/poky/scripts/lib/checklayer/cases/common.py b/poky/scripts/lib/checklayer/cases/common.py index 8ffe028b3..b82304e36 100644 --- a/poky/scripts/lib/checklayer/cases/common.py +++ b/poky/scripts/lib/checklayer/cases/common.py @@ -12,7 +12,7 @@ from checklayer.case import OECheckLayerTestCase class CommonCheckLayer(OECheckLayerTestCase): def test_readme(self): # The top-level README file may have a suffix (like README.rst or README.txt). - readme_files = glob.glob(os.path.join(self.tc.layer['path'], 'README*')) + readme_files = glob.glob(os.path.join(self.tc.layer['path'], '[Rr][Ee][Aa][Dd][Mm][Ee]*')) self.assertTrue(len(readme_files) > 0, msg="Layer doesn't contains README file.") diff --git a/poky/scripts/lib/devtool/standard.py b/poky/scripts/lib/devtool/standard.py index 96af48879..9eeaefb79 100644 --- a/poky/scripts/lib/devtool/standard.py +++ b/poky/scripts/lib/devtool/standard.py @@ -482,9 +482,9 @@ def symlink_oelocal_files_srctree(rd,srctree): addfiles.append(os.path.join(relpth, fn)) if addfiles: bb.process.run('git add %s' % ' '.join(addfiles), cwd=srctree) - useroptions = [] - oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=rd) - bb.process.run('git %s commit -a -m "Committing local file symlinks\n\n%s"' % (' '.join(useroptions), oe.patch.GitApplyTree.ignore_commit_prefix), cwd=srctree) + useroptions = [] + oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=rd) + bb.process.run('git %s commit -m "Committing local file symlinks\n\n%s"' % (' '.join(useroptions), oe.patch.GitApplyTree.ignore_commit_prefix), cwd=srctree) def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, workspace, fixed_setup, d, tinfoil, no_overrides=False): diff --git a/poky/scripts/lib/resulttool/log.py b/poky/scripts/lib/resulttool/log.py index 25c339671..2352c767d 100644 --- a/poky/scripts/lib/resulttool/log.py +++ b/poky/scripts/lib/resulttool/log.py @@ -16,6 +16,16 @@ def show_ptest(result, ptest, logger): print("ptest '%s' not found" % ptest) return 1 +def show_reproducible(result, reproducible, logger): + try: + print(result['reproducible'][reproducible]['diffoscope.text']) + return 0 + + except KeyError: + print("reproducible '%s' not found" % reproducible) + return 1 + + def log(args, logger): results = resultutils.load_resultsdata(args.source) @@ -40,17 +50,28 @@ def log(args, logger): with open(dest, 'w') as f: f.write(ptest['log']) - if args.raw: + if args.raw_ptest: if 'ptestresult.rawlogs' in r: print(r['ptestresult.rawlogs']['log']) else: - print('Raw logs not found') + print('Raw ptest logs not found') + return 1 + + if args.raw_reproducible: + if 'reproducible.rawlogs' in r: + print(r['reproducible.rawlogs']['log']) + else: + print('Raw reproducible logs not found') return 1 for ptest in args.ptest: if not show_ptest(r, ptest, logger): return 1 + for reproducible in args.reproducible: + if not show_reproducible(r, reproducible, logger): + return 1 + def register_commands(subparsers): """Register subcommands from this plugin""" parser = subparsers.add_parser('log', help='show logs', @@ -63,9 +84,15 @@ def register_commands(subparsers): help='show logs for a ptest') parser.add_argument('--dump-ptest', metavar='DIR', help='Dump all ptest log files to the specified directory.') + parser.add_argument('--reproducible', action='append', default=[], + help='show logs for a reproducible test') parser.add_argument('--prepend-run', action='store_true', help='''Dump ptest results to a subdirectory named after the test run when using --dump-ptest. Required if more than one test run is present in the result file''') parser.add_argument('--raw', action='store_true', - help='show raw logs') + help='show raw (ptest) logs. Deprecated. Alias for "--raw-ptest"', dest='raw_ptest') + parser.add_argument('--raw-ptest', action='store_true', + help='show raw ptest log') + parser.add_argument('--raw-reproducible', action='store_true', + help='show raw reproducible build logs') diff --git a/poky/scripts/lib/resulttool/report.py b/poky/scripts/lib/resulttool/report.py index f706280aa..8b03717d2 100644 --- a/poky/scripts/lib/resulttool/report.py +++ b/poky/scripts/lib/resulttool/report.py @@ -19,9 +19,9 @@ class ResultsTextReport(object): self.ptests = {} self.ltptests = {} self.ltpposixtests = {} - self.result_types = {'passed': ['PASSED', 'passed'], - 'failed': ['FAILED', 'failed', 'ERROR', 'error', 'UNKNOWN'], - 'skipped': ['SKIPPED', 'skipped']} + self.result_types = {'passed': ['PASSED', 'passed', 'PASS', 'XFAIL'], + 'failed': ['FAILED', 'failed', 'FAIL', 'ERROR', 'error', 'UNKNOWN', 'XPASS'], + 'skipped': ['SKIPPED', 'skipped', 'UNSUPPORTED', 'UNTESTED', 'UNRESOLVED']} def handle_ptest_result(self, k, status, result, machine): diff --git a/poky/scripts/lib/scriptutils.py b/poky/scripts/lib/scriptutils.py index e7e7021c2..c573dc7f6 100644 --- a/poky/scripts/lib/scriptutils.py +++ b/poky/scripts/lib/scriptutils.py @@ -16,12 +16,51 @@ import string import subprocess import sys import tempfile +import threading import importlib from importlib import machinery -def logger_create(name, stream=None): +class KeepAliveStreamHandler(logging.StreamHandler): + def __init__(self, keepalive=True, **kwargs): + super().__init__(**kwargs) + if keepalive is True: + keepalive = 5000 # default timeout + self._timeout = threading.Condition() + self._stop = False + + # background thread waits on condition, if the condition does not + # happen emit a keep alive message + def thread(): + while not self._stop: + with self._timeout: + if not self._timeout.wait(keepalive): + self.emit(logging.LogRecord("keepalive", logging.INFO, + None, None, "Keepalive message", None, None)) + + self._thread = threading.Thread(target = thread, daemon = True) + self._thread.start() + + def close(self): + # mark the thread to stop and notify it + self._stop = True + with self._timeout: + self._timeout.notify() + # wait for it to join + self._thread.join() + super().close() + + def emit(self, record): + super().emit(record) + # trigger timer reset + with self._timeout: + self._timeout.notify() + +def logger_create(name, stream=None, keepalive=None): logger = logging.getLogger(name) - loggerhandler = logging.StreamHandler(stream=stream) + if keepalive is not None: + loggerhandler = KeepAliveStreamHandler(stream=stream, keepalive=keepalive) + else: + loggerhandler = logging.StreamHandler(stream=stream) loggerhandler.setFormatter(logging.Formatter("%(levelname)s: %(message)s")) logger.addHandler(loggerhandler) logger.setLevel(logging.INFO) diff --git a/poky/scripts/lib/wic/canned-wks/qemuriscv.wks b/poky/scripts/lib/wic/canned-wks/qemuriscv.wks new file mode 100644 index 000000000..12c68b706 --- /dev/null +++ b/poky/scripts/lib/wic/canned-wks/qemuriscv.wks @@ -0,0 +1,3 @@ +# short-description: Create qcow2 image for RISC-V QEMU machines + +part / --source rootfs --fstype=ext4 --label root --align 4096 --size 5G diff --git a/poky/scripts/oe-selftest b/poky/scripts/oe-selftest index 57662b2f7..18ac0f586 100755 --- a/poky/scripts/oe-selftest +++ b/poky/scripts/oe-selftest @@ -33,7 +33,7 @@ scriptpath.add_bitbake_lib_path() from oeqa.utils import load_test_components from oeqa.core.exception import OEQAPreRun -logger = scriptutils.logger_create('oe-selftest', stream=sys.stdout) +logger = scriptutils.logger_create('oe-selftest', stream=sys.stdout, keepalive=True) def main(): description = "Script that runs unit tests against bitbake and other Yocto related tools. The goal is to validate tools functionality and metadata integrity. Refer to https://wiki.yoctoproject.org/wiki/Oe-selftest for more information." diff --git a/poky/scripts/runqemu b/poky/scripts/runqemu index e9b83737c..4415e9b69 100755 --- a/poky/scripts/runqemu +++ b/poky/scripts/runqemu @@ -82,7 +82,7 @@ of the following environment variables (in any order): bootparams= - specify custom kernel parameters during boot help, -h, --help: print this text -d, --debug: Enable debug output - -q, --quite: Hide most output except error messages + -q, --quiet: Hide most output except error messages Examples: runqemu @@ -145,8 +145,6 @@ class BaseConfig(object): # to be added with -drive if=pflash. # Found in the same places as the rootfs, with or without one of # these suffices: qcow2, bin. - # Setting one also adds "-vga std" because that is all that - # OVMF supports. self.ovmf_bios = [] # When enrolling default Secure Boot keys, the hypervisor # must provide the Platform Key and the first Key Exchange Key @@ -755,8 +753,8 @@ class BaseConfig(object): if s: self.set('QB_MEM', '-m %s' % s.group(1)) elif not self.get('QB_MEM'): - logger.info('QB_MEM is not set, use 512M by default') - self.set('QB_MEM', '-m 512') + logger.info('QB_MEM is not set, use 256M by default') + self.set('QB_MEM', '-m 256') # Check and remove M or m suffix qb_mem = self.get('QB_MEM') @@ -1283,10 +1281,6 @@ class BaseConfig(object): for ovmf in self.ovmf_bios: format = ovmf.rsplit('.', 1)[-1] self.qemu_opt += ' -drive if=pflash,format=%s,file=%s' % (format, ovmf) - if self.ovmf_bios: - # OVMF only supports normal VGA, i.e. we need to override a -vga vmware - # that gets added for example for normal qemux86. - self.qemu_opt += ' -vga std' self.qemu_opt += ' ' + self.qemu_opt_script -- cgit v1.2.3