diff options
author | Brad Bishop <bradleyb@fuzziesquirrel.com> | 2019-09-10 14:20:22 +0300 |
---|---|---|
committer | Brad Bishop <bradleyb@fuzziesquirrel.com> | 2019-09-10 14:21:34 +0300 |
commit | 79641f25e882b55af6a647cfc0bf6bcc025661a5 (patch) | |
tree | 2b5fb3c59e5b6e85fd6fb8a94b6e15989d974336 /poky/meta/lib/oeqa/core | |
parent | 39ae1a1dce50fde698688fa5ac8eb737b302f0fd (diff) | |
download | openbmc-79641f25e882b55af6a647cfc0bf6bcc025661a5.tar.xz |
poky: subtree update:c17113f1e2..3b87508a9a
Adrian Bunk (18):
help2man: Upgrade 1.47.10 -> 1.47.11
glib-networking: Upgrade 2.60.2 -> 2.60.3
flac: Upgrade 1.3.2 -> 1.3.3
subversion: Upgrade 1.12.0 -> 1.12.2
world-broken.inc: Remove
meta/recipes.txt: Stop documenting the long removed recipes-lsb4
python: Remove PYTHONLSBOPTS that hasn't been set for some time
systemd: Print a buildtime warning about the status with musl
linux-dtb.inc: Remove
systemd: Update to the latest 242 branch
Remove LSB support
Remove LSB support bits
libxcrypt: Fix the build with -Os
systemd: Update to the latest 242 branch
Remove leftovers from the eglinfo removal
Update to KERNEL_DEVICETREE variable.
ref-manual: Remove documentation for the removed bluez5 distro feature
ref-manual: Remove documentation for the removed gnome class
Alex Kiernan (1):
iputils: Set prefix correctly for usrmerge
Alexander Kanavin (23):
python3-numpy: update to 1.17.0
images: use 512M of RAM in qemu if 'opengl" is in DISTRO_FEATURES
boost: update to 1.71.0
beaglebone: use the default of 256M rather than 512M
rpm: resolve a host contamination issue for mono packaging
qemu: set default RAM to 256M for all machines
mesa: disable gallium swrast driver on x86 x32
ltp: move from python 2.x to python 3.x
lsb-release: fix upstream version check
libffi: fix upstream version check
qemu: switch to '-vga std' emulated hardware from vmware/cirrus for x86/mips
ca-certificates: use python 3 for builds
libpsl: use python 3 during builds
pseudo: use python 3 during builds
llvm: use python 3 during builds
virglrenderer: update 0.7.0 -> 0.8.0
webkitgtk: use Python 3 for builds
webkitgtk: update 2.24.3 -> 2.24.4
default-providers.inc: set mesa as the provider for native(sdk) virtual/libgl
libepoxy: enable x11 PACKAGECONFIG for native/nativesdk
eglinfo: remove the recipe
local.conf.sample: add a commented-out option to enable the gtk UI qemu frontend
package.bbclass: allow shell-style wildcards in PRIVATE_LIBS
Alistair Francis (2):
linux-yocto-dev: Mark qemuriscv64 as compatible
qemuriscv: Generate a wic rootFS with a larger filesystem
Andre McCurdy (11):
bitbake.conf: drop FETCHCMD, GITDIR, etc fetcher over-rides
tune-arm1136jf-s.inc: add hardfloat support
tune-arm1176jz-s.inc: add support for arm1176jz-s CPU specific tuning
tune-ep9312.inc: fix TUNEVALID description
packagegroup-core-boot: drop duplicate MACHINE_ESSENTIAL_EXTRA_RDEPENDS, etc
rootfs-postcommands.bbclass: add inittab tweak to read_only_rootfs_hook()
base-files: nsswitch.conf is glibc specific
sqlite3: make CFLAGS consistent across native, nativesdk and target cases
sqlite3: explicitly set target endian-ness
feature-arm-thumb.inc: fix ARM_THUMB_SUFFIX
dropbear: don't create invalid symlinks if ssh, scp, etc are disabled
Anuj Mittal (1):
libjpeg-turbo: upgrade 2.0.2 -> 2.0.3
Armin Kuster (4):
bitbake: tests/fetch: Resolve fetch error in bitbake-selftest
gcc-9.2: Security fix for CVE-2019-14250
gcc-8.3: Security fix for CVE-2019-14250
gcc-8.3: remove 8.3 from zeus
Bruce Ashfield (11):
bluez5: fix build against libc-headers 5.2+
kernel-devsrc: remove python2 dependency
perf: change dependencies on python to python3
perf: v5.3+ build fixes
devsrc: update arm64 scripts/prepare for kernels 5.3+
linux-libc-headers: update to v5.2 headers
linux-yocto: introduce 5.2 recipes
qemu: bump linux-yocto preferred version to 5.2
linux-yocto/5.2: update qemuriscv64 SRCREV
linux-yocto/5.2: config tweaks and kallsyms fix
linux-yocto: drop 5.0 recipes
Changhyeok Bae (1):
iputils: upgrade to s20190709
Changqing Li (8):
libedit: fix failed to get history
avahi-ui: support usrmerge
bootchart2: support usrmerge
systemd-bootchart: support usrmerge
bootchart2: switch to add patch from change source in do_install
mdadm: skip timeout testcase 11spare-migration
dnf.py: installroot support usrmerge
man-db: support usrmerge
Chen Qi (7):
bbtests.py: remove recipe-depends.dot
commands.py: fix typo
bitbake: cooker.py: remove generation of recipe-depends.dot
bind: fix build with python3 PACKAGECONFIG enabled
python-testtools: rdepend on doctest module
python-numpy: rdepend on multiprocessing module
python3: make misc package rdepend on pydoc and pickle modules
Christophe PRIOUZEAU (1):
common-licenses: update BSD-2-CLAUSE license text
Dan Callaghan (1):
autotools: always include config_site.d files in CONFIG_SITE
Daniel Klauer (1):
image.bbclass: Fix debug output for rootfs size
Denys Zagorui (1):
gzip: add nativesdk support
Drew Moseley (1):
runqemu: Fix typo in help text.
He Zhe (4):
ltp: cve/meltdown.c: Fix kernel symbol finding
ltp: syscalls: rt_sigwaitinfo01: Fix failure for MIPS arches
qemu: Add pkg-config handling for libgcrypt
qemu: Fix dependency of PACKAGECONFIG libcurl
Jacob Kroon (2):
btrfs-tools: Add PACKAGECONFIG knob for controlling python support
rm_work: Remove redundant 'after' in addtask statement
Jason Wessel (2):
serial-getty@.service: Allow device to fast fail if it does not exist
Revert "kernel.bbclass: adjust a condition checking"
Joshua Watt (9):
bitbake: bitbake: respect force flag in runall and runonly
oeqa: reproducible: Cleanup reproducible build
oeqa: reproducible: Use subTest for packages
oeqa: Set LD_LIBRARY_PATH when executing native commands
oeqa: reproducible: Record packages in test results
oeqa: Enable reproducible build test
oeqa: reproducible: Do two clean builds
oeqa: reproducible: Preserve sstate mirror for first build
resulttool: Add reproducible log extraction
Kai Kang (4):
avahi: launch avahi-daemon after connman
cups: redefine CUPS_SERVERBIN
texinfo: fix multilib file confliction
sysstat: 12.1.3 -> 12.1.6
Kevin Hao (1):
meta-yocto-bsp: Bump to the v5.2 kernel
Khem Raj (12):
go-1.12: update to 1.12.9 minor release
powerpc-tunes: Introduce bigendian tune
tune-riscv: Drop littleendian and introduce bigendian tune
gcc: Move STANDARD_STARTFILE_PREFIX_1 and STANDARD_STARTFILE_PREFIX_2 back to gcc.c
gcc: Fix ldbl-128 support for musl
libgpg-error: Fix build with gawk 5.x
libgpg-error: Fix build with gawk 5.x
musl: Update to latest master
gcc-9.2: Fix risc-v dynamic linker relocation
python3native, pythonnative: export PYTHON_LIBRARY and PYTHON_INCLUDE_DIR
oeqa: Recognise svlogd as another logger
python3-manifest.json: Fix typo in summary
Mark Asselstine (1):
sdkext: use simpler kernel module for devtool test
Mark Hatle (1):
yocto-check-layer: Allow any case for README file detection
Martin Hundebøll (1):
initscripts: remove warnings on read-only-rootfs (again)
Martin Jansa (3):
bitbake: utils: Fix movefile() exception handling with python3
bitbake: fetch2: show warning when renaming the archive with bad checksum failed
bitbake: bitbake-user-manual: Improve the example for Removal (Override Style Syntax)
Mazliana (1):
oeqa/kerneldevelopment: Able to apply a single patch to the Linux kernel source
Ming Liu (1):
weston: uprev to 7.0.0
Nathan Rossi (30):
binutils: Fix mips patch which changes default emulation
oeqa/utils/nfs: Add unfs_server function to setup a userspace NFS server
scripts/lib/resulttool/report.py: Add more result types
gcc-common.inc: Process staging fixme with correct target/native sysroot
oeqa/core/tests: Skip test_fail_duplicated_module
oeqa/core/tests: Fix test_data module tests
Clean up remnants of glibc-initial
oe-selftest: Implement console 'keepalive' output
dejagnu: Add dejagnu for binutils/gcc test suites
binutils-cross-testsuite: Create recipe for test suite execution
gcc-runtime: Add do_check task for executing gcc test suites
glibc-testsuite: Create a recipe to implement glibc test suite
gcc-testsuite.inc: Fix ssh.exp, ensure multilib_flags are populated
security_flags.inc: Add overrides for glibc-testsuite to match glibc
archiver.bbclass: Do not archive srpm's if there are no package tasks
oeqa/core: Implement proper extra result collection and serialization
oeqa/core: Rework OETestTag and remove unused OETestFilter
oeqa/selftest: Add test run filtering based on test tags
oeqa/core/runner.py: Fix OETestTag listing
oeqa/selftest/binutils: Create selftest case for binutils test suite
oeqa/selftest/gcc: Create selftest case for gcc test suite
oeqa/selftest/glibc: Create selftest case for glibc test suite
oeqa/selftest/cases/glibc.py: Don't install python3 and pexpect
oeqa/selftest: For toolchain tests do not fail if a test failed
oeqa/selftest: Toolchain tests suffix "-user" for qemu usermode results
oeqa/selftest: Use extraresults on self instead of self.tc
oeqa/selftest/context.py: Change -t/-T args to be optional
oeqa/core/decorator: Fix super class modifying subclass tags
oeqa/selftest/cases/gcc.py: Split into classes for parallelism
oeqa/selftest/cases/glibc.py: Rework and tag with toolchain-user/system
Neil Armstrong (1):
mesa: Add support for the panfrost PACKAGECONFIG
Oleksandr Kravchuk (8):
libsolv: update to 0.7.6
libevdev: update to 1.8.0
ell: update to 0.22
go-dep: update to 0.5.4
libsecret: 0.19.0
python3-pbr: update to 5.4.3
python3-git: update to 3.0.2
libxcrypt: update to 4.4.8
Peiran Hong (1):
diffutils: Added perl to support ptest & Skipped one test case
Peter Kjellerstedt (2):
libffi: Make it build for MIPS o32
devtool: Avoid failure for recipes with S == WORKDIR and no local files
Richard Purdie (11):
poky-altcfg: Replace poky-lsb with poky-altcfg
ca-certificates: Fix reproducibilty and multilib issue
gcc-cross: Clean up fixed-includes
systemtap: Use latest and greatest git version for 5.2 kernel support
gcc-cross: Fix header file corruption problems
poky: Use 5.2 kernel by default
binutils/glibc-testsuite: inherit nopackages to fix world builds
systemtap: Drop patches merged upstream
poky-tiny/poky-altcfg: Use busybox-mdev and systemd
oeqa/selftest: Markup 'machine' specific test cases
oeqa/selftest: Tweak binutils tests tags as toolchain/user and system
Robert Yang (3):
ccache: 3.7.1 -> 3.7.3
nfs-utils: 2.3.3 -> 2.4.1
build-compare: 2015.02.10 -> 2019.08.14
Ross Burton (11):
libedit: remove FETCHCMD append
libnl: upgrade to 3.5.0
libnl: don't patch libnl-3.0.pc
systemd: explicitly set the path to nologin
core-image-sato: don't use 512MB in qemumips
patch: add CVE tags to patches
pango: 1.44.6 upgrade
pango: add PACKAGECONFIG for libthai support
harfbuzz: upgrade to 2.6.1
systemd: ensure reproducible builds by clearly exposing the time epoch support
ref-system-requirements: add Debian 10 to supported distribution list
Scott Rifenbark (5):
ref-manual: Updated the "AS" variable
ref-manual: Updates to "Building Your Recipe"
dev-manual: Updated "Creating a Team Development Environment"
poky-end: Added "pylint3" package to Ubuntu essentials.
ref-manual: sync list of image types with source
Tim Blechmann (1):
icecc: blacklist systemtap
Trevor Gamblin (2):
lighttpd: drop hard-coded lua version in PACKAGECONFIG
libevent: add granularity to ptest log
Vijai Kumar K (1):
lib/oe/gpg_sign.py: Remove unwanted import
Wes Lindauer (1):
bitbake: bitbake: cooker: Ensure bbappends are found in stable order
Yi Zhao (1):
libbsd: upgrade 0.9.1 -> 0.10.0
Yuan Chao (1):
man-db: upgrade 2.8.6.1 -> 2.8.7
Zang Ruochen (7):
libarchive:upgrade 3.3.3 -> 3.4.0
python3-pip:upgrade 19.2.2 -> 19.2.3
logrotate:upgrade 3.15.0 -> 3.15.1
libogg:upgrade 1.3.3 -> 1.3.4
dtc:upgrade 1.5.0 -> 1.5.1
python3-dbus:upgrade 1.2.8 -> 1.2.10
meson:upgrade 0.51.1 -> 0.51.2
Change-Id: I341e8df65e8196114088a2d6d6eb3a33296188b4
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>
Diffstat (limited to 'poky/meta/lib/oeqa/core')
-rw-r--r-- | poky/meta/lib/oeqa/core/context.py | 4 | ||||
-rw-r--r-- | poky/meta/lib/oeqa/core/decorator/__init__.py | 21 | ||||
-rw-r--r-- | poky/meta/lib/oeqa/core/decorator/oetag.py | 27 | ||||
-rw-r--r-- | poky/meta/lib/oeqa/core/loader.py | 61 | ||||
-rw-r--r-- | poky/meta/lib/oeqa/core/runner.py | 60 | ||||
-rw-r--r-- | poky/meta/lib/oeqa/core/tests/cases/data.py | 2 | ||||
-rw-r--r-- | poky/meta/lib/oeqa/core/tests/cases/oetag.py | 21 | ||||
-rw-r--r-- | poky/meta/lib/oeqa/core/tests/common.py | 4 | ||||
-rwxr-xr-x | poky/meta/lib/oeqa/core/tests/test_data.py | 10 | ||||
-rwxr-xr-x | poky/meta/lib/oeqa/core/tests/test_decorators.py | 77 | ||||
-rwxr-xr-x | poky/meta/lib/oeqa/core/tests/test_loader.py | 26 | ||||
-rw-r--r-- | poky/meta/lib/oeqa/core/utils/concurrencytest.py | 61 |
12 files changed, 215 insertions, 159 deletions
diff --git a/poky/meta/lib/oeqa/core/context.py b/poky/meta/lib/oeqa/core/context.py index 68819cc33..14fc6a54f 100644 --- a/poky/meta/lib/oeqa/core/context.py +++ b/poky/meta/lib/oeqa/core/context.py @@ -64,12 +64,12 @@ class OETestContext(object): setattr(tclass, 'setUpHooker', skipfuncgen('Skip by the command line argument "%s"' % skip)) def loadTests(self, module_paths, modules=[], tests=[], - modules_manifest="", modules_required=[], filters={}): + modules_manifest="", modules_required=[], **kwargs): if modules_manifest: modules = self._read_modules_from_manifest(modules_manifest) self.loader = self.loaderClass(self, module_paths, modules, tests, - modules_required, filters) + modules_required, **kwargs) self.suites = self.loader.discover() def runTests(self, processes=None, skips=[]): diff --git a/poky/meta/lib/oeqa/core/decorator/__init__.py b/poky/meta/lib/oeqa/core/decorator/__init__.py index 923b21826..1a82518ab 100644 --- a/poky/meta/lib/oeqa/core/decorator/__init__.py +++ b/poky/meta/lib/oeqa/core/decorator/__init__.py @@ -6,6 +6,7 @@ from functools import wraps from abc import abstractmethod, ABCMeta +from oeqa.core.utils.misc import strToList decoratorClasses = set() @@ -63,12 +64,16 @@ class OETestDiscover(OETestDecorator): def discover(registry): return registry['cases'] -class OETestFilter(OETestDecorator): +def OETestTag(*tags): + expandedtags = [] + for tag in tags: + expandedtags += strToList(tag) + def decorator(item): + if hasattr(item, "__oeqa_testtags"): + # do not append, create a new list (to handle classes with inheritance) + item.__oeqa_testtags = list(item.__oeqa_testtags) + expandedtags + else: + item.__oeqa_testtags = expandedtags + return item + return decorator - # OETestLoader call it while loading the tests - # in loadTestsFromTestCase method, it needs to - # return a bool, True if needs to be filtered. - # This method must consume the filter used. - @abstractmethod - def filtrate(self, filters): - return False diff --git a/poky/meta/lib/oeqa/core/decorator/oetag.py b/poky/meta/lib/oeqa/core/decorator/oetag.py deleted file mode 100644 index 8c31138da..000000000 --- a/poky/meta/lib/oeqa/core/decorator/oetag.py +++ /dev/null @@ -1,27 +0,0 @@ -# -# Copyright (C) 2016 Intel Corporation -# -# SPDX-License-Identifier: MIT -# - -from . import OETestFilter, registerDecorator -from oeqa.core.utils.misc import strToList - -def _tagFilter(tags, filters): - return False if set(tags) & set(filters) else True - -@registerDecorator -class OETestTag(OETestFilter): - attrs = ('oetag',) - - def bind(self, registry, case): - super(OETestTag, self).bind(registry, case) - self.oetag = strToList(self.oetag, 'oetag') - - def filtrate(self, filters): - if filters.get('oetag'): - filterx = strToList(filters['oetag'], 'oetag') - del filters['oetag'] - if _tagFilter(self.oetag, filterx): - return True - return False diff --git a/poky/meta/lib/oeqa/core/loader.py b/poky/meta/lib/oeqa/core/loader.py index 7fea0585c..0d7970d49 100644 --- a/poky/meta/lib/oeqa/core/loader.py +++ b/poky/meta/lib/oeqa/core/loader.py @@ -16,7 +16,7 @@ from oeqa.core.utils.test import getSuiteModules, getCaseID from oeqa.core.exception import OEQATestNotFound from oeqa.core.case import OETestCase from oeqa.core.decorator import decoratorClasses, OETestDecorator, \ - OETestFilter, OETestDiscover + OETestDiscover # When loading tests, the unittest framework stores any exceptions and # displays them only when the run method is called. @@ -68,7 +68,7 @@ class OETestLoader(unittest.TestLoader): '_top_level_dir'] def __init__(self, tc, module_paths, modules, tests, modules_required, - filters, *args, **kwargs): + *args, **kwargs): self.tc = tc self.modules = _built_modules_dict(modules) @@ -76,13 +76,7 @@ class OETestLoader(unittest.TestLoader): self.tests = tests self.modules_required = modules_required - self.filters = filters - self.decorator_filters = [d for d in decoratorClasses if \ - issubclass(d, OETestFilter)] - self._validateFilters(self.filters, self.decorator_filters) - self.used_filters = [d for d in self.decorator_filters - for f in self.filters - if f in d.attrs] + self.tags_filter = kwargs.get("tags_filter", None) if isinstance(module_paths, str): module_paths = [module_paths] @@ -104,28 +98,6 @@ class OETestLoader(unittest.TestLoader): setattr(testCaseClass, 'td', self.tc.td) setattr(testCaseClass, 'logger', self.tc.logger) - def _validateFilters(self, filters, decorator_filters): - # Validate if filter isn't empty - for key,value in filters.items(): - if not value: - raise TypeError("Filter %s specified is empty" % key) - - # Validate unique attributes - attr_filters = [attr for clss in decorator_filters \ - for attr in clss.attrs] - dup_attr = [attr for attr in attr_filters - if attr_filters.count(attr) > 1] - if dup_attr: - raise TypeError('Detected duplicated attribute(s) %s in filter' - ' decorators' % ' ,'.join(dup_attr)) - - # Validate if filter is supported - for f in filters: - if f not in attr_filters: - classes = ', '.join([d.__name__ for d in decorator_filters]) - raise TypeError('Found "%s" filter but not declared in any of ' - '%s decorators' % (f, classes)) - def _registerTestCase(self, case): case_id = case.id() self.tc._registry['cases'][case_id] = case @@ -188,19 +160,20 @@ class OETestLoader(unittest.TestLoader): return True # Decorator filters - if self.filters and isinstance(case, OETestCase): - filters = self.filters.copy() - case_decorators = [cd for cd in case.decorators - if cd.__class__ in self.used_filters] - - # Iterate over case decorators to check if needs to be filtered. - for cd in case_decorators: - if cd.filtrate(filters): - return True - - # Case is missing one or more decorators for all the filters - # being used, so filter test case. - if filters: + if self.tags_filter is not None and callable(self.tags_filter): + alltags = set() + # pull tags from the case class + if hasattr(case, "__oeqa_testtags"): + for t in getattr(case, "__oeqa_testtags"): + alltags.add(t) + # pull tags from the method itself + if hasattr(case, test_name): + method = getattr(case, test_name) + if hasattr(method, "__oeqa_testtags"): + for t in getattr(method, "__oeqa_testtags"): + alltags.add(t) + + if self.tags_filter(alltags): return True return False diff --git a/poky/meta/lib/oeqa/core/runner.py b/poky/meta/lib/oeqa/core/runner.py index 930620ea1..f656e1a9c 100644 --- a/poky/meta/lib/oeqa/core/runner.py +++ b/poky/meta/lib/oeqa/core/runner.py @@ -43,6 +43,7 @@ class OETestResult(_TestResult): self.starttime = {} self.endtime = {} self.progressinfo = {} + self.extraresults = {} # Inject into tc so that TestDepends decorator can see results tc.results = self @@ -129,19 +130,51 @@ class OETestResult(_TestResult): return 'UNKNOWN', None - def addSuccess(self, test): + def extractExtraResults(self, test, details = None): + extraresults = None + if details is not None and "extraresults" in details: + extraresults = details.get("extraresults", {}) + elif hasattr(test, "extraresults"): + extraresults = test.extraresults + + if extraresults is not None: + for k, v in extraresults.items(): + # handle updating already existing entries (e.g. ptestresults.sections) + if k in self.extraresults: + self.extraresults[k].update(v) + else: + self.extraresults[k] = v + + def addError(self, test, *args, details = None): + self.extractExtraResults(test, details = details) + return super(OETestResult, self).addError(test, *args) + + def addFailure(self, test, *args, details = None): + self.extractExtraResults(test, details = details) + return super(OETestResult, self).addFailure(test, *args) + + def addSuccess(self, test, details = None): #Added so we can keep track of successes too self.successes.append((test, None)) - super(OETestResult, self).addSuccess(test) + self.extractExtraResults(test, details = details) + return super(OETestResult, self).addSuccess(test) + + def addExpectedFailure(self, test, *args, details = None): + self.extractExtraResults(test, details = details) + return super(OETestResult, self).addExpectedFailure(test, *args) + + def addUnexpectedSuccess(self, test, details = None): + self.extractExtraResults(test, details = details) + return super(OETestResult, self).addUnexpectedSuccess(test) def logDetails(self, json_file_dir=None, configuration=None, result_id=None, dump_streams=False): self.tc.logger.info("RESULTS:") - result = {} + result = self.extraresults logs = {} if hasattr(self.tc, "extraresults"): - result = self.tc.extraresults + result.update(self.tc.extraresults) for case_name in self.tc._registry['cases']: case = self.tc._registry['cases'][case_name] @@ -205,23 +238,20 @@ class OETestRunner(_TestRunner): self._walked_cases = self._walked_cases + 1 def _list_tests_name(self, suite): - from oeqa.core.decorator.oetag import OETestTag - self._walked_cases = 0 def _list_cases(logger, case): - oetag = None - - if hasattr(case, 'decorators'): - for d in case.decorators: - if isinstance(d, OETestTag): - oetag = d.oetag - - logger.info("%s\t\t%s" % (oetag, case.id())) + oetags = [] + if hasattr(case, '__oeqa_testtags'): + oetags = getattr(case, '__oeqa_testtags') + if oetags: + logger.info("%s (%s)" % (case.id(), ",".join(oetags))) + else: + logger.info("%s" % (case.id())) self.tc.logger.info("Listing all available tests:") self._walked_cases = 0 - self.tc.logger.info("id\ttag\t\ttest") + self.tc.logger.info("test (tags)") self.tc.logger.info("-" * 80) self._walk_suite(suite, _list_cases) self.tc.logger.info("-" * 80) diff --git a/poky/meta/lib/oeqa/core/tests/cases/data.py b/poky/meta/lib/oeqa/core/tests/cases/data.py index 0d8de87ae..61f88547f 100644 --- a/poky/meta/lib/oeqa/core/tests/cases/data.py +++ b/poky/meta/lib/oeqa/core/tests/cases/data.py @@ -5,7 +5,7 @@ # from oeqa.core.case import OETestCase -from oeqa.core.decorator.oetag import OETestTag +from oeqa.core.decorator import OETestTag from oeqa.core.decorator.data import OETestDataDepends class DataTest(OETestCase): diff --git a/poky/meta/lib/oeqa/core/tests/cases/oetag.py b/poky/meta/lib/oeqa/core/tests/cases/oetag.py index 4e1d08098..52f97dfda 100644 --- a/poky/meta/lib/oeqa/core/tests/cases/oetag.py +++ b/poky/meta/lib/oeqa/core/tests/cases/oetag.py @@ -5,10 +5,9 @@ # from oeqa.core.case import OETestCase -from oeqa.core.decorator.oetag import OETestTag +from oeqa.core.decorator import OETestTag class TagTest(OETestCase): - @OETestTag('goodTag') def testTagGood(self): self.assertTrue(True, msg='How is this possible?') @@ -17,5 +16,23 @@ class TagTest(OETestCase): def testTagOther(self): self.assertTrue(True, msg='How is this possible?') + @OETestTag('otherTag', 'multiTag') + def testTagOtherMulti(self): + self.assertTrue(True, msg='How is this possible?') + def testTagNone(self): self.assertTrue(True, msg='How is this possible?') + +@OETestTag('classTag') +class TagClassTest(OETestCase): + @OETestTag('otherTag') + def testTagOther(self): + self.assertTrue(True, msg='How is this possible?') + + @OETestTag('otherTag', 'multiTag') + def testTagOtherMulti(self): + self.assertTrue(True, msg='How is this possible?') + + def testTagNone(self): + self.assertTrue(True, msg='How is this possible?') + diff --git a/poky/meta/lib/oeqa/core/tests/common.py b/poky/meta/lib/oeqa/core/tests/common.py index 39efd504c..88cc758ad 100644 --- a/poky/meta/lib/oeqa/core/tests/common.py +++ b/poky/meta/lib/oeqa/core/tests/common.py @@ -30,9 +30,9 @@ class TestBase(unittest.TestCase): directory = os.path.dirname(os.path.abspath(__file__)) self.cases_path = os.path.join(directory, 'cases') - def _testLoader(self, d={}, modules=[], tests=[], filters={}): + def _testLoader(self, d={}, modules=[], tests=[], **kwargs): from oeqa.core.context import OETestContext tc = OETestContext(d, self.logger) tc.loadTests(self.cases_path, modules=modules, tests=tests, - filters=filters) + **kwargs) return tc diff --git a/poky/meta/lib/oeqa/core/tests/test_data.py b/poky/meta/lib/oeqa/core/tests/test_data.py index 50811bb38..ac74098b7 100755 --- a/poky/meta/lib/oeqa/core/tests/test_data.py +++ b/poky/meta/lib/oeqa/core/tests/test_data.py @@ -22,8 +22,9 @@ class TestData(TestBase): expectedException = "oeqa.core.exception.OEQAMissingVariable" tc = self._testLoader(modules=self.modules) - self.assertEqual(False, tc.runTests().wasSuccessful()) - for test, data in tc.errors: + results = tc.runTests() + self.assertFalse(results.wasSuccessful()) + for test, data in results.errors: expect = False if expectedException in data: expect = True @@ -35,8 +36,9 @@ class TestData(TestBase): d = {'IMAGE' : 'core-image-sato', 'ARCH' : 'arm'} tc = self._testLoader(d=d, modules=self.modules) - self.assertEqual(False, tc.runTests().wasSuccessful()) - for test, data in tc.failures: + results = tc.runTests() + self.assertFalse(results.wasSuccessful()) + for test, data in results.failures: expect = False if expectedError in data: expect = True diff --git a/poky/meta/lib/oeqa/core/tests/test_decorators.py b/poky/meta/lib/oeqa/core/tests/test_decorators.py index 499cd66ff..b798bf7d3 100755 --- a/poky/meta/lib/oeqa/core/tests/test_decorators.py +++ b/poky/meta/lib/oeqa/core/tests/test_decorators.py @@ -14,35 +14,58 @@ setup_sys_path() from oeqa.core.exception import OEQADependency from oeqa.core.utils.test import getCaseMethod, getSuiteCasesNames, getSuiteCasesIDs -class TestFilterDecorator(TestBase): - - def _runFilterTest(self, modules, filters, expect, msg): - tc = self._testLoader(modules=modules, filters=filters) - test_loaded = set(getSuiteCasesNames(tc.suites)) - self.assertEqual(expect, test_loaded, msg=msg) +class TestTagDecorator(TestBase): + def _runTest(self, modules, filterfn, expect): + tc = self._testLoader(modules = modules, tags_filter = filterfn) + test_loaded = set(getSuiteCasesIDs(tc.suites)) + self.assertEqual(expect, test_loaded) def test_oetag(self): - # Get all cases without filtering. - filter_all = {} - test_all = {'testTagGood', 'testTagOther', 'testTagNone'} - msg_all = 'Failed to get all oetag cases without filtering.' - - # Get cases with 'goodTag'. - filter_good = {'oetag':'goodTag'} - test_good = {'testTagGood'} - msg_good = 'Failed to get just one test filtering with "goodTag" oetag.' - - # Get cases with an invalid tag. - filter_invalid = {'oetag':'invalidTag'} - test_invalid = set() - msg_invalid = 'Failed to filter all test using an invalid oetag.' - - tests = ((filter_all, test_all, msg_all), - (filter_good, test_good, msg_good), - (filter_invalid, test_invalid, msg_invalid)) - - for test in tests: - self._runFilterTest(['oetag'], test[0], test[1], test[2]) + # get all cases without any filtering + self._runTest(['oetag'], None, { + 'oetag.TagTest.testTagGood', + 'oetag.TagTest.testTagOther', + 'oetag.TagTest.testTagOtherMulti', + 'oetag.TagTest.testTagNone', + 'oetag.TagClassTest.testTagOther', + 'oetag.TagClassTest.testTagOtherMulti', + 'oetag.TagClassTest.testTagNone', + }) + + # exclude any case with tags + self._runTest(['oetag'], lambda tags: tags, { + 'oetag.TagTest.testTagNone', + }) + + # exclude any case with otherTag + self._runTest(['oetag'], lambda tags: "otherTag" in tags, { + 'oetag.TagTest.testTagGood', + 'oetag.TagTest.testTagNone', + 'oetag.TagClassTest.testTagNone', + }) + + # exclude any case with classTag + self._runTest(['oetag'], lambda tags: "classTag" in tags, { + 'oetag.TagTest.testTagGood', + 'oetag.TagTest.testTagOther', + 'oetag.TagTest.testTagOtherMulti', + 'oetag.TagTest.testTagNone', + }) + + # include any case with classTag + self._runTest(['oetag'], lambda tags: "classTag" not in tags, { + 'oetag.TagClassTest.testTagOther', + 'oetag.TagClassTest.testTagOtherMulti', + 'oetag.TagClassTest.testTagNone', + }) + + # include any case with classTag or no tags + self._runTest(['oetag'], lambda tags: tags and "classTag" not in tags, { + 'oetag.TagTest.testTagNone', + 'oetag.TagClassTest.testTagOther', + 'oetag.TagClassTest.testTagOtherMulti', + 'oetag.TagClassTest.testTagNone', + }) class TestDependsDecorator(TestBase): modules = ['depends'] diff --git a/poky/meta/lib/oeqa/core/tests/test_loader.py b/poky/meta/lib/oeqa/core/tests/test_loader.py index 519ba9609..cb38ac845 100755 --- a/poky/meta/lib/oeqa/core/tests/test_loader.py +++ b/poky/meta/lib/oeqa/core/tests/test_loader.py @@ -15,31 +15,7 @@ from oeqa.core.exception import OEQADependency from oeqa.core.utils.test import getSuiteModules, getSuiteCasesIDs class TestLoader(TestBase): - - def test_fail_empty_filter(self): - filters = {'oetag' : ''} - expect = 'Filter oetag specified is empty' - msg = 'Expected TypeError exception for having invalid filter' - try: - # Must throw TypeError because empty filter - tc = self._testLoader(filters=filters) - self.fail(msg) - except TypeError as e: - result = True if expect in str(e) else False - self.assertTrue(result, msg=msg) - - def test_fail_invalid_filter(self): - filters = {'invalid' : 'good'} - expect = 'filter but not declared in any of' - msg = 'Expected TypeError exception for having invalid filter' - try: - # Must throw TypeError because invalid filter - tc = self._testLoader(filters=filters) - self.fail(msg) - except TypeError as e: - result = True if expect in str(e) else False - self.assertTrue(result, msg=msg) - + @unittest.skip("invalid directory is missing oetag.py") def test_fail_duplicated_module(self): cases_path = self.cases_path invalid_path = os.path.join(cases_path, 'loader', 'invalid') diff --git a/poky/meta/lib/oeqa/core/utils/concurrencytest.py b/poky/meta/lib/oeqa/core/utils/concurrencytest.py index 6bf771886..fa6fa34b0 100644 --- a/poky/meta/lib/oeqa/core/utils/concurrencytest.py +++ b/poky/meta/lib/oeqa/core/utils/concurrencytest.py @@ -21,6 +21,7 @@ import testtools import threading import time import io +import json import subunit from queue import Queue @@ -28,6 +29,8 @@ from itertools import cycle from subunit import ProtocolTestCase, TestProtocolClient from subunit.test_results import AutoTimingTestResultDecorator from testtools import ThreadsafeForwardingResult, iterate_tests +from testtools.content import Content +from testtools.content_type import ContentType from oeqa.utils.commands import get_test_layer import bb.utils @@ -70,6 +73,58 @@ class BBThreadsafeForwardingResult(ThreadsafeForwardingResult): self.semaphore.release() super(BBThreadsafeForwardingResult, self)._add_result_with_semaphore(method, test, *args, **kwargs) +class ProxyTestResult: + # a very basic TestResult proxy, in order to modify add* calls + def __init__(self, target): + self.result = target + + def _addResult(self, method, test, *args, **kwargs): + return method(test, *args, **kwargs) + + def addError(self, test, *args, **kwargs): + self._addResult(self.result.addError, test, *args, **kwargs) + + def addFailure(self, test, *args, **kwargs): + self._addResult(self.result.addFailure, test, *args, **kwargs) + + def addSuccess(self, test, *args, **kwargs): + self._addResult(self.result.addSuccess, test, *args, **kwargs) + + def addExpectedFailure(self, test, *args, **kwargs): + self._addResult(self.result.addExpectedFailure, test, *args, **kwargs) + + def addUnexpectedSuccess(self, test, *args, **kwargs): + self._addResult(self.result.addUnexpectedSuccess, test, *args, **kwargs) + + def __getattr__(self, attr): + return getattr(self.result, attr) + +class ExtraResultsDecoderTestResult(ProxyTestResult): + def _addResult(self, method, test, *args, **kwargs): + if "details" in kwargs and "extraresults" in kwargs["details"]: + if isinstance(kwargs["details"]["extraresults"], Content): + kwargs = kwargs.copy() + kwargs["details"] = kwargs["details"].copy() + extraresults = kwargs["details"]["extraresults"] + data = bytearray() + for b in extraresults.iter_bytes(): + data += b + extraresults = json.loads(data.decode()) + kwargs["details"]["extraresults"] = extraresults + return method(test, *args, **kwargs) + +class ExtraResultsEncoderTestResult(ProxyTestResult): + def _addResult(self, method, test, *args, **kwargs): + if hasattr(test, "extraresults"): + extras = lambda : [json.dumps(test.extraresults).encode()] + kwargs = kwargs.copy() + if "details" not in kwargs: + kwargs["details"] = {} + else: + kwargs["details"] = kwargs["details"].copy() + kwargs["details"]["extraresults"] = Content(ContentType("application", "json", {'charset': 'utf8'}), extras) + return method(test, *args, **kwargs) + # # We have to patch subunit since it doesn't understand how to handle addError # outside of a running test case. This can happen if classSetUp() fails @@ -116,7 +171,9 @@ class ConcurrentTestSuite(unittest.TestSuite): result.threadprogress = {} for i, (test, testnum) in enumerate(tests): result.threadprogress[i] = [] - process_result = BBThreadsafeForwardingResult(result, semaphore, i, testnum, totaltests) + process_result = BBThreadsafeForwardingResult( + ExtraResultsDecoderTestResult(result), + semaphore, i, testnum, totaltests) # Force buffering of stdout/stderr so the console doesn't get corrupted by test output # as per default in parent code process_result.buffer = True @@ -231,7 +288,7 @@ def fork_for_tests(concurrency_num, suite): # as per default in parent code subunit_client.buffer = True subunit_result = AutoTimingTestResultDecorator(subunit_client) - process_suite.run(subunit_result) + process_suite.run(ExtraResultsEncoderTestResult(subunit_result)) if ourpid != os.getpid(): os._exit(0) if newbuilddir: |