summaryrefslogtreecommitdiff
path: root/poky/meta/lib/oeqa/core
diff options
context:
space:
mode:
authorBrad Bishop <bradleyb@fuzziesquirrel.com>2019-09-10 14:20:22 +0300
committerBrad Bishop <bradleyb@fuzziesquirrel.com>2019-09-10 14:21:34 +0300
commit79641f25e882b55af6a647cfc0bf6bcc025661a5 (patch)
tree2b5fb3c59e5b6e85fd6fb8a94b6e15989d974336 /poky/meta/lib/oeqa/core
parent39ae1a1dce50fde698688fa5ac8eb737b302f0fd (diff)
downloadopenbmc-79641f25e882b55af6a647cfc0bf6bcc025661a5.tar.xz
poky: subtree update:c17113f1e2..3b87508a9a
Adrian Bunk (18): help2man: Upgrade 1.47.10 -> 1.47.11 glib-networking: Upgrade 2.60.2 -> 2.60.3 flac: Upgrade 1.3.2 -> 1.3.3 subversion: Upgrade 1.12.0 -> 1.12.2 world-broken.inc: Remove meta/recipes.txt: Stop documenting the long removed recipes-lsb4 python: Remove PYTHONLSBOPTS that hasn't been set for some time systemd: Print a buildtime warning about the status with musl linux-dtb.inc: Remove systemd: Update to the latest 242 branch Remove LSB support Remove LSB support bits libxcrypt: Fix the build with -Os systemd: Update to the latest 242 branch Remove leftovers from the eglinfo removal Update to KERNEL_DEVICETREE variable. ref-manual: Remove documentation for the removed bluez5 distro feature ref-manual: Remove documentation for the removed gnome class Alex Kiernan (1): iputils: Set prefix correctly for usrmerge Alexander Kanavin (23): python3-numpy: update to 1.17.0 images: use 512M of RAM in qemu if 'opengl" is in DISTRO_FEATURES boost: update to 1.71.0 beaglebone: use the default of 256M rather than 512M rpm: resolve a host contamination issue for mono packaging qemu: set default RAM to 256M for all machines mesa: disable gallium swrast driver on x86 x32 ltp: move from python 2.x to python 3.x lsb-release: fix upstream version check libffi: fix upstream version check qemu: switch to '-vga std' emulated hardware from vmware/cirrus for x86/mips ca-certificates: use python 3 for builds libpsl: use python 3 during builds pseudo: use python 3 during builds llvm: use python 3 during builds virglrenderer: update 0.7.0 -> 0.8.0 webkitgtk: use Python 3 for builds webkitgtk: update 2.24.3 -> 2.24.4 default-providers.inc: set mesa as the provider for native(sdk) virtual/libgl libepoxy: enable x11 PACKAGECONFIG for native/nativesdk eglinfo: remove the recipe local.conf.sample: add a commented-out option to enable the gtk UI qemu frontend package.bbclass: allow shell-style wildcards in PRIVATE_LIBS Alistair Francis (2): linux-yocto-dev: Mark qemuriscv64 as compatible qemuriscv: Generate a wic rootFS with a larger filesystem Andre McCurdy (11): bitbake.conf: drop FETCHCMD, GITDIR, etc fetcher over-rides tune-arm1136jf-s.inc: add hardfloat support tune-arm1176jz-s.inc: add support for arm1176jz-s CPU specific tuning tune-ep9312.inc: fix TUNEVALID description packagegroup-core-boot: drop duplicate MACHINE_ESSENTIAL_EXTRA_RDEPENDS, etc rootfs-postcommands.bbclass: add inittab tweak to read_only_rootfs_hook() base-files: nsswitch.conf is glibc specific sqlite3: make CFLAGS consistent across native, nativesdk and target cases sqlite3: explicitly set target endian-ness feature-arm-thumb.inc: fix ARM_THUMB_SUFFIX dropbear: don't create invalid symlinks if ssh, scp, etc are disabled Anuj Mittal (1): libjpeg-turbo: upgrade 2.0.2 -> 2.0.3 Armin Kuster (4): bitbake: tests/fetch: Resolve fetch error in bitbake-selftest gcc-9.2: Security fix for CVE-2019-14250 gcc-8.3: Security fix for CVE-2019-14250 gcc-8.3: remove 8.3 from zeus Bruce Ashfield (11): bluez5: fix build against libc-headers 5.2+ kernel-devsrc: remove python2 dependency perf: change dependencies on python to python3 perf: v5.3+ build fixes devsrc: update arm64 scripts/prepare for kernels 5.3+ linux-libc-headers: update to v5.2 headers linux-yocto: introduce 5.2 recipes qemu: bump linux-yocto preferred version to 5.2 linux-yocto/5.2: update qemuriscv64 SRCREV linux-yocto/5.2: config tweaks and kallsyms fix linux-yocto: drop 5.0 recipes Changhyeok Bae (1): iputils: upgrade to s20190709 Changqing Li (8): libedit: fix failed to get history avahi-ui: support usrmerge bootchart2: support usrmerge systemd-bootchart: support usrmerge bootchart2: switch to add patch from change source in do_install mdadm: skip timeout testcase 11spare-migration dnf.py: installroot support usrmerge man-db: support usrmerge Chen Qi (7): bbtests.py: remove recipe-depends.dot commands.py: fix typo bitbake: cooker.py: remove generation of recipe-depends.dot bind: fix build with python3 PACKAGECONFIG enabled python-testtools: rdepend on doctest module python-numpy: rdepend on multiprocessing module python3: make misc package rdepend on pydoc and pickle modules Christophe PRIOUZEAU (1): common-licenses: update BSD-2-CLAUSE license text Dan Callaghan (1): autotools: always include config_site.d files in CONFIG_SITE Daniel Klauer (1): image.bbclass: Fix debug output for rootfs size Denys Zagorui (1): gzip: add nativesdk support Drew Moseley (1): runqemu: Fix typo in help text. He Zhe (4): ltp: cve/meltdown.c: Fix kernel symbol finding ltp: syscalls: rt_sigwaitinfo01: Fix failure for MIPS arches qemu: Add pkg-config handling for libgcrypt qemu: Fix dependency of PACKAGECONFIG libcurl Jacob Kroon (2): btrfs-tools: Add PACKAGECONFIG knob for controlling python support rm_work: Remove redundant 'after' in addtask statement Jason Wessel (2): serial-getty@.service: Allow device to fast fail if it does not exist Revert "kernel.bbclass: adjust a condition checking" Joshua Watt (9): bitbake: bitbake: respect force flag in runall and runonly oeqa: reproducible: Cleanup reproducible build oeqa: reproducible: Use subTest for packages oeqa: Set LD_LIBRARY_PATH when executing native commands oeqa: reproducible: Record packages in test results oeqa: Enable reproducible build test oeqa: reproducible: Do two clean builds oeqa: reproducible: Preserve sstate mirror for first build resulttool: Add reproducible log extraction Kai Kang (4): avahi: launch avahi-daemon after connman cups: redefine CUPS_SERVERBIN texinfo: fix multilib file confliction sysstat: 12.1.3 -> 12.1.6 Kevin Hao (1): meta-yocto-bsp: Bump to the v5.2 kernel Khem Raj (12): go-1.12: update to 1.12.9 minor release powerpc-tunes: Introduce bigendian tune tune-riscv: Drop littleendian and introduce bigendian tune gcc: Move STANDARD_STARTFILE_PREFIX_1 and STANDARD_STARTFILE_PREFIX_2 back to gcc.c gcc: Fix ldbl-128 support for musl libgpg-error: Fix build with gawk 5.x libgpg-error: Fix build with gawk 5.x musl: Update to latest master gcc-9.2: Fix risc-v dynamic linker relocation python3native, pythonnative: export PYTHON_LIBRARY and PYTHON_INCLUDE_DIR oeqa: Recognise svlogd as another logger python3-manifest.json: Fix typo in summary Mark Asselstine (1): sdkext: use simpler kernel module for devtool test Mark Hatle (1): yocto-check-layer: Allow any case for README file detection Martin Hundebøll (1): initscripts: remove warnings on read-only-rootfs (again) Martin Jansa (3): bitbake: utils: Fix movefile() exception handling with python3 bitbake: fetch2: show warning when renaming the archive with bad checksum failed bitbake: bitbake-user-manual: Improve the example for Removal (Override Style Syntax) Mazliana (1): oeqa/kerneldevelopment: Able to apply a single patch to the Linux kernel source Ming Liu (1): weston: uprev to 7.0.0 Nathan Rossi (30): binutils: Fix mips patch which changes default emulation oeqa/utils/nfs: Add unfs_server function to setup a userspace NFS server scripts/lib/resulttool/report.py: Add more result types gcc-common.inc: Process staging fixme with correct target/native sysroot oeqa/core/tests: Skip test_fail_duplicated_module oeqa/core/tests: Fix test_data module tests Clean up remnants of glibc-initial oe-selftest: Implement console 'keepalive' output dejagnu: Add dejagnu for binutils/gcc test suites binutils-cross-testsuite: Create recipe for test suite execution gcc-runtime: Add do_check task for executing gcc test suites glibc-testsuite: Create a recipe to implement glibc test suite gcc-testsuite.inc: Fix ssh.exp, ensure multilib_flags are populated security_flags.inc: Add overrides for glibc-testsuite to match glibc archiver.bbclass: Do not archive srpm's if there are no package tasks oeqa/core: Implement proper extra result collection and serialization oeqa/core: Rework OETestTag and remove unused OETestFilter oeqa/selftest: Add test run filtering based on test tags oeqa/core/runner.py: Fix OETestTag listing oeqa/selftest/binutils: Create selftest case for binutils test suite oeqa/selftest/gcc: Create selftest case for gcc test suite oeqa/selftest/glibc: Create selftest case for glibc test suite oeqa/selftest/cases/glibc.py: Don't install python3 and pexpect oeqa/selftest: For toolchain tests do not fail if a test failed oeqa/selftest: Toolchain tests suffix "-user" for qemu usermode results oeqa/selftest: Use extraresults on self instead of self.tc oeqa/selftest/context.py: Change -t/-T args to be optional oeqa/core/decorator: Fix super class modifying subclass tags oeqa/selftest/cases/gcc.py: Split into classes for parallelism oeqa/selftest/cases/glibc.py: Rework and tag with toolchain-user/system Neil Armstrong (1): mesa: Add support for the panfrost PACKAGECONFIG Oleksandr Kravchuk (8): libsolv: update to 0.7.6 libevdev: update to 1.8.0 ell: update to 0.22 go-dep: update to 0.5.4 libsecret: 0.19.0 python3-pbr: update to 5.4.3 python3-git: update to 3.0.2 libxcrypt: update to 4.4.8 Peiran Hong (1): diffutils: Added perl to support ptest & Skipped one test case Peter Kjellerstedt (2): libffi: Make it build for MIPS o32 devtool: Avoid failure for recipes with S == WORKDIR and no local files Richard Purdie (11): poky-altcfg: Replace poky-lsb with poky-altcfg ca-certificates: Fix reproducibilty and multilib issue gcc-cross: Clean up fixed-includes systemtap: Use latest and greatest git version for 5.2 kernel support gcc-cross: Fix header file corruption problems poky: Use 5.2 kernel by default binutils/glibc-testsuite: inherit nopackages to fix world builds systemtap: Drop patches merged upstream poky-tiny/poky-altcfg: Use busybox-mdev and systemd oeqa/selftest: Markup 'machine' specific test cases oeqa/selftest: Tweak binutils tests tags as toolchain/user and system Robert Yang (3): ccache: 3.7.1 -> 3.7.3 nfs-utils: 2.3.3 -> 2.4.1 build-compare: 2015.02.10 -> 2019.08.14 Ross Burton (11): libedit: remove FETCHCMD append libnl: upgrade to 3.5.0 libnl: don't patch libnl-3.0.pc systemd: explicitly set the path to nologin core-image-sato: don't use 512MB in qemumips patch: add CVE tags to patches pango: 1.44.6 upgrade pango: add PACKAGECONFIG for libthai support harfbuzz: upgrade to 2.6.1 systemd: ensure reproducible builds by clearly exposing the time epoch support ref-system-requirements: add Debian 10 to supported distribution list Scott Rifenbark (5): ref-manual: Updated the "AS" variable ref-manual: Updates to "Building Your Recipe" dev-manual: Updated "Creating a Team Development Environment" poky-end: Added "pylint3" package to Ubuntu essentials. ref-manual: sync list of image types with source Tim Blechmann (1): icecc: blacklist systemtap Trevor Gamblin (2): lighttpd: drop hard-coded lua version in PACKAGECONFIG libevent: add granularity to ptest log Vijai Kumar K (1): lib/oe/gpg_sign.py: Remove unwanted import Wes Lindauer (1): bitbake: bitbake: cooker: Ensure bbappends are found in stable order Yi Zhao (1): libbsd: upgrade 0.9.1 -> 0.10.0 Yuan Chao (1): man-db: upgrade 2.8.6.1 -> 2.8.7 Zang Ruochen (7): libarchive:upgrade 3.3.3 -> 3.4.0 python3-pip:upgrade 19.2.2 -> 19.2.3 logrotate:upgrade 3.15.0 -> 3.15.1 libogg:upgrade 1.3.3 -> 1.3.4 dtc:upgrade 1.5.0 -> 1.5.1 python3-dbus:upgrade 1.2.8 -> 1.2.10 meson:upgrade 0.51.1 -> 0.51.2 Change-Id: I341e8df65e8196114088a2d6d6eb3a33296188b4 Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>
Diffstat (limited to 'poky/meta/lib/oeqa/core')
-rw-r--r--poky/meta/lib/oeqa/core/context.py4
-rw-r--r--poky/meta/lib/oeqa/core/decorator/__init__.py21
-rw-r--r--poky/meta/lib/oeqa/core/decorator/oetag.py27
-rw-r--r--poky/meta/lib/oeqa/core/loader.py61
-rw-r--r--poky/meta/lib/oeqa/core/runner.py60
-rw-r--r--poky/meta/lib/oeqa/core/tests/cases/data.py2
-rw-r--r--poky/meta/lib/oeqa/core/tests/cases/oetag.py21
-rw-r--r--poky/meta/lib/oeqa/core/tests/common.py4
-rwxr-xr-xpoky/meta/lib/oeqa/core/tests/test_data.py10
-rwxr-xr-xpoky/meta/lib/oeqa/core/tests/test_decorators.py77
-rwxr-xr-xpoky/meta/lib/oeqa/core/tests/test_loader.py26
-rw-r--r--poky/meta/lib/oeqa/core/utils/concurrencytest.py61
12 files changed, 215 insertions, 159 deletions
diff --git a/poky/meta/lib/oeqa/core/context.py b/poky/meta/lib/oeqa/core/context.py
index 68819cc33..14fc6a54f 100644
--- a/poky/meta/lib/oeqa/core/context.py
+++ b/poky/meta/lib/oeqa/core/context.py
@@ -64,12 +64,12 @@ class OETestContext(object):
setattr(tclass, 'setUpHooker', skipfuncgen('Skip by the command line argument "%s"' % skip))
def loadTests(self, module_paths, modules=[], tests=[],
- modules_manifest="", modules_required=[], filters={}):
+ modules_manifest="", modules_required=[], **kwargs):
if modules_manifest:
modules = self._read_modules_from_manifest(modules_manifest)
self.loader = self.loaderClass(self, module_paths, modules, tests,
- modules_required, filters)
+ modules_required, **kwargs)
self.suites = self.loader.discover()
def runTests(self, processes=None, skips=[]):
diff --git a/poky/meta/lib/oeqa/core/decorator/__init__.py b/poky/meta/lib/oeqa/core/decorator/__init__.py
index 923b21826..1a82518ab 100644
--- a/poky/meta/lib/oeqa/core/decorator/__init__.py
+++ b/poky/meta/lib/oeqa/core/decorator/__init__.py
@@ -6,6 +6,7 @@
from functools import wraps
from abc import abstractmethod, ABCMeta
+from oeqa.core.utils.misc import strToList
decoratorClasses = set()
@@ -63,12 +64,16 @@ class OETestDiscover(OETestDecorator):
def discover(registry):
return registry['cases']
-class OETestFilter(OETestDecorator):
+def OETestTag(*tags):
+ expandedtags = []
+ for tag in tags:
+ expandedtags += strToList(tag)
+ def decorator(item):
+ if hasattr(item, "__oeqa_testtags"):
+ # do not append, create a new list (to handle classes with inheritance)
+ item.__oeqa_testtags = list(item.__oeqa_testtags) + expandedtags
+ else:
+ item.__oeqa_testtags = expandedtags
+ return item
+ return decorator
- # OETestLoader call it while loading the tests
- # in loadTestsFromTestCase method, it needs to
- # return a bool, True if needs to be filtered.
- # This method must consume the filter used.
- @abstractmethod
- def filtrate(self, filters):
- return False
diff --git a/poky/meta/lib/oeqa/core/decorator/oetag.py b/poky/meta/lib/oeqa/core/decorator/oetag.py
deleted file mode 100644
index 8c31138da..000000000
--- a/poky/meta/lib/oeqa/core/decorator/oetag.py
+++ /dev/null
@@ -1,27 +0,0 @@
-#
-# Copyright (C) 2016 Intel Corporation
-#
-# SPDX-License-Identifier: MIT
-#
-
-from . import OETestFilter, registerDecorator
-from oeqa.core.utils.misc import strToList
-
-def _tagFilter(tags, filters):
- return False if set(tags) & set(filters) else True
-
-@registerDecorator
-class OETestTag(OETestFilter):
- attrs = ('oetag',)
-
- def bind(self, registry, case):
- super(OETestTag, self).bind(registry, case)
- self.oetag = strToList(self.oetag, 'oetag')
-
- def filtrate(self, filters):
- if filters.get('oetag'):
- filterx = strToList(filters['oetag'], 'oetag')
- del filters['oetag']
- if _tagFilter(self.oetag, filterx):
- return True
- return False
diff --git a/poky/meta/lib/oeqa/core/loader.py b/poky/meta/lib/oeqa/core/loader.py
index 7fea0585c..0d7970d49 100644
--- a/poky/meta/lib/oeqa/core/loader.py
+++ b/poky/meta/lib/oeqa/core/loader.py
@@ -16,7 +16,7 @@ from oeqa.core.utils.test import getSuiteModules, getCaseID
from oeqa.core.exception import OEQATestNotFound
from oeqa.core.case import OETestCase
from oeqa.core.decorator import decoratorClasses, OETestDecorator, \
- OETestFilter, OETestDiscover
+ OETestDiscover
# When loading tests, the unittest framework stores any exceptions and
# displays them only when the run method is called.
@@ -68,7 +68,7 @@ class OETestLoader(unittest.TestLoader):
'_top_level_dir']
def __init__(self, tc, module_paths, modules, tests, modules_required,
- filters, *args, **kwargs):
+ *args, **kwargs):
self.tc = tc
self.modules = _built_modules_dict(modules)
@@ -76,13 +76,7 @@ class OETestLoader(unittest.TestLoader):
self.tests = tests
self.modules_required = modules_required
- self.filters = filters
- self.decorator_filters = [d for d in decoratorClasses if \
- issubclass(d, OETestFilter)]
- self._validateFilters(self.filters, self.decorator_filters)
- self.used_filters = [d for d in self.decorator_filters
- for f in self.filters
- if f in d.attrs]
+ self.tags_filter = kwargs.get("tags_filter", None)
if isinstance(module_paths, str):
module_paths = [module_paths]
@@ -104,28 +98,6 @@ class OETestLoader(unittest.TestLoader):
setattr(testCaseClass, 'td', self.tc.td)
setattr(testCaseClass, 'logger', self.tc.logger)
- def _validateFilters(self, filters, decorator_filters):
- # Validate if filter isn't empty
- for key,value in filters.items():
- if not value:
- raise TypeError("Filter %s specified is empty" % key)
-
- # Validate unique attributes
- attr_filters = [attr for clss in decorator_filters \
- for attr in clss.attrs]
- dup_attr = [attr for attr in attr_filters
- if attr_filters.count(attr) > 1]
- if dup_attr:
- raise TypeError('Detected duplicated attribute(s) %s in filter'
- ' decorators' % ' ,'.join(dup_attr))
-
- # Validate if filter is supported
- for f in filters:
- if f not in attr_filters:
- classes = ', '.join([d.__name__ for d in decorator_filters])
- raise TypeError('Found "%s" filter but not declared in any of '
- '%s decorators' % (f, classes))
-
def _registerTestCase(self, case):
case_id = case.id()
self.tc._registry['cases'][case_id] = case
@@ -188,19 +160,20 @@ class OETestLoader(unittest.TestLoader):
return True
# Decorator filters
- if self.filters and isinstance(case, OETestCase):
- filters = self.filters.copy()
- case_decorators = [cd for cd in case.decorators
- if cd.__class__ in self.used_filters]
-
- # Iterate over case decorators to check if needs to be filtered.
- for cd in case_decorators:
- if cd.filtrate(filters):
- return True
-
- # Case is missing one or more decorators for all the filters
- # being used, so filter test case.
- if filters:
+ if self.tags_filter is not None and callable(self.tags_filter):
+ alltags = set()
+ # pull tags from the case class
+ if hasattr(case, "__oeqa_testtags"):
+ for t in getattr(case, "__oeqa_testtags"):
+ alltags.add(t)
+ # pull tags from the method itself
+ if hasattr(case, test_name):
+ method = getattr(case, test_name)
+ if hasattr(method, "__oeqa_testtags"):
+ for t in getattr(method, "__oeqa_testtags"):
+ alltags.add(t)
+
+ if self.tags_filter(alltags):
return True
return False
diff --git a/poky/meta/lib/oeqa/core/runner.py b/poky/meta/lib/oeqa/core/runner.py
index 930620ea1..f656e1a9c 100644
--- a/poky/meta/lib/oeqa/core/runner.py
+++ b/poky/meta/lib/oeqa/core/runner.py
@@ -43,6 +43,7 @@ class OETestResult(_TestResult):
self.starttime = {}
self.endtime = {}
self.progressinfo = {}
+ self.extraresults = {}
# Inject into tc so that TestDepends decorator can see results
tc.results = self
@@ -129,19 +130,51 @@ class OETestResult(_TestResult):
return 'UNKNOWN', None
- def addSuccess(self, test):
+ def extractExtraResults(self, test, details = None):
+ extraresults = None
+ if details is not None and "extraresults" in details:
+ extraresults = details.get("extraresults", {})
+ elif hasattr(test, "extraresults"):
+ extraresults = test.extraresults
+
+ if extraresults is not None:
+ for k, v in extraresults.items():
+ # handle updating already existing entries (e.g. ptestresults.sections)
+ if k in self.extraresults:
+ self.extraresults[k].update(v)
+ else:
+ self.extraresults[k] = v
+
+ def addError(self, test, *args, details = None):
+ self.extractExtraResults(test, details = details)
+ return super(OETestResult, self).addError(test, *args)
+
+ def addFailure(self, test, *args, details = None):
+ self.extractExtraResults(test, details = details)
+ return super(OETestResult, self).addFailure(test, *args)
+
+ def addSuccess(self, test, details = None):
#Added so we can keep track of successes too
self.successes.append((test, None))
- super(OETestResult, self).addSuccess(test)
+ self.extractExtraResults(test, details = details)
+ return super(OETestResult, self).addSuccess(test)
+
+ def addExpectedFailure(self, test, *args, details = None):
+ self.extractExtraResults(test, details = details)
+ return super(OETestResult, self).addExpectedFailure(test, *args)
+
+ def addUnexpectedSuccess(self, test, details = None):
+ self.extractExtraResults(test, details = details)
+ return super(OETestResult, self).addUnexpectedSuccess(test)
def logDetails(self, json_file_dir=None, configuration=None, result_id=None,
dump_streams=False):
self.tc.logger.info("RESULTS:")
- result = {}
+ result = self.extraresults
logs = {}
if hasattr(self.tc, "extraresults"):
- result = self.tc.extraresults
+ result.update(self.tc.extraresults)
for case_name in self.tc._registry['cases']:
case = self.tc._registry['cases'][case_name]
@@ -205,23 +238,20 @@ class OETestRunner(_TestRunner):
self._walked_cases = self._walked_cases + 1
def _list_tests_name(self, suite):
- from oeqa.core.decorator.oetag import OETestTag
-
self._walked_cases = 0
def _list_cases(logger, case):
- oetag = None
-
- if hasattr(case, 'decorators'):
- for d in case.decorators:
- if isinstance(d, OETestTag):
- oetag = d.oetag
-
- logger.info("%s\t\t%s" % (oetag, case.id()))
+ oetags = []
+ if hasattr(case, '__oeqa_testtags'):
+ oetags = getattr(case, '__oeqa_testtags')
+ if oetags:
+ logger.info("%s (%s)" % (case.id(), ",".join(oetags)))
+ else:
+ logger.info("%s" % (case.id()))
self.tc.logger.info("Listing all available tests:")
self._walked_cases = 0
- self.tc.logger.info("id\ttag\t\ttest")
+ self.tc.logger.info("test (tags)")
self.tc.logger.info("-" * 80)
self._walk_suite(suite, _list_cases)
self.tc.logger.info("-" * 80)
diff --git a/poky/meta/lib/oeqa/core/tests/cases/data.py b/poky/meta/lib/oeqa/core/tests/cases/data.py
index 0d8de87ae..61f88547f 100644
--- a/poky/meta/lib/oeqa/core/tests/cases/data.py
+++ b/poky/meta/lib/oeqa/core/tests/cases/data.py
@@ -5,7 +5,7 @@
#
from oeqa.core.case import OETestCase
-from oeqa.core.decorator.oetag import OETestTag
+from oeqa.core.decorator import OETestTag
from oeqa.core.decorator.data import OETestDataDepends
class DataTest(OETestCase):
diff --git a/poky/meta/lib/oeqa/core/tests/cases/oetag.py b/poky/meta/lib/oeqa/core/tests/cases/oetag.py
index 4e1d08098..52f97dfda 100644
--- a/poky/meta/lib/oeqa/core/tests/cases/oetag.py
+++ b/poky/meta/lib/oeqa/core/tests/cases/oetag.py
@@ -5,10 +5,9 @@
#
from oeqa.core.case import OETestCase
-from oeqa.core.decorator.oetag import OETestTag
+from oeqa.core.decorator import OETestTag
class TagTest(OETestCase):
-
@OETestTag('goodTag')
def testTagGood(self):
self.assertTrue(True, msg='How is this possible?')
@@ -17,5 +16,23 @@ class TagTest(OETestCase):
def testTagOther(self):
self.assertTrue(True, msg='How is this possible?')
+ @OETestTag('otherTag', 'multiTag')
+ def testTagOtherMulti(self):
+ self.assertTrue(True, msg='How is this possible?')
+
def testTagNone(self):
self.assertTrue(True, msg='How is this possible?')
+
+@OETestTag('classTag')
+class TagClassTest(OETestCase):
+ @OETestTag('otherTag')
+ def testTagOther(self):
+ self.assertTrue(True, msg='How is this possible?')
+
+ @OETestTag('otherTag', 'multiTag')
+ def testTagOtherMulti(self):
+ self.assertTrue(True, msg='How is this possible?')
+
+ def testTagNone(self):
+ self.assertTrue(True, msg='How is this possible?')
+
diff --git a/poky/meta/lib/oeqa/core/tests/common.py b/poky/meta/lib/oeqa/core/tests/common.py
index 39efd504c..88cc758ad 100644
--- a/poky/meta/lib/oeqa/core/tests/common.py
+++ b/poky/meta/lib/oeqa/core/tests/common.py
@@ -30,9 +30,9 @@ class TestBase(unittest.TestCase):
directory = os.path.dirname(os.path.abspath(__file__))
self.cases_path = os.path.join(directory, 'cases')
- def _testLoader(self, d={}, modules=[], tests=[], filters={}):
+ def _testLoader(self, d={}, modules=[], tests=[], **kwargs):
from oeqa.core.context import OETestContext
tc = OETestContext(d, self.logger)
tc.loadTests(self.cases_path, modules=modules, tests=tests,
- filters=filters)
+ **kwargs)
return tc
diff --git a/poky/meta/lib/oeqa/core/tests/test_data.py b/poky/meta/lib/oeqa/core/tests/test_data.py
index 50811bb38..ac74098b7 100755
--- a/poky/meta/lib/oeqa/core/tests/test_data.py
+++ b/poky/meta/lib/oeqa/core/tests/test_data.py
@@ -22,8 +22,9 @@ class TestData(TestBase):
expectedException = "oeqa.core.exception.OEQAMissingVariable"
tc = self._testLoader(modules=self.modules)
- self.assertEqual(False, tc.runTests().wasSuccessful())
- for test, data in tc.errors:
+ results = tc.runTests()
+ self.assertFalse(results.wasSuccessful())
+ for test, data in results.errors:
expect = False
if expectedException in data:
expect = True
@@ -35,8 +36,9 @@ class TestData(TestBase):
d = {'IMAGE' : 'core-image-sato', 'ARCH' : 'arm'}
tc = self._testLoader(d=d, modules=self.modules)
- self.assertEqual(False, tc.runTests().wasSuccessful())
- for test, data in tc.failures:
+ results = tc.runTests()
+ self.assertFalse(results.wasSuccessful())
+ for test, data in results.failures:
expect = False
if expectedError in data:
expect = True
diff --git a/poky/meta/lib/oeqa/core/tests/test_decorators.py b/poky/meta/lib/oeqa/core/tests/test_decorators.py
index 499cd66ff..b798bf7d3 100755
--- a/poky/meta/lib/oeqa/core/tests/test_decorators.py
+++ b/poky/meta/lib/oeqa/core/tests/test_decorators.py
@@ -14,35 +14,58 @@ setup_sys_path()
from oeqa.core.exception import OEQADependency
from oeqa.core.utils.test import getCaseMethod, getSuiteCasesNames, getSuiteCasesIDs
-class TestFilterDecorator(TestBase):
-
- def _runFilterTest(self, modules, filters, expect, msg):
- tc = self._testLoader(modules=modules, filters=filters)
- test_loaded = set(getSuiteCasesNames(tc.suites))
- self.assertEqual(expect, test_loaded, msg=msg)
+class TestTagDecorator(TestBase):
+ def _runTest(self, modules, filterfn, expect):
+ tc = self._testLoader(modules = modules, tags_filter = filterfn)
+ test_loaded = set(getSuiteCasesIDs(tc.suites))
+ self.assertEqual(expect, test_loaded)
def test_oetag(self):
- # Get all cases without filtering.
- filter_all = {}
- test_all = {'testTagGood', 'testTagOther', 'testTagNone'}
- msg_all = 'Failed to get all oetag cases without filtering.'
-
- # Get cases with 'goodTag'.
- filter_good = {'oetag':'goodTag'}
- test_good = {'testTagGood'}
- msg_good = 'Failed to get just one test filtering with "goodTag" oetag.'
-
- # Get cases with an invalid tag.
- filter_invalid = {'oetag':'invalidTag'}
- test_invalid = set()
- msg_invalid = 'Failed to filter all test using an invalid oetag.'
-
- tests = ((filter_all, test_all, msg_all),
- (filter_good, test_good, msg_good),
- (filter_invalid, test_invalid, msg_invalid))
-
- for test in tests:
- self._runFilterTest(['oetag'], test[0], test[1], test[2])
+ # get all cases without any filtering
+ self._runTest(['oetag'], None, {
+ 'oetag.TagTest.testTagGood',
+ 'oetag.TagTest.testTagOther',
+ 'oetag.TagTest.testTagOtherMulti',
+ 'oetag.TagTest.testTagNone',
+ 'oetag.TagClassTest.testTagOther',
+ 'oetag.TagClassTest.testTagOtherMulti',
+ 'oetag.TagClassTest.testTagNone',
+ })
+
+ # exclude any case with tags
+ self._runTest(['oetag'], lambda tags: tags, {
+ 'oetag.TagTest.testTagNone',
+ })
+
+ # exclude any case with otherTag
+ self._runTest(['oetag'], lambda tags: "otherTag" in tags, {
+ 'oetag.TagTest.testTagGood',
+ 'oetag.TagTest.testTagNone',
+ 'oetag.TagClassTest.testTagNone',
+ })
+
+ # exclude any case with classTag
+ self._runTest(['oetag'], lambda tags: "classTag" in tags, {
+ 'oetag.TagTest.testTagGood',
+ 'oetag.TagTest.testTagOther',
+ 'oetag.TagTest.testTagOtherMulti',
+ 'oetag.TagTest.testTagNone',
+ })
+
+ # include any case with classTag
+ self._runTest(['oetag'], lambda tags: "classTag" not in tags, {
+ 'oetag.TagClassTest.testTagOther',
+ 'oetag.TagClassTest.testTagOtherMulti',
+ 'oetag.TagClassTest.testTagNone',
+ })
+
+ # include any case with classTag or no tags
+ self._runTest(['oetag'], lambda tags: tags and "classTag" not in tags, {
+ 'oetag.TagTest.testTagNone',
+ 'oetag.TagClassTest.testTagOther',
+ 'oetag.TagClassTest.testTagOtherMulti',
+ 'oetag.TagClassTest.testTagNone',
+ })
class TestDependsDecorator(TestBase):
modules = ['depends']
diff --git a/poky/meta/lib/oeqa/core/tests/test_loader.py b/poky/meta/lib/oeqa/core/tests/test_loader.py
index 519ba9609..cb38ac845 100755
--- a/poky/meta/lib/oeqa/core/tests/test_loader.py
+++ b/poky/meta/lib/oeqa/core/tests/test_loader.py
@@ -15,31 +15,7 @@ from oeqa.core.exception import OEQADependency
from oeqa.core.utils.test import getSuiteModules, getSuiteCasesIDs
class TestLoader(TestBase):
-
- def test_fail_empty_filter(self):
- filters = {'oetag' : ''}
- expect = 'Filter oetag specified is empty'
- msg = 'Expected TypeError exception for having invalid filter'
- try:
- # Must throw TypeError because empty filter
- tc = self._testLoader(filters=filters)
- self.fail(msg)
- except TypeError as e:
- result = True if expect in str(e) else False
- self.assertTrue(result, msg=msg)
-
- def test_fail_invalid_filter(self):
- filters = {'invalid' : 'good'}
- expect = 'filter but not declared in any of'
- msg = 'Expected TypeError exception for having invalid filter'
- try:
- # Must throw TypeError because invalid filter
- tc = self._testLoader(filters=filters)
- self.fail(msg)
- except TypeError as e:
- result = True if expect in str(e) else False
- self.assertTrue(result, msg=msg)
-
+ @unittest.skip("invalid directory is missing oetag.py")
def test_fail_duplicated_module(self):
cases_path = self.cases_path
invalid_path = os.path.join(cases_path, 'loader', 'invalid')
diff --git a/poky/meta/lib/oeqa/core/utils/concurrencytest.py b/poky/meta/lib/oeqa/core/utils/concurrencytest.py
index 6bf771886..fa6fa34b0 100644
--- a/poky/meta/lib/oeqa/core/utils/concurrencytest.py
+++ b/poky/meta/lib/oeqa/core/utils/concurrencytest.py
@@ -21,6 +21,7 @@ import testtools
import threading
import time
import io
+import json
import subunit
from queue import Queue
@@ -28,6 +29,8 @@ from itertools import cycle
from subunit import ProtocolTestCase, TestProtocolClient
from subunit.test_results import AutoTimingTestResultDecorator
from testtools import ThreadsafeForwardingResult, iterate_tests
+from testtools.content import Content
+from testtools.content_type import ContentType
from oeqa.utils.commands import get_test_layer
import bb.utils
@@ -70,6 +73,58 @@ class BBThreadsafeForwardingResult(ThreadsafeForwardingResult):
self.semaphore.release()
super(BBThreadsafeForwardingResult, self)._add_result_with_semaphore(method, test, *args, **kwargs)
+class ProxyTestResult:
+ # a very basic TestResult proxy, in order to modify add* calls
+ def __init__(self, target):
+ self.result = target
+
+ def _addResult(self, method, test, *args, **kwargs):
+ return method(test, *args, **kwargs)
+
+ def addError(self, test, *args, **kwargs):
+ self._addResult(self.result.addError, test, *args, **kwargs)
+
+ def addFailure(self, test, *args, **kwargs):
+ self._addResult(self.result.addFailure, test, *args, **kwargs)
+
+ def addSuccess(self, test, *args, **kwargs):
+ self._addResult(self.result.addSuccess, test, *args, **kwargs)
+
+ def addExpectedFailure(self, test, *args, **kwargs):
+ self._addResult(self.result.addExpectedFailure, test, *args, **kwargs)
+
+ def addUnexpectedSuccess(self, test, *args, **kwargs):
+ self._addResult(self.result.addUnexpectedSuccess, test, *args, **kwargs)
+
+ def __getattr__(self, attr):
+ return getattr(self.result, attr)
+
+class ExtraResultsDecoderTestResult(ProxyTestResult):
+ def _addResult(self, method, test, *args, **kwargs):
+ if "details" in kwargs and "extraresults" in kwargs["details"]:
+ if isinstance(kwargs["details"]["extraresults"], Content):
+ kwargs = kwargs.copy()
+ kwargs["details"] = kwargs["details"].copy()
+ extraresults = kwargs["details"]["extraresults"]
+ data = bytearray()
+ for b in extraresults.iter_bytes():
+ data += b
+ extraresults = json.loads(data.decode())
+ kwargs["details"]["extraresults"] = extraresults
+ return method(test, *args, **kwargs)
+
+class ExtraResultsEncoderTestResult(ProxyTestResult):
+ def _addResult(self, method, test, *args, **kwargs):
+ if hasattr(test, "extraresults"):
+ extras = lambda : [json.dumps(test.extraresults).encode()]
+ kwargs = kwargs.copy()
+ if "details" not in kwargs:
+ kwargs["details"] = {}
+ else:
+ kwargs["details"] = kwargs["details"].copy()
+ kwargs["details"]["extraresults"] = Content(ContentType("application", "json", {'charset': 'utf8'}), extras)
+ return method(test, *args, **kwargs)
+
#
# We have to patch subunit since it doesn't understand how to handle addError
# outside of a running test case. This can happen if classSetUp() fails
@@ -116,7 +171,9 @@ class ConcurrentTestSuite(unittest.TestSuite):
result.threadprogress = {}
for i, (test, testnum) in enumerate(tests):
result.threadprogress[i] = []
- process_result = BBThreadsafeForwardingResult(result, semaphore, i, testnum, totaltests)
+ process_result = BBThreadsafeForwardingResult(
+ ExtraResultsDecoderTestResult(result),
+ semaphore, i, testnum, totaltests)
# Force buffering of stdout/stderr so the console doesn't get corrupted by test output
# as per default in parent code
process_result.buffer = True
@@ -231,7 +288,7 @@ def fork_for_tests(concurrency_num, suite):
# as per default in parent code
subunit_client.buffer = True
subunit_result = AutoTimingTestResultDecorator(subunit_client)
- process_suite.run(subunit_result)
+ process_suite.run(ExtraResultsEncoderTestResult(subunit_result))
if ourpid != os.getpid():
os._exit(0)
if newbuilddir: