From da295319aa739dfa657fb4a1ba7e81804ba48658 Mon Sep 17 00:00:00 2001 From: Patrick Williams Date: Tue, 5 Dec 2023 16:48:56 -0600 Subject: subtree updates meta-arm: 1dff3300fb..0b61cc659a: Ross Burton (1): meta-arm/selftest: add test that PAC/BTI instructions are used meta-openembedded: 991e6852a5..5ad7203f68: Alexander Kanavin (1): fio: revert "fio: upgrade 3.32 -> 2022" BELOUARGA Mohamed (1): monocypher: add crypto library recipe Dylan Turner (1): apache2: v2.4.57 to v2.4.58 to fix CVE-2023-43622 Hongxu Jia (1): freeradius: make sub packages to runtime depends on freeradius Kai Kang (1): libnma: remove conflict xml file Khem Raj (12): nlohmann-json: Fix SRCREV_FORMAT and do not package git metadata into ptests ptest-packagelists-meta-oe.inc: Move poco to slow tests sdbus-c++-libsystemd: Upgrade to 254 sdbus-c++-tools: Upgrade to 1.4.0 gstd: Fix systemd user unit packaging basu: Update to latest master sdbus-c++: Install ptests into PTEST_PATH liblognorm:Add asprintf to autoconf function check macro gnome-console,gnome-terminal: Depend on vte from core layer Revert "gnome-terminal: Remove recommendation on vte-prompt" vte9: Drop recipe basu: Update the SRCREV to get lld fix Luca Fancellu (1): linuxptp: Update downstream patches Markus Volk (9): libcacard: fix version string in libcacard.pc cups-filters: fix Makefile race condition system-config-printer: Add packageconfig for polkit pipewire: upgrade 0.3.85 > 1.0.0 libcacard: set meson version based on PV spice: Set meson version based on PV spice-gtk: Set meson version based on PV libdecor: update 0.2.0 -> 0.2.1 xdg-desktop-portal-gnome: upgrade 45.0 -> 45.1 Naveen Saini (2): tbb: upgrade 2021.9.0 -> 2021.11.0 tbb: enable NUMA/Hybrid CPU support Patrick Wicki (6): squid: update from v5.7 to v6.5 squid: add nm dispatcher reload hook squid: add auth packageconfig squid: move configs to sub package squid: add url-rewrite-helpers packageconfig squid: add systemd service Patrick Williams (1): glog: Disable 64bit atomics on armv{5,6} Peter Kjellerstedt (1): redis: Inherit pkgconfig Ross Burton (1): python3-validators: add new recipe Wang Mingyu (26): ctags: upgrade 6.0.20231119.0 -> 6.0.20231126.0 dnfdragora: upgrade 2.1.4 -> 2.1.5 gensio: upgrade 2.7.7 -> 2.8.0 frr: upgrade 9.0.1 -> 9.1 capnproto: upgrade 1.0.1 -> 1.0.1.1 libbpf: upgrade 1.2.2 -> 1.3.0 paho-mqtt-cpp: upgrade 1.2.0 -> 1.3.1 tomoyo-tools: upgrade 2.5.0 -> 2.6.1 python3-aiohttp: upgrade 3.9.0 -> 3.9.1 python3-bitstring: upgrade 4.1.2 -> 4.1.3 python3-dbus-fast: upgrade 2.14.0 -> 2.15.0 python3-humanize: upgrade 4.8.0 -> 4.9.0 python3-ipython: upgrade 8.17.2 -> 8.18.0 python3-mypy: upgrade 1.7.0 -> 1.7.1 python3-pdm: upgrade 2.10.3 -> 2.10.4 python3-pexpect: upgrade 4.8.0 -> 4.9.0 python3-pychromecast: upgrade 13.0.7 -> 13.0.8 python3-pydantic: upgrade 2.5.1 -> 2.5.2 python3-pymisp: upgrade 2.4.178 -> 2.4.179 python3-pytest-xdist: upgrade 3.4.0 -> 3.5.0 python3-sentry-sdk: upgrade 1.35.0 -> 1.37.1 python3-types-setuptools: upgrade 68.2.0.1 -> 68.2.0.2 python3-virtualenv: upgrade 20.24.6 -> 20.24.7 redis: upgrade 7.2.2 -> 7.2.3 ser2net: upgrade 4.5.1 -> 4.6.0 thingsboard-gateway: upgrade 3.4.2 -> 3.4.3.1 alperak (12): squashfs-tools-ng: upgrade 1.1.4 -> 1.2.0 tmate: Fix finding msgpack 6+ msgpack-c: upgrade 4.0.0 -> 6.0.0 msgpack-cpp: upgrade 4.1.1 -> 6.1.0 brotli: upgrade 1.0.9 -> 1.1.0 icewm: upgrade 2.9.9 -> 3.4.4 iotop: upgrade 1.21 -> 1.25 liblognorm: upgrade 1.0.1 -> 2.0.6 libmodbus: upgrade 3.1.7 -> 3.1.10 libpwquality: upgrade 1.4.4 -> 1.4.5 libspiro: upgrade 20200505 -> 20221101 gtkwave: upgrade 3.3.111 -> 3.3.117 poky: 2696bf8cf3..028b6f6226: Adrian Freihofer (1): cmake-qemu.bbclass: support qemu for cmake Alassane Yattara (9): bitbake: toaster/tests: Update methods wait_until_~ to skip using time.sleep bitbake: toaster/tests: Override table edit columns TestCase from image recipe page bitbake: toaster/tests: Test software recipe page bitbake: toaster/tests: Added Machine page TestCase bitbake: toaster/tests: Added Layers page TestCase bitbake: toaster/tests: Added distro page TestCase bitbake: toaster/tests: Bug-fix on tests/functional/test_project_page bitbake: toaster/tests: Test single layer page bitbake: toaster/tests: Test single recipe page Alex Kiernan (4): rust: Delete python2 configparser code path rust: Drop TARGET_VENDOR export eudev: Upgrade 3.2.12 -> 3.2.14 rust: Drop targets and hosts override magic Alexander Kanavin (15): python3-pyproject-hooks: fix upstream version check cmake: upgrade 3.27.5 -> 3.27.7 desktop-file-utils: upgrade 0.26 -> 0.27 erofs-utils: upgrade 1.6 -> 1.7.1 webkitgtk: update 2.40.5 -> 2.42.2 epiphany: upgrade 44.6 -> 45.1 virglrenderer: upgrade 0.10.4 -> 1.0.0 libxkbcommon: upgrade 1.5.0 -> 1.6.0 mpg123: upgrade 1.31.3 -> 1.32.3 icu: upgrade 73-2 -> 74-1 p11-kit: upgrade 0.25.0 -> 0.25.2 glib-2.0: install gio-querymodules into bindir as well as libexecdir for native meson: update 1.2.2 -> 1.3.0 repo: update 2.37 -> 2.39 rt-tests: update 2.5 -> 2.6 Bruce Ashfield (1): lttng-modules: fix build for v6.7+ Changhyeok Bae (1): iptables: upgrade 1.8.9 -> 1.8.10 Charlie Johnston (2): bitbake.conf: Add gsutil as hosttool for gcp fetcher. bitbake: fetch2: Ensure GCP fetcher checks if file exists before download. Jan Vermaete (1): systemd: fixed typo Joao Marcos Costa (1): documentation.conf: fix do_menuconfig description Joshua Watt (2): bitbake: bitbake-hashclient: Add commands to get hashes bitbake: hashserv: sqlite: Ensure sync propagates to database connections Julien Stephan (6): devtool: fix update-recipe dry-run mode lib/oe/recipeutils.py: remove trailing white-spaces devtool: finish/update-recipe: restrict mode srcrev to recipes fetched from SCM devtool: tag all submodules devtool: add support for git submodules oeqa/selftest/devtool: add test for git submodules Justin Bronder (1): contributor-guide: add License-Update tag Kareem Zarka (2): wic: bootimg-efi: Make kernel image installation configurable oeqa/selftest/wic: Add tests for kernel image installation Khem Raj (8): shared-mime-info: Fix build with clang-17+ libsoup-2.4: Fix build with clang-17 and libxml2-2.12 busybox: Enable utmp support on musl systems virglrenderer: Fix build with clang llvm: Upgrade to 17.0.6 rust-common.bbclass: Define rust arch for x32 platforms vte: Upgrade to 0.74.1 vte: Separate out gtk4 pieces of vte into individual packages Lee Chee Yang (3): wic: add test for partition hidden attributes migration-guides: add release notes for 4.3.1 openssl: upgrade to 3.2.0 Malte Schmidt (1): wic: rawcopy: add support for zstd decompression Marco Felsch (1): json-c: fix icecc compilation Markus Volk (3): bluez5: fix connection for ps5/dualshock controllers cups: Add root,sys,wheel to system groups vte: upgrade 0.72.2 -> 0.74.0 Martin Hundeb?ll (1): libpam: split /etc/environment into pam-plugin-env package Matsunaga-Shinji (1): cve-check: Modify judgment processing using "=" in version comparison Michael Opdenacker (4): systemd-compat-units.bb: fix postinstall script dev-manual: layers: update link to YP Compatible form contributor-guide: fix command option migration-guides: release 3.5 is actually 4.0 Niko Mauno (1): rust-llvm: Allow overriding LLVM target archs Patrick Williams (1): shared-mime-info-native: handle old GCC for AlmaLinux8 Peter Marko (2): cve-update-nvd2-native: remove unused variable CVE_SOCKET_TIMEOUT cve-update-nvd2-native: make number of fetch attemtps configurable Richard Haar (1): bitbake: bitbake: tests: Fix duplicate test_underscore_override test Richard Purdie (2): bitbake: ui/ncurses: Add missing function call to avoid traceback bitbake: cooker: Avoid eventlog variable listing lockups Robert Yang (2): gnu-config: Update to latest revision gettext: Upgrade 0.22 -> 0.22.3 Ross Burton (3): core-image-minimal-initramfs: don't install a kernel into the initramfs autoconf: upgrade to 2.72d Revert "cve-check: Modify judgment processing using "=" in version comparison" Sundeep KOKKONDA (3): rust: Split rustdoc into a separate package glibc: stable 2.38 branch updates binutils: stable 2.41 branch updates Tim Orling (8): python3-sphinxcontrib-applehelp: 1.0.4 -> 1.0.7 python3-sphinxcontrib-devhelp: 1.0.2 -> 1.0.5 python3-sphinxcontrib-htmlhelp: 2.0.1 -> 2.0.4 python3-sphinxcontrib-qthelp: 1.0.3 -> 1.0.6 python3-sphinxcontrib-serializinghtml: 1.1.5 -> 1.1.9 vim: upgrade 9.0.2068 -> 9.0.2130 python3-cryptography-vectors: add RECIPE_NO_UPDATE_REASON python3-cryptography{-vectors}: 41.0.5 -> 41.0.7 Trevor Gamblin (2): python3-ptest: skip test_storlines patchtest: shorten patch signed-off-by test output Viswanath Kraleti (1): systemd-boot: Fix build issues on armv7a-linux Wang Mingyu (27): bind: upgrade 9.18.19 -> 9.18.20 diffoscope: upgrade 251 -> 252 ell: upgrade 0.59 -> 0.60 git: upgrade 2.42.1 -> 2.43.0 gnutls: upgrade 3.8.1 -> 3.8.2 libdrm: upgrade 2.4.117 -> 2.4.118 libgcrypt: upgrade 1.10.2 -> 1.10.3 libksba: upgrade 1.6.4 -> 1.6.5 libxslt: upgrade 1.1.38 -> 1.1.39 log4cplus: upgrade 2.1.0 -> 2.1.1 python3-certifi: upgrade 2023.7.22 -> 2023.11.17 python3-setuptools: upgrade 68.2.2 -> 69.0.2 python3-wcwidth: upgrade 0.2.9 -> 0.2.11 python3-hypothesis: upgrade 6.89.0 -> 6.90.0 python3-pyasn1: upgrade 0.5.0 -> 0.5.1 python3-scons: upgrade 4.5.2 -> 4.6.0 python3-urllib3: upgrade 2.0.7 -> 2.1.0 ethtool: upgrade 6.5 -> 6.6 gi-docgen: upgrade 2023.1 -> 2023.3 init-system-helpers: upgrade 1.65.2 -> 1.66 libsolv: upgrade 0.7.26 -> 0.7.27 python3-idna: upgrade 3.4 -> 3.6 ofono: upgrade 2.1 -> 2.2 python3-sphinx-rtd-theme: upgrade 1.3.0 -> 2.0.0 python3-trove-classifiers: upgrade 2023.11.14 -> 2023.11.22 python3-wheel: upgrade 0.41.3 -> 0.42.0 resolvconf: upgrade 1.91 -> 1.92 Xiangyu Chen (2): shadow: Fix for CVE-2023-4641 bash: changes to SIGINT handler while waiting for a child Zahir Hussain (1): cmake: Unset CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES meta-raspberrypi: 8231f97534..fde68b24f0: Lorenzo Arena (1): docs: fix syntax for overriding fs type for initramfs image Change-Id: Idc6f6b1e913442bae03dfec9f207924c56f31056 Signed-off-by: Patrick Williams --- poky/bitbake/bin/bitbake-hashclient | 27 + poky/bitbake/lib/bb/cooker.py | 36 +- poky/bitbake/lib/bb/fetch2/gcp.py | 7 +- poky/bitbake/lib/bb/tests/data.py | 20 +- poky/bitbake/lib/bb/ui/ncurses.py | 3 + poky/bitbake/lib/hashserv/sqlite.py | 9 +- poky/bitbake/lib/hashserv/tests.py | 20 + .../toaster/tests/browser/selenium_helpers_base.py | 13 +- .../toaster/tests/functional/test_project_page.py | 553 ++++++++++++++++++++- 9 files changed, 639 insertions(+), 49 deletions(-) (limited to 'poky/bitbake') diff --git a/poky/bitbake/bin/bitbake-hashclient b/poky/bitbake/bin/bitbake-hashclient index 3ff7b76378..2cb6338666 100755 --- a/poky/bitbake/bin/bitbake-hashclient +++ b/poky/bitbake/bin/bitbake-hashclient @@ -52,6 +52,22 @@ def print_user(u): def main(): + def handle_get(args, client): + result = client.get_taskhash(args.method, args.taskhash, all_properties=True) + if not result: + return 0 + + print(json.dumps(result, sort_keys=True, indent=4)) + return 0 + + def handle_get_outhash(args, client): + result = client.get_outhash(args.method, args.outhash, args.taskhash) + if not result: + return 0 + + print(json.dumps(result, sort_keys=True, indent=4)) + return 0 + def handle_stats(args, client): if args.reset: s = client.reset_stats() @@ -189,6 +205,17 @@ def main(): subparsers = parser.add_subparsers() + get_parser = subparsers.add_parser('get', help="Get the unihash for a taskhash") + get_parser.add_argument("method", help="Method to query") + get_parser.add_argument("taskhash", help="Task hash to query") + get_parser.set_defaults(func=handle_get) + + get_outhash_parser = subparsers.add_parser('get-outhash', help="Get output hash information") + get_outhash_parser.add_argument("method", help="Method to query") + get_outhash_parser.add_argument("outhash", help="Output hash to query") + get_outhash_parser.add_argument("taskhash", help="Task hash to query") + get_outhash_parser.set_defaults(func=handle_get_outhash) + stats_parser = subparsers.add_parser('stats', help='Show server stats') stats_parser.add_argument('--reset', action='store_true', help='Reset server stats') diff --git a/poky/bitbake/lib/bb/cooker.py b/poky/bitbake/lib/bb/cooker.py index d658db9bd8..c5bfef55d6 100644 --- a/poky/bitbake/lib/bb/cooker.py +++ b/poky/bitbake/lib/bb/cooker.py @@ -102,12 +102,15 @@ class CookerFeatures(object): class EventWriter: def __init__(self, cooker, eventfile): - self.file_inited = None self.cooker = cooker self.eventfile = eventfile self.event_queue = [] - def write_event(self, event): + def write_variables(self): + with open(self.eventfile, "a") as f: + f.write("%s\n" % json.dumps({ "allvariables" : self.cooker.getAllKeysWithFlags(["doc", "func"])})) + + def send(self, event): with open(self.eventfile, "a") as f: try: str_event = codecs.encode(pickle.dumps(event), 'base64').decode('utf-8') @@ -117,28 +120,6 @@ class EventWriter: import traceback print(err, traceback.format_exc()) - def send(self, event): - if self.file_inited: - # we have the file, just write the event - self.write_event(event) - else: - # init on bb.event.BuildStarted - name = "%s.%s" % (event.__module__, event.__class__.__name__) - if name in ("bb.event.BuildStarted", "bb.cooker.CookerExit"): - with open(self.eventfile, "w") as f: - f.write("%s\n" % json.dumps({ "allvariables" : self.cooker.getAllKeysWithFlags(["doc", "func"])})) - - self.file_inited = True - - # write pending events - for evt in self.event_queue: - self.write_event(evt) - - # also write the current event - self.write_event(event) - else: - # queue all events until the file is inited - self.event_queue.append(event) #============================================================================# # BBCooker @@ -416,6 +397,7 @@ class BBCooker: def setupEventLog(self, eventlog): if self.eventlog and self.eventlog[0] != eventlog: bb.event.unregister_UIHhandler(self.eventlog[1]) + self.eventlog = None if not self.eventlog or self.eventlog[0] != eventlog: # we log all events to a file if so directed # register the log file writer as UI Handler @@ -423,7 +405,7 @@ class BBCooker: bb.utils.mkdirhier(os.path.dirname(eventlog)) writer = EventWriter(self, eventlog) EventLogWriteHandler = namedtuple('EventLogWriteHandler', ['event']) - self.eventlog = (eventlog, bb.event.register_UIHhandler(EventLogWriteHandler(writer))) + self.eventlog = (eventlog, bb.event.register_UIHhandler(EventLogWriteHandler(writer)), writer) def updateConfigOpts(self, options, environment, cmdline): self.ui_cmdline = cmdline @@ -1404,6 +1386,8 @@ class BBCooker: buildname = self.databuilder.mcdata[mc].getVar("BUILDNAME") if fireevents: bb.event.fire(bb.event.BuildStarted(buildname, [item]), self.databuilder.mcdata[mc]) + if self.eventlog: + self.eventlog[2].write_variables() bb.event.enable_heartbeat() # Execute the runqueue @@ -1547,6 +1531,8 @@ class BBCooker: for mc in self.multiconfigs: bb.event.fire(bb.event.BuildStarted(buildname, ntargets), self.databuilder.mcdata[mc]) + if self.eventlog: + self.eventlog[2].write_variables() bb.event.enable_heartbeat() rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist) diff --git a/poky/bitbake/lib/bb/fetch2/gcp.py b/poky/bitbake/lib/bb/fetch2/gcp.py index f42c81fda8..f40ce2eaa5 100644 --- a/poky/bitbake/lib/bb/fetch2/gcp.py +++ b/poky/bitbake/lib/bb/fetch2/gcp.py @@ -47,6 +47,7 @@ class GCP(FetchMethod): ud.basename = os.path.basename(ud.path) ud.localfile = d.expand(urllib.parse.unquote(ud.basename)) + ud.basecmd = "gsutil stat" def get_gcp_client(self): from google.cloud import storage @@ -61,7 +62,8 @@ class GCP(FetchMethod): if self.gcp_client is None: self.get_gcp_client() - bb.fetch2.check_network_access(d, "gsutil stat", ud.url) + bb.fetch2.check_network_access(d, ud.basecmd, f"gs://{ud.host}{ud.path}") + runfetchcmd("%s %s" % (ud.basecmd, f"gs://{ud.host}{ud.path}"), d) # Path sometimes has leading slash, so strip it path = ud.path.lstrip("/") @@ -88,7 +90,8 @@ class GCP(FetchMethod): if self.gcp_client is None: self.get_gcp_client() - bb.fetch2.check_network_access(d, "gsutil stat", ud.url) + bb.fetch2.check_network_access(d, ud.basecmd, f"gs://{ud.host}{ud.path}") + runfetchcmd("%s %s" % (ud.basecmd, f"gs://{ud.host}{ud.path}"), d) # Path sometimes has leading slash, so strip it path = ud.path.lstrip("/") diff --git a/poky/bitbake/lib/bb/tests/data.py b/poky/bitbake/lib/bb/tests/data.py index 98e430ce2a..cbc7c1ecd4 100644 --- a/poky/bitbake/lib/bb/tests/data.py +++ b/poky/bitbake/lib/bb/tests/data.py @@ -395,6 +395,16 @@ class TestOverrides(unittest.TestCase): self.d.setVar("OVERRIDES", "foo:bar:some_val") self.assertEqual(self.d.getVar("TEST"), "testvalue3") + # Test an override with _ in it based on a real world OE issue + def test_underscore_override_2(self): + self.d.setVar("TARGET_ARCH", "x86_64") + self.d.setVar("PN", "test-${TARGET_ARCH}") + self.d.setVar("VERSION", "1") + self.d.setVar("VERSION:pn-test-${TARGET_ARCH}", "2") + self.d.setVar("OVERRIDES", "pn-${PN}") + bb.data.expandKeys(self.d) + self.assertEqual(self.d.getVar("VERSION"), "2") + def test_remove_with_override(self): self.d.setVar("TEST:bar", "testvalue2") self.d.setVar("TEST:some_val", "testvalue3 testvalue5") @@ -416,16 +426,6 @@ class TestOverrides(unittest.TestCase): self.d.setVar("TEST:bar:append", "testvalue2") self.assertEqual(self.d.getVar("TEST"), "testvalue2") - # Test an override with _ in it based on a real world OE issue - def test_underscore_override(self): - self.d.setVar("TARGET_ARCH", "x86_64") - self.d.setVar("PN", "test-${TARGET_ARCH}") - self.d.setVar("VERSION", "1") - self.d.setVar("VERSION:pn-test-${TARGET_ARCH}", "2") - self.d.setVar("OVERRIDES", "pn-${PN}") - bb.data.expandKeys(self.d) - self.assertEqual(self.d.getVar("VERSION"), "2") - def test_append_and_unused_override(self): # Had a bug where an unused override append could return "" instead of None self.d.setVar("BAR:append:unusedoverride", "testvalue2") diff --git a/poky/bitbake/lib/bb/ui/ncurses.py b/poky/bitbake/lib/bb/ui/ncurses.py index cf1c876a51..18a706547a 100644 --- a/poky/bitbake/lib/bb/ui/ncurses.py +++ b/poky/bitbake/lib/bb/ui/ncurses.py @@ -227,6 +227,9 @@ class NCursesUI: shutdown = 0 try: + if not params.observe_only: + params.updateToServer(server, os.environ.copy()) + params.updateFromServer(server) cmdline = params.parseActions() if not cmdline: diff --git a/poky/bitbake/lib/hashserv/sqlite.py b/poky/bitbake/lib/hashserv/sqlite.py index f65036be93..f93cb2c1dd 100644 --- a/poky/bitbake/lib/hashserv/sqlite.py +++ b/poky/bitbake/lib/hashserv/sqlite.py @@ -109,11 +109,11 @@ class DatabaseEngine(object): ) def connect(self, logger): - return Database(logger, self.dbname) + return Database(logger, self.dbname, self.sync) class Database(object): - def __init__(self, logger, dbname, sync=True): + def __init__(self, logger, dbname, sync): self.dbname = dbname self.logger = logger @@ -121,6 +121,11 @@ class Database(object): self.db.row_factory = sqlite3.Row with closing(self.db.cursor()) as cursor: + cursor.execute("PRAGMA journal_mode = WAL") + cursor.execute( + "PRAGMA synchronous = %s" % ("NORMAL" if sync else "OFF") + ) + cursor.execute("SELECT sqlite_version()") version = [] diff --git a/poky/bitbake/lib/hashserv/tests.py b/poky/bitbake/lib/hashserv/tests.py index a9e6fdf9ff..869f7636c5 100644 --- a/poky/bitbake/lib/hashserv/tests.py +++ b/poky/bitbake/lib/hashserv/tests.py @@ -842,6 +842,26 @@ class TestHashEquivalenceClient(HashEquivalenceTestSetup, unittest.TestCase): def get_server_addr(self, server_idx): return "unix://" + os.path.join(self.temp_dir.name, 'sock%d' % server_idx) + def test_get(self): + taskhash, outhash, unihash = self.create_test_hash(self.client) + + p = self.run_hashclient(["--address", self.server_address, "get", self.METHOD, taskhash]) + data = json.loads(p.stdout) + self.assertEqual(data["unihash"], unihash) + self.assertEqual(data["outhash"], outhash) + self.assertEqual(data["taskhash"], taskhash) + self.assertEqual(data["method"], self.METHOD) + + def test_get_outhash(self): + taskhash, outhash, unihash = self.create_test_hash(self.client) + + p = self.run_hashclient(["--address", self.server_address, "get-outhash", self.METHOD, outhash, taskhash]) + data = json.loads(p.stdout) + self.assertEqual(data["unihash"], unihash) + self.assertEqual(data["outhash"], outhash) + self.assertEqual(data["taskhash"], taskhash) + self.assertEqual(data["method"], self.METHOD) + def test_stats(self): p = self.run_hashclient(["--address", self.server_address, "stats"], check=True) json.loads(p.stdout) diff --git a/poky/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py b/poky/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py index e0ac43768e..d9ea7fd108 100644 --- a/poky/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py +++ b/poky/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py @@ -71,7 +71,9 @@ class Wait(WebDriverWait): _TIMEOUT = 10 _POLL_FREQUENCY = 0.5 - def __init__(self, driver): + def __init__(self, driver, timeout=_TIMEOUT, poll=_POLL_FREQUENCY): + self._TIMEOUT = timeout + self._POLL_FREQUENCY = poll super(Wait, self).__init__(driver, self._TIMEOUT, self._POLL_FREQUENCY) def until(self, method, message=''): @@ -175,18 +177,19 @@ class SeleniumTestCaseBase(unittest.TestCase): """ Return the element which currently has focus on the page """ return self.driver.switch_to.active_element - def wait_until_present(self, selector): + def wait_until_present(self, selector, poll=0.5): """ Wait until element matching CSS selector is on the page """ is_present = lambda driver: self.find(selector) msg = 'An element matching "%s" should be on the page' % selector - element = Wait(self.driver).until(is_present, msg) + element = Wait(self.driver, poll=poll).until(is_present, msg) return element - def wait_until_visible(self, selector): + def wait_until_visible(self, selector, poll=1): """ Wait until element matching CSS selector is visible on the page """ is_visible = lambda driver: self.find(selector).is_displayed() msg = 'An element matching "%s" should be visible' % selector - Wait(self.driver).until(is_visible, msg) + Wait(self.driver, poll=poll).until(is_visible, msg) + time.sleep(poll) # wait for visibility to settle return self.find(selector) def wait_until_focused(self, selector): diff --git a/poky/bitbake/lib/toaster/tests/functional/test_project_page.py b/poky/bitbake/lib/toaster/tests/functional/test_project_page.py index 3edf967a2c..03f64f8fef 100644 --- a/poky/bitbake/lib/toaster/tests/functional/test_project_page.py +++ b/poky/bitbake/lib/toaster/tests/functional/test_project_page.py @@ -6,10 +6,17 @@ # SPDX-License-Identifier: GPL-2.0-only # +import random +import string import pytest +from time import sleep from django.urls import reverse +from django.utils import timezone +from selenium.webdriver.common.keys import Keys from selenium.webdriver.support.select import Select +from selenium.common.exceptions import NoSuchElementException, TimeoutException from tests.functional.functional_helpers import SeleniumFunctionalTestCase +from orm.models import Build, Project, Target from selenium.webdriver.common.by import By @@ -19,13 +26,18 @@ class TestProjectPage(SeleniumFunctionalTestCase): def setUp(self): super().setUp() release = '3' - project_name = 'projectmaster' + project_name = 'project_' + self.generate_random_string() self._create_test_new_project( project_name, release, False, ) + def generate_random_string(self, length=10): + characters = string.ascii_letters + string.digits # alphabetic and numerical characters + random_string = ''.join(random.choice(characters) for _ in range(length)) + return random_string + def _create_test_new_project( self, project_name, @@ -55,6 +67,205 @@ class TestProjectPage(SeleniumFunctionalTestCase): self.driver.find_element(By.ID, "create-project-button").click() + def _get_create_builds(self, **kwargs): + """ Create a build and return the build object """ + # parameters for builds to associate with the projects + now = timezone.now() + release = '3' + project_name = 'projectmaster' + self._create_test_new_project( + project_name+"2", + release, + False, + ) + + self.project1_build_success = { + 'project': Project.objects.get(id=1), + 'started_on': now, + 'completed_on': now, + 'outcome': Build.SUCCEEDED + } + + self.project1_build_failure = { + 'project': Project.objects.get(id=1), + 'started_on': now, + 'completed_on': now, + 'outcome': Build.FAILED + } + build1 = Build.objects.create(**self.project1_build_success) + build2 = Build.objects.create(**self.project1_build_failure) + + # add some targets to these builds so they have recipe links + # (and so we can find the row in the ToasterTable corresponding to + # a particular build) + Target.objects.create(build=build1, target='foo') + Target.objects.create(build=build2, target='bar') + + if kwargs: + # Create kwargs.get('success') builds with success status with target + # and kwargs.get('failure') builds with failure status with target + for i in range(kwargs.get('success', 0)): + now = timezone.now() + self.project1_build_success['started_on'] = now + self.project1_build_success[ + 'completed_on'] = now - timezone.timedelta(days=i) + build = Build.objects.create(**self.project1_build_success) + Target.objects.create(build=build, + target=f'{i}_success_recipe', + task=f'{i}_success_task') + + for i in range(kwargs.get('failure', 0)): + now = timezone.now() + self.project1_build_failure['started_on'] = now + self.project1_build_failure[ + 'completed_on'] = now - timezone.timedelta(days=i) + build = Build.objects.create(**self.project1_build_failure) + Target.objects.create(build=build, + target=f'{i}_fail_recipe', + task=f'{i}_fail_task') + return build1, build2 + + def _mixin_test_table_edit_column( + self, + table_id, + edit_btn_id, + list_check_box_id: list + ): + # Check edit column + edit_column = self.find(f'#{edit_btn_id}') + self.assertTrue(edit_column.is_displayed()) + edit_column.click() + # Check dropdown is visible + self.wait_until_visible('ul.dropdown-menu.editcol') + for check_box_id in list_check_box_id: + # Check that we can hide/show table column + check_box = self.find(f'#{check_box_id}') + th_class = str(check_box_id).replace('checkbox-', '') + if check_box.is_selected(): + # check if column is visible in table + self.assertTrue( + self.find( + f'#{table_id} thead th.{th_class}' + ).is_displayed(), + f"The {th_class} column is checked in EditColumn dropdown, but it's not visible in table" + ) + check_box.click() + # check if column is hidden in table + self.assertFalse( + self.find( + f'#{table_id} thead th.{th_class}' + ).is_displayed(), + f"The {th_class} column is unchecked in EditColumn dropdown, but it's visible in table" + ) + else: + # check if column is hidden in table + self.assertFalse( + self.find( + f'#{table_id} thead th.{th_class}' + ).is_displayed(), + f"The {th_class} column is unchecked in EditColumn dropdown, but it's visible in table" + ) + check_box.click() + # check if column is visible in table + self.assertTrue( + self.find( + f'#{table_id} thead th.{th_class}' + ).is_displayed(), + f"The {th_class} column is checked in EditColumn dropdown, but it's not visible in table" + ) + + def _get_config_nav_item(self, index): + config_nav = self.find('#config-nav') + return config_nav.find_elements(By.TAG_NAME, 'li')[index] + + def _navigate_to_config_nav(self, nav_id, nav_index): + # navigate to the project page + url = reverse("project", args=(1,)) + self.get(url) + self.wait_until_visible('#config-nav') + # click on "Software recipe" tab + soft_recipe = self._get_config_nav_item(nav_index) + soft_recipe.click() + self.wait_until_visible(f'#{nav_id}') + + def _mixin_test_table_show_rows(self, table_selector, **kwargs): + """ Test the show rows feature in the builds table on the all builds page """ + def test_show_rows(row_to_show, show_row_link): + # Check that we can show rows == row_to_show + show_row_link.select_by_value(str(row_to_show)) + self.wait_until_visible(f'#{table_selector} tbody tr', poll=2) + self.assertTrue( + len(self.find_all(f'#{table_selector} tbody tr')) == row_to_show + ) + self.wait_until_present(f'#{table_selector} tbody tr') + show_rows = self.driver.find_elements( + By.XPATH, + f'//select[@class="form-control pagesize-{table_selector}"]' + ) + rows_to_show = [10, 25, 50, 100, 150] + to_skip = kwargs.get('to_skip', []) + # Check show rows + for show_row_link in show_rows: + show_row_link = Select(show_row_link) + for row_to_show in rows_to_show: + if row_to_show not in to_skip: + test_show_rows(row_to_show, show_row_link) + + def _wait_until_build(self, state): + timeout = 10 + start_time = 0 + while True: + if start_time > timeout: + raise TimeoutException( + f'Build did not reach {state} state within {timeout} seconds' + ) + try: + last_build_state = self.driver.find_element( + By.XPATH, + '//*[@id="latest-builds"]/div[1]//div[@class="build-state"]', + ) + build_state = last_build_state.get_attribute( + 'data-build-state') + state_text = state.lower().split() + if any(x in str(build_state).lower() for x in state_text): + break + except NoSuchElementException: + continue + start_time += 1 + sleep(1) # take a breath and try again + + def _mixin_test_table_search_input(self, **kwargs): + input_selector, input_text, searchBtn_selector, table_selector, *_ = kwargs.values() + # Test search input + self.wait_until_visible(f'#{input_selector}') + recipe_input = self.find(f'#{input_selector}') + recipe_input.send_keys(input_text) + self.find(f'#{searchBtn_selector}').click() + self.wait_until_visible(f'#{table_selector} tbody tr') + rows = self.find_all(f'#{table_selector} tbody tr') + self.assertTrue(len(rows) > 0) + + def test_image_recipe_editColumn(self): + """ Test the edit column feature in image recipe table on project page """ + self._get_create_builds(success=10, failure=10) + + url = reverse('projectimagerecipes', args=(1,)) + self.get(url) + self.wait_until_present('#imagerecipestable tbody tr') + + column_list = [ + 'get_description_or_summary', 'layer_version__get_vcs_reference', + 'layer_version__layer__name', 'license', 'recipe-file', 'section', + 'version' + ] + + # Check that we can hide the edit column + self._mixin_test_table_edit_column( + 'imagerecipestable', + 'edit-columns-button', + [f'checkbox-{column}' for column in column_list] + ) + def test_page_header_on_project_page(self): """ Check page header in project page: - AT LEFT -> Logo of Yocto project, displayed, clickable @@ -184,11 +395,9 @@ class TestProjectPage(SeleniumFunctionalTestCase): self.wait_until_visible('#topbar-configuration-tab') config_tab = self.find('#topbar-configuration-tab') self.assertTrue(config_tab.get_attribute('class') == 'active') - self.assertTrue('Configuration' in config_tab.text) - config_tab_link = config_tab.find_element(By.TAG_NAME, 'a') + self.assertTrue('Configuration' in str(config_tab.text)) self.assertTrue( - f"/toastergui/project/1" in str(config_tab_link.get_attribute( - 'href')) + f"/toastergui/project/1" in str(self.driver.current_url) ) def get_tabs(): @@ -245,3 +454,337 @@ class TestProjectPage(SeleniumFunctionalTestCase): self.assertTrue( 'core-image-minimal' in str(last_build.text) ) + + def test_softwareRecipe_page(self): + """ Test software recipe page + - Check title "Compatible software recipes" is displayed + - Check search input + - Check "build recipe" button works + - Check software recipe table feature(show/hide column, pagination) + """ + self._navigate_to_config_nav('softwarerecipestable', 4) + # check title "Compatible software recipes" is displayed + self.assertTrue("Compatible software recipes" in self.get_page_source()) + # Test search input + self._mixin_test_table_search_input( + input_selector='search-input-softwarerecipestable', + input_text='busybox', + searchBtn_selector='search-submit-softwarerecipestable', + table_selector='softwarerecipestable' + ) + # check "build recipe" button works + rows = self.find_all('#softwarerecipestable tbody tr') + image_to_build = rows[0] + build_btn = image_to_build.find_element( + By.XPATH, + '//td[@class="add-del-layers"]//a[1]' + ) + build_btn.click() + self._wait_until_build('parsing starting cloning queued') + lastest_builds = self.driver.find_elements( + By.XPATH, + '//div[@id="latest-builds"]/div' + ) + self.assertTrue(len(lastest_builds) > 0) + + # check software recipe table feature(show/hide column, pagination) + self._navigate_to_config_nav('softwarerecipestable', 4) + column_list = [ + 'get_description_or_summary', + 'layer_version__get_vcs_reference', + 'layer_version__layer__name', + 'license', + 'recipe-file', + 'section', + 'version', + ] + self._mixin_test_table_edit_column( + 'softwarerecipestable', + 'edit-columns-button', + [f'checkbox-{column}' for column in column_list] + ) + self._navigate_to_config_nav('softwarerecipestable', 4) + # check show rows(pagination) + self._mixin_test_table_show_rows(table_selector='softwarerecipestable') + + def test_machines_page(self): + """ Test Machine page + - Check if title "Compatible machines" is displayed + - Check search input + - Check "Select machine" button works + - Check "Add layer" button works + - Check Machine table feature(show/hide column, pagination) + """ + self._navigate_to_config_nav('machinestable', 5) + # check title "Compatible software recipes" is displayed + self.assertTrue("Compatible machines" in self.get_page_source()) + # Test search input + self._mixin_test_table_search_input( + input_selector='search-input-machinestable', + input_text='qemux86-64', + searchBtn_selector='search-submit-machinestable', + table_selector='machinestable' + ) + # check "Select machine" button works + rows = self.find_all('#machinestable tbody tr') + machine_to_select = rows[0] + select_btn = machine_to_select.find_element( + By.XPATH, + '//td[@class="add-del-layers"]//a[1]' + ) + select_btn.send_keys(Keys.RETURN) + self.wait_until_visible('#config-nav') + project_machine_name = self.find('#project-machine-name') + self.assertTrue( + 'qemux86-64' in project_machine_name.text + ) + # check "Add layer" button works + self._navigate_to_config_nav('machinestable', 5) + # Search for a machine whit layer not in project + self._mixin_test_table_search_input( + input_selector='search-input-machinestable', + input_text='qemux86-64-tpm2', + searchBtn_selector='search-submit-machinestable', + table_selector='machinestable' + ) + rows = self.find_all('#machinestable tbody tr') + machine_to_add = rows[0] + add_btn = machine_to_add.find_element(By.XPATH, '//td[@class="add-del-layers"]') + add_btn.click() + self.wait_until_visible('#change-notification') + change_notification = self.find('#change-notification') + self.assertTrue( + f'You have added 1 layer to your project' in str(change_notification.text) + ) + # check Machine table feature(show/hide column, pagination) + self._navigate_to_config_nav('machinestable', 5) + column_list = [ + 'description', + 'layer_version__get_vcs_reference', + 'layer_version__layer__name', + 'machinefile', + ] + self._mixin_test_table_edit_column( + 'machinestable', + 'edit-columns-button', + [f'checkbox-{column}' for column in column_list] + ) + self._navigate_to_config_nav('machinestable', 5) + # check show rows(pagination) + self._mixin_test_table_show_rows(table_selector='machinestable') + + def test_layers_page(self): + """ Test layers page + - Check if title "Compatible layerss" is displayed + - Check search input + - Check "Add layer" button works + - Check "Remove layer" button works + - Check layers table feature(show/hide column, pagination) + """ + self._navigate_to_config_nav('layerstable', 6) + # check title "Compatible layers" is displayed + self.assertTrue("Compatible layers" in self.get_page_source()) + # Test search input + input_text='meta-tanowrt' + self._mixin_test_table_search_input( + input_selector='search-input-layerstable', + input_text=input_text, + searchBtn_selector='search-submit-layerstable', + table_selector='layerstable' + ) + # check "Add layer" button works + rows = self.find_all('#layerstable tbody tr') + layer_to_add = rows[0] + add_btn = layer_to_add.find_element( + By.XPATH, + '//td[@class="add-del-layers"]' + ) + add_btn.click() + # check modal is displayed + self.wait_until_visible('#dependencies-modal', poll=2) + list_dependencies = self.find_all('#dependencies-list li') + # click on add-layers button + add_layers_btn = self.driver.find_element( + By.XPATH, + '//form[@id="dependencies-modal-form"]//button[@class="btn btn-primary"]' + ) + add_layers_btn.click() + self.wait_until_visible('#change-notification') + change_notification = self.find('#change-notification') + self.assertTrue( + f'You have added {len(list_dependencies)+1} layers to your project: {input_text} and its dependencies' in str(change_notification.text) + ) + # check "Remove layer" button works + rows = self.find_all('#layerstable tbody tr') + layer_to_remove = rows[0] + remove_btn = layer_to_remove.find_element( + By.XPATH, + '//td[@class="add-del-layers"]' + ) + remove_btn.click() + self.wait_until_visible('#change-notification', poll=2) + change_notification = self.find('#change-notification') + self.assertTrue( + f'You have removed 1 layer from your project: {input_text}' in str(change_notification.text) + ) + # check layers table feature(show/hide column, pagination) + self._navigate_to_config_nav('layerstable', 6) + column_list = [ + 'dependencies', + 'revision', + 'layer__vcs_url', + 'git_subdir', + 'layer__summary', + ] + self._mixin_test_table_edit_column( + 'layerstable', + 'edit-columns-button', + [f'checkbox-{column}' for column in column_list] + ) + self._navigate_to_config_nav('layerstable', 6) + # check show rows(pagination) + self._mixin_test_table_show_rows(table_selector='layerstable') + + def test_distro_page(self): + """ Test distros page + - Check if title "Compatible distros" is displayed + - Check search input + - Check "Add layer" button works + - Check distro table feature(show/hide column, pagination) + """ + self._navigate_to_config_nav('distrostable', 7) + # check title "Compatible distros" is displayed + self.assertTrue("Compatible Distros" in self.get_page_source()) + # Test search input + input_text='poky-altcfg' + self._mixin_test_table_search_input( + input_selector='search-input-distrostable', + input_text=input_text, + searchBtn_selector='search-submit-distrostable', + table_selector='distrostable' + ) + # check "Add distro" button works + rows = self.find_all('#distrostable tbody tr') + distro_to_add = rows[0] + add_btn = distro_to_add.find_element( + By.XPATH, + '//td[@class="add-del-layers"]//a[1]' + ) + add_btn.click() + self.wait_until_visible('#change-notification', poll=2) + change_notification = self.find('#change-notification') + self.assertTrue( + f'You have changed the distro to: {input_text}' in str(change_notification.text) + ) + # check distro table feature(show/hide column, pagination) + self._navigate_to_config_nav('distrostable', 7) + column_list = [ + 'description', + 'templatefile', + 'layer_version__get_vcs_reference', + 'layer_version__layer__name', + ] + self._mixin_test_table_edit_column( + 'distrostable', + 'edit-columns-button', + [f'checkbox-{column}' for column in column_list] + ) + self._navigate_to_config_nav('distrostable', 7) + # check show rows(pagination) + self._mixin_test_table_show_rows( + table_selector='distrostable', + to_skip=[150] + ) + + def test_single_layer_page(self): + """ Test layer page + - Check if title is displayed + - Check add/remove layer button works + - Check tabs(layers, recipes, machines) are displayed + - Check left section is displayed + - Check layer name + - Check layer summary + - Check layer description + """ + url = reverse("layerdetails", args=(1, 8)) + self.get(url) + self.wait_until_visible('.page-header') + # check title is displayed + self.assertTrue(self.find('.page-header h1').is_displayed()) + + # check add layer button works + remove_layer_btn = self.find('#add-remove-layer-btn') + remove_layer_btn.click() + self.wait_until_visible('#change-notification', poll=2) + change_notification = self.find('#change-notification') + self.assertTrue( + f'You have removed 1 layer from your project' in str(change_notification.text) + ) + # check add layer button works, 18 is the random layer id + add_layer_btn = self.find('#add-remove-layer-btn') + add_layer_btn.click() + self.wait_until_visible('#change-notification') + change_notification = self.find('#change-notification') + self.assertTrue( + f'You have added 1 layer to your project' in str(change_notification.text) + ) + # check tabs(layers, recipes, machines) are displayed + tabs = self.find_all('.nav-tabs li') + self.assertEqual(len(tabs), 3) + # Check first tab + tabs[0].click() + self.assertTrue( + 'active' in str(self.find('#information').get_attribute('class')) + ) + # Check second tab + tabs[1].click() + self.assertTrue( + 'active' in str(self.find('#recipes').get_attribute('class')) + ) + # Check third tab + tabs[2].click() + self.assertTrue( + 'active' in str(self.find('#machines').get_attribute('class')) + ) + # Check left section is displayed + section = self.find('.well') + # Check layer name + self.assertTrue( + section.find_element(By.XPATH, '//h2[1]').is_displayed() + ) + # Check layer summary + self.assertTrue("Summary" in section.text) + # Check layer description + self.assertTrue("Description" in section.text) + + def test_single_recipe_page(self): + """ Test recipe page + - Check if title is displayed + - Check add recipe layer displayed + - Check left section is displayed + - Check recipe: name, summary, description, Version, Section, + License, Approx. packages included, Approx. size, Recipe file + """ + url = reverse("recipedetails", args=(1, 53428)) + self.get(url) + self.wait_until_visible('.page-header') + # check title is displayed + self.assertTrue(self.find('.page-header h1').is_displayed()) + # check add recipe layer displayed + add_recipe_layer_btn = self.find('#add-layer-btn') + self.assertTrue(add_recipe_layer_btn.is_displayed()) + # check left section is displayed + section = self.find('.well') + # Check recipe name + self.assertTrue( + section.find_element(By.XPATH, '//h2[1]').is_displayed() + ) + # Check recipe sections details info are displayed + self.assertTrue("Summary" in section.text) + self.assertTrue("Description" in section.text) + self.assertTrue("Version" in section.text) + self.assertTrue("Section" in section.text) + self.assertTrue("License" in section.text) + self.assertTrue("Approx. packages included" in section.text) + self.assertTrue("Approx. package size" in section.text) + self.assertTrue("Recipe file" in section.text) -- cgit v1.2.3