diff options
| author | Patrick Williams <patrick@stwcx.xyz> | 2021-08-08 03:21:33 +0300 |
|---|---|---|
| committer | Patrick Williams <patrick@stwcx.xyz> | 2021-08-11 23:39:32 +0300 |
| commit | 213cb2696d00a85cd48d356cb5131824a302d828 (patch) | |
| tree | bfdf8fcdfef3a88e392ca3acfca6dec4dc836c9f /poky/bitbake/lib | |
| parent | 40d8f44b51f83548f7ca8df062dd7435b784c0a8 (diff) | |
| download | openbmc-213cb2696d00a85cd48d356cb5131824a302d828.tar.xz | |
subtree updates
meta-raspberrypi: 8dc3a31088..c7f4c739a3:
Khem Raj (5):
linux-raspberrypi: Upgrade to 5.10.52
userland: Update to latest master branch
raspberrypi-firmware: Update to latest
raspberrypi-tools: Update to latest
sdcard_image-rpi.bbclass: Fix IMAGE_TYPEDEP override to use new syntax
Martin Jansa (4):
Convert to new override syntax
Manually fix conversion
layer.conf: Update to honister
userland: package man pages in PN-doc
Pierre-Jean Texier (2):
kas: local.conf: bump CONF_VERSION variable
kas: local.conf: disable prelink
poky: 17aabc0127..492205ea83:
Alexander Kanavin (17):
llvm: update 12.0.0 -> 12.0.1
systemd: update 248.3 -> 249.1
python3-testools: update 2.4.0 -> 2.5.0
libuv: update 1.41.0 -> 1.42.0
gnu-config: update to latest revision
vulkan-samples: update to latest revision
cmake: update 3.20.5 -> 3.21.0
cmake: update 3.21.0 -> 3.21.1
mtools: update 4.0.32 -> 4.0.34
util-linux: update 2.37 -> 2.37.1
iputils: update 20210202 -> 20210722
freetype: update 2.10.4 -> 2.11.0
devtool: print a warning on upgrades if PREFERRED_VERSION is set
rpm: do not RRECOMMEND rpm-build
selftest: add core-image-weston to no-gpl3-no-meta-gpl2 image test
shadow: update 4.8.1 -> 4.9
local.conf.sample: disable prelink
Bernhard Rosenkränzer (1):
gcc: update 11.1 -> 11.2
Bruce Ashfield (6):
linux-yocto/5.10: update to v5.10.53
linux-yocto/5.13: update to v5.13.5
linux-yocto/5.4: update to v5.4.135
linux-yocto-rt/5.10: update to -rt47
linux-yocto/5.13: enable TYPEC_TCPCI in usbc fragment
linux-yocto/5.10: enable TYPEC_TCPCI in usbc fragment
Changqing Li (1):
archiver.bbclass: fix do_ar_configured failure for kernel
Chen Qi (3):
zstd: fix CVE_PRODUCT
insane.bbclass: fix the file-rdeps QA message for the new override syntax
iputils: fix do_configure failure of missing ip command
Damian Wrobel (1):
rootfs: remove ldconfig auxiliary cache where appropriate
Denys Dmytriyenko (4):
meta: convert nested overrides leftovers to new syntax
convert-overrides.py: handle few more cases of overrides
libwpe: remove rpi-specific custom code
poky-tiny: drop uclibc override
Jon Mason (1):
parselogs.py: qemuarm should be qemuarmv5
Joshua Watt (4):
mesa: Fix v3d & vc4 dmabuf import
bitbake: bitbake: asyncrpc: Catch early SIGTERM
libxft: Fix bad PKG value
bitbake: contrib: vim: Update for new override syntax
Kai Kang (2):
u-boot_2021.07: set UBOOT_MACHINE for qemumips and qemumips64
python3-pytest: display correct version info
Kevin Hao (2):
meta-yocto-bsp: Introduce the v5.13 bbappend
meta-yocto-bsp: Bump to the v5.10.55
Khem Raj (10):
binutils: Upgrade to 2.37 branch
texinfo: Update gnulib to fix build with glibc 2.34
systemd: Fix build on musl
stress-ng: Drop defining daddr_t
stress-ng: Detemine minimal stack size via sysconf
mesa: Define a fallback for DRIDRIVERS
libssh2: Fix syntax for using ptest override
toaster-managed-mode.json: Correctly specify term with new override syntax
distrooverrides.bbclass: Correct override syntax
devtool.py: Correct override syntax
Lee Chee Yang (1):
aspell: fix CVE-2019-25051
Marek Vasut (2):
image_types: Restore pre-btrfs-tools 4.14.1 mkfs.btrfs shrink behavior
kernel-uboot: Handle gzip and lzo compression options
Martin Jansa (6):
convert-overrides.py: show processed file and version of this script
convert-overrides.py: remove base_dep_prepend and autotools_dep_prepend exception
convert-overrides.py: 0.9.1 include '(' as delimiter for shortvars
convert-overrides.py: allow specifying multiple target dirs
convert-overrides.py: allow dots before override in vars_re and shortvars_re
systemd-boot: use ld.bfd as efi-ld even when gold or lld is used in ${LD}
Matthias Klein (2):
runqemu: Fix typo in error message
runqemu: decouple bios and kernel options
Matthias Schiffer (3):
initscripts: populate-volatile.sh: do not log to tty0
initscripts: populate-volatile.sh: run create_file synchronously
initscripts: fix creation order for /var/log with VOLATILE_LOG_DIR=true
Michael Halstead (1):
releases: update to include 3.3.1
Michael Opdenacker (18):
oe-setup-builddir: update YP docs and OE URLs
conf-notes.txt: now suggesting to run 'runqemu qemux86-64'
test-manual: document LTO related reproducibility bug
quick start manual: update "source oe-init-build-env" output
dev-manual: fix wrong reference to class
documentation/README: improve BitBake manual referencing guidelines
manuals: simplify references to BitBake manual
manuals: remove explicit BitBake variable references
meta-skeleton: add recipe examples from documentation sources
bitbake: doc: bitbake-user-manual: fix syntax in example and improve description
bitbake: doc: bitbake-user-manual: update bitbake option help
bitbake: doc: bitbake-user-manual: grammar fix for the number of "metadata"
manuals: initial documentation for CVE management
ref-manual: remove example recipe source files
profile-manual: document how to build perf manpages on target
cve-check: fix comments
cve-check: update link to NVD website for CVE details
cve-check: improve comment about CVE patch file names
Mingli Yu (2):
perlcross: not break build if already patched
curl: Upgrade to 7.78.0
Nicolas Dechesne (4):
yocto-check-layer: improve missed dependencies
checklayer: new function get_layer_dependencies()
checklayer: rename _find_layer_depends
yocto-check-layer: ensure that all layer dependencies are tested too
Oleksandr Kravchuk (1):
bitbake.conf: change GNOME_MIRROR to new one
Patrick Williams (1):
pixman: re-disable iwmmxt
Paul Barker (4):
bitbake: asyncrpc: Fix bad message error in client
bitbake: asyncrpc: Set timeout when waiting for reply from server
bitbake: parse/ast: Substitute '~' when naming anonymous functions
kernel-yocto: Simplify no git repo case in do_kernel_checkout
Quentin Schulz (4):
bitbake: doc: Makefile: turn warnings into errors by default
bitbake: doc: bitbake-user-manual: ref-variables: order alphabetically the glossary sources
bitbake: doc: bitbake-user-manual: ref-variables: force glossary output to be alphabetically sorted
bitbake: doc: bitbake-user-manual: replace ``FOO`` by :term:`FOO` where possible
Richard Purdie (49):
Add MAINTAINERS.md file
yocto-check-layer: Remove duplicated code
libubootenv: Drop default-env RRECOMMENDS
bitbake: data_smart: Allow colon in variable expansion regex
meta-poky/meta-yocto-bsp: Convert to new override syntax
layer.conf: Update to honister
autotools/base/icecc: Remove prepend from function names
scripts/contrib: Add override conversion script
systemtap: Fix headers issue with x86 and 5.13 headers
migration-guides: Add start of 3.4 guide with override migration notes
common-tasks: Fix conversion error in npm example
bitbake: bitbake: Switch to using new override syntax
bitbake: doc/lib: Update to use new override syntax containing colons
bitbake: doc/lib: Add fixes for issues missed by the automated conversion
bitbake: bitbake: Update to version 1.51.1
layer.conf: Override changes mean we're only compatible with honister
Convert to new override syntax
meta: Manual override fixes
local.conf.sample: Bump version so users update their config
sanity.conf: Require bitbake 1.51.1
dropbear: Fix incorrect package override for postrm
convert-overrides: Allow script to handle patch/diffs
sdk: Decouple default install path from built in path
sstate: Fix rebuilds when changing layer config
populate_sdk_ext: Fix handling of TOOLCHAIN_HOST_TASK in the eSDK case
local.conf.sample: Bump version so users update their config
poky: Use SDKPATHINSTALL instead of SDKPATH
vim: Clarify where RDEPENDS/RRECOMMENDS apply
bitbake: data_smart: Fix inactive overide accidental variable value corruption
local.conf.sample: Fix missed override conversion
license: Exclude COPYING.MIT from pseudo
meta: Convert IMAGE_TYPEDEP to use override syntax
uboot-extlinux-config: Fix missing override conversion
image/image_types: Convert CONVERSION_CMD/COMPRESS_CMD to new override syntax
image: Drop COMPRESS_CMD
devupstream: Allow support of native class extensions
diffoscope: Upgrade 178 -> 179
strace: Upgrade 5.12 -> 5.13
valgrind: Add patches for glibc 2.34 support
bitbake: runqueue: Improve multiconfig deferred task issues
elfutils: Add patch from upstream for glibc 2.34 ptest fixes
bitbake: doc: Fix append/prepend/remove references
bitbake: fetch/tests/toaster: Override conversion fixups
bitbake: process: Improve traceback error reporting from main loop
bitbake: command: Ensure we catch/handle exceptions
bitbake: ui/taskexp: Improve startup exception handling
bitbake: ui/taskexp: Fix to work with empty build directories
oeqa/runtime/cases/ptest: Increase test timeout from 300s to 450s
packagedata: Fix after override syntax change
Ross Burton (2):
glew: fix Makefile race
libx11: fix xkb compilation with _EVDEVK symbols
Saul Wold (1):
MAINTAINERS: Saul will cover devtool and eSDK
Stefan Wiehler (1):
dev-manual: fix source release example script
Stefano Babic (1):
mtd-utils: upgrade 2.1.2 -> 2.1.3
Tim Orling (2):
python3-hypothesis: upgrade 6.14.3 -> 6.14.5
python3-importlib-metadata: upgrade 4.6.1 -> 4.6.3
Tony Battersby (2):
lto.inc: disable LTO for grub
gcc: Backport patch to make LTO builds more reproducible
Tony Tascioglu (6):
ffmpeg: fix-CVE-2020-20446
ffmpeg: fix CVE-2020-20453
ffmpeg: fix CVE-2020-22015
ffmpeg: fix CVE-2020-22021
ffmpeg: fix CVE-2020-22033 and CVE-2020-22019
ffmpeg: fix CVE-2021-33815
Trevor Woerner (1):
ffmpeg: add libatomic for armv5
Ulrich Ölmann (2):
initramfs-framework: fix whitespace issue
initramfs-framework/setup-live: fix shebang
Vinay Kumar (1):
glibc: Fix CVE-2021-33574
Vivien Didelot (1):
init-manager-systemd: define weak dev manager
Zqiang (1):
python3: use monotonic clock for condvar if possible
hongxu (1):
createrepo-c: fix createrepo-c failed in nativesdk
leimaohui (1):
archiver.bbclass: Fix patch error for recipes that inherit dos2unix.
wangmy (3):
bind: upgrade 9.16.18 -> 9.16.19
i2c-tools: upgrade 4.2 -> 4.3
diffoscope: upgrade 177 -> 178
zangrc (2):
python3-dbus: upgrade 1.2.16 -> 1.2.18
python3-pip: upgrade 21.1.3 -> 21.2.1
meta-openembedded: 8fbcfb9f02..3cf2475ea0:
Anastasios Kavoukis (1):
pm-qa: fix paths for shell scripts
Andreas Müller (3):
mozjs/0001-Port-build-to-python3.patch: Fix typos in description
jack: upgrade 1.19.18 -> 1.19.19
fluidsynth: upgrade 2.2.1 -> 2.2.2
Andrej Valek (1):
thrift: upgrade to 0.14.2
Andrew Jeffery (2):
python3-gmpy: Add native support
python3-ecdsa: Add native support
Armin Kuster (2):
hiawatha: fix url.
wireshark: update to 3.4.7
Ben Brown (1):
android-tools: fix install of adb client when TOOLS is overridden
Changqing Li (1):
apache2: upgrade 2.4.46 -> 2.4.48
Devendra Tewari (1):
Suppress eol in functionfs setup scripts (#147)
Gianfranco (1):
vboxguestdrivers: upgrade 6.1.22 -> 6.1.24
Joe Slater (2):
php: move to version 7.4.21
gtksourceview4: work around dependency deficiency
Johannes Obermüller (1):
evtest: fix timestamps in output
Kai Kang (2):
python3-blivet: 3.1.4 -> 3.4.0
python3-blivetgui: 2.1.10 -> 2.2.1
Khem Raj (23):
netperf: Update to latest
netperf: Add systemd unit file
packagegroup-meta-oe: Add lmdb
packagegroup-meta-oe: Add mbw
addcli: check for ns_get16 and ns_get32
fuse: Define closefrom if not available
autofs: Fix build with glibc 2.34+
ntp: Do not use PTHREAD_STACK_MIN on glibc
ntp: Fix make check
mongodb: Upgrade to 4.4.7
vboxguestdrivers: Remove __divmoddi4 patch
packagegroup-meta-oe: Add jemalloc
apitrace: Exclude from builds with glibc 2.34+
libhugetlbfs: Disable build with glibc 2.34+
fvwm: Package extra files and man pages
luajit: Fix override syntax
lua: Drop uclibc patch
packagegroup-meta-oe: Correct override name and fix syntax
recipes: Fix override syntax
emacs,libgpiod,cockpit: Fix override syntax in using FILES_${PN}
fvwm: Fix build time paths in target perl/python scripts
nis: Drop uclibc check in anon python function
jemalloc: Fix build on musl
Leon Anavi (3):
python3-networkx: Upgrade 2.6.1 -> 2.6.2
python3-pysonos: Upgrade 0.0.53 -> 0.0.54
python3-zeroconf: Upgrade 0.33.1 -> 0.33.2
Li Wang (1):
openlldp: fix segfault
Maksym Sloyko (1):
libusbgx: Configure the Devices Used
Martin Jansa (5):
Convert to new override syntax
layer.conf: Update to honister
mariadb: manually fix the conversion
packagegroup-meta-oe: manually finish override syntax conversion
klibc.bbclass, image_types_sparse.bbclass, packagegroup-meta-oe.bb: update the overrides syntax conversion
Mingli Yu (4):
mariadb: redefine log-error item
jemalloc: add new recipe
hdf5: improve reproducibility
mariadb: Update SRC_URI
Nicolas Dechesne (1):
mbw: add new recipe
Paulo Neves (1):
htop: Add ncurses-terminfo-base to RDEPENDS
Sakib Sajal (1):
lmdb: add recipe
Salman Ahmed (2):
nginx: upgrade 1.18.0 -> 1.20.1
nginx: upgrade 1.19.6 -> 1.21.1
Tony Battersby (1):
net-snmp: fix QA Issue after LDFLAGS change
Yi Zhao (3):
postfix: upgrade 3.6.1 -> 3.6.2
audit: upgrade 3.0.2 -> 3.0.3
audit: fix compile error for 2.8.5
Zang Ruochen (1):
python3-robotframework: upgrade 4.0.3 -> 4.1
wangmy (17):
evince: upgrade 40.2 -> 40.4
gnome-backgrounds: upgrade 3.36.0 -> 3.38.0
gnome-desktop3: upgrade 3.36.6 -> 3.38.8
cmark: upgrade 0.30.0 -> 0.30.1
ctags: upgrade 5.9.20210711.0 -> 5.9.20210718.0
libnet-dns-perl: upgrade 1.31 -> 1.32
libtalloc: upgrade 2.3.2 -> 2.3.3
nghttp2: upgrade 1.43.0 -> 1.44.0
bats: upgrade 1.3.0 -> 1.4.1
networkmanager: upgrade 1.32.2 -> 1.32.4
gensio: upgrade 2.2.7 -> 2.2.8
libmbim: upgrade 1.24.8 -> 1.26.0
fetchmail: upgrade 6.4.19 -> 6.4.20
ctags: upgrade 5.9.20210718.0 -> 5.9.20210801.0
libblockdev: upgrade 2.25 -> 2.26
libqmi: upgrade 1.28.6 -> 1.28.8
monit: upgrade 5.28.0 -> 5.28.1
zangrc (15):
python3-qrcode: upgrade 7.1 -> 7.2
python3-rdflib: upgrade 5.0.0 -> 6.0.0
python3-simplejson: upgrade 3.17.2 -> 3.17.3
python3-bitstring: upgrade 3.1.7 -> 3.1.9
python3-iso8601: upgrade 0.1.14 -> 0.1.16
python3-gmqtt: upgrade 0.6.9 -> 0.6.10
python3-graphviz: upgrade 0.16 -> 0.17
python3-smbus: upgrade 4.2 -> 4.3
python3-pandas: upgrade 1.3.0 -> 1.3.1
python3-progress: upgrade 1.5 -> 1.6
python3-sentry-sdk: upgrade 1.3.0 -> 1.3.1
python3-socketio: upgrade 5.3.0 -> 5.4.0
python3-tqdm: upgrade 4.61.2 -> 4.62.0
python3-twisted: upgrade 21.2.0 -> 21.7.0
python3-xlsxwriter: upgrade 1.4.4 -> 1.4.5
zhengruoqin (15):
live555: upgrade 20210710 -> 20210720
libtest-warnings-perl: upgrade 0.030 -> 0.031
python3-pybind11: upgrade 2.6.2 -> 2.7.0
python3-pymongo: upgrade 3.11.4 -> 3.12.0
python3-sqlalchemy: upgrade 1.4.20 -> 1.4.22
python3-sentry-sdk: upgrade 1.2.0 -> 1.3.0
libcurses-perl: upgrade 1.37 -> 1.38
libdbd-sqlite-perl: upgrade 1.66 -> 1.68
libencode-perl: upgrade 3.10 -> 3.11
python3-bitarray: upgrade 2.2.2 -> 2.2.3
python3-cbor2: upgrade 5.4.0 -> 5.4.1
python3-gast: upgrade 0.5.0 -> 0.5.1
poppler: upgrade 21.07.0 -> 21.08.0
valijson: upgrade 0.4 -> 0.5
xwd: upgrade 1.0.7 -> 1.0.8
meta-security: 152cdb506b..c885d399cd:
Armin Kuster (18):
suricata.inc: exclude ppc in rust version
suricata: Drop 4.1.x its EOL
add meta-rust
crowdsec: add pkg
packagegroup-core-security.bb: fix suricat-ptest inclusion
gitlab-ci.yml: streamline builds matrix
krill: Add new pkg
clamav: fix branch name and update
meta-security: Convert to new override syntax
meta-tpm: Convert to new override syntax
meta-integrity: Convert to new override syntax
meta-hardening: Convert to new override syntax
meta-security-isafw: Convert to new override syntax
meta-parsec: Convert to new override syntax
meta-security-compliance: Convert to new override syntax
dynamix-layers: Convert to new override syntax
kas: Convert to new override syntax
packagegroup-core-security.bb: only include suricat-ptest if rust is included
Martin Jansa (1):
layer.conf: Update to honister
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: Iec7301cf1c43b7cec462dcf88292a8b1b12a5045
Diffstat (limited to 'poky/bitbake/lib')
21 files changed, 258 insertions, 182 deletions
diff --git a/poky/bitbake/lib/bb/__init__.py b/poky/bitbake/lib/bb/__init__.py index a144bd609..c1e30697b 100644 --- a/poky/bitbake/lib/bb/__init__.py +++ b/poky/bitbake/lib/bb/__init__.py @@ -9,7 +9,7 @@ # SPDX-License-Identifier: GPL-2.0-only # -__version__ = "1.51.0" +__version__ = "1.51.1" import sys if sys.version_info < (3, 5, 0): diff --git a/poky/bitbake/lib/bb/asyncrpc/client.py b/poky/bitbake/lib/bb/asyncrpc/client.py index 79919c5be..3eb4fdde8 100644 --- a/poky/bitbake/lib/bb/asyncrpc/client.py +++ b/poky/bitbake/lib/bb/asyncrpc/client.py @@ -11,13 +11,14 @@ from . import chunkify, DEFAULT_MAX_CHUNK class AsyncClient(object): - def __init__(self, proto_name, proto_version, logger): + def __init__(self, proto_name, proto_version, logger, timeout=30): self.reader = None self.writer = None self.max_chunk = DEFAULT_MAX_CHUNK self.proto_name = proto_name self.proto_version = proto_version self.logger = logger + self.timeout = timeout async def connect_tcp(self, address, port): async def connect_sock(): @@ -70,14 +71,18 @@ class AsyncClient(object): async def send_message(self, msg): async def get_line(): - line = await self.reader.readline() + try: + line = await asyncio.wait_for(self.reader.readline(), self.timeout) + except asyncio.TimeoutError: + raise ConnectionError("Timed out waiting for server") + if not line: raise ConnectionError("Connection closed") line = line.decode("utf-8") if not line.endswith("\n"): - raise ConnectionError("Bad message %r" % msg) + raise ConnectionError("Bad message %r" % (line)) return line diff --git a/poky/bitbake/lib/bb/asyncrpc/serv.py b/poky/bitbake/lib/bb/asyncrpc/serv.py index ef20cb71d..4084f300d 100644 --- a/poky/bitbake/lib/bb/asyncrpc/serv.py +++ b/poky/bitbake/lib/bb/asyncrpc/serv.py @@ -9,6 +9,7 @@ import os import signal import socket import sys +import multiprocessing from . import chunkify, DEFAULT_MAX_CHUNK @@ -201,12 +202,14 @@ class AsyncServer(object): pass def signal_handler(self): + self.logger.debug("Got exit signal") self.loop.stop() def serve_forever(self): asyncio.set_event_loop(self.loop) try: self.loop.add_signal_handler(signal.SIGTERM, self.signal_handler) + signal.pthread_sigmask(signal.SIG_UNBLOCK, [signal.SIGTERM]) self.run_loop_forever() self.server.close() @@ -221,3 +224,21 @@ class AsyncServer(object): if self._cleanup_socket is not None: self._cleanup_socket() + + def serve_as_process(self, *, prefunc=None, args=()): + def run(): + if prefunc is not None: + prefunc(self, *args) + self.serve_forever() + + # Temporarily block SIGTERM. The server process will inherit this + # block which will ensure it doesn't receive the SIGTERM until the + # handler is ready for it + mask = signal.pthread_sigmask(signal.SIG_BLOCK, [signal.SIGTERM]) + try: + self.process = multiprocessing.Process(target=run) + self.process.start() + + return self.process + finally: + signal.pthread_sigmask(signal.SIG_SETMASK, mask) diff --git a/poky/bitbake/lib/bb/cache.py b/poky/bitbake/lib/bb/cache.py index 27eb27179..73bc6e966 100644 --- a/poky/bitbake/lib/bb/cache.py +++ b/poky/bitbake/lib/bb/cache.py @@ -53,12 +53,12 @@ class RecipeInfoCommon(object): @classmethod def pkgvar(cls, var, packages, metadata): - return dict((pkg, cls.depvar("%s_%s" % (var, pkg), metadata)) + return dict((pkg, cls.depvar("%s:%s" % (var, pkg), metadata)) for pkg in packages) @classmethod def taskvar(cls, var, tasks, metadata): - return dict((task, cls.getvar("%s_task-%s" % (var, task), metadata)) + return dict((task, cls.getvar("%s:task-%s" % (var, task), metadata)) for task in tasks) @classmethod diff --git a/poky/bitbake/lib/bb/command.py b/poky/bitbake/lib/bb/command.py index f530cf844..a81dcb136 100644 --- a/poky/bitbake/lib/bb/command.py +++ b/poky/bitbake/lib/bb/command.py @@ -65,9 +65,17 @@ class Command: # Ensure cooker is ready for commands if command != "updateConfig" and command != "setFeatures": - self.cooker.init_configdata() - if not self.remotedatastores: - self.remotedatastores = bb.remotedata.RemoteDatastores(self.cooker) + try: + self.cooker.init_configdata() + if not self.remotedatastores: + self.remotedatastores = bb.remotedata.RemoteDatastores(self.cooker) + except (Exception, SystemExit) as exc: + import traceback + if isinstance(exc, bb.BBHandledException): + # We need to start returning real exceptions here. Until we do, we can't + # tell if an exception is an instance of bb.BBHandledException + return None, "bb.BBHandledException()\n" + traceback.format_exc() + return None, traceback.format_exc() if hasattr(CommandsSync, command): # Can run synchronous commands straight away diff --git a/poky/bitbake/lib/bb/cooker.py b/poky/bitbake/lib/bb/cooker.py index 39e10e613..b2d69c28c 100644 --- a/poky/bitbake/lib/bb/cooker.py +++ b/poky/bitbake/lib/bb/cooker.py @@ -390,8 +390,7 @@ class BBCooker: dbfile = (self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE")) + "/hashserv.db" self.hashservaddr = "unix://%s/hashserve.sock" % self.data.getVar("TOPDIR") self.hashserv = hashserv.create_server(self.hashservaddr, dbfile, sync=False) - self.hashserv.process = multiprocessing.Process(target=self.hashserv.serve_forever) - self.hashserv.process.start() + self.hashserv.serve_as_process() self.data.setVar("BB_HASHSERVE", self.hashservaddr) self.databuilder.origdata.setVar("BB_HASHSERVE", self.hashservaddr) self.databuilder.data.setVar("BB_HASHSERVE", self.hashservaddr) diff --git a/poky/bitbake/lib/bb/data_smart.py b/poky/bitbake/lib/bb/data_smart.py index f48726a34..65528c6ae 100644 --- a/poky/bitbake/lib/bb/data_smart.py +++ b/poky/bitbake/lib/bb/data_smart.py @@ -26,9 +26,9 @@ from bb.COW import COWDictBase logger = logging.getLogger("BitBake.Data") -__setvar_keyword__ = ["_append", "_prepend", "_remove"] -__setvar_regexp__ = re.compile(r'(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$') -__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~]+?}") +__setvar_keyword__ = [":append", ":prepend", ":remove"] +__setvar_regexp__ = re.compile(r'(?P<base>.*?)(?P<keyword>:append|:prepend|:remove)(:(?P<add>[^A-Z]*))?$') +__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~:]+?}") __expand_python_regexp__ = re.compile(r"\${@.+?}") __whitespace_split__ = re.compile(r'(\s)') __override_regexp__ = re.compile(r'[a-z0-9]+') @@ -277,7 +277,7 @@ class VariableHistory(object): for (r, override) in d.overridedata[var]: for event in self.variable(r): loginfo = event.copy() - if 'flag' in loginfo and not loginfo['flag'].startswith("_"): + if 'flag' in loginfo and not loginfo['flag'].startswith(("_", ":")): continue loginfo['variable'] = var loginfo['op'] = 'override[%s]:%s' % (override, loginfo['op']) @@ -342,7 +342,7 @@ class VariableHistory(object): for event in history: if 'flag' in event: continue - if event['op'] == '_remove': + if event['op'] == ':remove': continue if isset and event['op'] == 'set?': continue @@ -481,7 +481,15 @@ class DataSmart(MutableMapping): def setVar(self, var, value, **loginfo): #print("var=" + str(var) + " val=" + str(value)) - var = var.replace(":", "_") + + if "_append" in var or "_prepend" in var or "_remove" in var: + info = "%s" % var + if "filename" in loginfo: + info += " file: %s" % loginfo[filename] + if "lineno" in loginfo: + info += " line: %s" % loginfo[lineno] + bb.fatal("Variable %s contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake." % info) + self.expand_cache = {} parsing=False if 'parsing' in loginfo: @@ -510,7 +518,7 @@ class DataSmart(MutableMapping): # pay the cookie monster # more cookies for the cookie monster - if '_' in var: + if ':' in var: self._setvar_update_overrides(base, **loginfo) if base in self.overridevars: @@ -521,27 +529,27 @@ class DataSmart(MutableMapping): self._makeShadowCopy(var) if not parsing: - if "_append" in self.dict[var]: - del self.dict[var]["_append"] - if "_prepend" in self.dict[var]: - del self.dict[var]["_prepend"] - if "_remove" in self.dict[var]: - del self.dict[var]["_remove"] + if ":append" in self.dict[var]: + del self.dict[var][":append"] + if ":prepend" in self.dict[var]: + del self.dict[var][":prepend"] + if ":remove" in self.dict[var]: + del self.dict[var][":remove"] if var in self.overridedata: active = [] self.need_overrides() for (r, o) in self.overridedata[var]: if o in self.overridesset: active.append(r) - elif "_" in o: - if set(o.split("_")).issubset(self.overridesset): + elif ":" in o: + if set(o.split(":")).issubset(self.overridesset): active.append(r) for a in active: self.delVar(a) del self.overridedata[var] # more cookies for the cookie monster - if '_' in var: + if ':' in var: self._setvar_update_overrides(var, **loginfo) # setting var @@ -567,8 +575,8 @@ class DataSmart(MutableMapping): def _setvar_update_overrides(self, var, **loginfo): # aka pay the cookie monster - override = var[var.rfind('_')+1:] - shortvar = var[:var.rfind('_')] + override = var[var.rfind(':')+1:] + shortvar = var[:var.rfind(':')] while override and __override_regexp__.match(override): if shortvar not in self.overridedata: self.overridedata[shortvar] = [] @@ -577,9 +585,9 @@ class DataSmart(MutableMapping): self.overridedata[shortvar] = list(self.overridedata[shortvar]) self.overridedata[shortvar].append([var, override]) override = None - if "_" in shortvar: - override = var[shortvar.rfind('_')+1:] - shortvar = var[:shortvar.rfind('_')] + if ":" in shortvar: + override = var[shortvar.rfind(':')+1:] + shortvar = var[:shortvar.rfind(':')] if len(shortvar) == 0: override = None @@ -590,8 +598,6 @@ class DataSmart(MutableMapping): """ Rename the variable key to newkey """ - key = key.replace(":", "_") - newkey = newkey.replace(":", "_") if key == newkey: bb.warn("Calling renameVar with equivalent keys (%s) is invalid" % key) return @@ -620,7 +626,7 @@ class DataSmart(MutableMapping): self.overridedata[newkey].append([v.replace(key, newkey), o]) self.renameVar(v, v.replace(key, newkey)) - if '_' in newkey and val is None: + if ':' in newkey and val is None: self._setvar_update_overrides(newkey, **loginfo) loginfo['variable'] = key @@ -632,15 +638,14 @@ class DataSmart(MutableMapping): def appendVar(self, var, value, **loginfo): loginfo['op'] = 'append' self.varhistory.record(**loginfo) - self.setVar(var + "_append", value, ignore=True, parsing=True) + self.setVar(var + ":append", value, ignore=True, parsing=True) def prependVar(self, var, value, **loginfo): loginfo['op'] = 'prepend' self.varhistory.record(**loginfo) - self.setVar(var + "_prepend", value, ignore=True, parsing=True) + self.setVar(var + ":prepend", value, ignore=True, parsing=True) def delVar(self, var, **loginfo): - var = var.replace(":", "_") self.expand_cache = {} loginfo['detail'] = "" @@ -649,9 +654,9 @@ class DataSmart(MutableMapping): self.dict[var] = {} if var in self.overridedata: del self.overridedata[var] - if '_' in var: - override = var[var.rfind('_')+1:] - shortvar = var[:var.rfind('_')] + if ':' in var: + override = var[var.rfind(':')+1:] + shortvar = var[:var.rfind(':')] while override and override.islower(): try: if shortvar in self.overridedata: @@ -661,14 +666,13 @@ class DataSmart(MutableMapping): except ValueError as e: pass override = None - if "_" in shortvar: - override = var[shortvar.rfind('_')+1:] - shortvar = var[:shortvar.rfind('_')] + if ":" in shortvar: + override = var[shortvar.rfind(':')+1:] + shortvar = var[:shortvar.rfind(':')] if len(shortvar) == 0: override = None def setVarFlag(self, var, flag, value, **loginfo): - var = var.replace(":", "_") self.expand_cache = {} if 'op' not in loginfo: @@ -679,7 +683,7 @@ class DataSmart(MutableMapping): self._makeShadowCopy(var) self.dict[var][flag] = value - if flag == "_defaultval" and '_' in var: + if flag == "_defaultval" and ':' in var: self._setvar_update_overrides(var, **loginfo) if flag == "_defaultval" and var in self.overridevars: self._setvar_update_overridevars(var, value) @@ -692,7 +696,6 @@ class DataSmart(MutableMapping): self.dict["__exportlist"]["_content"].add(var) def getVarFlag(self, var, flag, expand=True, noweakdefault=False, parsing=False, retparser=False): - var = var.replace(":", "_") if flag == "_content": cachename = var else: @@ -712,11 +715,11 @@ class DataSmart(MutableMapping): active = {} self.need_overrides() for (r, o) in overridedata: - # What about double overrides both with "_" in the name? + # FIXME What about double overrides both with "_" in the name? if o in self.overridesset: active[o] = r - elif "_" in o: - if set(o.split("_")).issubset(self.overridesset): + elif ":" in o: + if set(o.split(":")).issubset(self.overridesset): active[o] = r mod = True @@ -724,10 +727,10 @@ class DataSmart(MutableMapping): mod = False for o in self.overrides: for a in active.copy(): - if a.endswith("_" + o): + if a.endswith(":" + o): t = active[a] del active[a] - active[a.replace("_" + o, "")] = t + active[a.replace(":" + o, "")] = t mod = True elif a == o: match = active[a] @@ -746,31 +749,31 @@ class DataSmart(MutableMapping): value = copy.copy(local_var["_defaultval"]) - if flag == "_content" and local_var is not None and "_append" in local_var and not parsing: - if not value: - value = "" + if flag == "_content" and local_var is not None and ":append" in local_var and not parsing: self.need_overrides() - for (r, o) in local_var["_append"]: + for (r, o) in local_var[":append"]: match = True if o: - for o2 in o.split("_"): + for o2 in o.split(":"): if not o2 in self.overrides: match = False if match: + if value is None: + value = "" value = value + r - if flag == "_content" and local_var is not None and "_prepend" in local_var and not parsing: - if not value: - value = "" + if flag == "_content" and local_var is not None and ":prepend" in local_var and not parsing: self.need_overrides() - for (r, o) in local_var["_prepend"]: + for (r, o) in local_var[":prepend"]: match = True if o: - for o2 in o.split("_"): + for o2 in o.split(":"): if not o2 in self.overrides: match = False if match: + if value is None: + value = "" value = r + value parser = None @@ -779,12 +782,12 @@ class DataSmart(MutableMapping): if expand: value = parser.value - if value and flag == "_content" and local_var is not None and "_remove" in local_var and not parsing: + if value and flag == "_content" and local_var is not None and ":remove" in local_var and not parsing: self.need_overrides() - for (r, o) in local_var["_remove"]: + for (r, o) in local_var[":remove"]: match = True if o: - for o2 in o.split("_"): + for o2 in o.split(":"): if not o2 in self.overrides: match = False if match: @@ -820,7 +823,6 @@ class DataSmart(MutableMapping): return value def delVarFlag(self, var, flag, **loginfo): - var = var.replace(":", "_") self.expand_cache = {} local_var, _ = self._findVar(var) @@ -838,7 +840,6 @@ class DataSmart(MutableMapping): del self.dict[var][flag] def appendVarFlag(self, var, flag, value, **loginfo): - var = var.replace(":", "_") loginfo['op'] = 'append' loginfo['flag'] = flag self.varhistory.record(**loginfo) @@ -846,7 +847,6 @@ class DataSmart(MutableMapping): self.setVarFlag(var, flag, newvalue, ignore=True) def prependVarFlag(self, var, flag, value, **loginfo): - var = var.replace(":", "_") loginfo['op'] = 'prepend' loginfo['flag'] = flag self.varhistory.record(**loginfo) @@ -854,7 +854,6 @@ class DataSmart(MutableMapping): self.setVarFlag(var, flag, newvalue, ignore=True) def setVarFlags(self, var, flags, **loginfo): - var = var.replace(":", "_") self.expand_cache = {} infer_caller_details(loginfo) if not var in self.dict: @@ -869,13 +868,12 @@ class DataSmart(MutableMapping): self.dict[var][i] = flags[i] def getVarFlags(self, var, expand = False, internalflags=False): - var = var.replace(":", "_") local_var, _ = self._findVar(var) flags = {} if local_var: for i in local_var: - if i.startswith("_") and not internalflags: + if i.startswith(("_", ":")) and not internalflags: continue flags[i] = local_var[i] if expand and i in expand: @@ -886,7 +884,6 @@ class DataSmart(MutableMapping): def delVarFlags(self, var, **loginfo): - var = var.replace(":", "_") self.expand_cache = {} if not var in self.dict: self._makeShadowCopy(var) @@ -974,8 +971,8 @@ class DataSmart(MutableMapping): for (r, o) in self.overridedata[var]: if o in self.overridesset: overrides.add(var) - elif "_" in o: - if set(o.split("_")).issubset(self.overridesset): + elif ":" in o: + if set(o.split(":")).issubset(self.overridesset): overrides.add(var) for k in keylist(self.dict): diff --git a/poky/bitbake/lib/bb/fetch2/__init__.py b/poky/bitbake/lib/bb/fetch2/__init__.py index 0d49e1da3..ad898680f 100644 --- a/poky/bitbake/lib/bb/fetch2/__init__.py +++ b/poky/bitbake/lib/bb/fetch2/__init__.py @@ -1146,11 +1146,11 @@ def srcrev_internal_helper(ud, d, name): pn = d.getVar("PN") attempts = [] if name != '' and pn: - attempts.append("SRCREV_%s_pn-%s" % (name, pn)) + attempts.append("SRCREV_%s:pn-%s" % (name, pn)) if name != '': attempts.append("SRCREV_%s" % name) if pn: - attempts.append("SRCREV_pn-%s" % pn) + attempts.append("SRCREV:pn-%s" % pn) attempts.append("SRCREV") for a in attempts: diff --git a/poky/bitbake/lib/bb/parse/ast.py b/poky/bitbake/lib/bb/parse/ast.py index db2bdc35e..743ea0dfc 100644 --- a/poky/bitbake/lib/bb/parse/ast.py +++ b/poky/bitbake/lib/bb/parse/ast.py @@ -97,7 +97,6 @@ class DataNode(AstNode): def eval(self, data): groupd = self.groupd key = groupd["var"] - key = key.replace(":", "_") loginfo = { 'variable': key, 'file': self.filename, @@ -146,7 +145,7 @@ class DataNode(AstNode): data.setVar(key, val, parsing=True, **loginfo) class MethodNode(AstNode): - tr_tbl = str.maketrans('/.+-@%&', '_______') + tr_tbl = str.maketrans('/.+-@%&~', '________') def __init__(self, filename, lineno, func_name, body, python, fakeroot): AstNode.__init__(self, filename, lineno) @@ -208,7 +207,6 @@ class ExportFuncsNode(AstNode): def eval(self, data): for func in self.n: - func = func.replace(":", "_") calledfunc = self.classname + "_" + func if data.getVar(func, False) and not data.getVarFlag(func, 'export_func', False): diff --git a/poky/bitbake/lib/bb/runqueue.py b/poky/bitbake/lib/bb/runqueue.py index 6c41fe6d4..25e012125 100644 --- a/poky/bitbake/lib/bb/runqueue.py +++ b/poky/bitbake/lib/bb/runqueue.py @@ -2443,6 +2443,11 @@ class RunQueueExecute: if update_tasks: self.sqdone = False + for tid in [t[0] for t in update_tasks]: + h = pending_hash_index(tid, self.rqdata) + if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]: + self.sq_deferred[tid] = self.sqdata.hashes[h] + bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h])) update_scenequeue_data([t[0] for t in update_tasks], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) for (tid, harddepfail, origvalid) in update_tasks: @@ -2786,6 +2791,19 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq): sqdata.stamppresent = set() sqdata.valid = set() + sqdata.hashes = {} + sqrq.sq_deferred = {} + for mc in sorted(sqdata.multiconfigs): + for tid in sorted(sqdata.sq_revdeps): + if mc_from_tid(tid) != mc: + continue + h = pending_hash_index(tid, rqdata) + if h not in sqdata.hashes: + sqdata.hashes[h] = tid + else: + sqrq.sq_deferred[tid] = sqdata.hashes[h] + bb.note("Deferring %s after %s" % (tid, sqdata.hashes[h])) + update_scenequeue_data(sqdata.sq_revdeps, sqdata, rqdata, rq, cooker, stampcache, sqrq, summary=True) # Compute a list of 'stale' sstate tasks where the current hash does not match the one @@ -2850,32 +2868,20 @@ def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, sqrq, s sqdata.valid |= rq.validate_hashes(tocheck, cooker.data, len(sqdata.stamppresent), False, summary=summary) - sqdata.hashes = {} - sqrq.sq_deferred = {} - for mc in sorted(sqdata.multiconfigs): - for tid in sorted(sqdata.sq_revdeps): - if mc_from_tid(tid) != mc: - continue - if tid in sqdata.stamppresent: - continue - if tid in sqdata.valid: - continue - if tid in sqdata.noexec: - continue - if tid in sqrq.scenequeue_notcovered: - continue - if tid in sqrq.scenequeue_covered: - continue - - h = pending_hash_index(tid, rqdata) - if h not in sqdata.hashes: - if tid in tids: - sqdata.outrightfail.add(tid) - sqdata.hashes[h] = tid - else: - sqrq.sq_deferred[tid] = sqdata.hashes[h] - bb.note("Deferring %s after %s" % (tid, sqdata.hashes[h])) - + for tid in tids: + if tid in sqdata.stamppresent: + continue + if tid in sqdata.valid: + continue + if tid in sqdata.noexec: + continue + if tid in sqrq.scenequeue_covered: + continue + if tid in sqrq.scenequeue_notcovered: + continue + if tid in sqrq.sq_deferred: + continue + sqdata.outrightfail.add(tid) class TaskFailure(Exception): """ diff --git a/poky/bitbake/lib/bb/server/process.py b/poky/bitbake/lib/bb/server/process.py index a0955722e..6127fd40e 100644 --- a/poky/bitbake/lib/bb/server/process.py +++ b/poky/bitbake/lib/bb/server/process.py @@ -26,6 +26,7 @@ import errno import re import datetime import pickle +import traceback import bb.server.xmlrpcserver from bb import daemonize from multiprocessing import queues @@ -217,8 +218,9 @@ class ProcessServer(): self.command_channel_reply.send(self.cooker.command.runCommand(command)) serverlog("Command Completed") except Exception as e: - serverlog('Exception in server main event loop running command %s (%s)' % (command, str(e))) - logger.exception('Exception in server main event loop running command %s (%s)' % (command, str(e))) + stack = traceback.format_exc() + serverlog('Exception in server main event loop running command %s (%s)' % (command, stack)) + logger.exception('Exception in server main event loop running command %s (%s)' % (command, stack)) if self.xmlrpc in ready: self.xmlrpc.handle_requests() diff --git a/poky/bitbake/lib/bb/siggen.py b/poky/bitbake/lib/bb/siggen.py index 07692e673..3f9fe5064 100644 --- a/poky/bitbake/lib/bb/siggen.py +++ b/poky/bitbake/lib/bb/siggen.py @@ -228,7 +228,7 @@ class SignatureGeneratorBasic(SignatureGenerator): # self.dump_sigtask(fn, task, d.getVar("STAMP"), False) for task in taskdeps: - d.setVar("BB_BASEHASH_task-%s" % task, self.basehash[fn + ":" + task]) + d.setVar("BB_BASEHASH:task-%s" % task, self.basehash[fn + ":" + task]) def postparsing_clean_cache(self): # @@ -325,7 +325,7 @@ class SignatureGeneratorBasic(SignatureGenerator): h = hashlib.sha256(data.encode("utf-8")).hexdigest() self.taskhash[tid] = h - #d.setVar("BB_TASKHASH_task-%s" % task, taskhash[task]) + #d.setVar("BB_TASKHASH:task-%s" % task, taskhash[task]) return h def writeout_file_checksum_cache(self): diff --git a/poky/bitbake/lib/bb/tests/codeparser.py b/poky/bitbake/lib/bb/tests/codeparser.py index 826a2d2f6..f48520479 100644 --- a/poky/bitbake/lib/bb/tests/codeparser.py +++ b/poky/bitbake/lib/bb/tests/codeparser.py @@ -111,9 +111,9 @@ ${D}${libdir}/pkgconfig/*.pc self.assertExecs(set(["sed"])) def test_parameter_expansion_modifiers(self): - # - and + are also valid modifiers for parameter expansion, but are + # -,+ and : are also valid modifiers for parameter expansion, but are # valid characters in bitbake variable names, so are not included here - for i in ('=', ':-', ':=', '?', ':?', ':+', '#', '%', '##', '%%'): + for i in ('=', '?', '#', '%', '##', '%%'): name = "foo%sbar" % i self.parseExpression("${%s}" % name) self.assertNotIn(name, self.references) diff --git a/poky/bitbake/lib/bb/tests/data.py b/poky/bitbake/lib/bb/tests/data.py index 1d4a64b10..e667c7c7d 100644 --- a/poky/bitbake/lib/bb/tests/data.py +++ b/poky/bitbake/lib/bb/tests/data.py @@ -245,35 +245,35 @@ class TestConcatOverride(unittest.TestCase): def test_prepend(self): self.d.setVar("TEST", "${VAL}") - self.d.setVar("TEST_prepend", "${FOO}:") + self.d.setVar("TEST:prepend", "${FOO}:") self.assertEqual(self.d.getVar("TEST"), "foo:val") def test_append(self): self.d.setVar("TEST", "${VAL}") - self.d.setVar("TEST_append", ":${BAR}") + self.d.setVar("TEST:append", ":${BAR}") self.assertEqual(self.d.getVar("TEST"), "val:bar") def test_multiple_append(self): self.d.setVar("TEST", "${VAL}") - self.d.setVar("TEST_prepend", "${FOO}:") - self.d.setVar("TEST_append", ":val2") - self.d.setVar("TEST_append", ":${BAR}") + self.d.setVar("TEST:prepend", "${FOO}:") + self.d.setVar("TEST:append", ":val2") + self.d.setVar("TEST:append", ":${BAR}") self.assertEqual(self.d.getVar("TEST"), "foo:val:val2:bar") def test_append_unset(self): - self.d.setVar("TEST_prepend", "${FOO}:") - self.d.setVar("TEST_append", ":val2") - self.d.setVar("TEST_append", ":${BAR}") + self.d.setVar("TEST:prepend", "${FOO}:") + self.d.setVar("TEST:append", ":val2") + self.d.setVar("TEST:append", ":${BAR}") self.assertEqual(self.d.getVar("TEST"), "foo::val2:bar") def test_remove(self): self.d.setVar("TEST", "${VAL} ${BAR}") - self.d.setVar("TEST_remove", "val") + self.d.setVar("TEST:remove", "val") self.assertEqual(self.d.getVar("TEST"), " bar") def test_remove_cleared(self): self.d.setVar("TEST", "${VAL} ${BAR}") - self.d.setVar("TEST_remove", "val") + self.d.setVar("TEST:remove", "val") self.d.setVar("TEST", "${VAL} ${BAR}") self.assertEqual(self.d.getVar("TEST"), "val bar") @@ -281,42 +281,42 @@ class TestConcatOverride(unittest.TestCase): # (including that whitespace is preserved) def test_remove_inactive_override(self): self.d.setVar("TEST", "${VAL} ${BAR} 123") - self.d.setVar("TEST_remove_inactiveoverride", "val") + self.d.setVar("TEST:remove:inactiveoverride", "val") self.assertEqual(self.d.getVar("TEST"), "val bar 123") def test_doubleref_remove(self): self.d.setVar("TEST", "${VAL} ${BAR}") - self.d.setVar("TEST_remove", "val") + self.d.setVar("TEST:remove", "val") self.d.setVar("TEST_TEST", "${TEST} ${TEST}") self.assertEqual(self.d.getVar("TEST_TEST"), " bar bar") def test_empty_remove(self): self.d.setVar("TEST", "") - self.d.setVar("TEST_remove", "val") + self.d.setVar("TEST:remove", "val") self.assertEqual(self.d.getVar("TEST"), "") def test_remove_expansion(self): self.d.setVar("BAR", "Z") self.d.setVar("TEST", "${BAR}/X Y") - self.d.setVar("TEST_remove", "${BAR}/X") + self.d.setVar("TEST:remove", "${BAR}/X") self.assertEqual(self.d.getVar("TEST"), " Y") def test_remove_expansion_items(self): self.d.setVar("TEST", "A B C D") self.d.setVar("BAR", "B D") - self.d.setVar("TEST_remove", "${BAR}") + self.d.setVar("TEST:remove", "${BAR}") self.assertEqual(self.d.getVar("TEST"), "A C ") def test_remove_preserve_whitespace(self): # When the removal isn't active, the original value should be preserved self.d.setVar("TEST", " A B") - self.d.setVar("TEST_remove", "C") + self.d.setVar("TEST:remove", "C") self.assertEqual(self.d.getVar("TEST"), " A B") def test_remove_preserve_whitespace2(self): # When the removal is active preserve the whitespace self.d.setVar("TEST", " A B") - self.d.setVar("TEST_remove", "B") + self.d.setVar("TEST:remove", "B") self.assertEqual(self.d.getVar("TEST"), " A ") class TestOverrides(unittest.TestCase): @@ -329,70 +329,70 @@ class TestOverrides(unittest.TestCase): self.assertEqual(self.d.getVar("TEST"), "testvalue") def test_one_override(self): - self.d.setVar("TEST_bar", "testvalue2") + self.d.setVar("TEST:bar", "testvalue2") self.assertEqual(self.d.getVar("TEST"), "testvalue2") def test_one_override_unset(self): - self.d.setVar("TEST2_bar", "testvalue2") + self.d.setVar("TEST2:bar", "testvalue2") self.assertEqual(self.d.getVar("TEST2"), "testvalue2") - self.assertCountEqual(list(self.d.keys()), ['TEST', 'TEST2', 'OVERRIDES', 'TEST2_bar']) + self.assertCountEqual(list(self.d.keys()), ['TEST', 'TEST2', 'OVERRIDES', 'TEST2:bar']) def test_multiple_override(self): - self.d.setVar("TEST_bar", "testvalue2") - self.d.setVar("TEST_local", "testvalue3") - self.d.setVar("TEST_foo", "testvalue4") + self.d.setVar("TEST:bar", "testvalue2") + self.d.setVar("TEST:local", "testvalue3") + self.d.setVar("TEST:foo", "testvalue4") self.assertEqual(self.d.getVar("TEST"), "testvalue3") - self.assertCountEqual(list(self.d.keys()), ['TEST', 'TEST_foo', 'OVERRIDES', 'TEST_bar', 'TEST_local']) + self.assertCountEqual(list(self.d.keys()), ['TEST', 'TEST:foo', 'OVERRIDES', 'TEST:bar', 'TEST:local']) def test_multiple_combined_overrides(self): - self.d.setVar("TEST_local_foo_bar", "testvalue3") + self.d.setVar("TEST:local:foo:bar", "testvalue3") self.assertEqual(self.d.getVar("TEST"), "testvalue3") def test_multiple_overrides_unset(self): - self.d.setVar("TEST2_local_foo_bar", "testvalue3") + self.d.setVar("TEST2:local:foo:bar", "testvalue3") self.assertEqual(self.d.getVar("TEST2"), "testvalue3") def test_keyexpansion_override(self): self.d.setVar("LOCAL", "local") - self.d.setVar("TEST_bar", "testvalue2") - self.d.setVar("TEST_${LOCAL}", "testvalue3") - self.d.setVar("TEST_foo", "testvalue4") + self.d.setVar("TEST:bar", "testvalue2") + self.d.setVar("TEST:${LOCAL}", "testvalue3") + self.d.setVar("TEST:foo", "testvalue4") bb.data.expandKeys(self.d) self.assertEqual(self.d.getVar("TEST"), "testvalue3") def test_rename_override(self): - self.d.setVar("ALTERNATIVE_ncurses-tools_class-target", "a") + self.d.setVar("ALTERNATIVE:ncurses-tools:class-target", "a") self.d.setVar("OVERRIDES", "class-target") - self.d.renameVar("ALTERNATIVE_ncurses-tools", "ALTERNATIVE_lib32-ncurses-tools") - self.assertEqual(self.d.getVar("ALTERNATIVE_lib32-ncurses-tools"), "a") + self.d.renameVar("ALTERNATIVE:ncurses-tools", "ALTERNATIVE:lib32-ncurses-tools") + self.assertEqual(self.d.getVar("ALTERNATIVE:lib32-ncurses-tools"), "a") def test_underscore_override(self): - self.d.setVar("TEST_bar", "testvalue2") - self.d.setVar("TEST_some_val", "testvalue3") - self.d.setVar("TEST_foo", "testvalue4") + self.d.setVar("TEST:bar", "testvalue2") + self.d.setVar("TEST:some_val", "testvalue3") + self.d.setVar("TEST:foo", "testvalue4") self.d.setVar("OVERRIDES", "foo:bar:some_val") self.assertEqual(self.d.getVar("TEST"), "testvalue3") def test_remove_with_override(self): - self.d.setVar("TEST_bar", "testvalue2") - self.d.setVar("TEST_some_val", "testvalue3 testvalue5") - self.d.setVar("TEST_some_val_remove", "testvalue3") - self.d.setVar("TEST_foo", "testvalue4") + self.d.setVar("TEST:bar", "testvalue2") + self.d.setVar("TEST:some_val", "testvalue3 testvalue5") + self.d.setVar("TEST:some_val:remove", "testvalue3") + self.d.setVar("TEST:foo", "testvalue4") self.d.setVar("OVERRIDES", "foo:bar:some_val") self.assertEqual(self.d.getVar("TEST"), " testvalue5") def test_append_and_override_1(self): - self.d.setVar("TEST_append", "testvalue2") - self.d.setVar("TEST_bar", "testvalue3") + self.d.setVar("TEST:append", "testvalue2") + self.d.setVar("TEST:bar", "testvalue3") self.assertEqual(self.d.getVar("TEST"), "testvalue3testvalue2") def test_append_and_override_2(self): - self.d.setVar("TEST_append_bar", "testvalue2") + self.d.setVar("TEST:append:bar", "testvalue2") self.assertEqual(self.d.getVar("TEST"), "testvaluetestvalue2") def test_append_and_override_3(self): - self.d.setVar("TEST_bar_append", "testvalue2") + self.d.setVar("TEST:bar:append", "testvalue2") self.assertEqual(self.d.getVar("TEST"), "testvalue2") # Test an override with _<numeric> in it based on a real world OE issue @@ -400,11 +400,16 @@ class TestOverrides(unittest.TestCase): self.d.setVar("TARGET_ARCH", "x86_64") self.d.setVar("PN", "test-${TARGET_ARCH}") self.d.setVar("VERSION", "1") - self.d.setVar("VERSION_pn-test-${TARGET_ARCH}", "2") + self.d.setVar("VERSION:pn-test-${TARGET_ARCH}", "2") self.d.setVar("OVERRIDES", "pn-${PN}") bb.data.expandKeys(self.d) self.assertEqual(self.d.getVar("VERSION"), "2") + def test_append_and_unused_override(self): + # Had a bug where an unused override append could return "" instead of None + self.d.setVar("BAR:append:unusedoverride", "testvalue2") + self.assertEqual(self.d.getVar("BAR"), None) + class TestKeyExpansion(unittest.TestCase): def setUp(self): self.d = bb.data.init() @@ -498,7 +503,7 @@ class TaskHash(unittest.TestCase): d.setVar("VAR", "val") # Adding an inactive removal shouldn't change the hash d.setVar("BAR", "notbar") - d.setVar("MYCOMMAND_remove", "${BAR}") + d.setVar("MYCOMMAND:remove", "${BAR}") nexthash = gettask_bashhash("mytask", d) self.assertEqual(orighash, nexthash) diff --git a/poky/bitbake/lib/bb/tests/parse.py b/poky/bitbake/lib/bb/tests/parse.py index 9e21e1842..4d17f82ed 100644 --- a/poky/bitbake/lib/bb/tests/parse.py +++ b/poky/bitbake/lib/bb/tests/parse.py @@ -98,8 +98,8 @@ exportD = "d" overridetest = """ -RRECOMMENDS_${PN} = "a" -RRECOMMENDS_${PN}_libc = "b" +RRECOMMENDS:${PN} = "a" +RRECOMMENDS:${PN}:libc = "b" OVERRIDES = "libc:${PN}" PN = "gtk+" """ @@ -110,13 +110,13 @@ PN = "gtk+" self.assertEqual(d.getVar("RRECOMMENDS"), "b") bb.data.expandKeys(d) self.assertEqual(d.getVar("RRECOMMENDS"), "b") - d.setVar("RRECOMMENDS_gtk+", "c") + d.setVar("RRECOMMENDS:gtk+", "c") self.assertEqual(d.getVar("RRECOMMENDS"), "c") overridetest2 = """ EXTRA_OECONF = "" -EXTRA_OECONF_class-target = "b" -EXTRA_OECONF_append = " c" +EXTRA_OECONF:class-target = "b" +EXTRA_OECONF:append = " c" """ def test_parse_overrides(self): @@ -128,7 +128,7 @@ EXTRA_OECONF_append = " c" overridetest3 = """ DESCRIPTION = "A" -DESCRIPTION_${PN}-dev = "${DESCRIPTION} B" +DESCRIPTION:${PN}-dev = "${DESCRIPTION} B" PN = "bc" """ @@ -136,15 +136,15 @@ PN = "bc" f = self.parsehelper(self.overridetest3) d = bb.parse.handle(f.name, self.d)[''] bb.data.expandKeys(d) - self.assertEqual(d.getVar("DESCRIPTION_bc-dev"), "A B") + self.assertEqual(d.getVar("DESCRIPTION:bc-dev"), "A B") d.setVar("DESCRIPTION", "E") - d.setVar("DESCRIPTION_bc-dev", "C D") + d.setVar("DESCRIPTION:bc-dev", "C D") d.setVar("OVERRIDES", "bc-dev") self.assertEqual(d.getVar("DESCRIPTION"), "C D") classextend = """ -VAR_var_override1 = "B" +VAR_var:override1 = "B" EXTRA = ":override1" OVERRIDES = "nothing${EXTRA}" diff --git a/poky/bitbake/lib/bb/ui/taskexp.py b/poky/bitbake/lib/bb/ui/taskexp.py index 2b246710c..c00eaf663 100644 --- a/poky/bitbake/lib/bb/ui/taskexp.py +++ b/poky/bitbake/lib/bb/ui/taskexp.py @@ -8,6 +8,7 @@ # import sys +import traceback try: import gi @@ -196,6 +197,7 @@ def main(server, eventHandler, params): gtkgui.start() try: + params.updateToServer(server, os.environ.copy()) params.updateFromServer(server) cmdline = params.parseActions() if not cmdline: @@ -218,6 +220,9 @@ def main(server, eventHandler, params): except client.Fault as x: print("XMLRPC Fault getting commandline:\n %s" % x) return + except Exception as e: + print("Exception in startup:\n %s" % traceback.format_exc()) + return if gtkthread.quit.isSet(): return diff --git a/poky/bitbake/lib/bb/utils.py b/poky/bitbake/lib/bb/utils.py index 6ba1d2a37..e6e82d111 100644 --- a/poky/bitbake/lib/bb/utils.py +++ b/poky/bitbake/lib/bb/utils.py @@ -1178,7 +1178,7 @@ def edit_metadata(meta_lines, variables, varfunc, match_overrides=False): variables: a list of variable names to look for. Functions may also be specified, but must be specified with '()' at the end of the name. Note that the function doesn't have - any intrinsic understanding of _append, _prepend, _remove, + any intrinsic understanding of :append, :prepend, :remove, or overrides, so these are considered as part of the name. These values go into a regular expression, so regular expression syntax is allowed. diff --git a/poky/bitbake/lib/hashserv/tests.py b/poky/bitbake/lib/hashserv/tests.py index e2b762dbf..e851535c5 100644 --- a/poky/bitbake/lib/hashserv/tests.py +++ b/poky/bitbake/lib/hashserv/tests.py @@ -15,28 +15,32 @@ import tempfile import threading import unittest import socket +import time +import signal -def _run_server(server, idx): - # logging.basicConfig(level=logging.DEBUG, filename='bbhashserv.log', filemode='w', - # format='%(levelname)s %(filename)s:%(lineno)d %(message)s') +def server_prefunc(server, idx): + logging.basicConfig(level=logging.DEBUG, filename='bbhashserv.log', filemode='w', + format='%(levelname)s %(filename)s:%(lineno)d %(message)s') + server.logger.debug("Running server %d" % idx) sys.stdout = open('bbhashserv-%d.log' % idx, 'w') sys.stderr = sys.stdout - server.serve_forever() - class HashEquivalenceTestSetup(object): METHOD = 'TestMethod' server_index = 0 - def start_server(self, dbpath=None, upstream=None, read_only=False): + def start_server(self, dbpath=None, upstream=None, read_only=False, prefunc=server_prefunc): self.server_index += 1 if dbpath is None: dbpath = os.path.join(self.temp_dir.name, "db%d.sqlite" % self.server_index) - def cleanup_thread(thread): - thread.terminate() - thread.join() + def cleanup_server(server): + if server.process.exitcode is not None: + return + + server.process.terminate() + server.process.join() server = create_server(self.get_server_addr(self.server_index), dbpath, @@ -44,9 +48,8 @@ class HashEquivalenceTestSetup(object): read_only=read_only) server.dbpath = dbpath - server.thread = multiprocessing.Process(target=_run_server, args=(server, self.server_index)) - server.thread.start() - self.addCleanup(cleanup_thread, server.thread) + server.serve_as_process(prefunc=prefunc, args=(self.server_index,)) + self.addCleanup(cleanup_server, server) def cleanup_client(client): client.close() @@ -283,6 +286,33 @@ class HashEquivalenceCommonTests(object): self.assertClientGetHash(self.client, taskhash2, None) + def test_slow_server_start(self): + """ + Ensures that the server will exit correctly even if it gets a SIGTERM + before entering the main loop + """ + + event = multiprocessing.Event() + + def prefunc(server, idx): + nonlocal event + server_prefunc(server, idx) + event.wait() + + def do_nothing(signum, frame): + pass + + old_signal = signal.signal(signal.SIGTERM, do_nothing) + self.addCleanup(signal.signal, signal.SIGTERM, old_signal) + + _, server = self.start_server(prefunc=prefunc) + server.process.terminate() + time.sleep(30) + event.set() + server.process.join(300) + self.assertIsNotNone(server.process.exitcode, "Server did not exit in a timely manner!") + + class TestHashEquivalenceUnixServer(HashEquivalenceTestSetup, HashEquivalenceCommonTests, unittest.TestCase): def get_server_addr(self, server_idx): return "unix://" + os.path.join(self.temp_dir.name, 'sock%d' % server_idx) diff --git a/poky/bitbake/lib/toaster/orm/models.py b/poky/bitbake/lib/toaster/orm/models.py index 7f7e922ad..4c94b407d 100644 --- a/poky/bitbake/lib/toaster/orm/models.py +++ b/poky/bitbake/lib/toaster/orm/models.py @@ -1719,7 +1719,7 @@ class CustomImageRecipe(Recipe): """Generate the contents for the recipe file.""" # If we have no excluded packages we only need to _append if self.excludes_set.count() == 0: - packages_conf = "IMAGE_INSTALL_append = \" " + packages_conf = "IMAGE_INSTALL:append = \" " for pkg in self.appends_set.all(): packages_conf += pkg.name+' ' diff --git a/poky/bitbake/lib/toaster/toastergui/views.py b/poky/bitbake/lib/toaster/toastergui/views.py index 9a5e48e3b..74f9d5696 100644 --- a/poky/bitbake/lib/toaster/toastergui/views.py +++ b/poky/bitbake/lib/toaster/toastergui/views.py @@ -1708,7 +1708,7 @@ if True: except ProjectVariable.DoesNotExist: pass try: - return_data['image_install_append'] = ProjectVariable.objects.get(project = prj, name = "IMAGE_INSTALL_append").value, + return_data['image_install:append'] = ProjectVariable.objects.get(project = prj, name = "IMAGE_INSTALL:append").value, except ProjectVariable.DoesNotExist: pass try: @@ -1839,7 +1839,7 @@ if True: except ProjectVariable.DoesNotExist: pass try: - context['image_install_append'] = ProjectVariable.objects.get(project = prj, name = "IMAGE_INSTALL_append").value + context['image_install:append'] = ProjectVariable.objects.get(project = prj, name = "IMAGE_INSTALL:append").value context['image_install_append_defined'] = "1" except ProjectVariable.DoesNotExist: pass diff --git a/poky/bitbake/lib/toaster/toastermain/management/commands/buildimport.py b/poky/bitbake/lib/toaster/toastermain/management/commands/buildimport.py index 59da6ff7a..e25b55e5a 100644 --- a/poky/bitbake/lib/toaster/toastermain/management/commands/buildimport.py +++ b/poky/bitbake/lib/toaster/toastermain/management/commands/buildimport.py @@ -451,7 +451,7 @@ class Command(BaseCommand): # Catch vars relevant to Toaster (in case no Toaster section) self.update_project_vars(project,'DISTRO') self.update_project_vars(project,'MACHINE') - self.update_project_vars(project,'IMAGE_INSTALL_append') + self.update_project_vars(project,'IMAGE_INSTALL:append') self.update_project_vars(project,'IMAGE_FSTYPES') self.update_project_vars(project,'PACKAGE_CLASSES') # These vars are typically only assigned by Toaster |