diff options
| author | Patrick Williams <patrick@stwcx.xyz> | 2022-10-18 20:51:29 +0300 |
|---|---|---|
| committer | Patrick Williams <patrick@stwcx.xyz> | 2022-10-18 20:53:04 +0300 |
| commit | c67ef227fe09ebd2213c47709a37a70784232b12 (patch) | |
| tree | 5fd573a5df398965aa7e7f88b42a8db2144e8a0e /poky/bitbake/lib/bb | |
| parent | 53fdac2b0aee16e297ce86b473c56547ff1330ac (diff) | |
| download | openbmc-c67ef227fe09ebd2213c47709a37a70784232b12.tar.xz | |
subtree updates
poky: 387ab5f18b..eaf8ce9d39:
Alejandro Hernandez Samaniego (1):
rootfs.py: dont try to list installed packages for baremetal images
Alex Stewart (1):
maintainers: update opkg maintainer
Alexander Kanavin (26):
devtool/upgrade: correctly clean up when recipe filename isn't yet known
devtool/upgrade: catch bb.fetch2.decodeurl errors
scripts/oe-setup-builddir: make it known where configurations come from
bluez5: update 5.64 -> 5.65
libwpe: upgrade 1.12.0 -> 1.12.2
ell: upgrade 0.49 -> 0.50
iso-codes: upgrade 4.10.0 -> 4.11.0
libcap: upgrade 2.64 -> 2.65
libwebp: upgrade 1.2.2 -> 1.2.3
mobile-broadband-provider-info: upgrade 20220511 -> 20220725
webkitgtk: upgrade 2.36.4 -> 2.36.5
weston: upgrade 10.0.1 -> 10.0.2
tzdata: upgrade 2022a -> 2022b
xz: update 5.2.5 -> 5.2.6
gdk-pixbuf: upgrade 2.42.6 -> 2.42.8
gdk-pixbuf: update 2.42.8 -> 2.42.9
epiphany: upgrade 42.3 -> 42.4
glib-networking: upgrade 2.72.1 -> 2.72.2
libjpeg-turbo: upgrade 2.1.3 -> 2.1.4
libwebp: upgrade 1.2.3 -> 1.2.4
wireless-regdb: upgrade 2022.06.06 -> 2022.08.12
wpebackend-fdo: upgrade 1.12.0 -> 1.12.1
bind: upgrade 9.18.4 -> 9.18.5
lighttpd: upgrade 1.4.65 -> 1.4.66
rpm: update 4.17.0 -> 4.17.1
tzdata: update to 2022d
Alexandre Belloni (3):
ruby: drop capstone support
runqemu: display host uptime when starting
oeqa/runtime/dnf: fix typo
Andrei Gherzan (4):
linux-yocto: Fix COMPATIBLE_MACHINE regex match
shadow: Enable subid support
rootfspostcommands.py: Cleanup subid backup files generated by shadow-utils
shadow: Avoid nss warning/error with musl
Anuj Mittal (1):
poky.conf: add ubuntu-22.04 to tested distros
Aryaman Gupta (2):
bitbake: bitbake: runqueue: add cpu/io pressure regulation
bitbake: bitbake: runqueue: add memory pressure regulation
Awais Belal (1):
kernel-fitimage.bbclass: only package unique DTBs
Beniamin Sandu (1):
libpam: use /run instead of /var/run in systemd tmpfiles
Bertrand Marquis (1):
sysvinit-inittab/start_getty: Fix respawn too fast
Bruce Ashfield (16):
lttng-modules: fix 5.19+ build
lttng-modules: fix build against mips and v5.19 kernel
lttng-modules: replace mips compaction fix with upstream change
linux-yocto/5.15: update to v5.15.60
linux-yocto/5.15: update to v5.15.62
linux-yocto/5.10: update to v5.10.136
linux-yocto/5.10: update to v5.10.137
linux-yocto/5.10: update to v5.10.141
linux-yocto/5.10: update to v5.10.143
linux-yocto/5.15: update to v5.15.63
linux-yocto/5.15: update to v5.15.65
linux-yocto/5.15: update to v5.15.68
linux-yocto/5.15: cfg: fix ACPI warnings for -tiny
kernel-yocto: allow patch author date to be commit date
kern-tools: fix queue processing in relative TOPDIR configurations
kern-tools: allow 'y' or 'm' to avoid config audit warnings
Changqing Li (1):
apt: fix nativesdk-apt build failure during the second time build
Chee Yang Lee (1):
sqlite: add CVE-2022-35737 patch to SRC_URI
Daiane Angolini (1):
python3-pip: Fix RDEPENDS after the update
Daniel McGregor (1):
coreutils: add openssl PACKAGECONFIG
Denys Dmytriyenko (1):
glibc-locale: explicitly remove empty dirs in ${libdir}
Dmitry Baryshkov (2):
linux-firmware: upgrade 20220708 -> 20220913
linux-firmware: package new Qualcomm firmware
Enrico Scholz (5):
npm: replace 'npm pack' call by 'tar czf'
npm: return content of 'package.json' in 'npm_pack'
npm: take 'version' directly from 'package.json'
lib:npm_registry: initial checkin
npm: use npm_registry to cache package
Ernst Sjöstrand (1):
cve-check: Don't use f-strings
Florin Diaconescu (4):
expat: upgrade 2.4.7 -> 2.4.8
expat: upgrade 2.4.8 -> 2.4.9
rsync: update 3.2.3 -> 3.2.4
rsync: update 3.2.4 -> 3.2.5
Gennaro Iorio (1):
bitbake: fetch2: gitsm: fix incorrect handling of git submodule relative urls
He Zhe (3):
lttng-tools: Disable on qemuriscv32
stress-cpu: disable float128 math on powerpc64 to avoid SIGILL
lttng-tools: Disable on riscv32
Hitendra Prajapati (5):
gdk-pixbuf: CVE-2021-46829 a heap-based buffer overflow
gnutls: CVE-2022-2509 Double free during gnutls_pkcs7_verify
zlib: CVE-2022-37434 a heap-based buffer over-read
libtiff: CVE-2022-34526 A stack overflow was discovered
Revert "gdk-pixbuf: CVE-2021-46829 a heap-based buffer overflow"
Jacob Kroon (1):
bitbake: bitbake-user-manual: Correct description of the ??= operator
Jon Mason (2):
ref-manual: add numa to machine features
oeqa/parselogs: add qemuarmv5 arm-charlcd masking
Jose Quaresma (7):
archiver.bbclass: remove unsed do_deploy_archives[dirs]
create-spdx: ignore packing control files from ipk and deb
archiver.bbclass: some recipes that uses the kernelsrc bbclass uses the shared source
linux-yocto: prepend the the value with a space when append to KERNEL_EXTRA_ARGS
bitbake: bitbake: bitbake-user-manual: hashserv can be accessed on a dedicated domain
bitbake: bb/utils: remove: check the path again the expand python glob
bitbake: bb/utils: movefile: use the logger for printing
Joshua Watt (4):
bitbake: utils: Pass lock argument in fileslocked
classes: cve-check: Get shared database lock
oeqa: qemurunner: Report UNIX Epoch timestamp on login
bitbake: siggen: Fix insufficent entropy in sigtask file names
Kai Kang (1):
packagegroup-self-hosted: update for strace
Khem Raj (15):
libxml2: Ignore CVE-2016-3709
connman: Backports for security fixes
cracklib: Drop using register keyword
tcp-wrappers: Fix implicit-function-declaration warnings
xinetd: Pass missing -D_GNU_SOURCE
watchdog: Include needed system header for function decls
pinentry: enable _XOPEN_SOURCE on musl for wchar usage in curses
apr: Use correct strerror_r implementation based on libc type
gcr: Define _GNU_SOURCE
apr: Cache configure tests which use AC_TRY_RUN
autoconf: Fix strict prototype errors in generated tests
autoconf: Update K & R stype functions
webkitgtk: Upgrade to 2.36.6 minor update
webkitgtk: Update to 2.36.7
rpm: Remove -Wimplicit-function-declaration warnings
Kristian Amlie (1):
externalsrc: Don't wipe out src dir when EXPORT_FUNCTIONS is used.
LUIS ENRIQUEZ (1):
kernel-fitimage.bbclass: add padding algorithm property in config nodes
Mark Hatle (1):
runqemu: Add missing space on default display option
Martin Beeger (1):
cmake: remove CMAKE_ASM_FLAGS variable in toolchain file
Martin Jansa (2):
libxml2: Port gentest.py to Python-3
create-pull-request: don't switch the git remote protocol to git://
Mateusz Marciniec (1):
util-linux: Remove --enable-raw from EXTRA_OECONF
Michael Opdenacker (7):
migration guides: add missing release notes
bitbake: doc: bitbake-user-manual: add explicit target for crates fetcher
bitbake: doc: bitbake-user-manual: document npm and npmsw fetchers
bitbake: bitbake-user-manual: npm fetcher: improve description of SRC_URI format
poky.yaml.in: update version requirements
migration-guides: add 4.0.4 release notes
dev-manual: fix reference to BitBake user manual
Mihai Lindner (1):
create-spdx: Fix supplier field
Mikko Rapeli (7):
boost: fix install of fiber shared libraries
bitbake: event.py: ignore exceptions from stdout and sterr operations in atexit
u-boot: switch from append to += in SRC_URI
glibc-tests: use += instead of :append
go-native: switch from SRC_URI:append to SRC_URI +=
python3-rfc3986-validator: switch from SRC_URI:append to SRC_URI +=
linux-libc-headers: switch from SRC_URI:append to SRC_URI +=
Ming Liu (1):
meta: introduce UBOOT_MKIMAGE_KERNEL_TYPE
Mingli Yu (1):
busybox: add devmem 128-bit support
Neil Horman (1):
bitbake: Fix npm to use https rather than http
Ola x Nilsson (1):
bitbake: ConfHandler: Remove lingering close
Otavio Salvador (1):
bitbake: toaster: fix kirkstone version
Paul Eggleton (1):
relocate_sdk.py: ensure interpreter size error causes relocation to fail
Pavel Zhukov (4):
package_rpm: Do not replace square brackets in %files
parselogs: Ignore xf86OpenConsole error
core-image.bbclass: Exclude openssh complementary packages
bitbake: gitsm: Error out if submodule refers to parent repo
Peter Bergin (1):
rootfs-postcommands.bbclass: avoid moving ssh host keys if etc is writable
Peter Kjellerstedt (1):
cairo: Adapt the license information based on what is being built
Peter Marko (1):
create-spdx: handle links to inaccessible locations
Rajesh Dangi (2):
linux-yocto/5.15: update genericx86* machines to v5.15.59
linux-yocto/5.10: update genericx86* machines to v5.10.135
Randy MacLeod (1):
vim: update from 9.0.0063 to 9.0.0115
Rasmus Villemoes (1):
bitbake.conf: set BB_DEFAULT_UMASK using ??=
Richard Purdie (25):
nativesdk: Clear TUNE_FEATURES
selftest/wic: Tweak test case to not depend on kernel size
bitbake: runqueue: Change pressure file warning to a note
perf: Fix reproducibility issues with 5.19 onwards
vim: Upgrade 9.0.0115 -> 9.0.0242
vim: Upgrade 9.0.0242 -> 9.0.0341
pseudo: Update to include recent upstream minor fixes
bitbake: runqueue: Fix unihash cache mismatch issues
bitbake: cooker: Drop sre_constants usage
bitbake: ConfHandler/BBHandler: Improve comment error messages and add tests
bitbake: fetch2: Ensure directory exists before creating symlink
gcc-multilib-config: Fix i686 toolchain relocation issues
kernel: Always set CC and LD for the kernel build
kernel: Use consistent make flags for menuconfig
vim: Upgrade 9.0.0341 -> 9.0.0453
build-appliance-image: Update to kirkstone head revision
libpng: upgrade 1.6.37 -> 1.6.38
vim: Upgrade 9.0.453 -> 9.0.541
perf: Fix for recent kernel upgrades
vim: Upgrade 9.0.0541 -> 9.0.0598
bitbake: runqueue: Ensure deferred tasks are sorted by multiconfig
bitbake: runqueue: Improve deadlock warning messages
bitbake: runqueue: Drop deadlock breaking force fail
bitbake: bitbake: Add copyright headers where missing
bitbake: asyncrpc/client: Fix unix domain socket chdir race issues
Robert Joslyn (2):
curl: Backport patch for CVE-2022-35252
tzdata: Update from 2022b to 2022c
Roland Hieber (1):
devtool: error out when workspace is using old override syntax
Ross Burton (8):
oeqa/qemurunner: add run_serial() comment
oeqa/selftest: rename git.py to intercept.py
oeqa/gotoolchain: put writable files in the Go module cache
oeqa/gotoolchain: set CGO_ENABLED=1
wic: add target tools to PATH when executing native commands
wic/bootimg-efi: use cross objcopy when building unified kernel image
wic: depend on cross-binutils
cve-check: close cursors as soon as possible
Ruiqiang Hao (2):
gcc: add arm-v9 support
tune-neoversen2: support tune-neoversen2 base on armv9a
Sakib Sajal (9):
qemu: fix CVE-2021-3507
qemu: fix CVE-2021-3929
qemu: fix CVE-2021-4158
qemu: fix CVE-2022-0358
qemu: fix CVE-2022-0216
u-boot: fix CVE-2022-33103
u-boot: fix CVE-2022-30552
u-boot: fix CVE-2022-33967
go: update v1.17.12 -> v1.17.13
Samuli Piippo (2):
Revert "gcc-cross-canadian: Add symlink to real-ld alongside other symlinks"
gcc-cross-canadian: add default plugin linker
Shubham Kulkarni (1):
sanity: add a comment to ensure CONNECTIVITY_CHECK_URIS is correct
Steve Sakoman (3):
lttng-modules: fix build for kernel 5.10.137
poky.conf: bump version for 4.0.4
system-requirements.rst: Add Ubuntu 22.04 to list of supported distros
Sundeep KOKKONDA (1):
glibc: stable 2.35 branch updates.
Teoh Jay Shen (3):
go: fix CVE-2022-27664
inetutils: fix CVE-2022-39028 - remote DoS vulnerability in inetutils-telnetd
bind: upgrade 9.18.6 -> 9.18.7
Ulrich Ölmann (1):
scripts/runqemu.README: fix typos and trailing whitespaces
Xiangyu Chen (1):
ltp: Fix pread02 case trigger the glibc overflow detection
Yang Xu (1):
insane.bbclass: Skip patches not in oe-core by full path
Yongxin Liu (1):
grub2: fix several CVEs
ghassaneben (1):
sqlite: fix CVE-2022-35737
niko.mauno@vaisala.com (2):
systemd: Fix unwritable /var/lock when no sysvinit handling
systemd: Add 'no-dns-fallback' PACKAGECONFIG option
pgowda (3):
binutils : CVE-2022-38533
binutils: fix CVE-2022-38126
binutils : Fix CVE-2022-38127
wangmy (10):
libcap: upgrade 2.63 -> 2.64
libtasn1: upgrade 4.18.0 -> 4.19.0
liburcu: upgrade 0.13.1 -> 0.13.2
libwpe: upgrade 1.12.2 -> 1.12.3
libatomic-ops: upgrade 7.6.12 -> 7.6.14
lz4: upgrade 1.9.3 -> 1.9.4
cracklib: upgrade 2.9.7 -> 2.9.8
vala: upgrade 0.56.2 -> 0.56.3
lighttpd: upgrade 1.4.64 -> 1.4.65
bind: upgrade 9.18.5 -> 9.18.6
meta-raspberrypi: 0135a02ea5..dacad9302a:
Lluis Campos (1):
rpi-cmdline: do_compile: Use pure Python syntax to get `CMDLINE`
Vinicius Aquino (1):
raspberrypi-firmware: Update to 20220830 snapshot
meta-openembedded: acbe748798..744a4b6eda:
Changqing Li (2):
fuse3: support ptest
fuse3: fix ptest test_passthrough_hp failure
Chen Qi (1):
polkit: refresh patch
Enrico Scholz (1):
nodejs-oe-cache-native: initial checkin
Hitendra Prajapati (1):
wireshark: CVE-2022-3190 Infinite loop in legacy style dissector
Hitomi Hasegawa (1):
libsdl: add CVE-2019-14906 to allowlist
Jose Quaresma (2):
wireguard-module: 1.0.20210219 -> 1.0.20220627
wireguard-tools: Add a new package for wg-quick
Justin Bronder (1):
lmdb: only set SONAME on the shared library
Khem Raj (5):
audit: Upgrade to 3.0.8 and fix build with linux 5.17+
ntpsec: Add -D_GNU_SOURCE and fix building with devtool
gd: Fix build with clang-15
safec: Remove unused variable 'len'
audit: Revert the tweak done in configure step in do_install
Lei Maohui (1):
xrdp: Fix buildpaths warning.
Martin Jansa (1):
libcec: fix runtime dependencies for ${PN}-examples
Mingli Yu (1):
postgresql: make sure pam conf installed when pam enabled
Ovidiu Panait (1):
net-snmp: upgrade 5.9.1 -> 5.9.3
Richard Purdie (1):
lmdb: Don't inherit base
Sakib Sajal (1):
minicoredumper: retry elf parsing as long as needed
Saul Wold (10):
libipc-signal-perl: Fix LICENSE string
libdigest-hmac-perl: Fix LICENSE string
libio-socket-ssl-perl: Fix LICENSE string
libdigest-sha1-perl: Fix LICENSE string
libmime-types-perl: Fix LICENSE string
libauthen-sasl-perl: Fix LICENSE string
libnet-ldap-perl: Fix LICENSE string
libxml-libxml-perl: Fix LICENSE string
libnet-telnet-perl: Fix LICENSE string
libproc-waitstat-perl: Fix LICENSE string
Steffen Olsen (1):
postgreql: Fix pg_config not working after buildpaths patch
Wang Mingyu (3):
php: upgrade 8.1.8 -> 8.1.9
postgresql: upgrade 14.4 -> 14.5
tcpreplay: upgrade 4.4.1 -> 4.4.2
Yi Zhao (6):
libldb: upgrade 2.3.3 -> 2.3.4
samba: upgrade 4.14.13 -> 4.14.14
samba: fix buildpaths issue
frr: Security fix CVE-2022-37035
open-vm-tools: Security fix CVE-2022-31676
frr: Security fix CVE-2022-37032
wangmy (2):
php: upgrade 8.1.9 -> 8.1.10
dnsmasq: upgrade 2.86 -> 2.87
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Change-Id: I02f0e5b5dcf292a12933c694a10d0946b0edcbc4
Diffstat (limited to 'poky/bitbake/lib/bb')
25 files changed, 222 insertions, 42 deletions
diff --git a/poky/bitbake/lib/bb/COW.py b/poky/bitbake/lib/bb/COW.py index 23c22b65ef..76bc08a3ea 100644 --- a/poky/bitbake/lib/bb/COW.py +++ b/poky/bitbake/lib/bb/COW.py @@ -3,6 +3,8 @@ # # Copyright (C) 2006 Tim Ansell # +# SPDX-License-Identifier: GPL-2.0-only +# # Please Note: # Be careful when using mutable types (ie Dict and Lists) - operations involving these are SLOW. # Assign a file to __warn__ to get warnings about slow operations. diff --git a/poky/bitbake/lib/bb/asyncrpc/__init__.py b/poky/bitbake/lib/bb/asyncrpc/__init__.py index c2f2b3c00b..9a85e9965b 100644 --- a/poky/bitbake/lib/bb/asyncrpc/__init__.py +++ b/poky/bitbake/lib/bb/asyncrpc/__init__.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/poky/bitbake/lib/bb/asyncrpc/client.py b/poky/bitbake/lib/bb/asyncrpc/client.py index 34960197d1..fa042bbe87 100644 --- a/poky/bitbake/lib/bb/asyncrpc/client.py +++ b/poky/bitbake/lib/bb/asyncrpc/client.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # @@ -29,7 +31,17 @@ class AsyncClient(object): async def connect_unix(self, path): async def connect_sock(): - return await asyncio.open_unix_connection(path) + # AF_UNIX has path length issues so chdir here to workaround + cwd = os.getcwd() + try: + os.chdir(os.path.dirname(path)) + # The socket must be opened synchronously so that CWD doesn't get + # changed out from underneath us so we pass as a sock into asyncio + sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, 0) + sock.connect(os.path.basename(path)) + finally: + os.chdir(cwd) + return await asyncio.open_unix_connection(sock=sock) self._connect_sock = connect_sock @@ -148,14 +160,8 @@ class Client(object): setattr(self, m, self._get_downcall_wrapper(downcall)) def connect_unix(self, path): - # AF_UNIX has path length issues so chdir here to workaround - cwd = os.getcwd() - try: - os.chdir(os.path.dirname(path)) - self.loop.run_until_complete(self.client.connect_unix(os.path.basename(path))) - self.loop.run_until_complete(self.client.connect()) - finally: - os.chdir(cwd) + self.loop.run_until_complete(self.client.connect_unix(path)) + self.loop.run_until_complete(self.client.connect()) @property def max_chunk(self): diff --git a/poky/bitbake/lib/bb/asyncrpc/serv.py b/poky/bitbake/lib/bb/asyncrpc/serv.py index b4cffff213..e14df18e71 100644 --- a/poky/bitbake/lib/bb/asyncrpc/serv.py +++ b/poky/bitbake/lib/bb/asyncrpc/serv.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/poky/bitbake/lib/bb/codeparser.py b/poky/bitbake/lib/bb/codeparser.py index 3b3c3b41ff..9d66d3ae41 100644 --- a/poky/bitbake/lib/bb/codeparser.py +++ b/poky/bitbake/lib/bb/codeparser.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/poky/bitbake/lib/bb/compress/_pipecompress.py b/poky/bitbake/lib/bb/compress/_pipecompress.py index 5de17a82e2..4a403d62cf 100644 --- a/poky/bitbake/lib/bb/compress/_pipecompress.py +++ b/poky/bitbake/lib/bb/compress/_pipecompress.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # # Helper library to implement streaming compression and decompression using an diff --git a/poky/bitbake/lib/bb/compress/lz4.py b/poky/bitbake/lib/bb/compress/lz4.py index 0f6bc51a5b..88b0989322 100644 --- a/poky/bitbake/lib/bb/compress/lz4.py +++ b/poky/bitbake/lib/bb/compress/lz4.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/poky/bitbake/lib/bb/compress/zstd.py b/poky/bitbake/lib/bb/compress/zstd.py index 50c42133fb..cdbbe9d60f 100644 --- a/poky/bitbake/lib/bb/compress/zstd.py +++ b/poky/bitbake/lib/bb/compress/zstd.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/poky/bitbake/lib/bb/cooker.py b/poky/bitbake/lib/bb/cooker.py index 6da9291f9c..2adf4d297d 100644 --- a/poky/bitbake/lib/bb/cooker.py +++ b/poky/bitbake/lib/bb/cooker.py @@ -13,7 +13,6 @@ import sys, os, glob, os.path, re, time import itertools import logging import multiprocessing -import sre_constants import threading from io import StringIO, UnsupportedOperation from contextlib import closing @@ -1907,7 +1906,7 @@ class CookerCollectFiles(object): try: re.compile(mask) bbmasks.append(mask) - except sre_constants.error: + except re.error: collectlog.critical("BBMASK contains an invalid regular expression, ignoring: %s" % mask) # Then validate the combined regular expressions. This should never @@ -1915,7 +1914,7 @@ class CookerCollectFiles(object): bbmask = "|".join(bbmasks) try: bbmask_compiled = re.compile(bbmask) - except sre_constants.error: + except re.error: collectlog.critical("BBMASK is not a valid regular expression, ignoring: %s" % bbmask) bbmask = None diff --git a/poky/bitbake/lib/bb/daemonize.py b/poky/bitbake/lib/bb/daemonize.py index 4957bfd4b8..7689404436 100644 --- a/poky/bitbake/lib/bb/daemonize.py +++ b/poky/bitbake/lib/bb/daemonize.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/poky/bitbake/lib/bb/event.py b/poky/bitbake/lib/bb/event.py index df020551e3..97668601a1 100644 --- a/poky/bitbake/lib/bb/event.py +++ b/poky/bitbake/lib/bb/event.py @@ -132,8 +132,14 @@ def print_ui_queue(): if not _uiready: from bb.msg import BBLogFormatter # Flush any existing buffered content - sys.stdout.flush() - sys.stderr.flush() + try: + sys.stdout.flush() + except: + pass + try: + sys.stderr.flush() + except: + pass stdout = logging.StreamHandler(sys.stdout) stderr = logging.StreamHandler(sys.stderr) formatter = BBLogFormatter("%(levelname)s: %(message)s") diff --git a/poky/bitbake/lib/bb/exceptions.py b/poky/bitbake/lib/bb/exceptions.py index ecbad59970..801db9c82f 100644 --- a/poky/bitbake/lib/bb/exceptions.py +++ b/poky/bitbake/lib/bb/exceptions.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/poky/bitbake/lib/bb/fetch2/__init__.py b/poky/bitbake/lib/bb/fetch2/__init__.py index ac557176d7..a31406263f 100644 --- a/poky/bitbake/lib/bb/fetch2/__init__.py +++ b/poky/bitbake/lib/bb/fetch2/__init__.py @@ -1097,6 +1097,8 @@ def try_mirror_url(fetch, origud, ud, ld, check = False): def ensure_symlink(target, link_name): if not os.path.exists(link_name): + dirname = os.path.dirname(link_name) + bb.utils.mkdirhier(dirname) if os.path.islink(link_name): # Broken symbolic link os.unlink(link_name) diff --git a/poky/bitbake/lib/bb/fetch2/gitsm.py b/poky/bitbake/lib/bb/fetch2/gitsm.py index c5c23d5260..25d5db0e5b 100644 --- a/poky/bitbake/lib/bb/fetch2/gitsm.py +++ b/poky/bitbake/lib/bb/fetch2/gitsm.py @@ -88,7 +88,7 @@ class GitSM(Git): subrevision[m] = module_hash.split()[2] # Convert relative to absolute uri based on parent uri - if uris[m].startswith('..'): + if uris[m].startswith('..') or uris[m].startswith('./'): newud = copy.copy(ud) newud.path = os.path.realpath(os.path.join(newud.path, uris[m])) uris[m] = Git._get_repo_url(self, newud) @@ -115,6 +115,9 @@ class GitSM(Git): # This has to be a file reference proto = "file" url = "gitsm://" + uris[module] + if "{}{}".format(ud.host, ud.path) in url: + raise bb.fetch2.FetchError("Submodule refers to the parent repository. This will cause deadlock situation in current version of Bitbake." \ + "Consider using git fetcher instead.") url += ';protocol=%s' % proto url += ";name=%s" % module diff --git a/poky/bitbake/lib/bb/fetch2/npm.py b/poky/bitbake/lib/bb/fetch2/npm.py index 8f7c10ac9b..8a179a339a 100644 --- a/poky/bitbake/lib/bb/fetch2/npm.py +++ b/poky/bitbake/lib/bb/fetch2/npm.py @@ -156,7 +156,7 @@ class Npm(FetchMethod): raise ParameterError("Invalid 'version' parameter", ud.url) # Extract the 'registry' part of the url - ud.registry = re.sub(r"^npm://", "http://", ud.url.split(";")[0]) + ud.registry = re.sub(r"^npm://", "https://", ud.url.split(";")[0]) # Using the 'downloadfilename' parameter as local filename # or the npm package name. diff --git a/poky/bitbake/lib/bb/fetch2/osc.py b/poky/bitbake/lib/bb/fetch2/osc.py index eb0f82c8e6..bf4c2f0511 100644 --- a/poky/bitbake/lib/bb/fetch2/osc.py +++ b/poky/bitbake/lib/bb/fetch2/osc.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # """ diff --git a/poky/bitbake/lib/bb/parse/parse_py/BBHandler.py b/poky/bitbake/lib/bb/parse/parse_py/BBHandler.py index ee9bd760ce..68415735fd 100644 --- a/poky/bitbake/lib/bb/parse/parse_py/BBHandler.py +++ b/poky/bitbake/lib/bb/parse/parse_py/BBHandler.py @@ -178,10 +178,10 @@ def feeder(lineno, s, fn, root, statements, eof=False): if s and s[0] == '#': if len(__residue__) != 0 and __residue__[0][0] != "#": - bb.fatal("There is a comment on line %s of file %s (%s) which is in the middle of a multiline expression.\nBitbake used to ignore these but no longer does so, please fix your metadata as errors are likely as a result of this change." % (lineno, fn, s)) + bb.fatal("There is a comment on line %s of file %s:\n'''\n%s\n'''\nwhich is in the middle of a multiline expression. This syntax is invalid, please correct it." % (lineno, fn, s)) if len(__residue__) != 0 and __residue__[0][0] == "#" and (not s or s[0] != "#"): - bb.fatal("There is a confusing multiline, partially commented expression on line %s of file %s (%s).\nPlease clarify whether this is all a comment or should be parsed." % (lineno, fn, s)) + bb.fatal("There is a confusing multiline partially commented expression on line %s of file %s:\n%s\nPlease clarify whether this is all a comment or should be parsed." % (lineno - len(__residue__), fn, "\n".join(__residue__))) if s and s[-1] == '\\': __residue__.append(s[:-1]) diff --git a/poky/bitbake/lib/bb/parse/parse_py/ConfHandler.py b/poky/bitbake/lib/bb/parse/parse_py/ConfHandler.py index b895d5b5ef..451e68dd66 100644 --- a/poky/bitbake/lib/bb/parse/parse_py/ConfHandler.py +++ b/poky/bitbake/lib/bb/parse/parse_py/ConfHandler.py @@ -125,16 +125,21 @@ def handle(fn, data, include): s = f.readline() if not s: break + origlineno = lineno + origline = s w = s.strip() # skip empty lines if not w: continue s = s.rstrip() while s[-1] == '\\': - s2 = f.readline().rstrip() + line = f.readline() + origline += line + s2 = line.rstrip() lineno = lineno + 1 if (not s2 or s2 and s2[0] != "#") and s[0] == "#" : - bb.fatal("There is a confusing multiline, partially commented expression on line %s of file %s (%s).\nPlease clarify whether this is all a comment or should be parsed." % (lineno, fn, s)) + bb.fatal("There is a confusing multiline, partially commented expression starting on line %s of file %s:\n%s\nPlease clarify whether this is all a comment or should be parsed." % (origlineno, fn, origline)) + s = s[:-1] + s2 # skip comments if s[0] == '#': @@ -147,8 +152,6 @@ def handle(fn, data, include): if oldfile: data.setVar('FILE', oldfile) - f.close() - for f in confFilters: f(fn, data) diff --git a/poky/bitbake/lib/bb/process.py b/poky/bitbake/lib/bb/process.py index be2c15a188..4c7b6d39df 100644 --- a/poky/bitbake/lib/bb/process.py +++ b/poky/bitbake/lib/bb/process.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/poky/bitbake/lib/bb/runqueue.py b/poky/bitbake/lib/bb/runqueue.py index f34f1568e2..48e25401ba 100644 --- a/poky/bitbake/lib/bb/runqueue.py +++ b/poky/bitbake/lib/bb/runqueue.py @@ -24,6 +24,7 @@ import pickle from multiprocessing import Process import shlex import pprint +import time bblogger = logging.getLogger("BitBake") logger = logging.getLogger("BitBake.RunQueue") @@ -159,6 +160,55 @@ class RunQueueScheduler(object): self.buildable.append(tid) self.rev_prio_map = None + self.is_pressure_usable() + + def is_pressure_usable(self): + """ + If monitoring pressure, return True if pressure files can be open and read. For example + openSUSE /proc/pressure/* files have readable file permissions but when read the error EOPNOTSUPP (Operation not supported) + is returned. + """ + if self.rq.max_cpu_pressure or self.rq.max_io_pressure or self.rq.max_memory_pressure: + try: + with open("/proc/pressure/cpu") as cpu_pressure_fds, \ + open("/proc/pressure/io") as io_pressure_fds, \ + open("/proc/pressure/memory") as memory_pressure_fds: + + self.prev_cpu_pressure = cpu_pressure_fds.readline().split()[4].split("=")[1] + self.prev_io_pressure = io_pressure_fds.readline().split()[4].split("=")[1] + self.prev_memory_pressure = memory_pressure_fds.readline().split()[4].split("=")[1] + self.prev_pressure_time = time.time() + self.check_pressure = True + except: + bb.note("The /proc/pressure files can't be read. Continuing build without monitoring pressure") + self.check_pressure = False + else: + self.check_pressure = False + + def exceeds_max_pressure(self): + """ + Monitor the difference in total pressure at least once per second, if + BB_PRESSURE_MAX_{CPU|IO|MEMORY} are set, return True if above threshold. + """ + if self.check_pressure: + with open("/proc/pressure/cpu") as cpu_pressure_fds, \ + open("/proc/pressure/io") as io_pressure_fds, \ + open("/proc/pressure/memory") as memory_pressure_fds: + # extract "total" from /proc/pressure/{cpu|io} + curr_cpu_pressure = cpu_pressure_fds.readline().split()[4].split("=")[1] + curr_io_pressure = io_pressure_fds.readline().split()[4].split("=")[1] + curr_memory_pressure = memory_pressure_fds.readline().split()[4].split("=")[1] + exceeds_cpu_pressure = self.rq.max_cpu_pressure and (float(curr_cpu_pressure) - float(self.prev_cpu_pressure)) > self.rq.max_cpu_pressure + exceeds_io_pressure = self.rq.max_io_pressure and (float(curr_io_pressure) - float(self.prev_io_pressure)) > self.rq.max_io_pressure + exceeds_memory_pressure = self.rq.max_memory_pressure and (float(curr_memory_pressure) - float(self.prev_memory_pressure)) > self.rq.max_memory_pressure + now = time.time() + if now - self.prev_pressure_time > 1.0: + self.prev_cpu_pressure = curr_cpu_pressure + self.prev_io_pressure = curr_io_pressure + self.prev_memory_pressure = curr_memory_pressure + self.prev_pressure_time = now + return (exceeds_cpu_pressure or exceeds_io_pressure or exceeds_memory_pressure) + return False def next_buildable_task(self): """ @@ -172,6 +222,12 @@ class RunQueueScheduler(object): if not buildable: return None + # Bitbake requires that at least one task be active. Only check for pressure if + # this is the case, otherwise the pressure limitation could result in no tasks + # being active and no new tasks started thereby, at times, breaking the scheduler. + if self.rq.stats.active and self.exceeds_max_pressure(): + return None + # Filter out tasks that have a max number of threads that have been exceeded skip_buildable = {} for running in self.rq.runq_running.difference(self.rq.runq_complete): @@ -1699,6 +1755,9 @@ class RunQueueExecute: self.number_tasks = int(self.cfgData.getVar("BB_NUMBER_THREADS") or 1) self.scheduler = self.cfgData.getVar("BB_SCHEDULER") or "speed" + self.max_cpu_pressure = self.cfgData.getVar("BB_PRESSURE_MAX_CPU") + self.max_io_pressure = self.cfgData.getVar("BB_PRESSURE_MAX_IO") + self.max_memory_pressure = self.cfgData.getVar("BB_PRESSURE_MAX_MEMORY") self.sq_buildable = set() self.sq_running = set() @@ -1733,6 +1792,29 @@ class RunQueueExecute: if self.number_tasks <= 0: bb.fatal("Invalid BB_NUMBER_THREADS %s" % self.number_tasks) + lower_limit = 1.0 + upper_limit = 1000000.0 + if self.max_cpu_pressure: + self.max_cpu_pressure = float(self.max_cpu_pressure) + if self.max_cpu_pressure < lower_limit: + bb.fatal("Invalid BB_PRESSURE_MAX_CPU %s, minimum value is %s." % (self.max_cpu_pressure, lower_limit)) + if self.max_cpu_pressure > upper_limit: + bb.warn("Your build will be largely unregulated since BB_PRESSURE_MAX_CPU is set to %s. It is very unlikely that such high pressure will be experienced." % (self.max_cpu_pressure)) + + if self.max_io_pressure: + self.max_io_pressure = float(self.max_io_pressure) + if self.max_io_pressure < lower_limit: + bb.fatal("Invalid BB_PRESSURE_MAX_IO %s, minimum value is %s." % (self.max_io_pressure, lower_limit)) + if self.max_io_pressure > upper_limit: + bb.warn("Your build will be largely unregulated since BB_PRESSURE_MAX_IO is set to %s. It is very unlikely that such high pressure will be experienced." % (self.max_io_pressure)) + + if self.max_memory_pressure: + self.max_memory_pressure = float(self.max_memory_pressure) + if self.max_memory_pressure < lower_limit: + bb.fatal("Invalid BB_PRESSURE_MAX_MEMORY %s, minimum value is %s." % (self.max_memory_pressure, lower_limit)) + if self.max_memory_pressure > upper_limit: + bb.warn("Your build will be largely unregulated since BB_PRESSURE_MAX_MEMORY is set to %s. It is very unlikely that such high pressure will be experienced." % (self.max_io_pressure)) + # List of setscene tasks which we've covered self.scenequeue_covered = set() # List of tasks which are covered (including setscene ones) @@ -2172,10 +2254,9 @@ class RunQueueExecute: # No more tasks can be run. If we have deferred setscene tasks we should run them. if self.sq_deferred: - tid = self.sq_deferred.pop(list(self.sq_deferred.keys())[0]) - logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s" % tid) - if tid not in self.runq_complete: - self.sq_task_failoutright(tid) + deferred_tid = list(self.sq_deferred.keys())[0] + blocking_tid = self.sq_deferred.pop(deferred_tid) + logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s blocked by %s" % (deferred_tid, blocking_tid)) return True if self.failed_tids: @@ -2299,6 +2380,9 @@ class RunQueueExecute: self.rqdata.runtaskentries[hashtid].unihash = unihash bb.parse.siggen.set_unihash(hashtid, unihash) toprocess.add(hashtid) + if torehash: + # Need to save after set_unihash above + bb.parse.siggen.save_unitaskhashes() # Work out all tasks which depend upon these total = set() @@ -2438,11 +2522,14 @@ class RunQueueExecute: if update_tasks: self.sqdone = False - for tid in [t[0] for t in update_tasks]: - h = pending_hash_index(tid, self.rqdata) - if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]: - self.sq_deferred[tid] = self.sqdata.hashes[h] - bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h])) + for mc in sorted(self.sqdata.multiconfigs): + for tid in sorted([t[0] for t in update_tasks]): + if mc_from_tid(tid) != mc: + continue + h = pending_hash_index(tid, self.rqdata) + if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]: + self.sq_deferred[tid] = self.sqdata.hashes[h] + bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h])) update_scenequeue_data([t[0] for t in update_tasks], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False) for (tid, harddepfail, origvalid) in update_tasks: diff --git a/poky/bitbake/lib/bb/siggen.py b/poky/bitbake/lib/bb/siggen.py index 9fa568f614..9a20fc8e5f 100644 --- a/poky/bitbake/lib/bb/siggen.py +++ b/poky/bitbake/lib/bb/siggen.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # @@ -419,7 +421,7 @@ class SignatureGeneratorBasic(SignatureGenerator): bb.error("Taskhash mismatch %s versus %s for %s" % (computed_taskhash, self.taskhash[tid], tid)) sigfile = sigfile.replace(self.taskhash[tid], computed_taskhash) - fd, tmpfile = tempfile.mkstemp(dir=os.path.dirname(sigfile), prefix="sigtask.") + fd, tmpfile = bb.utils.mkstemp(dir=os.path.dirname(sigfile), prefix="sigtask.") try: with bb.compress.zstd.open(fd, "wt", encoding="utf-8", num_threads=1) as f: json.dump(data, f, sort_keys=True, separators=(",", ":"), cls=SetEncoder) diff --git a/poky/bitbake/lib/bb/tests/compression.py b/poky/bitbake/lib/bb/tests/compression.py index d3ddf67f1c..95af3f96d7 100644 --- a/poky/bitbake/lib/bb/tests/compression.py +++ b/poky/bitbake/lib/bb/tests/compression.py @@ -1,4 +1,6 @@ # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/poky/bitbake/lib/bb/tests/cooker.py b/poky/bitbake/lib/bb/tests/cooker.py index c82d4b7b81..9e524ae345 100644 --- a/poky/bitbake/lib/bb/tests/cooker.py +++ b/poky/bitbake/lib/bb/tests/cooker.py @@ -1,6 +1,8 @@ # # BitBake Tests for cooker.py # +# Copyright BitBake Contributors +# # SPDX-License-Identifier: GPL-2.0-only # diff --git a/poky/bitbake/lib/bb/tests/parse.py b/poky/bitbake/lib/bb/tests/parse.py index 2898f9bb14..1a3b74934d 100644 --- a/poky/bitbake/lib/bb/tests/parse.py +++ b/poky/bitbake/lib/bb/tests/parse.py @@ -194,3 +194,26 @@ deltask ${EMPTYVAR} self.assertTrue('addtask ignored: " do_patch"' in stdout) #self.assertTrue('dependent task do_foo for do_patch does not exist' in stdout) + broken_multiline_comment = """ +# First line of comment \\ +# Second line of comment \\ + +""" + def test_parse_broken_multiline_comment(self): + f = self.parsehelper(self.broken_multiline_comment) + with self.assertRaises(bb.BBHandledException): + d = bb.parse.handle(f.name, self.d)[''] + + + comment_in_var = """ +VAR = " \\ + SOMEVAL \\ +# some comment \\ + SOMEOTHERVAL \\ +" +""" + def test_parse_comment_in_var(self): + f = self.parsehelper(self.comment_in_var) + with self.assertRaises(bb.BBHandledException): + d = bb.parse.handle(f.name, self.d)[''] + diff --git a/poky/bitbake/lib/bb/utils.py b/poky/bitbake/lib/bb/utils.py index d11da978d7..92d44c5260 100644 --- a/poky/bitbake/lib/bb/utils.py +++ b/poky/bitbake/lib/bb/utils.py @@ -28,6 +28,8 @@ import signal import collections import copy import ctypes +import random +import tempfile from subprocess import getstatusoutput from contextlib import contextmanager from ctypes import cdll @@ -429,12 +431,14 @@ def better_eval(source, locals, extraglobals = None): return eval(source, ctx, locals) @contextmanager -def fileslocked(files): +def fileslocked(files, *args, **kwargs): """Context manager for locking and unlocking file locks.""" locks = [] if files: for lockfile in files: - locks.append(bb.utils.lockfile(lockfile)) + l = bb.utils.lockfile(lockfile, *args, **kwargs) + if l is not None: + locks.append(l) try: yield @@ -692,8 +696,8 @@ def remove(path, recurse=False, ionice=False): return if recurse: for name in glob.glob(path): - if _check_unsafe_delete_path(path): - raise Exception('bb.utils.remove: called with dangerous path "%s" and recurse=True, refusing to delete!' % path) + if _check_unsafe_delete_path(name): + raise Exception('bb.utils.remove: called with dangerous path "%s" and recurse=True, refusing to delete!' % name) # shutil.rmtree(name) would be ideal but its too slow cmd = [] if ionice: @@ -751,7 +755,7 @@ def movefile(src, dest, newmtime = None, sstat = None): if not sstat: sstat = os.lstat(src) except Exception as e: - print("movefile: Stating source file failed...", e) + logger.warning("movefile: Stating source file failed...", e) return None destexists = 1 @@ -779,7 +783,7 @@ def movefile(src, dest, newmtime = None, sstat = None): os.unlink(src) return os.lstat(dest) except Exception as e: - print("movefile: failed to properly create symlink:", dest, "->", target, e) + logger.warning("movefile: failed to properly create symlink:", dest, "->", target, e) return None renamefailed = 1 @@ -796,7 +800,7 @@ def movefile(src, dest, newmtime = None, sstat = None): except Exception as e: if e.errno != errno.EXDEV: # Some random error. - print("movefile: Failed to move", src, "to", dest, e) + logger.warning("movefile: Failed to move", src, "to", dest, e) return None # Invalid cross-device-link 'bind' mounted or actually Cross-Device @@ -808,13 +812,13 @@ def movefile(src, dest, newmtime = None, sstat = None): bb.utils.rename(destpath + "#new", destpath) didcopy = 1 except Exception as e: - print('movefile: copy', src, '->', dest, 'failed.', e) + logger.warning('movefile: copy', src, '->', dest, 'failed.', e) return None else: #we don't yet handle special, so we need to fall back to /bin/mv a = getstatusoutput("/bin/mv -f " + "'" + src + "' '" + dest + "'") if a[0] != 0: - print("movefile: Failed to move special file:" + src + "' to '" + dest + "'", a) + logger.warning("movefile: Failed to move special file:" + src + "' to '" + dest + "'", a) return None # failure try: if didcopy: @@ -822,7 +826,7 @@ def movefile(src, dest, newmtime = None, sstat = None): os.chmod(destpath, stat.S_IMODE(sstat[stat.ST_MODE])) # Sticky is reset on chown os.unlink(src) except Exception as e: - print("movefile: Failed to chown/chmod/unlink", dest, e) + logger.warning("movefile: Failed to chown/chmod/unlink", dest, e) return None if newmtime: @@ -1754,3 +1758,22 @@ def is_local_uid(uid=''): if str(uid) == line_split[2]: return True return False + +def mkstemp(suffix=None, prefix=None, dir=None, text=False): + """ + Generates a unique filename, independent of time. + + mkstemp() in glibc (at least) generates unique file names based on the + current system time. When combined with highly parallel builds, and + operating over NFS (e.g. shared sstate/downloads) this can result in + conflicts and race conditions. + + This function adds additional entropy to the file name so that a collision + is independent of time and thus extremely unlikely. + """ + entropy = "".join(random.choices("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890", k=20)) + if prefix: + prefix = prefix + entropy + else: + prefix = tempfile.gettempprefix() + entropy + return tempfile.mkstemp(suffix=suffix, prefix=prefix, dir=dir, text=text) |