diff options
author | Patrick Williams <patrick@stwcx.xyz> | 2024-03-01 17:54:02 +0300 |
---|---|---|
committer | Patrick Williams <patrick@stwcx.xyz> | 2024-03-01 19:12:33 +0300 |
commit | 396535664c5645a0b32bd0e06dbab3f1e2849b64 (patch) | |
tree | dcf5c28dc344e5906f266d9644e0cfab226134a0 /poky/bitbake/lib | |
parent | 73bd93f1d0a338767f36fd1acb54c52ad057db39 (diff) | |
download | openbmc-396535664c5645a0b32bd0e06dbab3f1e2849b64.tar.xz |
subtree updates
meta-arm: 025f76a14f..aba9250494:
Anusmita Dutta Mazumder (2):
arm-bsp/linux-yocto: Remove EOL Linux yocto kernel 6.1
arm-bsp/n1sdp: update to linux yocto kernel 6.6
Bence Balogh (1):
arm-bsp/trusted-firmware-m: disable libmetal doc generation
Drew Reed (5):
meta-arm: Support firmware building under a multiconfig
bsp,ci: Build Corstone-1000 firmware under multiconfig
bsp: Restore the ability to build firmware only
ci: Add back testing of firmware only builds
ci: Ensure tests are in the Corstone-1000 flash image
meta-raspberrypi: dbf1113a82..95a9103f91:
Khem Raj (1):
python3-sense-hat: Drop PYTHON_PN
Martin Jansa (2):
sdcard_image-rpi.bbclass: include ${IMAGE_NAME_SUFFIX} directly in both ${IMAGE_NAME} and ${IMAGE_LINK_NAME}
sdimage-raspberrypi.wks: increase /boot partition minimal size from 20 to 100
meta-openembedded: 528f273006..9f0e513211:
Andreas Mützel (1):
python3-pynacl: allow -native build
Chen Qi (1):
unixodbc: fix odbc.pc file generation
Daniel Ammann (1):
sdmon: add new package
Derek Straka (9):
python3-trustme: add runtime dependency for tests and re-add to ptest
python3-gunicorn: re-enable working ptests for the package
python-inotify: re-enable working ptests for the package
python3-license-expression: re-enable passing ptests for the package
python3-jdcal: re-add functional ptests
python3-msgpack: re-add functional ptests
python3-parse: re-add functional ptests
python3-typeguard: update ptest dependencies and re-enable functional tests
python3-service-identity: add missing ptest dependencies and re-enable functional tests
Jan Vermaete (1):
netdata: version bump 1.43.2 -> 1.44.3
Joerg Hofrichter (1):
python3-gevent: adding missing dependency to python3-zopeevent
Khawaja Shaheryar (2):
libdaq: add recipe
snort: add snort3 initial recipe
Khem Raj (25):
python3-pocketsphinx: Upgrade to 5.0.3
snort: Do not use llvm libunwind
snort3: Fix contains reference to TMPDIR [buildpaths] warnings
libcamera: Replace VLAs with alloca
dav1d: Inherit missing pkgconfig
webkitgtk3: Fix build on 32bit x86
ptest-packagelists-meta-oe: Remove oprofile for rv32/rv64
python3-jsmin: Fix ptests to run with python 3.12+
python3-ordered-set: Use automake formatter for ptest output
fuse3: Add missing runtime deps for ptests
python3-looseversion: Add recipe
sshfs-fuse: Fix ptest builds with python 3.12
meta-filesystems: Add meta-filesystems-image-ptest
meta-multimedia-image-ptest: Add images to enable BBCLASSEXTEND parallel execution
meta-networking-image-ptest: Add images to enable BBCLASSEXTEND parallel execution
python3-scapy: Add missing rdeps for ptests
ptest-packagelists-meta-oe.inc: Remove oprofile from PTESTS_PROBLEMS_META_OE
ptest-packagelists-meta-networking: firewalld hangs therefore disabled
ptest-packagelists-meta-perl.inc: Move couple of test to PTESTS_FAST_META_PERL
openhpi: Fix ptest run time failures
squid: Add missing bash dependency for ptest package
meta-networking: Express dependency on meta-python
ostree: Remove strace from ptest rdeps
python3-pydantic-core,python3-pydantic: Update to 2.16.3 and 2.6.3 respectively
python3-pydantic-core: Fix build for arches without 64bit atomics
Lei Maohui (1):
Fix install error when enable multilib.
Markus Volk (7):
iwd: update 2.13 -> 2.14
libgedit-gtksourceview: update 299.0.5 -> 299.1.0
gedit: update 46.1 -> 46.2
mutter: update 45.3 -> 45.4
gnome-shell: update 45.3 -> 45.4
gnome-control-center: update 45.2 -> 45.3
dav1d: update 1.3.0 -> 1.4.0
Martin Jansa (5):
python3-httpx: respect libdir in packaging
snort3: drop SRCPV from PV
snort3: fix snort.pc
gattlib: use python3native and depend on python3-packaging-native
networkmanager-fortisslvpn: use python3native and depend on python3-packaging-native
Mingli Yu (1):
mariadb: Upgrade to 10.11.7
Niko Mauno (2):
python3-pybind11: Migrate to python_setuptools_build_meta
python3-pybind11: Restore strip prevention patch
Oleh Matiusha (1):
yasm: improve reproducibility
Peter Marko (1):
dnsmasq: Upgrade 2.89 -> 2.90
Romain Naour (1):
wavemon: add recipe for version 0.9.5
Sascha Hauer (1):
signing.bbclass: fix wrong function name
Tim Orling (16):
python_mesonpy.bbclass: move to oe-core
python3-meson-python: move to oe-core
python3-pyproject-metadata: move to oe-core
meta-python: drop ${PYTHON_PN}
meta-oe: drop ${PYTHON_PN}
meta-filesystems: drop ${PYTHON_PN}
meta-networking: drop ${PYTHON_PN}
meta-gnome: drop ${PYTHON_PN}
python3-pytest-lazy-fixtures: add 1.0.5
python3-prettytable: upgrade 3.9.0 => 3.10.0; fix ptests
python3-pytest-lazy-fixture: drop recipe
meta-oe-image-ptest: add PTESTS_PROBLEMS_META_OE
meta-perl-image-ptest: add PTESTS_PROBLEMS_META_PERL
meta-python-image-ptest: add PTESTS_PROBLEMS_META_PYTHON
libencode-perl: drop recipe
libencode-locale-perl: drop recipe
Wang Mingyu (49):
babl: upgrade 0.1.106 -> 0.1.108
btop: upgrade 1.3.0 -> 1.3.2
gegl: upgrade 0.4.46 -> 0.4.48
gjs: upgrade 1.78.3 -> 1.78.4
gnome-bluetooth: upgrade 42.7 -> 42.8
gnome-keyring: upgrade 42.1 -> 46.1
isomd5sum: upgrade 1.2.3 -> 1.2.4
libei: upgrade 1.2.0 -> 1.2.1
libmanette: upgrade 0.2.6 -> 0.2.7
libmime-types-perl: upgrade 2.24 -> 2.26
logwatch: upgrade 7.9 -> 7.10
mpich: upgrade 4.1.2 -> 4.2.0
ostree: upgrade 2024.1 -> 2024.3
python3-aiohue: upgrade 4.7.0 -> 4.7.1
python3-awesomeversion: upgrade 23.11.0 -> 24.2.0
python3-bidict: upgrade 0.22.1 -> 0.23.0
python3-cantools: upgrade 39.4.3 -> 39.4.4
python3-cmake: upgrade 3.28.1 -> 3.28.3
python3-django: upgrade 5.0.1 -> 5.0.2
python3-dnspython: upgrade 2.5.0 -> 2.6.0
python3-elementpath: upgrade 4.2.0 -> 4.3.0
python3-engineio: upgrade 4.8.2 -> 4.9.0
python3-gevent: upgrade 23.9.1 -> 24.2.1
unbound: upgrade 1.19.0 -> 1.19.1
wireshark: upgrade 4.2.2 -> 4.2.3
protobuf: upgrade 4.25.2 -> 4.25.3
webkitgtk3: upgrade 2.42.4 -> 2.42.5
python3-tqdm: upgrade 4.66.1 -> 4.66.2
python3-google-api-python-client: upgrade 2.116.0 -> 2.118.0
python3-httpcore: upgrade 1.0.2 -> 1.0.3
python3-jsbeautifier: upgrade 1.14.11 -> 1.15.1
python3-langtable: upgrade 0.0.64 -> 0.0.65
python3-polyline: upgrade 2.0.1 -> 2.0.2
python3-protobuf: upgrade 4.25.2 -> 4.25.3
python3-pymisp: upgrade 2.4.184 -> 2.4.185
python3-pymodbus: upgrade 3.6.3 -> 3.6.4
python3-pytest-asyncio: upgrade 0.23.4 -> 0.23.5
python3-tox: upgrade 4.12.1 -> 4.13.0
python3-twine: upgrade 4.0.2 -> 5.0.0
python3-watchdog: upgrade 3.0.0 -> 4.0.0
python3-zopeinterface: upgrade 6.1 -> 6.2
remmina: upgrade 1.4.33 -> 1.4.34
sip: upgrade 6.8.2 -> 6.8.3
python3-google-auth: upgrade 2.27.0 -> 2.28.0
python3-gspread: upgrade 6.0.1 -> 6.0.2
python3-socketio: upgrade 5.11.0 -> 5.11.1
python3-sentry-sdk: upgrade 1.40.0 -> 1.40.4
python3-pydantic-core: upgrade 2.14.6 -> 2.16.1
python3-pydantic: upgrade 2.5.3 -> 2.6.0
William Lyu (1):
e2tools: Add ptest
Yi Zhao (1):
audit: upgrade 3.1.2 -> 4.0
Yoann Congal (2):
influxdb: Fix /etc files owner
influxdb: Add missing group to static id
chenheyun (1):
dropwatch: Use header files from sysroot instead of build host
poky: fc8e5d7c13..25d60ac6f6:
Adrian Freihofer (5):
devtool: ide-sdk python 3.12 escaping
sdk-manual: extensible.rst: cover devtool ide-sdk
devtool: ide-sdk launch.json per recipe only
devtool: ide-sdk prefer sources from workspace
oe-selftest devtool: ide-sdk tests
Alexander Kanavin (1):
dbus: disable assertions and enable only modular tests
Alexis Lothoré (7):
testimage: log exception when failing to retrieve artifacts
lib/oeqa: share get_json_result_dir helper
testimage: create a list of failed test post actions
oeqa/utils/postactions: isolate directory creation in dedicated action
oeqa/utils/postactions: add target disk usage stat as post action
oeqa/utils/postactions: testimage: add host disk usage stat as post action
oeqa/lib/utils/postactions: fix host disk usage stats retrieval
Bruce Ashfield (8):
linux-yocto/6.6: update to v6.6.17
linux-yocto/6.6: update CVE exclusions
linux-yocto/6.6: enable squashfs for selftests
linux-yocto/6.6: config: x86 tidy & consolidation
kern-tools: depend on git-replacement-native
linux-yocto/6.6: genericarm64 configuration/definition
linux-yocto/6.6: update to v6.6.18
linux-yocto/6.6: update CVE exclusions
Christoph Vogtländer (1):
overlayfs: add missing vardeps
Claus Stovgaard (1):
wpa-supplicant: Fix CVE-2023-52160
Eilís 'pidge' Ní Fhlannagáin (2):
creategroup*: Remove coreutils-native as a DEPENDS
selftest-users: Convoluted selftest for USERADD_DEPENDS
Emil Kronborg (1):
bluez5: remove configuration files from install task
Enguerrand de Ribaucourt (4):
devtool: ide: define compilerPath for meson projects
Revert "meson: use absolute cross-compiler paths"
bitbake: bitbake: progressbar: accept value over initial maxval
devtool: ide-sdk source mapping for vscode
Enrico Jörns (1):
wic: 'empty' plugin: fix typo in comment
Joe Slater (1):
qemuboot: predictable network interface names
Jonathan GUILLOT (2):
lib/oe/package: fix LOCALE_PATHS scan to create locale packages
glibc-locale: add an explicit dedicated package for locale.alias file
Jose Quaresma (1):
go: update 1.20.13 -> 1.20.14
Joshua Watt (1):
bitbake: asyncrpc: Add support for server headers
Khem Raj (6):
ncurses: Always pass -D_GNU_SOURCE
linux-yocto: Remove unused patch
ref-manual: variables: remove PYTHON_PN
python3-bcrypt: Fix build break on arches without 64 bit atomics
python3-maturin: Recognise riscv32 architecture
llvm: Update to 18.1.0 RC4
Lee Chee Yang (1):
migration-guide: add release notes for 4.3.3
Lei Maohui (1):
rpm: Fix the following error when run nativesdk-rpm in nativesdk environment.
Martin Jansa (1):
glib-2.0: backport a switch from distutils to packaging in codegen
Michael Halstead (1):
yocto-uninative: Update to 4.4 for glibc 2.39
Michael Opdenacker (5):
ref-manual: system-requirements: update packages to build docs
ref-manual: release-process: grammar fix
manuals: suppress excess use of "following" word
dev-manual: packages: clarify shared PR service constraint
dev-manual: packages: need enough free space
Munehisa Kamata (1):
kernel.bbclass: Set pkg-config variables for building modules
Nick Owens (1):
python3: dont disable readline module for editline
Philip Lorenz (1):
bitbake: fetch2: Ensure that git LFS objects are available
Piotr Łobacz (1):
useradd.bbclass: Fix order of postinst-useradd-*
Richard Purdie (6):
numactl: Upgrade 2.0.17 -> 2.0.18
lttng-ust: Upgrade 2.13.6 -> 2.13.7
oeqa/selftest/rust: Simplify the rust testsuite output gathering/processing
recipetool: Fix errors with meta-poky bbappend
bitbake: runqueue: Add support for BB_LOADFACTOR_MAX
mirrors: Switch llvm to use shallow cloning
Ross Burton (4):
base-files: add usage warning to motd
libexif: remove unused version_underscore
gstreamer1.0: skip a test that is known to be flaky
linux-firmware: split out more firmware pieces
Simone Weiß (6):
patchtest: provide further guidance for failed testcases
patchtest: Skip test for CVE_CHECK_IGNORE for older branches
meta: Remove some not needed CVE_STATUS
meta: Update CVE_STATUS for incorrect cpes
cve-check: Log if CVE_STATUS set but not reported for component
dev-manual: Rephrase spdx creation
Soumya Sambu (1):
bind: Upgrade 9.18.21 -> 9.18.24
Tim Orling (3):
bitbake: layerindexlib: fix missing layer branch backtrace
python3-cryptography{-vectors}: upgrade to 42.0.5
python3-attrs: disable Hypothesis deadline
Tobias Hagelborn (1):
bitbake: hashserv: Re-enable connection pooling with psycopg 3 driver
Trevor Gamblin (1):
python3-git: upgrade 3.1.41 -> 3.1.42
Trevor Woerner (1):
wic: allow imager-specific filename extensions
Ulrich Ölmann (1):
bitbake: taskexp_ncurses: fix execution example in introductory comment
Wang Mingyu (44):
bash-completion: upgrade 2.11 -> 2.12.0
ccache: upgrade 4.9 -> 4.9.1
createrepo-c: upgrade 1.0.3 -> 1.0.4
ed: upgrade 1.20 -> 1.20.1
efivar: upgrade 38 -> 39
gcr: upgrade 4.1.0 -> 4.2.0
git: upgrade 2.43.0 -> 2.44.0
libffi: upgrade 3.4.5 -> 3.4.6
libgpg-error: upgrade 1.47 -> 1.48
libhandy: upgrade 1.8.2 -> 1.8.3
libksba: upgrade 1.6.5 -> 1.6.6
libmicrohttpd: upgrade 0.9.77 -> 1.0.1
libpng: upgrade 1.6.41 -> 1.6.42
libsecret: upgrade 0.21.2 -> 0.21.4
libunistring: upgrade 1.1 -> 1.2
liburi-perl: upgrade 5.25 -> 5.27
libxext: upgrade 1.3.5 -> 1.3.6
libxkbfile: upgrade 1.1.2 -> 1.1.3
libxvmc: upgrade 1.0.13 -> 1.0.14
lighttpd: upgrade 1.4.73 -> 1.4.74
makedepend: upgrade 1.0.8 -> 1.0.9
mpg123: upgrade 1.32.4 -> 1.32.5
ofono: upgrade 2.3 -> 2.4
pango: upgrade 1.51.0 -> 1.52.0
pciutils: upgrade 3.10.0 -> 3.11.1
pkgconf: upgrade 2.1.0 -> 2.1.1
python3-beartype: upgrade 0.17.0 -> 0.17.2
python3-certifi: upgrade 2023.11.17 -> 2024.2.2
python3-dbusmock: upgrade 0.30.2 -> 0.31.1
python3-hypothesis: upgrade 6.97.3 -> 6.98.12
python3-pip: upgrade 23.3.2 -> 24.0
python3-pycairo: upgrade 1.25.1 -> 1.26.0
python3-pytest: upgrade 8.0.0 -> 8.0.2
python3-pytz: upgrade 2023.4 -> 2024.1
python3-setuptools-rust: upgrade 1.8.1 -> 1.9.0
python3-trove-classifiers: upgrade 2024.1.8 -> 2024.2.23
python3-typing-extensions: upgrade 4.9.0 -> 4.10.0
python3: upgrade 3.12.1 -> 3.12.2
python3-urllib3: upgrade 2.1.0 -> 2.2.1
python3-yamllint: upgrade 1.33.0 -> 1.35.1
swig: upgrade 4.2.0 -> 4.2.1
xkbcomp: upgrade 1.4.6 -> 1.4.7
xkeyboard-config: upgrade 2.40 -> 2.41
xprop: upgrade 1.2.6 -> 1.2.7
Xiangyu Chen (2):
systemd-systemctl: fix dead loop when multi services enable each other
libc-locale: fix ASCII compatible warning cause build failure.
Xiaotian Wu (2):
loongarch64: change -march to loongarch64
openssl: Match target name for loongarch64
Yash Shinde (3):
rust: Upgrade 1.74.1 -> 1.75.0
rust: Revert PGO to it's default
rust: reproducibility issue fix with v1.75
Yoann Congal (1):
waf: Improve version parsing to avoid failing on warnings
Change-Id: I6dfb848feb4ec8f5aae56a9ccbff475f4eb1edc6
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Diffstat (limited to 'poky/bitbake/lib')
-rw-r--r-- | poky/bitbake/lib/bb/asyncrpc/client.py | 33 | ||||
-rw-r--r-- | poky/bitbake/lib/bb/asyncrpc/serv.py | 16 | ||||
-rw-r--r-- | poky/bitbake/lib/bb/fetch2/git.py | 45 | ||||
-rw-r--r-- | poky/bitbake/lib/bb/runqueue.py | 16 | ||||
-rw-r--r-- | poky/bitbake/lib/bb/tests/fetch.py | 51 | ||||
-rwxr-xr-x | poky/bitbake/lib/bb/ui/taskexp_ncurses.py | 2 | ||||
-rw-r--r-- | poky/bitbake/lib/hashserv/sqlalchemy.py | 6 | ||||
-rw-r--r-- | poky/bitbake/lib/layerindexlib/__init__.py | 15 | ||||
-rw-r--r-- | poky/bitbake/lib/progressbar/progressbar.py | 2 |
9 files changed, 170 insertions, 16 deletions
diff --git a/poky/bitbake/lib/bb/asyncrpc/client.py b/poky/bitbake/lib/bb/asyncrpc/client.py index a6228bb0ba..29a5ab76aa 100644 --- a/poky/bitbake/lib/bb/asyncrpc/client.py +++ b/poky/bitbake/lib/bb/asyncrpc/client.py @@ -17,13 +17,24 @@ from .exceptions import ConnectionClosedError, InvokeError class AsyncClient(object): - def __init__(self, proto_name, proto_version, logger, timeout=30): + def __init__( + self, + proto_name, + proto_version, + logger, + timeout=30, + server_headers=False, + headers={}, + ): self.socket = None self.max_chunk = DEFAULT_MAX_CHUNK self.proto_name = proto_name self.proto_version = proto_version self.logger = logger self.timeout = timeout + self.needs_server_headers = server_headers + self.server_headers = {} + self.headers = headers async def connect_tcp(self, address, port): async def connect_sock(): @@ -61,9 +72,29 @@ class AsyncClient(object): async def setup_connection(self): # Send headers await self.socket.send("%s %s" % (self.proto_name, self.proto_version)) + await self.socket.send( + "needs-headers: %s" % ("true" if self.needs_server_headers else "false") + ) + for k, v in self.headers.items(): + await self.socket.send("%s: %s" % (k, v)) + # End of headers await self.socket.send("") + self.server_headers = {} + if self.needs_server_headers: + while True: + line = await self.socket.recv() + if not line: + # End headers + break + tag, value = line.split(":", 1) + self.server_headers[tag.lower()] = value.strip() + + async def get_header(self, tag, default): + await self.connect() + return self.server_headers.get(tag, default) + async def connect(self): if self.socket is None: self.socket = await self._connect_sock() diff --git a/poky/bitbake/lib/bb/asyncrpc/serv.py b/poky/bitbake/lib/bb/asyncrpc/serv.py index f0be9a6cdb..a66117acad 100644 --- a/poky/bitbake/lib/bb/asyncrpc/serv.py +++ b/poky/bitbake/lib/bb/asyncrpc/serv.py @@ -39,10 +39,14 @@ class AsyncServerConnection(object): "address": socket.address, }, ) + self.client_headers = {} async def close(self): await self.socket.close() + async def handle_headers(self, headers): + return {} + async def process_requests(self): try: self.logger.info("Client %r connected" % (self.socket.address,)) @@ -64,12 +68,20 @@ class AsyncServerConnection(object): ) return - # Read headers. Currently, no headers are implemented, so look for - # an empty line to signal the end of the headers + # Read headers + self.client_headers = {} while True: header = await self.socket.recv() if not header: + # Empty line. End of headers break + tag, value = header.split(":", 1) + self.client_headers[tag.lower()] = value.strip() + + if self.client_headers.get("needs-headers", "false") == "true": + for k, v in (await self.handle_headers(self.client_headers)).items(): + await self.socket.send("%s: %s" % (k, v)) + await self.socket.send("") # Handle messages while True: diff --git a/poky/bitbake/lib/bb/fetch2/git.py b/poky/bitbake/lib/bb/fetch2/git.py index df33fb6aeb..b43ee0da39 100644 --- a/poky/bitbake/lib/bb/fetch2/git.py +++ b/poky/bitbake/lib/bb/fetch2/git.py @@ -325,7 +325,10 @@ class Git(FetchMethod): return ud.clonedir def need_update(self, ud, d): - return self.clonedir_need_update(ud, d) or self.shallow_tarball_need_update(ud) or self.tarball_need_update(ud) + return self.clonedir_need_update(ud, d) \ + or self.shallow_tarball_need_update(ud) \ + or self.tarball_need_update(ud) \ + or self.lfs_need_update(ud, d) def clonedir_need_update(self, ud, d): if not os.path.exists(ud.clonedir): @@ -337,6 +340,15 @@ class Git(FetchMethod): return True return False + def lfs_need_update(self, ud, d): + if self.clonedir_need_update(ud, d): + return True + + for name in ud.names: + if not self._lfs_objects_downloaded(ud, d, name, ud.clonedir): + return True + return False + def clonedir_need_shallow_revs(self, ud, d): for rev in ud.shallow_revs: try: @@ -467,7 +479,7 @@ class Git(FetchMethod): if missing_rev: raise bb.fetch2.FetchError("Unable to find revision %s even from upstream" % missing_rev) - if self._contains_lfs(ud, d, ud.clonedir) and self._need_lfs(ud): + if self.lfs_need_update(ud, d): # Unpack temporary working copy, use it to run 'git checkout' to force pre-fetching # of all LFS blobs needed at the srcrev. # @@ -710,6 +722,35 @@ class Git(FetchMethod): raise bb.fetch2.FetchError("The command '%s' gave output with more then 1 line unexpectedly, output: '%s'" % (cmd, output)) return output.split()[0] != "0" + def _lfs_objects_downloaded(self, ud, d, name, wd): + """ + Verifies whether the LFS objects for requested revisions have already been downloaded + """ + # Bail out early if this repository doesn't use LFS + if not self._need_lfs(ud) or not self._contains_lfs(ud, d, wd): + return True + + # The Git LFS specification specifies ([1]) the LFS folder layout so it should be safe to check for file + # existence. + # [1] https://github.com/git-lfs/git-lfs/blob/main/docs/spec.md#intercepting-git + cmd = "%s lfs ls-files -l %s" \ + % (ud.basecmd, ud.revisions[name]) + output = runfetchcmd(cmd, d, quiet=True, workdir=wd).rstrip() + # Do not do any further matching if no objects are managed by LFS + if not output: + return True + + # Match all lines beginning with the hexadecimal OID + oid_regex = re.compile("^(([a-fA-F0-9]{2})([a-fA-F0-9]{2})[A-Fa-f0-9]+)") + for line in output.split("\n"): + oid = re.search(oid_regex, line) + if not oid: + bb.warn("git lfs ls-files output '%s' did not match expected format." % line) + if not os.path.exists(os.path.join(wd, "lfs", "objects", oid.group(2), oid.group(3), oid.group(1))): + return False + + return True + def _need_lfs(self, ud): return ud.parm.get("lfs", "1") == "1" diff --git a/poky/bitbake/lib/bb/runqueue.py b/poky/bitbake/lib/bb/runqueue.py index e86ccd8c61..6987de3e29 100644 --- a/poky/bitbake/lib/bb/runqueue.py +++ b/poky/bitbake/lib/bb/runqueue.py @@ -220,6 +220,16 @@ class RunQueueScheduler(object): bb.note("Pressure status changed to CPU: %s, IO: %s, Mem: %s (CPU: %s/%s, IO: %s/%s, Mem: %s/%s) - using %s/%s bitbake threads" % (pressure_state + pressure_values + (len(self.rq.runq_running.difference(self.rq.runq_complete)), self.rq.number_tasks))) self.pressure_state = pressure_state return (exceeds_cpu_pressure or exceeds_io_pressure or exceeds_memory_pressure) + elif self.rq.max_loadfactor: + limit = False + loadfactor = float(os.getloadavg()[0]) / os.cpu_count() + # bb.warn("Comparing %s to %s" % (loadfactor, self.rq.max_loadfactor)) + if loadfactor > self.rq.max_loadfactor: + limit = True + if hasattr(self, "loadfactor_limit") and limit != self.loadfactor_limit: + bb.note("Load average limiting set to %s as load average: %s - using %s/%s bitbake threads" % (limit, loadfactor, len(self.rq.runq_running.difference(self.rq.runq_complete)), self.rq.number_tasks)) + self.loadfactor_limit = limit + return limit return False def next_buildable_task(self): @@ -1822,6 +1832,7 @@ class RunQueueExecute: self.max_cpu_pressure = self.cfgData.getVar("BB_PRESSURE_MAX_CPU") self.max_io_pressure = self.cfgData.getVar("BB_PRESSURE_MAX_IO") self.max_memory_pressure = self.cfgData.getVar("BB_PRESSURE_MAX_MEMORY") + self.max_loadfactor = self.cfgData.getVar("BB_LOADFACTOR_MAX") self.sq_buildable = set() self.sq_running = set() @@ -1875,6 +1886,11 @@ class RunQueueExecute: bb.fatal("Invalid BB_PRESSURE_MAX_MEMORY %s, minimum value is %s." % (self.max_memory_pressure, lower_limit)) if self.max_memory_pressure > upper_limit: bb.warn("Your build will be largely unregulated since BB_PRESSURE_MAX_MEMORY is set to %s. It is very unlikely that such high pressure will be experienced." % (self.max_io_pressure)) + + if self.max_loadfactor: + self.max_loadfactor = float(self.max_loadfactor) + if self.max_loadfactor <= 0: + bb.fatal("Invalid BB_LOADFACTOR_MAX %s, needs to be greater than zero." % (self.max_loadfactor)) # List of setscene tasks which we've covered self.scenequeue_covered = set() diff --git a/poky/bitbake/lib/bb/tests/fetch.py b/poky/bitbake/lib/bb/tests/fetch.py index 5ed5b5607f..e988e26c0a 100644 --- a/poky/bitbake/lib/bb/tests/fetch.py +++ b/poky/bitbake/lib/bb/tests/fetch.py @@ -6,6 +6,7 @@ # SPDX-License-Identifier: GPL-2.0-only # +import contextlib import unittest import hashlib import tempfile @@ -2261,10 +2262,14 @@ class GitLfsTest(FetcherTest): bb.utils.mkdirhier(self.srcdir) self.git_init(cwd=self.srcdir) - with open(os.path.join(self.srcdir, '.gitattributes'), 'wt') as attrs: - attrs.write('*.mp3 filter=lfs -text') - self.git(['add', '.gitattributes'], cwd=self.srcdir) - self.git(['commit', '-m', "attributes", '.gitattributes'], cwd=self.srcdir) + self.commit_file('.gitattributes', '*.mp3 filter=lfs -text') + + def commit_file(self, filename, content): + with open(os.path.join(self.srcdir, filename), "w") as f: + f.write(content) + self.git(["add", filename], cwd=self.srcdir) + self.git(["commit", "-m", "Change"], cwd=self.srcdir) + return self.git(["rev-parse", "HEAD"], cwd=self.srcdir).strip() def fetch(self, uri=None, download=True): uris = self.d.getVar('SRC_URI').split() @@ -2285,6 +2290,44 @@ class GitLfsTest(FetcherTest): return unpacked_lfs_file @skipIfNoGitLFS() + def test_fetch_lfs_on_srcrev_change(self): + """Test if fetch downloads missing LFS objects when a different revision within an existing repository is requested""" + self.git(["lfs", "install", "--local"], cwd=self.srcdir) + + @contextlib.contextmanager + def hide_upstream_repository(): + """Hide the upstream repository to make sure that git lfs cannot pull from it""" + temp_name = self.srcdir + ".bak" + os.rename(self.srcdir, temp_name) + try: + yield + finally: + os.rename(temp_name, self.srcdir) + + def fetch_and_verify(revision, filename, content): + self.d.setVar('SRCREV', revision) + fetcher, ud = self.fetch() + + with hide_upstream_repository(): + workdir = self.d.getVar('WORKDIR') + fetcher.unpack(workdir) + + with open(os.path.join(workdir, "git", filename)) as f: + self.assertEqual(f.read(), content) + + commit_1 = self.commit_file("a.mp3", "version 1") + commit_2 = self.commit_file("a.mp3", "version 2") + + self.d.setVar('SRC_URI', "git://%s;protocol=file;lfs=1;branch=master" % self.srcdir) + + # Seed the local download folder by fetching the latest commit and verifying that the LFS contents are + # available even when the upstream repository disappears. + fetch_and_verify(commit_2, "a.mp3", "version 2") + # Verify that even when an older revision is fetched, the needed LFS objects are fetched into the download + # folder. + fetch_and_verify(commit_1, "a.mp3", "version 1") + + @skipIfNoGitLFS() @skipIfNoNetwork() def test_real_git_lfs_repo_succeeds_without_lfs_param(self): self.d.setVar('SRC_URI', "git://gitlab.com/gitlab-examples/lfs.git;protocol=https;branch=master") diff --git a/poky/bitbake/lib/bb/ui/taskexp_ncurses.py b/poky/bitbake/lib/bb/ui/taskexp_ncurses.py index dd91d26bc3..ea94a4987f 100755 --- a/poky/bitbake/lib/bb/ui/taskexp_ncurses.py +++ b/poky/bitbake/lib/bb/ui/taskexp_ncurses.py @@ -12,7 +12,7 @@ # # Execution example: -# $ bitbake -g -u taskexp_ncurses.py acl zlib +# $ bitbake -g -u taskexp_ncurses zlib acl # # Self-test example (executes a script of GUI actions): # $ TASK_EXP_UNIT_TEST=1 bitbake -g -u taskexp_ncurses zlib acl diff --git a/poky/bitbake/lib/hashserv/sqlalchemy.py b/poky/bitbake/lib/hashserv/sqlalchemy.py index fc3ae3d339..f7b0226a7a 100644 --- a/poky/bitbake/lib/hashserv/sqlalchemy.py +++ b/poky/bitbake/lib/hashserv/sqlalchemy.py @@ -129,7 +129,11 @@ class DatabaseEngine(object): return inspect(conn).has_table(name) self.logger.info("Using database %s", self.url) - self.engine = create_async_engine(self.url, poolclass=NullPool) + if self.url.drivername == 'postgresql+psycopg': + # Psygopg 3 (psygopg) driver can handle async connection pooling + self.engine = create_async_engine(self.url, max_overflow=-1) + else: + self.engine = create_async_engine(self.url, poolclass=NullPool) async with self.engine.begin() as conn: # Create tables diff --git a/poky/bitbake/lib/layerindexlib/__init__.py b/poky/bitbake/lib/layerindexlib/__init__.py index ac03d89876..c3265ddaa1 100644 --- a/poky/bitbake/lib/layerindexlib/__init__.py +++ b/poky/bitbake/lib/layerindexlib/__init__.py @@ -178,9 +178,9 @@ class LayerIndex(): '''Load the layerindex. indexURI - An index to load. (Use multiple calls to load multiple indexes) - + reload - If reload is True, then any previously loaded indexes will be forgotten. - + load - List of elements to load. Default loads all items. Note: plugs may ignore this. @@ -383,7 +383,14 @@ layerBranches set. If not, they are effectively blank.''' # Get a list of dependencies and then recursively process them for layerdependency in layerbranch.index.layerDependencies_layerBranchId[layerbranch.id]: - deplayerbranch = layerdependency.dependency_layerBranch + try: + deplayerbranch = layerdependency.dependency_layerBranch + except AttributeError as e: + logger.error('LayerBranch does not exist for dependent layer {}:{}\n' \ + ' Cannot continue successfully.\n' \ + ' You might be able to resolve this by checking out the layer locally.\n' \ + ' Consider reaching out the to the layer maintainers or the layerindex admins' \ + .format(layerdependency.dependency.name, layerbranch.branch.name)) if ignores and deplayerbranch.layer.name in ignores: continue @@ -846,7 +853,7 @@ class LayerIndexObj(): continue for layerdependency in layerbranch.index.layerDependencies_layerBranchId[layerbranch.id]: - deplayerbranch = layerdependency.dependency_layerBranch + deplayerbranch = layerdependency.dependency_layerBranch or None if ignores and deplayerbranch.layer.name in ignores: continue diff --git a/poky/bitbake/lib/progressbar/progressbar.py b/poky/bitbake/lib/progressbar/progressbar.py index e2b6ba1083..d4da10ab75 100644 --- a/poky/bitbake/lib/progressbar/progressbar.py +++ b/poky/bitbake/lib/progressbar/progressbar.py @@ -253,7 +253,7 @@ class ProgressBar(object): if (self.maxval is not UnknownLength and not 0 <= value <= self.maxval): - raise ValueError('Value out of range') + self.maxval = value self.currval = value |