diff options
| author | Brad Bishop <bradleyb@fuzziesquirrel.com> | 2019-08-19 20:50:42 +0300 |
|---|---|---|
| committer | Brad Bishop <bradleyb@fuzziesquirrel.com> | 2019-08-19 20:52:00 +0300 |
| commit | 96ff1984133494bf6a3451ddeb7f14548d3697e1 (patch) | |
| tree | f2c9093a4ddffe5fb78f5dccbba36fac85603f37 /poky/bitbake/lib/bb/runqueue.py | |
| parent | fd4f7537ebeee494d4dd91b7438ed9512eeda303 (diff) | |
| download | openbmc-96ff1984133494bf6a3451ddeb7f14548d3697e1.tar.xz | |
subtree updates
poky: 67266331b0..835f7eac06:
Adrian Bunk (9):
valgrind: Remove dependency on libx11
bluez5: Remove obsolete dependency on dbus-glib
python3-dbus: Remove obsolete dependency on dbus-glib
cups: Remove unnecessary dependency on dbus-glib
libnotify: Remove obsolete dependency on dbus-glib
unfs3: Switch to new upstream location
i2c-tools: Add alternative for i2ctransfer
meta: Remove remnants of bluez4 support
e2fsprogs: Remove patch that disabled 64bit for ext4 by default
Adrian Freihofer (1):
yocto-bsp: runqemu runs beaglebone-yocto
Adrian Ratiu (1):
opkg/package/rootfs_ipk: allow overwriting OPKGLIBDIR
Alejandro del Castillo (1):
opkg: upgrade to version 0.4.1
Alexander Kanavin (3):
rt-tests: exclude 1.4 version from upstream check as well
gtk-doc: correct the style.css permissions
mobile-broadband-provider-info: upgrade 20190116 -> 20190618
Alistair Francis (7):
mesa: Add support for the lima PACKAGECONFIG
u-boot: Update to 2019.07
packagegroup-core-sdk: Set blank sanitiser for RISC-V 32
opensbi: Update from 0.3 to 0.4
opensbi: Fix installed-vs-shipped warning
qemurunner.py: Be more verbose about problems
package_manager: Ensure the base-feed directory exists
Andrej Valek (2):
busybox: 1.30.1 -> 1.31.0
oe/copy_buildsystem: move layer into layers directory
Anuj Mittal (25):
gstreamer1.0-plugins-bad: depend on vulkan-loader now
vulkan-demos: depend on vulkan-loader
vulkan: remove
binutils: fix CVE-2019-12972 CVE-2019-9071
gnupg: upgrade 2.2.16 -> 2.2.17
libxslt: fix CVE-2019-13117 CVE-2019-13118
libva: upgrade 2.4.1 -> 2.5.0
libva-utils: upgrade 2.4.0 -> 2.5.0
nasm: fix CVE-2018-19755
python: fix CVE-2019-9740
python3: upgrade 3.7.3 -> 3.7.4
binutils: CVE-2019-9070 is same as CVE-2019-9071
qemu: fix CVE-2019-12155
bzip2: upgrade 1.0.7 -> 1.0.8
glib-2.0: upgrade 2.60.4 -> 2.60.5
vte: upgrade 0.56.1 -> 0.56.3
openssl: set CVE vendor to openssl
curl: upgrade 7.65.1 -> 7.65.2
rsync: fix CVEs for included zlib
glibc: CVE-2018-20796 is same as CVE-2019-9169
unzip: fix CVE-2019-13232
python: include CVE patches for python-native as well
gdb: fix CVE-2017-9778
iptables: upgrade 1.8.2 -> 1.8.3
piglit: fix SRC_URI
Armin Kuster (1):
timezone: update to 2019b
Bonnans, Laurent (1):
openssl: fix valgrind errors on v1.1.1c
Bruce Ashfield (5):
linux-yocto/5.0: bsp: add basic xilinx zynqmp support
linux-yocto/5.0: make scsi-debug include scsi core configs
linux-yocto: bsp/beaglebone: support qemu -machine virt
linux-yocto/4.19: update to 4.19.57 and -rt22
package: check PKG_ variables before executing ontarget postinst
CHerzig@Gauselmann.de (1):
bitbake: fetch2/clearcase: Fix class import errors
Changqing Li (5):
quilt: run-ptest remove Interactive Input
mdadm: fix systemd service start up failure
mdam: fix mdmonitor start up failure
opkg: make ptest output format align with common style
mdadm: make ptest output format align with common style
Chee Yang Lee (1):
wic: add support for kernel with initramfs bundled
Chen Qi (13):
target-sdk-provides-dummy: add libperl.so.5 64bit
devtool: warn user about multiple layer having the same base name
image.bbclass: fix systemd_preset_all
devtool.py: track to clean devtool.conf in test_create_workspace
grub-efi.bbclass: take into consideration of multilib
sysstat: use service file from source codes
xmlcatalog: hold libxml2-native dependency
oeqa/runtime/rpm: ensure no user process running before deleting user
oeqa/runtime/rpm: Move test_rpm_query_nonroot test case to RpmBasicTest
qemurunner.py: fix race condition at qemu startup
msmtp: use alternatives to manage /usr/lib/sendmail
runtime_test.py: use track_for_cleanup for temp dir
devtool: remove temp dir in upgrade
Fabio Berton (1):
mesa: Update 19.1.0 -> 19.1.1
Haiqing Bai (1):
sysstat: Use sysstat.service in source for cron with systemd
He Zhe (1):
ltp: file01: Fix in was not recognized
Hongzhi.Song (3):
ltp: fix shmctl01 failure when executed.
ltp: diotest4: Let kernel pick an address when calling mmap
ltp: getrlimit03: adjust-a-bit-of-code-to-compatiable-with mips32
Jason Wessel (5):
glibc: Fix multilibs + usrmerge builds
psmisc: Fix dependency for USE_NLS=no
glibc-locale: Fix build error with PACKAGE_NO_GCONV = "1"
glibc/glibc-locale: Fix do_stash_locale to work with usrmerge and multilibs
glibc / glibc-locale: Fix stash_locale determinism problems
Joe Slater (1):
libtool: remove host information from libtool
Jon Mason (1):
oe_syslog.py: Handle syslogd/klogd restart race
Joshua Watt (5):
python3: Fix .pyc file reproduciblility
oeqa: Test bitbake --skip-setsecene
bitbake: bitbake: Add --skip-setscene option
classes/icecc: Disable remote pre-processing by default
scripts/buildstats-diff: Add option to filter tasks
Joël Esponde (1):
package.bbclass: fix directories setuid and setgid bits
Jun Nie (1):
kernel-fitimage: uboot-sign: fix missing signature
Kai Kang (4):
rng-tools: fix rngd blocks system shutdown
openssl: fix multilib files conflict
webkitgtk: set incomptible with tune mips
defaultsetup.conf: enable select init manager
Khem Raj (10):
efibootmgr: Pass correct flags to compiler from pkg-config
mpeg2dec: Fix PIE build and avoid relocation in text section on ARM
Revert "unzip: fix CVE-2019-13232"
musl: Upgrade to 1.1.23+
mdadm: Include sys/sysmacros.h for major/minor definitions
sysvinit: Include sys/sysmacros.h for major/minor definitions on musl too
pam_systemd: Include missing.h for secure_getenv
musl-obstack: Add recipe
elfutils: Fix eu-* utils builds for musl
maintainers: Account for musl-obstack and libssp-nonshared
Li Zhou (2):
bc: dc: fix exit code of q command
iptables: Security Advisory - iptables - CVE-2019-11360
Luca Boccassi (1):
bitbake: tests/fetch.py: add missing skipIfNoNetwork tags to tests that try to git clone
Matthias Schiffer (1):
systemd: backport patch to fix sysctl warning on boot
Mike Crowe (4):
bitbake.conf: Stop exporting TARGET_ flags variables
image.bbclass: Only append to IMAGE_LINK_NAME if it was already set
rootfs-postcommands: Cope with empty IMAGE_LINK_NAME in write_image_manifest
rootfs-postcommands: Cope with empty IMAGE_LINK_NAME in write_image_test_data
Mikko Rapeli (3):
busybox: enable unicode support
cve-check.bbclass: initialize to_append
freetype: add --tag CC to libtool arguments
Mingli Yu (2):
go.bbclass: separate the ptest logic to go-ptest class
mdadm: fix ptest hang
Oleksandr Kravchuk (34):
mc: update to 4.8.23
encodings: update to 1.0.5
gawk: update to 5.0.1
libinput: update to 1.13.3
libxi: update to 1.7.10
libxt: update to 1.2.0
autoconf-archive: update to 2019.01.06
python3-mako: update to 1.0.12
python3-pbr: update to 5.3.1
python3-pygobject: update to 3.32.2
git: update to 2.22.0
eudev: update to 3.2.8
babeltrace: update to 1.5.7
dpkg: update to 1.19.7
apt: update to 1.2.31
libinput: update to 1.13.4
expat: update to 2.2.7
libsolf: update to 0.7.5
bison: update to 3.4.1
ruby: update to 2.5.5
quilt: update to 0.66
bzip2: update to 1.0.7
python3-mako: update to 1.0.13
ifupdown: update to 0.8.22
libdrm: update to 2.4.99
python3-pbr: update to 5.4.0
linux-firmware: bump to 20190618
iproute2: update to 5.2.0
udev-extraconf: do not mount swap partitions
python3-pbr: update to 5.4.1
xinput: update to 1.6.3
python3-scons: update to 3.1.0
python3-docutils: update to 0.15
python3-mako: update to 1.0.14
Pascal Bach (1):
cmake: 3.14.1 -> 3.14.5
Paul Eggleton (7):
libcap-ng: do not use symlink to share files with libcap-ng-python
scripts/contrib/ddimage: fix typo
scripts/contrib/ddimage: replace blacklist with mount check
scripts/contrib/ddimage: be explicit whether device doesn't exist or isn't writeable
list-packageconfig-flags: print PN instead of P
recipetool: ignore zero-length setup.py files
devtool: upgrade: fix handling of errors parsing upgraded recipe
Peter Kjellerstedt (4):
glib-2.0: Update to 2.60.4
glibc-package.inc: Do not use bitbake variable syntax for shell variables
meson.bbclass: Remove the MESON_*_ARGS variables
nativesdk-meson: Remove some unused variables
Pierre Le Magourou (10):
cve-update-db: Use std library instead of urllib3
cve-update-db: Manage proxy if needed.
cve-update-db: do_populate_cve_db depends on do_fetch
cve-update-db: Catch request.urlopen errors.
cve-check: Depends on cve-update-db-native
cve-update-db: Use NVD CPE data to populate PRODUCTS table
cve-check: Update unpatched CVE matching
cve-update-db-native: Skip recipe when cve-check class is not loaded.
cve-check: Replace CVE_CHECK_CVE_WHITELIST by CVE_CHECK_WHITELIST
cve-update-db-native: Remove hash column from database.
Ricardo Ribalda Delgado (4):
nfs-mountd: Add missing dependency on systemd service
systemd: Fix interface bring-up on kernels >= 5.2
wic: Fix (again) partition files UIDs on multi rootfs images
systemd-bootconf: Mark as machine specific
Ricardo Salveti (1):
gcc-9.1: add back GLIBC_DYNAMIC_LINKER riscv changes
Richard Purdie (58):
multilib_global: Fix multilib rebuild issue
multilib_global: Fix KERNEL_VERSION expansion problems
sysklogd: Fix init script races
busybox: Improve syslog restart handling
oeqa/runtime/syslog: Improve test debug messages
oeqa/runtime/oesyslog: systemd syslog restart doesn't change pid
oeqa/runtime/syslog: Add delay to test to avoid failures
busybox: Fix typo in syslog initscript
pigz: Add debug for autobuilder errors
staging: Code cleanup
package: Build pkgdata specific to the current recipe
Revert "pigz: Add debug for autobuilder errors"
grub2: Drop unneeded code
bitbake: event: Clear ui_queue after handling it
bitbake: main: Ensure log messages are printed when no UI starts
bitbake: main: Alter EOFError handling
core-image-sato-sdk-ptest: Reduce image padding size due to bootimg 4GB limit
oeqa/bbtests: Tweak test bitbake output pattern matching
sstate: Add tweak to avoid multiple sstate stats messages
bitbake: siggen: Fix default handler
bitbake: siggen: Use unique hashes for tasks
bitbake: runqueue: Tweak buildable variable handling in scheduler
bitbake: runqueue: Drop unused BB_SETSCENE_VERIFY_FUNCTION2
bitbake: runqueue: Remove now uneeded code
bitbake: runqueue: Move scenequeue data generation to a separate function
bitbake: runqueue: Remove unused function parameter
bitbake: runqueue: Factor out the process_setscene_whitelist checks
bitbake: runqueue: Uniquely namespace the scenequeue functions
bitbake: runqueue: Merge stats handling together for setscene/real tasks
bitbake: runqueue: Merge scenequeue and real task queue code together
bitbake: runqueue: Fix counter/task updating glitch
bitbake: runqueue: Remove RunQueueExecuteScenequeue and RunQueueExecuteTasks
bitbake: runqueue: Simplify _execute_runqueue logic
bitbake: runqueue: Fold remains of the scenequeue setup into RunQueueExecute
bitbake: event/runqueue: Drop StampUpdate event, its pointless/unused
bitbake: runqueue: Add covered_tasks (or 'collated_deps') to scenequeue data
bitbake: runqueue: Simplify scenequeue unskippable calculation
bitbake: runqueue: Tweak comments and debug code
bitbake: runqueue: Code simplification
bitbake: runqueue: Remove pointless variable
bitbake: runqueue: Further scheduler buildable tasks cleanup
bitbake: runqueue: Clarify scenequeue_covered vs. tasks_covered
bitbake: runqueue: Merge the queues and execute setscene and normal tasks in parallel
bitbake: runqueue: Alter setscenewhitelist handling
bitbake: runqueue: Complete the merge of scenequeue and normal task execution
bitbake: tests: Add initial scenario based test for runqueue
bitbake: uihelper: No longer listen to scenequeue task started
bitbake: runqueue: Simplify some convoluted logic
bitbake: runqueue: Whitespace fix
bitbake: runqueue: Abstract hash verification function
bitbake: runqueue: Optimise multiconfig with overlapping setscene
bitbake: tests/runqueue: Allow common sstate tasks to become valid
bitbake: runqueue: Fix non setscene tasks targets being lost
staging: Drop clean_recipe_sysroot
poky-lsb: Drop features already in poky
poky-lsb: Drop libx11 PREFERRED_PROVIDER
distro/include: Add poky-distro-alt-test-config.inc
bitbake: siggen: Fix handling of tainted sig files
Robert Yang (13):
update-alternatives.bbclass: run update-alternatives firstly in postinst script
busybox: make postinst run firstly before update-alternatives
multilib.bbclass: Reduce ALTERNATIVE_PRIORITY for extended recipes
bitbake: bitbake: lib: Cleanup /usr/bin/env python
bitbake: bitbake: toaster:tests: python -> python3
ksum.py: python -> python3
wic: python2 -> python3
ext-sdk-prepare.py: python2 -> python3
oeqa: Cleanup /usr/bin/env python
package_rpm.bbclass: python2 -> python3
bitbake: cache: Remove duplicated lines for provides and rprovides
bitbake: cache: Set packages for skipped recipes
bitbake: cache: Create a symlink for current cachefile
Ross Burton (56):
cve-check: be idiomatic
gtk-icon-cache: rename intercept to update_gtk_icon_cache
fortran-helloworld: add a very dumb Fortran Hello World for testing
oeqa/buildoptions: check that Fortran code actually cross-compiles
buildhistory: write the contents of the sysroot
buildhistory: report sysroot changes
perl: fix Upstream-Status tags
efivar: ensure that target security flags are not used to build native code
multilib_script: fix whitespace
buildhistory_analysis: ignore ownership for sysroot diffs
insane: use clean_path for the host contamination warnings
libsndfile1: disable use of sqlite3 by default
libsndfile1: remove redundant autoconf seeding
buildhistory: don't output ownership for the sysroot
buildhistory: filter out the unexpected prefix for native/cross sysroots
alsa-utils: disable tools using GTK+2
packagegroup-core-lsb: remove GTK+
recipetool: add MD5 hash for the line-wrapped MPL-1.1 license
oeqa/recipetool: change the CMake test to use taglib
gtk+: remove GTK+ 2
gnome-themes-standard: remove
Revert "sysstat: use service file from source codes"
libpsl: update Upstream-Status
grub: build with python 3
qemu: use Python 3 to build
ninja: use Python 3
conf/poky: add debian-10 to the supported distribution list
tiff: remove redundant patch
tiff: fix CVE-2019-6128
tiff: fix CVE-2019-7663
cve-check: remove redundant readline CVE whitelisting
cve-check-tool: remove
glibc: exclude child recipes from CVE scanning
libid3tag: CVE-2017-11551 is the same as CVE-2004-2779
libid3tag: handle unknown encodings (CVE-2017-11550)
subversion: set CVE vendor to Apache
boost: set CVE vendor to Boost
git: set CVE vendor to git-scm
ed: set CVE vendor to avoid false positives
cve-check: allow comparison of Vendor as well as Product
flex: set CVE_PRODUCT to include vendor
cve-update-db-native: use SQL placeholders instead of format strings
xkeyboard-config: remove redundant intltool dependency
piglit: upgrade to latest revision
pkgconf: upgrade 1.6.1 -> 1.6.3
conf/poky: add Fedora 30 and Opensuse Leap 15.1 to supported distributions
cve-update-db-native: use os.path.join instead of +
cve-update-db: actually inherit native
cve-update-db-native: use executemany() to optimise CPE insertion
cve-update-db-native: improve metadata parsing
cve-update-db-native: clean up JSON fetching
freetype: upgrade to 2.10.1
unfs3: set upstream tag regex to avoid false-positives
meson.bbclass: export STRIP=${BUILD_STRIP}
ffmpeg: don't use hardcoded lookup tables
ffmpeg: upgrade to 4.1.4
Sai Hari Chandana Kalluri (3):
devtool/standard.py: Update devtool modify to copy source from work-shared if its already downloaded
devtool/standard.py: Create a copy of kernel source within work-shared if not present
devtool: provide support for devtool menuconfig command
Scott Rifenbark (5):
overview-manual: Fixed manual history table
sdk-manual: Updated devtool to talk about oe-local-files.
dev-manual: Provided proper link title
ref-manual: Fixed typo for BBMULTICONFIG variable.
ref-manual: Removed "python2" mention in example.
Stefan Agner (1):
psplash: create psplash tmpfs mount directory in psplash-init
Tim Orling (3):
vulkan-headers: add recipe
vulkan-loader: add recipe
vulkan-tools: add recipe
Ulrich Ölmann (1):
squashfs-tools: upgrade to commit f95864afe883
William Bourque (2):
wic/plugins: Source that support both EFI and BIOS
meta/lib/oeqa: Test for bootimg-biosplusefi Source
Yi Zhao (2):
debianutils: upgrade 4.8.6.1 -> 4.8.6.3
ltp: upgrade 20190115 -> 20190517
Zang Ruochen (9):
nss: upgrade 3.44 -> 3.44.1
util-linux:upgrade 2.33.2 -> 2.34
librepo:upgrade 1.10.3 -> 1.10.4
sqlite3: Upgrade 3.28.0 -> 3.29.0
nss: Upgrade 3.44.1 -> 3.45
xauth:upgrade 1.0.10 -> 1.1
libice:upgrade 1.0.9 -> 1.0.10
xwininfo:upgrade 1.1.4 -> 1.1.5
libpciaccess:upgrade 0.14 -> 0.16
meta-phosphor: fe8cee7488..601f253a66:
Brad Bishop (1):
meta-phosphor: systemd: remove upstreamed patches
Change-Id: If591144821cd2e5b990a7aa49a1cf426f6a906de
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>
Diffstat (limited to 'poky/bitbake/lib/bb/runqueue.py')
| -rw-r--r-- | poky/bitbake/lib/bb/runqueue.py | 1244 |
1 files changed, 638 insertions, 606 deletions
diff --git a/poky/bitbake/lib/bb/runqueue.py b/poky/bitbake/lib/bb/runqueue.py index 010b085010..6a2de240cc 100644 --- a/poky/bitbake/lib/bb/runqueue.py +++ b/poky/bitbake/lib/bb/runqueue.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python """ BitBake 'RunQueue' implementation @@ -26,6 +25,7 @@ import subprocess import pickle from multiprocessing import Process import shlex +import pprint bblogger = logging.getLogger("BitBake") logger = logging.getLogger("BitBake.RunQueue") @@ -68,6 +68,14 @@ def build_tid(mc, fn, taskname): return "mc:" + mc + ":" + fn + ":" + taskname return fn + ":" + taskname +# Index used to pair up potentially matching multiconfig tasks +# We match on PN, taskname and hash being equal +def pending_hash_index(tid, rqdata): + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) + pn = rqdata.dataCaches[mc].pkg_fn[taskfn] + h = rqdata.runtaskentries[tid].hash + return pn + ":" + "taskname" + h + class RunQueueStats: """ Holds statistics on the tasks handled by the associated runQueue @@ -103,8 +111,6 @@ class RunQueueStats: # runQueue state machine runQueuePrepare = 2 runQueueSceneInit = 3 -runQueueSceneRun = 4 -runQueueRunInit = 5 runQueueRunning = 6 runQueueFailed = 7 runQueueCleanUp = 8 @@ -143,7 +149,8 @@ class RunQueueScheduler(object): Return the id of the first task we find that is buildable """ self.buildable = [x for x in self.buildable if x not in self.rq.runq_running] - if not self.buildable: + buildable = [x for x in self.buildable if (x in self.rq.tasks_covered or x in self.rq.tasks_notcovered)] + if not buildable: return None # Filter out tasks that have a max number of threads that have been exceeded @@ -159,8 +166,8 @@ class RunQueueScheduler(object): else: skip_buildable[rtaskname] = 1 - if len(self.buildable) == 1: - tid = self.buildable[0] + if len(buildable) == 1: + tid = buildable[0] taskname = taskname_from_tid(tid) if taskname in skip_buildable and skip_buildable[taskname] >= int(self.skip_maxthread[taskname]): return None @@ -175,7 +182,7 @@ class RunQueueScheduler(object): best = None bestprio = None - for tid in self.buildable: + for tid in buildable: taskname = taskname_from_tid(tid) if taskname in skip_buildable and skip_buildable[taskname] >= int(self.skip_maxthread[taskname]): continue @@ -1194,7 +1201,6 @@ class RunQueue: self.stamppolicy = cfgData.getVar("BB_STAMP_POLICY") or "perfile" self.hashvalidate = cfgData.getVar("BB_HASHCHECK_FUNCTION") or None - self.setsceneverify = cfgData.getVar("BB_SETSCENE_VERIFY_FUNCTION2") or None self.depvalidate = cfgData.getVar("BB_SETSCENE_DEPVALID") or None self.state = runQueuePrepare @@ -1203,7 +1209,7 @@ class RunQueue: # Invoked at regular time intervals via the bitbake heartbeat event # while the build is running. We generate a unique name for the handler # here, just in case that there ever is more than one RunQueue instance, - # start the handler when reaching runQueueSceneRun, and stop it when + # start the handler when reaching runQueueSceneInit, and stop it when # done with the build. self.dm = monitordisk.diskMonitor(cfgData) self.dm_event_handler_name = '_bb_diskmonitor_' + str(id(self)) @@ -1378,10 +1384,43 @@ class RunQueue: cache[tid] = iscurrent return iscurrent - def validate_hash(self, *, sq_fn, sq_task, sq_hash, sq_hashfn, siginfo, sq_unihash, d): + def validate_hashes(self, tocheck, data, presentcount=None, siginfo=False): + valid = set() + if self.hashvalidate: + sq_hash = [] + sq_hashfn = [] + sq_unihash = [] + sq_fn = [] + sq_taskname = [] + sq_task = [] + for tid in tocheck: + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) + + sq_fn.append(fn) + sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn]) + sq_hash.append(self.rqdata.runtaskentries[tid].hash) + sq_unihash.append(self.rqdata.runtaskentries[tid].unihash) + sq_taskname.append(taskname) + sq_task.append(tid) + + if presentcount is not None: + data.setVar("BB_SETSCENE_STAMPCURRENT_COUNT", presentcount) + + valid_ids = self.validate_hash(sq_fn, sq_taskname, sq_hash, sq_hashfn, siginfo, sq_unihash, data, presentcount) + + if presentcount is not None: + data.delVar("BB_SETSCENE_STAMPCURRENT_COUNT") + + for v in valid_ids: + valid.add(sq_task[v]) + + return valid + + def validate_hash(self, sq_fn, sq_task, sq_hash, sq_hashfn, siginfo, sq_unihash, d, presentcount): locs = {"sq_fn" : sq_fn, "sq_task" : sq_task, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "sq_unihash" : sq_unihash, "siginfo" : siginfo, "d" : d} + # Backwards compatibility hashvalidate_args = ("(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=siginfo, sq_unihash=sq_unihash)", "(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=siginfo)", "(sq_fn, sq_task, sq_hash, sq_hashfn, d)") @@ -1408,7 +1447,6 @@ class RunQueue: retval = True if self.state is runQueuePrepare: - self.rqexe = RunQueueExecuteDummy(self) # NOTE: if you add, remove or significantly refactor the stages of this # process then you should recalculate the weightings here. This is quite # easy to do - just change the next line temporarily to pass debug=True as @@ -1422,18 +1460,19 @@ class RunQueue: self.state = runQueueComplete else: self.state = runQueueSceneInit - self.rqdata.init_progress_reporter.next_stage() - - # we are ready to run, emit dependency info to any UI or class which - # needs it - depgraph = self.cooker.buildDependTree(self, self.rqdata.taskData) - self.rqdata.init_progress_reporter.next_stage() - bb.event.fire(bb.event.DepTreeGenerated(depgraph), self.cooker.data) if self.state is runQueueSceneInit: + self.rqdata.init_progress_reporter.next_stage() + + # we are ready to run, emit dependency info to any UI or class which + # needs it + depgraph = self.cooker.buildDependTree(self, self.rqdata.taskData) + self.rqdata.init_progress_reporter.next_stage() + bb.event.fire(bb.event.DepTreeGenerated(depgraph), self.cooker.data) + if not self.dm_event_handler_registered: res = bb.event.register(self.dm_event_handler_name, - lambda x: self.dm.check(self) if self.state in [runQueueSceneRun, runQueueRunning, runQueueCleanUp] else False, + lambda x: self.dm.check(self) if self.state in [runQueueRunning, runQueueCleanUp] else False, ('bb.event.HeartbeatEvent',)) self.dm_event_handler_registered = True @@ -1446,24 +1485,23 @@ class RunQueue: if 'printdiff' in dump: self.write_diffscenetasks(invalidtasks) self.state = runQueueComplete - else: - self.rqdata.init_progress_reporter.next_stage() - self.start_worker() - self.rqdata.init_progress_reporter.next_stage() - self.rqexe = RunQueueExecuteScenequeue(self) - - if self.state is runQueueSceneRun: - retval = self.rqexe.execute() - if self.state is runQueueRunInit: - if self.cooker.configuration.setsceneonly: - self.state = runQueueComplete - else: - # Just in case we didn't setscene - self.rqdata.init_progress_reporter.finish() - logger.info("Executing RunQueue Tasks") - self.rqexe = RunQueueExecuteTasks(self) - self.state = runQueueRunning + if self.state is runQueueSceneInit: + self.rqdata.init_progress_reporter.next_stage() + self.start_worker() + self.rqdata.init_progress_reporter.next_stage() + self.rqexe = RunQueueExecute(self) + + # If we don't have any setscene functions, skip execution + if len(self.rqdata.runq_setscene_tids) == 0: + logger.info('No setscene tasks') + for tid in self.rqdata.runtaskentries: + if len(self.rqdata.runtaskentries[tid].depends) == 0: + self.rqexe.setbuildable(tid) + self.rqexe.tasks_notcovered.add(tid) + self.rqexe.sqdone = True + logger.info('Executing Tasks') + self.state = runQueueRunning if self.state is runQueueRunning: retval = self.rqexe.execute() @@ -1479,11 +1517,12 @@ class RunQueue: if build_done and self.rqexe: self.teardown_workers() - if self.rqexe.stats.failed: - logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and %d failed.", self.rqexe.stats.completed + self.rqexe.stats.failed, self.rqexe.stats.skipped, self.rqexe.stats.failed) - else: - # Let's avoid the word "failed" if nothing actually did - logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and all succeeded.", self.rqexe.stats.completed, self.rqexe.stats.skipped) + if self.rqexe: + if self.rqexe.stats.failed: + logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and %d failed.", self.rqexe.stats.completed + self.rqexe.stats.failed, self.rqexe.stats.skipped, self.rqexe.stats.failed) + else: + # Let's avoid the word "failed" if nothing actually did + logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and all succeeded.", self.rqexe.stats.completed, self.rqexe.stats.skipped) if self.state is runQueueFailed: raise bb.runqueue.TaskFailure(self.rqexe.failed_tids) @@ -1566,16 +1605,8 @@ class RunQueue: def print_diffscenetasks(self): - valid = [] - sq_hash = [] - sq_hashfn = [] - sq_unihash = [] - sq_fn = [] - sq_taskname = [] - sq_task = [] noexec = [] - stamppresent = [] - valid_new = set() + tocheck = set() for tid in self.rqdata.runtaskentries: (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) @@ -1585,18 +1616,9 @@ class RunQueue: noexec.append(tid) continue - sq_fn.append(fn) - sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn]) - sq_hash.append(self.rqdata.runtaskentries[tid].hash) - sq_unihash.append(self.rqdata.runtaskentries[tid].unihash) - sq_taskname.append(taskname) - sq_task.append(tid) - - valid = self.validate_hash(sq_fn=sq_fn, sq_task=sq_taskname, sq_hash=sq_hash, sq_hashfn=sq_hashfn, - siginfo=True, sq_unihash=sq_unihash, d=self.cooker.data) + tocheck.add(tid) - for v in valid: - valid_new.add(sq_task[v]) + valid_new = self.validate_hashes(tocheck, self.cooker.data, None, True) # Tasks which are both setscene and noexec never care about dependencies # We therefore find tasks which are setscene and noexec and mark their @@ -1680,6 +1702,7 @@ class RunQueue: output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb) bb.plain("\nTask %s:%s couldn't be used from the cache because:\n We need hash %s, closest matching task was %s\n " % (pn, taskname, h, prevh) + '\n '.join(output)) + class RunQueueExecute: def __init__(self, rq): @@ -1691,6 +1714,10 @@ class RunQueueExecute: self.number_tasks = int(self.cfgData.getVar("BB_NUMBER_THREADS") or 1) self.scheduler = self.cfgData.getVar("BB_SCHEDULER") or "speed" + self.sq_buildable = set() + self.sq_running = set() + self.sq_live = set() + self.runq_buildable = set() self.runq_running = set() self.runq_complete = set() @@ -1698,9 +1725,15 @@ class RunQueueExecute: self.build_stamps = {} self.build_stamps2 = [] self.failed_tids = [] + self.sq_deferred = {} self.stampcache = {} + self.sqdone = False + + self.stats = RunQueueStats(len(self.rqdata.runtaskentries)) + self.sq_stats = RunQueueStats(len(self.rqdata.runq_setscene_tids)) + for mc in rq.worker: rq.worker[mc].pipe.setrunqueueexec(self) for mc in rq.fakeworker: @@ -1709,6 +1742,31 @@ class RunQueueExecute: if self.number_tasks <= 0: bb.fatal("Invalid BB_NUMBER_THREADS %s" % self.number_tasks) + # List of setscene tasks which we've covered + self.scenequeue_covered = set() + # List of tasks which are covered (including setscene ones) + self.tasks_covered = set() + self.tasks_scenequeue_done = set() + self.scenequeue_notcovered = set() + self.tasks_notcovered = set() + self.scenequeue_notneeded = set() + + self.coveredtopocess = set() + + schedulers = self.get_schedulers() + for scheduler in schedulers: + if self.scheduler == scheduler.name: + self.sched = scheduler(self, self.rqdata) + logger.debug(1, "Using runqueue scheduler '%s'", scheduler.name) + break + else: + bb.fatal("Invalid scheduler '%s'. Available schedulers: %s" % + (self.scheduler, ", ".join(obj.name for obj in schedulers))) + + if len(self.rqdata.runq_setscene_tids) > 0: + self.sqdata = SQData() + build_scenequeue_data(self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self) + def runqueue_process_waitpid(self, task, status): # self.build_stamps[pid] may not exist when use shared work directory. @@ -1716,10 +1774,17 @@ class RunQueueExecute: self.build_stamps2.remove(self.build_stamps[task]) del self.build_stamps[task] - if status != 0: - self.task_fail(task, status) + if task in self.sq_live: + if status != 0: + self.sq_task_fail(task, status) + else: + self.sq_task_complete(task) + self.sq_live.remove(task) else: - self.task_complete(task) + if status != 0: + self.task_fail(task, status) + else: + self.task_complete(task) return True def finish_now(self): @@ -1748,8 +1813,9 @@ class RunQueueExecute: def finish(self): self.rq.state = runQueueCleanUp - if self.stats.active > 0: - bb.event.fire(runQueueExitWait(self.stats.active), self.cfgData) + active = self.stats.active + self.sq_stats.active + if active > 0: + bb.event.fire(runQueueExitWait(active), self.cfgData) self.rq.read_workers() return self.rq.active_fds() @@ -1760,7 +1826,8 @@ class RunQueueExecute: self.rq.state = runQueueComplete return True - def check_dependencies(self, task, taskdeps, setscene = False): + # Used by setscene only + def check_dependencies(self, task, taskdeps): if not self.rq.depvalidate: return False @@ -1776,121 +1843,10 @@ class RunQueueExecute: return valid def can_start_task(self): - can_start = self.stats.active < self.number_tasks + active = self.stats.active + self.sq_stats.active + can_start = active < self.number_tasks return can_start -class RunQueueExecuteDummy(RunQueueExecute): - def __init__(self, rq): - self.rq = rq - self.stats = RunQueueStats(0) - - def finish(self): - self.rq.state = runQueueComplete - return - -class RunQueueExecuteTasks(RunQueueExecute): - def __init__(self, rq): - RunQueueExecute.__init__(self, rq) - - self.stats = RunQueueStats(len(self.rqdata.runtaskentries)) - - self.stampcache = {} - - initial_covered = self.rq.scenequeue_covered.copy() - - # Mark initial buildable tasks - for tid in self.rqdata.runtaskentries: - if len(self.rqdata.runtaskentries[tid].depends) == 0: - self.runq_buildable.add(tid) - if len(self.rqdata.runtaskentries[tid].revdeps) > 0 and self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered): - self.rq.scenequeue_covered.add(tid) - - found = True - while found: - found = False - for tid in self.rqdata.runtaskentries: - if tid in self.rq.scenequeue_covered: - continue - logger.debug(1, 'Considering %s: %s' % (tid, str(self.rqdata.runtaskentries[tid].revdeps))) - - if len(self.rqdata.runtaskentries[tid].revdeps) > 0 and self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered): - if tid in self.rq.scenequeue_notcovered: - continue - found = True - self.rq.scenequeue_covered.add(tid) - - logger.debug(1, 'Skip list (pre setsceneverify) %s', sorted(self.rq.scenequeue_covered)) - - # Allow the metadata to elect for setscene tasks to run anyway - covered_remove = set() - if self.rq.setsceneverify: - invalidtasks = [] - tasknames = {} - fns = {} - for tid in self.rqdata.runtaskentries: - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) - taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn] - fns[tid] = taskfn - tasknames[tid] = taskname - if 'noexec' in taskdep and taskname in taskdep['noexec']: - continue - if self.rq.check_stamp_task(tid, taskname + "_setscene", cache=self.stampcache): - logger.debug(2, 'Setscene stamp current for task %s', tid) - continue - if self.rq.check_stamp_task(tid, taskname, recurse = True, cache=self.stampcache): - logger.debug(2, 'Normal stamp current for task %s', tid) - continue - invalidtasks.append(tid) - - call = self.rq.setsceneverify + "(covered, tasknames, fns, d, invalidtasks=invalidtasks)" - locs = { "covered" : self.rq.scenequeue_covered, "tasknames" : tasknames, "fns" : fns, "d" : self.cooker.data, "invalidtasks" : invalidtasks } - covered_remove = bb.utils.better_eval(call, locs) - - def removecoveredtask(tid): - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) - taskname = taskname + '_setscene' - bb.build.del_stamp(taskname, self.rqdata.dataCaches[mc], taskfn) - self.rq.scenequeue_covered.remove(tid) - - toremove = covered_remove | self.rq.scenequeue_notcovered - for task in toremove: - logger.debug(1, 'Not skipping task %s due to setsceneverify', task) - while toremove: - covered_remove = [] - for task in toremove: - if task in self.rq.scenequeue_covered: - removecoveredtask(task) - for deptask in self.rqdata.runtaskentries[task].depends: - if deptask not in self.rq.scenequeue_covered: - continue - if deptask in toremove or deptask in covered_remove or deptask in initial_covered: - continue - logger.debug(1, 'Task %s depends on task %s so not skipping' % (task, deptask)) - covered_remove.append(deptask) - toremove = covered_remove - - logger.debug(1, 'Full skip list %s', self.rq.scenequeue_covered) - - - for mc in self.rqdata.dataCaches: - target_pairs = [] - for tid in self.rqdata.target_tids: - (tidmc, fn, taskname, _) = split_tid_mcfn(tid) - if tidmc == mc: - target_pairs.append((fn, taskname)) - - event.fire(bb.event.StampUpdate(target_pairs, self.rqdata.dataCaches[mc].stamp), self.cfgData) - - schedulers = self.get_schedulers() - for scheduler in schedulers: - if self.scheduler == scheduler.name: - self.sched = scheduler(self, self.rqdata) - logger.debug(1, "Using runqueue scheduler '%s'", scheduler.name) - break - else: - bb.fatal("Invalid scheduler '%s'. Available schedulers: %s" % - (self.scheduler, ", ".join(obj.name for obj in schedulers))) - def get_schedulers(self): schedulers = set(obj for obj in globals().values() if type(obj) is type and @@ -1964,65 +1920,137 @@ class RunQueueExecuteTasks(RunQueueExecute): def execute(self): """ - Run the tasks in a queue prepared by rqdata.prepare() + Run the tasks in a queue prepared by prepare_runqueue """ - if self.rqdata.setscenewhitelist is not None and not self.rqdata.setscenewhitelist_checked: - self.rqdata.setscenewhitelist_checked = True - - # Check tasks that are going to run against the whitelist - def check_norun_task(tid, showerror=False): - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) - # Ignore covered tasks - if tid in self.rq.scenequeue_covered: - return False - # Ignore stamped tasks - if self.rq.check_stamp_task(tid, taskname, cache=self.stampcache): - return False - # Ignore noexec tasks - taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn] - if 'noexec' in taskdep and taskname in taskdep['noexec']: - return False + self.rq.read_workers() - pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn] - if not check_setscene_enforce_whitelist(pn, taskname, self.rqdata.setscenewhitelist): - if showerror: - if tid in self.rqdata.runq_setscene_tids: - logger.error('Task %s.%s attempted to execute unexpectedly and should have been setscened' % (pn, taskname)) + task = None + if not self.sqdone and self.can_start_task(): + # Find the next setscene to run + for nexttask in self.rqdata.runq_setscene_tids: + if nexttask in self.sq_buildable and nexttask not in self.sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values(): + if nexttask not in self.sqdata.unskippable and len(self.sqdata.sq_revdeps[nexttask]) > 0 and self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.sqdata.sq_revdeps[nexttask]): + if nexttask not in self.rqdata.target_tids: + logger.debug(2, "Skipping setscene for task %s" % nexttask) + self.sq_task_skip(nexttask) + self.scenequeue_notneeded.add(nexttask) + if nexttask in self.sq_deferred: + del self.sq_deferred[nexttask] + return True + if nexttask in self.sq_deferred: + if self.sq_deferred[nexttask] not in self.runq_complete: + continue + logger.debug(1, "Task %s no longer deferred" % nexttask) + del self.sq_deferred[nexttask] + valid = self.rq.validate_hashes(set([nexttask]), self.cooker.data, None, False) + if not valid: + logger.debug(1, "%s didn't become valid, skipping setscene" % nexttask) + self.sq_task_failoutright(nexttask) + return True else: - logger.error('Task %s.%s attempted to execute unexpectedly' % (pn, taskname)) - return True - return False - # Look to see if any tasks that we think shouldn't run are going to - unexpected = False - for tid in self.rqdata.runtaskentries: - if check_norun_task(tid): - unexpected = True + self.sqdata.outrightfail.remove(nexttask) + if nexttask in self.sqdata.outrightfail: + logger.debug(2, 'No package found, so skipping setscene task %s', nexttask) + self.sq_task_failoutright(nexttask) + return True + if nexttask in self.sqdata.unskippable: + logger.debug(2, "Setscene task %s is unskippable" % nexttask) + task = nexttask break - if unexpected: - # Run through the tasks in the rough order they'd have executed and print errors - # (since the order can be useful - usually missing sstate for the last few tasks - # is the cause of the problem) - task = self.sched.next() - while task is not None: - check_norun_task(task, showerror=True) - self.task_skip(task, 'Setscene enforcement check') - task = self.sched.next() + if task is not None: + (mc, fn, taskname, taskfn) = split_tid_mcfn(task) + taskname = taskname + "_setscene" + if self.rq.check_stamp_task(task, taskname_from_tid(task), recurse = True, cache=self.stampcache): + logger.debug(2, 'Stamp for underlying task %s is current, so skipping setscene variant', task) + self.sq_task_failoutright(task) + return True - self.rq.state = runQueueCleanUp + if self.cooker.configuration.force: + if task in self.rqdata.target_tids: + self.sq_task_failoutright(task) + return True + + if self.rq.check_stamp_task(task, taskname, cache=self.stampcache): + logger.debug(2, 'Setscene stamp current task %s, so skip it and its dependencies', task) + self.sq_task_skip(task) return True - self.rq.read_workers() + if self.cooker.configuration.skipsetscene: + logger.debug(2, 'No setscene tasks should be executed. Skipping %s', task) + self.sq_task_failoutright(task) + return True - if self.stats.total == 0: - # nothing to do - self.rq.state = runQueueCleanUp + startevent = sceneQueueTaskStarted(task, self.sq_stats, self.rq) + bb.event.fire(startevent, self.cfgData) + + taskdepdata = self.sq_build_taskdepdata(task) + + taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn] + taskhash = self.rqdata.get_task_hash(task) + unihash = self.rqdata.get_task_unihash(task) + if 'fakeroot' in taskdep and taskname in taskdep['fakeroot'] and not self.cooker.configuration.dry_run: + if not mc in self.rq.fakeworker: + self.rq.start_fakeworker(self, mc) + self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>") + self.rq.fakeworker[mc].process.stdin.flush() + else: + self.rq.worker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>") + self.rq.worker[mc].process.stdin.flush() + + self.build_stamps[task] = bb.build.stampfile(taskname, self.rqdata.dataCaches[mc], taskfn, noextra=True) + self.build_stamps2.append(self.build_stamps[task]) + self.sq_running.add(task) + self.sq_live.add(task) + self.sq_stats.taskActive() + if self.can_start_task(): + return True + + if not self.sq_live and not self.sqdone and not self.sq_deferred: + logger.info("Setscene tasks completed") + logger.debug(1, 'We could skip tasks %s', "\n".join(sorted(self.scenequeue_covered))) + + completeevent = sceneQueueComplete(self.sq_stats, self.rq) + bb.event.fire(completeevent, self.cfgData) + + err = False + for x in self.rqdata.runtaskentries: + if x not in self.tasks_covered and x not in self.tasks_notcovered: + logger.error("Task %s was never moved from the setscene queue" % x) + err = True + if x not in self.tasks_scenequeue_done: + logger.error("Task %s was never processed by the setscene code" % x) + err = True + if len(self.rqdata.runtaskentries[x].depends) == 0 and x not in self.runq_buildable: + logger.error("Task %s was never marked as buildable by the setscene code" % x) + err = True + if err: + self.rq.state = runQueueFailed + return True + + if self.cooker.configuration.setsceneonly: + self.rq.state = runQueueComplete + return True + self.sqdone = True - task = self.sched.next() + if self.stats.total == 0: + # nothing to do + self.rq.state = runQueueComplete + return True + + if self.cooker.configuration.setsceneonly: + task = None + else: + task = self.sched.next() if task is not None: (mc, fn, taskname, taskfn) = split_tid_mcfn(task) - if task in self.rq.scenequeue_covered: + if self.rqdata.setscenewhitelist is not None: + if self.check_setscenewhitelist(task): + self.task_fail(task, "setscene whitelist") + return True + + if task in self.tasks_covered: logger.debug(2, "Setscene covered task %s", task) self.task_skip(task, "covered") return True @@ -2075,10 +2103,17 @@ class RunQueueExecuteTasks(RunQueueExecute): if self.can_start_task(): return True - if self.stats.active > 0: + if self.stats.active > 0 or self.sq_stats.active > 0: self.rq.read_workers() return self.rq.active_fds() + # No more tasks can be run. If we have deferred setscene tasks we should run them. + if self.sq_deferred: + tid = self.sq_deferred.pop(list(self.sq_deferred.keys())[0]) + logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s" % tid) + self.sq_task_failoutright(tid) + return True + if len(self.failed_tids) != 0: self.rq.state = runQueueFailed return True @@ -2087,9 +2122,9 @@ class RunQueueExecuteTasks(RunQueueExecute): for task in self.rqdata.runtaskentries: if task not in self.runq_buildable: logger.error("Task %s never buildable!", task) - if task not in self.runq_running: + elif task not in self.runq_running: logger.error("Task %s never ran!", task) - if task not in self.runq_complete: + elif task not in self.runq_complete: logger.error("Task %s never completed!", task) self.rq.state = runQueueComplete @@ -2131,270 +2166,100 @@ class RunQueueExecuteTasks(RunQueueExecute): #bb.note("Task %s: " % task + str(taskdepdata).replace("], ", "],\n")) return taskdepdata -class RunQueueExecuteScenequeue(RunQueueExecute): - def __init__(self, rq): - RunQueueExecute.__init__(self, rq) - - self.scenequeue_covered = set() - self.scenequeue_notcovered = set() - self.scenequeue_notneeded = set() - - # If we don't have any setscene functions, skip this step - if len(self.rqdata.runq_setscene_tids) == 0: - rq.scenequeue_covered = set() - rq.scenequeue_notcovered = set() - rq.state = runQueueRunInit - return - - self.stats = RunQueueStats(len(self.rqdata.runq_setscene_tids)) - - sq_revdeps = {} - sq_revdeps_new = {} - sq_revdeps_squash = {} - self.sq_harddeps = {} - self.stamps = {} - - # We need to construct a dependency graph for the setscene functions. Intermediate - # dependencies between the setscene tasks only complicate the code. This code - # therefore aims to collapse the huge runqueue dependency tree into a smaller one - # only containing the setscene functions. - - self.rqdata.init_progress_reporter.next_stage() - - # First process the chains up to the first setscene task. - endpoints = {} - for tid in self.rqdata.runtaskentries: - sq_revdeps[tid] = copy.copy(self.rqdata.runtaskentries[tid].revdeps) - sq_revdeps_new[tid] = set() - if (len(sq_revdeps[tid]) == 0) and tid not in self.rqdata.runq_setscene_tids: - #bb.warn("Added endpoint %s" % (tid)) - endpoints[tid] = set() - - self.rqdata.init_progress_reporter.next_stage() - - # Secondly process the chains between setscene tasks. - for tid in self.rqdata.runq_setscene_tids: - #bb.warn("Added endpoint 2 %s" % (tid)) - for dep in self.rqdata.runtaskentries[tid].depends: - if tid in sq_revdeps[dep]: - sq_revdeps[dep].remove(tid) - if dep not in endpoints: - endpoints[dep] = set() - #bb.warn(" Added endpoint 3 %s" % (dep)) - endpoints[dep].add(tid) - - self.rqdata.init_progress_reporter.next_stage() - - def process_endpoints(endpoints): - newendpoints = {} - for point, task in endpoints.items(): - tasks = set() - if task: - tasks |= task - if sq_revdeps_new[point]: - tasks |= sq_revdeps_new[point] - sq_revdeps_new[point] = set() - if point in self.rqdata.runq_setscene_tids: - sq_revdeps_new[point] = tasks - tasks = set() - continue - for dep in self.rqdata.runtaskentries[point].depends: - if point in sq_revdeps[dep]: - sq_revdeps[dep].remove(point) - if tasks: - sq_revdeps_new[dep] |= tasks - if len(sq_revdeps[dep]) == 0 and dep not in self.rqdata.runq_setscene_tids: - newendpoints[dep] = task - if len(newendpoints) != 0: - process_endpoints(newendpoints) - - process_endpoints(endpoints) - - self.rqdata.init_progress_reporter.next_stage() - - # Build a list of setscene tasks which are "unskippable" - # These are direct endpoints referenced by the build - endpoints2 = {} - sq_revdeps2 = {} - sq_revdeps_new2 = {} - def process_endpoints2(endpoints): - newendpoints = {} - for point, task in endpoints.items(): - tasks = set([point]) - if task: - tasks |= task - if sq_revdeps_new2[point]: - tasks |= sq_revdeps_new2[point] - sq_revdeps_new2[point] = set() - if point in self.rqdata.runq_setscene_tids: - sq_revdeps_new2[point] = tasks - for dep in self.rqdata.runtaskentries[point].depends: - if point in sq_revdeps2[dep]: - sq_revdeps2[dep].remove(point) - if tasks: - sq_revdeps_new2[dep] |= tasks - if (len(sq_revdeps2[dep]) == 0 or len(sq_revdeps_new2[dep]) != 0) and dep not in self.rqdata.runq_setscene_tids: - newendpoints[dep] = tasks - if len(newendpoints) != 0: - process_endpoints2(newendpoints) - for tid in self.rqdata.runtaskentries: - sq_revdeps2[tid] = copy.copy(self.rqdata.runtaskentries[tid].revdeps) - sq_revdeps_new2[tid] = set() - if (len(sq_revdeps2[tid]) == 0) and tid not in self.rqdata.runq_setscene_tids: - endpoints2[tid] = set() - process_endpoints2(endpoints2) - self.unskippable = [] - for tid in self.rqdata.runq_setscene_tids: - if sq_revdeps_new2[tid]: - self.unskippable.append(tid) - - self.rqdata.init_progress_reporter.next_stage(len(self.rqdata.runtaskentries)) - - for taskcounter, tid in enumerate(self.rqdata.runtaskentries): - if tid in self.rqdata.runq_setscene_tids: - deps = set() - for dep in sq_revdeps_new[tid]: - deps.add(dep) - sq_revdeps_squash[tid] = deps - elif len(sq_revdeps_new[tid]) != 0: - bb.msg.fatal("RunQueue", "Something went badly wrong during scenequeue generation, aborting. Please report this problem.") - self.rqdata.init_progress_reporter.update(taskcounter) - - self.rqdata.init_progress_reporter.next_stage() - - # Resolve setscene inter-task dependencies - # e.g. do_sometask_setscene[depends] = "targetname:do_someothertask_setscene" - # Note that anything explicitly depended upon will have its reverse dependencies removed to avoid circular dependencies - for tid in self.rqdata.runq_setscene_tids: - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) - realtid = tid + "_setscene" - idepends = self.rqdata.taskData[mc].taskentries[realtid].idepends - self.stamps[tid] = bb.build.stampfile(taskname + "_setscene", self.rqdata.dataCaches[mc], taskfn, noextra=True) - for (depname, idependtask) in idepends: - - if depname not in self.rqdata.taskData[mc].build_targets: + def scenequeue_process_notcovered(self, task): + if len(self.rqdata.runtaskentries[task].depends) == 0: + self.setbuildable(task) + notcovered = set([task]) + while notcovered: + new = set() + for t in notcovered: + for deptask in self.rqdata.runtaskentries[t].depends: + if deptask in notcovered or deptask in new or deptask in self.rqdata.runq_setscene_tids or deptask in self.tasks_notcovered: continue - - depfn = self.rqdata.taskData[mc].build_targets[depname][0] - if depfn is None: - continue - deptid = depfn + ":" + idependtask.replace("_setscene", "") - if deptid not in self.rqdata.runtaskentries: - bb.msg.fatal("RunQueue", "Task %s depends upon non-existent task %s:%s" % (realtid, depfn, idependtask)) - - if not deptid in self.sq_harddeps: - self.sq_harddeps[deptid] = set() - self.sq_harddeps[deptid].add(tid) - - sq_revdeps_squash[tid].add(deptid) - # Have to zero this to avoid circular dependencies - sq_revdeps_squash[deptid] = set() - - self.rqdata.init_progress_reporter.next_stage() - - for task in self.sq_harddeps: - for dep in self.sq_harddeps[task]: - sq_revdeps_squash[dep].add(task) - - self.rqdata.init_progress_reporter.next_stage() - - #for tid in sq_revdeps_squash: - # for dep in sq_revdeps_squash[tid]: - # data = data + "\n %s" % dep - # bb.warn("Task %s_setscene: is %s " % (tid, data - - self.sq_deps = {} - self.sq_revdeps = sq_revdeps_squash - self.sq_revdeps2 = copy.deepcopy(self.sq_revdeps) - - for tid in self.sq_revdeps: - self.sq_deps[tid] = set() - for tid in self.sq_revdeps: - for dep in self.sq_revdeps[tid]: - self.sq_deps[dep].add(tid) - - self.rqdata.init_progress_reporter.next_stage() - - for tid in self.sq_revdeps: - if len(self.sq_revdeps[tid]) == 0: - self.runq_buildable.add(tid) - - self.rqdata.init_progress_reporter.finish() - - self.outrightfail = [] - if self.rq.hashvalidate: - sq_hash = [] - sq_hashfn = [] - sq_unihash = [] - sq_fn = [] - sq_taskname = [] - sq_task = [] - noexec = [] - stamppresent = [] - for tid in self.sq_revdeps: - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) - - taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn] - - if 'noexec' in taskdep and taskname in taskdep['noexec']: - noexec.append(tid) - self.task_skip(tid) - bb.build.make_stamp(taskname + "_setscene", self.rqdata.dataCaches[mc], taskfn) - continue - - if self.rq.check_stamp_task(tid, taskname + "_setscene", cache=self.stampcache): - logger.debug(2, 'Setscene stamp current for task %s', tid) - stamppresent.append(tid) - self.task_skip(tid) - continue - - if self.rq.check_stamp_task(tid, taskname, recurse = True, cache=self.stampcache): - logger.debug(2, 'Normal stamp current for task %s', tid) - stamppresent.append(tid) - self.task_skip(tid) - continue - - sq_fn.append(fn) - sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn]) - sq_hash.append(self.rqdata.runtaskentries[tid].hash) - sq_unihash.append(self.rqdata.runtaskentries[tid].unihash) - sq_taskname.append(taskname) - sq_task.append(tid) - - self.cooker.data.setVar("BB_SETSCENE_STAMPCURRENT_COUNT", len(stamppresent)) - - valid = self.rq.validate_hash(sq_fn=sq_fn, sq_task=sq_taskname, sq_hash=sq_hash, sq_hashfn=sq_hashfn, - siginfo=False, sq_unihash=sq_unihash, d=self.cooker.data) - - self.cooker.data.delVar("BB_SETSCENE_STAMPCURRENT_COUNT") - - valid_new = stamppresent - for v in valid: - valid_new.append(sq_task[v]) - - for tid in self.sq_revdeps: - if tid not in valid_new and tid not in noexec: - logger.debug(2, 'No package found, so skipping setscene task %s', tid) - self.outrightfail.append(tid) - - logger.info('Executing SetScene Tasks') - - self.rq.state = runQueueSceneRun - - def scenequeue_updatecounters(self, task, fail = False): - for dep in self.sq_deps[task]: - if fail and task in self.sq_harddeps and dep in self.sq_harddeps[task]: + logger.debug(1, 'Task %s depends on non-setscene task %s so not skipping' % (t, deptask)) + new.add(deptask) + self.tasks_notcovered.add(deptask) + if len(self.rqdata.runtaskentries[deptask].depends) == 0: + self.setbuildable(deptask) + notcovered = new + + def scenequeue_process_unskippable(self, task): + # Look up the dependency chain for non-setscene things which depend on this task + # and mark as 'done'/notcovered + ready = set([task]) + while ready: + new = set() + for t in ready: + for deptask in self.rqdata.runtaskentries[t].revdeps: + if deptask in ready or deptask in new or deptask in self.tasks_scenequeue_done or deptask in self.rqdata.runq_setscene_tids: + continue + if self.rqdata.runtaskentries[deptask].depends.issubset(self.tasks_scenequeue_done): + new.add(deptask) + self.tasks_scenequeue_done.add(deptask) + self.tasks_notcovered.add(deptask) + #logger.warning("Up: " + str(deptask)) + ready = new + + def scenequeue_updatecounters(self, task, fail=False): + for dep in self.sqdata.sq_deps[task]: + if fail and task in self.sqdata.sq_harddeps and dep in self.sqdata.sq_harddeps[task]: logger.debug(2, "%s was unavailable and is a hard dependency of %s so skipping" % (task, dep)) - self.scenequeue_updatecounters(dep, fail) + self.sq_task_failoutright(dep) continue - if task not in self.sq_revdeps2[dep]: + if task not in self.sqdata.sq_revdeps2[dep]: # May already have been removed by the fail case above continue - self.sq_revdeps2[dep].remove(task) - if len(self.sq_revdeps2[dep]) == 0: - self.runq_buildable.add(dep) + self.sqdata.sq_revdeps2[dep].remove(task) + if len(self.sqdata.sq_revdeps2[dep]) == 0: + self.sq_buildable.add(dep) - def task_completeoutright(self, task): + next = set([task]) + while next: + new = set() + for t in next: + self.tasks_scenequeue_done.add(t) + # Look down the dependency chain for non-setscene things which this task depends on + # and mark as 'done' + for dep in self.rqdata.runtaskentries[t].depends: + if dep in self.rqdata.runq_setscene_tids or dep in self.tasks_scenequeue_done: + continue + if self.rqdata.runtaskentries[dep].revdeps.issubset(self.tasks_scenequeue_done): + new.add(dep) + #logger.warning(" Down: " + dep) + next = new + + if task in self.sqdata.unskippable: + self.scenequeue_process_unskippable(task) + + if task in self.scenequeue_notcovered: + logger.debug(1, 'Not skipping setscene task %s', task) + self.scenequeue_process_notcovered(task) + elif task in self.scenequeue_covered: + logger.debug(1, 'Queued setscene task %s', task) + self.coveredtopocess.add(task) + + for task in self.coveredtopocess.copy(): + if self.sqdata.sq_covered_tasks[task].issubset(self.tasks_scenequeue_done): + logger.debug(1, 'Processing setscene task %s', task) + covered = self.sqdata.sq_covered_tasks[task] + covered.add(task) + + # If a task is in target_tids and isn't a setscene task, we can't skip it. + cantskip = covered.intersection(self.rqdata.target_tids).difference(self.rqdata.runq_setscene_tids) + for tid in cantskip: + self.tasks_notcovered.add(tid) + self.scenequeue_process_notcovered(tid) + covered.difference_update(cantskip) + + # Remove notcovered tasks + covered.difference_update(self.tasks_notcovered) + self.tasks_covered.update(covered) + self.coveredtopocess.remove(task) + for tid in covered: + if len(self.rqdata.runtaskentries[tid].depends) == 0: + self.setbuildable(tid) + + def sq_task_completeoutright(self, task): """ Mark a task as completed Look at the reverse dependencies and mark any task with @@ -2403,9 +2268,10 @@ class RunQueueExecuteScenequeue(RunQueueExecute): logger.debug(1, 'Found task %s which could be accelerated', task) self.scenequeue_covered.add(task) + self.tasks_covered.add(task) self.scenequeue_updatecounters(task) - def check_taskfail(self, task): + def sq_check_taskfail(self, task): if self.rqdata.setscenewhitelist is not None: realtask = task.split('_setscene')[0] (mc, fn, taskname, taskfn) = split_tid_mcfn(realtask) @@ -2414,132 +2280,36 @@ class RunQueueExecuteScenequeue(RunQueueExecute): logger.error('Task %s.%s failed' % (pn, taskname + "_setscene")) self.rq.state = runQueueCleanUp - def task_complete(self, task): - self.stats.taskCompleted() - bb.event.fire(sceneQueueTaskCompleted(task, self.stats, self.rq), self.cfgData) - self.task_completeoutright(task) + def sq_task_complete(self, task): + self.sq_stats.taskCompleted() + bb.event.fire(sceneQueueTaskCompleted(task, self.sq_stats, self.rq), self.cfgData) + self.sq_task_completeoutright(task) - def task_fail(self, task, result): - self.stats.taskFailed() - bb.event.fire(sceneQueueTaskFailed(task, self.stats, result, self), self.cfgData) + def sq_task_fail(self, task, result): + self.sq_stats.taskFailed() + bb.event.fire(sceneQueueTaskFailed(task, self.sq_stats, result, self), self.cfgData) self.scenequeue_notcovered.add(task) + self.tasks_notcovered.add(task) self.scenequeue_updatecounters(task, True) - self.check_taskfail(task) + self.sq_check_taskfail(task) - def task_failoutright(self, task): - self.runq_running.add(task) - self.runq_buildable.add(task) - self.stats.taskSkipped() - self.stats.taskCompleted() + def sq_task_failoutright(self, task): + self.sq_running.add(task) + self.sq_buildable.add(task) + self.sq_stats.taskSkipped() + self.sq_stats.taskCompleted() self.scenequeue_notcovered.add(task) + self.tasks_notcovered.add(task) self.scenequeue_updatecounters(task, True) - def task_skip(self, task): - self.runq_running.add(task) - self.runq_buildable.add(task) - self.task_completeoutright(task) - self.stats.taskSkipped() - self.stats.taskCompleted() - - def execute(self): - """ - Run the tasks in a queue prepared by prepare_runqueue - """ - - self.rq.read_workers() - - task = None - if self.can_start_task(): - # Find the next setscene to run - for nexttask in self.rqdata.runq_setscene_tids: - if nexttask in self.runq_buildable and nexttask not in self.runq_running and self.stamps[nexttask] not in self.build_stamps.values(): - if nexttask in self.unskippable: - logger.debug(2, "Setscene task %s is unskippable" % nexttask) - if nexttask not in self.unskippable and len(self.sq_revdeps[nexttask]) > 0 and self.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.sq_revdeps[nexttask], True): - fn = fn_from_tid(nexttask) - foundtarget = False - - if nexttask in self.rqdata.target_tids: - foundtarget = True - if not foundtarget: - logger.debug(2, "Skipping setscene for task %s" % nexttask) - self.task_skip(nexttask) - self.scenequeue_notneeded.add(nexttask) - return True - if nexttask in self.outrightfail: - self.task_failoutright(nexttask) - return True - task = nexttask - break - if task is not None: - (mc, fn, taskname, taskfn) = split_tid_mcfn(task) - taskname = taskname + "_setscene" - if self.rq.check_stamp_task(task, taskname_from_tid(task), recurse = True, cache=self.stampcache): - logger.debug(2, 'Stamp for underlying task %s is current, so skipping setscene variant', task) - self.task_failoutright(task) - return True - - if self.cooker.configuration.force: - if task in self.rqdata.target_tids: - self.task_failoutright(task) - return True - - if self.rq.check_stamp_task(task, taskname, cache=self.stampcache): - logger.debug(2, 'Setscene stamp current task %s, so skip it and its dependencies', task) - self.task_skip(task) - return True - - startevent = sceneQueueTaskStarted(task, self.stats, self.rq) - bb.event.fire(startevent, self.cfgData) - - taskdepdata = self.build_taskdepdata(task) - - taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn] - taskhash = self.rqdata.get_task_hash(task) - unihash = self.rqdata.get_task_unihash(task) - if 'fakeroot' in taskdep and taskname in taskdep['fakeroot'] and not self.cooker.configuration.dry_run: - if not mc in self.rq.fakeworker: - self.rq.start_fakeworker(self, mc) - self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>") - self.rq.fakeworker[mc].process.stdin.flush() - else: - self.rq.worker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>") - self.rq.worker[mc].process.stdin.flush() - - self.build_stamps[task] = bb.build.stampfile(taskname, self.rqdata.dataCaches[mc], taskfn, noextra=True) - self.build_stamps2.append(self.build_stamps[task]) - self.runq_running.add(task) - self.stats.taskActive() - if self.can_start_task(): - return True - - if self.stats.active > 0: - self.rq.read_workers() - return self.rq.active_fds() - - #for tid in self.sq_revdeps: - # if tid not in self.runq_running: - # buildable = tid in self.runq_buildable - # revdeps = self.sq_revdeps[tid] - # bb.warn("Found we didn't run %s %s %s" % (tid, buildable, str(revdeps))) - - self.rq.scenequeue_covered = self.scenequeue_covered - self.rq.scenequeue_notcovered = self.scenequeue_notcovered - - logger.debug(1, 'We can skip tasks %s', "\n".join(sorted(self.rq.scenequeue_covered))) - - self.rq.state = runQueueRunInit - - completeevent = sceneQueueComplete(self.stats, self.rq) - bb.event.fire(completeevent, self.cfgData) + def sq_task_skip(self, task): + self.sq_running.add(task) + self.sq_buildable.add(task) + self.sq_task_completeoutright(task) + self.sq_stats.taskSkipped() + self.sq_stats.taskCompleted() - return True - - def runqueue_process_waitpid(self, task, status): - RunQueueExecute.runqueue_process_waitpid(self, task, status) - - - def build_taskdepdata(self, task): + def sq_build_taskdepdata(self, task): def getsetscenedeps(tid): deps = set() (mc, fn, taskname, _) = split_tid_mcfn(tid) @@ -2577,6 +2347,268 @@ class RunQueueExecuteScenequeue(RunQueueExecute): #bb.note("Task %s: " % task + str(taskdepdata).replace("], ", "],\n")) return taskdepdata + def check_setscenewhitelist(self, tid): + # Check task that is going to run against the whitelist + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) + # Ignore covered tasks + if tid in self.tasks_covered: + return False + # Ignore stamped tasks + if self.rq.check_stamp_task(tid, taskname, cache=self.stampcache): + return False + # Ignore noexec tasks + taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn] + if 'noexec' in taskdep and taskname in taskdep['noexec']: + return False + + pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn] + if not check_setscene_enforce_whitelist(pn, taskname, self.rqdata.setscenewhitelist): + if tid in self.rqdata.runq_setscene_tids: + msg = 'Task %s.%s attempted to execute unexpectedly and should have been setscened' % (pn, taskname) + else: + msg = 'Task %s.%s attempted to execute unexpectedly' % (pn, taskname) + logger.error(msg + '\nThis is usually due to missing setscene tasks. Those missing in this build were: %s' % pprint.pformat(self.scenequeue_notcovered)) + return True + return False + +class SQData(object): + def __init__(self): + # SceneQueue dependencies + self.sq_deps = {} + # SceneQueue reverse dependencies + self.sq_revdeps = {} + # Copy of reverse dependencies used by sq processing code + self.sq_revdeps2 = {} + # Injected inter-setscene task dependencies + self.sq_harddeps = {} + # Cache of stamp files so duplicates can't run in parallel + self.stamps = {} + # Setscene tasks directly depended upon by the build + self.unskippable = set() + # List of setscene tasks which aren't present + self.outrightfail = set() + # A list of normal tasks a setscene task covers + self.sq_covered_tasks = {} + +def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq): + + sq_revdeps = {} + sq_revdeps_squash = {} + sq_collated_deps = {} + + # We need to construct a dependency graph for the setscene functions. Intermediate + # dependencies between the setscene tasks only complicate the code. This code + # therefore aims to collapse the huge runqueue dependency tree into a smaller one + # only containing the setscene functions. + + rqdata.init_progress_reporter.next_stage() + + # First process the chains up to the first setscene task. + endpoints = {} + for tid in rqdata.runtaskentries: + sq_revdeps[tid] = copy.copy(rqdata.runtaskentries[tid].revdeps) + sq_revdeps_squash[tid] = set() + if (len(sq_revdeps[tid]) == 0) and tid not in rqdata.runq_setscene_tids: + #bb.warn("Added endpoint %s" % (tid)) + endpoints[tid] = set() + + rqdata.init_progress_reporter.next_stage() + + # Secondly process the chains between setscene tasks. + for tid in rqdata.runq_setscene_tids: + sq_collated_deps[tid] = set() + #bb.warn("Added endpoint 2 %s" % (tid)) + for dep in rqdata.runtaskentries[tid].depends: + if tid in sq_revdeps[dep]: + sq_revdeps[dep].remove(tid) + if dep not in endpoints: + endpoints[dep] = set() + #bb.warn(" Added endpoint 3 %s" % (dep)) + endpoints[dep].add(tid) + + rqdata.init_progress_reporter.next_stage() + + def process_endpoints(endpoints): + newendpoints = {} + for point, task in endpoints.items(): + tasks = set() + if task: + tasks |= task + if sq_revdeps_squash[point]: + tasks |= sq_revdeps_squash[point] + if point not in rqdata.runq_setscene_tids: + for t in tasks: + sq_collated_deps[t].add(point) + sq_revdeps_squash[point] = set() + if point in rqdata.runq_setscene_tids: + sq_revdeps_squash[point] = tasks + tasks = set() + continue + for dep in rqdata.runtaskentries[point].depends: + if point in sq_revdeps[dep]: + sq_revdeps[dep].remove(point) + if tasks: + sq_revdeps_squash[dep] |= tasks + if len(sq_revdeps[dep]) == 0 and dep not in rqdata.runq_setscene_tids: + newendpoints[dep] = task + if len(newendpoints) != 0: + process_endpoints(newendpoints) + + process_endpoints(endpoints) + + rqdata.init_progress_reporter.next_stage() + + # Build a list of setscene tasks which are "unskippable" + # These are direct endpoints referenced by the build + # Take the build endpoints (no revdeps) and find the sstate tasks they depend upon + new = True + for tid in rqdata.runtaskentries: + if len(rqdata.runtaskentries[tid].revdeps) == 0: + sqdata.unskippable.add(tid) + while new: + new = False + for tid in sqdata.unskippable.copy(): + if tid in rqdata.runq_setscene_tids: + continue + sqdata.unskippable.remove(tid) + if len(rqdata.runtaskentries[tid].depends) == 0: + # These are tasks which have no setscene tasks in their chain, need to mark as directly buildable + sqrq.tasks_notcovered.add(tid) + sqrq.tasks_scenequeue_done.add(tid) + sqrq.setbuildable(tid) + sqrq.scenequeue_process_unskippable(tid) + sqdata.unskippable |= rqdata.runtaskentries[tid].depends + new = True + + rqdata.init_progress_reporter.next_stage(len(rqdata.runtaskentries)) + + # Sanity check all dependencies could be changed to setscene task references + for taskcounter, tid in enumerate(rqdata.runtaskentries): + if tid in rqdata.runq_setscene_tids: + pass + elif len(sq_revdeps_squash[tid]) != 0: + bb.msg.fatal("RunQueue", "Something went badly wrong during scenequeue generation, aborting. Please report this problem.") + else: + del sq_revdeps_squash[tid] + rqdata.init_progress_reporter.update(taskcounter) + + rqdata.init_progress_reporter.next_stage() + + # Resolve setscene inter-task dependencies + # e.g. do_sometask_setscene[depends] = "targetname:do_someothertask_setscene" + # Note that anything explicitly depended upon will have its reverse dependencies removed to avoid circular dependencies + for tid in rqdata.runq_setscene_tids: + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) + realtid = tid + "_setscene" + idepends = rqdata.taskData[mc].taskentries[realtid].idepends + sqdata.stamps[tid] = bb.build.stampfile(taskname + "_setscene", rqdata.dataCaches[mc], taskfn, noextra=True) + for (depname, idependtask) in idepends: + + if depname not in rqdata.taskData[mc].build_targets: + continue + + depfn = rqdata.taskData[mc].build_targets[depname][0] + if depfn is None: + continue + deptid = depfn + ":" + idependtask.replace("_setscene", "") + if deptid not in rqdata.runtaskentries: + bb.msg.fatal("RunQueue", "Task %s depends upon non-existent task %s:%s" % (realtid, depfn, idependtask)) + + if not deptid in sqdata.sq_harddeps: + sqdata.sq_harddeps[deptid] = set() + sqdata.sq_harddeps[deptid].add(tid) + + sq_revdeps_squash[tid].add(deptid) + # Have to zero this to avoid circular dependencies + sq_revdeps_squash[deptid] = set() + + rqdata.init_progress_reporter.next_stage() + + for task in sqdata.sq_harddeps: + for dep in sqdata.sq_harddeps[task]: + sq_revdeps_squash[dep].add(task) + + rqdata.init_progress_reporter.next_stage() + + #for tid in sq_revdeps_squash: + # data = "" + # for dep in sq_revdeps_squash[tid]: + # data = data + "\n %s" % dep + # bb.warn("Task %s_setscene: is %s " % (tid, data)) + + sqdata.sq_revdeps = sq_revdeps_squash + sqdata.sq_revdeps2 = copy.deepcopy(sqdata.sq_revdeps) + sqdata.sq_covered_tasks = sq_collated_deps + + # Build reverse version of revdeps to populate deps structure + for tid in sqdata.sq_revdeps: + sqdata.sq_deps[tid] = set() + for tid in sqdata.sq_revdeps: + for dep in sqdata.sq_revdeps[tid]: + sqdata.sq_deps[dep].add(tid) + + rqdata.init_progress_reporter.next_stage() + + multiconfigs = set() + for tid in sqdata.sq_revdeps: + multiconfigs.add(mc_from_tid(tid)) + if len(sqdata.sq_revdeps[tid]) == 0: + sqrq.sq_buildable.add(tid) + + rqdata.init_progress_reporter.finish() + + if rq.hashvalidate: + noexec = [] + stamppresent = [] + tocheck = set() + + for tid in sqdata.sq_revdeps: + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid) + + taskdep = rqdata.dataCaches[mc].task_deps[taskfn] + + if 'noexec' in taskdep and taskname in taskdep['noexec']: + noexec.append(tid) + sqrq.sq_task_skip(tid) + bb.build.make_stamp(taskname + "_setscene", rqdata.dataCaches[mc], taskfn) + continue + + if rq.check_stamp_task(tid, taskname + "_setscene", cache=stampcache): + logger.debug(2, 'Setscene stamp current for task %s', tid) + stamppresent.append(tid) + sqrq.sq_task_skip(tid) + continue + + if rq.check_stamp_task(tid, taskname, recurse = True, cache=stampcache): + logger.debug(2, 'Normal stamp current for task %s', tid) + stamppresent.append(tid) + sqrq.sq_task_skip(tid) + continue + + tocheck.add(tid) + + valid = rq.validate_hashes(tocheck, cooker.data, len(stamppresent), False) + + valid_new = stamppresent + for v in valid: + valid_new.append(v) + + hashes = {} + for mc in sorted(multiconfigs): + for tid in sqdata.sq_revdeps: + if mc_from_tid(tid) != mc: + continue + if tid not in valid_new and tid not in noexec and tid not in sqrq.scenequeue_notcovered: + sqdata.outrightfail.add(tid) + + h = pending_hash_index(tid, rqdata) + if h not in hashes: + hashes[h] = tid + else: + sqrq.sq_deferred[tid] = hashes[h] + bb.warn("Deferring %s after %s" % (tid, hashes[h])) + + class TaskFailure(Exception): """ Exception raised when a task in a runqueue fails |