poky: subtree update:81f9e815d3..03d4d9d68f

Adrian Bunk (1):
      json-c: Don't --enable-rdrand

Alessio Igor Bogani (2):
      wic: Using the right rootfs size during prepare_rootfs
      rootfs-postcommands: Avoid use of an hard-coded value

Alexander Kanavin (1):
      binutils: drop UPSTREAM_VERSION_UNKNOWN

Alexandre Bard (1):
      systemd: Expose resolv-conf alternative only when resolved is built

Andre McCurdy (1):
      ffmpeg: enable more verbose build logs

André Draszik (4):
      ruby: drop long-merged CVE patches
      ruby: configure mis-detects isnan/isinf on musl
      ruby: fix non-IPv6 support
      packagegroup: fix a comment regarding PACKAGE_ARCH

Bruce Ashfield (6):
      linux-yocto/5.2: update to v5.2.13
      linux-yocto/4.19: update to v4.19.72
      linux-yocto/5.2: update to v5.2.14
      linux-yocto/5.2: update to v5.2.16
      linux-yocto/5.2: update to v5.2.17
      yocto-bsps: update to v5.2.17

Böszörményi Zoltán via Openembedded-core (1):
      classes/image-live.bbclass: Don't hardcode cpio.gz

Changqing Li (2):
      devtool.py: change to do clean before remove-layer
      devtool.py: fix buildclean test

Chen Qi (1):
      systemd: fix NFS regression

Dan Tran (1):
      unzip: Fix CVE-2019-13232

David Reyna (2):
      bitbake: toaster: issues in import layer when clicking 'add layer'
      bitbake: toaster: improve warnings when adding dependency to packages

Diego Rondini (2):
      initramfs-framework: fix var name
      initramfs-framework: support PARTLABEL option

Douglas Royds (1):
      icecc: Don't use icecc when INHIBIT_DEFAULT_DEPS is set

He Zhe (1):
      ltp: Fix hang of cve test cases

Heiko Schocher (1):
      kernel.fitimage.bbclass: remove ramdisk_ctype

Jacob Kroon (1):
      bitbake: tests/data: Test combinations of _append together with override

Joe Slater (1):
      bash-completion: add image feature

Jonathan Marler (1):
      package: Multiple shlib_providers for the same file should error

Joshua Watt (8):
      classes/reproducible_build: Move SDE deploy to another directory
      oeqa: Test multiconfig parsing
      bitbake: cookerdata: Add mc conffiles hashes to cache hash
      bitbake: hashserve: Add missing import
      bitbake: siggen: Fix attribute error when hashserver fails
      bitbake: hashserv: Don't daemonize server process
      local.conf.sample: Add Hash Equivalence
      classes/reproducible_build: Create SDE destination

Khem Raj (7):
      musl: Fix riscv64 CAS functions
      qemuriscv: Do not blacklist clang anymore
      sdk: Install nativesdk locales for all TCLIBC variants
      strace: Upgrade to 5.3
      packagegroups: All groups are not allarch
      musl: Fix __riscv_mc* containers to match glibc
      core-image-sato-sdk-ptest: Remove valgrind ptests for riscv

Konrad Scherer (1):
      gen-lockedsig-cache: Replace glob lookup with hash to filename lookup

Lei Maohui (1):
      bluez5: update patch to fix do_patch error when PATCHTOOL = "patch".

Li Zhou (1):
      shadow: use relaxed usernames for all

Limeng (1):
      u-boot: add CVE patches for u-boot

Nathan Rossi (2):
      oeqa/core/utils/concurrencytest.py: Handle exceptions and details
      oeqa/core/case.py: Encode binary data of log

Niclas Svensson (1):
      devtool: finish: Keep patches ordered when updating bbappend

Otavio Salvador (1):
      mesa: Add freedreno PACKAGECONFIG option

Peter Kjellerstedt (3):
      systemd: Make it build with hwdb disabled
      devtool: finish: Add suppport for the --no-clean option
      lib/oe/lsb: Make sure the distro ID is always lowercased

Randy MacLeod (1):
      ffmpeg: update from 4.2 to 4.2.1

Richard Purdie (17):
      Revert "meta-extsdk: Either an sstate task is a proper task or it isn't"
      sstatesig: Fix hash equivlanency locked signature issues
      oeqa/selftest/signing: Fix for hash equivlance server
      lib/sstatesig: Fix class inheritance problems
      populate_sdk_ext: Fix for hash equiv
      bitbake: runqueue: Fix task migration problems
      bitbake: siggen: Ensure setscenetasks list is available to worker context
      bitbake: runqueue: Change task migration behaviour for rerunning setscene tasks
      bitbake: siggen/runqueue: Fix signature mismatch issues
      bitbake: siggen: Avoid writing misleading sigdata files
      bitbake: runqueue: Save unihashes more frequently
      bitbake: runqueue: Small performance optimisation
      bitbake: siggen: Remove full path from unitaskhashes keys
      bitbake: tests/runqueue: Fix hashserve shutdown race
      base: Improve module import error message
      sanity.conf: Bump minimum bitbake version
      bitbake: bitbake: Bump verison 1.43.1 -> 1.43.2

Robert Yang (6):
      cases/bbtests.py: test_bitbake_g(): Check base-files rather than busybox
      expect: Fix configure error for nativesdk
      net-tools: Fix installed-vs-shipped for nativesdk
      expect: Fix buffer overflow error when build in long path
      apr: Check for libtoolize rather than libtool
      lttng-ust: Fix for --enable-python-agent

Ross Burton (12):
      oeqa/selftest/reproducible: test ipkgs too
      distcc: clean up the UI install logic
      distcc: use --enable-tcp-insecure instead of --make-me-a-botnet
      distcc: split into client and server packages
      json-c: clean up recipe
      json-c: use GitHub for upstream release checking
      bitbake: fetch2/git: refactor check for git-lfs command
      bitbake: tests/fetch: add test case for git-lfs handling
      python3: move runpy to core
      pango: fix the failing testiter test case
      opkg: remove redundant systemd inherit
      lttng-ust: update patch Signed-off-by

Trevor Gamblin (5):
      python3-subunit: ensure runtime dependencies are present
      python3-pip: ensure pickle is installed
      lighttpd: remove fam as a PACKAGECONFIG option
      tiff: fix CVE-2019-14973
      opkg: remove pathfinder PACKAGECONFIG option

Wang Quanyang (1):
      kexec-tools: fix arm kexec failure for __NR_kexec_file_load

Yi Zhao (1):
      python: add tk-lib as runtime dependency for python-tkinter

Change-Id: I0570125d49f7e4bc3bbf70508cbfd7e10bdbc032
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>

4 files changed, 70 insertions, 17 deletions

diff --git a/poky/scripts/gen-lockedsig-cache b/poky/scripts/gen-lockedsig-cache
index e3076e11a..48cb67112 100755
--- a/poky/scripts/gen-lockedsig-cache
+++ b/poky/scripts/gen-lockedsig-cache
@@ -5,9 +5,9 @@
 
 import os
 import sys
-import glob
 import shutil
 import errno
+import time
 
 def mkdir(d):
     try:
@@ -16,6 +16,36 @@ def mkdir(d):
         if e.errno != errno.EEXIST:
             raise e
 
+# extract the hash from past the last colon to last underscore
+def extract_sha(filename):
+    return filename.split(':')[7].split('_')[0]
+
+# get all files in a directory, extract hash and make
+# a map from hash to list of file with that hash
+def map_sha_to_files(dir_, prefix, sha_map):
+    sstate_prefix_path = dir_ + '/' + prefix + '/'
+    sstate_files = os.listdir(sstate_prefix_path)
+    for f in sstate_files:
+        try:
+            sha = extract_sha(f)
+            if sha not in sha_map:
+                sha_map[sha] = []
+            sha_map[sha].append(sstate_prefix_path + f)
+        except IndexError:
+            continue
+
+# given a prefix build a map of hash to list of files
+def build_sha_cache(prefix):
+    sha_map = {}
+
+    sstate_dir = sys.argv[2]
+    map_sha_to_files(sstate_dir, prefix, sha_map)
+
+    native_sstate_dir = sys.argv[2] + '/' + sys.argv[4]
+    map_sha_to_files(native_sstate_dir, prefix, sha_map)
+
+    return sha_map
+
 if len(sys.argv) < 5:
     print("Incorrect number of arguments specified")
     print("syntax: gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir> <nativelsbstring> [filterfile]")
@@ -41,12 +71,19 @@ with open(sys.argv[1]) as f:
                 sigs.append(sig)
 
 print('Gathering file list')
+start_time = time.perf_counter()
 files = set()
+sstate_content_cache = {}
 for s in sigs:
-    p = sys.argv[2] + "/" + s[:2] + "/*" + s + "*"
-    files |= set(glob.glob(p))
-    p = sys.argv[2] + "/%s/" % sys.argv[4] + s[:2] + "/*" + s + "*"
-    files |= set(glob.glob(p))
+    prefix = s[:2]
+    if prefix not in sstate_content_cache:
+        sstate_content_cache[prefix] = build_sha_cache(prefix)
+
+    for f in sstate_content_cache[prefix][s]:
+        files.add(f)
+
+elapsed = time.perf_counter() - start_time
+print("Gathering file list took %.1fs" % elapsed)
 
 print('Processing files')
 for f in files:
diff --git a/poky/scripts/lib/devtool/standard.py b/poky/scripts/lib/devtool/standard.py
index 9eeaefb79..60c9a046f 100644
--- a/poky/scripts/lib/devtool/standard.py
+++ b/poky/scripts/lib/devtool/standard.py
@@ -1619,17 +1619,17 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
                                           patches_dir, changed_revs)
         logger.debug('Pre-filtering: update: %s, new: %s' % (dict(upd_p), dict(new_p)))
         if filter_patches:
-            new_p = {}
-            upd_p = {k:v for k,v in upd_p.items() if k in filter_patches}
+            new_p = OrderedDict()
+            upd_p = OrderedDict((k,v) for k,v in upd_p.items() if k in filter_patches)
             remove_files = [f for f in remove_files if f in filter_patches]
         updatefiles = False
         updaterecipe = False
         destpath = None
         srcuri = (rd.getVar('SRC_URI', False) or '').split()
         if appendlayerdir:
-            files = dict((os.path.join(local_files_dir, key), val) for
+            files = OrderedDict((os.path.join(local_files_dir, key), val) for
                          key, val in list(upd_f.items()) + list(new_f.items()))
-            files.update(dict((os.path.join(patches_dir, key), val) for
+            files.update(OrderedDict((os.path.join(patches_dir, key), val) for
                               key, val in list(upd_p.items()) + list(new_p.items())))
             if files or remove_files:
                 removevalues = None
@@ -2008,7 +2008,7 @@ def finish(args, config, basepath, workspace):
         else:
             raise DevtoolError('Source tree is not clean:\n\n%s\nEnsure you have committed your changes or use -f/--force if you are sure there\'s nothing that needs to be committed' % dirty)
 
-    no_clean = False
+    no_clean = args.no_clean
     tinfoil = setup_tinfoil(basepath=basepath, tracking=True)
     try:
         rd = parse_recipe(config, tinfoil, args.recipename, True)
@@ -2282,6 +2282,7 @@ def register_commands(subparsers, context):
     parser_finish.add_argument('--mode', '-m', choices=['patch', 'srcrev', 'auto'], default='auto', help='Update mode (where %(metavar)s is %(choices)s; default is %(default)s)', metavar='MODE')
     parser_finish.add_argument('--initial-rev', help='Override starting revision for patches')
     parser_finish.add_argument('--force', '-f', action="store_true", help='Force continuing even if there are uncommitted changes in the source tree repository')
+    parser_finish.add_argument('--no-clean', '-n', action="store_true", help='Don\'t clean the sysroot to remove recipe output')
     parser_finish.add_argument('--no-overrides', '-O', action="store_true", help='Do not handle other override branches (if they exist)')
     parser_finish.add_argument('--dry-run', '-N', action="store_true", help='Dry-run (just report changes instead of writing them)')
     parser_finish.add_argument('--force-patch-refresh', action="store_true", help='Update patches in the layer even if they have not been modified (useful for refreshing patch context)')
diff --git a/poky/scripts/lib/resulttool/resultutils.py b/poky/scripts/lib/resulttool/resultutils.py
index 177fb25f9..7cb85a6aa 100644
--- a/poky/scripts/lib/resulttool/resultutils.py
+++ b/poky/scripts/lib/resulttool/resultutils.py
@@ -126,7 +126,11 @@ def decode_log(logdata):
         if "compressed" in logdata:
             data = logdata.get("compressed")
             data = base64.b64decode(data.encode("utf-8"))
-            return zlib.decompress(data).decode("utf-8")
+            data = zlib.decompress(data)
+            try:
+                return data.decode("utf-8")
+            except UnicodeDecodeError:
+                return data
     return None
 
 def ptestresult_get_log(results, section):
diff --git a/poky/scripts/lib/wic/partition.py b/poky/scripts/lib/wic/partition.py
index 2a71d7b1d..d809408e1 100644
--- a/poky/scripts/lib/wic/partition.py
+++ b/poky/scripts/lib/wic/partition.py
@@ -212,13 +212,24 @@ class Partition():
         if os.path.isfile(rootfs):
             os.remove(rootfs)
 
-        # If size is not specified compute it from the rootfs_dir size
         if not self.size and real_rootfs:
-            # Use the same logic found in get_rootfs_size()
-            # from meta/classes/image.bbclass
-            du_cmd = "du -ks %s" % rootfs_dir
-            out = exec_cmd(du_cmd)
-            self.size = int(out.split()[0])
+            # The rootfs size is not set in .ks file so try to get it
+            # from bitbake variable
+            rsize_bb = get_bitbake_var('ROOTFS_SIZE')
+            rdir = get_bitbake_var('IMAGE_ROOTFS')
+            if rsize_bb and rdir == rootfs_dir:
+                # Bitbake variable ROOTFS_SIZE is calculated in
+                # Image._get_rootfs_size method from meta/lib/oe/image.py
+                # using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT,
+                # IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE
+                self.size = int(round(float(rsize_bb)))
+            else:
+                # Bitbake variable ROOTFS_SIZE is not defined so compute it
+                # from the rootfs_dir size using the same logic found in
+                # get_rootfs_size() from meta/classes/image.bbclass
+                du_cmd = "du -ks %s" % rootfs_dir
+                out = exec_cmd(du_cmd)
+                self.size = int(out.split()[0])
 
         prefix = "ext" if self.fstype.startswith("ext") else self.fstype
         method = getattr(self, "prepare_rootfs_" + prefix)