poky: refresh thud: b904775c2b..7c76c5d78b

Update poky to thud HEAD.

Adam Trhon (1):
      icecc-env: don't raise error when icecc not installed

Alexander Kanavin (1):
      openssl10: update to 1.0.2q

Armin Kuster (1):
      perl: add testdepends for ssh

Bruce Ashfield (2):
      linux-yocto/4.18: update to v4.18.26
      linux-yocto/4.18: update to v4.18.27

Changqing Li (1):
      checklayer: generate locked-sigs.inc under builddir

Dan Dedrick (2):
      devtool: remove duplicate overrides
      devtool: improve git repo checks before check_commits logic

Daniel Ammann (1):
      ref-manual: Typo found and fixed.

Douglas Royds (2):
      openssl ptest: Strip build host paths from configdata.pm
      openssl: Strip perl version from installed ptest configdata.pm file

Dustin Bain (1):
      busybox: update to 1.29.3

Jan Kiszka (1):
      oe-git-proxy: Avoid resolving NO_PROXY against local files

Jens Rehsack (1):
      avahi: avoid depending on skipped package

Jonas Bonn (1):
      keymaps: tighten package write dependency

Kai Kang (1):
      selftest/wic: update test case test_qemu

Khem Raj (3):
      openssl10: Fix mutliple include assumptions for bn.h in opensslconf.h
      send-error-report: Use https instead of http protocol
      multilib_header_wrapper.h: Use #pragma once

Leonardo Augusto (1):
      scripts/lib/wic/engine: Fix cp's target path for ext* filesystems

Liu Haitao (1):
      iw: fix parsing of WEP keys

Mingli Yu (1):
      logrotate.py: restore /etc/logrotate.d/wtmp

Otavio Salvador (1):
      linux-firmware: Bump to 710963f revision

Ovidiu Panait (1):
      ghostscript: Fix CVE-2019-6116

Peter Kjellerstedt (1):
      libaio: Extend to native

Richard Purdie (23):
      package: Add pkg_postinst_ontarget to PACKAGEVARS
      oeqa/runtime/ptest: Avoid traceback for tests with no section
      oeqa/utils/logparser: Simplify ptest log parsing code
      oeqa/logparser: Further simplification/clarification
      oeqa/logparser: Reform the ptest results parser
      oeqa/utils/logparser: Add in support for duration, exitcode and logs by section
      oeqa/logparser: Improve results handling
      oeqa/logparser: Various misc cleanups
      oeqa/runtime/ptest: Ensure OOM errors are logged
      scripts/contrib/build-perf-test-wrapper.sh: Improve interaction with autobuilder automation
      scripts/contrib/build-perf-test.sh: Remove it
      oe-build-perf-report: Allow branch without hostname
      oe-build-perf-report: Allow commits from different branches
      oe-build-perf-report: Improve branch comparision handling
      oe-build-perf-report: Fix missing buildstats comparisions
      wic/engine: Fix missing parted autobuilder failures
      lib/buildstats: Improve error message
      scripts/oe-git-archive: Separate out functionality to library function
      oe-build-perf-report/gitarchive: Move common useful functions to library
      bitbake: runqueue: Fix dependency loop analysis 'hangs'
      bitbake: runqueue: Filter out multiconfig dependencies from BB_TASKDEPDATA
      bitbake: siggen: Fix multiconfig corner case
      bitbake: cooker: Tweak multiconfig dependency resolution

Robert Yang (5):
      bluez5: Fix a race issue for tools
      yocto-check-layer-wrapper: Fix path for oe-init-build-env
      checklayer: Avoid adding the layer if it is already present
      runqemu: Let qemuparams override default settings
      runqemu: Make QB_MEM easier to set

Ross Burton (3):
      e2fsprogs: fix file system generation with large files
      linux-firmware: recommend split up packages
      linux-firmware: split out liquidio firmware

Scott Rifenbark (2):
      poky.ent: Updated "meta-intel" version to "10.1"
      overview-manual, mega-manual: Updated Package Feeds diagram

Serhey Popovych (1):
      openssl: Skip assembler optimized code for powerpc64 with musl

William Bourque (1):
      wic/engine.py: Load paths from PATH environment variable

Xulin Sun (1):
      openssl: fix multilib file install conflicts

Zheng Ruoqin (1):
      mdadm: add init and service scripts

Change-Id: Ib14c2fb69d25d84aa3d4bf0a6715bba57d1eb900
Signed-off-by: Andrew Geissler <geissonator@yahoo.com>

1 files changed, 37 insertions, 87 deletions

diff --git a/poky/scripts/oe-build-perf-report b/poky/scripts/oe-build-perf-report
index 0bd05f44ef..f6fb458c2e 100755
--- a/poky/scripts/oe-build-perf-report
+++ b/poky/scripts/oe-build-perf-report
@@ -37,58 +37,18 @@ from buildstats import BuildStats, diff_buildstats, BSVerDiff
 scriptpath.add_oe_lib_path()
 
 from oeqa.utils.git import GitRepo, GitError
+import oeqa.utils.gitarchive as gitarchive
 
 
 # Setup logging
 logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
 log = logging.getLogger('oe-build-perf-report')
 
-
-# Container class for tester revisions
-TestedRev = namedtuple('TestedRev', 'commit commit_number tags')
-
-
-def get_test_runs(repo, tag_name, **kwargs):
-    """Get a sorted list of test runs, matching given pattern"""
-    # First, get field names from the tag name pattern
-    field_names = [m.group(1) for m in re.finditer(r'{(\w+)}', tag_name)]
-    undef_fields = [f for f in field_names if f not in kwargs.keys()]
-
-    # Fields for formatting tag name pattern
-    str_fields = dict([(f, '*') for f in field_names])
-    str_fields.update(kwargs)
-
-    # Get a list of all matching tags
-    tag_pattern = tag_name.format(**str_fields)
-    tags = repo.run_cmd(['tag', '-l', tag_pattern]).splitlines()
-    log.debug("Found %d tags matching pattern '%s'", len(tags), tag_pattern)
-
-    # Parse undefined fields from tag names
-    str_fields = dict([(f, r'(?P<{}>[\w\-.()]+)'.format(f)) for f in field_names])
-    str_fields['branch'] = r'(?P<branch>[\w\-.()/]+)'
-    str_fields['commit'] = '(?P<commit>[0-9a-f]{7,40})'
-    str_fields['commit_number'] = '(?P<commit_number>[0-9]{1,7})'
-    str_fields['tag_number'] = '(?P<tag_number>[0-9]{1,5})'
-    # escape parenthesis in fields in order to not messa up the regexp
-    fixed_fields = dict([(k, v.replace('(', r'\(').replace(')', r'\)')) for k, v in kwargs.items()])
-    str_fields.update(fixed_fields)
-    tag_re = re.compile(tag_name.format(**str_fields))
-
-    # Parse fields from tags
-    revs = []
-    for tag in tags:
-        m = tag_re.match(tag)
-        groups = m.groupdict()
-        revs.append([groups[f] for f in undef_fields] + [tag])
-
-    # Return field names and a sorted list of revs
-    return undef_fields, sorted(revs)
-
 def list_test_revs(repo, tag_name, verbosity, **kwargs):
     """Get list of all tested revisions"""
     valid_kwargs = dict([(k, v) for k, v in kwargs.items() if v is not None])
 
-    fields, revs = get_test_runs(repo, tag_name, **valid_kwargs)
+    fields, revs = gitarchive.get_test_runs(log, repo, tag_name, **valid_kwargs)
     ignore_fields = ['tag_number']
     if verbosity < 2:
         extra_fields = ['COMMITS', 'TEST RUNS']
@@ -133,36 +93,6 @@ def list_test_revs(repo, tag_name, verbosity, **kwargs):
 
     print_table(rows)
 
-def get_test_revs(repo, tag_name, **kwargs):
-    """Get list of all tested revisions"""
-    fields, runs = get_test_runs(repo, tag_name, **kwargs)
-
-    revs = {}
-    commit_i = fields.index('commit')
-    commit_num_i = fields.index('commit_number')
-    for run in runs:
-        commit = run[commit_i]
-        commit_num = run[commit_num_i]
-        tag = run[-1]
-        if not commit in revs:
-            revs[commit] = TestedRev(commit, commit_num, [tag])
-        else:
-            assert commit_num == revs[commit].commit_number, "Commit numbers do not match"
-            revs[commit].tags.append(tag)
-
-    # Return in sorted table
-    revs = sorted(revs.values(), key=attrgetter('commit_number'))
-    log.debug("Found %d tested revisions:\n    %s", len(revs),
-              "\n    ".join(['{} ({})'.format(rev.commit_number, rev.commit) for rev in revs]))
-    return revs
-
-def rev_find(revs, attr, val):
-    """Search from a list of TestedRev"""
-    for i, rev in enumerate(revs):
-        if getattr(rev, attr) == val:
-            return i
-    raise ValueError("Unable to find '{}' value '{}'".format(attr, val))
-
 def is_xml_format(repo, commit):
     """Check if the commit contains xml (or json) data"""
     if repo.rev_parse(commit + ':results.xml'):
@@ -427,8 +357,8 @@ def print_html_report(data, id_comp, buildstats):
 
             # Compare buildstats
             bs_key = test + '.' + meas
-            rev = metadata['commit_num']['value']
-            comp_rev = metadata['commit_num']['value_old']
+            rev = str(metadata['commit_num']['value'])
+            comp_rev = str(metadata['commit_num']['value_old'])
             if (rev in buildstats and bs_key in buildstats[rev] and
                     comp_rev in buildstats and bs_key in buildstats[comp_rev]):
                 new_meas['buildstats'] = BSSummary(buildstats[comp_rev][bs_key],
@@ -512,10 +442,10 @@ def auto_args(repo, args):
 
         key = split[0]
         val = split[1].strip()
-        if key == 'hostname':
+        if key == 'hostname' and not args.hostname:
             log.debug("Using hostname %s", val)
             args.hostname = val
-        elif key == 'branch':
+        elif key == 'branch' and not args.branch:
             log.debug("Using branch %s", val)
             args.branch = val
 
@@ -541,7 +471,8 @@ Examine build performance test results from a Git repository"""
                        default='{hostname}/{branch}/{machine}/{commit_number}-g{commit}/{tag_number}',
                        help="Tag name (pattern) for finding results")
     group.add_argument('--hostname', '-H')
-    group.add_argument('--branch', '-B', default='master')
+    group.add_argument('--branch', '-B', default='master', help="Branch to find commit in")
+    group.add_argument('--branch2', help="Branch to find comparision revisions in")
     group.add_argument('--machine', default='qemux86')
     group.add_argument('--history-length', default=25, type=int,
                        help="Number of tested revisions to plot in html report")
@@ -577,32 +508,51 @@ def main(argv=None):
     if not args.hostname:
         auto_args(repo, args)
 
-    revs = get_test_revs(repo, args.tag_name, hostname=args.hostname,
-                         branch=args.branch, machine=args.machine)
-    if len(revs) < 2:
-        log.error("%d tester revisions found, unable to generate report",
-                  len(revs))
-        return 1
+    revs = gitarchive.get_test_revs(log, repo, args.tag_name, hostname=args.hostname,
+                                    branch=args.branch, machine=args.machine)
+    if args.branch2:
+        revs2 = gitarchive.get_test_revs(log, repo, args.tag_name, hostname=args.hostname,
+                                         branch=args.branch2, machine=args.machine)
+        if not len(revs2):
+            log.error("No revisions found to compare against")
+            return 1
+        if not len(revs):
+            log.error("No revision to report on found")
+            return 1
+    else:
+        if len(revs) < 2:
+            log.error("Only %d tester revisions found, unable to generate report" % len(revs))
+            return 1
 
     # Pick revisions
     if args.commit:
         if args.commit_number:
             log.warning("Ignoring --commit-number as --commit was specified")
-        index1 = rev_find(revs, 'commit', args.commit)
+        index1 = gitarchive.rev_find(revs, 'commit', args.commit)
     elif args.commit_number:
-        index1 = rev_find(revs, 'commit_number', args.commit_number)
+        index1 = gitarchive.rev_find(revs, 'commit_number', args.commit_number)
     else:
         index1 = len(revs) - 1
 
+    if args.branch2:
+        revs2.append(revs[index1])
+        index1 = len(revs2) - 1
+        revs = revs2
+
     if args.commit2:
         if args.commit_number2:
             log.warning("Ignoring --commit-number2 as --commit2 was specified")
-        index2 = rev_find(revs, 'commit', args.commit2)
+        index2 = gitarchive.rev_find(revs, 'commit', args.commit2)
     elif args.commit_number2:
-        index2 = rev_find(revs, 'commit_number', args.commit_number2)
+        index2 = gitarchive.rev_find(revs, 'commit_number', args.commit_number2)
     else:
         if index1 > 0:
             index2 = index1 - 1
+            # Find the closest matching commit number for comparision
+            # In future we could check the commit is a common ancestor and
+            # continue back if not but this good enough for now
+            while index2 > 0 and revs[index2].commit_number > revs[index1].commit_number:
+                index2 = index2 - 1
         else:
             log.error("Unable to determine the other commit, use "
                       "--commit2 or --commit-number2 to specify it")