poky: refresh thud: b904775c2b..7c76c5d78b

Update poky to thud HEAD.

Adam Trhon (1):
      icecc-env: don't raise error when icecc not installed

Alexander Kanavin (1):
      openssl10: update to 1.0.2q

Armin Kuster (1):
      perl: add testdepends for ssh

Bruce Ashfield (2):
      linux-yocto/4.18: update to v4.18.26
      linux-yocto/4.18: update to v4.18.27

Changqing Li (1):
      checklayer: generate locked-sigs.inc under builddir

Dan Dedrick (2):
      devtool: remove duplicate overrides
      devtool: improve git repo checks before check_commits logic

Daniel Ammann (1):
      ref-manual: Typo found and fixed.

Douglas Royds (2):
      openssl ptest: Strip build host paths from configdata.pm
      openssl: Strip perl version from installed ptest configdata.pm file

Dustin Bain (1):
      busybox: update to 1.29.3

Jan Kiszka (1):
      oe-git-proxy: Avoid resolving NO_PROXY against local files

Jens Rehsack (1):
      avahi: avoid depending on skipped package

Jonas Bonn (1):
      keymaps: tighten package write dependency

Kai Kang (1):
      selftest/wic: update test case test_qemu

Khem Raj (3):
      openssl10: Fix mutliple include assumptions for bn.h in opensslconf.h
      send-error-report: Use https instead of http protocol
      multilib_header_wrapper.h: Use #pragma once

Leonardo Augusto (1):
      scripts/lib/wic/engine: Fix cp's target path for ext* filesystems

Liu Haitao (1):
      iw: fix parsing of WEP keys

Mingli Yu (1):
      logrotate.py: restore /etc/logrotate.d/wtmp

Otavio Salvador (1):
      linux-firmware: Bump to 710963f revision

Ovidiu Panait (1):
      ghostscript: Fix CVE-2019-6116

Peter Kjellerstedt (1):
      libaio: Extend to native

Richard Purdie (23):
      package: Add pkg_postinst_ontarget to PACKAGEVARS
      oeqa/runtime/ptest: Avoid traceback for tests with no section
      oeqa/utils/logparser: Simplify ptest log parsing code
      oeqa/logparser: Further simplification/clarification
      oeqa/logparser: Reform the ptest results parser
      oeqa/utils/logparser: Add in support for duration, exitcode and logs by section
      oeqa/logparser: Improve results handling
      oeqa/logparser: Various misc cleanups
      oeqa/runtime/ptest: Ensure OOM errors are logged
      scripts/contrib/build-perf-test-wrapper.sh: Improve interaction with autobuilder automation
      scripts/contrib/build-perf-test.sh: Remove it
      oe-build-perf-report: Allow branch without hostname
      oe-build-perf-report: Allow commits from different branches
      oe-build-perf-report: Improve branch comparision handling
      oe-build-perf-report: Fix missing buildstats comparisions
      wic/engine: Fix missing parted autobuilder failures
      lib/buildstats: Improve error message
      scripts/oe-git-archive: Separate out functionality to library function
      oe-build-perf-report/gitarchive: Move common useful functions to library
      bitbake: runqueue: Fix dependency loop analysis 'hangs'
      bitbake: runqueue: Filter out multiconfig dependencies from BB_TASKDEPDATA
      bitbake: siggen: Fix multiconfig corner case
      bitbake: cooker: Tweak multiconfig dependency resolution

Robert Yang (5):
      bluez5: Fix a race issue for tools
      yocto-check-layer-wrapper: Fix path for oe-init-build-env
      checklayer: Avoid adding the layer if it is already present
      runqemu: Let qemuparams override default settings
      runqemu: Make QB_MEM easier to set

Ross Burton (3):
      e2fsprogs: fix file system generation with large files
      linux-firmware: recommend split up packages
      linux-firmware: split out liquidio firmware

Scott Rifenbark (2):
      poky.ent: Updated "meta-intel" version to "10.1"
      overview-manual, mega-manual: Updated Package Feeds diagram

Serhey Popovych (1):
      openssl: Skip assembler optimized code for powerpc64 with musl

William Bourque (1):
      wic/engine.py: Load paths from PATH environment variable

Xulin Sun (1):
      openssl: fix multilib file install conflicts

Zheng Ruoqin (1):
      mdadm: add init and service scripts

Change-Id: Ib14c2fb69d25d84aa3d4bf0a6715bba57d1eb900
Signed-off-by: Andrew Geissler <geissonator@yahoo.com>

6 files changed, 352 insertions, 168 deletions

diff --git a/poky/meta/lib/oeqa/runtime/cases/logrotate.py b/poky/meta/lib/oeqa/runtime/cases/logrotate.py
index db6e695ee..d2666444e 100644
--- a/poky/meta/lib/oeqa/runtime/cases/logrotate.py
+++ b/poky/meta/lib/oeqa/runtime/cases/logrotate.py
@@ -9,8 +9,12 @@ from oeqa.runtime.decorator.package import OEHasPackage
 class LogrotateTest(OERuntimeTestCase):
 
     @classmethod
+    def setUpClass(cls):
+        cls.tc.target.run('cp /etc/logrotate.d/wtmp $HOME/wtmp.oeqabak')
+
+    @classmethod
     def tearDownClass(cls):
-        cls.tc.target.run('rm -rf $HOME/logrotate_dir')
+        cls.tc.target.run('mv -f $HOME/wtmp.oeqabak /etc/logrotate.d/wtmp && rm -rf $HOME/logrotate_dir')
 
     @OETestID(1544)
     @OETestDepends(['ssh.SSHTest.test_ssh'])
diff --git a/poky/meta/lib/oeqa/runtime/cases/perl.py b/poky/meta/lib/oeqa/runtime/cases/perl.py
index afeeb180e..be3287f22 100644
--- a/poky/meta/lib/oeqa/runtime/cases/perl.py
+++ b/poky/meta/lib/oeqa/runtime/cases/perl.py
@@ -1,11 +1,13 @@
 import os
 
 from oeqa.runtime.case import OERuntimeTestCase
+from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
 from oeqa.runtime.decorator.package import OEHasPackage
 
 class PerlTest(OERuntimeTestCase):
     @OETestID(208)
+    @OETestDepends(['ssh.SSHTest.test_ssh'])
     @OEHasPackage(['perl'])
     def test_perl_works(self):
         status, output = self.target.run("perl -e '$_=\"Uryyb, jbeyq\"; tr/a-zA-Z/n-za-mN-ZA-M/;print'")
diff --git a/poky/meta/lib/oeqa/runtime/cases/ptest.py b/poky/meta/lib/oeqa/runtime/cases/ptest.py
index 0972a583e..2a28ca59a 100644
--- a/poky/meta/lib/oeqa/runtime/cases/ptest.py
+++ b/poky/meta/lib/oeqa/runtime/cases/ptest.py
@@ -1,55 +1,16 @@
 import unittest
 import pprint
+import datetime
 
 from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
 from oeqa.core.decorator.data import skipIfNotFeature
 from oeqa.runtime.decorator.package import OEHasPackage
-from oeqa.utils.logparser import Lparser, Result
+from oeqa.utils.logparser import PtestParser
 
 class PtestRunnerTest(OERuntimeTestCase):
 
-    # a ptest log parser
-    def parse_ptest(self, logfile):
-        parser = Lparser(test_0_pass_regex="^PASS:(.+)",
-                         test_0_fail_regex="^FAIL:(.+)",
-                         test_0_skip_regex="^SKIP:(.+)",
-                         section_0_begin_regex="^BEGIN: .*/(.+)/ptest",
-                         section_0_end_regex="^END: .*/(.+)/ptest")
-        parser.init()
-        result = Result()
-
-        with open(logfile, errors='replace') as f:
-            for line in f:
-                result_tuple = parser.parse_line(line)
-                if not result_tuple:
-                    continue
-                result_tuple = line_type, category, status, name = parser.parse_line(line)
-
-                if line_type == 'section' and status == 'begin':
-                    current_section = name
-                    continue
-
-                if line_type == 'section' and status == 'end':
-                    current_section = None
-                    continue
-
-                if line_type == 'test' and status == 'pass':
-                    result.store(current_section, name, status)
-                    continue
-
-                if line_type == 'test' and status == 'fail':
-                    result.store(current_section, name, status)
-                    continue
-
-                if line_type == 'test' and status == 'skip':
-                    result.store(current_section, name, status)
-                    continue
-
-        result.sort_tests()
-        return result
-
     @OETestID(1600)
     @skipIfNotFeature('ptest', 'Test requires ptest to be in DISTRO_FEATURES')
     @OETestDepends(['ssh.SSHTest.test_ssh'])
@@ -60,8 +21,6 @@ class PtestRunnerTest(OERuntimeTestCase):
         if status != 0:
             self.skipTest("No -ptest packages are installed in the image")
 
-        import datetime
-
         test_log_dir = self.td.get('TEST_LOG_DIR', '')
         # The TEST_LOG_DIR maybe NULL when testimage is added after
         # testdata.json is generated.
@@ -69,9 +28,9 @@ class PtestRunnerTest(OERuntimeTestCase):
             test_log_dir = os.path.join(self.td.get('WORKDIR', ''), 'testimage')
         # Don't use self.td.get('DATETIME'), it's from testdata.json, not
         # up-to-date, and may cause "File exists" when re-reun.
-        datetime = datetime.datetime.now().strftime('%Y%m%d%H%M%S')
+        timestamp = datetime.datetime.now().strftime('%Y%m%d%H%M%S')
         ptest_log_dir_link = os.path.join(test_log_dir, 'ptest_log')
-        ptest_log_dir = '%s.%s' % (ptest_log_dir_link, datetime)
+        ptest_log_dir = '%s.%s' % (ptest_log_dir_link, timestamp)
         ptest_runner_log = os.path.join(ptest_log_dir, 'ptest-runner.log')
 
         status, output = self.target.run('ptest-runner', 0)
@@ -88,25 +47,36 @@ class PtestRunnerTest(OERuntimeTestCase):
         extras['ptestresult.rawlogs'] = {'log': output}
 
         # Parse and save results
-        parse_result = self.parse_ptest(ptest_runner_log)
-        parse_result.log_as_files(ptest_log_dir, test_status = ['pass','fail', 'skip'])
+        parser = PtestParser()
+        results, sections = parser.parse(ptest_runner_log)
+        parser.results_as_files(ptest_log_dir)
         if os.path.exists(ptest_log_dir_link):
             # Remove the old link to create a new one
             os.remove(ptest_log_dir_link)
         os.symlink(os.path.basename(ptest_log_dir), ptest_log_dir_link)
 
+        extras['ptestresult.sections'] = sections
+
         trans = str.maketrans("()", "__")
-        resmap = {'pass': 'PASSED', 'skip': 'SKIPPED', 'fail': 'FAILED'}
-        for section in parse_result.result_dict:
-            for test, result in parse_result.result_dict[section]:
-                testname = "ptestresult." + section + "." + "_".join(test.translate(trans).split())
-                extras[testname] = {'status': resmap[result]}
+        for section in results:
+            for test in results[section]:
+                result = results[section][test]
+                testname = "ptestresult." + (section or "No-section") + "." + "_".join(test.translate(trans).split())
+                extras[testname] = {'status': result}
 
         failed_tests = {}
-        for section in parse_result.result_dict:
-            failed_testcases = [ "_".join(test.translate(trans).split()) for test, result in parse_result.result_dict[section] if result == 'fail' ]
+        for section in results:
+            failed_testcases = [ "_".join(test.translate(trans).split()) for test in results[section] if results[section][test] == 'fail' ]
             if failed_testcases:
                 failed_tests[section] = failed_testcases
 
+        failmsg = ""
+        status, output = self.target.run('dmesg | grep "Killed process"', 0)
+        if output:
+            failmsg = "ERROR: Processes were killed by the OOM Killer:\n%s\n" % output
+
         if failed_tests:
-            self.fail("Failed ptests:\n%s" % pprint.pformat(failed_tests))
+            failmsg = failmsg + "Failed ptests:\n%s" % pprint.pformat(failed_tests)
+
+        if failmsg:
+            self.fail(failmsg)
diff --git a/poky/meta/lib/oeqa/selftest/cases/wic.py b/poky/meta/lib/oeqa/selftest/cases/wic.py
index 36ee5e5a1..79925f942 100644
--- a/poky/meta/lib/oeqa/selftest/cases/wic.py
+++ b/poky/meta/lib/oeqa/selftest/cases/wic.py
@@ -625,9 +625,11 @@ class Wic2(WicTestCase):
         self.remove_config(config)
 
         with runqemu('wic-image-minimal', ssh=False) as qemu:
-            cmd = "mount |grep '^/dev/' | cut -f1,3 -d ' ' | sort"
+            cmd = "mount | grep '^/dev/' | cut -f1,3 -d ' ' | egrep -c -e '/dev/sda1 /boot' " \
+                  "-e '/dev/root /|/dev/sda2 /' -e '/dev/sda3 /media' -e '/dev/sda4 /mnt'"
             status, output = qemu.run_serial(cmd)
-            self.assertEqual(output, '/dev/root /\r\n/dev/sda1 /boot\r\n/dev/sda3 /media\r\n/dev/sda4 /mnt')
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+            self.assertEqual(output, '4')
             cmd = "grep UUID= /etc/fstab"
             status, output = qemu.run_serial(cmd)
             self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
diff --git a/poky/meta/lib/oeqa/utils/gitarchive.py b/poky/meta/lib/oeqa/utils/gitarchive.py
new file mode 100644
index 000000000..ff614d06b
--- /dev/null
+++ b/poky/meta/lib/oeqa/utils/gitarchive.py
@@ -0,0 +1,244 @@
+#
+# Helper functions for committing data to git and pushing upstream
+#
+# Copyright (c) 2017, Intel Corporation.
+# Copyright (c) 2019, Linux Foundation
+#
+# This program is free software; you can redistribute it and/or modify it
+# under the terms and conditions of the GNU General Public License,
+# version 2, as published by the Free Software Foundation.
+#
+# This program is distributed in the hope it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+# more details.
+#
+
+import os
+import re
+import sys
+from operator import attrgetter
+from collections import namedtuple
+from oeqa.utils.git import GitRepo, GitError
+
+class ArchiveError(Exception):
+    """Internal error handling of this script"""
+
+def format_str(string, fields):
+    """Format string using the given fields (dict)"""
+    try:
+        return string.format(**fields)
+    except KeyError as err:
+        raise ArchiveError("Unable to expand string '{}': unknown field {} "
+                           "(valid fields are: {})".format(
+                               string, err, ', '.join(sorted(fields.keys()))))
+
+
+def init_git_repo(path, no_create, bare, log):
+    """Initialize local Git repository"""
+    path = os.path.abspath(path)
+    if os.path.isfile(path):
+        raise ArchiveError("Invalid Git repo at {}: path exists but is not a "
+                           "directory".format(path))
+    if not os.path.isdir(path) or not os.listdir(path):
+        if no_create:
+            raise ArchiveError("No git repo at {}, refusing to create "
+                               "one".format(path))
+        if not os.path.isdir(path):
+            try:
+                os.mkdir(path)
+            except (FileNotFoundError, PermissionError) as err:
+                raise ArchiveError("Failed to mkdir {}: {}".format(path, err))
+        if not os.listdir(path):
+            log.info("Initializing a new Git repo at %s", path)
+            repo = GitRepo.init(path, bare)
+    try:
+        repo = GitRepo(path, is_topdir=True)
+    except GitError:
+        raise ArchiveError("Non-empty directory that is not a Git repository "
+                           "at {}\nPlease specify an existing Git repository, "
+                           "an empty directory or a non-existing directory "
+                           "path.".format(path))
+    return repo
+
+
+def git_commit_data(repo, data_dir, branch, message, exclude, notes, log):
+    """Commit data into a Git repository"""
+    log.info("Committing data into to branch %s", branch)
+    tmp_index = os.path.join(repo.git_dir, 'index.oe-git-archive')
+    try:
+        # Create new tree object from the data
+        env_update = {'GIT_INDEX_FILE': tmp_index,
+                      'GIT_WORK_TREE': os.path.abspath(data_dir)}
+        repo.run_cmd('add .', env_update)
+
+        # Remove files that are excluded
+        if exclude:
+            repo.run_cmd(['rm', '--cached'] + [f for f in exclude], env_update)
+
+        tree = repo.run_cmd('write-tree', env_update)
+
+        # Create new commit object from the tree
+        parent = repo.rev_parse(branch)
+        git_cmd = ['commit-tree', tree, '-m', message]
+        if parent:
+            git_cmd += ['-p', parent]
+        commit = repo.run_cmd(git_cmd, env_update)
+
+        # Create git notes
+        for ref, filename in notes:
+            ref = ref.format(branch_name=branch)
+            repo.run_cmd(['notes', '--ref', ref, 'add',
+                          '-F', os.path.abspath(filename), commit])
+
+        # Update branch head
+        git_cmd = ['update-ref', 'refs/heads/' + branch, commit]
+        if parent:
+            git_cmd.append(parent)
+        repo.run_cmd(git_cmd)
+
+        # Update current HEAD, if we're on branch 'branch'
+        if not repo.bare and repo.get_current_branch() == branch:
+            log.info("Updating %s HEAD to latest commit", repo.top_dir)
+            repo.run_cmd('reset --hard')
+
+        return commit
+    finally:
+        if os.path.exists(tmp_index):
+            os.unlink(tmp_index)
+
+
+def expand_tag_strings(repo, name_pattern, msg_subj_pattern, msg_body_pattern,
+                       keywords):
+    """Generate tag name and message, with support for running id number"""
+    keyws = keywords.copy()
+    # Tag number is handled specially: if not defined, we autoincrement it
+    if 'tag_number' not in keyws:
+        # Fill in all other fields than 'tag_number'
+        keyws['tag_number'] = '{tag_number}'
+        tag_re = format_str(name_pattern, keyws)
+        # Replace parentheses for proper regex matching
+        tag_re = tag_re.replace('(', '\(').replace(')', '\)') + '$'
+        # Inject regex group pattern for 'tag_number'
+        tag_re = tag_re.format(tag_number='(?P<tag_number>[0-9]{1,5})')
+
+        keyws['tag_number'] = 0
+        for existing_tag in repo.run_cmd('tag').splitlines():
+            match = re.match(tag_re, existing_tag)
+
+            if match and int(match.group('tag_number')) >= keyws['tag_number']:
+                keyws['tag_number'] = int(match.group('tag_number')) + 1
+
+    tag_name = format_str(name_pattern, keyws)
+    msg_subj= format_str(msg_subj_pattern.strip(), keyws)
+    msg_body = format_str(msg_body_pattern, keyws)
+    return tag_name, msg_subj + '\n\n' + msg_body
+
+def gitarchive(data_dir, git_dir, no_create, bare, commit_msg_subject, commit_msg_body, branch_name, no_tag, tagname, tag_msg_subject, tag_msg_body, exclude, notes, push, keywords, log):
+
+    if not os.path.isdir(data_dir):
+        raise ArchiveError("Not a directory: {}".format(data_dir))
+
+    data_repo = init_git_repo(git_dir, no_create, bare, log)
+
+    # Expand strings early in order to avoid getting into inconsistent
+    # state (e.g. no tag even if data was committed)
+    commit_msg = format_str(commit_msg_subject.strip(), keywords)
+    commit_msg += '\n\n' + format_str(commit_msg_body, keywords)
+    branch_name = format_str(branch_name, keywords)
+    tag_name = None
+    if not no_tag and tagname:
+        tag_name, tag_msg = expand_tag_strings(data_repo, tagname,
+                                               tag_msg_subject,
+                                               tag_msg_body, keywords)
+
+    # Commit data
+    commit = git_commit_data(data_repo, data_dir, branch_name,
+                             commit_msg, exclude, notes, log)
+
+    # Create tag
+    if tag_name:
+        log.info("Creating tag %s", tag_name)
+        data_repo.run_cmd(['tag', '-a', '-m', tag_msg, tag_name, commit])
+
+    # Push data to remote
+    if push:
+        cmd = ['push', '--tags']
+        # If no remote is given we push with the default settings from
+        # gitconfig
+        if push is not True:
+            notes_refs = ['refs/notes/' + ref.format(branch_name=branch_name)
+                           for ref, _ in notes]
+            cmd.extend([push, branch_name] + notes_refs)
+        log.info("Pushing data to remote")
+        data_repo.run_cmd(cmd)
+
+# Container class for tester revisions
+TestedRev = namedtuple('TestedRev', 'commit commit_number tags')
+
+def get_test_runs(log, repo, tag_name, **kwargs):
+    """Get a sorted list of test runs, matching given pattern"""
+    # First, get field names from the tag name pattern
+    field_names = [m.group(1) for m in re.finditer(r'{(\w+)}', tag_name)]
+    undef_fields = [f for f in field_names if f not in kwargs.keys()]
+
+    # Fields for formatting tag name pattern
+    str_fields = dict([(f, '*') for f in field_names])
+    str_fields.update(kwargs)
+
+    # Get a list of all matching tags
+    tag_pattern = tag_name.format(**str_fields)
+    tags = repo.run_cmd(['tag', '-l', tag_pattern]).splitlines()
+    log.debug("Found %d tags matching pattern '%s'", len(tags), tag_pattern)
+
+    # Parse undefined fields from tag names
+    str_fields = dict([(f, r'(?P<{}>[\w\-.()]+)'.format(f)) for f in field_names])
+    str_fields['branch'] = r'(?P<branch>[\w\-.()/]+)'
+    str_fields['commit'] = '(?P<commit>[0-9a-f]{7,40})'
+    str_fields['commit_number'] = '(?P<commit_number>[0-9]{1,7})'
+    str_fields['tag_number'] = '(?P<tag_number>[0-9]{1,5})'
+    # escape parenthesis in fields in order to not messa up the regexp
+    fixed_fields = dict([(k, v.replace('(', r'\(').replace(')', r'\)')) for k, v in kwargs.items()])
+    str_fields.update(fixed_fields)
+    tag_re = re.compile(tag_name.format(**str_fields))
+
+    # Parse fields from tags
+    revs = []
+    for tag in tags:
+        m = tag_re.match(tag)
+        groups = m.groupdict()
+        revs.append([groups[f] for f in undef_fields] + [tag])
+
+    # Return field names and a sorted list of revs
+    return undef_fields, sorted(revs)
+
+def get_test_revs(log, repo, tag_name, **kwargs):
+    """Get list of all tested revisions"""
+    fields, runs = get_test_runs(log, repo, tag_name, **kwargs)
+
+    revs = {}
+    commit_i = fields.index('commit')
+    commit_num_i = fields.index('commit_number')
+    for run in runs:
+        commit = run[commit_i]
+        commit_num = run[commit_num_i]
+        tag = run[-1]
+        if not commit in revs:
+            revs[commit] = TestedRev(commit, commit_num, [tag])
+        else:
+            assert commit_num == revs[commit].commit_number, "Commit numbers do not match"
+            revs[commit].tags.append(tag)
+
+    # Return in sorted table
+    revs = sorted(revs.values(), key=attrgetter('commit_number'))
+    log.debug("Found %d tested revisions:\n    %s", len(revs),
+              "\n    ".join(['{} ({})'.format(rev.commit_number, rev.commit) for rev in revs]))
+    return revs
+
+def rev_find(revs, attr, val):
+    """Search from a list of TestedRev"""
+    for i, rev in enumerate(revs):
+        if getattr(rev, attr) == val:
+            return i
+    raise ValueError("Unable to find '{}' value '{}'".format(attr, val))
+
diff --git a/poky/meta/lib/oeqa/utils/logparser.py b/poky/meta/lib/oeqa/utils/logparser.py
index 0670627c3..32fde14a7 100644
--- a/poky/meta/lib/oeqa/utils/logparser.py
+++ b/poky/meta/lib/oeqa/utils/logparser.py
@@ -3,124 +3,86 @@
 import sys
 import os
 import re
-from . import ftools
-
 
 # A parser that can be used to identify weather a line is a test result or a section statement.
-class Lparser(object):
-
-    def __init__(self, test_0_pass_regex, test_0_fail_regex, test_0_skip_regex, section_0_begin_regex=None, section_0_end_regex=None, **kwargs):
-        # Initialize the arguments dictionary
-        if kwargs:
-            self.args = kwargs
-        else:
-            self.args = {}
-
-        # Add the default args to the dictionary
-        self.args['test_0_pass_regex'] = test_0_pass_regex
-        self.args['test_0_fail_regex'] = test_0_fail_regex
-        self.args['test_0_skip_regex'] = test_0_skip_regex
-        if section_0_begin_regex:
-            self.args['section_0_begin_regex'] = section_0_begin_regex
-        if section_0_end_regex:
-            self.args['section_0_end_regex'] = section_0_end_regex
-
-        self.test_possible_status = ['pass', 'fail', 'error', 'skip']
-        self.section_possible_status = ['begin', 'end']
-
-        self.initialized = False
-
-
-    # Initialize the parser with the current configuration
-    def init(self):
-
-        # extra arguments can be added by the user to define new test and section categories. They must follow a pre-defined pattern: <type>_<category_name>_<status>_regex
-        self.test_argument_pattern = "^test_(.+?)_(%s)_regex" % '|'.join(map(str, self.test_possible_status))
-        self.section_argument_pattern = "^section_(.+?)_(%s)_regex" % '|'.join(map(str, self.section_possible_status))
-
-        # Initialize the test and section regex dictionaries
-        self.test_regex = {}
-        self.section_regex ={}
-
-        for arg, value in self.args.items():
-            if not value:
-                raise Exception('The value of provided argument %s is %s. Should have a valid value.' % (key, value))
-            is_test =  re.search(self.test_argument_pattern, arg)
-            is_section = re.search(self.section_argument_pattern, arg)
-            if is_test:
-                if not is_test.group(1) in self.test_regex:
-                    self.test_regex[is_test.group(1)] = {}
-                self.test_regex[is_test.group(1)][is_test.group(2)] = re.compile(value)
-            elif is_section:
-                if not is_section.group(1) in self.section_regex:
-                    self.section_regex[is_section.group(1)] = {}
-                self.section_regex[is_section.group(1)][is_section.group(2)] = re.compile(value)
-            else:
-                # TODO: Make these call a traceback instead of a simple exception..
-                raise Exception("The provided argument name does not correspond to any valid type. Please give one of the following types:\nfor tests: %s\nfor sections: %s" % (self.test_argument_pattern, self.section_argument_pattern))
-
-        self.initialized = True
-
-    # Parse a line and return a tuple containing the type of result (test/section) and its category, status and name
-    def parse_line(self, line):
-        if not self.initialized:
-            raise Exception("The parser is not initialized..")
-
-        for test_category, test_status_list in self.test_regex.items():
-            for test_status, status_regex in test_status_list.items():
-                test_name = status_regex.search(line)
-                if test_name:
-                    return ['test', test_category, test_status, test_name.group(1)]
-
-        for section_category, section_status_list in self.section_regex.items():
-            for section_status, status_regex in section_status_list.items():
-                section_name = status_regex.search(line)
-                if section_name:
-                    return ['section', section_category, section_status, section_name.group(1)]
-        return None
-
-
-class Result(object):
-
+class PtestParser(object):
     def __init__(self):
-        self.result_dict = {}
-
-    def store(self, section, test, status):
-        if not section in self.result_dict:
-            self.result_dict[section] = []
-
-        self.result_dict[section].append((test, status))
-
-    # sort tests by the test name(the first element of the tuple), for each section. This can be helpful when using git to diff for changes by making sure they are always in the same order.
-    def sort_tests(self):
-        for package in self.result_dict:
-            sorted_results = sorted(self.result_dict[package], key=lambda tup: tup[0])
-            self.result_dict[package] = sorted_results
+        self.results = {}
+        self.sections = {}
+
+    def parse(self, logfile):
+        test_regex = {}
+        test_regex['PASSED'] = re.compile(r"^PASS:(.+)")
+        test_regex['FAILED'] = re.compile(r"^FAIL:(.+)")
+        test_regex['SKIPPED'] = re.compile(r"^SKIP:(.+)")
+
+        section_regex = {}
+        section_regex['begin'] = re.compile(r"^BEGIN: .*/(.+)/ptest")
+        section_regex['end'] = re.compile(r"^END: .*/(.+)/ptest")
+        section_regex['duration'] = re.compile(r"^DURATION: (.+)")
+        section_regex['exitcode'] = re.compile(r"^ERROR: Exit status is (.+)")
+        section_regex['timeout'] = re.compile(r"^TIMEOUT: .*/(.+)/ptest")
+
+        def newsection():
+            return { 'name': "No-section", 'log': "" }
+
+        current_section = newsection()
+
+        with open(logfile, errors='replace') as f:
+            for line in f:
+                result = section_regex['begin'].search(line)
+                if result:
+                    current_section['name'] = result.group(1)
+                    continue
+
+                result = section_regex['end'].search(line)
+                if result:
+                    if current_section['name'] != result.group(1):
+                        bb.warn("Ptest END log section mismatch %s vs. %s" % (current_section['name'], result.group(1)))
+                    if current_section['name'] in self.sections:
+                        bb.warn("Ptest duplicate section for %s" % (current_section['name']))
+                    self.sections[current_section['name']] = current_section
+                    del self.sections[current_section['name']]['name']
+                    current_section = newsection()
+                    continue
+
+                result = section_regex['timeout'].search(line)
+                if result:
+                    if current_section['name'] != result.group(1):
+                        bb.warn("Ptest TIMEOUT log section mismatch %s vs. %s" % (current_section['name'], result.group(1)))
+                    current_section['timeout'] = True
+                    continue
+
+                for t in ['duration', 'exitcode']:
+                    result = section_regex[t].search(line)
+                    if result:
+                        current_section[t] = result.group(1)
+                        continue
+
+                current_section['log'] = current_section['log'] + line 
+
+                for t in test_regex:
+                    result = test_regex[t].search(line)
+                    if result:
+                        if current_section['name'] not in self.results:
+                            self.results[current_section['name']] = {}
+                        self.results[current_section['name']][result.group(1)] = t
+
+        return self.results, self.sections
 
     # Log the results as files. The file name is the section name and the contents are the tests in that section.
-    def log_as_files(self, target_dir, test_status):
-        status_regex = re.compile('|'.join(map(str, test_status)))
-        if not type(test_status) == type([]):
-            raise Exception("test_status should be a list. Got " + str(test_status) + " instead.")
+    def results_as_files(self, target_dir):
         if not os.path.exists(target_dir):
             raise Exception("Target directory does not exist: %s" % target_dir)
 
-        for section, test_results in self.result_dict.items():
-            prefix = ''
-            for x in test_status:
-                prefix +=x+'.'
+        for section in self.results:
+            prefix = 'No-section'
             if section:
-                prefix += section
+                prefix = section
             section_file = os.path.join(target_dir, prefix)
             # purge the file contents if it exists
-            open(section_file, 'w').close()
-            for test_result in test_results:
-                (test_name, status) = test_result
-                # we log only the tests with status in the test_status list
-                match_status = status_regex.search(status)
-                if match_status:
-                    ftools.append_file(section_file, status + ": " + test_name)
-
-    # Not yet implemented!
-    def log_to_lava(self):
-        pass
+            with open(section_file, 'w') as f:
+                for test_name in sorted(self.results[section]):
+                    status = self.results[section][test_name]
+                    f.write(status + ": " + test_name + "\n")
+