summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorDylan Baker <dylan@pnwbakers.com>2016-05-09 21:42:21 (GMT)
committerDylan Baker <dylan@pnwbakers.com>2016-05-16 22:13:28 (GMT)
commit4515f3eeae79391eba4550d6825eba21d7256aa5 (patch)
tree10a58f0c761d458fbe4f05f4d65f254dbe2b488d
parent62857f092b7a0387bac1572b5193aa0f13c1226e (diff)
downloadpiglit-4515f3eeae79391eba4550d6825eba21d7256aa5.tar.gz
piglit-4515f3eeae79391eba4550d6825eba21d7256aa5.tar.xz
framework/results: Fix subtests masking crashes
In piglit when a test generates subtests we treat the "worst" subtest as the test status, and in most cases this works as expected. There is at least one case where this is not correct, and that is the case of crash. There are a couple of reasons that crash should not be masked. One is that it is generated by the framework when the test binary hits an assert or segfaults, or any number of similar cases. The second is that it may mean that all of the subtests did not run, as such we don't want the status to be masked by the "worst" subtest which would be fail at worst. This is maybe not the strictly correct way to handle this. Probably the better solution is to include the overall result in the max call if the overall result is not NOTRUN. But this fixes the problem and that can be done later. Signed-off-by: Dylan Baker <dylanx.c.baker@intel.com> Reviewed-by: Brian Paul <brianp@vmware.com> Tested-by: Brian Paul <brianp@vmware.com>
-rw-r--r--framework/results.py7
1 files changed, 5 insertions, 2 deletions
diff --git a/framework/results.py b/framework/results.py
index 469abeb..2095d90 100644
--- a/framework/results.py
+++ b/framework/results.py
@@ -173,10 +173,13 @@ class TestResult(object):
"""Return the result of the test.
If there are subtests return the "worst" value of those subtests. If
- there are not return the stored value of the test.
+ there are not return the stored value of the test. There is an
+ exception to this rule, and that's if the status is crash; since this
+ status is set by the framework, and can be generated even when some or
+ all unit tests pass.
"""
- if self.subtests:
+ if self.subtests and self.__result != status.CRASH:
return max(six.itervalues(self.subtests))
return self.__result