Pysys. Prevent execution of validate() method because of errors occured at execute()

Question

I'd want to prevent the execution of the validate() method of my testcase when something didn't go, in the execute() method, in the way I expected...

I've tried to do something like the following:

class MyTestCase(BaseTest):
   def execute(self):
      if somethingWentOK == False:
            self.addOutcome(BLOCKED)

And, from the validate() method:

   def validate(self):
      if self.getOutcome()!=BLOCKED:
           doValidationStuff()
      else:
           self.log.error("Something failed!!!")

But the output I get is different from what I expected:

2016-05-04 17:25:36,892 ERROR Testcase execution interrupted by previous errors!
2016-05-04 17:25:36,911 INFO
2016-05-04 17:25:36,933 INFO  Test duration 51.48 secs
2016-05-04 17:25:36,948 INFO  Test final outcome NOT VERIFIED
2016-05-04 17:25:36,950 INFO
2016-05-04 17:25:37,025 CRIT
2016-05-04 17:25:37,026 CRIT  Test duration: 51.60 (secs)
2016-05-04 17:25:37,026 CRIT
2016-05-04 17:25:37,026 CRIT  Summary of non passes and tests requiring inspection:

I thought I was going to get a BLOCKED outcome instead of the NOT VERIFIED one the framework produces...

I understand that I'm doing something not properly, so any feedback about that could be appreciated.

Answer 1

Basically the base test class has an attribute self.outcome which is a list of outcomes which can contain any of the following values in reverse precedence order - PASSED, INSPECT, NOTVERIFIED, FAILED, TIMEDOUT, DUMPEDCORE, BLOCKED and SKIPPED. When you call addOutcome() on the base test, you add a value to this list.

The overall outcome of the test is the entry in the list with highest precedence ie if the list contains [PASSED, FAILED, BLOCKED] then the overall outcome is BLOCKED. This is what is returned when you call getOutcome() on the base test. Any of the outcomes in BLOCKED, DUMPEDCORE, TIMEDOUT, FAILED are deemed to indicate a test failure. In the below example, which should do what you are wanting, we add a BLOCKED to the outcome list, and then in the validate method check to see if there is a BLOCKED entry in the outcome list and if so skip the validation.

from pysys.constants import *
from pysys.basetest import BaseTest

class PySysTest(BaseTest):
    somethingWentOK=False
    def execute(self):
        if self.somethingWentOK == False:
            self.log.error("Adding BLOCKED outcome ...")
            self.addOutcome(BLOCKED)

    def validate(self):
        self.log.info("Performing validation ...")
        if not BLOCKED in self.outcome:
            self.doValidationStuff()
        else:
            self.log.error("Something failed!!!")

    def doValidationStuff(self):
        pass

... which when run gives the following output;

C:\code\pysys-examples\tests>pysys.py run
2016:05:04 18:33:54 INFO  ==============================================================
2016:05:04 18:33:54 INFO  Id   : test_001
2016:05:04 18:33:54 INFO  ==============================================================
2016:05:04 18:33:54 ERROR Adding BLOCKED outcome ...
2016:05:04 18:33:54 INFO  Performing validation ...
2016:05:04 18:33:54 ERROR Something failed!!!
2016:05:04 18:33:54 INFO
2016:05:04 18:33:54 INFO  Test duration: 0.02 secs
2016:05:04 18:33:54 INFO  Test final outcome:  BLOCKED
2016:05:04 18:33:54 INFO
2016:05:04 18:33:54 CRIT
2016:05:04 18:33:54 CRIT  Test duration: 0.04 (secs)
2016:05:04 18:33:54 CRIT
2016:05:04 18:33:54 CRIT  Summary of non passes:
2016:05:04 18:33:54 CRIT    BLOCKED: test_001

PySys does by default execute the entire test before calculating the overall outcome and completing - ie it is not fail fast like a lot of unit test frameworks. In the latest release 1.1 there are fail fast semantics, such that as soon as an outcome is added to the outcome list which indicates a test failure, the test will complete immediately. You can run with fail fast semantics to the pysys.py run command using the -b option ie;

C:\code\pysys-examples\tests>pysys.py run  -b true
2016:05:04 18:36:25 INFO  ==============================================================
2016:05:04 18:36:25 INFO  Id   : test_001
2016:05:04 18:36:25 INFO  ==============================================================
2016:05:04 18:36:25 ERROR Adding BLOCKED outcome ...
2016:05:04 18:36:25 INFO  Aborting test due to abortOnError set to true ...
2016:05:04 18:36:25 INFO
2016:05:04 18:36:25 INFO  Test duration: 0.02 secs
2016:05:04 18:36:25 INFO  Test final outcome:  BLOCKED
2016:05:04 18:36:25 INFO
2016:05:04 18:36:25 CRIT
2016:05:04 18:36:25 CRIT  Test duration: 0.04 (secs)
2016:05:04 18:36:26 CRIT
2016:05:04 18:36:26 CRIT  Summary of non passes:
2016:05:04 18:36:26 CRIT    BLOCKED: test_001

In which case you don't need any conditional checks in your validate() method, as it wont be executed as soon as the BLOCKED outcome is added. If you want this on by default, you can set it in the pysysproject.xml file also.

property name="defaultAbortOnError" value="true"