« Classes, Objects, and QuickTest Pro 9
Using XML::XSLT to make your test results pretty »

reportStatus() - How we tie Test Level to Assertions

This gets to the heart of all our assert functions. As indicated in the first code snippet (the assertTrue function), all decisions regarding how to report an event are left up to reportStatus.

View as plain text
Visual Basic:
  1. Public Function assertTrue( bExpression, aOptions )
  2.     Set oOptions = GetOpts( ARRAY ( _
  3.         "sEvent", "AssertTrue", _
  4.         "sLogMessage", "Expected the given expression to return TRUE" _
  5.         ), aOptions)
  6.     'Return true or false
  7.     assertTrue = reportStatus(bExpression, oOptions("sEvent"), _
  8.                                            oOptions("sLogMessage"))
  9. End Function

Note that the return value is the same "true" or "false" that the expression evaluates to. This is so you can invoke the function via the following:

View as plain text
Visual Basic:
  1. If assertTrue(userMarcus.exists(NULL)) Then
  2.     userMarcus.edit(ARRAY("Role", "Agile SCRUM Master"))
  3. End If

The first lines use Will's ultra-handy named argument parsing to set default messages for the the result reporter. You are strongly encouraged to override these with custom messages, but you don't have to.

After that everything is very simple: reportStatus doesn't want to know anything other than "do you consider this a pass or a fail?" So, assertTrue just passes along the expression it received, which theoretically already contains "true" or "false". Then, it's reportStatus' job to decide how to report the pass/fail event to Reporter.ReportEvent.

This one function represents all the reasons we've stopped using QTP's built-in checkpoints: if the test is very strict by nature, the test will fail and exit. If the test requires flexibility (because it is being tested in an unfamiliar environment or database), problems will be ignored, and the test will do everything it can to move on and get to the end. If that doesn't make sense, go read our approach to strictness to get an idea of why we do things this way. It's worked very well for us so far.

The Test Level definitions live in a separate Constants.vbs file, containing the following (thanks to cytoe for pointing out that I forgot to define auNoReport):

View as plain text
Visual Basic:
  1. 'These are the Test Levels themselves
  2. auReportNothing = 0
  3. auReportNoWarnings = 1
  4. auReportWarnings = 2
  5. auReportFailures = 3
  6. auFailuresAreFatal = 4
  7.  
  8. 'This is used to determine whether ReportEvent should do anything
  9. auNoReport = -1

And they're meant to be self-explanatory. Going through the branches below, it's a very simple decision table, based on the current Test Level Environment variable, which then uses Mercury's built-in reporting constants.

One thing it also does is ensure that the ReporterFilter is fully enabled, then reset at the end of the function. We did this because QTP reports an awful lot of things we don't care about in its results files, and this allows us to turn it off in other places, but guarantee that it'll work like we expect here.

View as plain text
Visual Basic:
  1. 'This handles all test results, including reporting the
  2. ' events (or not) to the ReportEvent method
  3. Public Function reportStatus( bStatus, sEvent, sDetails )
  4.     iStatus = auNoReport
  5. sOldFilter = Reporter.Filter
  6. Reporter.Filter = rfEnableAll
  7.     If StringIsNotEmpty(Environment("TestLevel")) Then
  8.         Select Case CInt(Environment("TestLevel"))
  9.             Case auReportNothing
  10.                 iStatus = auNoReport
  11.             Case auReportNoWarnings
  12.                 iStatus = micDone
  13.             Case auReportWarnings
  14.                 If bStatus Then
  15.                     iStatus = micDone
  16.                 Else
  17.                     iStatus = micWarning
  18.                 End If
  19.             Case auReportFailures
  20.                 If bStatus Then
  21.                     iStatus = micPass
  22.                 Else
  23.                     iStatus = micFail
  24.                 End If
  25.             Case auFailuresAreFatal
  26.                 If bStatus Then
  27.                     iStatus = micPass
  28.                 Else
  29.                     iStatus = micFail
  30.                 End If
  31.         End Select
  32.         If iStatus <> auNoReport Then
  33.             Reporter.ReportEvent iStatus, sEvent, sDetails
  34.         End If
  35.         reportStatus = bStatus
  36.     End If
  37.     Reporter.Filter = rfEnableErrorsAndWarnings
  38. End Function

This thing has been rock solid for us for several months now, so we'd love to hear if anyone else has attempted a similar implementation that worked or didn't.

Tags: , , , ,

5 Comments for “reportStatus() - How we tie Test Level to Assertions”

  1. cytoe Says:
    September 28th, 2006 at 11:47 am

    In Function reportStatus, doesn’t auNoReport have to be set to something…perhaps “-1″? Also, if reportStatus should return true or false, shouldn’t reportStatus = bStatus ?

    Good stuff!
    thanks

    Reply to cytoe

  2. Marcus Says:
    September 28th, 2006 at 1:14 pm

    You’re absolutely right. I’ve been sitting here for 10 minutes trying to figure out why I originally made the decision to return iStatus instead of bStatus. The only thing I can remember is wanting to be able to test the return result for the auNoReport value, and offer the ability to make decisions based on that…

    But for the short term I don’t see much need for it. If anyone has particular conviction on this point, be sure to let me know.

    Thanks cytoe!

    Reply to Marcus

  3. cytoe Says:
    September 28th, 2006 at 4:52 pm

    While trying to implement this assert scheme, I found that I more often than not needed to override the defaults. Since vbs doesn’t do method overloading (grrrrr), I changed the following to make it easier to override sEvent and sDetail:

    Public Function AssertTrue( bExpression, sEvent, sDetail )
    aOptions = ARRAY ( _
    “sEvent”, sEvent, _
    “sLogMessage”, sDetail )
    Set oOptions = GetOpts( ARRAY ( _
    “sEvent”, “AssertTrue”, _
    “sLogMessage”, “Expected the given expression to return TRUE” _
    ), aOptions)
    ‘Return true or false
    assertTrue = reportStatus(bExpression, oOptions(”sEvent”), _
    oOptions(”sLogMessage”))
    End Function

    Public Function GetOpts(ByRef aDefaults, ByRef aCustom)
    Set oDefaults = DictBuild(aDefaults)
    Set oCustom = DictBuild(aCustom)
    For Each vKey in oDefaults
    If oCustom.Exists(vKey) and Not isNull(oCustom.Item(vKey))Then
    oDefaults.Item(vKey) = oCustom.Item(vKey)
    End If
    Next

    Set GetOpts = oDefaults
    End Function

    So if you want the defaults, just pass null for sEvent and sDetail, otherwise pass the overriding strings

    One problem I’m seeing is that ReportEvent output in the generated Test Results are all at the end of the report, not interspersed between the objects they are testing…so it might be hard to understand the results. Any ideas why this is?

    Reply to cytoe

  4. Marcus Says:
    September 28th, 2006 at 5:10 pm

    Sounds good… until recently we just passed on the strings sEvent and sDetail straight to the Reporter.ReportEvent call and it worked pretty well. I like what you’ve done with the defaults, it’s a slightly different spin that I think works pretty well.

    As far as the order of the test results, I can’t say that I’ve ever seen that before. Maybe you should try commenting out the part where Reporter.Filter is modified and see if that helps. If that’s what’s causing it, I’d call that a bug. In fact I think I’d call that a bug regardless, but I’d want to chase it down more to make sure.

    Let us know what else you find… so far all this stuff is just being used in house by a small but smart team of testers, so the more fresh eyes we have on it, the better.

    Thanks!

    Reply to Marcus

  5. cytoe Says:
    September 29th, 2006 at 10:52 am

    Yup…Reporter.Filter was causing the odd result behavior.

    Thanks

    Reply to cytoe

Leave a Reply