This gets to the heart of all our assert functions. As indicated in the first code snippet (the assertTrue function), all decisions regarding how to report an event are left up to reportStatus.

Note that the return value is the same "true" or "false" that the expression evaluates to. This is so you can invoke the function via the following:

The first lines use Will's ultra-handy named argument parsing to set default messages for the the result reporter. You are strongly encouraged to override these with custom messages, but you don't have to.

After that everything is very simple: reportStatus doesn't want to know anything other than "do you consider this a pass or a fail?" So, assertTrue just passes along the expression it received, which theoretically already contains "true" or "false". Then, it's reportStatus' job to decide how to report the pass/fail event to Reporter.ReportEvent.

This one function represents all the reasons we've stopped using QTP's built-in checkpoints: if the test is very strict by nature, the test will fail and exit. If the test requires flexibility (because it is being tested in an unfamiliar environment or database), problems will be ignored, and the test will do everything it can to move on and get to the end. If that doesn't make sense, go read our approach to strictness to get an idea of why we do things this way. It's worked very well for us so far.

The Test Level definitions live in a separate Constants.vbs file, containing the following (thanks to cytoe for pointing out that I forgot to define auNoReport):

And they're meant to be self-explanatory. Going through the branches below, it's a very simple decision table, based on the current Test Level Environment variable, which then uses Mercury's built-in reporting constants.

One thing it also does is ensure that the ReporterFilter is fully enabled, then reset at the end of the function. We did this because QTP reports an awful lot of things we don't care about in its results files, and this allows us to turn it off in other places, but guarantee that it'll work like we expect here.

This thing has been rock solid for us for several months now, so we'd love to hear if anyone else has attempted a similar implementation that worked or didn't.

Reading the post over at QA Forums makes me realize how long this is in coming. Hopefully this kind of interruption won't happen much more. The good news is that we've learned much, and had most of our theories about how to implement QTP proven successful.

So, the question was about using QTP's built-in Checkpoints, and how to code these things manually. It must have been the second or third day of my experience with QTP that I went back to look at a checkpoint (because something in the AUT changed, yada yada), and I realized you couldn't do much without going through all the windows. There wasn't much flexibility. What was worse: you didn't really have a good, reliable way to tell a Checkpoint, "okay, Checkpoint, I don't mean it this time. Don't report anything if there's a failure."

We test a lot of our QTP code against client databases, where custom fields are littered everywhere, list boxes contain varied and dynamic entries, and nothing is very reliable. Furthermore, we want to be able to hand our QTP scripts off to our Professional Services organization so they can try the scripts out without failures coming from everywhere.

Point being, we needed the ability to dig really deep inside the point, and change the very nature of what a Checkpoint does. Now that we have it implemented, I can't believe Mercury never tried to implement something like this.

Our solution was based on JUnit (or whatever system JUnit based their practice on). When you run unit tests in JUnit, you perform an operation, then assert some expression against the results.

When the assertion is true, the test passes. When the assertion is false, the test fails.

Let's go through an example:

Use Case

• Create a User with the user name of "foo" and password of "bar", assigned to the role of "Manager"
• Ensure that the user has been created in the interface

This means, after I've created the user, I want to go in and "assert" several things about that user. In our code, one such assertion would look like this (there would be more abstraction and no hard-coded strings, but you get the idea):

We have the following assert functions in our library (and we intend to write many more as the need arises):

Those last two are attached to most of the test objects via RegisterUserFunc. The idea is that you only need to qualify the object, then instead of this code:

you have this:

There are some ins and outs and gotchas, and the messages you send to the reporter.reportevent don't tend to be as clear (i.e. you can't say "this failed"--the message is going to be the same whether it passes or fails), but that's about the same as you get with Checkpoints. Checkpoints by default do better reporting than these assertions, but if you put enough information into the reportevent call, you won't miss that too much. Especially with the flexibility and freedom you get in return.

Soon, we'll post the code within the assertions (and I mean it this time).

We don't use Checkpoints at all in QTP. They're just not flexible to the degree that we need them to be.

Whenever you call a checkpoint, there's a lot of stuff going on, very little of which you have control over. You can put in some things that make it more flexible, like regular expressions and comparisons to other output values, but in the end you have this big black box of stuff that you can't really get at. Plus, in the end, you have a Reporter.ReportEvent that goes to the test results as a Pass or Fail, and you have no way to alter the strings.

The problem with that is, as I've said in previous posts, I want different tests to behave differently in different circumstances, but I don't want to rewrite any of the code. When I'm not running deep or strict tests, I want the test results to be sparse, just letting me know when the broad strokes of the tests have been completed. The Test Results Viewer that comes with QTP is difficult enough to use (e.g. it doesn't remember filter settings, etc.), so when I filter the test results based on pass/fail, I want to zero in quickly on the tests that were important to me when I ran the tests. Sometimes we use test actions as a means to an end, not an end in themselves, and in those cases I don't want to see every detail.

I have a lot of other problems with the checkpoints as they are, but some of my problems may be a lack of education. My team abandoned checkpoints too early to really explore the ins and outs and what-have-yous (mainly because of the reporting problem), so I'd like to hear from anyone who is still using them and gets what they need out of them. I'll state some of the problems below. I'm not 100% sure that I'm right on these, so pretend that with all of them I've added "This is the experience I've had, and I've seen no evidence to contradict it. If you have some knowledge to impart, by all means let me know."

Some of these problems are:

• The Standard Checkpoints seem to be very, very slow. It's nice that you can look into specific object properties and all that, but the fact that it slows down the script is a real pain when you have a lot of them in there at once
• For Table Checkpoints, you can't be at all flexible in column or row checking. Our application allows the user to change the order of columns, etc., which means we need to
• For Text checkpoints, you must specify the nebulous "text before" and "text after", when, in my AUT at least, those things are seldom reliable or consistent
• Worse yet, when it comes to localization, they'll be in a different language. You could specify values from a data table, etc., but sometimes I would rather be able to say "the text to the right of the object with XYZ properties", or "the text that is in this frame, and I don't care about the words, only that the font color is red and the CSS class reference is std_WarnText"

Like I said, there may be other ways to do this, but I haven't found them. When I have coded around things to shoehorn my checkpoints into working, I've often later found them to be very fragile. I had an occurrence recently of a long, drawn-out test script, that probably contained over a thousand lines of code. The one line of code that broke consistently and with no good error message was the one leftover checkpoint from my earlier days. I finally removed it and replaced it with our new, preferred method.

What is that new, preferred method? I'll talk about that soon. To give you a hint, think about JUnit.

In the meantime I want to talk a little more about Test Level, and the different levels we've defined. There are 6 levels, and we're retaining the option to have more. Each level gives an indication of how strictly it behaves, and how its behavior is recorded into the test results.

• Test Level 0: Loosest testing possible. Try at all costs to be smart about selecting options, and just get through the code by any means necessary. Also, don't log anything that's not an error (and that's a QTP Run error, not a "test failed" error)
• Test Level 1: Be as loose as level 0, but report all non-errors as "micDone" entries in the results. This way, we've at least kept track of all the things we've done, but when you filter by Warnings and Errors, you'll only get the stuff that was considered Very Important by the test developer
• Test Level 2: Be as loose as level 0, but report all "smart" behaviors as warnings. When something doesn't exist in a WebList and you have to guess at it by clicking the first non-empty string, report that fact as a "micWarning". If the specified value was in the list, report it as a "micDone"
• Test Level 3: Again, be as loose as level 0, but report all "smart" behaviors as "micFail", and all correctly specified values as "micPass"
• Test Level 4: This time, don't be at all smart about how values are selected. If the developer specified something to be a certain way, report the problem as "micFail", then exit the test iteration (like QuickTest would by default, except that the intent is to give very detailed error information about what we were looking for and what appeared instead)
• Test Level 5: Same as Level 4, but exit all test iterations immediately

The nice thing is that you can ratchet up or down the Test Level as the test goes on. You can test shallow in order to achieve some sort of setup, like object or scenario creation, then knock it up a level or two when you start the really important part of the test.

Our company has tasked us with asking a bit more out of our automated tests than is normally recommended. We understand that you're not usually successful when you try to do outrageously large regression tests, or to simulate manual scripts. But they want something in between "normal" automation and 100% regression coverage, and techniques like this allow us to move much closer to that goal than ever before.

The ability to take the same scripts that perform the low-level field validation, then employ them as "just a step along the way", has been paramount. It takes a while to develop the initial tests, but the development gets faster and faster as we build our library. Then, with very little effort, we can build a smart suite that can be reused against new builds, against customer-deployed sites, and against new versions of the product.

And we don't use Mercury's checkpoints anywhere.

Next, I'll post an example of another Smart method, the one we use for WebLists. It employs heavy use of Test Level to determine its behavior and logging.