The Inquisitors are proud to introduce a bit of code we have been working on. We are calling it the Software Inquisition Foundation Library for QTP, but you can call it SIFL (pronounced like "sniffle", but without the "n").

SIFL is a collection of QTP functions that we have been using for a while as a base framework for our QTP tests. Having eschewed checkpoints, the object repository, and the idea of record-and-playback testing, we needed a uniform way to write tests. SIFL is what we came up with.

Before I get into the details, let me show you a sample test to give an idea of what we are looking at. This is a quick example that uses the all-too-familiar Mercury Tours web site. We will open the site, navigate to the cruises page and make sure the departure time for the Skagway cruise is 5 pm.

If you are used to seeing QTP code, that should look somewhat – but not entirely – familiar. I’ll go line-by-line to show you what is happening.

BrowserDefault.Navigate "http://newtours.mercuryinteractive.com"

The “BrowserDefault” function returns the current default browser. Because we have not already defined a default browser, it will open a new instance of IE for us and make it the default for future calls to BrowserDefault. The “Navigate” portion of that line is just the browser object’s standard navigate method. This causes the newly opened instance of IE to navigate to the Mercury Tours site.

BrowserDefault.Link("innerText:=Cruises").Click

Here we are calling the default browser again. This time we are clicking a link in the default browser.

Set oTable = BrowserDefault.WebTable("innerText:=Cruise Itinerary.*")

Once again, this is pretty standard. We are just assigning this web table instance to oTable. This step isn’t really necessary, but I wanted to keep the next line short

sDeparture = oTable.GetXWhereYisString("Departure", "Port of Call", "Skagway, Alaska")

Here we are using the SIFL method “GetXWhereYisString”. To put it simply, this gets the data from the “Departure” column in the row where the “Port of Call” column is “Skagway, Alaska”. If everything is kosher on the site, sDeparture should be a string with the value “5 pm”.

assertStringEquals sDeparture, "5 pm"

Here we assert that the string sDeparture equals “5 pm” and note the results in the log. If the assertion is correct, it passes, if not, it fails. Note the fine control we have over the types involved in this homemade "checkpoint". This is what allows us to execute tests across continents without a complicated source tree.

I am writing this on a plane that is about to land, so that's it for now. Expect to see more on SIFL in the next few days along with a link to download it for yourself.

This gets to the heart of all our assert functions. As indicated in the first code snippet (the assertTrue function), all decisions regarding how to report an event are left up to reportStatus.

Note that the return value is the same "true" or "false" that the expression evaluates to. This is so you can invoke the function via the following:

The first lines use Will's ultra-handy named argument parsing to set default messages for the the result reporter. You are strongly encouraged to override these with custom messages, but you don't have to.

After that everything is very simple: reportStatus doesn't want to know anything other than "do you consider this a pass or a fail?" So, assertTrue just passes along the expression it received, which theoretically already contains "true" or "false". Then, it's reportStatus' job to decide how to report the pass/fail event to Reporter.ReportEvent.

This one function represents all the reasons we've stopped using QTP's built-in checkpoints: if the test is very strict by nature, the test will fail and exit. If the test requires flexibility (because it is being tested in an unfamiliar environment or database), problems will be ignored, and the test will do everything it can to move on and get to the end. If that doesn't make sense, go read our approach to strictness to get an idea of why we do things this way. It's worked very well for us so far.

The Test Level definitions live in a separate Constants.vbs file, containing the following (thanks to cytoe for pointing out that I forgot to define auNoReport):

And they're meant to be self-explanatory. Going through the branches below, it's a very simple decision table, based on the current Test Level Environment variable, which then uses Mercury's built-in reporting constants.

One thing it also does is ensure that the ReporterFilter is fully enabled, then reset at the end of the function. We did this because QTP reports an awful lot of things we don't care about in its results files, and this allows us to turn it off in other places, but guarantee that it'll work like we expect here.

This thing has been rock solid for us for several months now, so we'd love to hear if anyone else has attempted a similar implementation that worked or didn't.

Reading the post over at QA Forums makes me realize how long this is in coming. Hopefully this kind of interruption won't happen much more. The good news is that we've learned much, and had most of our theories about how to implement QTP proven successful.

So, the question was about using QTP's built-in Checkpoints, and how to code these things manually. It must have been the second or third day of my experience with QTP that I went back to look at a checkpoint (because something in the AUT changed, yada yada), and I realized you couldn't do much without going through all the windows. There wasn't much flexibility. What was worse: you didn't really have a good, reliable way to tell a Checkpoint, "okay, Checkpoint, I don't mean it this time. Don't report anything if there's a failure."

We test a lot of our QTP code against client databases, where custom fields are littered everywhere, list boxes contain varied and dynamic entries, and nothing is very reliable. Furthermore, we want to be able to hand our QTP scripts off to our Professional Services organization so they can try the scripts out without failures coming from everywhere.

Point being, we needed the ability to dig really deep inside the point, and change the very nature of what a Checkpoint does. Now that we have it implemented, I can't believe Mercury never tried to implement something like this.

Our solution was based on JUnit (or whatever system JUnit based their practice on). When you run unit tests in JUnit, you perform an operation, then assert some expression against the results.

When the assertion is true, the test passes. When the assertion is false, the test fails.

Let's go through an example:

Use Case

• Create a User with the user name of "foo" and password of "bar", assigned to the role of "Manager"
• Ensure that the user has been created in the interface

This means, after I've created the user, I want to go in and "assert" several things about that user. In our code, one such assertion would look like this (there would be more abstraction and no hard-coded strings, but you get the idea):

We have the following assert functions in our library (and we intend to write many more as the need arises):

Those last two are attached to most of the test objects via RegisterUserFunc. The idea is that you only need to qualify the object, then instead of this code:

you have this:

There are some ins and outs and gotchas, and the messages you send to the reporter.reportevent don't tend to be as clear (i.e. you can't say "this failed"--the message is going to be the same whether it passes or fails), but that's about the same as you get with Checkpoints. Checkpoints by default do better reporting than these assertions, but if you put enough information into the reportevent call, you won't miss that too much. Especially with the flexibility and freedom you get in return.

Soon, we'll post the code within the assertions (and I mean it this time).