How do you measure the overall quality of a product? One way is to take the spec, take the final product, and compare the two feature by feature. Well, QTP 9 went GA today, and we’ve pored over it so thoroughly you’d think we’ve had it for months. First I’ll present the summary, then over the next few days we will dissect the minutae of what they did, how well they did it, and whether or not it’s going to make your lives easier. Here’s a bullet-by-bullet summary of how well they did against their own press release:

• Open and edit multiple Object Repositories - yes, the fact is that you can do this, but it’s not always easy, and there were a lot of missed opportunities to go from good to great
• Multiple Object Repositories per test asset - I hate the term test asset. I think in this case they mean Action, because as far as I can tell, there is no other “thing” you can associate an OR with, and in version 9 you can associate multiple ORs with any given action. There are problems with this, but I’ll get to those later
• Easy (Object Repository) conversion to/from XML - Yep, for what it’s worth, the conversion is easy. It is very very very slow, and the schema is complicated, but both of those are pretty understandable.
• Easily copy/move objects between Object Repositories - Yes, you can even drag/drop objects from one repository to another, but maybe more importantly, you can reparent an object… this is super-duper cool, because you can now take all the objects under “Frame_4″ and “Frame_37″ and consolidate them under “Frame”. There are some HUGE problems with this feature that will probably be made easier in future releases, but it’s still possible and nice when you just have to do one or two objects
• Manage functions and keywords centrally - I don’t know what this means, outside from the ability to view/edit functions from within the main IDE. If that’s the case, they’re kinda cheating in making this a separate bullet point from the “Step Into…” point below. If that’s not the case, I have no idea what this means and shall declare it an example of “silly marketingspeak”
• New user interface - sure, OK. It’s not like the difference between vi and Visual Studio .NET, and they haven’t provided an Eclipse plugin yet (note to self: write Eclipse plugin for QTP), but I guess they’ve made some changes
• Missing resources panel - This is très cool, but not nearly as cool as the way they make it sound
• Comment and uncomment multiple lines in Expert View - One of those little nice things they’ve done to provide features we’re used to in every other programmer IDE on the market
• Indent and un-indent multiple lines in Expert View - ibid
• Pass parameters between Actions - I have no idea what this means, since you could do this before. Maybe they didn’t broadcast that you could do it before. Whatever
• Multiple Document Interface for function libraries - I guess this means you can open more than one function lib at a time. That’s yet another restatement of the “Manage Centrally” nonsense from the earlier bullet. But I’m guessing they list it three separate times because it’s simply the single best feature of version 9. It’s changed the way we compose our tests, which is funny, because it essentially means that we use Mercury’s scaffolding a lot less because they’ve made it easier for us to write our own
• Step Into Function Definition Code While Debugging the Test - Yes, THIS RULES!
• Enhanced IntelliSense Support - the main thing I notice here is that your user-defined RegisterUserFunc methods now appear in the code. Maybe that was there before, but I don’t know. I could never make it work

In our QTP framework, “strictness” has two dimensions: Test Level and Test Weight. Test Level refers to how loose the test is allowed to be when, say, selecting specific values from a WebList or WebRadioGroup. Test Weight refers to how important a given test is, and whether or not it will be run in the current cycle.

When we run tests, we run them at a default Test Level of 0, which is to say, “when you encounter a WebList or other input type, if the value I have specified does not exist, or only exists partially, just select a valid response and move on.”

The Test Weight is 200 (out of a possible 1000). The lower the Test Weight, the more important the test. When you run a 200-weight suite, you are essentially running a smoke test, where you want to run the least number of tests to validate that the system is up and running. This is what we run every time Cruise Control deploys a new site. A suite level of 800 or 1000 tells it to run every stinkin’ test we’ve written, and it takes much longer. That’s what we run at 2 am when nobody is here.

Let’s say I’m writing a QTP test where I create 15 new user accounts (each with different access features), log in as each, then perform existence checks on objects (buttons, checkboxes, etc) that should be enabled/disabled according to the access features. Given the number of different roles, features, and grants in our system, there can be over 800 permutations. Further, since we have between 4 and 8 hosted customer sites, this number mushrooms. As we host more customers, we want to scale our tests to be able to handle any number of customers, and we don’t want to have to change to code to do so.

Even further, we want to design the code so that we can re-use all these actions (User creation, logging in, access checking, etc.) in other scripts where we don’t want to run any checks beyond the most basic “did it work?” functionality.

Test Weight and Test Level are Environment variables, set in XML files, and altered at run-time to fit our needs (via an external utility). Each action has its own internal Test Weight, which is compared at run-time to Environment variable. If this internal Test Weight is greater than the weight of the suite being run, the action will short circuit and return to immediately without reporting anything or executing. If the internal Test Weight is less than or equal to the Environment variable, the test will execute.

Test Level is a lot more low-level. It is checked on a function-by-function basis, and the function in question always executes. The difference is, the Test Level will tell the function how to execute, then how to report its progress. An example would be (as above) a WebList, where you specify that “abc” be selected. If Test Level is 0 and “abc” doesn’t exist in the list, it will take the first non-empty string in the list and just select it without looking deeper into the matter. The idea is that if I’m testing a site in a different language, or a customer-deployed site that contains completely different information, the test will still work just fine.

Having these two dimensions on our tests ensures that we can run exactly the same scripts on multiple databases with varying degrees of thoroughness as the situation dictates.

Next, I’ll delve a little deeper into the different Test Levels we use, then I’ll post the code for this WebList.SmartSelect method, and how it implements the concept.

We've had to solve this problem quite a bit, so Will finally turned it into an object method.

Here's the problem:

You want to be able to say "Select the 'Butter' radio button", but the enterprising and oh-so-efficient programmer has made it so that each option has a value that corresponds to some GUID (e.g. "AFED-0839-02E8A5-0923CDB"), and "Option A" is nowhere to be found anywhere in the object's properties.

The method below allows you to send the text you wish to select, and the radio group will know exactly which button to select.

Select a Radio Button based on the adjacent descriptive text
- The WebRadioGroup to use
- The text to look for
- optional: The sWhere statement for getAdjacentText. Default is afterEnd
- optional: The Index number of the matched button to select. 0 selects first match, 1 selects second. Default is 0

Now, to select the radio button I want, I just to this:

Note that this performs an exact string match. The next step is to create our WebRadioGroup.SmartSelect function, which would be much more flexible, and would allow for partial matches of either adjascent text or the traditional button values. It would also allow for pattern matching via regular expressions. We'll post that one later.