In our QTP framework, “strictness” has two dimensions: Test Level and Test Weight. Test Level refers to how loose the test is allowed to be when, say, selecting specific values from a WebList or WebRadioGroup. Test Weight refers to how important a given test is, and whether or not it will be run in the current cycle.

When we run tests, we run them at a default Test Level of 0, which is to say, “when you encounter a WebList or other input type, if the value I have specified does not exist, or only exists partially, just select a valid response and move on.”

The Test Weight is 200 (out of a possible 1000). The lower the Test Weight, the more important the test. When you run a 200-weight suite, you are essentially running a smoke test, where you want to run the least number of tests to validate that the system is up and running. This is what we run every time Cruise Control deploys a new site. A suite level of 800 or 1000 tells it to run every stinkin’ test we’ve written, and it takes much longer. That’s what we run at 2 am when nobody is here.

Let’s say I’m writing a QTP test where I create 15 new user accounts (each with different access features), log in as each, then perform existence checks on objects (buttons, checkboxes, etc) that should be enabled/disabled according to the access features. Given the number of different roles, features, and grants in our system, there can be over 800 permutations. Further, since we have between 4 and 8 hosted customer sites, this number mushrooms. As we host more customers, we want to scale our tests to be able to handle any number of customers, and we don’t want to have to change to code to do so.

Even further, we want to design the code so that we can re-use all these actions (User creation, logging in, access checking, etc.) in other scripts where we don’t want to run any checks beyond the most basic “did it work?” functionality.

Test Weight and Test Level are Environment variables, set in XML files, and altered at run-time to fit our needs (via an external utility). Each action has its own internal Test Weight, which is compared at run-time to Environment variable. If this internal Test Weight is greater than the weight of the suite being run, the action will short circuit and return to immediately without reporting anything or executing. If the internal Test Weight is less than or equal to the Environment variable, the test will execute.

Test Level is a lot more low-level. It is checked on a function-by-function basis, and the function in question always executes. The difference is, the Test Level will tell the function how to execute, then how to report its progress. An example would be (as above) a WebList, where you specify that “abc” be selected. If Test Level is 0 and “abc” doesn’t exist in the list, it will take the first non-empty string in the list and just select it without looking deeper into the matter. The idea is that if I’m testing a site in a different language, or a customer-deployed site that contains completely different information, the test will still work just fine.

Having these two dimensions on our tests ensures that we can run exactly the same scripts on multiple databases with varying degrees of thoroughness as the situation dictates.

Next, I’ll delve a little deeper into the different Test Levels we use, then I’ll post the code for this WebList.SmartSelect method, and how it implements the concept.

Tags: Functions, QTP 9, QTP-8.2, QuickTest Pro, Software-Testing, Test-Level