My team has recently taken on the challenge of conducting performance testing for our product. Performance and web-app security testing have been both been put under the same umbrella. That’s great–lots of efficiencies to be gained from our different skills, but at the same time, I’m no expert in either field. I have a perf test engineer who is good with Empirix e-Load, and he’s teaching me how we do things. We have a couple of books (including this one), and we’ve received some tool-specific training, but none of it has been enough as yet to convince me that we’re doing everything we can to test page load times accurately.

This is also one of the reasons we haven’t been posting much. With new responsibility comes new drains on time. (more…)

One limitation of QTP that has always bothered me is the inability to reuse its web objects after the browser reloads. I spend countless cpu cycles parsing the DOM to find an element, grab enough properties to make it unique, build a description with those properties and finally get the right micobject. Then I perform some action and that reloads the page, and my object is useless. If I want to use it, I have to start the process all over again.

After years of struggle to find an answer, I packed some gear, hired a Sherpa and set off to find a guru who could give me an answer.
(more…)

Despite persistent the rumors to the contrary, you really can execute Javascript from QTP, and not just using Web Extensibility. Here is a simple function that will run some javascript in IE and return the result.

(more...)

This gets to the heart of all our assert functions. As indicated in the first code snippet (the assertTrue function), all decisions regarding how to report an event are left up to reportStatus.

Note that the return value is the same "true" or "false" that the expression evaluates to. This is so you can invoke the function via the following:

The first lines use Will's ultra-handy named argument parsing to set default messages for the the result reporter. You are strongly encouraged to override these with custom messages, but you don't have to.

After that everything is very simple: reportStatus doesn't want to know anything other than "do you consider this a pass or a fail?" So, assertTrue just passes along the expression it received, which theoretically already contains "true" or "false". Then, it's reportStatus' job to decide how to report the pass/fail event to Reporter.ReportEvent.

This one function represents all the reasons we've stopped using QTP's built-in checkpoints: if the test is very strict by nature, the test will fail and exit. If the test requires flexibility (because it is being tested in an unfamiliar environment or database), problems will be ignored, and the test will do everything it can to move on and get to the end. If that doesn't make sense, go read our approach to strictness to get an idea of why we do things this way. It's worked very well for us so far.

The Test Level definitions live in a separate Constants.vbs file, containing the following (thanks to cytoe for pointing out that I forgot to define auNoReport):

And they're meant to be self-explanatory. Going through the branches below, it's a very simple decision table, based on the current Test Level Environment variable, which then uses Mercury's built-in reporting constants.

One thing it also does is ensure that the ReporterFilter is fully enabled, then reset at the end of the function. We did this because QTP reports an awful lot of things we don't care about in its results files, and this allows us to turn it off in other places, but guarantee that it'll work like we expect here.

This thing has been rock solid for us for several months now, so we'd love to hear if anyone else has attempted a similar implementation that worked or didn't.

VBScript is considered to be "object-based", meaning you can use classes, and those classes can have methods and members, but you can't get nearly the kind of re-use and extensibility you could out of a real OOP language. You don't get inheritance, you don't get polymorphism. In short, you get encapsulation, and that's about it. To me that's pretty important, the abilty to wrap up a bunch of common functions and attributes into a single entity called "an object" and throw it around kinda like you would in Java or Ruby. It allows me to create (instantiate) an object in line 10, then count on it being persisted three levels deep and 200 lines later, and never need to look inside the box to find out how it works.

I'll give some code first.

I'm using VBScript classes right now to model our AUT and make the code prettier. For example, if I want to create a user, it looks like this:

This snippet opens a browser, logs in as an administrator, creates a new User object (basically a variable that holds attributes about a user), then uses that data (if I had supplied any) to actually create user from within the GUI. After the user is created and declared "valid", I call the "addRole()" method off of the same user. The User object knows how to fill out all the form fields. It knows how to add a role through the security interface. How? That's the beauty: I don't have to know. Someone else can develop the library of objects and maintain it, and all I have to know how to do is invoke it.

I know, I know, it's the same with functions. Yes, it is. But here they're all wrapped up together. Consider this:

Later, if I want to know the name of that user, I can just ask the object:

Rather than have to code up a function to go and look it up somewhere. It's the persistance of data between states that I'm most interested in. The lack of inheritance and polymorphism is bad, but the most important problems are solved.

Here's the problem: in QTP, you don't just have the limitations of VBScript, you also have some severe limitations built in to (or rather "excluded from") QTP.

The class declaration (in a file called User.vbs) looks like this:

The first limitation:
That last function is there because QTP doesn't have the ability to use classes from external vbs files within the function libraries. You must give the test script access to the above function, which has access to the class, in order for your test script to be able to "see" the class.

The second limitation:
There's a bug in QTP where you can't, no matter what you do, use the debugger to step through class functions. That was almost a showstopper for me when doing this, before I learned just how few of me teammates use the debugger. When I found out I was the only one I decided that for the greater good, OO was a good direction to go in, if for no other reason than to make the code seem more familiar to us non-VBScript types.

So, in short, it's possible for us to get what we need to when we confront OO concepts, but we're severely limited by VBScript as well as Mercury. I'll submit defects on both of these issues, but I'm not sure whether I'd be optimistic about my chances. We seem to be the only ones trying to do this. Let me know if you know different!

Reading the post over at QA Forums makes me realize how long this is in coming. Hopefully this kind of interruption won't happen much more. The good news is that we've learned much, and had most of our theories about how to implement QTP proven successful.

So, the question was about using QTP's built-in Checkpoints, and how to code these things manually. It must have been the second or third day of my experience with QTP that I went back to look at a checkpoint (because something in the AUT changed, yada yada), and I realized you couldn't do much without going through all the windows. There wasn't much flexibility. What was worse: you didn't really have a good, reliable way to tell a Checkpoint, "okay, Checkpoint, I don't mean it this time. Don't report anything if there's a failure."

We test a lot of our QTP code against client databases, where custom fields are littered everywhere, list boxes contain varied and dynamic entries, and nothing is very reliable. Furthermore, we want to be able to hand our QTP scripts off to our Professional Services organization so they can try the scripts out without failures coming from everywhere.

Point being, we needed the ability to dig really deep inside the point, and change the very nature of what a Checkpoint does. Now that we have it implemented, I can't believe Mercury never tried to implement something like this.

Our solution was based on JUnit (or whatever system JUnit based their practice on). When you run unit tests in JUnit, you perform an operation, then assert some expression against the results.

When the assertion is true, the test passes. When the assertion is false, the test fails.

Let's go through an example:

Use Case

• Create a User with the user name of "foo" and password of "bar", assigned to the role of "Manager"
• Ensure that the user has been created in the interface

This means, after I've created the user, I want to go in and "assert" several things about that user. In our code, one such assertion would look like this (there would be more abstraction and no hard-coded strings, but you get the idea):

We have the following assert functions in our library (and we intend to write many more as the need arises):

Those last two are attached to most of the test objects via RegisterUserFunc. The idea is that you only need to qualify the object, then instead of this code:

you have this:

There are some ins and outs and gotchas, and the messages you send to the reporter.reportevent don't tend to be as clear (i.e. you can't say "this failed"--the message is going to be the same whether it passes or fails), but that's about the same as you get with Checkpoints. Checkpoints by default do better reporting than these assertions, but if you put enough information into the reportevent call, you won't miss that too much. Especially with the flexibility and freedom you get in return.

Soon, we'll post the code within the assertions (and I mean it this time).

I wrote this function this afternoon after being frustrated by the lack of decent checkpoints in QuickTest. I hope you find it insightful.

When deciding whether a piece of code should go in a reusable action or a function, we usually favored functions in QTP 8.2. When Mercury introduced QuickTest 9, they made the argument for functions even more compelling with the Multi Document Interface. Now we can open as many function libraries as we want, but we are still limited to one test. So, if you want to edit a reusable action in another test, you have to close your current test, open the second test, make your changes, save and close the second test then reopen your original test.

In addition to that new argument for functions in QTP 9, the same old arguments from 8.2 still apply. I'll sum up a few of the key differences between reusable actions and functions here.

Passing Data

Actions - can only accept primative data types as parameters (strings, integers, etc)

Functions - can accept arrays, dictionary objects and test objects (i.e. Pages, Frames, WebRadioGroups, etc.)

Resource Usage

Actions - For each action, there’s a folder, three separate vbs files, a local Object Repository, a subfolder containing snapshots, an Excel spreadsheet, and a few seconds of load time.

Functions - There’s the code contained in the Function, and that’s all

API

Actions - You cannot insert calls to Existing Actions through the QTP API, you can only do it through the interface

Functions - You can attach Function Libraries to Test Scripts through the QTP API

Return Values

Actions - Return values are difficult to understand and read in the syntax

Functions - Return values work like they do in other languages (i.e. as lvalues)

UPDATE:

In comments, Jugular Bean points out a significant downside to functions that I neglected above. You cannot change the number of parameters that a function accepts without affecting all calls to that function. That is a problem that used to drive me crazy. It seems like every time I put a new function in a library, the next test I write needs that function with one extra option. Now that I use the getopts function to define the parameters in my functions, I can easily add new optional parameters without messing up other tests that already call the function.

This is first in a series of analyses of the new features of QTP 9. Over the next few days we'll be posting more in varying levels of deatil, and within a week or two we hope to have just about all of them covered. We'll try to address where they went right, but we're not going to leave out where they could have gone farther. Please add comments where you agree/disagree with out conclusions. We want to be an accurate source of information, so if we're wrong, TELL US!

In my opinion, the Multiple Document Interface is the best feature of version 9. It is going to alter how we address the question of External Actions vs Functions.

With the new interface you can have one test script open, and then any number of Function Libraries. Under the FileOpen and FileNew menu items, there's now a separate entry for Function Libraries, and the libraries do appear in the Recent File list. The function libraries then appear as tabs across the top. From these tabs, you can right-click and it will bring up a small menu with "Save" and "Close" for options. There's also a "Save All" button in the toolbar. All in all, it's nice.

But more importantly than any of these small victories, there are the larger implications of the feature: you can now view, develop, execute, and debug your functions.

In the past, I've written the function inside my test script, made sure it worked properly, then put it in the VBS file. Then, if I find a bug in my function, I've had to copy it back into my test script, then repeat the whole thing ad nauseum. It's even one of the things Mercury advises in their training materials.

Allowing me to write and debug the code in situ means I don't need to be afraid to write new functions, or groan when I find a bug. In fact, if I get to a breakpoint where I'm calling an external function, it will open the library and allow me to step into the code without the extra step of opening it myself.

With this feature, we plan to shift the way we develop our framework: since we have full freedom to develop code in small chunks, we don't always want to spend the necessary overhead to create External Actions. We'll only create actions in certain situations. I'll discuss this topic in detail under a different heading. For now, I just wanted to post a description of this nice new feature, and compliment Mercury on implementing it very effectively.

Next, I'll cover in detail the little interface changes they've made. Some good, some not so good.

In our QTP framework, "strictness" has two dimensions: Test Level and Test Weight. Test Level refers to how loose the test is allowed to be when, say, selecting specific values from a WebList or WebRadioGroup. Test Weight refers to how important a given test is, and whether or not it will be run in the current cycle.

When we run tests, we run them at a default Test Level of 0, which is to say, "when you encounter a WebList or other input type, if the value I have specified does not exist, or only exists partially, just select a valid response and move on."

The Test Weight is 200 (out of a possible 1000). The lower the Test Weight, the more important the test. When you run a 200-weight suite, you are essentially running a smoke test, where you want to run the least number of tests to validate that the system is up and running. This is what we run every time Cruise Control deploys a new site. A suite level of 800 or 1000 tells it to run every stinkin' test we've written, and it takes much longer. That's what we run at 2 am when nobody is here.

Let's say I'm writing a QTP test where I create 15 new user accounts (each with different access features), log in as each, then perform existence checks on objects (buttons, checkboxes, etc) that should be enabled/disabled according to the access features. Given the number of different roles, features, and grants in our system, there can be over 800 permutations. Further, since we have between 4 and 8 hosted customer sites, this number mushrooms. As we host more customers, we want to scale our tests to be able to handle any number of customers, and we don't want to have to change to code to do so.

Even further, we want to design the code so that we can re-use all these actions (User creation, logging in, access checking, etc.) in other scripts where we don't want to run any checks beyond the most basic "did it work?" functionality.

Test Weight and Test Level are Environment variables, set in XML files, and altered at run-time to fit our needs (via an external utility). Each action has its own internal Test Weight, which is compared at run-time to Environment variable. If this internal Test Weight is greater than the weight of the suite being run, the action will short circuit and return to immediately without reporting anything or executing. If the internal Test Weight is less than or equal to the Environment variable, the test will execute.

Test Level is a lot more low-level. It is checked on a function-by-function basis, and the function in question always executes. The difference is, the Test Level will tell the function how to execute, then how to report its progress. An example would be (as above) a WebList, where you specify that "abc" be selected. If Test Level is 0 and "abc" doesn't exist in the list, it will take the first non-empty string in the list and just select it without looking deeper into the matter. The idea is that if I'm testing a site in a different language, or a customer-deployed site that contains completely different information, the test will still work just fine.

Having these two dimensions on our tests ensures that we can run exactly the same scripts on multiple databases with varying degrees of thoroughness as the situation dictates.

Next, I'll delve a little deeper into the different Test Levels we use, then I'll post the code for this WebList.SmartSelect method, and how it implements the concept.