When deciding whether a piece of code should go in a reusable action or a function, we usually favored functions in QTP 8.2. When Mercury introduced QuickTest 9, they made the argument for functions even more compelling with the Multi Document Interface. Now we can open as many function libraries as we want, but we are still limited to one test. So, if you want to edit a reusable action in another test, you have to close your current test, open the second test, make your changes, save and close the second test then reopen your original test.

In addition to that new argument for functions in QTP 9, the same old arguments from 8.2 still apply. I'll sum up a few of the key differences between reusable actions and functions here.

Passing Data

Actions - can only accept primative data types as parameters (strings, integers, etc)

Functions - can accept arrays, dictionary objects and test objects (i.e. Pages, Frames, WebRadioGroups, etc.)

Resource Usage

Actions - For each action, there’s a folder, three separate vbs files, a local Object Repository, a subfolder containing snapshots, an Excel spreadsheet, and a few seconds of load time.

Functions - There’s the code contained in the Function, and that’s all

API

Actions - You cannot insert calls to Existing Actions through the QTP API, you can only do it through the interface

Functions - You can attach Function Libraries to Test Scripts through the QTP API

Return Values

Actions - Return values are difficult to understand and read in the syntax

Functions - Return values work like they do in other languages (i.e. as lvalues)

UPDATE:

In comments, Jugular Bean points out a significant downside to functions that I neglected above. You cannot change the number of parameters that a function accepts without affecting all calls to that function. That is a problem that used to drive me crazy. It seems like every time I put a new function in a library, the next test I write needs that function with one extra option. Now that I use the getopts function to define the parameters in my functions, I can easily add new optional parameters without messing up other tests that already call the function.

My company is good to me. They give me two 19 inch LCDs to look at all day. Unfortunately, Mercury Interactive isn't as good to their employees. If they were, their developers would write software that works well on a dual monitor system.

I've spent quite some time struggling with a problem in QTP 9 where it won't click on a button in a dialog box. I didn't think to try moving the application to the other monitor until I called support. Fortunately, I happened to mention that the dialog was on my right monitor which tipped the support engineer off that I have multiple monitors, and he was able to tell me what the problem is. He said there is little chance of Mercury fixing the problem anytime soon because there is a workaround -- only use one monitor.

I'm not writing this to get you to all sympathize with me, but to tell you that there is a way out of this single-monitor bondage. After finding out this is a monitor related issue, I tried switching monitors. Previously I had QuickTest on the primary display and IE on the secondary. I switched that around so that QuickTest is on the right (secondary) monitor and IE is on the left, and it is working beautifully.

This is first in a series of analyses of the new features of QTP 9. Over the next few days we'll be posting more in varying levels of deatil, and within a week or two we hope to have just about all of them covered. We'll try to address where they went right, but we're not going to leave out where they could have gone farther. Please add comments where you agree/disagree with out conclusions. We want to be an accurate source of information, so if we're wrong, TELL US!

In my opinion, the Multiple Document Interface is the best feature of version 9. It is going to alter how we address the question of External Actions vs Functions.

With the new interface you can have one test script open, and then any number of Function Libraries. Under the FileOpen and FileNew menu items, there's now a separate entry for Function Libraries, and the libraries do appear in the Recent File list. The function libraries then appear as tabs across the top. From these tabs, you can right-click and it will bring up a small menu with "Save" and "Close" for options. There's also a "Save All" button in the toolbar. All in all, it's nice.

But more importantly than any of these small victories, there are the larger implications of the feature: you can now view, develop, execute, and debug your functions.

In the past, I've written the function inside my test script, made sure it worked properly, then put it in the VBS file. Then, if I find a bug in my function, I've had to copy it back into my test script, then repeat the whole thing ad nauseum. It's even one of the things Mercury advises in their training materials.

Allowing me to write and debug the code in situ means I don't need to be afraid to write new functions, or groan when I find a bug. In fact, if I get to a breakpoint where I'm calling an external function, it will open the library and allow me to step into the code without the extra step of opening it myself.

With this feature, we plan to shift the way we develop our framework: since we have full freedom to develop code in small chunks, we don't always want to spend the necessary overhead to create External Actions. We'll only create actions in certain situations. I'll discuss this topic in detail under a different heading. For now, I just wanted to post a description of this nice new feature, and compliment Mercury on implementing it very effectively.

Next, I'll cover in detail the little interface changes they've made. Some good, some not so good.

How do you measure the overall quality of a product? One way is to take the spec, take the final product, and compare the two feature by feature. Well, QTP 9 went GA today, and we've pored over it so thoroughly you'd think we've had it for months. First I'll present the summary, then over the next few days we will dissect the minutae of what they did, how well they did it, and whether or not it's going to make your lives easier. Here's a bullet-by-bullet summary of how well they did against their own press release:

• Open and edit multiple Object Repositories - yes, the fact is that you can do this, but it's not always easy, and there were a lot of missed opportunities to go from good to great
• Multiple Object Repositories per test asset - I hate the term test asset. I think in this case they mean Action, because as far as I can tell, there is no other "thing" you can associate an OR with, and in version 9 you can associate multiple ORs with any given action. There are problems with this, but I'll get to those later
• Easy (Object Repository) conversion to/from XML - Yep, for what it's worth, the conversion is easy. It is very very very slow, and the schema is complicated, but both of those are pretty understandable.
• Easily copy/move objects between Object Repositories - Yes, you can even drag/drop objects from one repository to another, but maybe more importantly, you can reparent an object... this is super-duper cool, because you can now take all the objects under "Frame_4" and "Frame_37" and consolidate them under "Frame". There are some HUGE problems with this feature that will probably be made easier in future releases, but it's still possible and nice when you just have to do one or two objects
• Manage functions and keywords centrally - I don't know what this means, outside from the ability to view/edit functions from within the main IDE. If that's the case, they're kinda cheating in making this a separate bullet point from the "Step Into..." point below. If that's not the case, I have no idea what this means and shall declare it an example of "silly marketingspeak"
• New user interface - sure, OK. It's not like the difference between vi and Visual Studio .NET, and they haven't provided an Eclipse plugin yet (note to self: write Eclipse plugin for QTP), but I guess they've made some changes
• Missing resources panel - This is très cool, but not nearly as cool as the way they make it sound
• Comment and uncomment multiple lines in Expert View - One of those little nice things they've done to provide features we're used to in every other programmer IDE on the market
• Indent and un-indent multiple lines in Expert View - ibid
• Pass parameters between Actions - I have no idea what this means, since you could do this before. Maybe they didn't broadcast that you could do it before. Whatever
• Multiple Document Interface for function libraries - I guess this means you can open more than one function lib at a time. That's yet another restatement of the "Manage Centrally" nonsense from the earlier bullet. But I'm guessing they list it three separate times because it's simply the single best feature of version 9. It's changed the way we compose our tests, which is funny, because it essentially means that we use Mercury's scaffolding a lot less because they've made it easier for us to write our own
• Step Into Function Definition Code While Debugging the Test - Yes, THIS RULES!
• Enhanced IntelliSense Support - the main thing I notice here is that your user-defined RegisterUserFunc methods now appear in the code. Maybe that was there before, but I don't know. I could never make it work

We don't use Checkpoints at all in QTP. They're just not flexible to the degree that we need them to be.

Whenever you call a checkpoint, there's a lot of stuff going on, very little of which you have control over. You can put in some things that make it more flexible, like regular expressions and comparisons to other output values, but in the end you have this big black box of stuff that you can't really get at. Plus, in the end, you have a Reporter.ReportEvent that goes to the test results as a Pass or Fail, and you have no way to alter the strings.

The problem with that is, as I've said in previous posts, I want different tests to behave differently in different circumstances, but I don't want to rewrite any of the code. When I'm not running deep or strict tests, I want the test results to be sparse, just letting me know when the broad strokes of the tests have been completed. The Test Results Viewer that comes with QTP is difficult enough to use (e.g. it doesn't remember filter settings, etc.), so when I filter the test results based on pass/fail, I want to zero in quickly on the tests that were important to me when I ran the tests. Sometimes we use test actions as a means to an end, not an end in themselves, and in those cases I don't want to see every detail.

I have a lot of other problems with the checkpoints as they are, but some of my problems may be a lack of education. My team abandoned checkpoints too early to really explore the ins and outs and what-have-yous (mainly because of the reporting problem), so I'd like to hear from anyone who is still using them and gets what they need out of them. I'll state some of the problems below. I'm not 100% sure that I'm right on these, so pretend that with all of them I've added "This is the experience I've had, and I've seen no evidence to contradict it. If you have some knowledge to impart, by all means let me know."

Some of these problems are:

• The Standard Checkpoints seem to be very, very slow. It's nice that you can look into specific object properties and all that, but the fact that it slows down the script is a real pain when you have a lot of them in there at once
• For Table Checkpoints, you can't be at all flexible in column or row checking. Our application allows the user to change the order of columns, etc., which means we need to
• For Text checkpoints, you must specify the nebulous "text before" and "text after", when, in my AUT at least, those things are seldom reliable or consistent
• Worse yet, when it comes to localization, they'll be in a different language. You could specify values from a data table, etc., but sometimes I would rather be able to say "the text to the right of the object with XYZ properties", or "the text that is in this frame, and I don't care about the words, only that the font color is red and the CSS class reference is std_WarnText"

Like I said, there may be other ways to do this, but I haven't found them. When I have coded around things to shoehorn my checkpoints into working, I've often later found them to be very fragile. I had an occurrence recently of a long, drawn-out test script, that probably contained over a thousand lines of code. The one line of code that broke consistently and with no good error message was the one leftover checkpoint from my earlier days. I finally removed it and replaced it with our new, preferred method.

What is that new, preferred method? I'll talk about that soon. To give you a hint, think about JUnit.

In the meantime I want to talk a little more about Test Level, and the different levels we've defined. There are 6 levels, and we're retaining the option to have more. Each level gives an indication of how strictly it behaves, and how its behavior is recorded into the test results.

• Test Level 0: Loosest testing possible. Try at all costs to be smart about selecting options, and just get through the code by any means necessary. Also, don't log anything that's not an error (and that's a QTP Run error, not a "test failed" error)
• Test Level 1: Be as loose as level 0, but report all non-errors as "micDone" entries in the results. This way, we've at least kept track of all the things we've done, but when you filter by Warnings and Errors, you'll only get the stuff that was considered Very Important by the test developer
• Test Level 2: Be as loose as level 0, but report all "smart" behaviors as warnings. When something doesn't exist in a WebList and you have to guess at it by clicking the first non-empty string, report that fact as a "micWarning". If the specified value was in the list, report it as a "micDone"
• Test Level 3: Again, be as loose as level 0, but report all "smart" behaviors as "micFail", and all correctly specified values as "micPass"
• Test Level 4: This time, don't be at all smart about how values are selected. If the developer specified something to be a certain way, report the problem as "micFail", then exit the test iteration (like QuickTest would by default, except that the intent is to give very detailed error information about what we were looking for and what appeared instead)
• Test Level 5: Same as Level 4, but exit all test iterations immediately

The nice thing is that you can ratchet up or down the Test Level as the test goes on. You can test shallow in order to achieve some sort of setup, like object or scenario creation, then knock it up a level or two when you start the really important part of the test.

Our company has tasked us with asking a bit more out of our automated tests than is normally recommended. We understand that you're not usually successful when you try to do outrageously large regression tests, or to simulate manual scripts. But they want something in between "normal" automation and 100% regression coverage, and techniques like this allow us to move much closer to that goal than ever before.

The ability to take the same scripts that perform the low-level field validation, then employ them as "just a step along the way", has been paramount. It takes a while to develop the initial tests, but the development gets faster and faster as we build our library. Then, with very little effort, we can build a smart suite that can be reused against new builds, against customer-deployed sites, and against new versions of the product.

And we don't use Mercury's checkpoints anywhere.

Next, I'll post an example of another Smart method, the one we use for WebLists. It employs heavy use of Test Level to determine its behavior and logging.

In our QTP framework, "strictness" has two dimensions: Test Level and Test Weight. Test Level refers to how loose the test is allowed to be when, say, selecting specific values from a WebList or WebRadioGroup. Test Weight refers to how important a given test is, and whether or not it will be run in the current cycle.

When we run tests, we run them at a default Test Level of 0, which is to say, "when you encounter a WebList or other input type, if the value I have specified does not exist, or only exists partially, just select a valid response and move on."

The Test Weight is 200 (out of a possible 1000). The lower the Test Weight, the more important the test. When you run a 200-weight suite, you are essentially running a smoke test, where you want to run the least number of tests to validate that the system is up and running. This is what we run every time Cruise Control deploys a new site. A suite level of 800 or 1000 tells it to run every stinkin' test we've written, and it takes much longer. That's what we run at 2 am when nobody is here.

Let's say I'm writing a QTP test where I create 15 new user accounts (each with different access features), log in as each, then perform existence checks on objects (buttons, checkboxes, etc) that should be enabled/disabled according to the access features. Given the number of different roles, features, and grants in our system, there can be over 800 permutations. Further, since we have between 4 and 8 hosted customer sites, this number mushrooms. As we host more customers, we want to scale our tests to be able to handle any number of customers, and we don't want to have to change to code to do so.

Even further, we want to design the code so that we can re-use all these actions (User creation, logging in, access checking, etc.) in other scripts where we don't want to run any checks beyond the most basic "did it work?" functionality.

Test Weight and Test Level are Environment variables, set in XML files, and altered at run-time to fit our needs (via an external utility). Each action has its own internal Test Weight, which is compared at run-time to Environment variable. If this internal Test Weight is greater than the weight of the suite being run, the action will short circuit and return to immediately without reporting anything or executing. If the internal Test Weight is less than or equal to the Environment variable, the test will execute.

Test Level is a lot more low-level. It is checked on a function-by-function basis, and the function in question always executes. The difference is, the Test Level will tell the function how to execute, then how to report its progress. An example would be (as above) a WebList, where you specify that "abc" be selected. If Test Level is 0 and "abc" doesn't exist in the list, it will take the first non-empty string in the list and just select it without looking deeper into the matter. The idea is that if I'm testing a site in a different language, or a customer-deployed site that contains completely different information, the test will still work just fine.

Having these two dimensions on our tests ensures that we can run exactly the same scripts on multiple databases with varying degrees of thoroughness as the situation dictates.

Next, I'll delve a little deeper into the different Test Levels we use, then I'll post the code for this WebList.SmartSelect method, and how it implements the concept.