I have spent most of the last few weeks working on QTP libraries, but very little of that time was spent wrestling with Mercury’s beast of an IDE. Aside from actually running my tests, I use QuickTest for very little these days. Instead, I am using a new editor called Test Design Studio. Everybody who uses QuickTest Pro’s expert view should try Test Design Studio. Once you do, you won’t go back to QTP.

Before we get too far into this, you need to know that Test Design Studio (TDS) isn’t a complete QTP replacement. TDS replaces QuickTest’s editor and library management, but it doesn’t actually run the tests. You use TDS to write tests and QTP to run them.

I could go on for pages about TDS’s features, but I won’t. You can see all of that at Patterson-consulting.net. Instead I will give you my first-ever top 10 list.

Top Ten Test Design Studio features that QTP is missing

10. Edit two tests at once – QTP is already on version 9.2 and they still haven’t figured out that users programmers might want to copy text from one test to another.

9. Excellent technical support – This isn’t technically a product feature, but it makes a big difference in using the product. I have submitted my share of help requests and bugs to Mercury, and I don’t think I have ever had an issue resolved within a day…rarely within a week. Patterson Consulting is another story. They will bend over backward to help you. Send an email to support or post in their help forum, and you have a good chance of getting a resolution within the hour. I haven’t called their support number, but I think I’m safe in assuming that if I do I won’t have to navigate through multiple levels of tech support before getting a resolution. The same can’t be said for Mercury.

8. Documentation Wizard – Automatically generate a help file from your code comments. If you maintain a large library, you know what a pain the documentation can be. TDS’s Documentation Wizard takes 90% of the work out of it.

7. IntelliSense for every class and function in your library – Don’t confuse this with QTP’s IntelliSense. TDS goes far beyond what QTP offers. With QTP, if you type Browser(oDescription), you will get IntelliSense for the various methods registered to the Browser type, but what if you have a function that returns a Browser object? You are out of luck with QTP, but TDS is able to tell that the function will return a browser. For instance, I often call the function BrowserDefault, which returns my default browser. With TDS, as soon as I type the period after the function name, I see all of Browser’s registered methods.

I can’t do justice to TDS’s IntelliSense here. You’ll have to see it for yourself.

6. Customizable menus – You can put almost anything you can think of on any toolbar or menu. I recently worked on customizing the xslt file that defines the layout of TDS’s Object Browser. Instead of restarting TDS to try out changes, I added the “Reload ObjectBrowser.xslt” command to my tools menu. For most applications, that would be an internal-only call, but it looks like Patterson Consulting has made a point to make everything possible available to their users.

5. Hooks for external tools – There are still a few things that Test Design Studio doesn’t do. For that, there are external tools. For instance, I have an external tool set up that grabs all the vbs files in my project and adds them as resources on the active test. Another external tool is a vbs script that will open and run the current test in QTP.

4. Customizable Hot Keys – One of the few things I don’t care for in TDS are the default hot-keys assignments, but thanks to the customizable hot keys, I don’t have to remember that Ctrl+E,C is the command to comment out a block of text. It took about five minutes to make TDS match the Eclipse hot keys that I am used to.
Also, by combining this with the external tools mentioned above, I just have to press F5 to run the current test in QTP.

3. Code snippets – If you use Visual Studio, you already know what these are. I had not used code snippets before TDS, but I have come to find them very useful. I’m not sure that I can adequately explain code snippets in this post, so I’ll let Patterson explain it.

2. Macros – If you like to do the same thing over and over and over and over, this isn’t for you.

1. Search and replace throughout the project – Search and replace throughout the project is huge. Shortly after I started evaluating TDS, I changed the signature on a function. It literally took less than a minute to find and update hundreds of calls to that function spread over dozens of files.

There you have my first-ever top ten list. I hope you enjoyed it. I probably shouldn’t have numbered them because I just wrote them in the order that I thought of them. I also ran out of numbers before I ran out of features. If the list were a bit longer I would have included Object Browser, collapsible code outlining and xml-comments.

This script is tightly coupled with the QTP.pm module in a previous post.

Note that the action refers to "ExecQTPTest.plx". I'm using the PLX extension because that's how you get IIS to use the Perl DLL instead of the executable. For some vague reason I've had a ton of trouble using the executable--I believe it has to do with Win32::OLE. It ends up executing the QTP code twice through, and I just don't know why. After much tinkering, I found the PLX solution and it seems to work much better.

Like I said before, I don't know of any reason you couldn't use Apache as your web server, but I haven't actually tested it.

There are a couple pieces of information missing from here: what is the big textarea called "XML"? How does the stylesheet get passed in? What's all this XSLT nonsense?

The answers for some of these questions are already up, some others only drafted, but I've just got to face the fact that we have a lot of information to put out there, and it's a slow process to turn it from in-house slop-code to polished, ready-for-primetime Open Source software. Our apologies in advance for the inevitable delays...

We launch by pointing a web browser to this URL:

http://servername/qtp/ExecQTPTest.plx?testname=LoginTest&dest=outputDirectory/results.html&redirect_results=1

After the test is finished (give it a minute or two), the browser is redirected to the html output generated by QTP.pm.

There may be bugs in there, because I had to strip out some company-proprietary stuff, so let me know if you have problems.

By the way, this is a good solution for executing tests in a targeted, specific manner, but we only use this rarely nowadays. The new methodology involves a queueing system, test harness-independent test execution, web service result reporting, and dynamic QTP script generation. It's awesome, but it's not ready for public consumption yet.

So, with this merger comes some questions, particularly about the future of Quality Center/WinRunner/QuickTest Pro, etc.

My reading of both the press release and articles like this tells me that the acquisition was much more about SOA than about testing. So where does that leave us grunts? Here is a list of questions I have about this. Note just how much this makes me look like a guy who sticks to functional test automation, and not a guy who speaks in acronyms and sees everything "from 40,000 feet".

Since it doesn't exactly fit within the ITG category, will it be spun off into another company or something?

I want better APIs and more web services. The APIs available now are decent, but they're very 1990s, very COM, very Web 1.0. I understand the industry is in a bit of a transition right now, but is there going to be more or less of an effort in this direction as a result of the merger?

My understanding of the Mercury policy on this kind of information is roughly, "take the $2500 training course if you have high-fallutin' questions." They don't write books, they don't offer (good) Best Practice advice for free, and they aren't extremely good at helping advanced users (at least with QuickTest Pro--there isn't even a QTP 9 Expert course out there yet).

I want to write a book about QTP--"The stuff they don't tell you" or something like that--but I don't know if any publisher would be allowed to release it at the moment (O'Reilly already passed on the idea, both because it's Mercury and because of the "small size of the testing market"--the latter of which I believe will change as more companies move to automation/outsourcing).

I want to know whether or not they will let up on the whole "fair use" thing and allow publishers to release some better information about how to use their products. People seem to be screaming for this kind of information, based if nothing else on the number of hits we get from specific, targeted Google searching.

That's all I have for now. Anyone else want to weigh in? Maybe I'll be able to get some of these questions answered at Mercury World.

Jeff posted a comment about my implication that we have integrated QuickTest Pro into our build processes. I thought I'd elaborate on that a little.

The first thing I should say is that it's not perfect, that we cobbled it together in between developing our testing strategy for our current release and writing regression test cases for some legacy products. We've got plans to improve it, we know pretty much what's missing, but, as with most meta-testing projects, IT WORKS.

The second thing I should say is that we're trying to follow an Agile methodology here, and while we haven't worked it all out, we at least use some of the terminology. If you see some term you're not familiar with, it probably comes from that school.

I'll start with an overview of the process, then discuss some of the steps in detail:

• At about 2am, CruiseControl starts what we call the "Nightly" build
• The Nightly build is just an Ant task that starts by building the software (it's a Tomcat webapp, but that's not critical to the process discussed here)
• A blank test database is created
• The build is unit-tested, then deployed to a test server
• QuickTest is called upon to run a test that just verified the most basic things: a user logs in, then several objects are tested for existence and properties
• The build is declared "Valid", and a more comprehensive suite of tests is run (via QuickTest Pro and other automated testing tools
• After each test is run, test results are reported back to Rally (our Agile project management software)

As you may see, these processes take the place of many Mercury products (Quality Center, IT Governance, etc.). We're a small shop and can't afford all those tools, and even with the amount of man hours spent on these side projects, I'll bet we've only spent a fraction of what we would spend were we to buy them. I can't quantify that, and I'm sure someone at Mercury would argue that I'm not factoring in anything, but one thing I know for sure: there is a heck of a lot about our needs that Mercury doesn't factor in when they charge their astronomical fees and we do actually pay them. They've provided some tools that allow us to get this stuff done, so we're taking full advantage.

And that should put to bed any speculation that we're secretly in league with Mercury and just using this forum to promote their products.

[/rant]

Here's the list of things we used to do it:

• Perl
• XML-XSLT
• Apache Ant
• CruiseControl
• Rally
• IIS (could as easily have been Apache)

We've written a web-based interface to our suite of QuickTest Pro tests, which allows anyone to run any test at any time on the box running QTP. It's a simple cgi script that takes several parameters: you select the test, enter where you want the results stored, and when the test is complete, it redirects you to an html representation of the test results.

What that really get us is the ability to launch any QTP script from any application that can make an HTTP request. Ant can do this, so we just have a series of Ant tasks that call the GET method, like so:

The @dest attribute just tells Ant where to put the resulting document on the build machine. We post it at a location that will be stored as a link in the Artifacts folder of the given build.

So, a Perl script, a VB script, heck, a wget command can launch QTP. That means you can integrate with nearly any product out there.

Now, as to the Perl script itself? It's kinda involved. I'll post that later when I'm able to cover it in more detail. The plot summary is that it uses the Win32::OLE library (available from CPAN, but usually distributed with any ActiveState version of Perl) to do exactly the same kind of interaction with QTP as you could do with VBS. The fact that it's in Perl means we don't have to do any special magic to get to it from within our test harness or any of our other interfaces, most of which are written in Perl.

The ability to integrate QTP tests with our existing build process was invaluable. It allowed us to keep on the track we were on before, using Open Source and keeping tight control over the minutae of our processes.