A central server manages every aspect of the queue. All interactions with the queue take place via Apache over port 80, using very simple LWP::Simple (HTTP) requests. Nearly every CGI script in this discussion was written using basic Perl.

We started off with the same machine handling both the management of the queue and the test execution itself, but once we procured some hardware from elsewhere in the company, we were able to add 2 more “testrunner” machines.

These testrunner machines each have an instance of QuickTest Pro running off of our concurrent license server, and they each have a scheduled task that peeks into the queue every minute. If no job is available, the testrunner just waits another minute before checking again.

The task itself is just a perl script that does an HTTP GET against a specific cgi script on the queue server. The server responds with an XML file which contains everything the testrunner needs to run the job (including VBScript code) and report the results.

The testrunner takes the code from the XML file, generates a file called Script.mts, which it places inside a QTP test directory, in the Action0 folder, which already has all libraries and environment variables associated. The Perl script then uses the Win32::OLE module to talk to QTP via the COM library, which performs the following steps:

• Launches QTP with various execution options
• Executes the test
• Takes the Results.xml file from the Res1 folder (which gets overwritten each time a test is executed)
• Transforms the Results.xml file into HTML by using a slightly altered version of PShort.xsl (available in the dat folder of your QTP installation)
• Submits the HTML either/both via email, and/or to Rally, as a Test Case Result for that Test Case, complete with Verdict, Time/Date stamp, Build number, and the Notes as rich text (i.e. limited HTML)

So, when you get to the point (which we have) where you have a large cache of objects and a few hundred test cases stored in Rally, you can just call up any given test case and hit “Go”, and your result will come back to you in a few minutes.

Now, that’s an overview, and I want to explain some of the features in more detail. We wrote the system for how we needed to utilize it, which forced to come up with some pretty good functionality.

• You can specify either All Tests, Smoke Tests, one or more specific Packages, a list of test cases by number, or by Rally Query string
• The list of Rally Packages available is guaranteed to include only Packages for which there are automated test cases
• The tests will all execute against the default “nightly trunk” server, but you can choose instead to execute against a number of other predefined test sites, or you can select a profile’s Environment Variable set, then override the base URL - these Environment Variables will be passed into the script when the job is passed to the queue–more on all of this site information stuff in a future post
• You can choose to execute the tests against the GUI or against the API, using the exact same test scripts–more on this in a future post as well. It’s one of my favorite features of our framework, borne of careful planning and a little bit of luck
• If you’re executing your tests against an old version of our software, where the functionality changes were so significant that we had to branch our test automation code, you can specify that, and the same test scripts will still execute the old code–there are caveats to this, to be discussed, well, in a future post
• Finally, you specify how you want the results to be dealt with–either via email or via Rally
• We also have two “developer only” options that we don’t explain to anyone outside our team. If they figure them out, hey… they’re welcome: the first is the ability to just display the code generated by this test case number. This gives you the ability to look over the syntax to make sure things look okay. It’s very useful, but much less so since we got TDS and started writing tools for it. Now it’s a last resort. This doesn’t actually execute the test or put it in the queue
• The second “developer only” option is a hidden text field, revealed by JavaScript if a particular link is clicked, where you can just enter code. This option, when used, disables the Rally reporting option, and sends you an email with the results. It’s awesome for just ripping out some one-off code or trying to test whether the latest library changes have been pushed to the testrunner machines

Once all of these options have been dealt with, you hit Submit. Then, you start to see the test cases come back one by one–the CGI script is figuring out which tests are trying to execute, then downloading the code, then creating entries in the queue for immediate execution by the testrunners.

Once a job is taken, it goes from “queued” state to “pending” state. The server then awaits a final message that will take the job to “completed”, but truth be told we haven’t implemented that last step yet, and we haven’t needed to until relatively recently. Just knowing that we’ve delivered the jobs to the testrunner machines is good enough.

Each testrunner machine processes its own test results–it sends the emails, it sends the REST request to Rally, etc. That’s one of the things we’re hoping to change. We ultimately want a system that is completely “repository agnostic”. We don’t care where you store your automated test cases, we don’t care what platform you use to execute them, and we don’t care where you store the results. I think the best way to handle this will be to have the queue server handle the results processing (anticipating license issues with various platforms), but we haven’t needed to since we’ve integrated so thoroughly with Rally.

Nearly every point I’ve discussed here merits its own post. I hope we can take the time to share all of it over time, because we’re pretty proud of what we’ve done, and we believe it’s not just our company who should benefit from it. We’ve managed to solve a lot of the problems people run into in this business, and its in our very selfish interests, as test automation engineers/architects, to make sure that people out there use test automation rather more effectively than most groups seem to use it at the moment.
MM

Tags: QTP-9.5, quicktest professional, test automation, test automation framework, test automation process, test management