The QTP 9.5 evaluation is finally available for download on HP’s site. I haven’t downloaded this one yet, but if it is like the one I downloaded from HP a few weeks ago, you should be able to use it with the same license you used for 9.2.
I have spent most of the last few weeks working on QTP libraries, but very little of that time was spent wrestling with Mercury’s beast of an IDE. Aside from actually running my tests, I use QuickTest for very little these days. Instead, I am using a new editor called Test Design Studio. Everybody who uses QuickTest Pro’s expert view should try Test Design Studio. Once you do, you won’t go back to QTP.
Before we get too far into this, you need to know that Test Design Studio (TDS) isn’t a complete QTP replacement. TDS replaces QuickTest’s editor and library management, but it doesn’t actually run the tests. You use TDS to write tests and QTP to run them.
I could go on for pages about TDS’s features, but I won’t. You can see all of that at Patterson-consulting.net. Instead I will give you my first-ever top 10 list.
10. Edit two tests at once – QTP is already on version 9.2 and they still haven’t figured out that users programmers might want to copy text from one test to another.
9. Excellent technical support – This isn’t technically a product feature, but it makes a big difference in using the product. I have submitted my share of help requests and bugs to Mercury, and I don’t think I have ever had an issue resolved within a day…rarely within a week. Patterson Consulting is another story. They will bend over backward to help you. Send an email to support or post in their help forum, and you have a good chance of getting a resolution within the hour. I haven’t called their support number, but I think I’m safe in assuming that if I do I won’t have to navigate through multiple levels of tech support before getting a resolution. The same can’t be said for Mercury.
8. Documentation Wizard – Automatically generate a help file from your code comments. If you maintain a large library, you know what a pain the documentation can be. TDS’s Documentation Wizard takes 90% of the work out of it.
7. IntelliSense for every class and function in your library – Don’t confuse this with QTP’s IntelliSense. TDS goes far beyond what QTP offers. With QTP, if you type Browser(oDescription), you will get IntelliSense for the various methods registered to the Browser type, but what if you have a function that returns a Browser object? You are out of luck with QTP, but TDS is able to tell that the function will return a browser. For instance, I often call the function BrowserDefault, which returns my default browser. With TDS, as soon as I type the period after the function name, I see all of Browser’s registered methods.
I can’t do justice to TDS’s IntelliSense here. You’ll have to see it for yourself.
6. Customizable menus – You can put almost anything you can think of on any toolbar or menu. I recently worked on customizing the xslt file that defines the layout of TDS’s Object Browser. Instead of restarting TDS to try out changes, I added the “Reload ObjectBrowser.xslt” command to my tools menu. For most applications, that would be an internal-only call, but it looks like Patterson Consulting has made a point to make everything possible available to their users.
5. Hooks for external tools – There are still a few things that Test Design Studio doesn’t do. For that, there are external tools. For instance, I have an external tool set up that grabs all the vbs files in my project and adds them as resources on the active test. Another external tool is a vbs script that will open and run the current test in QTP.
4. Customizable Hot Keys – One of the few things I don’t care for in TDS are the default hot-keys assignments, but thanks to the customizable hot keys, I don’t have to remember that Ctrl+E,C is the command to comment out a block of text. It took about five minutes to make TDS match the Eclipse hot keys that I am used to.
Also, by combining this with the external tools mentioned above, I just have to press F5 to run the current test in QTP.
3. Code snippets – If you use Visual Studio, you already know what these are. I had not used code snippets before TDS, but I have come to find them very useful. I’m not sure that I can adequately explain code snippets in this post, so I’ll let Patterson explain it.
2. Macros – If you like to do the same thing over and over and over and over, this isn’t for you.
1. Search and replace throughout the project – Search and replace throughout the project is huge. Shortly after I started evaluating TDS, I changed the signature on a function. It literally took less than a minute to find and update hundreds of calls to that function spread over dozens of files.
There you have my first-ever top ten list. I hope you enjoyed it. I probably shouldn’t have numbered them because I just wrote them in the order that I thought of them. I also ran out of numbers before I ran out of features. If the list were a bit longer I would have included Object Browser, collapsible code outlining and xml-comments.
The Inquisitors are proud to introduce a bit of code we have been working on. We are calling it the Software Inquisition Foundation Library for QTP, but you can call it SIFL (pronounced like "sniffle", but without the "n").
SIFL is a collection of QTP functions that we have been using for a while as a base framework for our QTP tests. Having eschewed checkpoints, the object repository, and the idea of record-and-playback testing, we needed a uniform way to write tests. SIFL is what we came up with.
Before I get into the details, let me show you a sample test to give an idea of what we are looking at. This is a quick example that uses the all-too-familiar Mercury Tours web site. We will open the site, navigate to the cruises page and make sure the departure time for the Skagway cruise is 5 pm.
If you are used to seeing QTP code, that should look somewhat – but not entirely – familiar. I’ll go line-by-line to show you what is happening.
BrowserDefault.Navigate "http://newtours.mercuryinteractive.com"
The “BrowserDefault” function returns the current default browser. Because we have not already defined a default browser, it will open a new instance of IE for us and make it the default for future calls to BrowserDefault. The “Navigate” portion of that line is just the browser object’s standard navigate method. This causes the newly opened instance of IE to navigate to the Mercury Tours site.
BrowserDefault.Link("innerText:=Cruises").Click
Here we are calling the default browser again. This time we are clicking a link in the default browser.
Set oTable = BrowserDefault.WebTable("innerText:=Cruise Itinerary.*")
Once again, this is pretty standard. We are just assigning this web table instance to oTable. This step isn’t really necessary, but I wanted to keep the next line short
sDeparture = oTable.GetXWhereYisString("Departure", "Port of Call", "Skagway, Alaska")
Here we are using the SIFL method “GetXWhereYisString”. To put it simply, this gets the data from the “Departure” column in the row where the “Port of Call” column is “Skagway, Alaska”. If everything is kosher on the site, sDeparture should be a string with the value “5 pm”.
assertStringEquals sDeparture, "5 pm"
Here we assert that the string sDeparture equals “5 pm” and note the results in the log. If the assertion is correct, it passes, if not, it fails. Note the fine control we have over the types involved in this homemade "checkpoint". This is what allows us to execute tests across continents without a complicated source tree.
I am writing this on a plane that is about to land, so that's it for now. Expect to see more on SIFL in the next few days along with a link to download it for yourself.
I am in New York at the Google Test Automation Conference. There is little talk of QTP here, but the material is relevant to most software test automation - no matter what tool you use. The conference sessions cover web-based testing as well as test driven development.
The talks will be posted on the Youtube Google channel soon after they occur. I recommend looking at them, and I will post a link when it is available.
I'm particularly interested in seeing a talk that Simon Stewart is doing on WebDriver. It is a Java based tool for web testing. Simon says it will soon be available for Ruby as well.
UPDATE: There are many links to GTAC related info on the google testing blog.
According to Google Analytics, about 36% of you understand the title of this post. For the rest of you, it is Hindi for "We welcome our Indian friends".
Over the past 18 months, there we have had almost as many Indian visitors as US visitors. Considering this is a US site written in English, it amazes me that 7500 people in India have visited us and read 25,000 articles (or at least looked at them). I had heard that there is a lot of software development and testing happening in India, but seeing this many Indian visitors makes it real to me. Also making it real to me are daily phone calls and IMs with three coworkers in Bangalore: Ashok, Sridhar and Shiva. They make up 3/5 of our test automation team.
The top six cities that The Software Inquisition receives visitors from are in India: Bangalore, Chennai, Mumbai, Hyderabad, Delhi and Pune. The first US city on the list is New York at number 7.
I have been extensively using the method for vbscript optional arguments that I described in March. I want to expand on this a little with some recent changes we have made to how we use this.
Functions that use this method have an aOptions parameter in their signatures. That means, that even if we are not going to specify any optional arguments, we have to pass something, so we pass a Null. The problem is that I end up passing Null most of the time, and I don't like having to do that - especially for our smoke tests where are code is visible to manual testers and developers.
For example, I will use a simple function called WebEditFillByID that simply finds a WebEdit based on it's html id and fills it with the text supplied. There is also an aOptions parameter in its signature, but I don't usually set any options. So, when I am filling out a form it looks something like this.
That doesn't look too bad, but it would be much more legible without the Nulls in the way. So, to get aOptions out of WebEditFillByID's signature I create a new function called WebEditFillByIDAdv that is identical to WebEditFillByID. Directly beneath WebEditFillByIDAdv, I put a second function named WebEditFillByID with the same signature minus the aOptions. All WebEditFillByID does is call WebEditFillByID with the same arguments it was passed and adds a Null for the aOptions in WebEditFillByID's signature.
Now I can fill the same form without the Nulls.
In my opinion, this is much more pleasant to deal with. For reference, here are the new functions.
To read more about named optional arguments in vbscript, see my previous post about it.
Short Story - we had a great time and learned a lot. It probably didn't suit us as well as it could have (since we're not currently QC customers--just QTP), but a few of the track sessions were excellent. There was one I would even call "validating" to both our approach to Mercury's products, and to the direction we're headed in as a QA organization.
Will and I gave a presentation on the All-Pairs technique, something we haven't really talked about here but which we're both big fans of. The slides are up now that the conference is over, and I put a lot of notes in there to fill in the blanks for those who missed the talk.
Long Story
There wasn't much going on with the conference Saturday or Sunday (registration and some 1-on-1 time with Mercury people), so I sat in the sports book at Bellagio and made modest wagers on various sports teams, enjoying the accoutrements supplied by the generous waitstaff. That is to say, if you've got a couple days to kill, go get blitzed at the Bellagio. The drinks are free as long as you're gambling!
Monday morning was mostly about the big-time speakers, the Mercury CEO, Jared Diamond, and a couple others. At the first track session (I won't mention which, because there's no need to call anyone out after the fact), I walked out during the Q&A, wondering whether the presenter(s) was/were idiots or I had just misunderstood. After I got out, the VP of IT for Citigroup was nearby, commenting on how that session was a bunch of BS, and how did it ever make it into the program... So, at least I can say I reached the same conclusion as a Citigroup VP...
The next session was purportedly an examination of Agile methods in conjunction with Quality Center and Business Process Testing. I didn't see anything in there that told me the presenter had ever really been a part of or studied Agile methodology. He kept referring to "ways to keep your agility in testing", and "how to stay flexible in light of constant change", and that's good stuff, but it's a far cry from talking about SCRUMs, iterative development, feature-complete delivery at milestones, story cards, etc. It ended up just being the third time we got a demo of Quality Center. I left during the Q&A there too... very disappointing first set of track sessions.
The next morning started great--"Functional Testing of Web Services" by a guy from Mercury. Fantastic session. He covered just about every question I had had about the QTP Web Services plug-in, and got me excited about using it. We have a home-grown tool we use to test web services, and it's good (I wrote it ;) ), but it's not as polished as this one. This would also give us the ability to unify the web services test results with the web app test results.
We skipped around the next session (figuring that nearly every one listed would just be product demos we'd seen before--and we were right), but at 2 there was an excellent talk by Ryan English from SPI Dynamics. He covered ~8 different types of security defects common in web applications, along with both automated and manual suggestions for how to test for them. I'm disappointed that the slides aren't online, because it was a crash course on everything that's wrong with web software (from a security standpoint).
Wednesday morning we rehearsed our own presentation and watched as Marika Lazi, VP of IT for Bear Stearns, rehearsed hers. It was called "Automation of Edge-to-Edge Testing", and covered many of the kinds of problems Will and I have when trying to automate our own testing without the suite of QC products. She was sharp as a tack and very realistic in her approach to automated testing. I was glad to see that it wasn't just a sales pitch by a sales person, but a real stand-and-deliver from someone who sees the trenches every day even if she doesn't live in them.
After lunch, Will and I gave our presentation. I have to say that the A/V equipment and management at the Bellagio is probably about the best it could possibly be. World class, really.
I think our talk went pretty well overall, though I was disappointed that nobody in the audience had ever heard of the Inquisition! We met a bunch of people afterwards who had some great comments and interesting insights, and my hope is that we hear from one Martin O'Brien again, who showed us an amazing Neural-Network-based test case processor-generator that could execute billions of test cases in short amounts of time. He left us with a million questions, but no contact info.
Did I mention the amazing food? At lunch every day and at dinner Monday and Wednesday, I enjoyed some of the finest dining I've ever been around... and not just because it was free. It was great all the way around. Mercury sure knows how to throw a party.
I could talk about the gambling, the drinking, the fact that I've officially been to both a Vegas nightclub and a Vegas ultra-lounge (and my company didn't pay for a dime of it!), the ~20 hours of football I watched, only sleeping about 12 hours from Sunday to Thursday, actually winning money at poker, and all sorts of other stuff, but I'll just leave it in Vegas. I had a great time, didn't spend nearly as much of my company's money as I could have, and didn't even put an ounce stress on any marriage vows. All in all, if you have to spend a week attending 20 presentations on a fairly dry topic like automated testing, I can think of worse ways to spend said weeks.
Hope everyone got as much as I did out of it, and I hope to see you all at the Venetian next June!!
This script is tightly coupled with the QTP.pm module in a previous post.
Note that the action refers to "ExecQTPTest.plx". I'm using the PLX extension because that's how you get IIS to use the Perl DLL instead of the executable. For some vague reason I've had a ton of trouble using the executable--I believe it has to do with Win32::OLE. It ends up executing the QTP code twice through, and I just don't know why. After much tinkering, I found the PLX solution and it seems to work much better.
Like I said before, I don't know of any reason you couldn't use Apache as your web server, but I haven't actually tested it.
There are a couple pieces of information missing from here: what is the big textarea called "XML"? How does the stylesheet get passed in? What's all this XSLT nonsense?
The answers for some of these questions are already up, some others only drafted, but I've just got to face the fact that we have a lot of information to put out there, and it's a slow process to turn it from in-house slop-code to polished, ready-for-primetime Open Source software. Our apologies in advance for the inevitable delays...
We launch by pointing a web browser to this URL:
http://servername/qtp/ExecQTPTest.plx?testname=LoginTest&dest=outputDirectory/results.html&redirect_results=1
After the test is finished (give it a minute or two), the browser is redirected to the html output generated by QTP.pm.
There may be bugs in there, because I had to strip out some company-proprietary stuff, so let me know if you have problems.
By the way, this is a good solution for executing tests in a targeted, specific manner, but we only use this rarely nowadays. The new methodology involves a queueing system, test harness-independent test execution, web service result reporting, and dynamic QTP script generation. It's awesome, but it's not ready for public consumption yet.
This gets to the heart of all our assert functions. As indicated in the first code snippet (the assertTrue function), all decisions regarding how to report an event are left up to reportStatus.
Note that the return value is the same "true" or "false" that the expression evaluates to. This is so you can invoke the function via the following:
The first lines use Will's ultra-handy named argument parsing to set default messages for the the result reporter. You are strongly encouraged to override these with custom messages, but you don't have to.
After that everything is very simple: reportStatus doesn't want to know anything other than "do you consider this a pass or a fail?" So, assertTrue just passes along the expression it received, which theoretically already contains "true" or "false". Then, it's reportStatus' job to decide how to report the pass/fail event to Reporter.ReportEvent.
This one function represents all the reasons we've stopped using QTP's built-in checkpoints: if the test is very strict by nature, the test will fail and exit. If the test requires flexibility (because it is being tested in an unfamiliar environment or database), problems will be ignored, and the test will do everything it can to move on and get to the end. If that doesn't make sense, go read our approach to strictness to get an idea of why we do things this way. It's worked very well for us so far.
The Test Level definitions live in a separate Constants.vbs file, containing the following (thanks to cytoe for pointing out that I forgot to define auNoReport):
And they're meant to be self-explanatory. Going through the branches below, it's a very simple decision table, based on the current Test Level Environment variable, which then uses Mercury's built-in reporting constants.
One thing it also does is ensure that the ReporterFilter is fully enabled, then reset at the end of the function. We did this because QTP reports an awful lot of things we don't care about in its results files, and this allows us to turn it off in other places, but guarantee that it'll work like we expect here.
This thing has been rock solid for us for several months now, so we'd love to hear if anyone else has attempted a similar implementation that worked or didn't.
VBScript is considered to be "object-based", meaning you can use classes, and those classes can have methods and members, but you can't get nearly the kind of re-use and extensibility you could out of a real OOP language. You don't get inheritance, you don't get polymorphism. In short, you get encapsulation, and that's about it. To me that's pretty important, the abilty to wrap up a bunch of common functions and attributes into a single entity called "an object" and throw it around kinda like you would in Java or Ruby. It allows me to create (instantiate) an object in line 10, then count on it being persisted three levels deep and 200 lines later, and never need to look inside the box to find out how it works.
I'll give some code first.
I'm using VBScript classes right now to model our AUT and make the code prettier. For example, if I want to create a user, it looks like this:
This snippet opens a browser, logs in as an administrator, creates a new User object (basically a variable that holds attributes about a user), then uses that data (if I had supplied any) to actually create user from within the GUI. After the user is created and declared "valid", I call the "addRole()" method off of the same user. The User object knows how to fill out all the form fields. It knows how to add a role through the security interface. How? That's the beauty: I don't have to know. Someone else can develop the library of objects and maintain it, and all I have to know how to do is invoke it.
I know, I know, it's the same with functions. Yes, it is. But here they're all wrapped up together. Consider this:
Later, if I want to know the name of that user, I can just ask the object:
Rather than have to code up a function to go and look it up somewhere. It's the persistance of data between states that I'm most interested in. The lack of inheritance and polymorphism is bad, but the most important problems are solved.
Here's the problem: in QTP, you don't just have the limitations of VBScript, you also have some severe limitations built in to (or rather "excluded from") QTP.
The class declaration (in a file called User.vbs) looks like this:
The first limitation:
That last function is there because QTP doesn't have the ability to use classes from external vbs files within the function libraries. You must give the test script access to the above function, which has access to the class, in order for your test script to be able to "see" the class.
The second limitation:
There's a bug in QTP where you can't, no matter what you do, use the debugger to step through class functions. That was almost a showstopper for me when doing this, before I learned just how few of me teammates use the debugger. When I found out I was the only one I decided that for the greater good, OO was a good direction to go in, if for no other reason than to make the code seem more familiar to us non-VBScript types.
So, in short, it's possible for us to get what we need to when we confront OO concepts, but we're severely limited by VBScript as well as Mercury. I'll submit defects on both of these issues, but I'm not sure whether I'd be optimistic about my chances. We seem to be the only ones trying to do this. Let me know if you know different!
