Posts Tagged ‘QTP-Automation’

Mere Bharatiye sathiyon ka swagat hai

Friday, August 3rd, 2007

According to Google Analytics, about 36% of you understand the title of this post. For the rest of you, it is Hindi for “We welcome our Indian friends”.

Over the past 18 months, there we have had almost as many Indian visitors as US visitors. Considering this is a US site written in English, it amazes me that 7500 people in India have visited us and read 25,000 articles (or at least looked at them). I had heard that there is a lot of software development and testing happening in India, but seeing this many Indian visitors makes it real to me. Also making it real to me are daily phone calls and IMs with three coworkers in Bangalore: Ashok, Sridhar and Shiva. They make up 3/5 of our test automation team.

The top six cities that The Software Inquisition receives visitors from are in India: Bangalore, Chennai, Mumbai, Hyderabad, Delhi and Pune. The first US city on the list is New York at number 7.

SI country stats

Tags: , , , ,
Posted in Uncategorized | 3 Comments »

Mercury World - The Report

Monday, October 16th, 2006

Short Story - we had a great time and learned a lot. It probably didn’t suit us as well as it could have (since we’re not currently QC customers–just QTP), but a few of the track sessions were excellent. There was one I would even call “validating” to both our approach to Mercury’s products, and to the direction we’re headed in as a QA organization.

Will and I gave a presentation on the All-Pairs technique, something we haven’t really talked about here but which we’re both big fans of. The slides are up now that the conference is over, and I put a lot of notes in there to fill in the blanks for those who missed the talk.

Long Story

There wasn’t much going on with the conference Saturday or Sunday (registration and some 1-on-1 time with Mercury people), so I sat in the sports book at Bellagio and made modest wagers on various sports teams, enjoying the accoutrements supplied by the generous waitstaff. That is to say, if you’ve got a couple days to kill, go get blitzed at the Bellagio. The drinks are free as long as you’re gambling!

Monday morning was mostly about the big-time speakers, the Mercury CEO, Jared Diamond, and a couple others. At the first track session (I won’t mention which, because there’s no need to call anyone out after the fact), I walked out during the Q&A, wondering whether the presenter(s) was/were idiots or I had just misunderstood. After I got out, the VP of IT for Citigroup was nearby, commenting on how that session was a bunch of BS, and how did it ever make it into the program… So, at least I can say I reached the same conclusion as a Citigroup VP…

The next session was purportedly an examination of Agile methods in conjunction with Quality Center and Business Process Testing. I didn’t see anything in there that told me the presenter had ever really been a part of or studied Agile methodology. He kept referring to “ways to keep your agility in testing”, and “how to stay flexible in light of constant change”, and that’s good stuff, but it’s a far cry from talking about SCRUMs, iterative development, feature-complete delivery at milestones, story cards, etc. It ended up just being the third time we got a demo of Quality Center. I left during the Q&A there too… very disappointing first set of track sessions.

The next morning started great–”Functional Testing of Web Services” by a guy from Mercury. Fantastic session. He covered just about every question I had had about the QTP Web Services plug-in, and got me excited about using it. We have a home-grown tool we use to test web services, and it’s good (I wrote it ;) ), but it’s not as polished as this one. This would also give us the ability to unify the web services test results with the web app test results.

We skipped around the next session (figuring that nearly every one listed would just be product demos we’d seen before–and we were right), but at 2 there was an excellent talk by Ryan English from SPI Dynamics. He covered ~8 different types of security defects common in web applications, along with both automated and manual suggestions for how to test for them. I’m disappointed that the slides aren’t online, because it was a crash course on everything that’s wrong with web software (from a security standpoint).

Wednesday morning we rehearsed our own presentation and watched as Marika Lazi, VP of IT for Bear Stearns, rehearsed hers. It was called “Automation of Edge-to-Edge Testing”, and covered many of the kinds of problems Will and I have when trying to automate our own testing without the suite of QC products. She was sharp as a tack and very realistic in her approach to automated testing. I was glad to see that it wasn’t just a sales pitch by a sales person, but a real stand-and-deliver from someone who sees the trenches every day even if she doesn’t live in them.

After lunch, Will and I gave our presentation. I have to say that the A/V equipment and management at the Bellagio is probably about the best it could possibly be. World class, really.

I think our talk went pretty well overall, though I was disappointed that nobody in the audience had ever heard of the Inquisition! We met a bunch of people afterwards who had some great comments and interesting insights, and my hope is that we hear from one Martin O’Brien again, who showed us an amazing Neural-Network-based test case processor-generator that could execute billions of test cases in short amounts of time. He left us with a million questions, but no contact info.

Did I mention the amazing food? At lunch every day and at dinner Monday and Wednesday, I enjoyed some of the finest dining I’ve ever been around… and not just because it was free. It was great all the way around. Mercury sure knows how to throw a party.

I could talk about the gambling, the drinking, the fact that I’ve officially been to both a Vegas nightclub and a Vegas ultra-lounge (and my company didn’t pay for a dime of it!), the ~20 hours of football I watched, only sleeping about 12 hours from Sunday to Thursday, actually winning money at poker, and all sorts of other stuff, but I’ll just leave it in Vegas. I had a great time, didn’t spend nearly as much of my company’s money as I could have, and didn’t even put an ounce stress on any marriage vows. All in all, if you have to spend a week attending 20 presentations on a fairly dry topic like automated testing, I can think of worse ways to spend said weeks.

Hope everyone got as much as I did out of it, and I hope to see you all at the Venetian next June!!

Tags: , , ,
Posted in Uncategorized | 4 Comments »

A CGI Script to Launch QTP Tests

Friday, September 29th, 2006

This script is tightly coupled with the QTP.pm module in a previous post.

Note that the action refers to "ExecQTPTest.plx". I'm using the PLX extension because that's how you get IIS to use the Perl DLL instead of the executable. For some vague reason I've had a ton of trouble using the executable--I believe it has to do with Win32::OLE. It ends up executing the QTP code twice through, and I just don't know why. After much tinkering, I found the PLX solution and it seems to work much better.

Like I said before, I don't know of any reason you couldn't use Apache as your web server, but I haven't actually tested it.

There are a couple pieces of information missing from here: what is the big textarea called "XML"? How does the stylesheet get passed in? What's all this XSLT nonsense?

The answers for some of these questions are already up, some others only drafted, but I've just got to face the fact that we have a lot of information to put out there, and it's a slow process to turn it from in-house slop-code to polished, ready-for-primetime Open Source software. Our apologies in advance for the inevitable delays...

We launch by pointing a web browser to this URL:

http://servername/qtp/ExecQTPTest.plx?testname=LoginTest&dest=outputDirectory/results.html&redirect_results=1

After the test is finished (give it a minute or two), the browser is redirected to the html output generated by QTP.pm.

View as plain text
PERL:
  1. use strict;
  2. use QTP;
  3. use CGI qw ( :standard );
  4. use CGI::Carp qw ( fatalsToBrowser );
  5. use FileHandle;
  6.  
  7. # PARAMS TO EXPECT
  8. # SiteName     - unique name of site to deploy
  9. # testpath     - path of the test to be run
  10. # visible      - whether or not to run QTP in "Visible" mode
  11.  
  12. my $results;
  13. my $quick_results;
  14.  
  15. my $q                = new CGI;
  16. my $default_test     = 'C:\dev\qtp\LoginTest';
  17. my $test             = $q->param('test');
  18. my $redirect_results = $q->param('redirect_results');
  19. my $stylesheet       = $q->param('stylesheet');
  20. my $args =
  21. {
  22.     testpath => $test,
  23.     visible  => $q->param('visible'),
  24. };
  25.  
  26. my $xml = $q->param('xml');
  27. if ($xml)
  28. {
  29.     my $xml_file = 'QTP_Script.xml';
  30.     my $ofh = FileHandle->new(">$xml_file");
  31.     print $ofh $xml;
  32.     $ofh->close;
  33.     my $qtp = QTP->new($args);
  34.     $qtp->generate_new_test($xml_file);
  35. }
  36.  
  37. if ($test)
  38. {
  39.     my $qtp = QTP->new($args);
  40.     $qtp->run_test(Test => $test);
  41.     $quick_results = $qtp->{last_status};
  42.  
  43.     my @time_stamp = localtime;
  44.     my $file_identifier = $time_stamp[4] . $time_stamp[3] . 'a';
  45.  
  46.     $results = $qtp->get_results($stylesheet, $file_identifier);
  47.  
  48.     if ($redirect_results)
  49.     {
  50.         if ( -e $results )
  51.         {
  52.             print redirect($results);
  53.             exit;
  54.         }
  55.     }
  56. }
  57.  
  58. print $q->header;
  59. print "<body bgcolor=#FFFFFF text=#000000 link=Blue " .
  60.             "vLink=MediumSpringGreen aLink=Purple>";
  61.  
  62. print $q->h1( { -align => "CENTER" }, "Test Launch Console");
  63.  
  64. my $get_time = localtime;
  65. print $q->p( { -align => "CENTER" }, $get_time );
  66. print $q->br;
  67. print $q->hr;
  68. if (-e $results)
  69. {
  70.     print $q->h3("Last Run Results: " . $quick_results),
  71.           $q->br,
  72.           $q->a({-href=>$results,-target=>$results},
  73.             "View Results");
  74.          
  75. }
  76.  
  77. print <<HERE;
  78.     <form method=GET action="ExecQTPTest.plx">
  79.     <h3>Test</h3>
  80.     <input type=text name=test size=80 value=$default_test />
  81.     <!-- <input type=text name=starting_url size=80><p> -->
  82.  
  83.     Stylesheet:
  84. <select name=results_stylesheet>
  85.         <option value="C:\<path-to-QTP>\dat\PShort.xsl" selected>Short
  86.         <option value="C:\<path-to-QTP>\dat\PDetails.xsl">Details
  87.         <option value="C:\<path-to-QTP>\dat\PSelection.xsl">Selection
  88.     </select>
  89.     <input type=checkbox name=redirect_results value=1>Redirect to results<p>
  90.    
  91.     <p>
  92.     <input type=submit value="Submit"> * <input type=reset>
  93.     <h3>XML:</h3>
  94.     <textarea name=xml rows=30 cols=100 value=$default_test></textarea>
  95. </form>
  96. HERE
  97.  
  98. print $q->end_html;

There may be bugs in there, because I had to strip out some company-proprietary stuff, so let me know if you have problems.

By the way, this is a good solution for executing tests in a targeted, specific manner, but we only use this rarely nowadays. The new methodology involves a queueing system, test harness-independent test execution, web service result reporting, and dynamic QTP script generation. It's awesome, but it's not ready for public consumption yet.

Tags: , , , ,
Posted in Uncategorized | 6 Comments »

Using XML::XSLT to make your test results pretty

Monday, September 25th, 2006

This question came up at QAForums recently:

Q: Is it possible to export QTP results to HTML format?

A: Mercury has provided nearly everything you'd ever need to make this happen, and it's highly customizable if you take the time to learn XSLT.

XSLT is a simple technology, used to transform XML documents into other kinds of documents. QTP comes with three XSL files, which transform the Results.xml file into perfectly readable HTML code.

These stylesheets live in the <QTP root>\dat\ folder (on mine, it's C:\Program Files\Mercury Interactive\QuickTest Professional\dat):
PShort.xsl
PDetails.xsl
PSelection.xsl

When the test run report is generated, it's stored in a directory something like this:
<test folder>\Res1\Report\Results.xml

To see it, you have to add the following line below the XML declaration in Results.xml, so it looks like this:

View as plain text
XML:
  1. <?xml version="1.0"?>
  2. <!-- This assumes you've made the PShort.xsl file available via a web server!! -->
  3. <?xml-stylesheet href="http://localhost/qtp/PShort.xsl" type="text/xsl"?>

Then view the Results.xml file in IE (which has an XSL engine built into it).

What we do is, we use an external XSLT transformation tool called 'xsltproc' (which comes from the LibXML2 suite at XMLSoft) to generate an HTML file, which we then upload to a server that archives all our test result information. That way nobody needs Mercury's Results Viewer app (that blasted thing takes you through too many clicks to get what you want, and it doesn't remember your filter preferences from one session to the next... GARR!!!!)

This is much better for everyone on my team.

So, it's not difficult, and you have everything you need already. Now if only there were a way to have QTP insert that stylesheet declaration by default into every test...

Tags: , , ,
Posted in Uncategorized | 6 Comments »

Integrating QTP with Non-Mercury Products: Some Code

Tuesday, August 22nd, 2006

Continuing the discussion about QTP automation, I'll give a little bit of the code I use to manage the application. Short version: I treat it like any other object. The calling script just requests a QTP (COM/Win32::OLE) object, then tells it which test to run, passing in some options along the way. It's pretty Perl-advanced, and brings in a lot of other technologies, so pay close attention. Or, just copy it into a file and call it like a black-box.

This first snippet is in a file called QTP.pm. It is just a package that gets blessed as a 'QTP' object. The cgi script is the second snippet, which invokes the class and runs operations on it.

Update: At Bob's suggestion, I've cleaned up a few things and added comments. I've got more to do, and it's still in Perl, but hopefully this will make it slightly more than "Read Never".

Note the places commented "TODO". This indicates you need to replace the value in <> with values appropriate for your environment.

View as plain text
PERL:
  1. package QTP;
  2.  
  3. # Standard Perl/CPAN modules
  4. use Win32::OLE;
  5. use Win32::OLE::Variant qw(:DEFAULT nothing);
  6. use FileHandle;
  7. use File::Spec;
  8. use File::Copy;
  9. use Data::Dumper;
  10. use XML::XSLT;
  11.  
  12. # Set OLE warn level to "warn", not "fail" on minor errors
  13. $Win32::OLE::Warn = 3;
  14.  
  15. # constructor
  16. sub new
  17. {
  18. my ($proto, $args) = @_;
  19.  
  20.     my $class = ref($proto) || $proto;
  21.  
  22.     # $args is a hashref containing settings for the constructor
  23.     my $self  = bless {
  24.  
  25.         # visible defaults to false--QTP runs faster when it's off, but
  26.         #  is useful when debugging
  27.         visible  => $args->{visible}  || 0,
  28.  
  29.         # the path to the Object Repository (optional)
  30.         ORPath   => $args->{ORPath},
  31.  
  32.         # path to the test
  33.         testpath => $args->{testpath},
  34.     }, $class;
  35.  
  36.     # Get an instance of QTP through Win32::OLE
  37.     eval
  38.     {
  39.         $self->{qtp} = Win32::OLE->new( 'Quicktest.Application', 'Quit' );
  40.     };
  41.     die $@ if ($@);
  42.  
  43.     return $self;
  44. }
  45.  
  46. # gather information about the test sites. This should be abstracted to an
  47. #  external module, as it is not QTP-specific
  48. sub get_site_info
  49. {
  50. my $self = shift;
  51.     my $site_name = shift;
  52.  
  53.     # Create a user agent object
  54.     use LWP::UserAgent;
  55.  
  56.     $ua = LWP::UserAgent->new;
  57.  
  58.     # Create a request
  59.     my $req = HTTP::Request->new(GET => 'http://<hostname>/'.
  60.         $site_name . '.txt'); # TODO
  61.     $req->content_type('text/plain');
  62.  
  63.     # Pass request to the user agent and get a response back
  64.     my $res = $ua->request($req);
  65.  
  66.     # Check the outcome of the response
  67.     if ($res->is_success) {
  68.         my @results = split /\n/, $res->content;
  69.         for my $line (@results)
  70.         {
  71.             # The information comes back in tab separated key-value pairs
  72.             my ($key, $value) = split /\t/, $line;
  73.             $ENV{$key} = $value;
  74.         }
  75.     }
  76.     else {
  77.         # if there was an error
  78.         print $res->status_line, "\n";
  79.     }
  80. }
  81.  
  82.  
  83. # last_status is stored after the most recent test run
  84. #  would like to get this from the QTP application itself,
  85. #  but normally it's already been destroyed by this time
  86. sub get_last_status
  87. {
  88.     my $self = shift;
  89.     return $self->{last_status};
  90. }
  91.  
  92. # This method performs the bulk of the work
  93. #  Because of the limitations of Win32::OLE, the application
  94. #  object gets launched and destroyed in the course of this method,
  95. #  so any information I want needs to be gathered while it's still hanging
  96. #  around. The other methods about getting status and test results just
  97. #  query members of the class that are captured in this method.
  98. sub run_test
  99. {
  100. my $self = shift;
  101.  
  102.     # This hash holds the arguments
  103.     my %args = @_;
  104.  
  105.     #Test Definitions here - these tests become hard-coded, so we don't
  106.     #  recommend using them. This is deprecated, and I'll post the update
  107.     #  the moment I refactor it out.
  108.     my $AllPredefinedTests =
  109.     {
  110.         Login =>
  111.         {
  112.                 Path  => 'C:\<path>', # TODO
  113.                 RO    => 1,
  114.                 Arg   => 0,
  115.         },
  116.     };
  117.     my $test = $args{Test};
  118.     my $testDetails;
  119.  
  120.     # Check to see whether we have a valid path or not (whether in the
  121.     #  predefined tests or in a path sent in through the method invocation)
  122.     if (exists $AllPredefinedTests->{$test})
  123.     {
  124.         $testDetails = $AllPredefinedTests->{$test};
  125.     }
  126.     else
  127.     {
  128.         if (-e $args{Test})
  129.         {
  130.             $testDetails =
  131.             {
  132.                 Path => $args{Test},
  133.                 RO   => 1,
  134.                 Arg  => 0,
  135.             }
  136.         }
  137.     }
  138.  
  139.     # Tell the QTP object to launch the application according to the scrubbed
  140.     #  data from above
  141.     $self->{qtp}->Open(
  142.             $testDetails->{'Path'},
  143.             $testDetails->{'RO'},
  144.             $testDetails->{'Arg'} );
  145.     $self->{qtp}->Launch;
  146.  
  147.     # Make the application visible or invisible
  148.     $self->{qtp}->{Visible} = $self->{visible};
  149.  
  150.     # RunResultsOptions is a separate COM object.
  151.     #  No later reference to it is needed, so we instantiate it here and let it die.
  152.     #  This object allows us to set the Test Results path to whatever we need it to be
  153.     my $qtResultsOpt = Win32::OLE->new( 'QuickTest.RunResultsOptions', 'Quit' );
  154.     $qtResultsOpt->{ResultsLocation} = '<results path>'; # TODO
  155.  
  156.     # Execute the test
  157.     $self->{qtp}->Test->Run($qtResultsOpt);
  158.  
  159.     # capture the test run results for later use
  160.     $self->{last_status} = $self->{qtp}->Test->LastRunResults->{Status};
  161.  
  162.  
  163.     return $self;
  164. }
  165.  
  166. # Since we put the results in a place we specified, it won't get rolled up into
  167. #  the 'Res1', 'Res2', etc., folders. Now we know where to get them and can
  168. #  transform them however we want
  169. sub get_results
  170. {
  171. my $self = shift;
  172.  
  173.     # $args is a hashref
  174.     my $args = shift;
  175.  
  176.     # by the end of this code block, we'll have the name of a stylesheet.
  177.     #  If it's invalid, the XSLT processor will pick it up. We could go crazy
  178.     #  trying to determine whether or not it's invalid, so we'll punt
  179.     # Default to a known good stylesheet (PShort.xsl, distributed with QTP)
  180.     my $stylesheet = $args->{stylesheet} ||
  181.         'http://<xsl doc server>/PShort.xsl'; # TODO
  182.  
  183.     # put together the results path, with the default pointing to the same
  184.     # directory as above
  185.     my $results_path = $args->{results_path} || File::Spec->catfile
  186.     (
  187.         '<results path>', # TODO
  188.         'Report',
  189.         'Results.xml'
  190.     );
  191.  
  192.     # This invoked the LibXML2 app 'xsltproc'. In other places in our library,
  193.     #  we've replaced this with XML::LibXSLT so we don't have to launch
  194.     #  an external process. I'll refactor this to use that one day.
  195.     my @results = `xsltproc $stylesheet $results_path`;
  196.     my $results = join '', @status;
  197.  
  198.     # if xsltproc hit an error message
  199.     ($?>> 8 ) and die "xsltproc returned the following status code: $results";
  200.  
  201.     return $results;
  202. }
  203.  
  204. # A little Perl 6 before Perl 6 is available
  205. sub slurp
  206. {
  207.     my $filename = shift;
  208.  
  209.     # concatenate the file into one large string
  210.     open(FILE, "<$filename");
  211.     my $return;
  212.     while (<FILE>) { $return .= $_ }
  213.  
  214.     return $return;
  215. }
  216.  
  217. # don't forget this line:
  218. 1;

Tags: ,
Posted in Uncategorized | 3 Comments »

Integrating QTP With Non-Mercury Products

Friday, June 9th, 2006

Jeff posted a comment about my implication that we have integrated QuickTest Pro into our build processes. I thought I'd elaborate on that a little.

The first thing I should say is that it's not perfect, that we cobbled it together in between developing our testing strategy for our current release and writing regression test cases for some legacy products. We've got plans to improve it, we know pretty much what's missing, but, as with most meta-testing projects, IT WORKS.

The second thing I should say is that we're trying to follow an Agile methodology here, and while we haven't worked it all out, we at least use some of the terminology. If you see some term you're not familiar with, it probably comes from that school.

I'll start with an overview of the process, then discuss some of the steps in detail:

  • At about 2am, CruiseControl starts what we call the "Nightly" build
  • The Nightly build is just an Ant task that starts by building the software (it's a Tomcat webapp, but that's not critical to the process discussed here)
  • A blank test database is created
  • The build is unit-tested, then deployed to a test server
  • QuickTest is called upon to run a test that just verified the most basic things: a user logs in, then several objects are tested for existence and properties
  • The build is declared "Valid", and a more comprehensive suite of tests is run (via QuickTest Pro and other automated testing tools
  • After each test is run, test results are reported back to Rally (our Agile project management software)

As you may see, these processes take the place of many Mercury products (Quality Center, IT Governance, etc.). We're a small shop and can't afford all those to