We all get raises at the same time of year. The buzz is big around the time it happens… whispers of flat 2% or 1% across the board, rumors that “we’re all getting screwed”… you know, standard stuff. They are entered into our payroll software before they’re communicated to us, but we don’t have access, and there’s always a 7-10 day period between the decision and the communication. What, do we have to wait until we get the check to see what it’s going to be?

Well, in fact, no we don’t.

I found what I would like to call a “soft hack”… no injection or script-insertion, no session hijacking or privilege escalation. I didn’t even need a proxy to manipulate the parameters. I just poked around the site that stores our benefits information. See, the benefits information interfaces with the payroll information, which means the numbers I need are already in there. I know that because I’m a manager and I already saw my employees’ increases, but comp numbers are hidden from individuals. So mine is in there somewhere… somewhere.

I look at withholding - if I see how much they’re taking out of the check for taxes of social security, I’m home free. I can do that kind of math. No luck–that’s hidden too. Medical insurance? No luck–it’s a flat rate. Flex spending? Nope–another flat rate.

401(k) contributions? That would do it, right? If I’m withholding 6% and they print the dollar amount, the amount will represent 6% of the new number, right? Of course not–they just print the percentage.

Finally I look at our Life Insurance coverage amounts. My company is awesome–just for working here we get automatic life insurance, gratis, equal to TWO TIMES OUR ANNUAL SALARY.

So, the number listed there–I divide it in half, and there it is, my new salary. Thanks, boss!

Because I’m a software tester first and foremost, I want to verify my methodology: I’m able to look at my benefits “as of” arbitrary dates, so I set the date back a month or so, divided the life insurance in half, and boom… the old salary. We have a soft hack.

And that’s what worries me: even when they don’t give you the information, they give you the information. The thing about this hack, in light of my recent acquisition of the web app penetration testing unit, is that no tool, not AppScan or WebInspect, not burp suite or paros or httprint or nessus or nikto or IEWatch HTTPWatch. What is my AUT trying to tell the world, that I don’t want to tell them? Some little nugget sitting out there in plain site, waiting for the right person to come along and do some second-grade math?

It looks like the best security testing tool you possess is your own mad desire to get the data. Conjuring this motivation is difficult: I’m definitely more interested in my new paycheck numbers than I am in whether some theoretical user can escalate his permissions to perform admin tasks. It’s almost like I need to have a Stanislavski moment before I start this hack-testing: imagine I’m sitting in a dark room somewhere, logged into the target site using “rsmith/password”, and I’m not about to let rsmith’s meager permissions get in my way. I know that information is on that server, and they’ve opened up port 80 for me to get to it. There’s only a thin layer of imperfection sitting between me and the database. It’s only a matter of time.

Should I coin a new term? “Method testing”? Those words are all over these Internets, but they mean something quite different from what I’m spelling out here.

You heard it here first.

This XML file is then copied to a directory, accessible either via HTTP by the queueing system or a web browser. Anyone in the engineering department can go to this page and, using IE/Firefox's built-in XSLT processor, view it as a nice little HTML table. It shows a happy face if the site is up, or a Jolly Roger if the site is down.

But the queue server only cares about the data--and it's awesome having the same tool performing multiple jobs.

If you look at it with a browser, you get this:

If the queue is looking at it, it sees this:

The data file that drives the heartbeat script also contains a list of usernames and passwords by role. If you need to test something manually, you go to this site, find the URL you want, then hover over the name of the site. A JavaScript popup appears, listing the roles, usernames, and passwords for the site. This allows you to login as, say, an Employee, or as a Super Administrator, to any site on the list:

This information is also made available to the queue system. It allows us to use a line of code like this:

LoginProfile "SUPER"

or

LoginProfile "RECRUITER"

Either line of code will log you in to the site via a Profile or Alias as defined in the data file--allowing you to reuse your test script against any site for which you have provided login information by role. We never hard-code usernames or passwords into our scripts. Further, our engineering dept has a set of default sample data that we use for most test sites. If you just throw together a site using the default database, and want to execute our tests against it, you simply have to pass the URL to the test queueing script. This script will look for a match on the heartbeat page, and not finding one, it will take these defaults (and still check this site for availability before executing the tests).

Our AUT contains a flat text file, accessible to anyone, which gives the build number, the time/date of the build, several non-critical global options. If you notice from the screen shot, the build number is there (useful for writing up or verifying defects, the time of the last build, the product version, and which applications have been activated.

Lastly, and possibly most importantly (since our product is actually a suite of products), this status page also includes a list of which sub-products has been "activated" or "licensed" for this deployment. These are indicated by the checkboxes on the right, and as "1" or "0" in the xml. Users can go to this page any time to see "what site can I use to test this Recruiting feature against?"

It also allows us, when asked to execute a test suite, to execute or skip a test case if the target product isn't activated. The script that adds jobs to our test queue looks at these attributes for the requested test site, then performs an "eligibility check" on each test. If the test case is within the "Recruiting.Administration" package (in Rally), but the test site does not have the Recruiting app available, this test case will be declared ineligible, and will be skipped. This results in very few false positives, since the test case will never be run against a site that can't handle it.

A lot of information, but this little set of perl/xml/xslt scripts has become the backbone of our little operation. If I can pretty up the code and document it better I'll post it some day. You know how things go around here.

This is also one of the reasons we haven't been posting much. With new responsibility comes new drains on time.

The issue we’re struggling with now—and think we’ve solved—is about how to satisfy management that our numbers are accurate, and how to provide numbers that not only talk about server response times under stress, but also reflect the real end-user experience times. For years, we've used e-Load to run a bunch of tests every week, then reported that “all pages return in under 2.7 seconds”. Every user in the field, every manual tester, every single person who ever used our software knew that was BS—because the pages themselves take up to 9 seconds to render, with no other load on the server.

Here's the problem: AJAX and client-heavy JavaScript make the server response time numbers less useful than they used to be. Empirix has no ability (that we know of) to measure the amount of time between when the page is done being downloaded to the client machine, and when all the HTML, JavaScript, and images actually finish rendering. Therefore, my directive is to produce numbers the executives could a) believe, b) have confidence in, and c) present to customers as "writ in stone"--including this new kind of stat.

I think we’ve solved it, but now we need to put down our pencils and look around... we need now to ask the question: are we on the right track, or are we full of it? Does anyone out there have a solution for testing server capacity and response time under load and the ability to measure comprehensive client page load times simultaneously?

My background, as you would gather from this site, is all in functional testing. We’ve created a unique solution for automated functional testing, but it doesn't help us one bit in determining how our AUT performs under stress. That's where we had to get creative, and to get educated.

We needed a way to execute a few tests by hand while we simulate 1000 users hammering the site with requests. That brought us back to our core strengths: our functional tests are executed on a real web browser, not just through a proxy or through multi-threaded HTTP processes… so, it occurred to us that they could give us our client-side execution numbers, if only we had a way to guarantee that our numbers are accurate.

Once again, the flexibility and generosity of our engineering department has saved us. Our AUT developers injected several timing AJAX calls into every page load served to the user. When the page is finished downloading, an event fires that says "Page Download Complete", complete with a count of the number of milliseconds it took. When all the objects on the page, HTML, JavaScript, images, are finished loading, the "please wait while page loads" DIV fires off another event, "Page Rendering Complete", with the timing included. These numbers are stored in our AUT database, by page name, and exportable to csv format.

What we are left with is a set of data that tells us the following:

myDashboardPage: 247 ms to download, 3893 ms to render

Huh, so before, when we were just using the proxy playback method of perf testing with Empirix, it told us that this page took only 247 ms to download. Meanwhile, our users were seeing 3.8 seconds. We were off in our report by approximately 1576%. Can you see why nobody believed us?

What we do now is, we fire up 1,000 VUs with Empirix, all running tests and hammering the server. At the same time, we use our queuing system to launch our ~500 automated functional tests (4-5 at a time), and have the real-time client-side rendering numbers reported in Excel on a page-by-page basis.

We believe this is the right approach to take, but we still have some concerns:

• Not having employed or interacted with any experts in performance testing on any level, I don’t know if this is brilliant or just something that sounds-good-but-doesn’t-work-like-I-think-it-should
• Crunching the numbers I get into a report palatable to the VP-level and above
• Ensuring that the Empirix (soon to be JMeter) scripts we develop are doing what we want them to do
• Ensuring that these scripts are the right scripts to run
• Using the Empirix numbers in concert with the Client-side numbers to extrapolate information about server (and cluster) capacity, browser compatibility, implications of various configuration changes, bottlenecks, and about a hundred other things—I think this data is trying to tell us what we need to know, I’m just not a good enough statistician to know how to get it out
• How do we know when we’re “done” with a given release? We’ve found the bottlenecks, we’ve told the VPs what they wanted to hear… my instinct is to keep pushing, even though we think we’ll meet all SLAs, but maybe the SLAs only meet the minimum and we want to do more. (This would be a good problem to have, of course)

Anyway, these are my preliminary thoughts on the subject. We're making the transition from Empirix to jmeter, which is pretty exciting. As Scott Barber says, "the craft isn't about the tool". We're just starting out on this task, but I think we're off to a good start.

Thoughts? Ideas?

If you aren't using SIFL, change the first line to your preferred means of opening a browser and navigating to news.google.com.

This will open news.google.com and get the first two headline divs using the Div add-in and WebElement and show you how long it takes to execute. For me, the first two times using the Div method took 2.12 and 0.86 seconds. The second two checks using the WebElement method took 4.08 and 4.05 seconds.

The reason the first time using the WebExt takes longer is it has to attach the javascript for the Div class to IE. After it does that, future executions are much speedier.

In conclusion, this is the long, boring way to say that WebElement is slow and Web Extensibility objects can be much faster.

At the time, I said this function was only a few lines long. I was wrong. Somebody with more javascript experience could probably golf this down to a few lines, but my version is pretty long.

I found the meat of this function at permadi.com and modified it for my own use.

I'll break down some of the features that were discussed.

IDE Improvements

• Adding a TODO pane populated by comments prefixed with TODO: .  This is similar to what Test Design Studio and many other IDEs do.
• Dynamic Surrounding - I'm unsure about this feature, as I haven't heard the term Dynamic Surrounding before.  The presenter indicated that it basically meant auto indenting code within If statements and loops, but QTP's IDE already does that, so I must have misunderstood.  Maybe this is something like the surround snippets in Test Design Studio.  HP would do well to include something like that in QTP.
• IntelliSense Improvements - three areas where IntelliSense will be improved
  • Adding second-level IntelliSense -- example given was if an objects property is also an object, then IntelliSense will be provided for that object as well.
  • Inner Object IntelliSense.  So you won't lose IntelliSense after you type WebEdit().Object
  • Com object IntelliSense.  IntelliSense will be provided for any COM object defined with createobject()
• Toolbar Customization - You will be able to customize the toolbar to include buttons for launching various other applications.  I think this may be something along the lines of Test Design Studio's external functions.

Report Improvements

• Export reports to PDF and Doc files
• Define standard report formats
• Embed your own images and screenshots in reports
• Jump to a script line in QTP from a report

Quality Center

I'm not a Quality Center user, so I don't know what some of these features mean.  I'm just transcribing them for those of you who do.

The presenter said that Quality Center Integration is the main area of focus for Atlantis.  QC Atlantis will release at the same time as QTP Atlantis.

• External Resources Module in QC. Allows you to manage QC integrations.
  QC will understand what the assets of QTP tests are and how to manage them
• Use QC’s internal versioning to version tests and assets
• Comparison tool, compare QTP tests and assets to choose which one to use
• Add sharing of assets between projects
• Manage QC from within QTP
• From with QTP, open a specific QC asset, including specifying the revision from QCs version control
• When you open an asset in QTP, it will automatically be checked out from QC

I look just like this, except I'm not wearing glasses.

He showed SPI's SQL Injector tool, and I could feel the fear coming from part of the audience. I suspect he is going to ease their fear by telling them how they can buy his product.

1:12p - Now a blindfolded tightrope walker is going over the stage while the zombies below try to get high enough to get at him. Presumably to make him part of their zombie army.

1:14p - Oh no, he slipped...but he seems to have grabbed on to the tight rope with his hands. That was a close call.

1:16p - A bunch of preppy types in orange shirts and khakis are dispatching the zombies, and somehow our tightrope walker has shed his blindfold.

1:17p - David Gee just took the stage. He says that whole thing was some sort of metaphor.

1:20p - David Gee says this is going to be the biggest and best HPSU ever, and he seems very thankful to a bunch of sponsors.

1:21p - The first speaker is Tom Hogan, senior vp of software at HP.

1:24p - I'm drifting away...too much marketing speak...

1:26p - Moving on to statistics now. All the statistics seem to spell doom for every business ever.

1:28p - Kids these days...they don't use email...they use text messaging

1:29p - battery is running low and i left my power cord in the hotel room.

1:49p - I got the power cord. Now that I'm back, they've moved on to Q&A. One question on the board, that I want to hear the answer to is why their support site sucks. I bet he doesn't answer that one.

1:51p - Addressing support:
- They are trying to replace a bunch of people they lost in the transition. I wonder why they lost people. Do they not know how to hang on to their talent?
- They are throwing money at the support site.
- Getting some new helpdesk software.
Sounds like a non-answer to me.

2:02 - This part is wrapping up and none too soon. This hour has been largely free of actual content.

I don't know if it qualifies as live blogging, but I will be posting a few updates throughout the week about goings on here.

UPDATE:

You can also read coverage of HPSU 2008 on loadtester.com. They also have a HPSU bingo card for those attending. One of the squares on the card is in the title of my presentation: Web 2.0.