Rally is the centralized repository for our… everything. We use it for our Story Cards, Defects, Test Cases… everything. Our test execution platform doesn’t require it, but their simple API makes things extremely easy to access via script. It doesn’t do real Configuration Management - it doesn’t build the software or perform source control. It just helps us manage the process and all of the test data we generate.

I won’t go into the details of Rally’s interface. Our system is plug-in driven, which means that in order to be able to write a plug-in for a particular test case repository, it needs to meet certain requirements. Here are the main points–all of which Rally satisfies:

• A robust Web-Service API (or other method of getting information via script)
• Plain text export on some field where automation code can be posted
• The ability to add and manipulate custom fields
• Results storage, tracking, and reporting

When you go to create a test case in Rally, you do the basic stuff: name, description, objective, regression/acceptance/performance/security, and a few other things. The only thing our framework cares about are the Test Case Steps.

Traditionally, Rally separates each step into an “Input” and an “Expected Result”. You can have 0-n steps in a test case, and they are stored as plain text, as opposed to the Rich Text format supported by most other fields. This is key for our export–we can’t parse through the rtf html-like tags to get to the code.

We’ve essentially named the “Input” step to “Manual Step”, and the “Expected Result” step to “Automated Step”. This way, when you look at a test case, you see all the information you would normally see, and if you just follow the steps on the left-hand side of the screen, you’ll be able to execute any of our test cases maually. We enforce a set of standards as to how detailed these manual steps are written, so that any time an automated test shows a failure, the test can be executed manually to try to reproduce the problem.

So, the QA team gets word of a new feature that they have to test. They create all of their test cases for it, and then those with test automation training can go back through and write the automation steps in parallel with the manual steps. This means we always have a set of manual test cases, which we can then automate incrementally as our bandwidth allows (also, not all manual test cases are good automation candidates).

Once they write the code, they change the “Method” for the test case from “Manual” to “Automated”. We added a custom drop-down field to every Rally test case called “Automation Type”, which includes QTP, Empirix, Pith, API, and several others. Each of these types is a Plug-in, but for the moment, only QTP really works. The rest are stubbed out so that we can develop them as we have time. We intend to add JMeter, AppScan, BurpSuite, and just about any other test harness type that can be executed remotely.

So, we now have a test case in Rally, which has automation code, a unique test case number, and an Automation Type of “QTP”. Now it’s time to execute it.

I’ll go over that next. I promise. There’s a lot to it, and I didn’t want to combine it with this and go over 2,000 words.

Tags: qtp, quicktest professional, test automation, test automation framework, test framework, test management