Our Automated Test Process

It's been a long time since we did regular updates, and that's about to change. We've spent the past 18 months building and solidifying a team structure and a test platform, and it's now time to start talking about it. We intend to release some of our work under the GNU public license, and this background information should help you in making the decision to use it.

We will post a series of articles, which will describe our processes in detail. Then we'll round it out with several examples of how this framework solved most of the problems that typically plague test automation efforts. Comments and suggestions are welcome.

Here's how our team works:

We have four test library developers, each of whom "owns" various pieces of the AUT from a functional point of view. One person may own User admin and role-based security, another may own Workflow Definitions, etc
The library consists of several dozen VBS files, each of which more or less contains a single Class
Each Class represents some sort of functional object in the AUT, e.g. we have a class for "User", which contains members such as "UserID", "password", etc, and methods such as "commit" (like create), "delete", etc
When the AUT changes, our test scripts start failing, and it is the responsibility of one of the 4 core team members to file a defect (in the case of a product error), or update the corresponding library file (in the case of simple product change)
These scripts utilize the classes above in a simplified way--with no references to navigation. That's a key point in this discussion
We, along with the rest of the Quality Assurance team, write test scripts into a centrally located repository (Rally), each of which can be executed from a single server, independently from the test developer's workstation

So, here's an example of how a simple test script would look:

View as plain text

Visual Basic:

'Log in to the application as the super administrator
LoginProfile "SUPER"
'Create a User, and set a few key attributes
Set oUser = NewUser
oUser.sUserID = "inquisitor"
oUser.sGivenName = "Marcus"
oUser.sFamilyName = "Inquisitor"
oUser.sPassword = "qq"
oUser.commit
Logoff
'Log back in as this new user
Login "inquisitor", "qq"
'Check to make sure the page title greets the new user by name
ComparePageTitleWith "Welcome, Marcus Inquisitor"

Things to note:

No reference is made to a URL or test site location in this code--we maintain a list of test sites against which our tests can be executed, and this information is generated and passed to the LoginProfile method at run-time (via Environment variables)
The syntax is a little programm-y, but we've made attempts to make it as unintimidating as possible. It's still a hurdle for people with no programming background, but there aren't any loops or egregious indentation schemes or anything
At no point do you see the word "click" or "mouse over" or "in this field, enter "
The simple object syntax allows you to add and edit attributes without regard to the web form's tabbed interface or to the exact data type of the attribute--the data is mildly scrubbed within the object's commit method
Until the object's commit method is called, no work is being done against the AUT. This allows you to instantiate, for example, an existing object (one that's already in the AUT database) by setting its attributes, then performing another operation on it, such as delete or edit.

This setup allows us to create a number of entities very quickly, very simply, and have these entities interact with each other in ways that test the business logic. By abstracting the navigational steps, we lose the ability to do deeply integrated tests of the creation of these entities (checking input validation and the like), but we get a lot more out of our test automation this way, since it's more oriented toward the interaction of various entities with each other in ways that replicate user actions.

To put it another way, you can easily test how a user interacts with the system throughout the lifecycle, rather than just testing whether or not you can create a user.

This system is well-suited for teams who would like to use Business Process Testing (BPT) from HP, but can't afford it. That's why we wrote it. This allows us to write test cases from Business Analyst-provided Use Cases at a very high level. It doesn't catch some of the details, but it does catch the real problems that prohibit our users from interacting meaningfully with our system.

Next up I'll talk about our standard interface, more detail about how we use Rally, and the queueing system.

Tags: test automation management, test framework, test management

5 Comments for “Our Automated Test Process”

Stefan Says:
June 3rd, 2008 at 1:42 am
Very interesting reading. I am looking forward to read about the queueing system since I am about to implement one of my own.

I like your new site design as well as the content of your articles.

:-)

/Stefan
Reply to Stefan
Basim Baassiri Says:
June 4th, 2008 at 7:32 am
I’m curious to know if a custom keyword driven approach was explored and if so, what were the pros and cons and the reasons why you went with VB way of doing things. I’ve done our automation framework where by we read keywords from an excel sheet and the framework does whats in the excel. Addin test cases is a matter of copying and pasting in excel

Your thoughts?
Reply to Basim Baassiri
Marcus Merrell Says:
June 4th, 2008 at 11:48 am
That sounds pretty interesting. We don’t store out test scripts in Excel, but we certainly could. I don’t see any reason why that approach wouldn’t work.

If I understand your question and process correctly, the pros for starting the way we’ve started are, we have very fine-grained control over the code both within the libraries and within the test scripts. The cons are, the code is not quite as easy to write for testers who don’t have programming backgrounds.

The key is, as I’ll be explaining in future posts, that the source of your scripts, i.e. the repository they are stored in, doesn’t matter. We have plug-ins that utilize a standard API interface to “deliver” the script code to QTP.

As long as the code is formatted and delivered properly, we could, in theory, use something like email to send code to the queueing & execution system. It doesn’t matter how it gets the code, as long as the libraries are associated and the code is well-formed.

Let me know if I didn’t answer your question.
Reply to Marcus Merrell
Alister Scott Says:
June 4th, 2008 at 5:33 pm I’m interested to know if your framework supports negative testing as well. For instance, in your User class what happens if a mandatory piece of information (like sUserID) is not provided? What if you would like to test a scenario to make sure that this piece of information is needed to be provided and that your AUT provides an appropriate error message? Also, I’m interested in what you see are the benefits of your ‘rally’ system over something free like SVN or CVS.
Reply to Alister Scott
1. Marcus Merrell Says:
  June 4th, 2008 at 10:25 pm
  Let me address the Rally question first: Rally is hosted Project Management software, which captures information about features, story cards, use cases, tasks, projects, releases, test cases, and test case results. It does no version control at all, in fact, we use SVN for that, and have integrated some of our SVN calls with the Rally API.
  
  I’ll cover that more in the next couple of posts, and I hope to make it clear.
  
  So, for a negative test, we’ll use a set of assertions to ensure the error message has appeared.
  
  Our developers were kind enough to include a standard hidden element on any page which contains an error message. This hidden field is guaranteed to contain the exact text of any error message that appears on a form. We wrote into our libraries as a small set of assertions. There’s a small catch, that if there is more than one error, only the first will be contained in this hidden field. We decided to live with that for the time being.
  
  Say, for example, you have the snippet like the following. The AUT enforces the User ID, the First Name, and the Last Name all as required fields.
  
  All I do here is leave the User ID out, and the form field gets left blank–we’re very particular about not enforcing much business logic in our code. If you want to create a user with no user ID, we’ve made certain that you’ll be allowed to.
  
  Set oUser = NewUser oUser.sFirstName = "first" oUser.sLastName = "last" oUser.commit
  assertErrorMessageContains "Error: The User ID field cannot be left blank"
  
  What happens is, the first three lines create the User, but only in QTP’s memory. Once the commit method is reached, the UI navigates to the area where the users are listed, clicks the Add button, fills out all the fields for which the attribute is a non-empty string, and clicks the Save button.
  
  The assert method checks the hidden Error Message field to make sure it contains the text provided.
  
  There’s a small problem in that, this test case is limited to English only. Most of our tests don’t have this limitation (I’ll explain that later, too), but like a lot of things we just had to compromise.
  
  I don’t know if I’ve explained this thoroughly enough. If I haven’t, I’m sure I can give more thorough examples of how we do negative testing. It’s one of the things we specifically accounted for in the framework, though sometimes we have to be clever in how we write the scripts.
  Reply to Marcus Merrell