Saturday, January 13, 2007

The Low Cost Test Automation Alternative

I won’t rehash the entire history of automation or the well documented reasons why most of the companies who try automation abandon it in the first year. It’s a good read and I suggest that people new to QA do a little research on their own.

What I will try to do is identify the single most significant reason most automation projects fail, and then offer an alternative. I want to warn you ahead of time that what I am proposing is not the ‘right way’ to implement automation, but I do believe it has a place with ‘young’ QA Departments under pressure to roll out an automation solution.

The Single Most Significant Reason Automation Fails:
Automation is a Development Project.

Successful automation is based on the same principles as successful development projects: Reuse, Extensibility, and Maintenance.

If you skimp on the design and quality of your company’s software development, you end up with crappy software. Once again the business cost of doing a poor job designing software is well documented so I won’t go into it here.

If you know you’re not qualified to design and implement a robust, reusable, extensible and maintainable QA Automation Framework, what can you do when the company refuses to assign development resources to the project and it ends up being all you?

You punt. In American football the punt occurs when things are looking badly for the team and the decision is made to give up your current drive and kick the ball to the other team. The idea is to buy some time to regroup instead of taking a big risk and ending up in a worse situation.

All Management wants is ‘Automation’. It’s all very clear here on this checklist.

  • Requirements
  • Product Design
  • Product Development
  • Ship to QA
  • Automation
  • Profit

As you can see, you are the only thing standing between the company and its profit. Not the best time to try and explain how automation is really a development project.

Since we have established that it is unlikely you are going to be able to implement Automation the ‘Right Way’. Don’t take the big risk; buy some time for your department by using Automation to enhance your manual testing. After’Automation’ is checked off the list, you will have the time (and now experience) to take a stab at a better solution.

Solution – Record and Playback:
As we all well know, ‘Record and Playback’ is the single most vilified method of automation. Its critics say it is too simplistic, it is not reusable, it is not extensible and it is certainly not maintainable. All the scripts generated are useless once the UI Changes! Sounds perfect for our project!

The idea is to reduce the number of hours spent retesting the entire application every time a change is made. We all have been burned when some obscure code change breaks a feature in another part of the application. We can use this method to automate all those mundane tasks and let us focus on the more important areas.

Step 1 - Make Test Cases
In order to do any automation, you have to define the test cases you want to cover. Use whatever tool you want (Excel works fine) and list out all of your test cases. Each test case should meet the following criteria.

  • Test cases must be written so that they are atomic and each one has only a single possible success path.
  • The positive and negative test cases must be separated.
  • The test case must have the same result every time it is manually run.
  • Avoid hard tests; the power of this method is in automate the mundane tests. If it is hard to test something, it will be even harder to automate it.
  • Don’t try to identify everything at once. Start out with just 1-2 test cases for every major feature. You want to cover all the major areas, then fill in the gaps later.

Step 2 - Record Manual Testing
When you begin the manual testing, sit in front of the automation tool, turn on the recorder and record yourself as you execute a single test case. Once you have completed the test case and it passes, then you save the script and move on to the next test.

Step 3 -Playback the Tests
When a new drop of the product makes it to QA (Patch, DevDrop, Daily Build) a QA Engineer sets up the test environment (installs new version, resets Database, etc..) and starts the automated tests. After the test starts they move on to their normal manual testing.

Step 4 –Failing Tests
After all the tests are executed a QA Engineer needs to review the report. Any failing tests should be retested manually by QA. If there is a defect it is entered, if the test is broken (UI changed, new functionality) then the QA Engineer DELETES the test script, and re-records it again manually.

Notice I said delete the script. It is important to avoid the temptation to start modifying the scripts. It will always be much faster (and safer) to re-record it. Eventually you will get so sick of fixing the same script issues that you will want to add some helper scripts, manage variables, and load values from a config file. Now you are spending all your time building a crappy framework and not testing.

Step 5 – Present Results to Management
At this point, I hope you see the benefit for yourself. You are not spending a significant amount time on the automation process, so you have more time for testing. The time previously spent retesting the common features, is now spent tracking down the more difficult defects.

As far as management is concerned you have done a great job. You added automation, you have X number of test cases and hopefully you are actually getting more QA done on the project than before.

No comments: