Automated testing is a valuable tool that can provide a great return on investment if used effectively. At MERAK, there are several strategies we use with our Agile projects:
Plan from the start
Plan for Automation at the story estimation stage. This stage helps identify what should be automated, whether there will be smoke testing of builds, and if regression, load/stress tests are necessary. When these important pieces of the puzzle are known from the start, unexpected delays can be avoided and appropriate testing time can be allocated. We recommend 10 to 20% of testers’ time be allocated for automated testing in each sprint.
When creating an automated test, a tester may only focus on the story being tested and not consider any other stories or sprints. Although it’s not always possible, automated tests can often follow an Agile approach where the Automated Test scripts are built incrementally similar to the application under test. The result is that at the end of the sprint cycles, you have a fully formed Automated test that assesses the fully-formed application.
The final decision to automate should be at the sprint stage. It may not be until a tester writes test cases that it is clear that a user story is a good candidate for automation. Conversely there could be hidden complexity that is too difficult or time consuming to automate.
Although the iterative development appears to be modular, quite often the code base is common across multiple sprints so regression testing is important and automating it becomes almost standard activity. Multiple sprints also mean multiple runs of the automated script which quickly provides a return on the investment and quite often, the smoke test may also use automated regression tests from the previous sprint. With a new build every three weeks in Agile development (or more, if fixes are deployed in a sprint), developing an automated smoke test is often a standard activity. If there are issues with a build, an automated test can redirect the process back to development faster than manual testing.
Automated testing sprint
With stress and load testing, exclusive use of the test environment may be required which means upfront planning is critical. Any manual users could interfere and possibly skew the results. If multiple test environments are available, Automated tests can be run concurrently.
Think outside the sprint
A MERAK client had an application that had a drop-list scheme that contained thousands of permutations. Automated testing checked the permutations accurately and in less time it would take to test manually. This was also a high risk, core function for the client. The script was planned for and developed outside of the sprint timeline, tested in one sprint, and when the functionality was ready, run during a later sprint.
Think beyond the project
While the initial output of time to automate testing may seem to outweigh the benefits (particularly if it would take less time to test manually), there are often considerations beyond the project.
- Is there a second phase where the scripts could be used again?
- If this is a new development, could the scripts be used for maintenance?
Knowing these things in advance, and planning for them, can help maximize the return on investment and eliminate additional work down the road.