Often with time your tests don’t uncover bugs anymore. You might feel a gerbil on the treadmill sprint after sprint. Why?
One of the answers is test repeatability. Have you heard about Boris Beizer’s pesticide paradox?
Farmers use pesticide to get rid of bugs. The poison works perfectly. But since the time the more of the same poison they use the less effective it becomes. Bugs develop immunity to the poison.
Speaking the QA language, we have the following:
- We carry out a project, testing a piece of software. To do our best we apply a mix of white-box and black-box tests for this software. The tests detect bugs – we report – everyone is happy.
- Since the software is getting more mature, new features are added to it. In the short run tests part of the tests is broken, however after being fixed they still run the same piece of functionality. They start to lack additional integration flows introduced by the added features.
- Another obvious (but often missed) case is when we sign off the releases based on the same standardized set of Acceptance test cases. No new bugs are detected. But does it really mean they don’t exist? It may happen some new bugs appeared when the fix and new changes were applied to the system. So the old set of test cases can’t detect the new bugs.
What is the way out of this situation?
- Once per release spend several hours to change testing steps in your cases. Try to reach the expected result in more sophisticated way.
- Re-arrange the test suite. Sometimes this simple measure enables you to catch specific pre-condition state and uncover the issue.
- Try to use different approach: test in black-box what was previously covered by white-box scenario.
- Use 80 / 20 proportion for test cases running and adhoc exploration. If you have experienced testers on board, adhoc is often effective when used as an addition. On contrary for junior testers, try to avoid adhoc wherever is possible.
by Artyom Prishchepov, QA Manager at Qaprosoft