So you get to the point when you have automated test cases. Great. I have a question for you:
How do you measure automation success?
Some metrics seem to be obvious:
– # of tests
– defects found
Do you report based on these?
Did you take into account that:
– Having a large number of tests doesn’t mean that the right things are being tested.
– Not finding any issues with the automated tests doesn’t necessary mean that there is no issues.
Once you have a picture on which features, business paths are covered by the tests, you might consider to introduce % of path covered. Do you include this into your report?
Knowing the time saved by running the automation tests is also a good indicator. This is the time that testers can spend on focusing on other areas of testing.
On the other hand it is also good to keep in mind that automated tests might need test fix or update. Do you know how much time you spend on updating/fixing existing tests?
So. How do you measure automation success?