Measuring Automation Success

So you get to the point when you have automated test cases. Great. I have a question for you:

How do you measure automation success?

Some metrics seem to be obvious:
– # of tests
– defects found


Do you report based on these?
Did you take into account that:
– Having a large number of tests doesn’t mean that the right things are being tested.
– Not finding any issues with the automated tests doesn’t necessary mean that there is no issues.

Once you have a picture on which features, business paths are covered by the tests, you might consider to introduce % of path covered. Do you include this into your report?

Knowing the time saved by running the automation tests is also a good indicator. This is the time that testers can spend on focusing on other areas of testing.
On the other hand it is also good to keep in mind that automated tests might need test fix or update. Do you know how much time you spend on updating/fixing existing tests?

So. How do you measure automation success?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s