Skip to main content

How good is your security guard at his job?

· 3 min read
Puneet Khanduri
Evaluating Quality Engineering investments and measuring the quality of Application Quality Monitoring systems

Let’s think of some questions we can use to determine how good a job your security guard has been doing.

How many burglars did you catch last quarter?

Obviously, this is not a good way to measure your security guard’s performance. Developers sometimes ask variations of this question like “how many bugs will this tool catch?”, or “how long do I have to wait before it catches one?”, and “how long do I have to wait before that Aha! moment?”

A well-meaning QA Engineer will likely respond with “I hope you never have that Aha! moment” because when it comes to testing and security, boring is good. In order for any quality-related investment to demonstrate ROI (by these standards), the developers really have to screw up often.

How many thefts happened while you were on duty?

This is a much better criteria and sets a lower bound on performance expectations. Good QA Engineers aspire to prevent all bugs from ever reaching prod environments. Good metrics around this criteria are “how many bugs shipped to production last sprint?” or better yet “how many sprints since last bug was shipped to production?”

How many legit visitors did you mistake for burglars?

A paranoid guard who mistakes your regular electrician and plumber for burglars and calls the cops is being sincere but is a bit painful to deal with. He is like the boy who cried wolf — eventually when there is an actual burglar, you won’t believe him. Tools used by QA Engineers can sometimes be noisy and produce false positives. These noisy reports and flaky tests erode trust and teams quickly learn to tune them out.

We are really talking about using precision and recall as metrics for evaluation of your QA investments. Just to be clear, we are not talking about code coverage here (why not is an entire separate post). Most teams don’t apply this level of mathematical rigor to evaluate their investments. Eventually, when you start doing this you realize two things:

  1. You have some blind spots in your investments that make it impossible to catch some classes of bugs.

  2. You also have some redundancy in your investments where the same class of bugs will be caught by multiple investments independently.

Applying quantitative principles to your quality engineering investments allows to discover optimal tradeoffs and maximize your ROI.

Stay tuned for a detailed case study!