Entomology: The Art of Classifying Bugs

All software contains bugs. Period. Some bugs require immediate attention, once discovered. And, some bugs aren’t worth fixing at all. How you differentiate between the two makes all the difference inside and outside of your Engineering department. This is how we do entomology at Entelo…

A Framework for Classifying Bugs

We classify a bug by severity and scope in order to assign it a priority.

• Severity is a measurement of a bug’s impact on the usefulness of the software. A high severity bug significantly curtails the user’s ability to derive value from the system. A moderate severity bug limits a user’s access to a major feature. And, a low severity bug is a minor nuisance, but not blocking use of any features.
• Scope is a measurement of the number of users impacted by the bug. The more users affected, the broader the scope. The fewer users affected, the narrower the scope.
• Priority is used to set expectations between departments (and with our customers) as to when an issue will receive attention. Essentially, a priority level is shorthand for a service level agreement and the actions we take in response to a bug.

We define priority levels as follows:

A few notes on scope:

• If we think a bug affects all users, we treat it as such until we discover otherwise. If we discover that, in fact, the bug did not affect all users, we will downgrade the bug as appropriate.
• If we can reproduce a bug, we treat it as though at least a subset of users (some) are affected, even if only a single user has reported the issue.
• If we cannot reproduce a bug that was reported by a single user, we treat it as such.

Service Level Agreements

We define our default service levels as follows:

Note that “receives attention” means that the Product team will prioritize and the Engineering team will look into the bug within the specified time frame. It does not promise resolution within that time. Some bugs are harder to fix than others.

Adjusting Service Levels Based on Circumstances

For the vast majority of bugs, the default service level should be fine. But, from time to time, we may encounter an issue that requires more urgent attention than the default SLA affords. Specifically, the Product Manager may decide (in consultation with Customer Success) to prioritize a bug sooner than the SLA would warrant, based on the presence of one or more aggravating circumstances, including but not limited to:

• The bug affects our relationship with a strategic customer.
• The bug affects the renewal of a large account.

It is important to note that aggravating circumstances only affect a bug’s SLA. They do not change a bug’s priority level, since priority levels govern the type of response Engineering is required to undertake for SOC 2 compliance. This is important because the additional overhead associated with P0 and P1 bugs means that Engineering can actually resolve more P2 bugs than P0 and P1 bugs in the same amount of time.

Conclusion

Actually, this is just the beginning. We’ve just rolled out the framework across the company. Our hope is that it will reduce communication friction between departments by setting priority levels objectively, based on severity and scope.