Using ATT&CK Evaluations

Our ATT&CK Evaluations results are detailed and may be different from other evaluations you’ve seen, so you will likely use our results differently. Our evaluations look at each vendor’s capabilities within their own context, while doing so in a way that is consistent across vendors. Our evaluation results describe how product users could detect specific ATT&CK instantiations under perfect circumstances with knowledge of the adversary and without environmental noise. The results serve as an example of what a capability could do, although we realize the environment in which we test isn’t entirely realistic.

Direct comparison between vendor capabilities is complicated, and we encourage anyone using our results to consider other factors we didn’t evaluate. Our evaluations are narrowly focused on the technical ability to detect adversary behavior. There are other factors we are not accounting for in our evaluations that should be considered by decision makers as they decide which tool best fits their needs. You should consider factors such as cost of ownership, sophistication of your Security Operations Center, environmental noise, integration with other tools, user interface, security policies, and other factors. One product may not fit every need, and products can address different needs in different ways.

We realize many people will want to use our raw results to develop scores, rankings, or ratings. Should you decide to do this, we encourage you to consider each detection independently and rank it on how useful it would be to meet your unique requirements. Though we categorized detections, not all detections in the same category are created equal – some detections may be more useful to you than others. We’ve given you a head start by evaluating the technical capabilities, but this only provides a piece of the story, and we encourage you to consider these additional factors.

Our evaluations and methodology can assist organizations as they make critical decisions about which vendor capabilities best suit their needs. By looking at the detection abilities of these products and weighing their unique constraints, organizations may be able to “down-select” products that appear to best meet their requirements. We encourage using our methodology to test capabilities in your environment. This allows false positives, environmental noise, user interface, and operational impact to be considered in a way that is tailored to your organization.