MITRE released its latest Enterprise MITRE ATT&CK Evaluations in December of 2024. At that time, we published a blog with a quick overview of the results. Today, we’re excited to announce that we have released three new pieces of research about this round of evaluations:

  1. Analysis Of The 2024 MITRE Engenuity ATT&CK Evaluation — an overview of this round’s evaluations with an explanation of the changes in methodology and an interpretation of the results.
  2. 2024 MITRE Engenuity ATT&CK Enterprise Evaluation Results — a report that visualizes some key data from the evaluation.
  3. The 2024 MITRE Engenuity ATT&CK Enterprise Evaluation Cost Calculation Tool — a tool to help calculate the costs of using each of the vendor technologies in the evaluation to detect the attacks included in the evaluation.

These reports break down the results, give insights into how to interpret them, and provide ways to quantify the impact of using these tools. If you want a tl;dr version, this blog covers a few things we learned from this evaluation data.

True correlation remains elusive, if not nonexistent.

The alert volume generated by some of the participating vendors is ridiculous. A few of the vendors were able to reduce the number of alerts per attack to single digits while still providing full visibility into the attack. This is ideal but was extremely rare in this evaluation.

Most vendors surfaced a detection every single time they identified a step as potentially malicious, including those that were not correlated into an incident. This inflated their alert volume numbers significantly, requiring a lot of manual effort to correlate.

High alert volumes will create misery for incident responders.

While MITRE is not evaluating user experience — and, to be quite honest, cannot be expected to — we at Forrester do evaluate vendors on their user experience, among other features. Some vendors triggered thousands of alerts during this evaluation, with one vendor triggering over a million alerts. Surfacing so many alerts for just three scenarios is ridiculous. Even if you only consider the high- and critical-severity alerts, the numbers still don’t add up. Individual vendor results vary from one vendor with over 5,000 high- and critical-severity alerts to other vendors that had under 10.

number of alerts from vendors in the MITRE engenuity ATT&CK evaluations

The true cost of excess alerting comes down to dollars and cents.

Let’s assume that your team is bringing relevant alerts into the security information and event management (SIEM) platform for additional correlation. Every alert generated by these tools is an added cost (as if the user experience costs were not enough). For example, say that the cost to ingest and store in the SIEM per GB is $0.30 and that the average endpoint alert is 1 KB. If 10,000 endpoints are hit with LockBit, the cost to bring those alerts into the SIEM ranges from $0.006 with some vendors to $471.192 with others. That’s the cost for just one attack. Consider this cost when you’re evaluating tools or trying to reduce your SIEM budget. Our research includes a tool for calculating this cost.

Dive deeper.

Forrester clients who want deeper guidance about the evaluation and the vendors included can book an inquiry or guidance session with me.