MITRE Engenuity last month released the latest MITRE ATT&CK evaluations of endpoint security products, and the results contain some pretty big surprises.
MITRE doesn’t analyze or comment on the evaluations and instead just makes the data available for vendors and cybersecurity buyers to use as they see fit. That said, the raw numbers alone contain some interesting insights. In the weeks ahead, we’ll be using the results as we re-examine our top endpoint detection and response (EDR) and antivirus product lists. Also see our analysis of MITRE’s new endpoint security protection tests.
For the 2022 MITRE endpoint security results, see Latest MITRE Endpoint Security Results Show Some Familiar Names on Top
The latest evaluations were dubbed Carbanak+FIN7 and were modeled after threat groups that target the banking and retail industries. Previous evaluation rounds were modeled after the APT29 and APT3 threat groups connected to the governments of Russia and China, respectively. APT 29 was behind two of the more notorious attacks of recent years, the 2016 DNC and 2020 SolarWinds hacks, so the MITRE evaluations are among the most rigorous testing a cybersecurity product can face.
MITRE results analyzed
Here are the surprising – and not-so surprising – results in our analysis of the Carbanak+FIN7 evaluations. We focused on two data points: the attack techniques detected by telemetry, meaning little processing was required to detect the technique, and those detections that required analytic processing. Those two were combined in a category MITRE called “visibility,” which is the overall detection rate of the 174 attack techniques the vendors were tested on. See a chart of the MITRE results at the bottom of this article.
SentinelOne tops the charts
In overall performance, SentinelOne detected every technique, 164 by telemetry and 174 by telemetry or analytic processing. For a vendor we already rate highly, SentinelOne’s results are still surprising. SentinelOne’s users are already among the happiest in the EDR space; they now have another reason to smile.
Palo Alto Networks remains king
As impressive as SentinelOne’s performance was, it wasn’t enough to knock Palo Alto Networks from the top of the pack. After years of stellar independent testing, Palo Alto’s second-place finish continues to paint a picture of consistent excellence against a wide range of attack techniques. Palo Alto detected 169 of the 174 MITRE evaluation techniques, good for a 97% overall rate. All vendors had their own spin on the results, and Palo Alto noted that it blocked 100% of attacks in the MITRE protection evaluation.
Symantec and McAfee still got game
In a market full of new and exciting upstarts, Symantec and McAfee showed that being an established tech vendor doesn’t mean you can’t innovate and stay on top of the market. Of the 29 EDR vendors that participated in the evaluations, Symantec came in sixth place with an overall score of 91%, while McAfee was thirteenth with just under 87%. Considering the size of their combined customer base, that’s good news for cybersecurity in general.
Cybereason makes a statement
Cybereason continues to climb in the rankings. The nine-year-old Boston-based company came out on top in NSS Labs testing last year, and now they’re adding to that success with a fifth-place MITRE showing and 92% detection rate. The company is showing that “next-gen EDR” is more than a marketing slogan.
Leaders stay leaders
A number of our other top EDR vendors placed in the top 10 in the latest MITRE evaluations: Trend Micro, Check Point, VMware CarbonBlack and CrowdStrike were also in the top 10, followed closely by F-Secure and Microsoft (yes, that free antivirus software that comes packaged with Windows 10 is very good). The only top 10 MITRE finisher not on our top EDR products list somewhere is six-year-old Cynet, so you can bet they’ll now get another look; they’re already making waves in the XDR space.
Some need improvement
One reason MITRE doesn’t analyze results or rank participants is to encourage participation – vendors are expected to use the results to improve their products, which in turn is good for their customers (and their customers’ customers, which is pretty much all of us). That said, a few well-known vendors had overall detection rates of around 70% in the latest MIRE evaluations: Cisco, Sophos, Fortinet, Micro Focus and Malwarebytes. They’re likely already using the evaluations to improve their products, and we’d like to see them return to the MITRE evaluations in future rounds with strong results. We’ve seen a number of impressive test results in the past from Sophos and Fortinet in particular, so users can expect quick action there. And the presence of Micro Focus in the evaluations is interesting, as the company is largely a SIEM vendor – it would be valuable info if MITRE were to conduct evaluations targeted at other critical cybersecurity systems like SIEM and firewalls. Thus far the MITRE evaluations have overwhelmingly attracted EDR vendors.
And one noteworthy vendor skipped the latest MITRE round: Kaspersky, which has previously participated in both MITRE and NSS Labs tests, with good results.
