MSSPs Fare Well in First MITRE Evaluations

eSecurity Planet content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

If MITRE Engenuity’s new MSSP evaluations are any indication, managed security service providers are a little like children from Lake Wobegon: They’re all above average.

Of the 15 MSSPs that participated in MITRE’s first-ever security services testing, only three failed to report attack techniques in all 10 of the evaluation steps, and in two of those cases it was because the test didn’t successfully execute because of a web shell failure.

While the sample is small – by some estimates there are roughly 10,000 MSSPs – it nonetheless should be reassuring to MSSP customers that the vendors charged with defending their networks have demonstrable cybersecurity expertise. As there are few measures of security effectiveness, and none better than MITRE, it would benefit information-starved security buyers if more service providers participated in future rounds.

Ashwin Radhakrishnan, general manager of MITRE Engenuity’s ATT&CK Evals, said in a statement that the organization decided to evaluate MSSPs because of their growing importance.

“More than half of organizations use security service providers to protect their data and networks,” Radhakrishnan said. “We wanted to research how they are employing threat-informed defense practices for their clients. We don’t rank the vendors in our evaluations. Organizations, however, can use the Evals to determine which service providers may best address their cybersecurity gaps and fit their particular business needs.”

See the Best Managed Detection and Response (MDR) Services and the Top MSSPs

MSSP Tests Look At Reporting, Not Detection

MITRE is best-known for its endpoint security product evaluations, but there are some important differences between the organization’s product and services evaluations.

The MSSP evaluations examined how vendors performed under techniques that simulated attacks from the OilRig Iranian threat group, which was chosen because of its “evasion and persistence techniques, its complexity, and its relevancy to industry,” MITRE said.

The evaluation examined the MSSPs’ ability to report ATT&CK Techniques across 74 techniques and 10 steps, from initial compromise through lateral movement, exfiltration and cleanup.

An important emphasis in the new tests is on the word “report” rather than the detections measured in MITRE’s endpoint tests. MITRE purple teamers evaluated whether an ATT&CK Technique was reported or not, rather than whether it was detected by the service provider, MITRE said.

“In many cases, the service provider may have detected the ATT&CK Technique under test but chose not to report it to MITRE Engenuity because they believe it is unnecessary information, or they believe it can be implied or assumed by other information provided to MITRE Engenuity,” MITRE said on the MSSP evaluation’s overview page. “In order for an ATT&CK Technique to be considered Reported, the activity provided to MITRE Engenuity must contain sufficient context to explain the activity. Things like raw telemetry with no added analysis provided by the service provider were not considered Reported.”

That means the data provided by the tests isn’t as clear as it is in the product evaluations. So while we’ve recorded below the number and percentage of techniques reported by the MSSPs, as always it’s important to dig into the data and find what’s relevant for your organization’s needs.

In a blog on interpreting the results, Radhakrishnan noted a number of important considerations, among them:

• Not all techniques are equal: “A service provider reporting on Process Discovery might not have the same value as a service provider reporting on Credential Dumping due to the severity of the action.”
• Not all procedures are equal: “Process Discovery (T1057) via Command-Line Interface (T1059) can be detected with most process monitoring. Process Discovery via API (T1106) would need API monitoring. A service provider could have reported one, but not the other.”

The Results

With those significant caveats, here is some basic data from the evaluations.

Only one MSSP – BlackBerry – failed to report any findings on one of the 10 steps, the five techniques where the attackers download and install a web shell on the Exchange Web Server (EWS) for persistence. BlackBerry found plenty in the other 9 steps, however.

Palo Alto Networks and NVISO couldn’t participate in a handful of the 74 techniques, which couldn’t be executed because of a web shell failure.

And a 16th vendor, Trend Micro, did not have its results published after inadvertently finding “sensitive information.”

“Although Trend Micro participated and completed testing for this inaugural round, after an unintended situation, Trend Micro promptly and responsibly shared that their team had found sensitive information to MITRE Engenuity,” Radhakrishnan told eSecurity Planet. “Based on the agreement between MITRE Engenuity and Trend Micro, MITRE Engenuity did not publish Trend Micro’s results.”

So with those caveats, here are the raw numbers and percentages of the 74 attack techniques reported by the MSSPs: