MITRE is moving beyond its well-regarded endpoint security evaluations and will soon be testing other security services and products.
MITRE recently issued a call for participation for ATT&CK Evaluations for Managed Services, designed to reveal how managed security service providers (MSSPs) and managed detection and response (MDR) respond to adversarial attacks. Except unlike its Enterprise evaluations, managed services participants won’t know the adversary emulated until the testing is complete, “though it will be based upon publicly available threat intelligence.”
The services evaluation will be focused entirely on understanding adversary activity, and remediation/prevention will be prohibited, the call for participation said.
“During a post-mortem purple team, MITRE Engenuity will disclose the adversary emulated, all behavior performed, and disclose how MITRE Engenuity mapped participant provided analysis to that behavior,” MITRE said. “MITRE Engenuity will work with participants to enhance their detection capability during this period, as participants are encouraged to ask questions regarding the execution.”
MITRE also announced the new ATT&CK Evaluation Trials, in which MITRE will evaluate technologies that don’t fall into its current evaluation programs. First up will be deception tools.
Tests Aren’t Competitive
MITRE’s assessments do not include a competitive analysis. There are no rankings, scores, or ratings. Rather, they demonstrate how each vendor handles threat detection using the ATT&CK knowledge base. They are able to give an unbiased assessment of detection and protection capabilities, as well as identify potential gaps, by selectively picking adversaries and freely sharing results.
In 2018, MITRE Engenuity ATT&CK Evaluations were launched, focusing on the endpoint protection and detection markets. During the evaluations, it became evident that while other types of security solutions were beneficial, they did not meet the project’s standards.
Also read: Top Endpoint Detection & Response (EDR) Solutions
Testing Deception
As a result, MITRE devised the new ATT&CK Evaluations Trials program to assess the capabilities of each technology. The trials project is a research-focused expansion to the ATT&CK Evaluations landscape, involving a collaboration with vendors to develop new evaluation methodologies that will better capture their value propositions in an open manner. Each trial will have its own set of objectives, designs, and outcomes that will showcase the benefits of each technology.
MITRE is developing a deception approach that will provide end-users with relevant findings, define important distinctions in vendor product strategies, and do so in “a fair and open manner.”
According to MITRE, “Deception technology offers a unique value to organizations seeking to understand adversary behavior. It can dramatically increase analyst confidence in detection via high-fidelity tripwires, causing the adversary to waste time, money, or capability, and potentially provide us critical new insights into adversary behavior. Each of these use cases starts to put power into the defenders’ hands when they have long since been forced to be reactionary.”
The trails will attempt to answer two fundamental questions:
- Did the adversary encounter the deception (i.e., could the deception capability affect the adversary)?
- Did the adversary engage the deception?
Determining whether or not the adversary encountered deception is a straightforward question that can be answered from a threat-informed perspective. It can be determined by using the adversary technique and documenting whether or not it notices anything different from a scenario that does not use deception.
Engagement, on the other hand, can be more difficult to quantify due to the human factor. MITRE is taking into account a number of factors in this regard, including:
- Did the adversary engage it out of happenstance, or did they make the conscious decision to pick it because it seemed like the better target?
- Would they have engaged the deception again if they were presented with the same choice again?
- Would a different tester make the same choice?
- Would that choice change if they were aware, or not, that there was deception technology deployed?
- Was the effect a short-term inconvenience, or did it affect their long-term mission?
There are some challenges when it comes to representing results, how to do so in a way that is uniformly fair. This is because vendors assess success in different ways due to the wide range of products on the market.
MITRE will need to identify common measures that will allow them to talk about products in a similar language while still recognizing each vendor’s unique capabilities and use cases, given the diversity of the outcomes. Deception has a variety of value propositions, which is why MITRE is exploring it as a research project. The following are some of the areas they will assess:
- Detection based on a high level of confidence tripwires
- Interaction that keeps the attackers engaged and wastes their time and resources
Attivo and CounterCraft Sign Up
Attivo Networks and CounterCraft Security have confirmed their participation in the ATT&CK Evaluations Trials program.
According to Frank Duff, General Manager of ATT&CK Evaluations, Deception is the only current Trials program underway, There have also been talks with a number of other vendors about other partial research opportunities, some of which could look at similar threats but with a different value propositions (e.g., NDR), others could be extensions/improvements for their current methodology (e.g., fairly and effectively evaluating false positives).
MITRE also has one other evaluation going on, an emulation of the Wizard Spider and Sandstorm threat groups. Thirty endpoint security vendors have signed up for that one, with results expected in 2022.
Further reading: Latest MITRE EDR Evaluations Contain Some Surprises
