MITRE’s MDR stress-test winners combine human intelligence and AI for stronger cybersecurity

Cyberattacks succeed by using social engineering and spear-phishing to find and exploit gaps in corporate IT environments, endpoints and identities. They often launch persistent threats immediately and then steal credentials to move laterally across networks undetected. MITRE chose this breach sequence for its first-ever closed-book “MITRE ATT&CK Evaluations for Security Service Provider.”

The goal of the ATT&CK evaluation is to test providers’ cybersecurity effectiveness. How ready, able and accurate are these solutions at identifying and stopping a breach attempt without knowing when and how it will occur?

MITRE Engenuity ATT&CK evaluations are based on a knowledge base of tactics, techniques and sub-techniques to keep evaluations open and fair. MITRE’s ATT&CKMatrixfor Enterprise is the most commonly used framework for evaluating enterprise systems and software security. 

Stress-testing managed services and MDR 

Historically, MITRE ATT&CK evaluations have informed security vendors upfront — before the active testing — what intrusion and breach attempts they will be tested on and why. With that advance information, vendors have been known to game evaluations, leading to inaccurate results.

In a closed-book evaluation, vendors do not have advance knowledge of what threats they will face in the test. MITRE ATT&CK Evaluations for Security Service Providers is the first closed-book evaluation designed to stress-test the technical efficacy and real-world capabilities of vendors’ Managed Services or Managed Detection and Response (MDR) solutions.

>>Don’t miss our new special issue: Zero trust: The new security paradigm.<<

Closed-book evaluations provide the most realistic reflection of how a security vendor would perform in a customer environment. “The closed book test provides an opportunity to show how security platforms operate against adversary tradecraft in a real-world setting, as vendors have no prior knowledge to guide their actions,” said Michael Sentonas, chief technology officer at CrowdStrike.

MITRE’s assessment of MDRs is particularly relevant, given that chronic cybersecurity skills shortages put organizations at a higher risk of breaches. According to the (ISC)² Cybersecurity Workforce Study, “3.4 million more cybersecurity workers are needed to secure assets effectively.” Managed detection and response (MDR) provides organizations with an effective way to close the skills gap and improve business resiliency.

The MITRE Security Service Providers evaluation lasted five days, with a 24-hour reporting window. Sixteen MDR vendors participating in the program had no prior understanding of the adversary or its tactics, techniques and procedures (TTPs). They were each graded on 10 steps comprised of 76 events, including 10 unique ATT&CK tactics and 48 unique ATT&CK techniques.

“We selected OilRig based on their defense evasion and persistence techniques, their complexity, and their relevancy across industry verticals,” writes Ashwin Radhakrishnan of MITRE Engenuity. The first round of MITRE ATT&CK Evaluations tested vendors by emulating the TTPs of OilRig (also known as HELIX KITTEN), the adversary group with operations aligned to the strategic objectives of the Iranian government.

The attack scenario started with a spear-phishing attack against a national organization using malware associated with HELIX KITTEN campaigns. Next, the simulated threat attack initiated lateral movement across networks to identify and collect critical information, with the final goal of data exfiltration.

Real-time threat intelligence shared across platforms and Managed Services teams are critical to stopping sophisticated cyberattacks. CrowdStrike’s Falcon Complete team collaborated in real time with the Falcon OverWatch threat-hunting service creating an incident diagram and mapping out adversary activity throughout the infrastructure.

Combining human intelligence with AI and ML delivers the best results

MDR vendors with multiple product generations of platform and Managed Services experience, using a combination of artificial intelligence/machine learning (AI/ML) and human intelligence in real time, did the best in the MITRE evaluation. The top four vendors, those that detected the greatest number of the 76 adversary techniques, were CrowdStrike Falcon Complete, Microsoft, SentinelOne and Palo Alto Networks.

These MDR providers rely on insights and intelligence from senior security analysts who use AI/ML apps and techniques designed to analyze telemetry captured from endpoints, networks and cloud infrastructure. The result: AI-assisted threat-hunting expertise that enables their solutions to identify and thwart breaches. 

MITRE Engenuity summarizes its testing results in ATT&CK® Evaluations: Managed Services — OilRig (2022) and the Top 10 Ways to Interpret the Results. This document provides an overview of the methodology and the interpretation of results. MITRE also makes the layer file graphic available for further analysis in its ATT&CK Navigator, shown below.

For the Managed Services — OilRig evaluation, 38 ATT&CK techniques and 26 sub-techniques across 12 ATT&CK tactics were in-scope. Source: ATT&CK Navigator

The results of the 16 vendors who participated in the MITRE ATT&CK Evaluations for Security Service Providers showed the factors that enabled vendors to do well. Vendors that did the best are experienced operators of their own security technologies. They deliver a holistic range of capabilities from across their security portfolios. These vendors continually produced the best security outcomes with the highest detection coverage in the study.

CrowdStrike led all vendors in this category by reporting 75 of the 76 advisory techniques used during the MITRE ATT&CK evaluation. Additionally, consistent with the fact that the highest performing vendors have designed real-time threat intelligence into their platforms and managed services, CrowdStrike was able to internally identify the emulated nation-state adversary in under 13 minutes. 

For an MDR, AI-assisted threat intelligence is key

Getting right the convergence of AI, ML and human intelligence in an integrated MDR solution is the future of cybersecurity. Therefore, product lifecycles for cybersecurity platforms need to be tightly integrated into MDR workflows. That way, valuable capabilities — like native, first-party threat intelligence — become truly actionable.

The evaluation showed how MDR solutions that can generate or create, and then vet, threat intelligence succeed in identifying the most events. CrowdStrike’s reliance on Indicators of Compromise (IOCs) and other strategic insights integrated throughout their products shows how threat intelligence can be scaled across an MDR solution. Identifying the nuanced aspects of MDR solutions, and what enterprises need to look for in a solution, is why the MITRE ATT&CK Evaluations for Security Service Providers are so valuable for organizations looking to these benchmarks for guidance.