The Results Are In: Critical Start MDR Reports Adversary Activity Across All Steps of the MITRE ENGENUITY™ ATT&CK® Evaluations for Managed Services

The very first MITRE ENGENUITY ATT&CK Evaluations program for Managed Services recently wrapped up and the results published on November 9. The team here at Critical Start has been an enthusiastic participant since the program kicked off earlier this year, and we’re excited to share details on the outcomes. We’ll provide a summary here, but we’re also publishing an in-depth eBook that provides an overview into the step-by-step process the MITRE ENGENUITY team employs to carry out realistic simulations of real-world intrusion attacks and data exfiltration exploits.

Critical Start – a leading provider in MDR Services for endpoint, user, identity, applications, network infrastructure, cloud security tools, SIEM and XDR – reported adversary activity across all steps of the MITRE ENGENUITY ATT&CK Evaluations for Managed Services.

In addition, as you will see here, our service delivery approach provides our customers with additional contextual value information beyond security alerts from vendor’s consoles, to include an analysis on behaviors and observations, potential risk impact, guided recommendations, and if enabled, vendor-agnostics response and remediation capabilities using web interface, native mobile application, security orchestration and ticketing systems integrations, and open APIs.

The MITRE ENGENUITY ATT&CK Evaluations program brings together product and service providers with MITRE experts to collaborate in evaluating security solutions. The evaluations process applies a systematic methodology using a threat-informed purple teaming approach to capture critical context around a solution’s ability to detect or protect against known adversary behavior as defined by the MITRE ATT&CK knowledge base. Results from each evaluation are thoroughly documented and openly published.

How It Works: The MITRE ATT&CK Evaluation Process

MITRE ATT&CK Evaluations for Managed Services assess the capabilities of participating MDR and Managed Security Service Provider (MSSP) vendors in their ability to analyze and describe adversary behavior. Adversary activity emulated by the MITRE ENGENUITY red team, and correlating context provided by the participants, was mapped to the MITRE ATT&CK knowledge base. Participants leveraged a self-supplied toolset to enable their detection capabilities and provide the relevant analysis in the same format they provide to their customers. Examples include—but are not limited to—real-time alerts, daily roll-up reports, dashboard access, etc.

Evaluation Parameters and Priorities

The core goals of the evaluation were a) to detect the effectiveness of MDR and MSSP Service Level Agreements (SLAs) for Endpoint Detection and Response (EDR) threat detection (not response), and b) to assess MDR and MSSPs in their ability to analyze and describe adversary behavior. The evaluation focused on people, process and technology as part of a good security program, and the same principles of the MITRE ATT&CK Evaluation Enterprise program and other MITRE ATT&CK best practices were applied.

The Managed Services Evaluation employed a “closed book version” of adversary emulation, whereby the vendor participants did not know the emulated adversary until after the execution is complete, though it will be based upon publicly available threat intelligence. This methodology was a departure for MITRE ENGENUITY, as all previous evaluations were conducted “open book”—whereby vendors knew going into the process what adversaries would be emulated and provided with information on Technique Scope, which defined the techniques that could be included in the evaluation.

Summary of Emulated Adversarial Activity

Critical Start correctly identified the emulated adversarial group, initial access vectors and target objectives in our submission documents at the completion of the evaluation period.

• Group(s): OilRig / Hafnium – a suspected Iranian threat group that has targeted Middle Eastern and international victims since at least 2014.
• Initial Access: a spearphish email to download an infected document that initiated the follow-on adversary activities using SideTwist, TwoFace, RDAT, mimikatz and renamed Windows binaries.
• Target Objectives: perform reconnaissance in the network, steal credential to move laterally using native Windows services, access a SQL Server database and exfiltrate the database backup over email communication channels.

In a multi-stage adversarial campaign spanning days, Critical Start was able to identify and recommend immediate response actions in the initial execution that would have completely stopped the adversary through web and native mobile interfaces – quarantine of the infected documents, isolation of the impacted workstations and force logout and force password change for compromised user accounts.

Critical Start MDR Service Contextual Value Delivery

Additionally, Critical Start SOC analysts added additional context on additional behaviors observed, behaviors & evidence, organizational risk and guided recommended actions. Organizations deploying security tools already get alerts in those tools’ consoles. The original intent by MITRE for this Managed Services Evaluation is that MDR providers must go above-and-beyond forwarding alerts emails and provide actionable guidance in their service delivery, else organizations will face the same alert fatigue and alert overload from their providers as they do from their own internal tools.

The images below are actual screen shots from Critical Start’s platform web and mobile interfaces submitted to MITRE ENGENUITY for this Evaluation that illustrate what we provide to customers in our customary services delivery agnostic of the underlying security tools deployed by an organization.

CRITICALSTART® Threat Navigator and MITRE ATT&CK® Framework Mapping

The Critical Start Platform uses a unique capability we developed called Threat Navigator that maps all vendor security alerts – from any security product – plus Critical Start’s custom threat detection content into the MITRE ATT&CK Framework. The threat detections are mapped by security product category, vendor, vendor product name and vendor product module allowing organizations to determine their current and potential future coverage across the MITRE ATT&CK Framework.

This approach normalizes the delivery of threat content from detection engineering across any security product deployed by the organization – currently deployed or for future vendor migrations – providing visibility in threat detection coverage across the MITRE ATT&CK Framework coupled with adversary activity detection.

The screen shot below shows the vendor and custom detections mapped to MITRE ATT&CK Technique ID values for one security product utilized by Critical Start in this Managed Services Evaluation. This provides a time-line history (from left-to-right) of the adversary activity across the attack chain from Reconnaissance to Exfiltration.

Conclusion

In this retrospective of an emulated adversary attack executed in the MITRE ENGENUITY ATT&CK Evaluations for Managed Services, Critical Start presents key considerations and best practices for organizations adopting Managed Detection and Response services:

• The Evaluation seeks to present a qualitative assessment of method in which the service provider participants reported attacks to their customers – not a quantitative result of specific detections.
• Critical Start reported adversarial activity across all Steps of the Evaluation delivering end-to-end visibility into the adversary behavior.
• Organizations seeking a managed services provider should evaluate whether that communications add additional context, risk analysis and guided recommendations (plus response actions) beyond screen shots and emails of alerts that come from the underlying security products.
• While the contract language of this MITRE Evaluation does not allow participants to disclose the use of third-party security tools they utilized, Critical Start’s platform approach coupled with our 24/7 Security Operations Center analysts provide the same high-level of Managed Detection and Response service delivery regardless of the underlying security products used by organizations – including supporting many of the endpoint vendors in this Evaluation.
• Critical Start provides MDR services for security products that are outside the scope of the endpoint-oriented focus of this Evaluation, including alerts from user activity, identity providers, infected documents, user-reported phishing emails, on-premises and cloud applications, network infrastructure, cloud infrastructure and more.