Major Cyber Threat Detection Vendors Pull Out of MITRE Evaluations Test
Three major providers of cybersecurity solutions have decided not to take part in the 2025 edition of MITRE’s annual endpoint detection and response (EDR) solution test. After Microsoft announced it would not participate in MITRE Engenuity ATT&CK Evaluations: Enterprise 2025 in June, SentinelOne and Palo Alto Networks confirmed on September 12 they were also pulling out of the test for this year. These decisions have raised concerns among the cybersecurity community about the program’s future and relevancy. It is a particularly surprising decision for Microsoft, which used its ranking in the test to promote its solution, Microsoft Defender XDR, as recently as December 2024. Interestingly, all three companies justified the move by saying they wanted to prioritize product development and innovation. However, experts have suggested that other factors may also be at play, including the tests becoming increasingly seen as promotional rather than achieving real security gains. Infosecurity spoke with Charles Clancy, MITRE CTO and SVP of MITRE Labs, who shared key elements of the evolution of the evaluation test that could explain the decisions ahead of the results of this year’s test in December 2025. Backstory of ATT&CK Evaluations: Enterprise MITRE Corporation is a US-based non-profit organization running many cybersecurity programs, including some on behalf of the US government. MITRE introduced its ATT&CK framework in 2015, which quickly became the standard tool in the cybersecurity industry for mapping real-world cyber adversaries’ techniques, tactics and procedures (TTPs). In 2019, MITRE ATT&CK launched its first Evaluations program to “fill a gap in the security testing market,” Clancy argued. “There were many types of third-party testing out there for cybersecurity products, but each one of them had their own process and scoring methodology, leading to inconsistent results and a lack of rigor that wasn’t driving the industry forward,” he explained. MITRE Engenuity ATT&CK Evaluations: Enterprise is the most regular of all Evaluations tests, occurring every year since its launch. In a LinkedIn post, Igal Gofman, the director of engineering at CrowdStrike and a former security researcher at Microsoft and Tenable, called the test the “Olympics of cybersecurity.” Among the 1000 people working in MITRE’s cybersecurity practice, 133 are dedicated to MITRE ATT&CK, of whom 12 to 15 people are working on the Evaluations tests, Clancy told Infosecurity. Each year, the team behind the testing program picks one of several real-life adversaries and/or attack chains based on their TTPs mapped in ATT&CK. They then test the EDR solutions of participating vendors in simulated attacks using Caldera, MITRE’s own automated adversary emulation platform, according to several criteria, including detection results, false positives and true negatives. Although this test can be used to compare how effective EDR solutions are, Clancy noted it should not be seen as a longitudinal benchmark because each annual test differs greatly from the previous one. “The ethos we’re trying to drive in the testing is comparison of an individual product to detect a particular threat actor. Simulating different adversaries year over year is really important to understand different classes of emerging threats,” Clancy said. Inside the Test's 2024 and 2025 Editions In 2024, MITRE ATT&CK Evaluations: Enterprise emulated 14 techniques across 7 tactics from known North Korean-affiliated hackers, 16 techniques across 7 tactics from the CL0P ransomware group and 31 techniques across 11 tactics from the LockBit ransomware group. CrowdStrike, one of the leading EDR providers, did not take part in that year’s edition, with one member of the CrowdStrike subreddit – who claimed to be working for the company – suggesting that the evaluation was set to take place shortly after the July 19 global outage that affected the company’s EDR product. In 2025, the ATT&CK Evaluations team has selected two scenarios: A financially motivated cyberciminal collective scenario: multi-faceted intrusion in a hybrid environment that features social engineering, cloud infrastructure exploitation, identity abuse and living off the land (LOTL) techniques A Chinese-aligned cyber-espionage scenario: evasive intrusion highlighting the adversary’s adept use of social engineering, abuse of legitimate applications and services, establishing persistent mechanisms and employing custom malware to evade detection While he admitted vendors can vary year-over-year, Clancy assured they can rely on “a lot of repeat customers.” Why Vendors Are Pulling Out of MITRE’s Test However, this year’s edition, the results of which are expected in December, will be missing three major players: Microsoft, SentinelOne and Palo Alto Networks. Microsoft announced it will not take part in this year’s test on June 13, claiming that this decision “allows us to focus all our resources on the Secure Future Initiative and on delivering product innovation to our customers.” On September 12, SentinelOne and Palo Alto released similar statements. The former said it wanted to “prioritize our product and engineering resources on customer-focused initiatives while accelerating our platform roadmap,” while the latter explained that this decision “enables us to further accelerate critical platform innovations that directly address our customers' most pressing security challenges and respond even faster to the evolving threat landscape.” When contacted by Infosecurity, SentinelOne and Palo Alto Networks declined to provide further comment. Microsoft did not respond to a request for comment. However, MITRE’s Clancy said he is in close contact with the three vendors and believes he knows the reasons that made them pull out of this year’s test. First, as the vendors said in their statements, taking part in MITRE ATT&CK Evaluations program requires a resource-intensive commitment, suggesting that the time and personnel dedicated to it are lost on other projects. Then, Clancy said that the team behind the test strives to make it harder every year and conceded they may have pushed it too far this year. “Each year, we want to design a test that’s harder than the year before in order to drive the whole industry forward, since the test can offer an opportunity for vendors to upgrade their products in preparation for the test and once they get the results. And sometimes, we don’t get the balance quite right,” he explained. Speaking to Infosecurity, Vishal Santharam, a senior product manager for endpoint security products at ManageEngine, elaborated on Clancy’s point. “In 2024, MITRE started recording the volume of alerts in the evaluations, which is always a challenge for a vendor to tune in to. More alerts mean increased alert fatigue,” he said, referring to a Forrester study decoding the 2024 MITRE Evaluations: Enterprise based on alert volume. Additionally, Santharam noted that the 2025 Evaluations: Enterprise test included cloud environment, “which is untested territory and requires even more attention from vendors.” Finally, Clancy told Infosecurity that his team used to run a vendor forum each year to prepare for the MITRE ATT&CK Evaluations: Enterprise test. “This forum, which was helpful in working with industry to set the objectives of the test each year, fell off over the last couple of years,” Clancy admitted. On LinkedIn, CrowdStrike’s Gofman argued that the MITRE Evaluations tests were initially a great initiative to benchmark security solutions, but they turned into “vendor theater” in recent years. “Vendors investing huge resources for PR wins, not real security improvements. With MITRE and CISA under pressure from budget cuts and changes, some vendors likely saw an opportunity to step back,” he said. “The concept of TTP-based testing is still valuable, but the way it’s evolved, outdated, overly endpoint-focused, detached from real-world threats is far less so,” he added. Patrick Garrity, a vulnerability researcher at VulnCheck, corroborated this view: “[It] sounds like this benchmarking activity has become a giant distraction to building better products in exchange for publicity,” he said in another LinkedIn post. Despite these concerns, Clancy confirmed that a dozen cybersecurity vendors were still taking part in the 2025 edition of the test. MITRE to Reboot Vendor Forum in 2026 Clancy told Infosecurity that his team intended to re-establish the vendor forum ahead of MITRE ATT&CK Evaluations: Enterprise 2026. “This is something we’re already working to re-establish for the 2026 edition,” he said. He later made this ambition public in a LinkedIn post published on September 18, after SentinelOne and Palo Alto announced they would not participate in the 2025 edition. Santharam also told Infosecurity that ManageEngine was working on an EDR solution and intended to participate in MITRE Engenuity ATT&CK Evaluations: Enterprise in 2026. "The Advanced Anti-Malware and Next-Gen AV products from ManageEngine were certified by AV-Comparatives on their first try. The solution paves the way for our next EDR offering while also providing comprehensive protection against malware and ransomware," he said. "We are also gearing up to take part in the forthcoming Gartner Magic Quadrant for Endpoint Protection Platform (EPP), and the MITRE ATT&CK tests. In addition to proving the robustness and reliability of our technology, these independent assessments also assist clients in developing confidence in our EDR capabilities."
infosecurity-magazine.comSep 22, 2025extracted