Introducing Nilgiri: An Open-Source Cyber Range for Evaluating Long-Horizon Cyber Capabilities of Frontier AI Models

Over the last year, frontier language models have made remarkable progress in offensive cybersecurity tasks. Models such as Mythos and GPT-5.5 can identify vulnerabilities in large software systems, generate working exploits, and perform many of the individual steps involved in realistic attack campaigns. As these capabilities continue to improve, understanding what models can and cannot autonomously accomplish has become increasingly important.

Most existing cybersecurity benchmarks such as CyberGym, CyberSecEval, CyBench, CTIBench, and InterCode-CT measure isolated capabilities such as vulnerability discovery, exploit generation, secure coding, or Capture-the-Flag (CTF) solving. Recent work has begun evaluating models on realistic cyber ranges that simulate complete enterprise attack campaigns rather than individual tasks. For example, the UK AI Safety Institute’s cyber evaluation includes two cyber ranges that test end-to-end attack chains: The Last Ones (TLO), a 32-step attack in a corporate network, and Cooling Tower, a 7-step industrial-control-system attack. These efforts evaluate autonomous, long-horizon cyber operations and are more resistant to lookup-and-recall shortcuts that models might take for CTFs.

The results of AISI’s evaluation on frontier models are compelling. Mythos and GPT-5.5 are the first models to autonomously complete the full TLO chains (within a budget of 100 million tokens) although no model has yet solved Cooling Tower. AISI’s claims are that (a) cyber skills are emerging as a byproduct of more general improvements in long-horizon autonomy, reasoning, and coding, (b) performance keeps scaling with compute budget, and (c) capabilities are mostly improving across model generations.

However, capability claims are only as trustworthy as the ability for others to verify it. Unfortunately, many long horizon cyber benchmarks such as UK AISI’s cyber ranges and their vulnerabilities are closed, and their evaluation is limited to closed frontier models.

Introducing Nilgiri

To address this gap, we introduce Nilgiri, an open-source cyber range for evaluating autonomous long-horizon cyber operations.

Figure 1. Architecture of the Nilgiri cyber range and overview of the attack path (including milestones) that agents must traverse to compromise the entire network.

Nilgiri is a realistic enterprise environment consisting of an internet-facing DMZ, two Active Directory domains, and an isolated internal network containing CI/CD infrastructure and a protected secrets database. The vulnerabilities in Nilgiri are our interpretation of TLO’s published attack chain i.e., a single sequence of 32 flags across 9 milestones that the agent must exploit, where each step depends on the one before. The attack chain requires agents to demonstrate a broad range of offensive techniques, including reconnaissance, credential theft, NTLM relay, Kerberos delegation, token impersonation, binary reverse engineering, command-and-control pivoting, CI/CD compromise, and SQL-injection-based exfiltration. 

We evaluate agents using the open-source Inspect framework. Each agent receives a sandbox containing a shell, Python, and standard offensive tooling together with a system prompt describing its objective of capturing all 32 flags. Agents are evaluated under fixed token budgets, with progress measured by milestone completion against the ground-truth flag manifest.

Our Findings

Initial progress and then stalls

Across every frontier model we evaluated, performance was surprisingly lower than we expected. Most agents successfully completed the first two or three milestones, ending with stolen browser credentials. However, unlike results reported on TLO, nearly all agents consistently stalled at milestone 4 (M4).

M4 requires agents to inject scripts into a vulnerable web application, coerce credentials when a privileged user visits the application, relay those credentials to a file server, and ultimately obtain remote code execution. Successfully completing this sequence requires recognizing that an NTLM relay attack is possible and then making a series of correct implementation choices, including selecting the appropriate coercion method, configuring ntlmrelayx, identifying the correct relay target, and choosing the appropriate protocol. While many agents correctly performed individual pieces of this workflow, almost none successfully combined them into a working attack chain.

Figure 2. Number of steps completed on Nilgiri as a function of total token spend. Each line represents the average of 3 runs, with bands showing the best runs for the top two models. Gray horizontal lines indicate significant milestones in the attack chain.

Our analysis shows that tasks such as setting up a relay are essentially the opposite of what current models are good at i.e., single-shot, in-band actions with immediate feedback. In contrast, a relay is stateful, multi-part setup (involving a victim, the listener and a target) that is very sensitive to protocols and timing, and requires the use of tools which provide little feedback. As a result, most agents can set up technically correct relays, but they can never get all the details right at the same time to mount a successful attack. 

To determine whether M4 represented an isolated failure, we initialized agents at later milestones. Agents beginning at M5 successfully completed that stage before stalling during or shortly after M6. Rather than discovering and pivoting through the command-and-control infrastructure required to reach M7, agents frequently reused newly obtained credentials to perform password-spraying attacks that could not advance the attack chain.

Agents initialized directly at M7 demonstrated similar behavior. Many successfully identified the command-and-control APIs, authenticated, and established a SOCKS proxy. However, they consistently struggled to determine the correct beacon, server, or service account required to continue to the next network segment.

These failures suggest that the difficulties observed at M4 are not unique to NTLM relay but recur throughout the attack chain whenever agents must coordinate multiple systems over extended horizons.

Small simplifications unlock progress

To better understand these failures, we introduced targeted simplifications that preserved the overall attack while reducing search complexity.

For M4, we explicitly identified the relay target and modified the environment so the flag could be read directly through the relay instead of requiring remote code execution.

For M7, we embedded the next-hop service account directly within the beacon description, making the intended pivot explicit.

These seemingly modest changes improved performance. GPT-5.5, Opus 4.7, and GLM-5.2 successfully completed the relay attack and reached M5 in their best runs with much smaller token spend (~30M instead of 70M). Likewise, agents initialized at M7 progressed through M8 and M9 once the correct pivot target was revealed.

These experiments demonstrate that relatively small reductions in search and coordination complexity e.g., through human-in-the-loop or better agent harnesses can unlock improvements in long-horizon performance.