Our Evaluation of Cybersecurity Capabilities of GPT-5.6 and Kimi-K3
July 20, 2026
In our last post, we introduced Nilgiri, our open-source cyber range for evaluating the cybersecurity capabilities of frontier models. Our main finding was that all agents we tested were able to perform simple offensive tasks such as reconnaissance, lateral movement, and browser-credential theft, but stalled at more complex attacks that require coordinated actions across multiple servers, such as using an NTLM credential relay to obtain remote code execution.
We have since run two new models against the same range with the same 100M-token budget: OpenAI’s GPT-5.6-Sol and Moonshot’s Kimi-K3. The two land on opposite sides of the wall we identified last time, and together they sharpen where the frontier actually sits.
Figure 1. Number of steps completed on Nilgiri as a function of total token spend. Each line represents the average of 3 runs, with bands showing the best runs for the top two models. Gray horizontal lines indicate significant milestones in the attack chain
For the first time in Nilgiri, an agent based on GPT-5.6-Sol succeeded at milestone 4 (M4) consistently. The agent coerced an account, relayed the authentication to the file server, and landed remote code execution over the relayed session. It cleared this milestone without requiring hints or relying on simplifications.
Breaking M4 in our range unlocked more milestones downstream. GPT-5.6-Sol completes M5, which requires the agent to perform a SQL-injection-driven coercion, xp_cmdshell RCE, secret extraction, and Kerberos delegation. In its best run, it also completes M6, which requires reverse-engineering a system binary, extracting ciphertext containing secrets, and obtaining decryption keys via privilege escalation. GPT-5.5 was able to complete M5 and M6, but only when starting at M5.
After M6, GPT-5.6-Sol stalled while pivoting to M7. As with M4, this pivot requires the agent to sustain a multi-step attack across multiple systems: use a previously obtained credential to authenticate to a command-and-control (C2) server, select a live beacon, open a SOCKS proxy through it, and route tools into an otherwise unreachable network segment. At this point, GPT-5.6-Sol reverts to single-shot, immediate-feedback “poke the endpoint” behavior instead of holding and executing that plan, even though it already has the credentials. So the frontier in Nilgiri essentially shifted from running relays to the more challenging reserve a C2 and pivot.
Kimi-K3: reaching the M4 wall, but not crossing it
Kimi-K3 is a counterpoint; in all its runs, Kimi-K3 captured 11 flags, cleanly completing M1 through M3. It then broke into M4: it planted the malicious payload, and coerced an admin. But it never converted that coerced authentication into a relayed remote code execution. Kimi-K3 stalls at the exact step we called out last time.
Efficiency
We also investigated the relative efficiency of GPT-5.6-Sol and Kimi-K3, and the gap with previous models is striking. Both GPT-5.6-Sol and Kimi-K3 capture 11 flags by ~8M and ~10M tokens respectively, and GPT-5.6-Sol captures 15 flags by ~10M. Even GPT-5.6-Sol’s median run outperforms GPT-5.5’s best.
Lessons
There is much to learn from these new models. First, cyber capability is still scaling, and quickly. In roughly a month, an agent went from solving simple offensive tasks and then stalling to executing a complex, multi-part offensive task. However, the barrier has moved up to more complex operations: the limitations that remain cluster around sustaining coordinated attacks across systems with little feedback. Also, run-to-run variability still exists. Third, Kimi-K3 shows that open weight models have yet to cross the previous barrier, and GPT-5.6-Sol’s breakthrough reflects a real capability difference.
This makes an even clearer case for open and robust cyber evaluation. Today, we’re publishing a live Nilgiri leaderboard. It ranks agents by how far they get through Nilgiri’s 9-milestone, 32-flag chain. We invite the community to run their agents against the range and put the results on the leaderboard. Please see the contribution guide for details. If your agent can cross the M4 relay, land the C2 pivot, or push past M7 where GPT-5.6-Sol stalls, we want to see it. In the meanwhile, we’ll keep expanding our range with more milestones, and we’ll keep the board up-to-date as new models arrive.