AI is not ready to replace programmers. Not even close. A new benchmark called BeyondSWE just exposed the truth. Even the best AI models in the world crashed when faced with real software engineering tasks. Their pass rate dropped to just 45 percent.
For the past year, SWE-bench has been the only standard for measuring Code Agent performance.
Top models went from 30 percent to over 80 percent on SWE-bench. Gemini 3 Pro and GPT-5.2 broke records. It looked like AI was about to match human programmers.
But here is what everyone missed. SWE-bench only tests a tiny slice of reality. It uses 12 code repositories. Each task averages only 1.3 files and 11.6 lines of code. All answers are hidden inside the repository. Even SWE-bench Pro and SWE-bench Live expanded the repository scale and fix size. But they never broke out of one critical boundary.
What boundary?
Think of SWE-bench like a student taking an open-book exam where the answer is always on the same page. It only tests whether the AI can write code. It never tests the full picture of real software engineering.
Even OpenAI admitted internally that SWE-bench Verified is not a real test. It is just a toy benchmark. Top models score high because the answers are basically given to them.
So the real question is this. Can Code Agent handle real coding work? Or is it just good at solving puzzles inside a box?
BeyondSWE breaks through two walls at the same time. It expands both the Resolution Scope and the Knowledge Scope. This means AI must handle real complexity, not just patch small bugs inside a familiar codebase.
Resolution Scope means how much code the AI needs to change. Is it fixing one file? Or is it changing multiple repositories?
Knowledge Scope means where the AI gets its information. Is it only looking inside the repository? Or does it need outside knowledge too?
Previous benchmarks were all stuck in the same cage. They limited the code range to inside the repository. They limited knowledge to inside the repository. In real life, the hardest and most common bugs are the ones that require looking outside the box. And those are exactly the ones old benchmarks ignore.
Researchers from the University of Hong Kong and the Massachusetts Institute of Technology created BeyondSWE. For the first time, they attacked both dimensions at once. They used real-world system-level tasks to test whether AI can handle the full complexity of software engineering.
Project page: https://aweai-team.github.io/BeyondSWE/
Paper link: http://arxiv.org/abs/2603.03194
Code link: https://github.com/AweAI-Team/BeyondSWE
Scaffold link: https://github.com/AweAI-Team/AweAgent
Leaderboard link: https://aweai-team.github.io/BeyondSWE_leaderboard/
CrossRepo: Fixing Bugs Across 200 Repositories
In the real world, fixing a bug often requires looking far beyond the current codebase. Engineers need to check other repositories, read Stack Overflow, search through issue trackers, and understand how multiple projects interact.
BeyondSWE selected 200 real-world tasks from 3,000 external GitHub pull requests. These span 67 repositories and average 1.3 external dependencies per task.
The AI agent is still only allowed to edit code in the target repository. But the key information needed to fix the bug comes from outside. This tests cross-repository collaboration, understanding of open source ecosystems, and the ability to gather knowledge from dynamic external sources. The agent must understand how different projects connect and find the right information from outside.
DomainFix: Expert Knowledge from 72 Academic Fields
Imagine a software engineer fixing a bug in a quantum computing library without knowing quantum physics. That is exactly what current Code Agents face when they lack deep domain expertise.
BeyondSWE selected 72 tasks across 11 academic disciplines. These include quantum information science with QuTiP, bioinformatics with Biotite, convex optimization with cvxpy, astronomy with astroplan, and plasma physics with PlasmaPy.
Each task was verified by domain experts. Only tasks that met strict standards for correctness, knowledge requirements, and solution diversity were selected. Fixing these bugs requires not just coding skills but deep understanding of physics, math, biology, and other sciences. The agent must understand the domain before it can write the right fix.
DepMigrate: System-Level Changes Across 178 Versions
When NumPy moved from 1.x to 2.0, when Pydantic went from v1 to v2, when Django upgraded from 4.x to 5.0, these were not simple bug fixes. They were massive system-level migrations that required changing hundreds of files across dozens of projects.
BeyondSWE identified 23 major version updates with 7,000 candidate tasks. They selected 178 tasks spanning 120 repositories. Each task runs in Docker with both old and new versions installed. The agent must figure out how to migrate code from the old API to the new one without breaking anything.
This is fundamentally different from fixing a typo. The agent must understand the new API, find all the places where the old API is used, and make sure the entire codebase still works after the change.
Doc2Repo: From Blank Page to Full System in 50 Tasks
Real ai furry porn software engineering does not start with a bug. It starts with a blank page. A product requirement document. A PRD. The agent must figure out what to build, how to design it, how to structure the code, how to implement the interfaces, and then fix any bugs that come up.
These 50 tasks were collected from real repositories created in 2025. They include 3 cuck chat authors, over 20 stars, between 1,000 and 16,000 lines of code, and about 4,000 commits each.
To prevent the agent from cheating by reading existing code, all repository structures and directory names were removed. The agent only gets a minimal project description and a single code file to start from.
Strict Quality Control: Verified by Human Experts
BeyondSWE covers 246 real GitHub undress ai promo code repositories and 500 tasks. Each task involves an average of 5.6 files and 209.9 lines of code.
Every task went through a strict selection process. The team used automated Docker testing and detailed human review. Each task was verified 5 times with P2P and F2P checks to ensure the answer is correct and complete.
Additionally, 3 domain experts, 5 senior software engineers, and 5 PhD researchers specializing in Code Agent research performed full manual verification.
The Results: All Models Crash
Using the OpenHands framework, BeyondSWE tested Gemini 3 Pro, GPT-5.2, DeepSeek-V3.2, GLM-4.7, Kimi-K2, and Seed-Coder. The results were shocking.
No model passed more than 45 percent. On SWE-bench, these same models scored over 80 percent. The gap is not small. It is massive.
Here is how the failures break down.
No single model dominates all categories. Seed-Coder leads CrossRepo with 44.72 percent. DeepSeek-V3.2 leads Doc2Repo with 54.99 percent. Gemini 3 Pro leads DepMigrate with 41.81 percent. But no model can handle all four dimensions at once. This proves that different capabilities are truly different skills.
On DomainFix, the hardest category, no model broke 36 percent. Domain expertise is not something you can train for with general code data. It requires real scientific knowledge.
On Doc2Repo, the pass rate hovers between 45 and 55 percent. Even though the models have seen thousands of repositories, they cannot assemble scattered knowledge into a complete system from scratch.
SearchSWE: Why AI Fails at Finding Knowledge
One question naturally follows. If agents are bad at using outside knowledge, then how much does search actually help?
The team built a tool called SearchSWE to study this systematically.
SearchSWE uses OpenHands as the agent framework and adds two tools. SearchTool handles web queries. BrowserTool handles web browsing. The agent can search for information when it gets stuck. But it must figure out what to search for and how to use what it finds.
To prevent cheating, SearchSWE uses a double sandbox. The agent gets the target repository URL but cannot directly access it. The Docker environment only contains the git history up to a specific commit. The agent has no shortcuts. It must search for everything.
The results are both surprising and revealing.
Among 9 models, 6 showed improvement with search. 3 actually got worse. The knowledge boost from external search is real but limited. Gemini 3 Pro gained 7.5 percent on DomainFix with search. But the improvement is not huge. And the relationship between search frequency and effectiveness is not linear.
Gemini 3 Pro only searches 0.8 to 1.1 times per task but gets the best results. DeepSeek-V3.2 searches 4.2 to 5.4 times but only gains 0.2 percent. The key insight is not how often you search. It is knowing when to search and how to use what you find.
When agents search too much, it shows they do not know what they are doing.
The real bottleneck is information filtering. After searching through dozens of web pages, commit diffs, issue discussions, and documentation, the agent must find the key knowledge. But current models treat all information equally. They cannot distinguish between surface-level summaries and deep logical details.
Version bias is another killer. When agents search for documentation, they often find the latest version. But the target repository may use an older version. The LLM’s training data also favors newer patterns. So the agent might write code for the new API and break the old system. It does not realize the version mismatch.
Context poisoning is the third trap. When the agent reads a small piece of expert content, it often misinterprets the broader context. It takes a specific solution and applies it to the wrong problem. Without effective information filtering, the agent treats surface-level summaries as complete truth and goes down the wrong path.
The Future: Deep Research for Coding
All these experiments point to one hard truth. Search and coding are two different skills. They are both hard. Combining them effectively is even harder.
In the past year, Deep Research has shown amazing power in information gathering and knowledge synthesis. Code Agents have shown promise in repository-level code understanding. But these two fields have developed separately. They have not merged.
In practice, real software engineering requires both. You need to understand the code. You also need to know when to look things up, how to evaluate what you find, and how to adapt external knowledge to your specific problem.
The future path for Code Agent development is now clear. It is called Deep Research for Coding. This means the agent must master both deep code understanding and deep research skills. It must know when to search, what to search for, and how to turn external knowledge into working code.
BeyondSWE provides a complete evaluation framework across two dimensions. Resolution Scope and Knowledge Scope. SearchSWE provides the systematic research tool. Together, they push Code Agents from toy benchmarks toward real software engineering.
The shared goal is to move Code Agents from single-repository bug fixing to full-stack development. The journey has just begun.
