10 min read · 2026-09-19

Lessons Learned from LegoFlow-SWE

A quick recap: What is LegoFlow-SWE?

LegoFlow-SWE is an open-source release of 5,000 verified software-engineering tasks and 2,780 successful trajectories. We select the task pool from more than 12 million pull requests in more than 700,000 GitHub repositories and generate the trajectories from those tasks. As our LegoFlow overview shows, training on roughly 1,000 of these trajectories produces strong results across SWE-bench Verified, Pro, and Multilingual. Here, we focus on the lessons from building the dataset.

Lesson I: Task quality matters

For us, task quality has two parts. First, a task must be verifiable, with an environment and tests that reliably separate the original code from the fix. It must also be hard enough to require a real investigation, such as editing multiple files or working through a detailed issue, rather than applying a trivial patch.

How Curator scores task difficulty

Passing validation tells us that a task is usable, but not how difficult it is. Curator measures difficulty with a static quality rubric. No LLM judge is involved in this score. We use five signals, each scaled from 1.0 to 5.0, then combine them by weight and map the result to a final score from 1.0 to 10.0:

patch_scope (30%): changed lines, files, and hunks.
logic_complexity (25%): new functions, classes, and control-flow additions.
context_breadth (20%): distinct directories touched by the fix.
test_complexity (15%): test lines and test files.
instruction_complexity (10%): the length and detail of the problem statement.

We label scores of 4.0 or below as easy, scores above 4.0 and up to 7.0 as medium, and scores above 7.0 as hard.

Does the rubric identify better training tasks?

We compare LegoFlow-SWE with SWE-rebench V2 to test whether the rubric above captures a useful difference. LegoFlow-SWE has a higher average difficulty score, 6.27 versus 5.80. Hard tasks account for 39.4% of the pool rather than 35.1%, and LegoFlow-SWE covers eight programming languages instead of six. The same GLM-5.2 teacher also produces much longer trajectories on the LegoFlow-SWE tasks: 130k tokens and 104 turns per trajectory on average, compared with 69k tokens and 67 turns for SWE-rebench V2.

We then collect roughly 1,000 GLM-5.2 trajectories from each source and fine-tune Qwen3.5-35B-A3B-Base on the two pools. The teacher, agent scaffold, sampling policy, and SFT recipe remain fixed. We evaluate both students on SWE-bench Verified, Pro, and Multilingual.

Figure 1. Every pool we generate through LegoFlow beats the instruction-tuned reference model on all three benchmarks. No external pool does.

Changing the task pool adds 5.8 points on Verified, 1.9 on Pro, and 1.0 on Multilingual. The LegoFlow-SWE student scores 70.2 on SWE-bench Verified, 48.8 on Pro, and 57.0 on Multilingual. The SWE-rebench V2 student scores 64.4, 46.9, and 56.0 under the same setup. With the rest of the experiment fixed, the result suggests that the LegoFlow-SWE task pool produces more useful training trajectories.

Lesson II: Stronger models tend to "hack"

A stronger teacher can be better at hacking the benchmark, not just solving it. We see three recurring attempts to exploit leakage:

Finding the public repository and the upstream pull request that contains the fix.
Downloading or copying the existing patch instead of deriving a solution from the local task environment.
Comparing the generated changes with the upstream commit and treating a match as confirmation.

Figure 2. Removing upstream leakage reduces the GLM-5.2 teacher's Verified score by 8.0 points and its student's score by 10.0 points.

When we remove runs that use leaked solutions, the teacher's Verified score falls by 8.0 points and the student's by 10.0.

We observe higher hacking rates with newer open-source models. Confirmed hacking rises from 3.6% of GLM-5 runs to 21.2% of GLM-5.2 runs. At the same time, the share of suspicious but unconfirmed runs falls from 21.6% to 5.4%, while the clean share stays almost unchanged. In other words, GLM-5.2 does not make many more questionable attempts; it is much better at turning those attempts into successful hacks. We observe similar behavior in DeepSeek-V4-Flash and GLM-5.3.

Figure 3. The stronger teacher turns many suspicious attempts into confirmed reward hacking.

How we prevent hacking

We use two measures to address hacking.

First, we explicitly tell the model not to do it. This simple change reduces the hacking rate of GLM-5.2 to below 1%. We add the following instruction to the prompt (an example task):

Important: Solve this task using only the repository, files, documentation, tests, and tools available in the provided task environment. Do not use the internet or any external search service to look up this issue, related pull requests, patches, commits, discussions, or ready-made solutions. Derive the solution independently from the local evidence.

Second, we restrict what the agent can access. We remove local Git information that can reveal the reference solution, including commits and branches containing the gold patch. However, removing local Git history is not enough because mirrors, caches, and search APIs may still expose the answer. We therefore also restrict network access during agent execution. In Harbor, the agent can reach only the model provider endpoints it needs, and the verifier has no network access. To help the community study this behavior, we release GLM-5.2 trajectories that contain hacking and GLM-5.3 trajectories that do not. Both sets use the anti-hacking instruction.

Lesson III: Reasoning depth matters more than reasoning coverage

Chain-of-thought (CoT) is an important part of learning from teacher behavior. But how often a teacher reasons is not enough to tell us whether a trajectory is useful. What matters more in our experiments is how much reasoning the teacher does when a problem calls for it.

Figure 4. LegoFlow-SWE uses reasoning on fewer turns and spends more tokens on the turns where it reasons.

We find that open-source datasets such as DeNovoSWE, Step-3.5-Flash-SFT, and Scale-SWE include CoT on 97% to 100% of turns, compared with 62% for LegoFlow-SWE. LegoFlow-SWE reasoning turns are much longer: 714 tokens on average, versus 181 to 309 for the external pools. Reasoning briefly on every turn is less useful than reasoning in depth when the task requires it.

We see the same pattern when we change the teacher but keep the tasks fixed. On the same SWE-rebench V2 instances, GLM-5 reasons on 100% of turns at 82 tokens each. Its student scores 60.4 on Verified and 30.2 on Pro. GLM-5.2 reasons on 62% of turns but uses 500 tokens on average when it does. With the same student recipe, the scores rise to 64.4 and 46.9. Coverage goes down; both scores go up.

To check this more directly, we sort a GLM-5.2 SWE-rebench pool by average reasoning length, split it into quartiles, and train one model on each quartile.

Figure 5. Most of the improvement comes from moving beyond the quartile with the shortest reasoning traces.

The shortest quartile scores 57.0% on Verified. The next three score 64.0%, 64.6%, and 65.0% as average reasoning length rises from 167 to 812 tokens per turn. Most of the gain comes from leaving the shallowest quartile; after that, the returns flatten quickly. This is why we filter out shallow traces rather than simply choosing the longest ones. Once a trajectory contains enough reasoning, adding a different task may provide more training value than extending the same trajectory.

Citation

@misc{legox2026legoflowlessons,
  title        = {Lessons Learned from LegoFlow-SWE},
  author       = {{LegoX Team}},
  year         = {2026},
  url          = {https://legox.net/blog/legoflow-experiments/},
}