Terminal-Bench 2.1terminalGit repository format
Replace secrets in the dclm repository
Find API credentials in the repository and replace each with a consistent service-specific placeholder without changing uncontaminated files.
sanitize-git-repo
Normalized profile
What this task asks for
Find API credentials in the repository and replace each with a consistent service-specific placeholder without changing uncontaminated files. Replace AWS access IDs, AWS secret keys, GitHub tokens, and Hugging Face tokens with the prescribed placeholders. Remove every original sensitive value. Use consistent placeholders across files. Leave uncontaminated files unchanged.
Intent
- security remediation
Work surfaces
- repository files
- secret values
Technology
- Git
- AWS credentials
- GitHub tokens
- Hugging Face tokens
Expected artifacts
- sanitized dclm repository
What makes it difficult
- strict behavioral compatibility or exact-output validation
Published evidence
Model × harness outcomes
Counts are tied to these exact configurations. Harnesses are not treated as equivalent across benchmarks.
- Models
- 13
- Configurations
- 20
- Trials
- 100
Showing 20 of 20 configurations
| Model | Harness | Effort | Result | Passes |
|---|---|---|---|---|
| claude-fable-5anthropic · 2026-06-07 | claude-code2.1.167 | xhigh | Mixed | 3/5 |
| claude-fable-5anthropic · 2026-06-05 | terminus-22.0.0 | high | Mixed | 2/5 |
| claude-opus-4-7anthropic · 2026-05-01 | claude-code2.1.123 | max | All passed | 5/5 |
| claude-opus-4-7anthropic · 2026-05-01 | terminus-22.0.0 | max | Mixed | 4/5 |
| claude-opus-4-8anthropic · 2026-07-09 | claude-code2.1.205 | high | All passed | 5/5 |
| claude-sonnet-5anthropic · 2026-07-09 | claude-code2.1.205 | high | Mixed | 3/5 |
| gemini-3-pro-previewgemini · 2026-05-01 | gemini-cli0.40.0 | high | Mixed1 errors | 2/5 |
| gemini-3-pro-previewgemini · 2026-05-01 | terminus-22.0.0 | high | Mixed | 2/5 |
| gemini-3.1-pro-previewgemini · 2026-05-05 | gemini-cli0.40.0 | high | Mixed | 2/5 |
| gemini-3.1-pro-previewgemini · 2026-05-05 | terminus-22.0.0 | high | No passes | 0/5 |
| glm-5.1zai · 2026-05-01 | claude-code2.1.123 | max | No passes | 0/5 |
| gpt-5.5openai · 2026-05-01 | codex0.125.0 | xhigh | Mixed | 4/5 |
| gpt-5.5openai · 2026-05-01 | terminus-22.0.0 | xhigh | Mixed | 4/5 |
| gpt-5.6-lunaopenai · 2026-07-10 | codex0.144.0 | max | Mixed | 2/5 |
| gpt-5.6-lunaopenai · 2026-07-11 | codex0.144.1 | max | Mixed | 3/5 |
| gpt-5.6-solopenai · 2026-07-10 | codex0.144.0 | max | Mixed1 errors | 2/5 |
| gpt-5.6-terraopenai · 2026-07-10 | codex0.144.0 | max | No passes1 errors | 0/5 |
| gpt-5.6-terraopenai · 2026-07-11 | codex0.144.1 | max | No passes1 errors | 0/5 |
| grok-4.5cursor · 2026-07-09 | cursor-cli2026.07.08-0c04a8a | default | Mixed | 4/5 |
| muse-spark-1.1openai · 2026-07-09 | mini-swe-agent2.4.5 | xhigh | Mixed | 1/5 |