Terminal-Bench 2.1terminalSchemePython
Implement a self-interpreting Scheme evaluator
Write a Scheme metacircular evaluator for the supplied language that runs all programs, forwards remaining input and output, and can interpret itself recursively.
schemelike-metacircular-eval
Normalized profile
What this task asks for
Write a Scheme metacircular evaluator for the supplied language that runs all programs, forwards remaining input and output, and can interpret itself recursively. Read exactly one program path line before forwarding remaining input. Match interp.py semantics for every program under test/. Send interpreted output to standard output. Support eval.scm interpreting another eval.scm.
Intent
- feature implementation
- test and validation
Work surfaces
- language interpreter
- standard input and output
Technology
- Scheme
- metacircular evaluation
Expected artifacts
- eval.scm
What makes it difficult
- strict behavioral compatibility or exact-output validation
- quantitative performance, accuracy, or resource constraints
- cross-language or cross-format integration
Published evidence
Model × harness outcomes
Counts are tied to these exact configurations. Harnesses are not treated as equivalent across benchmarks.
- Models
- 13
- Configurations
- 20
- Trials
- 100
Showing 20 of 20 configurations
| Model | Harness | Effort | Result | Passes |
|---|---|---|---|---|
| claude-fable-5anthropic · 2026-06-07 | claude-code2.1.167 | xhigh | Mixed1 errors | 4/5 |
| claude-fable-5anthropic · 2026-06-05 | terminus-22.0.0 | high | Mixed2 errors | 3/5 |
| claude-opus-4-7anthropic · 2026-05-01 | claude-code2.1.123 | max | Mixed3 errors | 2/5 |
| claude-opus-4-7anthropic · 2026-05-01 | terminus-22.0.0 | max | No passes5 errors | 0/5 |
| claude-opus-4-8anthropic · 2026-07-09 | claude-code2.1.205 | high | Mixed | 4/5 |
| claude-sonnet-5anthropic · 2026-07-09 | claude-code2.1.205 | high | Mixed1 errors | 3/5 |
| gemini-3-pro-previewgemini · 2026-05-01 | gemini-cli0.40.0 | high | All passed | 5/5 |
| gemini-3-pro-previewgemini · 2026-05-01 | terminus-22.0.0 | high | Mixed | 3/5 |
| gemini-3.1-pro-previewgemini · 2026-05-05 | gemini-cli0.40.0 | high | All passed | 5/5 |
| gemini-3.1-pro-previewgemini · 2026-05-05 | terminus-22.0.0 | high | Mixed | 3/5 |
| glm-5.1zai · 2026-05-01 | claude-code2.1.123 | max | No passes5 errors | 0/5 |
| gpt-5.5openai · 2026-05-01 | codex0.125.0 | xhigh | All passed | 5/5 |
| gpt-5.5openai · 2026-05-01 | terminus-22.0.0 | xhigh | All passed | 5/5 |
| gpt-5.6-lunaopenai · 2026-07-10 | codex0.144.0 | max | Mixed | 4/5 |
| gpt-5.6-lunaopenai · 2026-07-11 | codex0.144.1 | max | Mixed | 4/5 |
| gpt-5.6-solopenai · 2026-07-10 | codex0.144.0 | max | All passed | 5/5 |
| gpt-5.6-terraopenai · 2026-07-10 | codex0.144.0 | max | All passed | 5/5 |
| gpt-5.6-terraopenai · 2026-07-11 | codex0.144.1 | max | Mixed | 4/5 |
| grok-4.5cursor · 2026-07-09 | cursor-cli2026.07.08-0c04a8a | default | Mixed | 4/5 |
| muse-spark-1.1openai · 2026-07-09 | mini-swe-agent2.4.5 | xhigh | Mixed2 errors | 2/5 |