Terminal-Bench 2.1terminalC
Implement compact GPT-2 inference in C
Write a dependency-free C program under 5,000 bytes that loads GPT-2 TensorFlow checkpoints and BPE vocabulary and emits 20 argmax tokens.
gpt2-codegolf
Normalized profile
What this task asks for
Write a dependency-free C program under 5,000 bytes that loads GPT-2 TensorFlow checkpoints and BPE vocabulary and emits 20 argmax tokens. Remain under 5,000 source bytes. Compile with gcc -O3 -lm without dependencies. Read the checkpoint, BPE file, and prompt from command-line arguments. Continue output for exactly 20 argmax-selected tokens.
Intent
- feature implementation
- performance optimization
Work surfaces
- model inference
- checkpoint loader
- tokenizer
Technology
- GPT-2
- TensorFlow checkpoints
- BPE
- GCC
Expected artifacts
- /app/gpt2.c
What makes it difficult
- strict behavioral compatibility or exact-output validation
- quantitative performance, accuracy, or resource constraints
- native compilation and dependency integration
- changes span multiple components or interfaces
Published evidence
Model × harness outcomes
Counts are tied to these exact configurations. Harnesses are not treated as equivalent across benchmarks.
- Models
- 13
- Configurations
- 20
- Trials
- 100
Showing 20 of 20 configurations
| Model | Harness | Effort | Result | Passes |
|---|---|---|---|---|
| claude-fable-5anthropic · 2026-06-07 | claude-code2.1.167 | xhigh | All passed | 5/5 |
| claude-fable-5anthropic · 2026-06-05 | terminus-22.0.0 | high | Mixed2 errors | 3/5 |
| claude-opus-4-7anthropic · 2026-05-01 | claude-code2.1.123 | max | No passes5 errors | 0/5 |
| claude-opus-4-7anthropic · 2026-05-01 | terminus-22.0.0 | max | No passes5 errors | 0/5 |
| claude-opus-4-8anthropic · 2026-07-09 | claude-code2.1.205 | high | Mixed4 errors | 1/5 |
| claude-sonnet-5anthropic · 2026-07-09 | claude-code2.1.205 | high | No passes5 errors | 0/5 |
| gemini-3-pro-previewgemini · 2026-05-01 | gemini-cli0.40.0 | high | Mixed3 errors | 1/5 |
| gemini-3-pro-previewgemini · 2026-05-01 | terminus-22.0.0 | high | No passes5 errors | 0/5 |
| gemini-3.1-pro-previewgemini · 2026-05-05 | gemini-cli0.40.0 | high | Mixed2 errors | 1/5 |
| gemini-3.1-pro-previewgemini · 2026-05-05 | terminus-22.0.0 | high | No passes4 errors | 0/5 |
| glm-5.1zai · 2026-05-01 | claude-code2.1.123 | max | No passes5 errors | 0/5 |
| gpt-5.5openai · 2026-05-01 | codex0.125.0 | xhigh | All passed | 5/5 |
| gpt-5.5openai · 2026-05-01 | terminus-22.0.0 | xhigh | No passes5 errors | 0/5 |
| gpt-5.6-lunaopenai · 2026-07-10 | codex0.144.0 | max | No passes5 disqualified | 0/5 |
| gpt-5.6-lunaopenai · 2026-07-11 | codex0.144.1 | max | Mixed4 errors | 1/5 |
| gpt-5.6-solopenai · 2026-07-10 | codex0.144.0 | max | Mixed3 disqualified | 2/5 |
| gpt-5.6-terraopenai · 2026-07-10 | codex0.144.0 | max | Mixed3 disqualified | 2/5 |
| gpt-5.6-terraopenai · 2026-07-11 | codex0.144.1 | max | Mixed1 errors | 4/5 |
| grok-4.5cursor · 2026-07-09 | cursor-cli2026.07.08-0c04a8a | default | No passes5 disqualified | 0/5 |
| muse-spark-1.1openai · 2026-07-09 | mini-swe-agent2.4.5 | xhigh | No passes5 errors | 0/5 |