Terminal-Bench 2.1terminalPythonProtocol Buffers
Run a gRPC key-value service
Define, generate, and run a Python gRPC service that stores integer values under string keys and exposes GetVal and SetVal RPCs.
kv-store-grpc
Normalized profile
What this task asks for
Define, generate, and run a Python gRPC service that stores integer values under string keys and exposes GetVal and SetVal RPCs. Define KVStore with GetVal and SetVal. Use string keys and integer values. Implement class Server over a Python dictionary. Leave the service running on port 5328.
Intent
- feature implementation
- systems setup
Work surfaces
- RPC API
- in-memory store
- background service
Technology
- gRPC 1.73.0
- Protocol Buffers
- Python
Expected artifacts
- /app/kv-store.proto
- generated Python stubs
- /app/server.py
- service on port 5328
What makes it difficult
- quantitative performance, accuracy, or resource constraints
- specialized algorithmic or domain reasoning
- cross-language or cross-format integration
- changes span multiple components or interfaces
Published evidence
Model × harness outcomes
Counts are tied to these exact configurations. Harnesses are not treated as equivalent across benchmarks.
- Models
- 13
- Configurations
- 20
- Trials
- 100
Showing 20 of 20 configurations
| Model | Harness | Effort | Result | Passes |
|---|---|---|---|---|
| claude-fable-5anthropic · 2026-06-07 | claude-code2.1.167 | xhigh | All passed | 5/5 |
| claude-fable-5anthropic · 2026-06-05 | terminus-22.0.0 | high | All passed | 5/5 |
| claude-opus-4-7anthropic · 2026-05-01 | claude-code2.1.123 | max | No passes5 errors | 0/5 |
| claude-opus-4-7anthropic · 2026-05-01 | terminus-22.0.0 | max | All passed | 5/5 |
| claude-opus-4-8anthropic · 2026-07-09 | claude-code2.1.205 | high | All passed | 5/5 |
| claude-sonnet-5anthropic · 2026-07-09 | claude-code2.1.205 | high | All passed | 5/5 |
| gemini-3-pro-previewgemini · 2026-05-01 | gemini-cli0.40.0 | high | No passes | 0/5 |
| gemini-3-pro-previewgemini · 2026-05-01 | terminus-22.0.0 | high | All passed | 5/5 |
| gemini-3.1-pro-previewgemini · 2026-05-05 | gemini-cli0.40.0 | high | No passes | 0/5 |
| gemini-3.1-pro-previewgemini · 2026-05-05 | terminus-22.0.0 | high | Mixed | 4/5 |
| glm-5.1zai · 2026-05-01 | claude-code2.1.123 | max | Mixed | 4/5 |
| gpt-5.5openai · 2026-05-01 | codex0.125.0 | xhigh | Mixed | 3/5 |
| gpt-5.5openai · 2026-05-01 | terminus-22.0.0 | xhigh | All passed | 5/5 |
| gpt-5.6-lunaopenai · 2026-07-10 | codex0.144.0 | max | Mixed | 1/5 |
| gpt-5.6-lunaopenai · 2026-07-11 | codex0.144.1 | max | Mixed | 1/5 |
| gpt-5.6-solopenai · 2026-07-10 | codex0.144.0 | max | No passes | 0/5 |
| gpt-5.6-terraopenai · 2026-07-10 | codex0.144.0 | max | No passes | 0/5 |
| gpt-5.6-terraopenai · 2026-07-11 | codex0.144.1 | max | No passes | 0/5 |
| grok-4.5cursor · 2026-07-09 | cursor-cli2026.07.08-0c04a8a | default | No passes5 errors | 0/5 |
| muse-spark-1.1openai · 2026-07-09 | mini-swe-agent2.4.5 | xhigh | Mixed | 4/5 |