Add hierarchical evaluation cancellation to Boa
Add cancellable evaluation handles that propagate through nested script, module, and job execution.
boa-hierarchical-evaluation-cancellation
Normalized profile
What this task asks for
Add cancellable evaluation handles that propagate through nested script, module, and job execution. `Context::{eval_with_evaluation, enqueue_job_with_evaluation, run_jobs_with_evaluation}` must each return a fallible result with the same result-shape category as its non-handle analog. `cancellation_reason(context)` must return an optional value (`None` when not cancelled, `Some(reason)` when cancelled). For an already-cancelled handle, `Module::evaluate_with_evaluation` must still return success with a rejected promise. `Context::enqueue_job_with_evaluation(job, handle)` must fail immediately when `handle` is already cancelled and must not enqueue that job. If cancellation happens without a custom reason, `cancellation_reason(context)` must produce an Error-like value whose string contains `AbortError`. `Context::run_jobs_with_evaluation(handle)` must fail immediately when `handle` is already cancelled and must not drain queued jobs in that failed call.
Intent
- feature implementation
- configuration tuning
- data transformation
Work surfaces
- library API
Technology
- Boa
Expected artifacts
- repository patch
What makes it difficult
- concurrency, cancellation, or distributed-state coordination
Published evidence
Model × harness outcomes
Counts are tied to these exact configurations. Harnesses are not treated as equivalent across benchmarks.
- Models
- 13
- Configurations
- 41
- Trials
- 163
Showing 41 of 41 configurations
| Model | Harness | Effort | Result | Passes |
|---|---|---|---|---|
| claude-fable-5vertex_ai | mini-swe-agentpublished configuration | low | Mixed | 1/4 |
| claude-fable-5vertex_ai | mini-swe-agentpublished configuration | medium | Mixed | 3/4 |
| claude-fable-5vertex_ai | mini-swe-agentpublished configuration | high | Mixed | 3/4 |
| claude-fable-5vertex_ai | mini-swe-agentpublished configuration | xhigh | All passed | 4/4 |
| claude-fable-5vertex_ai | mini-swe-agentpublished configuration | max | All passed | 4/4 |
| claude-opus-4-8anthropic | mini-swe-agentpublished configuration | low | Mixed | 1/4 |
| claude-opus-4-8anthropic | mini-swe-agentpublished configuration | medium | Mixed | 1/4 |
| claude-opus-4-8anthropic | mini-swe-agentpublished configuration | high | Mixed | 2/4 |
| claude-opus-4-8anthropic | mini-swe-agentpublished configuration | xhigh | Mixed | 3/4 |
| claude-opus-4-8anthropic | mini-swe-agentpublished configuration | max | All passed | 4/4 |
| claude-sonnet-4-6anthropic | mini-swe-agentpublished configuration | high | No passes | 0/4 |
| claude-sonnet-5anthropic | mini-swe-agentpublished configuration | low | No passes | 0/4 |
| claude-sonnet-5anthropic | mini-swe-agentpublished configuration | medium | No passes | 0/4 |
| claude-sonnet-5anthropic | mini-swe-agentpublished configuration | high | No passes | 0/4 |
| claude-sonnet-5anthropic | mini-swe-agentpublished configuration | xhigh | No passes | 0/4 |
| claude-sonnet-5anthropic | mini-swe-agentpublished configuration | max | No passes | 0/4 |
| gemini-3-1-pro-previewvertex_ai | mini-swe-agentpublished configuration | high | No passes | 0/4 |
| gemini-3-5-flashvertex_ai | mini-swe-agentpublished configuration | medium | Mixed | 1/4 |
| glm-5-2zai | mini-swe-agentpublished configuration | high | Mixed | 2/4 |
| glm-5-2zai | mini-swe-agentpublished configuration | max | Mixed | 1/4 |
| gpt-5-4openai | mini-swe-agentpublished configuration | xhigh | Mixed | 1/4 |
| gpt-5-5openai | mini-swe-agentpublished configuration | low | No passes | 0/4 |
| gpt-5-5openai | mini-swe-agentpublished configuration | medium | Mixed | 2/4 |
| gpt-5-5openai | mini-swe-agentpublished configuration | high | Mixed | 2/4 |
| gpt-5-5openai | mini-swe-agentpublished configuration | xhigh | All passed | 4/4 |
| gpt-5-6-lunaopenai | mini-swe-agentpublished configuration | low | No passes | 0/4 |
| gpt-5-6-lunaopenai | mini-swe-agentpublished configuration | medium | No passes | 0/4 |
| gpt-5-6-lunaopenai | mini-swe-agentpublished configuration | high | Mixed | 2/4 |
| gpt-5-6-lunaopenai | mini-swe-agentpublished configuration | xhigh | Mixed | 2/4 |
| gpt-5-6-lunaopenai | mini-swe-agentpublished configuration | max | All passed | 4/4 |
| gpt-5-6-solopenai | mini-swe-agentpublished configuration | low | Mixed | 2/4 |
| gpt-5-6-solopenai | mini-swe-agentpublished configuration | medium | Mixed | 2/4 |
| gpt-5-6-solopenai | mini-swe-agentpublished configuration | high | Mixed | 2/4 |
| gpt-5-6-solopenai | mini-swe-agentpublished configuration | xhigh | All passed | 4/4 |
| gpt-5-6-solopenai | mini-swe-agentpublished configuration | max | All passed | 4/4 |
| gpt-5-6-terraopenai | mini-swe-agentpublished configuration | low | No passes | 0/3 |
| gpt-5-6-terraopenai | mini-swe-agentpublished configuration | medium | Mixed | 1/4 |
| gpt-5-6-terraopenai | mini-swe-agentpublished configuration | high | Mixed | 2/4 |
| gpt-5-6-terraopenai | mini-swe-agentpublished configuration | xhigh | Mixed | 2/4 |
| gpt-5-6-terraopenai | mini-swe-agentpublished configuration | max | Mixed | 3/4 |
| kimi-k2-7-codeopenrouter | mini-swe-agentpublished configuration | default | Mixed | 1/4 |