All evidence cases
DeepSWE v1.1repositoryRust

Add hierarchical evaluation cancellation to Boa

Add cancellable evaluation handles that propagate through nested script, module, and job execution.

boa-hierarchical-evaluation-cancellation

Normalized profile

What this task asks for

Add cancellable evaluation handles that propagate through nested script, module, and job execution. `Context::{eval_with_evaluation, enqueue_job_with_evaluation, run_jobs_with_evaluation}` must each return a fallible result with the same result-shape category as its non-handle analog. `cancellation_reason(context)` must return an optional value (`None` when not cancelled, `Some(reason)` when cancelled). For an already-cancelled handle, `Module::evaluate_with_evaluation` must still return success with a rejected promise. `Context::enqueue_job_with_evaluation(job, handle)` must fail immediately when `handle` is already cancelled and must not enqueue that job. If cancellation happens without a custom reason, `cancellation_reason(context)` must produce an Error-like value whose string contains `AbortError`. `Context::run_jobs_with_evaluation(handle)` must fail immediately when `handle` is already cancelled and must not drain queued jobs in that failed call.

  • feature implementation
  • configuration tuning
  • data transformation
  • library API
  • Boa
  • repository patch
  • concurrency, cancellation, or distributed-state coordination

Published evidence

Model × harness outcomes

Counts are tied to these exact configurations. Harnesses are not treated as equivalent across benchmarks.

Models
13
Configurations
41
Trials
163

Showing 41 of 41 configurations

ModelHarnessEffortResultPasses
claude-fable-5vertex_aimini-swe-agentpublished configurationlowMixed1/4
claude-fable-5vertex_aimini-swe-agentpublished configurationmediumMixed3/4
claude-fable-5vertex_aimini-swe-agentpublished configurationhighMixed3/4
claude-fable-5vertex_aimini-swe-agentpublished configurationxhighAll passed4/4
claude-fable-5vertex_aimini-swe-agentpublished configurationmaxAll passed4/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationlowMixed1/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationmediumMixed1/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationhighMixed2/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationxhighMixed3/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationmaxAll passed4/4
claude-sonnet-4-6anthropicmini-swe-agentpublished configurationhighNo passes0/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationlowNo passes0/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationmediumNo passes0/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationhighNo passes0/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationxhighNo passes0/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationmaxNo passes0/4
gemini-3-1-pro-previewvertex_aimini-swe-agentpublished configurationhighNo passes0/4
gemini-3-5-flashvertex_aimini-swe-agentpublished configurationmediumMixed1/4
glm-5-2zaimini-swe-agentpublished configurationhighMixed2/4
glm-5-2zaimini-swe-agentpublished configurationmaxMixed1/4
gpt-5-4openaimini-swe-agentpublished configurationxhighMixed1/4
gpt-5-5openaimini-swe-agentpublished configurationlowNo passes0/4
gpt-5-5openaimini-swe-agentpublished configurationmediumMixed2/4
gpt-5-5openaimini-swe-agentpublished configurationhighMixed2/4
gpt-5-5openaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationlowNo passes0/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationmediumNo passes0/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationhighMixed2/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationxhighMixed2/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationmaxAll passed4/4
gpt-5-6-solopenaimini-swe-agentpublished configurationlowMixed2/4
gpt-5-6-solopenaimini-swe-agentpublished configurationmediumMixed2/4
gpt-5-6-solopenaimini-swe-agentpublished configurationhighMixed2/4
gpt-5-6-solopenaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-6-solopenaimini-swe-agentpublished configurationmaxAll passed4/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationlowNo passes0/3
gpt-5-6-terraopenaimini-swe-agentpublished configurationmediumMixed1/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationhighMixed2/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationxhighMixed2/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationmaxMixed3/4
kimi-k2-7-codeopenroutermini-swe-agentpublished configurationdefaultMixed1/4