All evidence cases
DeepSWE v1.1repositoryPython

Add grouped test phases with synchronized barriers

Add grouped execution phases with explicit synchronization barriers and per-group setup and teardown.

mobly-grouped-test-barriers

Normalized profile

What this task asks for

Add grouped execution phases with explicit synchronization barriers and per-group setup and teardown. Run global setup and teardown around no-entry, implicit-group, or explicit-group execution modes. Run explicit group participants concurrently between one group setup and one group teardown. Resolve participant group and ID from configuration entries and expose device context only in group phases and tests. Synchronize explicit participants by instance, group, hook or test, and step name with reusable barriers. Reject invalid synchronization contexts and timeouts with the documented exception types and messages. Continue other groups after group setup failures and always run the applicable teardown hooks.

  • feature implementation
  • bug repair
  • test and validation
  • configuration
  • task scheduler
  • library API
  • build system
  • Mobly
  • repository patch
  • concurrency, cancellation, or distributed-state coordination
  • quantitative performance, accuracy, or resource constraints
  • consistent behavior across lifecycle stages or execution modes
  • changes span multiple components or interfaces

Published evidence

Model × harness outcomes

Counts are tied to these exact configurations. Harnesses are not treated as equivalent across benchmarks.

Models
13
Configurations
41
Trials
164

Showing 41 of 41 configurations

ModelHarnessEffortResultPasses
claude-fable-5vertex_aimini-swe-agentpublished configurationlowAll passed4/4
claude-fable-5vertex_aimini-swe-agentpublished configurationmediumMixed2/4
claude-fable-5vertex_aimini-swe-agentpublished configurationhighAll passed4/4
claude-fable-5vertex_aimini-swe-agentpublished configurationxhighMixed3/4
claude-fable-5vertex_aimini-swe-agentpublished configurationmaxAll passed4/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationlowMixed1/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationmediumMixed1/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationhighMixed2/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationxhighMixed2/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationmaxMixed2/4
claude-sonnet-4-6anthropicmini-swe-agentpublished configurationhighNo passes0/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationlowMixed2/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationmediumAll passed4/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationhighAll passed4/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationxhighAll passed4/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationmaxMixed3/4
gemini-3-1-pro-previewvertex_aimini-swe-agentpublished configurationhighNo passes0/4
gemini-3-5-flashvertex_aimini-swe-agentpublished configurationmediumMixed3/4
glm-5-2zaimini-swe-agentpublished configurationhighNo passes0/4
glm-5-2zaimini-swe-agentpublished configurationmaxNo passes0/4
gpt-5-4openaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-5openaimini-swe-agentpublished configurationlowMixed1/4
gpt-5-5openaimini-swe-agentpublished configurationmediumMixed3/4
gpt-5-5openaimini-swe-agentpublished configurationhighAll passed4/4
gpt-5-5openaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationlowNo passes0/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationmediumNo passes0/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationhighMixed2/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationmaxAll passed4/4
gpt-5-6-solopenaimini-swe-agentpublished configurationlowMixed2/4
gpt-5-6-solopenaimini-swe-agentpublished configurationmediumMixed1/4
gpt-5-6-solopenaimini-swe-agentpublished configurationhighAll passed4/4
gpt-5-6-solopenaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-6-solopenaimini-swe-agentpublished configurationmaxAll passed4/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationlowNo passes0/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationmediumMixed2/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationhighNo passes0/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationmaxMixed2/4
kimi-k2-7-codeopenroutermini-swe-agentpublished configurationdefaultNo passes0/4