All evidence cases
DeepSWE v1.1repositoryJavaScript

Add bail-on-test-failure handling to Testem

Add configurable early bailout on test failures across runners, reporters, and exit codes.

testem-bail-on-test-failure

Normalized profile

What this task asks for

Add configurable early bailout on test failures across runners, reporters, and exit codes. Add `bail_on_test_failure` with false, true, or positive-integer thresholds and validate invalid values. Stop on the configured non-skipped failure count and expose resettable reason, launcher, count, and failed-test state. Emit bailout summaries through TAP, Dot, TeamCity, and XUnit reporters. Make runner, server, and application abort operations idempotent and suppress later results or errors. Guard Mocha, Jasmine2, and QUnit browser emissions after abort, including deferred callbacks. Return a distinct bail exit error using only the reported bail reason and pre-bail test count.

  • feature implementation
  • configuration tuning
  • test and validation
  • HTTP server
  • configuration
  • network service
  • test runner
  • library API
  • Testem
  • TeamCity
  • xUnit
  • TAP
  • repository patch
  • changes span multiple components or interfaces

Published evidence

Model × harness outcomes

Counts are tied to these exact configurations. Harnesses are not treated as equivalent across benchmarks.

Models
13
Configurations
41
Trials
164

Showing 41 of 41 configurations

ModelHarnessEffortResultPasses
claude-fable-5vertex_aimini-swe-agentpublished configurationlowMixed1/4
claude-fable-5vertex_aimini-swe-agentpublished configurationmediumMixed1/4
claude-fable-5vertex_aimini-swe-agentpublished configurationhighMixed2/4
claude-fable-5vertex_aimini-swe-agentpublished configurationxhighMixed3/4
claude-fable-5vertex_aimini-swe-agentpublished configurationmaxAll passed4/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationlowNo passes0/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationmediumMixed2/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationhighNo passes0/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationxhighMixed2/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationmaxNo passes0/4
claude-sonnet-4-6anthropicmini-swe-agentpublished configurationhighNo passes0/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationlowNo passes0/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationmediumNo passes0/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationhighMixed1/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationxhighMixed1/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationmaxMixed1/4
gemini-3-1-pro-previewvertex_aimini-swe-agentpublished configurationhighNo passes0/4
gemini-3-5-flashvertex_aimini-swe-agentpublished configurationmediumMixed1/4
glm-5-2zaimini-swe-agentpublished configurationhighNo passes0/4
glm-5-2zaimini-swe-agentpublished configurationmaxNo passes0/4
gpt-5-4openaimini-swe-agentpublished configurationxhighNo passes0/4
gpt-5-5openaimini-swe-agentpublished configurationlowNo passes0/4
gpt-5-5openaimini-swe-agentpublished configurationmediumNo passes0/4
gpt-5-5openaimini-swe-agentpublished configurationhighNo passes0/4
gpt-5-5openaimini-swe-agentpublished configurationxhighMixed1/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationlowNo passes0/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationmediumNo passes0/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationhighNo passes0/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationxhighNo passes0/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationmaxMixed2/4
gpt-5-6-solopenaimini-swe-agentpublished configurationlowNo passes0/4
gpt-5-6-solopenaimini-swe-agentpublished configurationmediumNo passes0/4
gpt-5-6-solopenaimini-swe-agentpublished configurationhighNo passes0/4
gpt-5-6-solopenaimini-swe-agentpublished configurationxhighNo passes0/4
gpt-5-6-solopenaimini-swe-agentpublished configurationmaxNo passes0/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationlowNo passes0/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationmediumNo passes0/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationhighNo passes0/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationxhighNo passes0/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationmaxNo passes0/4
kimi-k2-7-codeopenroutermini-swe-agentpublished configurationdefaultNo passes0/4