Add bail-on-test-failure handling to Testem
Add configurable early bailout on test failures across runners, reporters, and exit codes.
testem-bail-on-test-failure
Normalized profile
What this task asks for
Add configurable early bailout on test failures across runners, reporters, and exit codes. Add `bail_on_test_failure` with false, true, or positive-integer thresholds and validate invalid values. Stop on the configured non-skipped failure count and expose resettable reason, launcher, count, and failed-test state. Emit bailout summaries through TAP, Dot, TeamCity, and XUnit reporters. Make runner, server, and application abort operations idempotent and suppress later results or errors. Guard Mocha, Jasmine2, and QUnit browser emissions after abort, including deferred callbacks. Return a distinct bail exit error using only the reported bail reason and pre-bail test count.
Intent
- feature implementation
- configuration tuning
- test and validation
Work surfaces
- HTTP server
- configuration
- network service
- test runner
- library API
Technology
- Testem
- TeamCity
- xUnit
- TAP
Expected artifacts
- repository patch
What makes it difficult
- changes span multiple components or interfaces
Published evidence
Model × harness outcomes
Counts are tied to these exact configurations. Harnesses are not treated as equivalent across benchmarks.
- Models
- 13
- Configurations
- 41
- Trials
- 164
Showing 41 of 41 configurations
| Model | Harness | Effort | Result | Passes |
|---|---|---|---|---|
| claude-fable-5vertex_ai | mini-swe-agentpublished configuration | low | Mixed | 1/4 |
| claude-fable-5vertex_ai | mini-swe-agentpublished configuration | medium | Mixed | 1/4 |
| claude-fable-5vertex_ai | mini-swe-agentpublished configuration | high | Mixed | 2/4 |
| claude-fable-5vertex_ai | mini-swe-agentpublished configuration | xhigh | Mixed | 3/4 |
| claude-fable-5vertex_ai | mini-swe-agentpublished configuration | max | All passed | 4/4 |
| claude-opus-4-8anthropic | mini-swe-agentpublished configuration | low | No passes | 0/4 |
| claude-opus-4-8anthropic | mini-swe-agentpublished configuration | medium | Mixed | 2/4 |
| claude-opus-4-8anthropic | mini-swe-agentpublished configuration | high | No passes | 0/4 |
| claude-opus-4-8anthropic | mini-swe-agentpublished configuration | xhigh | Mixed | 2/4 |
| claude-opus-4-8anthropic | mini-swe-agentpublished configuration | max | No passes | 0/4 |
| claude-sonnet-4-6anthropic | mini-swe-agentpublished configuration | high | No passes | 0/4 |
| claude-sonnet-5anthropic | mini-swe-agentpublished configuration | low | No passes | 0/4 |
| claude-sonnet-5anthropic | mini-swe-agentpublished configuration | medium | No passes | 0/4 |
| claude-sonnet-5anthropic | mini-swe-agentpublished configuration | high | Mixed | 1/4 |
| claude-sonnet-5anthropic | mini-swe-agentpublished configuration | xhigh | Mixed | 1/4 |
| claude-sonnet-5anthropic | mini-swe-agentpublished configuration | max | Mixed | 1/4 |
| gemini-3-1-pro-previewvertex_ai | mini-swe-agentpublished configuration | high | No passes | 0/4 |
| gemini-3-5-flashvertex_ai | mini-swe-agentpublished configuration | medium | Mixed | 1/4 |
| glm-5-2zai | mini-swe-agentpublished configuration | high | No passes | 0/4 |
| glm-5-2zai | mini-swe-agentpublished configuration | max | No passes | 0/4 |
| gpt-5-4openai | mini-swe-agentpublished configuration | xhigh | No passes | 0/4 |
| gpt-5-5openai | mini-swe-agentpublished configuration | low | No passes | 0/4 |
| gpt-5-5openai | mini-swe-agentpublished configuration | medium | No passes | 0/4 |
| gpt-5-5openai | mini-swe-agentpublished configuration | high | No passes | 0/4 |
| gpt-5-5openai | mini-swe-agentpublished configuration | xhigh | Mixed | 1/4 |
| gpt-5-6-lunaopenai | mini-swe-agentpublished configuration | low | No passes | 0/4 |
| gpt-5-6-lunaopenai | mini-swe-agentpublished configuration | medium | No passes | 0/4 |
| gpt-5-6-lunaopenai | mini-swe-agentpublished configuration | high | No passes | 0/4 |
| gpt-5-6-lunaopenai | mini-swe-agentpublished configuration | xhigh | No passes | 0/4 |
| gpt-5-6-lunaopenai | mini-swe-agentpublished configuration | max | Mixed | 2/4 |
| gpt-5-6-solopenai | mini-swe-agentpublished configuration | low | No passes | 0/4 |
| gpt-5-6-solopenai | mini-swe-agentpublished configuration | medium | No passes | 0/4 |
| gpt-5-6-solopenai | mini-swe-agentpublished configuration | high | No passes | 0/4 |
| gpt-5-6-solopenai | mini-swe-agentpublished configuration | xhigh | No passes | 0/4 |
| gpt-5-6-solopenai | mini-swe-agentpublished configuration | max | No passes | 0/4 |
| gpt-5-6-terraopenai | mini-swe-agentpublished configuration | low | No passes | 0/4 |
| gpt-5-6-terraopenai | mini-swe-agentpublished configuration | medium | No passes | 0/4 |
| gpt-5-6-terraopenai | mini-swe-agentpublished configuration | high | No passes | 0/4 |
| gpt-5-6-terraopenai | mini-swe-agentpublished configuration | xhigh | No passes | 0/4 |
| gpt-5-6-terraopenai | mini-swe-agentpublished configuration | max | No passes | 0/4 |
| kimi-k2-7-codeopenrouter | mini-swe-agentpublished configuration | default | No passes | 0/4 |