Partition report files by launcher and expand report templates
Split report output into per-launcher files with template expansion, launcher-safe filenames, and per-launcher summaries.
testem-per-launcher-reports
Normalized profile
What this task asks for
Split report output into per-launcher files with template expansion, launcher-safe filenames, and per-launcher summaries. Report_file must support <launcher>, <date>, <timestamp> template variables. When <launcher> is present, Reporter must create separate files and route each browser's results to its own file. Finish() must be idempotent. Launcher names in filenames must be filesystem-safe (each /\:*?"<>|() becomes one underscore, and consecutive whitespace becomes one underscore). The internal "testem" launcher must not produce a file. TAP reporter must optionally show per-launcher pass/fail/skip counts.
Intent
- feature implementation
- configuration tuning
- data transformation
- test and validation
Work surfaces
- serialization
- configuration
- filesystem
- test runner
Technology
- Testem
- XML
- xUnit
- TAP
Expected artifacts
- repository patch
What makes it difficult
- changes span multiple components or interfaces
Published evidence
Model × harness outcomes
Counts are tied to these exact configurations. Harnesses are not treated as equivalent across benchmarks.
- Models
- 13
- Configurations
- 41
- Trials
- 164
Showing 41 of 41 configurations
| Model | Harness | Effort | Result | Passes |
|---|---|---|---|---|
| claude-fable-5vertex_ai | mini-swe-agentpublished configuration | low | All passed | 4/4 |
| claude-fable-5vertex_ai | mini-swe-agentpublished configuration | medium | All passed | 4/4 |
| claude-fable-5vertex_ai | mini-swe-agentpublished configuration | high | Mixed | 3/4 |
| claude-fable-5vertex_ai | mini-swe-agentpublished configuration | xhigh | All passed | 4/4 |
| claude-fable-5vertex_ai | mini-swe-agentpublished configuration | max | All passed | 4/4 |
| claude-opus-4-8anthropic | mini-swe-agentpublished configuration | low | All passed | 4/4 |
| claude-opus-4-8anthropic | mini-swe-agentpublished configuration | medium | All passed | 4/4 |
| claude-opus-4-8anthropic | mini-swe-agentpublished configuration | high | All passed | 4/4 |
| claude-opus-4-8anthropic | mini-swe-agentpublished configuration | xhigh | All passed | 4/4 |
| claude-opus-4-8anthropic | mini-swe-agentpublished configuration | max | All passed | 4/4 |
| claude-sonnet-4-6anthropic | mini-swe-agentpublished configuration | high | All passed | 4/4 |
| claude-sonnet-5anthropic | mini-swe-agentpublished configuration | low | No passes | 0/4 |
| claude-sonnet-5anthropic | mini-swe-agentpublished configuration | medium | Mixed | 2/4 |
| claude-sonnet-5anthropic | mini-swe-agentpublished configuration | high | No passes | 0/4 |
| claude-sonnet-5anthropic | mini-swe-agentpublished configuration | xhigh | Mixed | 2/4 |
| claude-sonnet-5anthropic | mini-swe-agentpublished configuration | max | Mixed | 2/4 |
| gemini-3-1-pro-previewvertex_ai | mini-swe-agentpublished configuration | high | No passes | 0/4 |
| gemini-3-5-flashvertex_ai | mini-swe-agentpublished configuration | medium | All passed | 4/4 |
| glm-5-2zai | mini-swe-agentpublished configuration | high | Mixed | 2/4 |
| glm-5-2zai | mini-swe-agentpublished configuration | max | Mixed | 3/4 |
| gpt-5-4openai | mini-swe-agentpublished configuration | xhigh | All passed | 4/4 |
| gpt-5-5openai | mini-swe-agentpublished configuration | low | No passes | 0/4 |
| gpt-5-5openai | mini-swe-agentpublished configuration | medium | Mixed | 2/4 |
| gpt-5-5openai | mini-swe-agentpublished configuration | high | All passed | 4/4 |
| gpt-5-5openai | mini-swe-agentpublished configuration | xhigh | All passed | 4/4 |
| gpt-5-6-lunaopenai | mini-swe-agentpublished configuration | low | No passes | 0/4 |
| gpt-5-6-lunaopenai | mini-swe-agentpublished configuration | medium | No passes | 0/4 |
| gpt-5-6-lunaopenai | mini-swe-agentpublished configuration | high | Mixed | 3/4 |
| gpt-5-6-lunaopenai | mini-swe-agentpublished configuration | xhigh | Mixed | 3/4 |
| gpt-5-6-lunaopenai | mini-swe-agentpublished configuration | max | Mixed | 3/4 |
| gpt-5-6-solopenai | mini-swe-agentpublished configuration | low | No passes | 0/4 |
| gpt-5-6-solopenai | mini-swe-agentpublished configuration | medium | Mixed | 3/4 |
| gpt-5-6-solopenai | mini-swe-agentpublished configuration | high | All passed | 4/4 |
| gpt-5-6-solopenai | mini-swe-agentpublished configuration | xhigh | All passed | 4/4 |
| gpt-5-6-solopenai | mini-swe-agentpublished configuration | max | All passed | 4/4 |
| gpt-5-6-terraopenai | mini-swe-agentpublished configuration | low | No passes | 0/4 |
| gpt-5-6-terraopenai | mini-swe-agentpublished configuration | medium | No passes | 0/4 |
| gpt-5-6-terraopenai | mini-swe-agentpublished configuration | high | Mixed | 2/4 |
| gpt-5-6-terraopenai | mini-swe-agentpublished configuration | xhigh | All passed | 4/4 |
| gpt-5-6-terraopenai | mini-swe-agentpublished configuration | max | All passed | 4/4 |
| kimi-k2-7-codeopenrouter | mini-swe-agentpublished configuration | default | Mixed | 1/4 |