Implement AFAB pipeline parallelism for LLaMA
Implement one LLaMA training step that balances layers across ranks and executes all microbatch forwards before all backwards with correct distributed tensors.
torch-pipeline-parallelism
Normalized profile
What this task asks for
Implement one LLaMA training step that balances layers across ranks and executes all microbatch forwards before all backwards with correct distributed tensors. Export train_step_pipeline_afab with the specified signature. Partition layers roughly evenly for world sizes one and two. Perform every forward before any backward. Communicate hidden states and gradients with the specified shapes. Compute and microbatch-scale cross-entropy on the last rank without hooks.
Intent
- feature implementation
- performance optimization
- test and validation
Work surfaces
- distributed training
- model layers
- tensor communication
Technology
- PyTorch
- torch.distributed
- LLaMA
- pipeline parallelism
Expected artifacts
- /app/pipeline_parallel.py
What makes it difficult
- concurrency, cancellation, or distributed-state coordination
- changes span multiple components or interfaces
Published evidence
Model × harness outcomes
Counts are tied to these exact configurations. Harnesses are not treated as equivalent across benchmarks.
- Models
- 13
- Configurations
- 20
- Trials
- 100
Showing 20 of 20 configurations
| Model | Harness | Effort | Result | Passes |
|---|---|---|---|---|
| claude-fable-5anthropic · 2026-06-07 | claude-code2.1.167 | xhigh | All passed | 5/5 |
| claude-fable-5anthropic · 2026-06-05 | terminus-22.0.0 | high | All passed | 5/5 |
| claude-opus-4-7anthropic · 2026-05-01 | claude-code2.1.123 | max | Mixed1 errors | 2/5 |
| claude-opus-4-7anthropic · 2026-05-01 | terminus-22.0.0 | max | No passes4 errors | 0/5 |
| claude-opus-4-8anthropic · 2026-07-09 | claude-code2.1.205 | high | All passed | 5/5 |
| claude-sonnet-5anthropic · 2026-07-09 | claude-code2.1.205 | high | Mixed | 3/5 |
| gemini-3-pro-previewgemini · 2026-05-01 | gemini-cli0.40.0 | high | No passes1 errors | 0/5 |
| gemini-3-pro-previewgemini · 2026-05-01 | terminus-22.0.0 | high | Mixed | 3/5 |
| gemini-3.1-pro-previewgemini · 2026-05-05 | gemini-cli0.40.0 | high | No passes | 0/5 |
| gemini-3.1-pro-previewgemini · 2026-05-05 | terminus-22.0.0 | high | Mixed1 errors | 1/5 |
| glm-5.1zai · 2026-05-01 | claude-code2.1.123 | max | No passes5 errors | 0/5 |
| gpt-5.5openai · 2026-05-01 | codex0.125.0 | xhigh | All passed | 5/5 |
| gpt-5.5openai · 2026-05-01 | terminus-22.0.0 | xhigh | Mixed4 errors | 1/5 |
| gpt-5.6-lunaopenai · 2026-07-10 | codex0.144.0 | max | Mixed1 disqualified | 2/5 |
| gpt-5.6-lunaopenai · 2026-07-11 | codex0.144.1 | max | All passed | 5/5 |
| gpt-5.6-solopenai · 2026-07-10 | codex0.144.0 | max | No passes5 disqualified | 0/5 |
| gpt-5.6-terraopenai · 2026-07-10 | codex0.144.0 | max | Mixed3 disqualified | 1/5 |
| gpt-5.6-terraopenai · 2026-07-11 | codex0.144.1 | max | No passes | 0/5 |
| grok-4.5cursor · 2026-07-09 | cursor-cli2026.07.08-0c04a8a | default | No passes4 disqualified | 0/5 |
| muse-spark-1.1openai · 2026-07-09 | mini-swe-agent2.4.5 | xhigh | No passes | 0/5 |