All evidence cases
DeepSWE v1.1repositoryGo

Add action pinning linting for actions and reusable workflows

Add a configurable lint rule that enforces pinned versions for action and reusable workflow references.

actionlint-action-pinning-lint

Normalized profile

What this task asks for

Add a configurable lint rule that enforces pinned versions for action and reusable workflow references. Check action and reusable-workflow `uses:` references with the `action-pinning` lint rule. Support `major-minor`, `semver`, and `commit-sha` levels, defaulting to `semver`. Require full 40-character lowercase hexadecimal values at the `commit-sha` level. Skip local and Docker references, but report dynamic version expressions that cannot be verified. Merge allow and deny lists across matching configurations with denials taking precedence. Let per-path configuration and `-action-pinning-level` enable or override only the pinning level.

  • feature implementation
  • security remediation
  • configuration tuning
  • data transformation
  • CLI
  • serialization
  • configuration
  • actionlint
  • Docker
  • repository patch
  • layered configuration precedence and inheritance semantics
  • changes span multiple components or interfaces

Published evidence

Model × harness outcomes

Counts are tied to these exact configurations. Harnesses are not treated as equivalent across benchmarks.

Models
13
Configurations
41
Trials
164

Showing 41 of 41 configurations

ModelHarnessEffortResultPasses
claude-fable-5vertex_aimini-swe-agentpublished configurationlowMixed3/4
claude-fable-5vertex_aimini-swe-agentpublished configurationmediumAll passed4/4
claude-fable-5vertex_aimini-swe-agentpublished configurationhighMixed3/4
claude-fable-5vertex_aimini-swe-agentpublished configurationxhighMixed2/4
claude-fable-5vertex_aimini-swe-agentpublished configurationmaxAll passed4/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationlowAll passed4/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationmediumAll passed4/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationhighAll passed4/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationxhighMixed2/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationmaxAll passed4/4
claude-sonnet-4-6anthropicmini-swe-agentpublished configurationhighMixed3/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationlowMixed2/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationmediumAll passed4/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationhighMixed2/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationxhighMixed3/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationmaxMixed2/4
gemini-3-1-pro-previewvertex_aimini-swe-agentpublished configurationhighMixed1/4
gemini-3-5-flashvertex_aimini-swe-agentpublished configurationmediumAll passed4/4
glm-5-2zaimini-swe-agentpublished configurationhighMixed1/4
glm-5-2zaimini-swe-agentpublished configurationmaxMixed3/4
gpt-5-4openaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-5openaimini-swe-agentpublished configurationlowAll passed4/4
gpt-5-5openaimini-swe-agentpublished configurationmediumAll passed4/4
gpt-5-5openaimini-swe-agentpublished configurationhighMixed3/4
gpt-5-5openaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationlowNo passes0/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationmediumMixed3/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationhighAll passed4/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationmaxAll passed4/4
gpt-5-6-solopenaimini-swe-agentpublished configurationlowAll passed4/4
gpt-5-6-solopenaimini-swe-agentpublished configurationmediumAll passed4/4
gpt-5-6-solopenaimini-swe-agentpublished configurationhighMixed3/4
gpt-5-6-solopenaimini-swe-agentpublished configurationxhighMixed3/4
gpt-5-6-solopenaimini-swe-agentpublished configurationmaxMixed3/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationlowMixed3/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationmediumMixed3/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationhighAll passed4/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationxhighMixed3/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationmaxAll passed4/4
kimi-k2-7-codeopenroutermini-swe-agentpublished configurationdefaultAll passed4/4