All evidence cases
DeepSWE v1.1repositoryTypeScript

Add scoped per-rule ignore markers to Obsidian Linter

Implement standalone comment markers that disable and re-enable specific lint rules with nested scope handling.

obsidian-linter-scoped-ignore-markers

Normalized profile

What this task asks for

Implement standalone comment markers that disable and re-enable specific lint rules with nested scope handling. The linter must recognize both HTML comment markers and Obsidian comment markers. Markers that occur inside YAML frontmatter, fenced or indented code blocks, inline code, or math blocks must be ignored. Marker lines must never be modified by any rule, regardless of whether the marker disables that rule. `N` must be a positive base-10 integer, otherwise the marker has no effect. Rule lists must be normalized case-insensitively, with duplicates removed, and trailing commas / empty entries ignored. Disabling all rules and re-enabling specific rules within that scope must be supported.

  • feature implementation
  • test and validation
  • library API
  • Obsidian Linter
  • HTML
  • YAML
  • repository patch
  • quantitative performance, accuracy, or resource constraints

Published evidence

Model × harness outcomes

Counts are tied to these exact configurations. Harnesses are not treated as equivalent across benchmarks.

Models
13
Configurations
41
Trials
164

Showing 41 of 41 configurations

ModelHarnessEffortResultPasses
claude-fable-5vertex_aimini-swe-agentpublished configurationlowAll passed4/4
claude-fable-5vertex_aimini-swe-agentpublished configurationmediumAll passed4/4
claude-fable-5vertex_aimini-swe-agentpublished configurationhighAll passed4/4
claude-fable-5vertex_aimini-swe-agentpublished configurationxhighAll passed4/4
claude-fable-5vertex_aimini-swe-agentpublished configurationmaxAll passed4/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationlowAll passed4/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationmediumAll passed4/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationhighAll passed4/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationxhighAll passed4/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationmaxAll passed4/4
claude-sonnet-4-6anthropicmini-swe-agentpublished configurationhighNo passes0/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationlowMixed3/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationmediumMixed2/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationhighMixed3/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationxhighMixed3/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationmaxMixed2/4
gemini-3-1-pro-previewvertex_aimini-swe-agentpublished configurationhighMixed1/4
gemini-3-5-flashvertex_aimini-swe-agentpublished configurationmediumAll passed4/4
glm-5-2zaimini-swe-agentpublished configurationhighMixed2/4
glm-5-2zaimini-swe-agentpublished configurationmaxMixed2/4
gpt-5-4openaimini-swe-agentpublished configurationxhighMixed2/4
gpt-5-5openaimini-swe-agentpublished configurationlowMixed1/4
gpt-5-5openaimini-swe-agentpublished configurationmediumMixed2/4
gpt-5-5openaimini-swe-agentpublished configurationhighMixed3/4
gpt-5-5openaimini-swe-agentpublished configurationxhighMixed3/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationlowNo passes0/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationmediumNo passes0/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationhighMixed1/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationxhighMixed2/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationmaxNo passes0/4
gpt-5-6-solopenaimini-swe-agentpublished configurationlowMixed2/4
gpt-5-6-solopenaimini-swe-agentpublished configurationmediumMixed3/4
gpt-5-6-solopenaimini-swe-agentpublished configurationhighAll passed4/4
gpt-5-6-solopenaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-6-solopenaimini-swe-agentpublished configurationmaxAll passed4/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationlowNo passes0/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationmediumMixed1/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationhighMixed3/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationmaxMixed2/4
kimi-k2-7-codeopenroutermini-swe-agentpublished configurationdefaultMixed1/4