All evidence cases
DeepSWE v1.1repositoryGo

Add go:embed directive support for interpreted packages

Support //go:embed directives so interpreted package variables can receive embedded file contents and embed.FS values.

yaegi-go-embed-directives

Normalized profile

What this task asks for

Support //go:embed directives so interpreted package variables can receive embedded file contents and embed.FS values. Support `//go:embed` directives that embed file contents into package-level variables. The variable must hold its embedded content by the time the first interpreted statement executes. The interpreter's standard variable initialization must not overwrite it. `embed.FS` -- one or more files as a read-only filesystem. For `string` and `[]byte`, patterns must resolve to exactly one file. Opened directories implement `fs.ReadDirFile`.

  • feature implementation
  • parser
  • filesystem
  • runtime
  • Yaegi
  • repository patch
  • strict behavioral compatibility or exact-output validation
  • changes span multiple components or interfaces

Published evidence

Model × harness outcomes

Counts are tied to these exact configurations. Harnesses are not treated as equivalent across benchmarks.

Models
13
Configurations
41
Trials
164

Showing 41 of 41 configurations

ModelHarnessEffortResultPasses
claude-fable-5vertex_aimini-swe-agentpublished configurationlowMixed3/4
claude-fable-5vertex_aimini-swe-agentpublished configurationmediumMixed3/4
claude-fable-5vertex_aimini-swe-agentpublished configurationhighMixed2/4
claude-fable-5vertex_aimini-swe-agentpublished configurationxhighMixed2/4
claude-fable-5vertex_aimini-swe-agentpublished configurationmaxMixed1/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationlowMixed2/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationmediumMixed1/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationhighMixed2/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationxhighMixed1/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationmaxMixed1/4
claude-sonnet-4-6anthropicmini-swe-agentpublished configurationhighMixed1/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationlowMixed3/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationmediumMixed3/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationhighMixed1/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationxhighMixed3/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationmaxMixed2/4
gemini-3-1-pro-previewvertex_aimini-swe-agentpublished configurationhighMixed1/4
gemini-3-5-flashvertex_aimini-swe-agentpublished configurationmediumMixed1/4
glm-5-2zaimini-swe-agentpublished configurationhighMixed2/4
glm-5-2zaimini-swe-agentpublished configurationmaxMixed2/4
gpt-5-4openaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-5openaimini-swe-agentpublished configurationlowMixed3/4
gpt-5-5openaimini-swe-agentpublished configurationmediumMixed3/4
gpt-5-5openaimini-swe-agentpublished configurationhighAll passed4/4
gpt-5-5openaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationlowNo passes0/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationmediumMixed1/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationhighMixed2/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationxhighMixed3/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationmaxMixed3/4
gpt-5-6-solopenaimini-swe-agentpublished configurationlowMixed2/4
gpt-5-6-solopenaimini-swe-agentpublished configurationmediumMixed3/4
gpt-5-6-solopenaimini-swe-agentpublished configurationhighMixed3/4
gpt-5-6-solopenaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-6-solopenaimini-swe-agentpublished configurationmaxAll passed4/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationlowMixed1/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationmediumAll passed4/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationhighAll passed4/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationxhighAll passed4/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationmaxMixed3/4
kimi-k2-7-codeopenroutermini-swe-agentpublished configurationdefaultMixed1/4