All evidence cases
DeepSWE v1.1repositoryPython

Persist the fitted feature schema across evaluate, predict, serve, and export

Persist and reuse the fitted feature schema so evaluation, prediction, serving, and export all apply the same canonicalized inputs.

igel-persist-feature-schema

Normalized profile

What this task asks for

Persist and reuse the fitted feature schema so evaluation, prediction, serving, and export all apply the same canonicalized inputs. When fit runs with dataset.features configured, the selected raw feature schema is not persisted. After fit, write feature_schema.joblib in the results directory and record feature_schema_path, input_features, dropped_features, and duplicate_feature_aliases in description.json. These rules must hold for single-target, multi-target, and clustering models. Extra raw columns must be ignored. Missing required selected features must raise an error naming them. If multiple duplicate sources are supplied they must agree row-wise for every row or raise an error naming the conflicting columns.

  • feature implementation
  • configuration tuning
  • test and validation
  • validation layer
  • igel
  • JSON
  • HTTP
  • repository patch
  • consistent behavior across lifecycle stages or execution modes

Published evidence

Model × harness outcomes

Counts are tied to these exact configurations. Harnesses are not treated as equivalent across benchmarks.

Models
13
Configurations
41
Trials
164

Showing 41 of 41 configurations

ModelHarnessEffortResultPasses
claude-fable-5vertex_aimini-swe-agentpublished configurationlowMixed2/4
claude-fable-5vertex_aimini-swe-agentpublished configurationmediumMixed3/4
claude-fable-5vertex_aimini-swe-agentpublished configurationhighMixed3/4
claude-fable-5vertex_aimini-swe-agentpublished configurationxhighAll passed4/4
claude-fable-5vertex_aimini-swe-agentpublished configurationmaxMixed3/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationlowMixed1/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationmediumMixed3/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationhighAll passed4/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationxhighMixed1/4
claude-opus-4-8anthropicmini-swe-agentpublished configurationmaxAll passed4/4
claude-sonnet-4-6anthropicmini-swe-agentpublished configurationhighNo passes0/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationlowAll passed4/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationmediumAll passed4/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationhighAll passed4/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationxhighMixed3/4
claude-sonnet-5anthropicmini-swe-agentpublished configurationmaxAll passed4/4
gemini-3-1-pro-previewvertex_aimini-swe-agentpublished configurationhighNo passes0/4
gemini-3-5-flashvertex_aimini-swe-agentpublished configurationmediumMixed3/4
glm-5-2zaimini-swe-agentpublished configurationhighAll passed4/4
glm-5-2zaimini-swe-agentpublished configurationmaxMixed3/4
gpt-5-4openaimini-swe-agentpublished configurationxhighMixed2/4
gpt-5-5openaimini-swe-agentpublished configurationlowMixed1/4
gpt-5-5openaimini-swe-agentpublished configurationmediumNo passes0/4
gpt-5-5openaimini-swe-agentpublished configurationhighMixed2/4
gpt-5-5openaimini-swe-agentpublished configurationxhighNo passes0/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationlowNo passes0/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationmediumMixed1/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationhighMixed1/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationxhighNo passes0/4
gpt-5-6-lunaopenaimini-swe-agentpublished configurationmaxMixed1/4
gpt-5-6-solopenaimini-swe-agentpublished configurationlowMixed1/4
gpt-5-6-solopenaimini-swe-agentpublished configurationmediumMixed3/4
gpt-5-6-solopenaimini-swe-agentpublished configurationhighMixed2/4
gpt-5-6-solopenaimini-swe-agentpublished configurationxhighMixed2/4
gpt-5-6-solopenaimini-swe-agentpublished configurationmaxNo passes0/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationlowMixed2/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationmediumMixed2/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationhighMixed3/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationxhighNo passes0/4
gpt-5-6-terraopenaimini-swe-agentpublished configurationmaxNo passes0/4
kimi-k2-7-codeopenroutermini-swe-agentpublished configurationdefaultNo passes0/4