Evidence browser
Cases, minus the archaeology.
Search compact task profiles, see what makes them difficult, and compare published model outcomes.
- Cases
- 202
- Suites
- 2
- Published trials
- 27424
202 cases
Showing 202 of 202 evidence cases.
| Task profile | Benchmark | Technology | Published passes | Open case |
|---|---|---|---|---|
| Harden module loading, cache introspection, and script flags Harden ABS module resolution and caching, expose cache introspection APIs, and make module flags work correctly in script mode. | DeepSWEv1.1 · repository | ABSGo | 113/16469% | |
| Add stepped slices for arrays and strings Add stepped slice parsing, slicing, and assignment for arrays and strings with rune-correct indexing. | DeepSWEv1.1 · repository | ABSGo | 115/16470% | |
| Add action pinning linting for actions and reusable workflows Add a configurable lint rule that enforces pinned versions for action and reusable workflow references. | DeepSWEv1.1 · repository | actionlintGo | 131/16480% | |
| Add input key aliases to name mapping Add load-only alias support to name mapping so fields can resolve from alternate input keys. | DeepSWEv1.1 · repository | AdaptixPython | 101/16462% | |
| Add task snapshots, inspection, and diffing to aiomonitor Add point-in-time task snapshots with interactive inspection, diffing, deletion, and web/CLI access. | DeepSWEv1.1 · repository | aiomonitorPython | 103/16363% | |
| Add default arguments to Anko function parameters Add parsing and call-time evaluation of default parameter values in Anko functions. | DeepSWEv1.1 · repository | AnkoGo | 113/16469% | |
| Add typed variable bindings to Anko Add typed `var` declarations in Anko and enforce declared type constraints on assignment when TypedBindings is enabled. | DeepSWEv1.1 · repository | AnkoGo | 67/16441% | |
| Add drift detection and compliance baselines Implement baseline capture, drift comparison, and compliance tracking for container configurations. | DeepSWEv1.1 · repository | ArcaneGo | 106/16465% | |
| Add JSON Schema refs and dependency keywords Add JSON Schema dependency keywords, local $defs/$ref resolution, and conditional schema handling. | DeepSWEv1.1 · repository | ArkTypeTypeScript | 60/16437% | |
| Add dependency-aware async initialization to the container Add dependency-aware asynchronous container initialization with ordered startup, concurrency limits, and rollback on failure. | DeepSWEv1.1 · repository | AwilixTypeScript | 57/16435% | |
| Add incremental cache controls to Bandit Add incremental analysis caching with cache invalidation, import/export, pruning, and cache inspection CLI options. | DeepSWEv1.1 · repository | BanditPython | 83/16451% | |
| Add interprocedural taint checks for Bandit injection sinks Track user input through variables and calls so Bandit flags injection sinks beyond string literals. | DeepSWEv1.1 · repository | BanditPython | 86/16452% | |
| Add structured nosec directives for regions and next line Add region and next-line nosec directives with selector expressions and ignore-nosec handling. | DeepSWEv1.1 · repository | BanditPython | 8/1645% | |
| Add hierarchical evaluation cancellation to Boa Add cancellable evaluation handles that propagate through nested script, module, and job execution. | DeepSWEv1.1 · repository | BoaRust | 70/16343% | |
| Add partial structuring with error recovery to cattrs Add `partial_structure` and `PartialResult` for recoverable, field-level structuring with nested partial results. | DeepSWEv1.1 · repository | cattrsPython | 97/16459% | |
| Add async autocomplete options and fetch lifecycle handling Add async option fetching with caching, retries, debouncing, and loading state to AutocompletePrompt. | DeepSWEv1.1 · repository | ClackTypeScript | 44/16427% | |
| Implement recursive agent delegation through delegate_task tool calls Run delegated sub-agents, feed their results back as tool_result, and handle unknown agents, failures, and circular delegation. | DeepSWEv1.1 · repository | Claude Code by AgentsTypeScript | 89/16355% | |
| Add config file parsing to Cliffy commands Add command-level config file loading, parsing, merging, and precedence handling. | DeepSWEv1.1 · repository | CliffyTypeScript | 75/16446% | |
| Add shorthand expansion and compression to the lexer Add lexer methods to expand CSS shorthands into longhands and compress longhands back into shorthand values. | DeepSWEv1.1 · repository | CSSTreeJavaScript | 48/16429% | |
| Add HTML document format handling to Dasel Add read and write support for HTML documents with Dasel's format handling. | DeepSWEv1.1 · repository | DaselGo | 71/16443% | |
| Add RFC 5545 timezone interoperability to dateutil recurrence parsing Extend rrule and rruleset to serialize, parse, and compare RFC 5545 timezone-aware recurrence data. | DeepSWEv1.1 · repository | python-dateutilPython | 59/16436% | |
| Add typed window function builders with OVER clauses Add typed query-builder helpers for SQL window functions, named windows, and frame specs. | DeepSWEv1.1 · repository | Drizzle ORMTypeScript | 139/16485% | |
| Add conditional required attributes to schemas Add `requiredIf`-based conditional attribute enforcement across schema validation, parsing, updates, and JSON Schema export. | DeepSWEv1.1 · repository | DynamoDB ToolboxTypeScript | 112/16369% | |
| Add lazy recursive schemas with DTO and JSON Schema export Add a lazy schema type for self-referencing recursive data with full serialization, validation, and export support. | DeepSWEv1.1 · repository | DynamoDB ToolboxTypeScript | 103/16064% | |
| Add SSE streaming endpoints to HttpApi Add typed Server-Sent Events streaming endpoints, encoders/decoders, and client handling to HttpApi. | DeepSWEv1.1 · repository | EffectTypeScript | 25/16415% | |
| Add keyset cursor pagination to `$find` Add cursor-based keyset pagination to `$find` with nextCursor handling and validation. | DeepSWEv1.1 · repository | EICRUDTypeScript | 41/16425% | |
| Add XML diff, patch, and merge operations to etree Add recursive XML diffing, patch generation and application, reverse patching, three-way merge, and diff summaries. | DeepSWEv1.1 · repository | etreeGo | 93/16457% | |
| Add try/catch error recovery to expr Add expression and block-level error recovery with try, catch, finally, throw, retry, and errtype. | DeepSWEv1.1 · repository | ExprGo | 42/16426% | |
| Add deprecation, sunset, and successor headers to FastAPI routes Add runtime Deprecation, Sunset, and Link headers plus OpenAPI metadata and tracking middleware for deprecated routes. | DeepSWEv1.1 · repository | FastAPIPython | 71/16443% | |
| Add implicit HEAD and automatic OPTIONS responses to FastAPI routes Add configurable implicit HEAD handling and automatic OPTIONS responses for FastAPI routes, routers, and included routers. | DeepSWEv1.1 · repository | FastAPIPython | 80/16449% | |
| Add deterministic multi-key sorting to fd Add repeatable multi-key sorting controls to fd output with deterministic tie-breaking and seeded random order. | DeepSWEv1.1 · repository | fdRust | 96/16459% | |
| Add ShapeIndex encoding and decoding Add stable ShapeIndex Encode/Decode support so indices round-trip without rebuilding. | DeepSWEv1.1 · repository | GeoGo | 116/16471% | |
| Add a checker for broken doc comment links Add a diagnostic checker that validates doc comment symbol links against package and type information. | DeepSWEv1.1 · repository | go-criticGo | 82/16450% | |
| Expose accumulated streamed function-call args in SDK surfaces Expose fully accumulated streamed function-call arguments across streaming, live sessions, and chat history. | DeepSWEv1.1 · repository | Go GenAIGo | 111/16468% | |
| Add worktree merge conflict handling Add worktree merge support with conflict detection, merge head handling, and index stage updates. | DeepSWEv1.1 · repository | go-gitGo | 98/16460% | |
| Add retry-aware publishing audit logs Implement per-artifact retries for uploads, artifactory, and blobs while recording deterministic publish attempt history. | DeepSWEv1.1 · repository | GoReleaserGo | 128/16478% | |
| Add GraphQL incremental delivery with @defer and @stream Add incremental GraphQL response handling with @defer and @stream across HTTP multipart, WebSocket transport, and the DSL. | DeepSWEv1.1 · repository | gqlPython | 6/1644% | |
| Abort pending body reads on shutdown Ensure interrupted request and response body reads, formData parsing, and discarded timers abort cleanly during shutdown. | DeepSWEv1.1 · repository | Happy DOMTypeScript | 150/16491% | |
| Implement a deterministic IntersectionObserver in Happy DOM Implement a real, deterministic IntersectionObserver with async delivery, thresholds, root margins, and target tracking. | DeepSWEv1.1 · repository | Happy DOMTypeScript | 50/16430% | |
| Add configurable array merge strategies to Helm value coalescing Add chart-scoped append and key-based merge strategies for coalescing arrays during Helm value merging. | DeepSWEv1.1 · repository | HelmGo | 26/16416% | |
| Add unified manifest stream output across Helm commands Add a stable unified manifest stream for template, dry-run install/upgrade, and get manifest output. | DeepSWEv1.1 · repository | HelmGo | 125/16476% | |
| Add a deterministic CookieStore with modern Set-Cookie parsing Add a deterministic CookieStore with modern Set-Cookie parsing, cookie eviction, and request header handling. | DeepSWEv1.1 · repository | HTTPXTypeScript | 132/16480% | |
| Add multipart response parsing to HTTPX Add Response iterators that parse multipart HTTP response bodies into parts. | DeepSWEv1.1 · repository | HTTPXPython | 107/16465% | |
| Add streaming JSON iteration to HTTPX responses Add response iterators that incrementally parse JSON values from supported streaming media types. | DeepSWEv1.1 · repository | HTTPXPython | 63/16438% | |
| Persist the fitted feature schema across evaluate, predict, serve, and export Persist and reuse the fitted feature schema so evaluation, prediction, serving, and export all apply the same canonicalized inputs. | DeepSWEv1.1 · repository | igelPython | 80/16449% | |
| Add CSS Grid layout to the Box component Add CSS Grid layout parsing and placement to the Box component, including track sizing, gaps, and explicit child positioning. | DeepSWEv1.1 · repository | InkTypeScript | 49/16430% | |
| Add session bundle recording and replay to IPython Add `.ipybundle` recording, validation, and replay APIs for IPython sessions. | DeepSWEv1.1 · repository | IPythonPython | 93/16457% | |
| Add `\multicolumn` column spans to array-like environments Add `\multicolumn` parsing and rendering for array-like environments with span-aware alignment and errors. | DeepSWEv1.1 · repository | KaTeXJavaScript | 40/16424% | |
| Add multiplexed ordered streams over KCP Add a multiplexing layer that carries many ordered sub-streams over one KCP connection with flow control, priority scheduling, and SNMP counters. | DeepSWEv1.1 · repository | kcp-goGo | 115/16470% | |
| Add atomic signal selectors to Kea Introduce fine-grained atomic selector tracking with dependency health, circular detection, and unchanged Kea lifecycle behavior. | DeepSWEv1.1 · repository | KeaTypeScript | 52/16432% | |
| Add consistent hash policy support to TrafficPolicy Add spec.consistentHash to TrafficPolicy and merge it into route hash_policy generation. | DeepSWEv1.1 · repository | kgatewayGo | 60/16437% | |
| Add single-active-consumer priority and cancel tracking to virtual transports Add single-active-consumer semantics, consumer priority selection, cancel notifications, and lifecycle tracking to virtual transports. | DeepSWEv1.1 · repository | KombuPython | 85/16452% | |
| Add dead-lettering, TTL, and overflow handling to virtual queues Add dead-letter routing, TTL expiry, and max-length overflow handling to the virtual transport layer. | DeepSWEv1.1 · repository | KombuPython | 29/16418% | |
| Add composite trait aspects to Koota Add createAspect and aspect-aware query operations that merge constituent traits and propagate writes and lifecycle events. | DeepSWEv1.1 · repository | KootaTypeScript | 50/16430% | |
| Add a deferred mutation buffer to batch entity changes Add a deferred command buffer that batches entity mutations during query iteration and flushes them at defined boundaries. | DeepSWEv1.1 · repository | KootaTypeScript | 41/16425% | |
| Add entity snapshot and rollback APIs to Koota Add snapshot, rollback, and diff APIs for entities and worlds, including registry-based trait and relation serialization. | DeepSWEv1.1 · repository | KootaPython | 126/16477% | |
| Add pair-level relation tracking modifiers Tracking modifiers should distinguish changes to specific relation pairs, not just trait-level additions and removals. | DeepSWEv1.1 · repository | KootaTypeScript | 29/16418% | |
| Add value-based query predicates to Koota Add composable value-based entity predicates with dependency tracking and change transitions. | DeepSWEv1.1 · repository | KootaTypeScript | 30/16418% | |
| Add grouping-set and window-frame SQL helpers Add grouped aggregation clauses, window frame builders, null-handling modifiers, and related SQL function helpers. | DeepSWEv1.1 · repository | KyselyTypeScript | 105/16464% | |
| Add request coalescing to `Runnable` Add `Runnable.with_coalesce()` so concurrent identical inputs share one execution across sync, async, streaming, and batch APIs. | DeepSWEv1.1 · repository | LangChainPython | 58/16435% | |
| Add flattened dataclass fields to Mashumaro field options Add field_options support for flattening nested dataclass fields into parent dictionaries with prefix and rename validation. | DeepSWEv1.1 · repository | MashumaroPython | 77/16447% | |
| Add explicit resource management declarations to the parser Add parsing and AST support for `using` and `await using` declarations, including context-sensitive errors. | DeepSWEv1.1 · repository | MeriyahTypeScript | 40/16424% | |
| Validate daemon watch, status, and log lifecycle Add daemon watch validation, state tracking, logging, and lifecycle commands with non-blocking run-once processing. | DeepSWEv1.1 · repository | mnamerPython | 85/16452% | |
| Add grouped test phases with synchronized barriers Add grouped execution phases with explicit synchronization barriers and per-group setup and teardown. | DeepSWEv1.1 · repository | MoblyPython | 94/16457% | |
| Add rolling min, max, median, and quantile methods Add the remaining rolling window methods to Expr and Series with consistent validation and backend delegation. | DeepSWEv1.1 · repository | NarwhalsPython | 143/16487% | |
| Add boundary modes to `@stencil` Add configurable boundary handling modes to `@stencil` for out-of-bounds accesses. | DeepSWEv1.1 · repository | NumbaPython | 97/16459% | |
| Add automatic table of contents generation for Obsidian linter Implement an opt-in rule that generates and updates a TOC from document headings. | DeepSWEv1.1 · repository | Obsidian LinterTypeScript | 1/1641% | |
| Add link format conversion between wiki and markdown syntax Add a Link Style rule that converts between Obsidian wiki links or embeds and markdown links or images while preserving ignored regions and edge cases. | DeepSWEv1.1 · repository | Obsidian LinterTypeScript | 48/16429% | |
| Add scoped per-rule ignore markers to Obsidian Linter Implement standalone comment markers that disable and re-enable specific lint rules with nested scope handling. | DeepSWEv1.1 · repository | Obsidian LinterTypeScript | 104/16463% | |
| Add a per-origin circuit breaker to ofetch Add an opt-in per-origin circuit breaker for fetch requests with half-open probing and shared state across clients. | DeepSWEv1.1 · repository | ofetchTypeScript | 125/16476% | |
| Add transparent encryption to dump uploads Encrypt database dumps before upload and add keyed config, loading, and decryptable streaming support. | DeepSWEv1.1 · repository | onedumpGo | 44/16427% | |
| Add rule evaluation profiling to Rego Add opt-in per-rule evaluation profiling to Rego results with profiling stats and diff helpers. | DeepSWEv1.1 · repository | Open Policy AgentGo | 97/16459% | |
| Reconstruct template strings in partial evaluation output Rebuild lowered template strings into normal Rego syntax in partial evaluation outputs and support modules. | DeepSWEv1.1 · repository | Open Policy AgentGo | 90/16455% | |
| Add conditional option dependencies to Optique Add conditional dependencies so options can require or hide based on other option presence or values. | DeepSWEv1.1 · repository | OptiqueTypeScript | 44/16427% | |
| Preserve structure needed by stylesheet selectors Keep rewrite optimizations from breaking structure-dependent stylesheet selector matching. | DeepSWEv1.1 · repository | oxvgRust | 17/16310% | |
| Add build-time grammar conflict analysis to participle Add build-time static analysis that detects ambiguous participle grammars and reports conflicts. | DeepSWEv1.1 · repository | ParticipleGo | 35/16421% | |
| Add durability callbacks and wait APIs for sync writes Add batch durability callbacks, durability wait methods, notifications, and metrics for sync write commits. | DeepSWEv1.1 · repository | PebbleGo | 90/16455% | |
| Coalesce qualifying choices into character classes Add optimizer passes that collapse qualifying choice chains into merged character and negated character classes. | DeepSWEv1.1 · repository | PestRust | 21/16413% | |
| Add transactional reload status and rollback tracking to Prometheus Add an opt-in transactional config reload mode with durable reload outcomes, rollback tracking, and an HTTP status endpoint. | DeepSWEv1.1 · repository | PrometheusTypeScript | 31/16419% | |
| Fix PromQL label sorting across typed and untyped values PromQL label sorting must order mixed typed and untyped label values with stable typed comparison rules. | DeepSWEv1.1 · repository | PrometheusGo | 76/16446% | |
| Add typed blend range access and blend-if compositing Add typed blend range objects, persist them on layers, and apply blend-if during compositing. | DeepSWEv1.1 · repository | psd-toolsPython | 131/16480% | |
| Add tube multiplexing to pwntools Add a TubeMultiplexer and MuxChannel API for multiple bidirectional logical channels over one tube, with flow control and close propagation. | DeepSWEv1.1 · repository | pwntoolsPython | 52/16332% | |
| Add scoped state data to state machine callbacks and history Add per-state scoped data ownership with lifecycle resets, callback injection, history restoration, and validation. | DeepSWEv1.1 · repository | python-statemachinePython | 45/16427% | |
| Preserve restored query state in persisted snapshots Preserve full persisted query state, including errors, counters, timestamps, and infinite pagination, during restoration and cache rebuilds. | DeepSWEv1.1 · repository | TanStack QueryTypeScript | 113/16469% | |
| Reuse one toolbar across multiple Quill editors Allow multiple Quill editors to share one toolbar container while routing toolbar actions and UI state to the currently active editor. | DeepSWEv1.1 · repository | QuillTypeScript | 42/16426% | |
| Add an error-accumulating Validated container Add a Validated container that accumulates errors while preserving standard container APIs. | DeepSWEv1.1 · repository | ReturnsPython | 131/16480% | |
| Add bounded-memory spilling to SCC aggregation Add an opt-in bounded-memory mode that spills per-file aggregation results to disk without changing output. | DeepSWEv1.1 · repository | sccGo | 97/16459% | |
| Add method declarations and interface dispatch to Scriggo Implement method declarations with receiver handling, method expressions, and interface satisfaction for Scriggo-defined types. | DeepSWEv1.1 · repository | ScriggoGo | 88/16454% | |
| Add duration encoding to TableVectorizer Add a DurationEncoder, duration selector, and duration routing in TableVectorizer. | DeepSWEv1.1 · repository | skrubPython | 87/16453% | |
| Format BigQuery pipe syntax queries correctly Add parsing and formatting support for BigQuery pipe syntax queries without changing traditional SQL formatting. | DeepSWEv1.1 · repository | SQL FormatterTypeScript | 131/16480% | |
| Format CREATE TABLE DDL and add DDL parsing helpers Format CREATE TABLE statements with DDL-aware line breaking and add parsing models for table columns and constraints. | DeepSWEv1.1 · repository | sqlfmtPython | 36/16422% | |
| Add safe import checkpoints and invariant validation Add safe bulk import checkpoints, invariant validation, and rollback-on-failure behavior. | DeepSWEv1.1 · repository | sqlite-utilsPython | 79/16448% | |
| Add error stack serialization to SuperJSON Add configurable serialization and restoration of error stacks, stack frames, causes, and sanitization in SuperJSON. | DeepSWEv1.1 · repository | SuperJSONTypeScript | 55/16434% | |
| Add task graph export with JSON, DOT, and text output Add a graph export command for tasks with JSON, DOT, and text output, including reverse traversal and status control. | DeepSWEv1.1 · repository | TaskGo | 116/16471% | |
| Fix isolated Go-side calls for Tengo callables and closures Enable Go-side invocation of exported Tengo functions and closures while preserving runtime context and isolating compiled-instance state. | DeepSWEv1.1 · repository | TengoGo | 97/16459% | |
| Add destructuring bindings to Tengo Add destructuring bindings for `:=` in arrays, maps, and function parameters. | DeepSWEv1.1 · repository | TengoGo | 79/16448% | |
| Preserve ANSI resets during truncation and styling Add ANSI tokenization, reset-preserving truncation, and style-aware output helpers. | DeepSWEv1.1 · repository | termenvGo | 27/16416% | |
| Add bail-on-test-failure handling to Testem Add configurable early bailout on test failures across runners, reporters, and exit codes. | DeepSWEv1.1 · repository | TestemJavaScript | 22/16413% | |
| Partition report files by launcher and expand report templates Split report output into per-launcher files with template expansion, launcher-safe filenames, and per-launcher summaries. | DeepSWEv1.1 · repository | TestemJavaScript | 107/16465% | |
| Complete Kitty keyboard phases and stable fallback key metadata Add full Kitty keyboard phase handling and stable fallback key metadata for keys and shortcuts. | DeepSWEv1.1 · repository | TextualPython | 46/16428% | |
| Restore RichLog follow-state parity and expand reflow behavior Restore RichLog follow-mode parity with Log and preserve expand reflow behavior for justified writes. | DeepSWEv1.1 · repository | TextualPython | 106/16465% | |
| Add bidirectional TOML table converters Add in-place conversion helpers between standard tables, inline tables, dotted keys, and super tables while preserving comments and round-trip integrity. | DeepSWEv1.1 · repository | TOMLKitPython | 95/16458% | |
| Add iterable collection combinators to true-myth Add iterable-aware sequence, traverse, zip, filtering, and task combinators across Maybe, Result, Task, and toolbelt APIs. | DeepSWEv1.1 · repository | True MythTypeScript | 141/16486% | |
| Add `matchEach` to ts-pattern Add a new `matchEach` matcher that evaluates all matching clauses and returns all results in order. | DeepSWEv1.1 · repository | ts-patternTypeScript | 117/16471% | |
| Add policy-based alerting for failures, latency, and SSL expiry Add policy-driven alerting with consecutive failure, latency, recovery, and SSL expiry notifications. | DeepSWEv1.1 · repository | UpdoGo | 19/16412% | |
| Add recursive schema composition to Valibot Add first-class recursive schema composition with `Recur`, `recursive(...)`, and `recursiveAsync(...)` wrappers. | DeepSWEv1.1 · repository | ValibotTypeScript | 104/16463% | |
| Add duration-aware sharding to Vitest Add duration-aware sharding strategies and duration history handling for Vitest. | DeepSWEv1.1 · repository | VitestTypeScript | 120/16473% | |
| Add a persistent analysis cache to Vulture Add a persistent cache so Vulture can reuse unchanged analysis across runs. | DeepSWEv1.1 · repository | VulturePython | 53/16432% | |
| Add trap coredump generation to wasmi Generate opt-in Wasm coredumps on traps and attach the bytes to errors. | DeepSWEv1.1 · repository | wasmiRust | 100/16461% | |
| Add multi-module memory snapshots to wazero Add coordinated capture, restore, diff, and serialization for multi-module memory snapshots. | DeepSWEv1.1 · repository | wazeroGo | 122/16474% | |
| Add go:embed directive support for interpreted packages Support //go:embed directives so interpreted package variables can receive embedded file contents and embed.FS values. | DeepSWEv1.1 · repository | YaegiGo | 97/16459% | |
| Add deterministic map conflict detection to Y.Map writes Add strict, deterministic conflict detection for Y.Map key writes with collect and error policies. | DeepSWEv1.1 · repository | YjsJavaScript | 120/16473% | |
| Add JSONPath query APIs to orderedmap and Starlark modules Add orderedmap and Starlark JSONPath query APIs with selectors, filters, and syntax errors. | DeepSWEv1.1 · repository | yttGo | 121/16474% | |
| Implement adaptive rejection sampling in R Implement a modular R sampler for log-concave densities with input checks, distribution validation, formal tests, and generated sample output. | Terminal-Bench2.1 · terminal | adaptive rejection samplingR | 75/10075% | |
| Recover and intervene on a Bayesian network Recover a six-edge DAG from observations, fit its Bayesian network, intervene on Y at zero, and draw 10,000 samples from the result. | Terminal-Bench2.1 · terminal | Bayesian networksCSV | 98/10098% | |
| Create HTML that bypasses a JavaScript filter Create an HTML document that still executes an automatic alert after the supplied in-place JavaScript filter processes it. | Terminal-Bench2.1 · terminal | HTMLHTML · JavaScript · Python | 84/10084% | |
| Build pyknotid Cython extensions for NumPy 2.3 Patch and install pyknotid 0.5.3 from source so its three Cython extensions work with NumPy 2.3.0 in the global Python environment. | Terminal-Bench2.1 · terminal | pyknotidPython · C · Cython | 82/10082% | |
| Build headless pMARS from Debian sources Build and install pMARS from Debian source packages with X11 disabled while retaining a working debugger and source tree. | Terminal-Bench2.1 · terminal | pMARSC · Makefile | 99/10099% | |
| Build POV-Ray 2.2 on a modern system Locate, patch, compile, and install the legacy POV-Ray 2.2 source so it reproduces the supplied reference scene. | Terminal-Bench2.1 · terminal | POV-Ray 2.2C | 63/10063% | |
| Train a CPU-only Caffe model on CIFAR-10 Build BVLC Caffe 1.0.0 for CPU execution and train CIFAR-10 for exactly 500 iterations while meeting the stated accuracy bounds. | Terminal-Bench2.1 · terminal | BVLC Caffe 1.0.0Protocol Buffers | 76/10076% | |
| Run bounded async tasks with cancellation cleanup Implement a Python async task runner that limits concurrency and guarantees cleanup for running and queued work when execution is cancelled. | Terminal-Bench2.1 · terminal | Python asyncioPython | 62/10062% | |
| Find the best move from a chessboard image Read a white-to-move chess position from an image and write every winning move in coordinate notation. | Terminal-Bench2.1 · terminal | chessplain text | 67/10067% | |
| Build a gate circuit for Fibonacci of integer square root Describe a logic-gate circuit that maps a 32-bit integer to fib(isqrt(N)) modulo 2^32 within the simulator's line and step limits. | Terminal-Bench2.1 · terminal | logic gatesgate netlist | 85/10184% | |
| Reimplement COBOL file processing in Python Reproduce a GnuCOBOL program in Python so it performs byte-identical updates to the account, book, and transaction data files. | Terminal-Bench2.1 · terminal | GnuCOBOL 3COBOL · Python | 91/10091% | |
| Execute pseudocode recovered from an image Recover the intended pseudocode from an image, implement its cryptographic computation, and write the resulting value. | Terminal-Bench2.1 · terminal | OCRplain text | 91/10091% | |
| Build CompCert 3.13.1 from source Configure and build the CompCert 3.13.1 verified C compiler for the host operating system and architecture. | Terminal-Bench2.1 · terminal | CompCert 3.13.1C · OCaml · Coq | 63/10063% | |
| Deploy Git pushes through Nginx Configure a Git repository and post-receive deployment so pushes to master become static content served by Nginx on port 8080. | Terminal-Bench2.1 · terminal | Gitshell | 45/10045% | |
| Schedule a constrained team meeting Find the earliest valid one-hour slot for three attendees, apply availability and preference tie-breakers, and emit a standards-compliant calendar event. | Terminal-Bench2.1 · terminal | iCalendarICS | 98/10098% | |
| Count science-domain DeepSeek tokens Count Qwen2.5-1.5B-Instruct tokens in the science subset of the OpenThoughts-1k-sample dataset and write the exact integer total. | Terminal-Bench2.1 · terminal | Hugging Face datasetsplain text | 79/10079% | |
| Recover a password-protected 7z secret Recover the password for the supplied 7z archive and extract the word stored in secret_file.txt. | Terminal-Bench2.1 · terminal | 7-Zipplain text | 90/10090% | |
| Fix a release-only C++ heap crash Repair a release-only C++ crash involving custom libstdc++ memory initialization while limiting edits to user.cpp and eliminating leaks. | Terminal-Bench2.1 · terminal | GCCC++ | 83/10083% | |
| Recover an encrypted SQLite WAL Repair the encrypted or corrupted SQLite write-ahead log and export all eleven database rows as sorted JSON. | Terminal-Bench2.1 · terminal | SQLiteJSON | 41/10041% | |
| Find a distribution with dual KL divergence 10 Numerically construct a valid 150,000-element probability distribution whose forward and backward KL divergences from uniform both equal 10 within 0.001. | Terminal-Bench2.1 · terminal | NumPyPython · NumPy binary format | 96/10096% | |
| Design primers for Golden Gate plasmid assembly Design the minimum primer set for one-pot BsaI-HF v2 assembly of the supplied plasmid and protein fragments into the target sequence. | Terminal-Bench2.1 · terminal | PCRFASTA · DNA | 52/10052% | |
| Design Q5 mutagenesis primers Design the minimum primer set that converts the circular input plasmid to the target sequence using the Q5 site-directed mutagenesis kit. | Terminal-Bench2.1 · terminal | PCRFASTA · DNA | 13/10013% | |
| Extract initialized memory values from an ELF binary Implement a Node.js utility that reads an ELF executable and emits accurate address-to-integer mappings for at least 75 percent of its initialized memory. | Terminal-Bench2.1 · terminal | ELFJavaScript · C · JSON | 35/10035% | |
| Transcribe Zork moves from a video Transcribe every command entered during the supplied Zork gameplay video and write the moves in chronological order. | Terminal-Bench2.1 · terminal | video processingplain text | 17/10017% | |
| Recover a FEAL round key with differential cryptanalysis Implement a chosen-plaintext differential attack against the supplied FEAL-like cipher that returns the sixth 32-bit round key within 30 seconds. | Terminal-Bench2.1 · terminal | FEALPython | 95/10095% | |
| Recover a FEAL key with linear cryptanalysis Use known plaintext-ciphertext pairs to recover the FEAL-like cipher key and decrypt every supplied ciphertext. | Terminal-Bench2.1 · terminal | FEALC | 80/10080% | |
| Remove JavaScript from HTML safely Implement an in-place HTML sanitizer that removes executable JavaScript while preserving legitimate structure, content, attributes, and formatting. | Terminal-Bench2.1 · terminal | HTMLPython · HTML · JavaScript | 3/1003% | |
| Classify financial documents and summarize invoices Classify mixed PDF and JPEG documents, move them by type, extract invoice totals and tax, and generate an aggregate CSV summary. | Terminal-Bench2.1 · terminal | OCRCSV | 86/10086% | |
| Fix CRLF injection in Bottle headers Identify and repair the Bottle header-validation flaw that permits CRLF injection, report its CWE, and preserve the repository's test behavior. | Terminal-Bench2.1 · terminal | BottlePython · JSON Lines | 95/10095% | |
| Recover detached Git work into master Locate personal-site changes left on an unreachable or detached commit and merge them into the master branch. | Terminal-Bench2.1 · terminal | GitGit object format | 100/100100% | |
| Repair run-length compression in the OCaml garbage collector Debug a broken major-heap sweep optimization so the OCaml compiler bootstraps and its basic tests run cleanly. | Terminal-Bench2.1 · terminal | OCaml compilerOCaml · C | 97/10097% | |
| Decode printed text from G-code Analyze Prusa MK4S movement commands to determine the text deposited by the supplied G-code and write the decoded string. | Terminal-Bench2.1 · terminal | G-codeG-code | 34/10034% | |
| Recover and purge a leaked Git secret Recover a secret from unreachable Git objects, save it separately, and remove every repository copy without disturbing unrelated history. | Terminal-Bench2.1 · terminal | GitGit object format | 99/10099% | |
| Deploy Git branches to separate HTTPS paths Configure an SSH Git server whose post-receive hook deploys main and dev branches to distinct Nginx HTTPS locations within three seconds. | Terminal-Bench2.1 · terminal | Gitshell | 97/10097% | |
| Implement compact GPT-2 inference in C Write a dependency-free C program under 5,000 bytes that loads GPT-2 TensorFlow checkpoints and BPE vocabulary and emits 20 argmax tokens. | Terminal-Bench2.1 · terminal | GPT-2C | 25/10025% | |
| Implement a persistent headless terminal Implement the supplied terminal interface over an interactive Bash process with persistent state, startup-file loading, interactive programs, and control-key support. | Terminal-Bench2.1 · terminal | BashPython | 82/10082% | |
| Serve Hugging Face sentiment inference Download the specified DistilBERT sentiment model and run a background Flask API that returns positive and negative confidence scores. | Terminal-Bench2.1 · terminal | Hugging Face TransformersPython · JSON | 74/10074% | |
| Run Windows 3.11 in a controllable QEMU VM Boot the supplied Windows 3.11 image in snapshot mode with VNC, an Nginx web interface, and a monitor socket for programmatic keyboard input. | Terminal-Bench2.1 · terminal | QEMU 5.2.0disk image | 41/10041% | |
| Run a gRPC key-value service Define, generate, and run a Python gRPC service that stores integer values under string keys and exposes GetVal and SetVal RPCs. | Terminal-Bench2.1 · terminal | gRPC 1.73.0Python · Protocol Buffers | 52/10052% | |
| Transform a million-row CSV with Vim macros Create a restricted Vim script containing three compact macros that transforms input.csv to the expected file byte for byte. | Terminal-Bench2.1 · terminal | VimVim script · CSV | 95/10095% | |
| Accelerate dominant eigenpair computation Implement a faster Python entrypoint for the largest-magnitude eigenpair of small real nonsymmetric matrices while retaining complex-valued correctness. | Terminal-Bench2.1 · terminal | NumPyPython | 80/10080% | |
| Optimize shape-aware LLM inference batches Pack every request from two buckets into static-shape batches that obey alignment and shape limits while meeting cost, padding, latency, and time thresholds. | Terminal-Bench2.1 · terminal | LLM inferenceJSON Lines · Python | 87/10087% | |
| Summarize log severities by date range Count ERROR, WARNING, and INFO events across five fixed date ranges from date-stamped logs and emit rows in the required CSV order. | Terminal-Bench2.1 · terminal | CSVCSV | 94/10094% | |
| Configure a confirmed-opt-in Mailman list Configure Postfix and Mailman 3 for a local reading-group list with confirmed join and leave messages and subscriber announcements delivered to Unix mailboxes. | Terminal-Bench2.1 · terminal | Mailman 3INI · email | 78/10078% | |
| Cross-compile DOOM for a MIPS emulator Build the supplied DOOM source into the expected MIPS ELF so vm.js runs it and the custom frame writer produces bitmap frames. | Terminal-Bench2.1 · terminal | DOOMC · JavaScript | 1/1001% | |
| Implement a MIPS interpreter that boots DOOM Implement a JavaScript MIPS ELF interpreter with system calls and file access that boots the supplied DOOM binary and saves rendered frames. | Terminal-Bench2.1 · terminal | MIPSJavaScript · C | 51/10051% | |
| Sample a hierarchical beta-binomial model with RStan Implement and run the specified hierarchical Bayesian model in RStan, then save reproducible posterior means for alpha and beta. | Terminal-Bench2.1 · terminal | RStan 2.32.7R · Stan · CSV | 90/10090% | |
| Merge Git bundles and implement the inferred grid mapping Import two bundles as named branches, merge them with conflicts resolved, and implement the grid transformation implied by the ARC-style examples. | Terminal-Bench2.1 · terminal | GitPython · JSON | 95/10095% | |
| Extract hidden weights from a ReLU network Query a black-box one-hidden-layer ReLU function to recover its first-layer weight matrix up to neuron permutation and scaling. | Terminal-Bench2.1 · terminal | ReLU neural networksPython · NumPy binary format | 31/10031% | |
| Modernize a Python 2 climate analysis script Reimplement the legacy climate analysis for current Python using pandas and pathlib while preserving station coverage and formatted mean-temperature output. | Terminal-Bench2.1 · terminal | pandasPython · CSV · INI | 100/100100% | |
| Find the top Scandinavian MTEB embedding model Identify the complete-coverage model with the highest Mean (Task) score on the Scandinavian MTEB leaderboard as of August 2025. | Terminal-Bench2.1 · terminal | MTEBplain text | 68/10068% | |
| Retrieve the fifth-nearest document with MTEB Encode the query and line-delimited corpus with the pinned Chinese BGE model and return the document ranked fifth by cosine similarity. | Terminal-Bench2.1 · terminal | MTEB 1.36.8plain text | 67/10067% | |
| Merge prioritized user records across three formats Normalize and merge JSON, CSV, and Parquet user data by user ID with deterministic source priority and a complete field-level conflict report. | Terminal-Bench2.1 · terminal | JSONJSON · CSV · Parquet · Python | 99/10099% | |
| Configure Nginx request logging and rate limits Run Nginx on port 8080 with static content, detailed access logs, a custom 404 page, and per-IP request limiting. | Terminal-Bench2.1 · terminal | NginxNginx configuration · HTML | 100/100100% | |
| Generate and verify an internal TLS certificate Create a 2048-bit RSA key and one-year self-signed certificate for the specified internal host, plus combined and human-readable verification artifacts. | Terminal-Bench2.1 · terminal | OpenSSLPython | 94/10094% | |
| Eliminate LaTeX overfull boxes with allowed synonyms Replace words in input.tex only with their permitted synonyms until the document compiles without overfull hbox warnings. | Terminal-Bench2.1 · terminal | LaTeXTeX | 71/10071% | |
| Recover a deleted password from a disk image Perform file-system forensics to recover candidate values from a deleted launchcode.txt and write every value matching the exact password pattern. | Terminal-Bench2.1 · terminal | disk forensicsplain text | 92/10092% | |
| Reconstruct a path-traced image algorithmically Write a compact standalone C renderer that reproduces the supplied PPM image with at least 0.99 normalized L2 similarity without reading it at runtime. | Terminal-Bench2.1 · terminal | path tracingC | 63/10063% | |
| Recreate a compiled path tracer in C Reverse-engineer the mystery executable into compact standalone C source whose isolated behavior and generated output are identical. | Terminal-Bench2.1 · terminal | binary analysisC | 59/10059% | |
| Write a Python and C Fibonacci polyglot Create one source file that runs under Python 3.12.3 and compiles under GCC 13.2.0 to print the requested Fibonacci number. | Terminal-Bench2.1 · terminal | Python 3.12.3Python · C | 98/10098% | |
| Write a Rust and C++ Fibonacci polyglot Create one source file accepted by Rust 1.75.0 and G++ 13.2.0 that prints the same shifted Fibonacci sequence value. | Terminal-Bench2.1 · terminal | Rust 1.75.0Rust · C++ | 83/10083% | |
| Accelerate portfolio calculations with a C extension Complete the Python extension so portfolio risk and return match the baseline within 1e-10 and run at least 1.2 times faster at scale. | Terminal-Bench2.1 · terminal | Python C APIC · Python | 97/10097% | |
| Design a FRET fusion-protein gBlock Design a codon sequence for the prescribed five-part fusion protein with exact spectral choices, linker ordering, binding functions, and synthesis constraints. | Terminal-Bench2.1 · terminal | FRETDNA · FASTA · GenBank | 30/10030% | |
| Complete a Coq proof of addition commutativity Fill the missing inductive proof steps for natural-number addition commutativity and compile the verified theorem. | Terminal-Bench2.1 · terminal | CoqCoq | 99/10099% | |
| Host an installable vectorops package Create vectorops 0.1.0 with a dot-product API and host it on a local simple-package index reachable by pip on port 8080. | Terminal-Bench2.1 · terminal | Python packagingPython | 58/10058% | |
| Run MNIST PyTorch inference from a C CLI Convert the supplied PyTorch model weights to JSON and implement a command-line executable that predicts one MNIST digit from an image. | Terminal-Bench2.1 · terminal | PyTorchC · Python · JSON | 84/10084% | |
| Recover and tune a PyTorch model architecture Reconstruct the network represented by a state dictionary, freeze all but output_layer, improve dataset MSE, and export the result as TorchScript. | Terminal-Bench2.1 · terminal | PyTorchPython | 82/10082% | |
| Boot Alpine in QEMU with SSH access Run the supplied Alpine ISO in QEMU and configure a password-authenticated root SSH shell forwarded to localhost port 2222. | Terminal-Bench2.1 · terminal | QEMUISO disk image | 58/10058% | |
| Expose an Alpine QEMU console over Telnet Boot the supplied Alpine ISO in a background QEMU process and expose its ready login console over Telnet on localhost port 6665. | Terminal-Bench2.1 · terminal | QEMUISO disk image | 74/10074% | |
| Optimize an Open English WordNet SQL query Rewrite the supplied SQLite query for maximum efficiency while preserving its exact result against the immutable OEWN database. | Terminal-Bench2.1 · terminal | SQLiteSQL | 56/10056% | |
| Fit Raman G and 2D peaks Fit Lorentzian models to the graphene spectrum's G and 2D peaks and export each peak's center, width, amplitude, and offset. | Terminal-Bench2.1 · terminal | Raman spectroscopyJSON · Python | 17/10017% | |
| Generate legal chess positions with regex substitutions Encode a complete white move generator as ordered regex replacements that transforms FEN into every legal successor position. | Terminal-Bench2.1 · terminal | regular expressionsregular expressions · JSON · Python | 51/10150% | |
| Match the last valid date on IPv4 log lines Write one Python-compatible multiline regex that returns only the last valid date from each line containing a boundary-safe IPv4 address. | Terminal-Bench2.1 · terminal | regular expressionsregular expressions · Python | 86/10086% | |
| Reshard and restore C4 dataset directories Implement reversible dataset resharding that caps directory fanout and file size, recreates the original tree exactly, and runs through a locked uv environment. | Terminal-Bench2.1 · terminal | C4 datasetPython · TOML | 99/10099% | |
| Port an RStan Gaussian process to PyStan Translate the supplied RStan Gaussian-process analysis to PyStan 3.10.0 with equivalent model, data, hyperparameters, sampling, and posterior summaries. | Terminal-Bench2.1 · terminal | PyStan 3.10.0Python · R · Stan · CSV · JSON | 92/10092% | |
| Refine cell boxes into MobileSAM contours Implement a CPU MobileSAM pipeline that converts every rectangular or polygonal cell annotation into one nonoverlapping contiguous polyline and updates its geometry fields. | Terminal-Bench2.1 · terminal | MobileSAMPython · CSV | 88/10088% | |
| Replace secrets in the dclm repository Find API credentials in the repository and replace each with a consistent service-specific placeholder without changing uncontaminated files. | Terminal-Bench2.1 · terminal | GitGit repository format | 48/10048% | |
| Implement a self-interpreting Scheme evaluator Write a Scheme metacircular evaluator for the supplied language that runs all programs, forwards remaining input and output, and can interpret itself recursively. | Terminal-Bench2.1 · terminal | SchemeScheme · Python | 70/10070% | |
| Query eligible EU university professors Write a SPARQL query returning full professors who currently work in an EU university department with more than ten enrolled students. | Terminal-Bench2.1 · terminal | SPARQLSPARQL · Turtle | 96/10096% | |
| Recover rows from a truncated SQLite database Recover as many records as possible from the binary-truncated SQLite file and serialize each word and numeric value to JSON. | Terminal-Bench2.1 · terminal | SQLiteJSON | 99/10099% | |
| Build gcov-instrumented SQLite Compile the pre-vendored SQLite source with gcov instrumentation and make the resulting command-line tool available on PATH. | Terminal-Bench2.1 · terminal | SQLiteC | 92/10092% | |
| Implement AFAB pipeline parallelism for LLaMA Implement one LLaMA training step that balances layers across ranks and executes all microbatch forwards before all backwards with correct distributed tensors. | Terminal-Bench2.1 · terminal | PyTorchPython | 38/10038% | |
| Implement row and column tensor-parallel linear layers Implement distributed PyTorch linear layers with correct master-weight sharding, collective-style outputs, zero bias initialization, and gradient behavior. | Terminal-Bench2.1 · terminal | PyTorchPython | 75/10075% | |
| Train a compact Yelp FastText classifier Train a FastText text classifier on the supplied Yelp reviews that exceeds 0.62 held-out accuracy while keeping the model under 150MB. | Terminal-Bench2.1 · terminal | FastTextFastText binary format | 15/10015% | |
| Accelerate a MuJoCo model without changing physics Tune the MJCF model so a two-second simulation runs in at most 60 percent of the reference time while reaching the same finite physics state. | Terminal-Bench2.1 · terminal | MuJoCoXML · Python | 90/10090% | |
| Detect hurdle takeoff and landing frames Implement an OpenCV script that analyzes a stationary-camera hurdle video and records the single jump's takeoff and landing frame numbers in TOML. | Terminal-Bench2.1 · terminal | OpenCVPython · TOML | 28/10028% | |
| Extract a hidden flag from a vulnerable executable Analyze and interact with the supplied executable to recover its FLAG-formatted secret and save the exact value. | Terminal-Bench2.1 · terminal | binary analysisplain text | 94/10094% | |
| Write a Core War warrior with target win rates Create a Redcode warrior that meets separate 100-round win-rate thresholds against five supplied pMARS opponents without modifying them. | Terminal-Bench2.1 · terminal | pMARSRedcode | 63/10063% | |
| Create data for a custom decompressor Reverse-engineer the supplied decompression format and produce a compact file that expands byte-for-byte to data.txt. | Terminal-Bench2.1 · terminal | custom compressionC | 80/10080% |