Evidence browser

Cases, minus the archaeology.

Search compact task profiles, see what makes them difficult, and compare published model outcomes.

Cases
202
Suites
2
Published trials
27424

202 cases

Showing 202 of 202 evidence cases.

compact profile · exact revision
Task profileBenchmarkTechnologyPublished passesOpen case
Harden module loading, cache introspection, and script flags

Harden ABS module resolution and caching, expose cache introspection APIs, and make module flags work correctly in script mode.

DeepSWEv1.1 · repositoryABSGo113/16469%
Add stepped slices for arrays and strings

Add stepped slice parsing, slicing, and assignment for arrays and strings with rune-correct indexing.

DeepSWEv1.1 · repositoryABSGo115/16470%
Add action pinning linting for actions and reusable workflows

Add a configurable lint rule that enforces pinned versions for action and reusable workflow references.

DeepSWEv1.1 · repositoryactionlintGo131/16480%
Add input key aliases to name mapping

Add load-only alias support to name mapping so fields can resolve from alternate input keys.

DeepSWEv1.1 · repositoryAdaptixPython101/16462%
Add task snapshots, inspection, and diffing to aiomonitor

Add point-in-time task snapshots with interactive inspection, diffing, deletion, and web/CLI access.

DeepSWEv1.1 · repositoryaiomonitorPython103/16363%
Add default arguments to Anko function parameters

Add parsing and call-time evaluation of default parameter values in Anko functions.

DeepSWEv1.1 · repositoryAnkoGo113/16469%
Add typed variable bindings to Anko

Add typed `var` declarations in Anko and enforce declared type constraints on assignment when TypedBindings is enabled.

DeepSWEv1.1 · repositoryAnkoGo67/16441%
Add drift detection and compliance baselines

Implement baseline capture, drift comparison, and compliance tracking for container configurations.

DeepSWEv1.1 · repositoryArcaneGo106/16465%
Add JSON Schema refs and dependency keywords

Add JSON Schema dependency keywords, local $defs/$ref resolution, and conditional schema handling.

DeepSWEv1.1 · repositoryArkTypeTypeScript60/16437%
Add dependency-aware async initialization to the container

Add dependency-aware asynchronous container initialization with ordered startup, concurrency limits, and rollback on failure.

DeepSWEv1.1 · repositoryAwilixTypeScript57/16435%
Add incremental cache controls to Bandit

Add incremental analysis caching with cache invalidation, import/export, pruning, and cache inspection CLI options.

DeepSWEv1.1 · repositoryBanditPython83/16451%
Add interprocedural taint checks for Bandit injection sinks

Track user input through variables and calls so Bandit flags injection sinks beyond string literals.

DeepSWEv1.1 · repositoryBanditPython86/16452%
Add structured nosec directives for regions and next line

Add region and next-line nosec directives with selector expressions and ignore-nosec handling.

DeepSWEv1.1 · repositoryBanditPython8/1645%
Add hierarchical evaluation cancellation to Boa

Add cancellable evaluation handles that propagate through nested script, module, and job execution.

DeepSWEv1.1 · repositoryBoaRust70/16343%
Add partial structuring with error recovery to cattrs

Add `partial_structure` and `PartialResult` for recoverable, field-level structuring with nested partial results.

DeepSWEv1.1 · repositorycattrsPython97/16459%
Add async autocomplete options and fetch lifecycle handling

Add async option fetching with caching, retries, debouncing, and loading state to AutocompletePrompt.

DeepSWEv1.1 · repositoryClackTypeScript44/16427%
Implement recursive agent delegation through delegate_task tool calls

Run delegated sub-agents, feed their results back as tool_result, and handle unknown agents, failures, and circular delegation.

DeepSWEv1.1 · repositoryClaude Code by AgentsTypeScript89/16355%
Add config file parsing to Cliffy commands

Add command-level config file loading, parsing, merging, and precedence handling.

DeepSWEv1.1 · repositoryCliffyTypeScript75/16446%
Add shorthand expansion and compression to the lexer

Add lexer methods to expand CSS shorthands into longhands and compress longhands back into shorthand values.

DeepSWEv1.1 · repositoryCSSTreeJavaScript48/16429%
Add HTML document format handling to Dasel

Add read and write support for HTML documents with Dasel's format handling.

DeepSWEv1.1 · repositoryDaselGo71/16443%
Add RFC 5545 timezone interoperability to dateutil recurrence parsing

Extend rrule and rruleset to serialize, parse, and compare RFC 5545 timezone-aware recurrence data.

DeepSWEv1.1 · repositorypython-dateutilPython59/16436%
Add typed window function builders with OVER clauses

Add typed query-builder helpers for SQL window functions, named windows, and frame specs.

DeepSWEv1.1 · repositoryDrizzle ORMTypeScript139/16485%
Add conditional required attributes to schemas

Add `requiredIf`-based conditional attribute enforcement across schema validation, parsing, updates, and JSON Schema export.

DeepSWEv1.1 · repositoryDynamoDB ToolboxTypeScript112/16369%
Add lazy recursive schemas with DTO and JSON Schema export

Add a lazy schema type for self-referencing recursive data with full serialization, validation, and export support.

DeepSWEv1.1 · repositoryDynamoDB ToolboxTypeScript103/16064%
Add SSE streaming endpoints to HttpApi

Add typed Server-Sent Events streaming endpoints, encoders/decoders, and client handling to HttpApi.

DeepSWEv1.1 · repositoryEffectTypeScript25/16415%
Add keyset cursor pagination to `$find`

Add cursor-based keyset pagination to `$find` with nextCursor handling and validation.

DeepSWEv1.1 · repositoryEICRUDTypeScript41/16425%
Add XML diff, patch, and merge operations to etree

Add recursive XML diffing, patch generation and application, reverse patching, three-way merge, and diff summaries.

DeepSWEv1.1 · repositoryetreeGo93/16457%
Add try/catch error recovery to expr

Add expression and block-level error recovery with try, catch, finally, throw, retry, and errtype.

DeepSWEv1.1 · repositoryExprGo42/16426%
Add deprecation, sunset, and successor headers to FastAPI routes

Add runtime Deprecation, Sunset, and Link headers plus OpenAPI metadata and tracking middleware for deprecated routes.

DeepSWEv1.1 · repositoryFastAPIPython71/16443%
Add implicit HEAD and automatic OPTIONS responses to FastAPI routes

Add configurable implicit HEAD handling and automatic OPTIONS responses for FastAPI routes, routers, and included routers.

DeepSWEv1.1 · repositoryFastAPIPython80/16449%
Add deterministic multi-key sorting to fd

Add repeatable multi-key sorting controls to fd output with deterministic tie-breaking and seeded random order.

DeepSWEv1.1 · repositoryfdRust96/16459%
Add ShapeIndex encoding and decoding

Add stable ShapeIndex Encode/Decode support so indices round-trip without rebuilding.

DeepSWEv1.1 · repositoryGeoGo116/16471%
Add a checker for broken doc comment links

Add a diagnostic checker that validates doc comment symbol links against package and type information.

DeepSWEv1.1 · repositorygo-criticGo82/16450%
Expose accumulated streamed function-call args in SDK surfaces

Expose fully accumulated streamed function-call arguments across streaming, live sessions, and chat history.

DeepSWEv1.1 · repositoryGo GenAIGo111/16468%
Add worktree merge conflict handling

Add worktree merge support with conflict detection, merge head handling, and index stage updates.

DeepSWEv1.1 · repositorygo-gitGo98/16460%
Add retry-aware publishing audit logs

Implement per-artifact retries for uploads, artifactory, and blobs while recording deterministic publish attempt history.

DeepSWEv1.1 · repositoryGoReleaserGo128/16478%
Add GraphQL incremental delivery with @defer and @stream

Add incremental GraphQL response handling with @defer and @stream across HTTP multipart, WebSocket transport, and the DSL.

DeepSWEv1.1 · repositorygqlPython6/1644%
Abort pending body reads on shutdown

Ensure interrupted request and response body reads, formData parsing, and discarded timers abort cleanly during shutdown.

DeepSWEv1.1 · repositoryHappy DOMTypeScript150/16491%
Implement a deterministic IntersectionObserver in Happy DOM

Implement a real, deterministic IntersectionObserver with async delivery, thresholds, root margins, and target tracking.

DeepSWEv1.1 · repositoryHappy DOMTypeScript50/16430%
Add configurable array merge strategies to Helm value coalescing

Add chart-scoped append and key-based merge strategies for coalescing arrays during Helm value merging.

DeepSWEv1.1 · repositoryHelmGo26/16416%
Add unified manifest stream output across Helm commands

Add a stable unified manifest stream for template, dry-run install/upgrade, and get manifest output.

DeepSWEv1.1 · repositoryHelmGo125/16476%
Add a deterministic CookieStore with modern Set-Cookie parsing

Add a deterministic CookieStore with modern Set-Cookie parsing, cookie eviction, and request header handling.

DeepSWEv1.1 · repositoryHTTPXTypeScript132/16480%
Add multipart response parsing to HTTPX

Add Response iterators that parse multipart HTTP response bodies into parts.

DeepSWEv1.1 · repositoryHTTPXPython107/16465%
Add streaming JSON iteration to HTTPX responses

Add response iterators that incrementally parse JSON values from supported streaming media types.

DeepSWEv1.1 · repositoryHTTPXPython63/16438%
Persist the fitted feature schema across evaluate, predict, serve, and export

Persist and reuse the fitted feature schema so evaluation, prediction, serving, and export all apply the same canonicalized inputs.

DeepSWEv1.1 · repositoryigelPython80/16449%
Add CSS Grid layout to the Box component

Add CSS Grid layout parsing and placement to the Box component, including track sizing, gaps, and explicit child positioning.

DeepSWEv1.1 · repositoryInkTypeScript49/16430%
Add session bundle recording and replay to IPython

Add `.ipybundle` recording, validation, and replay APIs for IPython sessions.

DeepSWEv1.1 · repositoryIPythonPython93/16457%
Add `\multicolumn` column spans to array-like environments

Add `\multicolumn` parsing and rendering for array-like environments with span-aware alignment and errors.

DeepSWEv1.1 · repositoryKaTeXJavaScript40/16424%
Add multiplexed ordered streams over KCP

Add a multiplexing layer that carries many ordered sub-streams over one KCP connection with flow control, priority scheduling, and SNMP counters.

DeepSWEv1.1 · repositorykcp-goGo115/16470%
Add atomic signal selectors to Kea

Introduce fine-grained atomic selector tracking with dependency health, circular detection, and unchanged Kea lifecycle behavior.

DeepSWEv1.1 · repositoryKeaTypeScript52/16432%
Add consistent hash policy support to TrafficPolicy

Add spec.consistentHash to TrafficPolicy and merge it into route hash_policy generation.

DeepSWEv1.1 · repositorykgatewayGo60/16437%
Add single-active-consumer priority and cancel tracking to virtual transports

Add single-active-consumer semantics, consumer priority selection, cancel notifications, and lifecycle tracking to virtual transports.

DeepSWEv1.1 · repositoryKombuPython85/16452%
Add dead-lettering, TTL, and overflow handling to virtual queues

Add dead-letter routing, TTL expiry, and max-length overflow handling to the virtual transport layer.

DeepSWEv1.1 · repositoryKombuPython29/16418%
Add composite trait aspects to Koota

Add createAspect and aspect-aware query operations that merge constituent traits and propagate writes and lifecycle events.

DeepSWEv1.1 · repositoryKootaTypeScript50/16430%
Add a deferred mutation buffer to batch entity changes

Add a deferred command buffer that batches entity mutations during query iteration and flushes them at defined boundaries.

DeepSWEv1.1 · repositoryKootaTypeScript41/16425%
Add entity snapshot and rollback APIs to Koota

Add snapshot, rollback, and diff APIs for entities and worlds, including registry-based trait and relation serialization.

DeepSWEv1.1 · repositoryKootaPython126/16477%
Add pair-level relation tracking modifiers

Tracking modifiers should distinguish changes to specific relation pairs, not just trait-level additions and removals.

DeepSWEv1.1 · repositoryKootaTypeScript29/16418%
Add value-based query predicates to Koota

Add composable value-based entity predicates with dependency tracking and change transitions.

DeepSWEv1.1 · repositoryKootaTypeScript30/16418%
Add grouping-set and window-frame SQL helpers

Add grouped aggregation clauses, window frame builders, null-handling modifiers, and related SQL function helpers.

DeepSWEv1.1 · repositoryKyselyTypeScript105/16464%
Add request coalescing to `Runnable`

Add `Runnable.with_coalesce()` so concurrent identical inputs share one execution across sync, async, streaming, and batch APIs.

DeepSWEv1.1 · repositoryLangChainPython58/16435%
Add flattened dataclass fields to Mashumaro field options

Add field_options support for flattening nested dataclass fields into parent dictionaries with prefix and rename validation.

DeepSWEv1.1 · repositoryMashumaroPython77/16447%
Add explicit resource management declarations to the parser

Add parsing and AST support for `using` and `await using` declarations, including context-sensitive errors.

DeepSWEv1.1 · repositoryMeriyahTypeScript40/16424%
Validate daemon watch, status, and log lifecycle

Add daemon watch validation, state tracking, logging, and lifecycle commands with non-blocking run-once processing.

DeepSWEv1.1 · repositorymnamerPython85/16452%
Add grouped test phases with synchronized barriers

Add grouped execution phases with explicit synchronization barriers and per-group setup and teardown.

DeepSWEv1.1 · repositoryMoblyPython94/16457%
Add rolling min, max, median, and quantile methods

Add the remaining rolling window methods to Expr and Series with consistent validation and backend delegation.

DeepSWEv1.1 · repositoryNarwhalsPython143/16487%
Add boundary modes to `@stencil`

Add configurable boundary handling modes to `@stencil` for out-of-bounds accesses.

DeepSWEv1.1 · repositoryNumbaPython97/16459%
Add automatic table of contents generation for Obsidian linter

Implement an opt-in rule that generates and updates a TOC from document headings.

DeepSWEv1.1 · repositoryObsidian LinterTypeScript1/1641%
Add link format conversion between wiki and markdown syntax

Add a Link Style rule that converts between Obsidian wiki links or embeds and markdown links or images while preserving ignored regions and edge cases.

DeepSWEv1.1 · repositoryObsidian LinterTypeScript48/16429%
Add scoped per-rule ignore markers to Obsidian Linter

Implement standalone comment markers that disable and re-enable specific lint rules with nested scope handling.

DeepSWEv1.1 · repositoryObsidian LinterTypeScript104/16463%
Add a per-origin circuit breaker to ofetch

Add an opt-in per-origin circuit breaker for fetch requests with half-open probing and shared state across clients.

DeepSWEv1.1 · repositoryofetchTypeScript125/16476%
Add transparent encryption to dump uploads

Encrypt database dumps before upload and add keyed config, loading, and decryptable streaming support.

DeepSWEv1.1 · repositoryonedumpGo44/16427%
Add rule evaluation profiling to Rego

Add opt-in per-rule evaluation profiling to Rego results with profiling stats and diff helpers.

DeepSWEv1.1 · repositoryOpen Policy AgentGo97/16459%
Reconstruct template strings in partial evaluation output

Rebuild lowered template strings into normal Rego syntax in partial evaluation outputs and support modules.

DeepSWEv1.1 · repositoryOpen Policy AgentGo90/16455%
Add conditional option dependencies to Optique

Add conditional dependencies so options can require or hide based on other option presence or values.

DeepSWEv1.1 · repositoryOptiqueTypeScript44/16427%
Preserve structure needed by stylesheet selectors

Keep rewrite optimizations from breaking structure-dependent stylesheet selector matching.

DeepSWEv1.1 · repositoryoxvgRust17/16310%
Add build-time grammar conflict analysis to participle

Add build-time static analysis that detects ambiguous participle grammars and reports conflicts.

DeepSWEv1.1 · repositoryParticipleGo35/16421%
Add durability callbacks and wait APIs for sync writes

Add batch durability callbacks, durability wait methods, notifications, and metrics for sync write commits.

DeepSWEv1.1 · repositoryPebbleGo90/16455%
Coalesce qualifying choices into character classes

Add optimizer passes that collapse qualifying choice chains into merged character and negated character classes.

DeepSWEv1.1 · repositoryPestRust21/16413%
Add transactional reload status and rollback tracking to Prometheus

Add an opt-in transactional config reload mode with durable reload outcomes, rollback tracking, and an HTTP status endpoint.

DeepSWEv1.1 · repositoryPrometheusTypeScript31/16419%
Fix PromQL label sorting across typed and untyped values

PromQL label sorting must order mixed typed and untyped label values with stable typed comparison rules.

DeepSWEv1.1 · repositoryPrometheusGo76/16446%
Add typed blend range access and blend-if compositing

Add typed blend range objects, persist them on layers, and apply blend-if during compositing.

DeepSWEv1.1 · repositorypsd-toolsPython131/16480%
Add tube multiplexing to pwntools

Add a TubeMultiplexer and MuxChannel API for multiple bidirectional logical channels over one tube, with flow control and close propagation.

DeepSWEv1.1 · repositorypwntoolsPython52/16332%
Add scoped state data to state machine callbacks and history

Add per-state scoped data ownership with lifecycle resets, callback injection, history restoration, and validation.

DeepSWEv1.1 · repositorypython-statemachinePython45/16427%
Preserve restored query state in persisted snapshots

Preserve full persisted query state, including errors, counters, timestamps, and infinite pagination, during restoration and cache rebuilds.

DeepSWEv1.1 · repositoryTanStack QueryTypeScript113/16469%
Reuse one toolbar across multiple Quill editors

Allow multiple Quill editors to share one toolbar container while routing toolbar actions and UI state to the currently active editor.

DeepSWEv1.1 · repositoryQuillTypeScript42/16426%
Add an error-accumulating Validated container

Add a Validated container that accumulates errors while preserving standard container APIs.

DeepSWEv1.1 · repositoryReturnsPython131/16480%
Add bounded-memory spilling to SCC aggregation

Add an opt-in bounded-memory mode that spills per-file aggregation results to disk without changing output.

DeepSWEv1.1 · repositorysccGo97/16459%
Add method declarations and interface dispatch to Scriggo

Implement method declarations with receiver handling, method expressions, and interface satisfaction for Scriggo-defined types.

DeepSWEv1.1 · repositoryScriggoGo88/16454%
Add duration encoding to TableVectorizer

Add a DurationEncoder, duration selector, and duration routing in TableVectorizer.

DeepSWEv1.1 · repositoryskrubPython87/16453%
Format BigQuery pipe syntax queries correctly

Add parsing and formatting support for BigQuery pipe syntax queries without changing traditional SQL formatting.

DeepSWEv1.1 · repositorySQL FormatterTypeScript131/16480%
Format CREATE TABLE DDL and add DDL parsing helpers

Format CREATE TABLE statements with DDL-aware line breaking and add parsing models for table columns and constraints.

DeepSWEv1.1 · repositorysqlfmtPython36/16422%
Add safe import checkpoints and invariant validation

Add safe bulk import checkpoints, invariant validation, and rollback-on-failure behavior.

DeepSWEv1.1 · repositorysqlite-utilsPython79/16448%
Add error stack serialization to SuperJSON

Add configurable serialization and restoration of error stacks, stack frames, causes, and sanitization in SuperJSON.

DeepSWEv1.1 · repositorySuperJSONTypeScript55/16434%
Add task graph export with JSON, DOT, and text output

Add a graph export command for tasks with JSON, DOT, and text output, including reverse traversal and status control.

DeepSWEv1.1 · repositoryTaskGo116/16471%
Fix isolated Go-side calls for Tengo callables and closures

Enable Go-side invocation of exported Tengo functions and closures while preserving runtime context and isolating compiled-instance state.

DeepSWEv1.1 · repositoryTengoGo97/16459%
Add destructuring bindings to Tengo

Add destructuring bindings for `:=` in arrays, maps, and function parameters.

DeepSWEv1.1 · repositoryTengoGo79/16448%
Preserve ANSI resets during truncation and styling

Add ANSI tokenization, reset-preserving truncation, and style-aware output helpers.

DeepSWEv1.1 · repositorytermenvGo27/16416%
Add bail-on-test-failure handling to Testem

Add configurable early bailout on test failures across runners, reporters, and exit codes.

DeepSWEv1.1 · repositoryTestemJavaScript22/16413%
Partition report files by launcher and expand report templates

Split report output into per-launcher files with template expansion, launcher-safe filenames, and per-launcher summaries.

DeepSWEv1.1 · repositoryTestemJavaScript107/16465%
Complete Kitty keyboard phases and stable fallback key metadata

Add full Kitty keyboard phase handling and stable fallback key metadata for keys and shortcuts.

DeepSWEv1.1 · repositoryTextualPython46/16428%
Restore RichLog follow-state parity and expand reflow behavior

Restore RichLog follow-mode parity with Log and preserve expand reflow behavior for justified writes.

DeepSWEv1.1 · repositoryTextualPython106/16465%
Add bidirectional TOML table converters

Add in-place conversion helpers between standard tables, inline tables, dotted keys, and super tables while preserving comments and round-trip integrity.

DeepSWEv1.1 · repositoryTOMLKitPython95/16458%
Add iterable collection combinators to true-myth

Add iterable-aware sequence, traverse, zip, filtering, and task combinators across Maybe, Result, Task, and toolbelt APIs.

DeepSWEv1.1 · repositoryTrue MythTypeScript141/16486%
Add `matchEach` to ts-pattern

Add a new `matchEach` matcher that evaluates all matching clauses and returns all results in order.

DeepSWEv1.1 · repositoryts-patternTypeScript117/16471%
Add policy-based alerting for failures, latency, and SSL expiry

Add policy-driven alerting with consecutive failure, latency, recovery, and SSL expiry notifications.

DeepSWEv1.1 · repositoryUpdoGo19/16412%
Add recursive schema composition to Valibot

Add first-class recursive schema composition with `Recur`, `recursive(...)`, and `recursiveAsync(...)` wrappers.

DeepSWEv1.1 · repositoryValibotTypeScript104/16463%
Add duration-aware sharding to Vitest

Add duration-aware sharding strategies and duration history handling for Vitest.

DeepSWEv1.1 · repositoryVitestTypeScript120/16473%
Add a persistent analysis cache to Vulture

Add a persistent cache so Vulture can reuse unchanged analysis across runs.

DeepSWEv1.1 · repositoryVulturePython53/16432%
Add trap coredump generation to wasmi

Generate opt-in Wasm coredumps on traps and attach the bytes to errors.

DeepSWEv1.1 · repositorywasmiRust100/16461%
Add multi-module memory snapshots to wazero

Add coordinated capture, restore, diff, and serialization for multi-module memory snapshots.

DeepSWEv1.1 · repositorywazeroGo122/16474%
Add go:embed directive support for interpreted packages

Support //go:embed directives so interpreted package variables can receive embedded file contents and embed.FS values.

DeepSWEv1.1 · repositoryYaegiGo97/16459%
Add deterministic map conflict detection to Y.Map writes

Add strict, deterministic conflict detection for Y.Map key writes with collect and error policies.

DeepSWEv1.1 · repositoryYjsJavaScript120/16473%
Add JSONPath query APIs to orderedmap and Starlark modules

Add orderedmap and Starlark JSONPath query APIs with selectors, filters, and syntax errors.

DeepSWEv1.1 · repositoryyttGo121/16474%
Implement adaptive rejection sampling in R

Implement a modular R sampler for log-concave densities with input checks, distribution validation, formal tests, and generated sample output.

Terminal-Bench2.1 · terminaladaptive rejection samplingR75/10075%
Recover and intervene on a Bayesian network

Recover a six-edge DAG from observations, fit its Bayesian network, intervene on Y at zero, and draw 10,000 samples from the result.

Terminal-Bench2.1 · terminalBayesian networksCSV98/10098%
Create HTML that bypasses a JavaScript filter

Create an HTML document that still executes an automatic alert after the supplied in-place JavaScript filter processes it.

Terminal-Bench2.1 · terminalHTMLHTML · JavaScript · Python84/10084%
Build pyknotid Cython extensions for NumPy 2.3

Patch and install pyknotid 0.5.3 from source so its three Cython extensions work with NumPy 2.3.0 in the global Python environment.

Terminal-Bench2.1 · terminalpyknotidPython · C · Cython82/10082%
Build headless pMARS from Debian sources

Build and install pMARS from Debian source packages with X11 disabled while retaining a working debugger and source tree.

Terminal-Bench2.1 · terminalpMARSC · Makefile99/10099%
Build POV-Ray 2.2 on a modern system

Locate, patch, compile, and install the legacy POV-Ray 2.2 source so it reproduces the supplied reference scene.

Terminal-Bench2.1 · terminalPOV-Ray 2.2C63/10063%
Train a CPU-only Caffe model on CIFAR-10

Build BVLC Caffe 1.0.0 for CPU execution and train CIFAR-10 for exactly 500 iterations while meeting the stated accuracy bounds.

Terminal-Bench2.1 · terminalBVLC Caffe 1.0.0Protocol Buffers76/10076%
Run bounded async tasks with cancellation cleanup

Implement a Python async task runner that limits concurrency and guarantees cleanup for running and queued work when execution is cancelled.

Terminal-Bench2.1 · terminalPython asyncioPython62/10062%
Find the best move from a chessboard image

Read a white-to-move chess position from an image and write every winning move in coordinate notation.

Terminal-Bench2.1 · terminalchessplain text67/10067%
Build a gate circuit for Fibonacci of integer square root

Describe a logic-gate circuit that maps a 32-bit integer to fib(isqrt(N)) modulo 2^32 within the simulator's line and step limits.

Terminal-Bench2.1 · terminallogic gatesgate netlist85/10184%
Reimplement COBOL file processing in Python

Reproduce a GnuCOBOL program in Python so it performs byte-identical updates to the account, book, and transaction data files.

Terminal-Bench2.1 · terminalGnuCOBOL 3COBOL · Python91/10091%
Execute pseudocode recovered from an image

Recover the intended pseudocode from an image, implement its cryptographic computation, and write the resulting value.

Terminal-Bench2.1 · terminalOCRplain text91/10091%
Build CompCert 3.13.1 from source

Configure and build the CompCert 3.13.1 verified C compiler for the host operating system and architecture.

Terminal-Bench2.1 · terminalCompCert 3.13.1C · OCaml · Coq63/10063%
Deploy Git pushes through Nginx

Configure a Git repository and post-receive deployment so pushes to master become static content served by Nginx on port 8080.

Terminal-Bench2.1 · terminalGitshell45/10045%
Schedule a constrained team meeting

Find the earliest valid one-hour slot for three attendees, apply availability and preference tie-breakers, and emit a standards-compliant calendar event.

Terminal-Bench2.1 · terminaliCalendarICS98/10098%
Count science-domain DeepSeek tokens

Count Qwen2.5-1.5B-Instruct tokens in the science subset of the OpenThoughts-1k-sample dataset and write the exact integer total.

Terminal-Bench2.1 · terminalHugging Face datasetsplain text79/10079%
Recover a password-protected 7z secret

Recover the password for the supplied 7z archive and extract the word stored in secret_file.txt.

Terminal-Bench2.1 · terminal7-Zipplain text90/10090%
Fix a release-only C++ heap crash

Repair a release-only C++ crash involving custom libstdc++ memory initialization while limiting edits to user.cpp and eliminating leaks.

Terminal-Bench2.1 · terminalGCCC++83/10083%
Recover an encrypted SQLite WAL

Repair the encrypted or corrupted SQLite write-ahead log and export all eleven database rows as sorted JSON.

Terminal-Bench2.1 · terminalSQLiteJSON41/10041%
Find a distribution with dual KL divergence 10

Numerically construct a valid 150,000-element probability distribution whose forward and backward KL divergences from uniform both equal 10 within 0.001.

Terminal-Bench2.1 · terminalNumPyPython · NumPy binary format96/10096%
Design primers for Golden Gate plasmid assembly

Design the minimum primer set for one-pot BsaI-HF v2 assembly of the supplied plasmid and protein fragments into the target sequence.

Terminal-Bench2.1 · terminalPCRFASTA · DNA52/10052%
Design Q5 mutagenesis primers

Design the minimum primer set that converts the circular input plasmid to the target sequence using the Q5 site-directed mutagenesis kit.

Terminal-Bench2.1 · terminalPCRFASTA · DNA13/10013%
Extract initialized memory values from an ELF binary

Implement a Node.js utility that reads an ELF executable and emits accurate address-to-integer mappings for at least 75 percent of its initialized memory.

Terminal-Bench2.1 · terminalELFJavaScript · C · JSON35/10035%
Transcribe Zork moves from a video

Transcribe every command entered during the supplied Zork gameplay video and write the moves in chronological order.

Terminal-Bench2.1 · terminalvideo processingplain text17/10017%
Recover a FEAL round key with differential cryptanalysis

Implement a chosen-plaintext differential attack against the supplied FEAL-like cipher that returns the sixth 32-bit round key within 30 seconds.

Terminal-Bench2.1 · terminalFEALPython95/10095%
Recover a FEAL key with linear cryptanalysis

Use known plaintext-ciphertext pairs to recover the FEAL-like cipher key and decrypt every supplied ciphertext.

Terminal-Bench2.1 · terminalFEALC80/10080%
Remove JavaScript from HTML safely

Implement an in-place HTML sanitizer that removes executable JavaScript while preserving legitimate structure, content, attributes, and formatting.

Terminal-Bench2.1 · terminalHTMLPython · HTML · JavaScript3/1003%
Classify financial documents and summarize invoices

Classify mixed PDF and JPEG documents, move them by type, extract invoice totals and tax, and generate an aggregate CSV summary.

Terminal-Bench2.1 · terminalOCRCSV86/10086%
Fix CRLF injection in Bottle headers

Identify and repair the Bottle header-validation flaw that permits CRLF injection, report its CWE, and preserve the repository's test behavior.

Terminal-Bench2.1 · terminalBottlePython · JSON Lines95/10095%
Recover detached Git work into master

Locate personal-site changes left on an unreachable or detached commit and merge them into the master branch.

Terminal-Bench2.1 · terminalGitGit object format100/100100%
Repair run-length compression in the OCaml garbage collector

Debug a broken major-heap sweep optimization so the OCaml compiler bootstraps and its basic tests run cleanly.

Terminal-Bench2.1 · terminalOCaml compilerOCaml · C97/10097%
Decode printed text from G-code

Analyze Prusa MK4S movement commands to determine the text deposited by the supplied G-code and write the decoded string.

Terminal-Bench2.1 · terminalG-codeG-code34/10034%
Recover and purge a leaked Git secret

Recover a secret from unreachable Git objects, save it separately, and remove every repository copy without disturbing unrelated history.

Terminal-Bench2.1 · terminalGitGit object format99/10099%
Deploy Git branches to separate HTTPS paths

Configure an SSH Git server whose post-receive hook deploys main and dev branches to distinct Nginx HTTPS locations within three seconds.

Terminal-Bench2.1 · terminalGitshell97/10097%
Implement compact GPT-2 inference in C

Write a dependency-free C program under 5,000 bytes that loads GPT-2 TensorFlow checkpoints and BPE vocabulary and emits 20 argmax tokens.

Terminal-Bench2.1 · terminalGPT-2C25/10025%
Implement a persistent headless terminal

Implement the supplied terminal interface over an interactive Bash process with persistent state, startup-file loading, interactive programs, and control-key support.

Terminal-Bench2.1 · terminalBashPython82/10082%
Serve Hugging Face sentiment inference

Download the specified DistilBERT sentiment model and run a background Flask API that returns positive and negative confidence scores.

Terminal-Bench2.1 · terminalHugging Face TransformersPython · JSON74/10074%
Run Windows 3.11 in a controllable QEMU VM

Boot the supplied Windows 3.11 image in snapshot mode with VNC, an Nginx web interface, and a monitor socket for programmatic keyboard input.

Terminal-Bench2.1 · terminalQEMU 5.2.0disk image41/10041%
Run a gRPC key-value service

Define, generate, and run a Python gRPC service that stores integer values under string keys and exposes GetVal and SetVal RPCs.

Terminal-Bench2.1 · terminalgRPC 1.73.0Python · Protocol Buffers52/10052%
Transform a million-row CSV with Vim macros

Create a restricted Vim script containing three compact macros that transforms input.csv to the expected file byte for byte.

Terminal-Bench2.1 · terminalVimVim script · CSV95/10095%
Accelerate dominant eigenpair computation

Implement a faster Python entrypoint for the largest-magnitude eigenpair of small real nonsymmetric matrices while retaining complex-valued correctness.

Terminal-Bench2.1 · terminalNumPyPython80/10080%
Optimize shape-aware LLM inference batches

Pack every request from two buckets into static-shape batches that obey alignment and shape limits while meeting cost, padding, latency, and time thresholds.

Terminal-Bench2.1 · terminalLLM inferenceJSON Lines · Python87/10087%
Summarize log severities by date range

Count ERROR, WARNING, and INFO events across five fixed date ranges from date-stamped logs and emit rows in the required CSV order.

Terminal-Bench2.1 · terminalCSVCSV94/10094%
Configure a confirmed-opt-in Mailman list

Configure Postfix and Mailman 3 for a local reading-group list with confirmed join and leave messages and subscriber announcements delivered to Unix mailboxes.

Terminal-Bench2.1 · terminalMailman 3INI · email78/10078%
Cross-compile DOOM for a MIPS emulator

Build the supplied DOOM source into the expected MIPS ELF so vm.js runs it and the custom frame writer produces bitmap frames.

Terminal-Bench2.1 · terminalDOOMC · JavaScript1/1001%
Implement a MIPS interpreter that boots DOOM

Implement a JavaScript MIPS ELF interpreter with system calls and file access that boots the supplied DOOM binary and saves rendered frames.

Terminal-Bench2.1 · terminalMIPSJavaScript · C51/10051%
Sample a hierarchical beta-binomial model with RStan

Implement and run the specified hierarchical Bayesian model in RStan, then save reproducible posterior means for alpha and beta.

Terminal-Bench2.1 · terminalRStan 2.32.7R · Stan · CSV90/10090%
Merge Git bundles and implement the inferred grid mapping

Import two bundles as named branches, merge them with conflicts resolved, and implement the grid transformation implied by the ARC-style examples.

Terminal-Bench2.1 · terminalGitPython · JSON95/10095%
Extract hidden weights from a ReLU network

Query a black-box one-hidden-layer ReLU function to recover its first-layer weight matrix up to neuron permutation and scaling.

Terminal-Bench2.1 · terminalReLU neural networksPython · NumPy binary format31/10031%
Modernize a Python 2 climate analysis script

Reimplement the legacy climate analysis for current Python using pandas and pathlib while preserving station coverage and formatted mean-temperature output.

Terminal-Bench2.1 · terminalpandasPython · CSV · INI100/100100%
Find the top Scandinavian MTEB embedding model

Identify the complete-coverage model with the highest Mean (Task) score on the Scandinavian MTEB leaderboard as of August 2025.

Terminal-Bench2.1 · terminalMTEBplain text68/10068%
Retrieve the fifth-nearest document with MTEB

Encode the query and line-delimited corpus with the pinned Chinese BGE model and return the document ranked fifth by cosine similarity.

Terminal-Bench2.1 · terminalMTEB 1.36.8plain text67/10067%
Merge prioritized user records across three formats

Normalize and merge JSON, CSV, and Parquet user data by user ID with deterministic source priority and a complete field-level conflict report.

Terminal-Bench2.1 · terminalJSONJSON · CSV · Parquet · Python99/10099%
Configure Nginx request logging and rate limits

Run Nginx on port 8080 with static content, detailed access logs, a custom 404 page, and per-IP request limiting.

Terminal-Bench2.1 · terminalNginxNginx configuration · HTML100/100100%
Generate and verify an internal TLS certificate

Create a 2048-bit RSA key and one-year self-signed certificate for the specified internal host, plus combined and human-readable verification artifacts.

Terminal-Bench2.1 · terminalOpenSSLPython94/10094%
Eliminate LaTeX overfull boxes with allowed synonyms

Replace words in input.tex only with their permitted synonyms until the document compiles without overfull hbox warnings.

Terminal-Bench2.1 · terminalLaTeXTeX71/10071%
Recover a deleted password from a disk image

Perform file-system forensics to recover candidate values from a deleted launchcode.txt and write every value matching the exact password pattern.

Terminal-Bench2.1 · terminaldisk forensicsplain text92/10092%
Reconstruct a path-traced image algorithmically

Write a compact standalone C renderer that reproduces the supplied PPM image with at least 0.99 normalized L2 similarity without reading it at runtime.

Terminal-Bench2.1 · terminalpath tracingC63/10063%
Recreate a compiled path tracer in C

Reverse-engineer the mystery executable into compact standalone C source whose isolated behavior and generated output are identical.

Terminal-Bench2.1 · terminalbinary analysisC59/10059%
Write a Python and C Fibonacci polyglot

Create one source file that runs under Python 3.12.3 and compiles under GCC 13.2.0 to print the requested Fibonacci number.

Terminal-Bench2.1 · terminalPython 3.12.3Python · C98/10098%
Write a Rust and C++ Fibonacci polyglot

Create one source file accepted by Rust 1.75.0 and G++ 13.2.0 that prints the same shifted Fibonacci sequence value.

Terminal-Bench2.1 · terminalRust 1.75.0Rust · C++83/10083%
Accelerate portfolio calculations with a C extension

Complete the Python extension so portfolio risk and return match the baseline within 1e-10 and run at least 1.2 times faster at scale.

Terminal-Bench2.1 · terminalPython C APIC · Python97/10097%
Design a FRET fusion-protein gBlock

Design a codon sequence for the prescribed five-part fusion protein with exact spectral choices, linker ordering, binding functions, and synthesis constraints.

Terminal-Bench2.1 · terminalFRETDNA · FASTA · GenBank30/10030%
Complete a Coq proof of addition commutativity

Fill the missing inductive proof steps for natural-number addition commutativity and compile the verified theorem.

Terminal-Bench2.1 · terminalCoqCoq99/10099%
Host an installable vectorops package

Create vectorops 0.1.0 with a dot-product API and host it on a local simple-package index reachable by pip on port 8080.

Terminal-Bench2.1 · terminalPython packagingPython58/10058%
Run MNIST PyTorch inference from a C CLI

Convert the supplied PyTorch model weights to JSON and implement a command-line executable that predicts one MNIST digit from an image.

Terminal-Bench2.1 · terminalPyTorchC · Python · JSON84/10084%
Recover and tune a PyTorch model architecture

Reconstruct the network represented by a state dictionary, freeze all but output_layer, improve dataset MSE, and export the result as TorchScript.

Terminal-Bench2.1 · terminalPyTorchPython82/10082%
Boot Alpine in QEMU with SSH access

Run the supplied Alpine ISO in QEMU and configure a password-authenticated root SSH shell forwarded to localhost port 2222.

Terminal-Bench2.1 · terminalQEMUISO disk image58/10058%
Expose an Alpine QEMU console over Telnet

Boot the supplied Alpine ISO in a background QEMU process and expose its ready login console over Telnet on localhost port 6665.

Terminal-Bench2.1 · terminalQEMUISO disk image74/10074%
Optimize an Open English WordNet SQL query

Rewrite the supplied SQLite query for maximum efficiency while preserving its exact result against the immutable OEWN database.

Terminal-Bench2.1 · terminalSQLiteSQL56/10056%
Fit Raman G and 2D peaks

Fit Lorentzian models to the graphene spectrum's G and 2D peaks and export each peak's center, width, amplitude, and offset.

Terminal-Bench2.1 · terminalRaman spectroscopyJSON · Python17/10017%
Generate legal chess positions with regex substitutions

Encode a complete white move generator as ordered regex replacements that transforms FEN into every legal successor position.

Terminal-Bench2.1 · terminalregular expressionsregular expressions · JSON · Python51/10150%
Match the last valid date on IPv4 log lines

Write one Python-compatible multiline regex that returns only the last valid date from each line containing a boundary-safe IPv4 address.

Terminal-Bench2.1 · terminalregular expressionsregular expressions · Python86/10086%
Reshard and restore C4 dataset directories

Implement reversible dataset resharding that caps directory fanout and file size, recreates the original tree exactly, and runs through a locked uv environment.

Terminal-Bench2.1 · terminalC4 datasetPython · TOML99/10099%
Port an RStan Gaussian process to PyStan

Translate the supplied RStan Gaussian-process analysis to PyStan 3.10.0 with equivalent model, data, hyperparameters, sampling, and posterior summaries.

Terminal-Bench2.1 · terminalPyStan 3.10.0Python · R · Stan · CSV · JSON92/10092%
Refine cell boxes into MobileSAM contours

Implement a CPU MobileSAM pipeline that converts every rectangular or polygonal cell annotation into one nonoverlapping contiguous polyline and updates its geometry fields.

Terminal-Bench2.1 · terminalMobileSAMPython · CSV88/10088%
Replace secrets in the dclm repository

Find API credentials in the repository and replace each with a consistent service-specific placeholder without changing uncontaminated files.

Terminal-Bench2.1 · terminalGitGit repository format48/10048%
Implement a self-interpreting Scheme evaluator

Write a Scheme metacircular evaluator for the supplied language that runs all programs, forwards remaining input and output, and can interpret itself recursively.

Terminal-Bench2.1 · terminalSchemeScheme · Python70/10070%
Query eligible EU university professors

Write a SPARQL query returning full professors who currently work in an EU university department with more than ten enrolled students.

Terminal-Bench2.1 · terminalSPARQLSPARQL · Turtle96/10096%
Recover rows from a truncated SQLite database

Recover as many records as possible from the binary-truncated SQLite file and serialize each word and numeric value to JSON.

Terminal-Bench2.1 · terminalSQLiteJSON99/10099%
Build gcov-instrumented SQLite

Compile the pre-vendored SQLite source with gcov instrumentation and make the resulting command-line tool available on PATH.

Terminal-Bench2.1 · terminalSQLiteC92/10092%
Implement AFAB pipeline parallelism for LLaMA

Implement one LLaMA training step that balances layers across ranks and executes all microbatch forwards before all backwards with correct distributed tensors.

Terminal-Bench2.1 · terminalPyTorchPython38/10038%
Implement row and column tensor-parallel linear layers

Implement distributed PyTorch linear layers with correct master-weight sharding, collective-style outputs, zero bias initialization, and gradient behavior.

Terminal-Bench2.1 · terminalPyTorchPython75/10075%
Train a compact Yelp FastText classifier

Train a FastText text classifier on the supplied Yelp reviews that exceeds 0.62 held-out accuracy while keeping the model under 150MB.

Terminal-Bench2.1 · terminalFastTextFastText binary format15/10015%
Accelerate a MuJoCo model without changing physics

Tune the MJCF model so a two-second simulation runs in at most 60 percent of the reference time while reaching the same finite physics state.

Terminal-Bench2.1 · terminalMuJoCoXML · Python90/10090%
Detect hurdle takeoff and landing frames

Implement an OpenCV script that analyzes a stationary-camera hurdle video and records the single jump's takeoff and landing frame numbers in TOML.

Terminal-Bench2.1 · terminalOpenCVPython · TOML28/10028%
Extract a hidden flag from a vulnerable executable

Analyze and interact with the supplied executable to recover its FLAG-formatted secret and save the exact value.

Terminal-Bench2.1 · terminalbinary analysisplain text94/10094%
Write a Core War warrior with target win rates

Create a Redcode warrior that meets separate 100-round win-rate thresholds against five supplied pMARS opponents without modifying them.

Terminal-Bench2.1 · terminalpMARSRedcode63/10063%
Create data for a custom decompressor

Reverse-engineer the supplied decompression format and produce a compact file that expands byte-for-byte to data.txt.

Terminal-Bench2.1 · terminalcustom compressionC80/10080%