Authors:Emmanuel Sérié (CMAP, CNRS, École Polytechnique, Institut Polytechnique de Paris)

View PDF HTML (experimental)

Abstract:Make-lineage workflow runners decide whether a job must re-run from file-modification time (mtime, a timestamp) -- a broken proxy for the question that matters: did the content change? A git checkout, a tree copy, or a backup restore rewrites mtimes without touching content, forcing spurious re-execution; and in the reverse case -- when an output looks newer than its inputs but its content is stale -- the stale output is silently reused. (Snakemake 7's per-output provenance survives this churn, as local bookkeeping; GNU Make and pure-mtime fast paths are where it bites.) OxyMake, a single-binary Rust workflow engine, replaces the proxy with a content-addressed cache key: a BLAKE3 hash of rule source, input content, parameters, environment, and platform. Because the key is a pure function of these declared inputs, the caching decision survives mtime churn and travels across same-platform machines and shared caches. Phantom re-runs vanish for declared inputs (no sandbox: an undeclared input is invisible to the key). The spec stays declarative and statically parseable, keeping the Make rule model so Snakemake pipelines port directly. DAG resolution is an order of magnitude faster than Snakemake's on large graphs, but a cold end-to-end run is slower -- the price of content-addressed bookkeeping -- repaid several-fold on the warm re-run that caching exists to serve (exact figures, hardware, and a bundled reproducer are in the evaluation). Execution is daemon-free via a cooperative claim/reclaim protocol (sessions claim jobs, reclaiming stalled ones); today two sessions duplicate work safely rather than coordinate, and wiring the protocol as a hard execution gate is staged, not yet done. Cross-session safety is specified in TLA+ and model-checked over all interleavings for 2-3 sessions, assuming atomic state commits. An this http URL plan-of-record and NDJSON event stream record exactly what ran.

Submission history

From: Sérié Emmanuel [view email]
[v1] Thu, 18 Jun 2026 23:45:52 UTC (108 KB)