A developer building a debugging tool called Rewind VM has found that many of the hardest engineering problems in building a debugger were already solved by Nix, the package manager and build system known for reproducible builds. The result is a deterministic virtual machine that can replay a program’s execution, including its exact thread scheduling, making notoriously hard-to-reproduce bugs like race conditions far easier to catch and inspect.

This matters because race conditions are among the most frustrating bugs in software: they often vanish when you try to reproduce them, appear only under specific timing, and are nearly impossible to debug with traditional tools. By combining a deterministic VM with Nix’s existing infrastructure for tracking dependencies, source code, and debug symbols, the project author built a surprisingly capable debugger without writing most of the underlying plumbing from scratch.

What Rewind VM actually does

Rewind VM is a virtual machine designed so that every run of a Nix build is a pure function of its inputs, down to the exact thread schedule. In a normal multi-threaded program, the order in which threads run can change from one execution to the next, which is exactly why race conditions are so hard to pin down. Rewind’s VM runs on a single CPU and can replay a program step by step, letting a developer rewind and inspect exactly what happened at any point in execution.

The tool has grown features quickly: a source panel, stack frame inspection, bookmarks, a “Compare” tab for diffing runs, expanded support for the GNU Debugger (gdb), “thread lanes” that visualize which thread held the CPU at each step, and a “Check from here” feature that narrows down the exact step where a race condition occurs.

Why a race condition is a good test case

The article illustrates the problem with a classic example: two threads depositing money into the same bank account. Each deposit reads the current balance, writes a line to a ledger, then stores the balance plus the deposit. If one thread runs in between another thread’s read and write, it overwrites the balance with stale data, silently losing money.

According to the author, on a 16-core laptop this bug caused the simulated bank to lose money in 396 out of 1,000 runs. When the program was pinned to a single core using the taskset utility, it failed in none of 1,000 runs, because a single core rarely switches threads at the exact wrong moment. This is precisely why Rewind deliberately perturbs the thread schedule: its rewind check command reruns a build under different simulated schedules, asking the guest kernel to reschedule threads at different points, until it isolates the exact step where the failure first occurs. In the author’s test, this check took 11 seconds and narrowed the divergence between a passing and failing run down to a single step, step 3237.

How Nix does the heavy lifting

The central claim of the article is that Nix already solved most of the hard problems a debugger needs to solve. A debugger needs the exact inputs of a program, its debug symbols, its full source tree including every library it depends on, and a reliable way to hand all of that to someone else’s machine. That is essentially what a Nix derivation already provides.

The rewind nix command takes a derivation, which can be a flake reference, and uses Nix’s knowledge of that derivation’s inputs, including source code, compiler, libraries, kernel, and VM configuration, to assemble everything needed for a reproducible run. It packs this closure into a read-only erofs image and boots the VM from it. Each run’s ID is a hash of its inputs, similar to how Nix store paths work, and the rewind show command can print the exact command needed to reproduce that run again.

When a build finishes, the guest VM reports a NAR hash for each output, and rewind nix checks that hash against both the host machine’s copy and any binary caches that Nix would substitute from, fetching only the lightweight .narinfo metadata to do so. This confirms that what ran inside the VM matches what Nix would have built on the host, and that the result is reproducible.

Source code and live debugging with gdb

Because nixpkgs builds packages with separateDebugInfo, debug information is cached separately on cache.nixos.org as a dedicated output. Rewind leverages debuginfod, an existing protocol for fetching debug info and source code by build ID, to display the source of any binary running in the VM, including the Linux kernel itself.

For deeper inspection, rewind gdb opens gdb on a fork of the recorded run at the current playhead position, with access to every thread in the process. From there, a developer can set breakpoints and watchpoints, and inspect memory, registers, and variables, without altering the underlying recording. In the bank account example, this lets a developer watch the balance variable at the exact step before a second thread regains the CPU, and step forward until the race condition’s effect becomes visible.

What this means for developers

The key insight is architectural, not just technical: rather than building a custom dependency-resolution and source-fetching system, Rewind VM treats Nix derivations as the natural specification of everything a debugger needs. This suggests a broader pattern for tool builders in the Nix ecosystem, where reproducibility guarantees that Nix already provides for builds can be repurposed for entirely different problems, like deterministic replay debugging.

For developers already using Nix or nixpkgs, this could mean lower-effort access to powerful debugging capabilities, since much of the setup Rewind needs, such as debug symbols and cached sources, may already exist for packages they use. For teams outside the Nix ecosystem, the caveat is that replicating this approach would likely require adopting Nix-style reproducible builds first, since the article’s author is explicit that Nix’s existing infrastructure, not new code, is what made these features tractable to build quickly.

Frequently asked questions

What is a race condition? It is a bug that occurs when the behavior of a program depends on the unpredictable timing or ordering of multiple threads, often causing data to be lost or corrupted only under specific, hard-to-reproduce conditions.

Why does pinning to one CPU core change the bug’s frequency? With only one core, the operating system rarely interrupts a thread in the exact middle of a sensitive operation, so the specific interleaving that triggers the bug almost never happens, even though the underlying flaw is still present in the code.

Do I need to use Nix to try something like this? Based on the article, Rewind VM’s integration depends heavily on Nix derivations, debuginfod, and nixpkgs’ debug info caching, so getting the full benefit described would require working within or alongside the Nix ecosystem.

Source: Nix wrote half of my debugger