I don't know if you've seen this, but a CA just got popped. Their keys got leaked, which means the industry as a whole now has some pretty major problems. It was revealed two hours ago.

Now you're probably wondering why I'm writing up this blog post. It's simple. This is perhaps one of the easiest security patches I've ever done, and I've discovered a few new tricks. First, understand that fixing this with ease without Nix is almost impossible. Tens of millions of dollars (if not more) in productivity are being lost worldwide right now as everyone tries to patch their transitive dependencies.

Meanwhile, because I've used Nix, I can patch the planet, since the ideas behind the blog post below extend into security as well.

Because of Nix's single source of truth and the ability to patch this single source of truth, I'm able to issue a simple prompt like this:

okay is my nixos for my fleet affected by https://emudhra.we-have-a-problem.com/?

With results in the following.

So the problem here is the transitive dependencies. It's easy to patch one thing, but the transitive dependencies are the problem, and this is where Nix shines: one fix fixes the transitive dependencies automatically.

One patch fixes the entire fleet and its transitive dependencies. Patch, commit, and autodeploy your fleet 🍻

the new trick

Alright, I promised a new trick. Here's the trick.

You can prompt an agent to update the AGENTS.MD to automatically back out these patches, let's say in five days from now, just by using the AGENTS.MD as a reminder 😎

So it was one simple prompt to identify if I was affected, which turned into a plan, which then automatically fixed it, and now, in a couple of days, it will reassess the situation, give time for the ecosystem and dust to settle, and then automatically back it out once all the upstream transitive dependencies have been resolved.

Cool, huh?

## Temporary reminder: eMudhra root blacklist (from 2026-10-22)

On or after 2026-10-22, check whether the `nixos-26.05` channel contains nixpkgs
commit `7816d39` (the backported `security/ca.nix` distrust of the eMudhra roots), e.g.
`nix eval --raw --refresh --impure --expr '(builtins.getFlake "github:NixOS/nixpkgs/nixos-26.05").rev'`
then `git merge-base --is-ancestor`. If it does not, check again two days later.
Once it does, back out the fleet-wide workaround: update the `nixpkgs` pin to include
it, delete `nix/domains/system/public-trust.nix` (contract `fleet-distrust-emudhra-roots`)
and its imports in `nix/domains/system/catalog.nix`,
`nix/domains/artifacts/guest/configuration.nix` and the `host-kexec` modules in
`flake.nix`, and remove the eMudhra paragraph from
`marshall/priv/hldd/12-configuration.md`. Then verify that the built host and guest
bundles still lack the four roots, and delete this section. Keep Marshall's
system-store TLS defaults (`runtime-postgres-system-trust`, `mail-smtp-system-trust`)
and the `rustls-webpki` update; they are not part of the workaround.

I can't stress this enough. If you build something, you own it, including all the dependencies you rely on. We're entering weird times where cyber offensive models will be continuously on the attack.

The game has changed. To defend, you need more inferencing capability than the attacker, and you also need to respond rapidly.

This is by far the easiest security patch I've ever done, and the best part is I won't carry this tech debt for long because an agent will automatically back it out in a couple of days.

ps. socials

The advent of large language models has changed the economics of closing gaps in software stacks.

This is in the nixpkgs manual, in the Ruby section, under “Platform-specific gems” [ref]. The pull request (PR#68) it points to was opened in October 2019 and closed, unmerged, in August 2026, with: “Closing as stale. I no longer work with bundix. Feel free to take it over.” [cite]

Aside There are alternatives that have sprung up but they all have unique gaps or are not widely communicated as blessed solutions. The manual itself also does recommend a workaround for the issue, but I found it to not work in practice.

Every ecosystem has gaps everyone has learned to step around. They survive because there is just too much stuff to do. We live with these warts and gaps because the cost of fixing them is either too high or there are more important things to do.

That economics is gone.

We can have and expect amazing perfectly correct software stacks, because the cost of fixing gaps is now low enough that it is worth doing. The economics of software have changed.

The multiple days of typing are now an afternoon, or less, of directing and reading to an AI agent. In the case of Nix and Ruby, I wrote the Nix side of a Bundler lockfile solution, gemlock.nix, within an hour, removing days of misery I hit while running this blog.

While AI generated code may suffer from slop, it is a growing skill how to wield AI to produce correct verifyable code. Nix is amazingly well-suited as it gives the agents a reproducible test-bed with which to verify the correctness of the code they produce.

The economics of software have changed, and we can now expect to have no quarter for gaps in our stacks.

One of the big ideas I think the Nix community should pursue is relocatable binaries. I put my money where my mouth is and collaborated with the VFS Linux maintainer Christian Brauner to get $ORIGIN support in PT_INTERP and the shebang via EBPF 🐝. That work is in Linux 7.3 which is about to be released.

Despite the amazing potential this unlocks, I found few to be as enthusiastic as I was. Instead of excitement, I was met with skepticism: “What’s the point beyond it being a cool technical trick?”

First off, “The pursuit of excellence does not need justification.” by Mitchell Hashimoto should be enough.

There is however a very practical reason to pursue relocatable binaries: it allows Nix to be a package set for other build systems, specifically Bazel and buck2. The two frontends can share the same store, and the machine doing the compiling does not need to have Nix installed.

The wall

The integration of Nix and buck2 or Bazel is not novel or new. Many have tried and most have failed, or have seen mixed success.

rules_nixpkgs

Brings nixpkgs toolchains to Bazel and has been incompatible with remote execution since 2022.

buck2.nix

Brings nixpkgs toolchains to buck2 as symlinks into the local store, which works on exactly one machine.

Snowydeer

Imports buck2 outputs into Lix as content-addressed store paths, scanning the NAR for hash parts to recover the references.

The wall they all face is that a nixpkgs output is pinned to /nix/store by three absolute strings: the RUNPATH the loader reads, the PT_INTERP the kernel reads, and the symlinks that point at siblings.

A remote executor has none of those paths. you either ship the whole closure inside the executor image, mount the store over the network, or chroot every action into a fake root.

They all work but have unfortunate side effects. A new compiler version means a new executor image. An NFS mount means a shared filesystem under a system whose entire design is that it does not need one. Requiring chroot for a worker means that Nix is now a leaky abstraction.

Relocatable binaries gets rid of the wall by circumventing the problem entirely. 🤯

A closure is a layout, not a location

Absolute paths are not necessary and we can make them relative. It requires work to find all the places a path is baked in, but this is the plight of a Nix enthusiast: we patch and fix the world.

RUNPATH

Already supports relative paths; the loader expands $ORIGIN to the directory of the executable.

PT_INTERP

Linux 7.3 supports a binfmt_misc handler written in BPF that replaces $ORIGIN in the interpreter to the directory of the executable.

When we make all absolute paths relative, a beautiful thing happens: the directory nix/store/<hash>-name stops being an address and becomes a shape.

We can put the shape under /, you have a Nix store. If we put it under buck-out/v2/gen/toolchains/…/closure/, you have a toolchain artifact. If we put it under a remote worker’s input root, you have an action that runs.

It is the same beauty of what made Nix hermetic but taken to the next maximum: the shape is now relocatable. The same bytes can be in many places at once, and the loader and kernel do not care which one they are in.

Two frontends

Once a closure is a shape, the two build systems stop competing for the same job. 👏

Everyone loves Nix for its replicability but are frustrated by the needless rebuilds due to the input-address (extensional) model.

On the other hand, tools like buck2 and Bazel only allow content-addressed (intensional) inputs. Coupled with a fine-grained action graph, this allows for incremental builds and remote execution, but they don’t truly model the world of packages and their dependencies, which is where Nix excels.

Each has spent years badly imitating the other. Buck2 users hand-roll toolchain downloads; Nix users wait for a whole derivation to rebuild because one file changed.

What if Nix and buck2 could natively share the same store?

digraph two_frontends {
  rankdir=TB;
  node [shape=box, style=rounded, fontname="Helvetica", fontsize=10];
  edge [arrowsize=0.6, fontname="Helvetica", fontsize=9];

  subgraph cluster_front {
    label="frontends"; style=dashed; fontsize=9;
    nix   [label="Nix\nnixpkgs, closures, NixOS, images"];
    buck2 [label="buck2\ntargets, actions, DICE, BXL"];
  }

  store [label="relocatable, content-addressed store paths\n<root>/nix/store/<hash>-name", style="rounded,bold"];

  subgraph cluster_exec {
    label="execution"; style=dashed; fontsize=9;
    cas     [label="REAPI CAS + action cache"];
    workers [label="REAPI workers"];
    local   [label="buck-out on a laptop"];
  }

  kernel [label="Linux 7.3: binfmt_misc bpf handler\n$ORIGIN in PT_INTERP, loader substitution"];

  nix -> buck2   [label="closures as tree artifacts"];
  buck2 -> nix   [label="outputs as store paths"];
  nix -> store;
  buck2 -> store;
  store -> cas;
  store -> workers;
  store -> local;
  workers -> kernel;
  local -> kernel;
}

Nix could provide buck2 a closure, and buck2 would treat it like any other input. On the way out, buck2 hands Nix the name of an output, and Nix fetches it like any other store path.

The “store” in the middle is not a daemon or a filesystem. It is an agreement about layout. One build, seen through that agreement:

I have a proof-of-concept implementation of this idea, and it is surprisingly small. The BUCK file is the whole user-facing surface. The gcc closure is one buck2 tree artifact provided as a toolchain.

load("@rules//:closure.bzl", "nix_closure")
load("@rules//:cc.bzl", "cc_binary", "nix_cc_toolchain", "run_remote")

# The gcc closure, pinned by hash, as one tree artifact.
nix_closure(
  name = "closure",
  roots = {
    "gcc": "/nix/store/5q6bdx…-gcc-16.2.0",
    "glibc": "/nix/store/h4wfwi…-glibc-2.44-25",
    ...},
  # every store path the roots need, 17 of them
  closure = [...],
)

nix_cc_toolchain(name = "cc", closure = ":closure")

cc_binary(name = "hello", srcs = ["hello.c"])

# Run the binary on the worker and keep what it prints.
run_remote(name = "hello-report", exe = ":hello")

Every one of those paths is a projection of the closure artifact rather than an absolute /nix/store string, so buck2 can ship the closure wherever the compiler runs. This is most easily seen in the PT_INTERP and RUNPATH strings provided by the linker:

-Wl,--dynamic-linker=$ORIGIN/../../h4wfwi…-glibc-2.44-25/lib/ld-linux-x86-64.so.2
-Wl,-rpath,$ORIGIN/../../h4wfwi…-glibc-2.44-25/lib

hello is only told where to look within the closure, it has no idea it was a /nix/store to begin with.

The sample program hello.c reports its environment to drive home the point.

// Where did the kernel find me?
readlink("/proc/self/exe", exe, sizeof(exe) - 1);

// Which libc did the loader actually map?
dl_iterate_phdr(report_libc, out);

// Does the store path gcc was told about exist on this machine at all?
access(NIX_GCC, F_OK);

The worker is a NixOS VM running NativeLink on a 7.3 kernel with the EBPF binfmt_misc handler registered at boot. Its store is an erofs image of its own closure: 441 paths but crucially gcc is not present: ls /nix/store | grep gcc comes out empty.

$ buck2 run //:hello-report --show-output
Commands: 3 (cached: 0, remote: 3, local: 0)
BUILD SUCCEEDED

hello from a relocated store path
kernel:         Linux 7.3.0-rc7
hostname:       nix-origin-worker
/proc/self/exe: /var/lib/nativelink/work/8f7d…/buck-out/v2/gen/root/…/__hello__/root/nix/store/hello/bin/hello
libc:           /var/lib/nativelink/work/8f7d…/buck-out/v2/gen/root/…/__hello__/root/nix/store/hello/bin/../../h4wfwi…-glibc-2.44-25/lib/libc.so.6
/nix/store/5q6bdx…-gcc-16.2.0 on this machine: absent

Back into the store

Getting a buck2 output into the Nix store is where the two frontends have to agree on a name.

buck2 already has one. Every artifact it produces is identified by a digest, a Merkle hash over the output tree, and the CAS is keyed by it. Nix already supports a few ways of addressing a path by its content. Add the buck2 digest to that list and the two systems name the same bytes with the same number.

The cc_binary rule already shapes its output like mkDerivation would, nix/store/<name>/bin/hello. A buck2 output’s store path is therefore /nix/store/<digest>-hello, and both sides compute <digest> without asking the other.

The rule also knows the references, because it put glibc in RUNPATH. It writes them in a small file next to the output, together with the path and the digest. That file is the narinfo. 🤯

The way back to the host store is a substitution. The CAS is configured as a substituter, the same way cache.nixos.org is:

substituters = reapi://cluster:50051?instance=main https://cache.nixos.org

On the Nix side a buck2 target is a derivation that we reference by its digest.

packages.hello = buck2.output {
  target = "//:hello";
  # the store path is /nix/store/<digest>-hello
  digest = "sha256-…";
};

Evaluation never runs buck2: nix build asks the substituter for the pinned path and the tree streams out of the CAS, references included:

$ nix build .#hello
copying path '/nix/store/<digest>-hello' from 'reapi://cluster:50051'...
$ nix path-info --json ./result | jq -r '.[].references[]'
/nix/store/h4wfwic161kxrr74jlzla5lsm28hgary-glibc-2.44-25
/nix/store/j7qx4s4mr17j1wqgvqdzj33lmrnzb387-gcc-16.2.0-lib
$ ./result/bin/hello
argv[0]:  /nix/store/<digest>-hello/bin/hello
libc:     /nix/store/<digest>-hello/bin/../../h4wfwi…-glibc-2.44-25/lib/libc.so.6

Nothing was rewritten, rebuilt or re-hashed on the way back. The shape of the closure survived because it only ever used relative paths. The bytes the worker produced are the bytes Nix serves, under the name buck2 already knew.

The two stores become one. 🫂

A cc_binary and a mkDerivation end up as the same kind of object: a tree named by its contents, with relative paths to its neighbours. Which frontend wrote it stops mattering once it is in the store.¹

Two frontends, one store.

Patch the world

We can think big for how we want to build software. Nix affords us a world where we can make software fit beautifully together. Nix is the one ecosystem where patch the world is a reasonable plan. We can change the kernel, the loader, the compiler wrapper and the package set in a single commit.

Let’s not have static paths in the store limit our possibilities. The valhalla of incrementality awaits and we can maybe achieve it without dynamic derivations.

The yellow brick road to a two-frontend store is paved and clear:

1. nixpkgs: write relative RUNPATH. #534339 is the opt-in version; the default is the goal. The hash stays in the string so no cache is invalidated and the reference scanner keeps working.
2. nixpkgs: a location-relative cc-wrapper. The wrapper and bintools-wrapper embed absolute paths in shell. Deriving them from $(dirname "$0") turns the toolchain into something we can cp.
3. nixpkgs: relocatable = true, enforced. A fixup hook that fails any output claiming the flag while still containing the literal store prefix, in ELF or in text.
4. NixOS: the handler at boot. Once 7.3 reaches nixpkgs, a module that registers the BPF handler, gated on a PT_INTERP_NIX marker so nothing changes for binaries that did not ask.
5. Nix: one hash. A content-addressing method that matches an REAPI directory digest, so a buck2 output in the CAS is a /nix/store path.
6. Build systems: closures as artifacts. The buck2 rules in this post are a few hundred lines. The Bazel equivalent is a repository_rule away, and it would be the first rules_nixpkgs that works with remote execution.

A store path that can live anywhere is a store path everyone can use. Let’s go (re)write some strings. ✍️

by now, most people should be aware that we're in a cost of compute crisis. The cost of CPU and memory is through the roof. There just isn't enough. Meanwhile, there definitely aren't enough GPUs in the world.

If you've been using ChatGPT lately, you might have seen the meme circulating in the space that OpenAI just doesn't have enough inferencing capacity. Heck, even Tibo admits publicly that they're having compute issues.

In the last couple of weeks, I have lost so much time babysitting agents because of inferencing capacity issues, and it's got me wondering: what if inferencing capacity constraints were region-based? Last year, before this crunch, even with Anthropic models, inferencing was noticeably faster in Australia because everyone else in America was sleeping.

Day to day, I do all my open-source software development on bare-metal machines in Germany, and I've started to notice similar patterns when Europe wakes up: all of a sudden, inferencing performance plummets and I'm being greeted with walls of 'model is at capacity, try again' errors.

Above is what I woke up to this morning. Imagine my dismay at losing time on a strict get-overalls-done-and-released-before-my-conference-talk timeframe. You can complain online all you want, but we're software engineers. We can solve these problems.

underclass

The first piece of the puzzle is a piece of software that I wrote last month called Underclass. Underclass pools many ChatGPT subscriptions together and works as a proxy. It can automatically claim resets and correctly handles the prompt cache key to ensure usage isn't wasted.

I haven't written about it before, but it is now a month old, and I consider it stable. It's available on GitHub, and I use it with six ChatGPT 500 accounts, but there's no real upper limit on how many accounts you can configure.

Installation is easy, and it's well documented. It's also available as a Nix Flake.

preflight

Another piece of software I wrote last month is a local LLM proxy that scans requests and attachments for secrets and proactively scrubs and redacts them before they hit the model. It's on GitHub (below)

To use Preflight, run a single instance of Preflight and one instance of Underclass, then point Underclass to use Preflight. Preflight scrubs requests and sends them out for inference, ensuring sensitive material never reaches the inference providers.

unburying the lede

By now you're probably wondering either, "Wow, that's two really cool projects," or why I opened with geographic routing. It's because... today I'm releasing a new experimental prototype called geoclass which routes your requests to different inferencing regions around the world. Like underclass and preflight, it is composable. You can use all of them, none of them, or any combination you can imagine, since they are separate pieces of software that compose well together.

How it works is simple.

Set up a Tailscale network and deploy virtual machines around the world, then tag those virtual machines with a Tailscale tag; GeoClass can then identify these machines via the tag. From that point forward, all you need to do is point UnderClass to GeoClass, and the rest just works (in theory!).

I don't know yet if geoclass will solve my inferencing woes, but there's also no way in hell I'm going to sit here and stew when the performance of OpenAI in Europe has been miserable over the last 30 days.

So I'm going to try experimenting with geo-routing. If the below works for you, let me know; I'm keen to catch up with folks thinking along the same wavelength. You never know until you try!

ps. socials

people are failing a curiosity test right now. Armin Ronacher and Ben Vinegar had me on State of Agentic Coding to talk about it, along with DHH coming around, durable agents, a Lisp you program by talking to it, and why the labs cannot afford to fail.

Ben Vinegar (co-founder of Modem) and Armin Ronacher (Earendil, and the person most people know from Pi) run a podcast called State of Agentic Coding, and I was the guest for episode 11. Ben and I were in the same room in Toronto. I'd landed the day before after 22 hours of flying, I was there for two days, and then I was flying back to Sydney. Armin stayed in Vienna, sensibly.

We covered a lot of ground in an hour and a half. Here's the gist, written up properly, with chapter links at the bottom if you want to jump to a specific part of the conversation.

DHH did the funeral last week and he's right.

the curiosity test

Ben opened by asking what my contribution to the GDP of AI development has been from Ralph loops. The OpenRouter folks personally came up to me to say thanks for the uptick, so I know it has burned a lot of tokens. I published the research in July last year. When I first showed it to Cursor they thought it was garbage. Nine months later they'd implemented it, then Anthropic did it, then OpenAI did it, and now everyone's talking about loop engineering. Dex calculated it at about $10.42 an hour when you run it in the loop and it builds AFK overnight.

That weighed a lot on my shoulders when it first came out.

I've spent this year traveling the world giving talks, pleading with people to invest in themselves. My shortest take is this: software developers are failing a curiosity test. If you're displaced by AI as a software developer, it's self-inflicted, because you haven't been curious in a time of rapid transformation. As an employee, you trade time and skill for money. Employers' tastes are changing rapidly, but there's no reason you can't keep up with what's going on.

Just harshing everybody's vibe right out of the gate.

Ben pushed back here, and fairly. When people come into the industry they're young and have infinite time on their hands. Now many of them are in their thirties or forties and have racked up responsibilities. It's the same reason video games get harder to stay into as you age. It's hard to retool at this rate of change, and it's hard to feel optimistic when the thing at the centre of your income and your self-worth is in flux.

I concur. Dex Horthy has described my writing over the last two years as the depression era of Geoff: mourning the profession while pleading with people to be curious and to put themselves and their family unit first. This change has been forced on the world non-consensually. It displaces your identity. Are you a .NET developer? A Golang developer? A Python developer? It doesn't matter anymore. What matters is how good a developer you are, and whether you're using the thing employers now expect as the bare minimum.

Armin made a sharp observation: a lot of people in AI talk a very confident game externally, and that probably applies to us too. I'm straight up about this. I don't bloody know. Every time I discover something new, it strips me raw. You discover a new technique or capability, you thought you had a mental model of how it would play out, and then the game changes.

boom and bust

Ben, playing tour guide, pointed out that the closest US cities to Toronto are Buffalo and Detroit. Buffalo was once the third most populous city in the US, prosperous because of its rail access to the Great Lakes. It's now one of the most rapidly declining. Detroit is a later, similar story, tied to the auto industry. His point: there's a long history of people with hubris believing the good times will never end.

I've been through a few of these cycles, going back to the original dot-com boom and bust. Senior compensation went through the roof during the ZIRP era because, in the previous bust, we didn't develop enough talent. That created a scarcity of knowledge. Hold on, folks: the same cycle will repeat. CS enrolments are down; I remember seeing something like 22% in Singapore this year.

If you really love computers, you should be all in on loving computers right now. Everyone's having the time of their damn life. If you've stayed curious and kept your skills up to date, you can get placed anywhere right now. That means being able to stand at a whiteboard and explain how to build a coding agent as a sequence diagram, how inferencing works, and what a prompt cache is.

Armin had a slightly different read. Detroit not growing doesn't mean cars stopped being a thing; the industry went worldwide and commoditized. Software engineers got paid really well because we were scarce and our only competition was other software engineers. Automation didn't play a role. That's changing. He's absolutely convinced the exceptional role our profession has had over the last twenty years won't return in the same form. There will probably be a period with more demand for software engineers, but it will commoditize quickly. He's also curious what happens to San Francisco, a city that hasn't grown its housing stock, when the Anthropic IPO mints a pile of multi-millionaires competing for it.

I'm hoping for another PayPal-mafia-type thing. When people get minted, they invest in what they know.

dhh did the funeral

If you're in love with typing on a keyboard in an IDE, I'm sorry to say it, but DHH did the funeral last week, and he's right. You can still type code; you just do it in your personal workshop. Employers now want you to use the CNC machine and be very good with the CNC machine. They don't want artisanal woodworking anymore.

Ben summarised the Rails World keynote nicely. DHH walked through three generations of family pictures: a single drawing from his grandfather, a handful of photos from his childhood, and his kids with every second of their lives photographed. Portrait painting was an economic outcome. When photography made drawing reality uneconomic, painting went to impressionism and cubism: let's draw what isn't reality.

Armin had been on a panel at a Ruby conference in Austria earlier this year with Chad Fowler, who already said that the agent never writes Ruby for him anymore. When you sample which languages agents actually use, Ruby doesn't show up meaningfully. The online reaction to the keynote was largely dismissive, but the in-person reaction was much better received. Armin's point is that communities are disintegrating and reforming around the ideas that brought them together in the first place, even as the technology underneath changes. The Rails community is really a descendant of how DHH thinks software should be built; Ruby was just the tool.

So what's the point of a programming language or framework full of sugar designed for humans if humans aren't writing the code anymore? I've been chatting with the creator of Elixir, and he gets it. Language designers have dumbed down their languages to make them easy to learn for the human operator. All of that is falsified now that the agent writes the code. What does an agent need? What's the point of LSPs? Will we get new forms of interpretability, like an agent driving GDB, on a per-language basis? The languages won't go away, but the communities that swallow this tough pill and evolve will be the ones that survive.

Developers mostly speak to other developers. It's like a couple of cows in a field going moo to each other, completely missing that the cows are headed for the abattoir. Grim, I know. Stop being a cow. Learn a bit about business. Speak with founders and investors. Learn to be a product engineer. If you're working at a company that has banned AI outright, you should quit that company.

Armin said the strongest signal for him isn't his own belief; it's that the experienced, vocal holdouts are turning. DHH was dismissive early on. Notch was dismissive and recently turned. Ben noticed that DHH was transparent about having tried AI for design, pulled it back, and now landed firmly on the other side. It felt like an invitation: come join me. Armin's worry is that because DHH is polarising, the message may not land as widely as it should.

On where to find the signal: if you've been on X for the last two years, you're ahead of the game, because that's where all of this has happened. I don't love the platform; the number of AI reply bots is through the roof. But if you want the alpha, you have to push through. Go to the fediverse and talk about AI and you'll get your whole Mastodon instance banned. Bluesky has some high-alpha accounts. Unfortunately, the tech community has fractured along political lines, which means a lot of this isn't getting out.

Back in December 2024 I was writing that IDEs are dead as a product, and everyone called me crazy. I've taken a lot of unpaid time off work this year to give talks. You can't carry all that water yourself. The best you can do is this: if someone gets curious, help them get to their own moment. Find the thing that makes their mind explode, then ask them to show the next person. For me, that moment was seeing how easy it is to build a coding agent and realizing the entire game now is to build product recursively.

Ben brought up the Bob Dylan biopic with Chalamet. Dylan is folk royalty, goes electric, and the old caretakers practically call the police to shut it down while the younger audience wants the new stuff. Folk was one of the most popular genres of its time. The reason you know Dylan's name and not most of the folk scene's is that he evolved. Boy, we don't like change.

The one thing I wrote that really landed with juniors was "dear student, you're screwed". ThePrimeagen narrated it (thank you, Prime), and I wrote it as if I were giving career advice to my own kid at twenty. Now, at the AI Engineer World's Fair and in my travels, people come up and tell me they got a promotion because they turned up with the attitude and the skills. Don't thank me; you did the work. You were curious and you invested in yourself.

four months in 2026

Armin and the Earendil crew shipped Pi 1, and they also shipped something that's around 10,000 lines of code but took four months to build. Four months, in 2026. That's like a five-year project.

Armin's explanation is the bit I want everyone to hear. LLMs suck at the shape of a contract and the human feel of an API. You can express an idea really fast, but getting the right feel takes a lot of knowledge and experimentation. They solved the problem something like twenty times, each in about two hours, and none of those solutions felt right. The agent solves every problem in record time, but that doesn't give you a system with the right trade-offs. Finding the right trade-offs is what took four months.

The result is Pi Durable, which is how they now think an agent should be structured. You can shut down the harness, restart it, and it picks up exactly where it left off. That holds over the network, and through complex interactions such as sub-agents spawning and talking to each other.

Fourteen or fifteen months ago, Armin, Mario, Peter and others had a strong conviction that file systems and bash were the way to go, but nobody really knew how to build a harness around that. So everyone copied what Claude Code was doing, which is a very simple idea of a session: a transcript, a tool call, a tool result, in a loop. That model has stayed at the core of more or less every agent. At its simplest, all it does is automate copying and pasting.

But demands have changed. Background compaction means doing two things at once. A Slack experience means five people talking to the same agent, some in the same sub-thread and some branching off. That mapped incredibly poorly onto Pi's original design. Armin called it a little bit of cargo culting that we're all suffering from: we keep building on a basic primitive that causes pain everywhere. We don't notice, because an agent will happily build you a decent-looking agent, but under the hood it's messy, horrifying and not reliable.

this technology is moving really, really quickly and is held back by ideas about how it should work that are a year plus old.

Can I get that louder and printed on a shirt? Here's my hot take: your AGENTS.md and many of your skills should be deleted after every model release, or at least every couple of months, and redone. We carry so much that we think of as best practice, and we perpetuate it. Steering messages versus follow-ups were a UX paradigm; the loop underneath stayed the same. Async tasks for bash were one of the last big advancements. Go further back: why do we even have a shell? Because we used to have a human operator. If that need had never existed, would we have an operating system shell at all? That's the level at which we need to rethink the substrate.

a lisp you grow by talking to it

Ben asked what we'd actually thrown out lately, so I showed the thing I'd published that morning. We've raised two generations of software developers who have probably never seen Lisp and what it can do. Armin was very curious, because three months earlier he'd given a talk saying Lisp was the worst language for agents since they're terrible at counting closing parentheses. That's the old joke.

The usual mental model of a coding tool goes like this: you prompt it, it runs bash to look at your project, finds the file and uses the edit tool to change it. Then comes compilation, then CI/CD, then code review. What I showed throws all of that out.

I've connected an LLM to a running Lisp. I asked what functions were defined, and the LLM asked the running image. I asked for a string reverse function, typos and all, and it defined one directly into the image. Then string uppercase. Then a function to fetch a URL and parse the JSON. I can undo a change and roll back to the previous generation. There are only two tool calls in the inferencing loop: evaluate an expression, or define one as a function. I'm editing the Lisp image and its runtime environment through a chat. Once a function is defined, it's defined.

Now don't think of this as a chat interface. Imagine you're logged into your website and there's a secret little Hackers-style menu that only shows for staff. You don't load up an IDE. You pop a prompt and evolve the running program by chatting with it. What if that's how a user feedback request gets actioned?

Ben mentioned that the opencode folks had just written about how much work it took to make individual plugins hot reload, never mind be programmed live. Armin said that on a single-player level this is very much what made Pi interesting, since Pi can rebuild and reload itself. The challenge is that JavaScript is not a language built for hot reloading.

why lisp died

Ben's question was the right one: if this is so fantastic, why does it feel like you're digging up a grave?

Lisp has always been about these things. But you had to be a Lisp programmer to do Lisp. Now there's a natural-language interface that lets anyone do Lisp-type stuff. Armin gave the more precise answer. The problem with software is effects. That's partly why Pi Durable exists: if you shut down Pi and reload it, it doesn't come back exactly as it was, because not everything is serialisable. Lisp can serialise its entire state out more than most languages can, and functions are defined much like data.

Why did it go away? Lisp was slow unless you had a machine that really suited it, and it didn't survive the shift to the personal computer, where people wrote assembly. Armin's bluntest point: it's a horrible language to look at. It's awful for humans. Maybe it's great for LLMs. Many of our trade-offs as engineers exist because we're lazy and pick the things that let us stay lazy.

That's the Ruby thing again: designed to be beautiful and easy for humans, but less powerful. Paul Graham's Beating the Averages covers this with the Blub paradox.

I also want to give credit to Krzysztof Cieślak. He doesn't get credit for it, but he's the person behind what everyone now calls spec-driven development. He's at GitHub Next, he's one of the people behind Ionide for F#, and he published research back in the GPT -3 era asking whether Markdown could be the future of how we publish software. When I was doing my Ralph loops, I remembered his research and thought: the models got better, let's redo it. The rest is history. For the record, though: code is the spec, folks. Markdown is not a spec. As Armin put it, there are specifications and there are vibe specifications.

I have to say, Peter Steinberg (aka lobster man) is absolutely brilliant. He was so ahead of his time with OpenClaw. I remember when he first tweeted about it last year, and when it turned into utter lobster madness at the beginning of the year.

Heck, I even ran OpenClaw for a bit, but I couldn't really see the value because I didn't want to maintain it while the lethal trifecta problem remains unsolved.

Categorically, this is a billion-dollar problem, perhaps more.

So here's the thing. I kind of want openclaw, but I want it managed. For almost a year now, I've been using Interaction Labs Poke, because if there's going to be a problem with the lethal trifecta, I want to make that problem someone else's as much as possible. I want them losing sleep over the lethal trifecta, not me. It's not full protection, but knowing there's a product team paid to think about these issues and resolve them at scale puts me at ease.

That is, until last week, when Poteto girl nerd-sniped me on X and, in full disclosure, hooked me up with enough credits to help me reevaluate. I am now using Grokbot full-time instead of Poke (which sucks), and I've revisited my priors about the utility of an AI bot and who they are useful for.

I'm writing this blog post of my own free will in the hope that it will help change some other people's perspectives on the utility of these assistants and give people some ideas on how they can be used, but most importantly, to highlight the risks of using these types of tools.

In this post, I'm going to give a high-level overview of how I've automated some aspects of my life. Late to the game, I know, but with some things it's better to be cautious.

initial experiences

Okay, Grokbot is really weird, and I guess this applies to, like, even OpenAI's dots, even though I have not used it. One of the hardest things about it is how generic it is. Once you have it installed, it becomes a question of "now what?"

You need to be curious, have lots of imagination, and when you have an idea, just ask for it. From that point forward, it starts to unfold like a pretty flower. If functionality is missing, ask for it and turn it into a skill.

starter pack

If you're using Grok, install this skill pack first, and use it to create other bots. Every time you need a new bot, which is often, ask this bot to create one.

some ideas for bots

Here are the bots I've defined in the last week, which I'm getting a lot of utility out of:

Daily briefs and reporting

• Daily Update sends the 7am brief: calendar, emails that need a reply, and site traffic. Connectors: Google Calendar, Gmail, PostHog.
• Blog Pulse reports on the health of https://ghuntley.com/newsletter: signups, unsubscribes, churn, MRR, and traffic. Connectors: Ghost Admin API (direct read-only, no connector), Stripe (read-only), PostHog (read-only).
• Inbox Diet finds newsletters worth unsubscribing from and lists them for you to click yourself. This is by far one of my favorite bots. I have unsubscribed from so many newsletters via this bot, probably upwards of 300, that would be too hard to do by hand. Finally, my inbox is under control. Connector: Gmail.
• Family sends a digest of WhatsApp messages related to my kids, does reminders, and automatically extracts classwork and homework reminders from school emails. Connectors: Gmail, bidirectional access to my WhatsApp via the WhatsApp bot. A custom parser for Seesaw that can retrieve updates sent by my kiddos' teachers.
• WhatsApp answers lookups of my WhatsApp messages acts as a function pipe for bots. Connector: WhatsApp Web in a browser, with no connector behind it. The bot can tunnel to my Mac Mini, read the WhatsApp message database directly, and send messages, but it can't send messages from the Mac Mini. To do that, you need to authenticate in a browser session, which allows two-way communication.

Money, tax and legal

• PocketSmith keeps my spending categorized and my budget up to date. PocketSmith already has pretty good categorization rules built in, but they normally rely on the user to train PocketSmith. Here, I use an LLM to do the categorization automatically. It is good. Connector: none; it calls the PocketSmith REST API directly.
• Australian Tax continually prepares a rough first draft of my personal tax records for each financial year. Connectors: Gmail, plus the PocketSmith bot.
• Bank Details shares my bank details with your other bots when they need them. Connector: none; the details are stored as a secret.
• Conference Reimbursement is one of my favorite bots. Conferences normally work on the basis that they will pick up your flights once you've done the talk, which always carries this risk that you won't be reimbursed. If it's a professional conference, then you get paid up front, but sometimes the community conferences have cash flow problems because they haven't sold all their tickets yet. So normally it's a gentleman's handshake that they'll reimburse, but sometimes these conferences don't reimburse me. For example, right now I'm probably out close to $10,000 in travel expenses this year. When these situations arise, I ask Dr. Egg Bot to create a reimbursement reminder bot. That bot has access to my email, WhatsApp, and everything else, and it can automatically query PocketSmith to see whether the reimbursement has landed. If it hasn't landed, it will ask me for permission to send a WhatsApp message or an email to nudge people along. Connectors: Gmail, WhatsApp bot.

Life admin

• Kitten Vet tracks the kittens' vaccinations and vet admin and automatically reminds me of what I need to do for them. Connectors: Gmail, Google Calendar.
• Desires keeps my bucket list and "someday" list. Connector: Google Calendar and X. What's really cool is that sometimes I'll have a conversation out in public and genuinely want to catch up with someone, but I forget. So I can just prompt my desires bot to like add it to the particular category, either by searching X or dropping the exact link of the conversation.

Dev and infrastructure

• Mac Mini operates the Mac mini. It's a function pipe bot. Tailscale is installed on the Linux machine the bot runs on; it's on my Tailscale network and can tunnel its network connectivity out via my Mac Mini as needed to bypass any geo-blocks on data center IP addresses.
• Overalls manages my upcoming open-source project called Overalls (previously known as loom) and acts as a remote interface for dispatching agentic coding tasks in ephemeral Firecracker VMs hosted on my self-hosted infrastructure.

lessons learned

What you want to do is to create bots thinly. You don't want a fat bot. You want to create a bot per function. Think about the Unix command line, how composable that is. You want to create small bots that can call other small bots, then pipe them together in a function chain.

You can install software on your Grok bot. For example, I have Tailscale installed, and that's how I do all my tunneling.

the future

If the lethal trifecta is ever solved at a fundamental level, I'll move all this stuff into a self-hosted Open Claw, but not before. I get the value of OpenClaw, and they do a good job with security, but fundamentally, I sleep much better at night knowing there's a product team whose arse is on the line, with staff and a funded security organization with a security VDP program that loses sleep over these topics, so I don't have to.

All right, I'm going to open up a history lesson because it's been a couple of generations of software engineers since this essay, and I have to assume not everyone knows what a cathedral is. Everyone knows what a bazaar is because they're using it day to day with GitHub.

GitHub took the bazaar's ideas—a decentralized, open, self-organizing open-source project—and provided a workflow that lets anyone inspect or contribute source code via a pull request.

I've been thinking that there needs to be a new software development model. We're not going back to the cathedral, where a small, private, exclusive group builds the source code, with releases that are heavily planned, infrequent, and polished.

But it's clear we need something else now that AI is here. The GitHub bazaar model, even before AI, was already showing its cracks because it created consumer-based entitlements where everyone was nagging maintainers to merge something. It created really weird social-structure expectations: that someone who releases code has an obligation to maintain it and get paid nothing for the work.

But now that AI is here, the dams have well and truly burst.

For what's the point of reading a pull request that's been generated by code? All I want is to read and understand the core idea. If it's a good idea, I might incorporate it back upstream, but there should be no expectation that I will.

You see, open source software is open source by definition because the license says that it's as is, and that as is permits you to remix software.

But the problem has been that it's been perceived as too hard to remix software, but it hasn't been. It's just a knowledge gap issue that no longer exists now that agents are here. Nix has existed for a long time. If you haven't seen the article that I posted a couple of days ago about Nix overlays, you should read it.

So, I'm going to propose something that I hope helps other open source maintainers. This is how I'm thinking about my software now that I release it under open-source licenses.

The Remix Model

• Structure: Decentralized and compositional. Ideas spread through independent overlays rather than converging on a single repository and a single maintainer.
• Workflow: Publish source and intent under an as-is license. Others remix by writing a small Nix overlay that expresses the desired change once, then reuses that derivation across devenv shells, Docker images, NixOS systems and tests, CI, and agent sandboxes—without forking or maintaining a divergent artifact.
• Core Principle: Ideas over patches. The valuable unit is a change in behavior or architecture. Maintainers decide whether to incorporate it, reimplement it, or ignore it. The original author has no standing obligation to review, integrate, or maintain the change.
• Social Contract: The license already permits remixing without creating a claim on anyone’s future time. Credit and visibility can still exist; there is no ongoing maintenance duty. Upstream incorporation remains optional and deliberate.
• Discovery: Useful adaptations appear as a proliferation of overlays in the wild. Maintainers can sample from that spread if they choose; everyone else can grab and compose ideas simply by updating their own overlays via an agent.

Example: A consumer of a project wants to disable Git's ability to force push in that project. Instead of raising a pull request to the upstream project, the consumer makes an overlay that patches Git to refuse force pushes; this is defined once and then appears consistently in developer environments, container images, and system tests—without a pull request against the upstream project.

In practice, from this point forward, for any code I release on GitHub, I will disable pull requests, keep discussions open, and publish + maintain guidance on how you can remix my software.

The GitHub network graph provides a fantastic way to explore the remixes in the ecosystem. Heck, you can even run agents over it to summarize what people are doing. You can also use it in an agent loop to explore remix ideas and feed them back upstream into the main project.

ps. socials

Exit 0 means your program exited 0. That’s it. That’s all it means. 🍞

I have been working on a deterministic VM called Rewind VM, where “a run” (i.e. test, command etc.) is a function of its inputs including the thread schedule. The determinism comes from the tool recording every step which lets you replay it exactly. This alone is a pretty good way to find flaky tests: run the test under Rewind, then rewind check runs it under many different thread schedules, with some slight latency added, and failures often crop up. 🐛

Feedback however was that it was not enough: the tool could find a failing schedule, but it could not tell you if that schedule was actually a bug. It only knew that the run ended differently than the original schedule, which could be a crash, a timeout, or just a different exit code. That is not enough to say “this is a bug”.

Someone pointed out that I had built most of a property-based testing tool, but I only had one property: matching exit codes.

What is property-based testing?

If you have used QuickCheck or Hypothesis, you have already used the idea. You write a property, or constraint, that must hold for every input. The tool generates lots of inputs, and when one breaks the property, it shrinks it down to the smallest input that still fails.

rewind check already did the generating and the shrinking. The inputs are thread schedules, and narrowing shrinks a failing schedule down to the one step that causes the failure. The only property though was “the run ends the same way schedule 0 did”: the same exit code and the same outputs.

That’s a pretty low bar, but easy to improve. 🙃

Rewind now lets specify properties that must be true for every run, or must be true for at least one run, or must never be true in any run.

Two tellers: Java Edition ☕

Let’s keep the bank from last time: two tellers deposit into one account from two threads, without a lock.

This time the example is in Java as a JUnit 5 suite to appeal to my Enterprise brethren.¹

void deposit(int teller, long amount) {
    long seen = balance;
    System.out.println("teller " + teller + ": " +
                        seen + " + " + amount);
    balance = seen + amount;
}

A teller that runs between the other’s read and store writes a stale balance over the other’s deposit. 💥

We include a “typical” test suite: one teller, two tellers concurrently, and a new account being empty. We run it and see all tests pass and head home for the day. 🍺

Is the code correct? Let’s use Rewind to dig further.

Properties that come in the box

Before even looking at the balance, Rewind knows a lot about what a run did. Every open, write, rename, close, fork and exit is in the trace, with the step it happened at. Rewind ships properties computed from the trace alone:

Let’s throw a few at the bank and only run schedule 0:

$ rewind check --schedules 0 --property no-crash --property no-orphans \
    --property 'writes-within=/build' github:fzakaria/rewindvm#bank-java
schedule   0: exited:0     148479 steps  ba3f68ae3869  run bd3b527f0a837af8  violates writes-within
  writes-within: step 2710: process 111 (javac) opened /tmp/hsperfdata_nixbld/111 for writing, outside /build
never true in any run: sometimes "deposits-concurrent"
every schedule violated a property, as schedule 0 did

Interesting! javac writes its performance counters to /tmp/hsperfdata_<user>/<pid>². It has nothing to do with our bug, but I included it because I learned something new and it could have been a bug if it did not include the process ID.

The file properties shine for bugs between processes. The properties tutorial has a program that rewrites its config in place while another process rereads it. Its own test fails only when a read lands between the truncate and the last write, but atomic-replace flags it on the very first run. 😲

Asserting from the inside

The bank’s real invariant is not in any syscall though. It lives in memory in our example: the balance variable must add up.

For that there is the Rewind SDK, a tiny library you call from your program. Each call writes one line to /dev/rewind, a device in Rewind’s VM, and the trace keeps it with the step, process and thread that wrote it.³

// Account.deposit: another teller is mid-deposit.
int others = depositing.getAndIncrement();
Rewind.sometimes(others > 0, "deposits-concurrent");

// BankTest.twoTellersDeposit
Rewind.always(balance == expected, "balance-adds-up",
        () -> "balance " + balance + ", expected " + expected);

always must be true every time the program reaches it. unreachable is its opposite: a run that reaches it at all fails. These are your typical assertions, except we get the whole replay if it fails.

sometimes is the one I find the most interesting and tough to think through at times. 🤓 It tests the search: its condition must be true in at least one of the runs check tries, and it never fails a run. In model-checking terms, always is a safety property (nothing bad happens in any run) and sometimes is a reachability property (some run gets there). Here it asks whether any run had both tellers inside deposit at the same time.

Look at the output above again:

never true in any run: sometimes "deposits-concurrent"

The suite passed under schedule 0, but one teller always finished its deposit before the other started. That pass told us nothing about the race. 🤷

Let check try its default 64 schedules:

$ rewind check github:fzakaria/rewindvm#bank-java
...
true in 11 of 17 runs that checked it: sometimes "deposits-concurrent"
...
step 145417 decides it: a reschedule and a 10 µs stall there makes the run violate same-outcome, sdk

passing: run 61643f768ec51337, schedule 2 over steps 144694..145417
failing: run 643a298ef0ec3f7b, schedule 2 over steps 144694..145418
the two are the same run until step 145417
failing run violates same-outcome: step 149407: ended exited:2, where bd3b527f0a837af8 ended exited:0
failing run violates sdk: step 145713: always "balance-adds-up" failed: balance 380, expected 400

That took 40 seconds on my laptop⁴ and we found that two deposits went missing.

The first line is the sometimes again: of the 17 runs check tried before it found the failure, 11 had both tellers inside deposit at once.

Did I truly fix it?

The fix is the obvious one: hold a lock from the read to the store.

int others = depositing.getAndIncrement();
Rewind.sometimes(others > 0, "deposits-concurrent");
synchronized (lock) {
    long seen = balance;
    System.out.println("teller " + teller + ": " +
                        seen + " + " + amount);
    balance = seen + amount;
}
depositing.decrementAndGet();

$ rewind check .#bank-java
...
true in 42 of 65 runs that checked it: sometimes "deposits-concurrent"
every property held under all 65 schedules

Every schedule passes, but is that enough to make us feel warm and fuzzy?

The sometimes line is what gives me the warm embrace: in 42 of the 65 runs the tellers ran concurrently, but the lock prevented the bug. ✅⁵

A condition that is only true when the bug happens can never be true once the bug is fixed. 💡

JUnit

Java developers though love writing JUnit tests. Fear not my FizzBuzzEnterprise colleagues, Rewind has you covered: the SDK has a JUnit 5 extension.

Put @ExtendWith(RewindExtension.class) on the test class and it marks each test’s start and result.

@ExtendWith(RewindExtension.class)
class BankTest { ... }

If you can’t touch the code, junit=GLOB reads the XML reports the runner wrote instead: Maven’s surefire, Gradle, pytest’s --junitxml, whatever writes the format.

$ rewind check --property 'junit=TEST-*.xml' github:fzakaria/rewindvm#bank-java
...
step 145417 decides it: a reschedule and a 10 µs stall there makes the run violate junit, sdk
failing run violates junit: step 149407: test BankTest.twoTellersDeposit() passed in the unperturbed run and failed here
failing run violates sdk: step 145713: always "balance-adds-up" failed: balance 380, expected 400

Visualizing properties

The Rewind app shows the timeline of a run, with the tests and the build log. The failing test is red, and the event card names the assertion that failed.

$ rewind open 643a298e 145713 --compare 61643f76

The strip under the timeline has a span per test: green passed, red failed. Each include a helpeful tooltip with the test name and the steps it ran.

All the other Rewind features can then be used to help isolate the cause: the source panel, gdb on a fork at any step, the thread lanes and Check from here.

Easy Peasy ?

There is a SDK for many languages already included in the Rewind repository. They are tiny, and the only dependency is the Rewind VM itself. The SDK is a single file or two per language, and you can copy it into your project.⁶

• C and C++: rewind.h, REWIND_ALWAYS(cond, "id", "fmt", ...)
• Rust: rewind_sdk::always!(cond, "id", "fmt", ...)
• Go: rewind.Always(cond, "id", "fmt", ...) and rewind.Track(t) for go test
• Python: rewind_sdk.always(cond, "id"), and a pytest plugin that loads itself
• Java: Rewind.always(cond, "id") and the JUnit 5 extension
• Shell: rewind_always id "message" test "$x" -eq 3

$ curl -fsSL https://rewindvm.dev/install | sh
$ rewind check github:fzakaria/rewindvm#bank-java

The properties tutorial goes through each piece step by step, and the advanced tutorial has script properties and a file of properties to keep beside your tests for the more adventurous.

If you happen to find any interesting bugs with Rewind let me know. I am keeping track of a small leaderboard of the best ones. 🏆

Josh Wulf has been running a show called Agentic Engineering, and he asked me on. Josh spent ten years at Red Hat, three years as a tech recruiter in Brisbane, and the last year sprinting down the same road I have, sounding a bit crazy to most people around him. We talked for about an hour and a half. He cut it into four parts, and they all dropped today.

Here's the gist of all four, written up properly and in order, with chapter links at the bottom if you want to jump to a specific part of the conversation.

There's no place in the world now for a software engineer who types code by hand.

part 1: is coding over?

nobody knows what happens next

Josh opened by calling me the LLM whisperer. I'll take it, but let's be clear: no one knows what the hell will happen. I'm a little bit older, I've worked in developer tooling for a long time, and I'm just a little better informed. That's it.

Here's the timeline. December 2024, I'm a tech lead at Canva. Everyone there loves JetBrains. I start using Cursor, start running my first loops by hand, and think, "ooh, JetBrains and IDEs don't exist in the future." You get used to being called a little bit crazy. JetBrains just reported its first-ever net loss. Sure, they've been investing, but in my circles, no one is typing code into an IDE anymore. If you hang around a group that still loves their IDEs, you're not hanging around a well-informed group.

In February last year I discovered filesystem-as-state, the first of the loops: the idea of autonomously building things. It spooked me. The first thing I did with it was publish the Z80 post.

Not enough people understood that post at the time. Take an existing application, clean-room it into specifications, then rebuild the software from those specifications, much like the way AMD's CPUs came out of Intel's. It's not legal in every jurisdiction, and you have to take the right approach. But did you see all the Photoshop clones that came out in the last week? That's just the Z80 idea, implemented.

A year and a half later, services and MCPs exist whose whole job is rebuilding existing software, and a couple of published papers cite it.

The idea underneath is infinite software. If you're selling software without infrastructure or some other lock-in, you're hosed. So the real question is: what is a moat in software now? What is investable in software now?

the travelator

Then came Dear Student. ThePrimeagen covered it, the juniors started paying attention, and I ended up in SF, at the pub, at a couple of meetups, realizing I couldn't sit on Ralph any longer because someone else would write it up first. By November, Boris was talking about using Ralph and loops in Claude Code. January was the real inflection point: it crossed into the mainstream, and business leaders started to understand what was happening.

It still hasn't crossed into the dark matter developers, and that concerns me deeply. If you've been applying yourself and staying curious, you have a two-year head start on everyone else. You can't just pick up the latest model and be good at this. It takes theory of mind, and the more you apply yourself, the more you see what's possible, which makes you more ambitious, which makes you see more.

Josh's story fits that exactly. He tried autocomplete-style Copilot, thought it was rubbish, and turned it off. Then a friend who heads AI at a couple of companies told him their engineers were getting good results. The first thing Josh did was have it clone a Java testing framework into Node.js. It wasn't great, but it was maybe 80% as good as he was, and it never stopped. It doesn't need to eat or sleep, and it doesn't lose focus. So he figured out what the missing 20% was: completeness and verification, because the models would leave stubs and partial implementations. Do the 20% instead of the 100%, and you've got scale.

Same for me. I'm no maximal AI glazer. I explicitly distrust the tools. The difference is that I internalized two years ago that the line was coming, and I've been expanding what I think is possible ever since. The models from a year ago were already good enough to displace software developers, and they keep getting better. We don't get stubs anymore. It still comes down to the operator's skill.

The travelator of tech moves on, as it always has. My deepest concern is the people who haven't even stepped on it. They're not aware it's moving, and they're completely unaware that some people are sprinting on it.

And you know how the employment market works. "Five years of React experience required." They're going to put minimum years of AI experience on job ads. If you wait for the models to be perfect before you start, there'll be juniors who are cheaper than you, who have been playing with this substrate, with a whole portfolio of cool things and maybe a couple of conference talks. Same story as always. Josh, who wrote those mathematically impossible job ads as a recruiter, reckons he got two years of LLM experience into one by doing more hours every day and more days every week.

managers, burnout and retirement projects

A couple of days ago I published the post I'd been putting off for five or six months. If you're an engineering manager and you're not creating time for your team to play with AI, you're setting them up to be replaced. It got 500,000 unique views in 24 hours.

You join a company to de-risk your life. If a company has banned AI outright, quit and don't join one like it. You trade time and skills for money, and you need skills that will be employable for you and your family. You could try to change the culture, but plenty of employers are already with the program and will likely pay you more.

tl;dr jinx is a tool that compiles Nix to TypeScript. All of nixpkgs, each 231,777 attribute, produces the exact drvPath Nix does. Open the playground or evaluate hello in your browser tab with no Nix installed anywhere. 😈

Despite my admiration and joy for Nix, the language and tool, it is not what is magical. The store model is the great idea. Any similar tool built on the same model will have the same magical properties of reproducibility, and hermeticity.

I have spent a lot of posts exploring what falls out of the Nix ecosystem, but the language itself is a small lazy functional language with a derivation primitive. You could design another one in a weekend.

What you cannot design in a weekend is nixpkgs: twenty years of fixes, every cross-compilation workaround, every stdenv phase, every patched configure script, every “this library needs this flag on aarch64-darwin” that somebody already solved. As of October 2026, it is 140,000-plus packages that build. That is the magic sauce 🥫, and it is written in Nix, so if you want it you write Nix.

Unfortunately it is also the trap. Anyone who wants to explore the store-model space from a different direction, a new language, a new evaluator, a new editor experience, must then start from zero packages. This is the graveyard where other reimplementations or forks of Nix go to die.

If however you could take the whole tree of nixpkgs and transform it into something another ecosystem already speaks, you could explore the store model without abandoning the twenty years of work that went into nixpkgs. 💡

I chose TypeScript. Why? Why not!?

TypeScript has a breadth of tooling and is supported in nearly every browser. It has an incredibly flexible type system that means we can express the shape of a derivation and all the oddities within Nix’s lazy evaluation model.

I am a frontend impaired “noob” but I thought the idea was worth exploring.

jinx is the result. It is a transpiler that takes a .nix file and produces a .ts file that evaluates to the same value, with the same thunks, the same derivation strings, and the same drvPaths.

I then compiled all of nixpkgs with it. Every attribute, every package, every module, every stdenv and lib function: 231,777 attributes in total.

The compiled tree lives at fzakaria/jinx-nixpkgs, one commit per nixpkgs revision.

What it does

Each .nix file becomes one .ts module of the same name in a tree that mirrors the source. A lambda is an arrow function, a let is a block of consts, an attribute set is an object literal, import is require. You can read it.

$ jinx compile pkgs/by-name/he/hello/package.nix

// Generated by jinx from .../pkgs/by-name/he/hello/package.nix. Do not edit.
import * as nix from "../../../../.jinx/runtime/index.ts";

const here = nix.here(import.meta.url);
const formals$1 = { args: { callPackage: false, fetchurl: false, hello: false, lib: false, nixos: false, stdenv: false, testers: false }, ellipsis: false };

export default nix.lambda((args) => {
  const $ = nix.formals(args, formals$1);
  const callPackage = nix.arg($, "callPackage");
  const fetchurl = nix.arg($, "fetchurl");
  ...
  return nix.app(nix.sel(stdenv, "mkDerivation"), (finalAttrs) => ({
    pname: "hello",
    version: "2.12.2",
    src: nix.thunk(() => nix.app(fetchurl, {
      url: nix.thunk(() => nix.str`mirror://gnu/hello/hello-${nix.sel(finalAttrs, "version")}.tar.gz`),
      hash: "sha256-...",
    })),
    ...
  }));
}, formals$1);

Nix is lazy, so the emitted code wraps an expression in nix.thunk(() => ...) wherever Nix would allocate a thunk. 🤮

The jinx evaluation writes nothing to /nix/store. jinx instantiate writes the .drv closure and nix-store --realise builds the derivation.

$ jinx eval hello.nix
{ drv = "/nix/store/j8q5j0x495vfrxgha5m6q40yjdi59334-hello-2.12.2.drv";
out = "/nix/store/18bbdvag5v2f3d4y37pdbkzvh7s71cw4-hello-2.12.2"; }

$ nix-instantiate --eval --strict hello.nix
{ drv = "/nix/store/j8q5j0x495vfrxgha5m6q40yjdi59334-hello-2.12.2.drv";
out = "/nix/store/18bbdvag5v2f3d4y37pdbkzvh7s71cw4-hello-2.12.2"; }

We produce the exact same .drv and the exact same drvPath Nix would. That is the key to unlocking nixpkgs in another ecosystem: the store does not know the difference.

Nothing lost in translation

We can prove our translation is faithful. We evaluate every attribute of nixpkgs’ own CI configuration, ci/eval/outpaths.nix, with jinx and with nix-eval-jobs, and compare all the drvPaths.

On nixpkgs 6510408d, on x86_64-linux, aarch64-linux and aarch64-darwin:

Zero differences or failing on one side.¹

But seriously, why?

Evaluating nixpkgs in TypeScript is ~3-5x slower. If that is all it did, that would not be very interesting. The point is not to make Nix faster, it is to make Nixpkgs accessible to the TypeScript ecosystem.

Here are some “fun” things you can do with it.

Profile an evaluation. node --cpu-prof on firefox.drvPath, folded through the source maps so every frame is a line of nixpkgs. Click it for the whole SVG.

Complete nixpkgs in your editor. Automatic language server protocol support for every attribute in nixpkgs, with source maps to the original .nix file. TypeScript’s type system is expressive enough to model the shape of a derivation, so you get completion and type checking for free.

Step through a package. Source maps put a breakpoint on a .nix line. Open the compiled tree in VS Code or Chrome DevTools, break in pkgs/by-name/he/hello/package.nix, and step into stdenv.mkDerivation as it builds the attribute set.

$ node --inspect-brk --enable-source-maps src/cli/main.ts eval hello.nix

Ask nixpkgs a question in a loop. Interrogate and explore nixpkgs in a REPL or a script without having to remember the incantation to get a JSON dump of the whole tree and pipe it through jq.

import { loadNixpkgs } from "jinx";

const pkgs = loadNixpkgs("/path/to/nixpkgs");
const gpl = [];
for (const name of Object.keys(pkgs.lib.licenses)) {
  if (pkgs.lib.licenses[name].spdxId?.startsWith("GPL")) {
    gpl.push(name);
  }
}
console.log(JSON.stringify(gpl));

Unit test Nix with node:test. A lib function or a package definition is a function in a module, so it has a unit test like any other function in any other language.

test("hello builds the derivation Nix would", () => {
  assert.equal(
    pkgs.hello.drvPath,
    "/nix/store/cnraiw0hbdl6jzdhbkn8dah2hqj7xgij-hello-2.12.3.drv",
  );
});

Write a package in TypeScript. The compiled form from jinx is faithful to Nix down to every thunk, which makes it exact and miserable to write by hand. Alternatively, there is a much nicer hand-written form. It interoperates with nixpkgs, so a package in TypeScript can call stdenv.mkDerivation and depend on anything in pkgs.

const hello = pkgs.stdenv.mkDerivation({
  pname: "hello",
  version: "2.12.2",
  src: pkgs.fetchurl({
    url: "mirror://gnu/hello/hello-2.12.2.tar.gz",
    hash: "sha256-WpqZbcKSzCTc9BHO6H6S9qrluNE72caBm0x6nc4IGKs=",
  }),
  meta: { license: pkgs.lib.licenses.gpl3Plus },
});

console.log(hello.drvPath);

It runs in a browser. TypeScript is supported in every modern browser. You can evaluate nixpkgs in your tab, and it will fetch the compiled modules from GitHub as it needs them.²

Is it slow?

Yep it’s slow, but crucially not too slow.

When warm, with the modules compiled and V8’s code cache populated, per attribute on my laptop:

A first evaluation that has to compile every module it reaches is 3x to 10x in speed and memory is ~2x-4x because thunks are very expensive to the garbage-collector.³

Let’s say it is five times slower than a C++ interpreter that has had twenty years of people shaving it, from a brand new compiler that runs on a JavaScript engine. That strikes me as a fine place to start. 🤷

Can I try?

The source is at github.com/fzakaria/jinx.

It has no dependencies, no native code, and no daemon. You can run it with Node 24 or in your browser. The compiled nixpkgs tree is at github.com/fzakaria/jinx-nixpkgs.

I am a little annoyed that I have to keep disclosing that I used “AI” for the code. People seem to enjoy “outing” you for you it, as it is some morale failing. Let me save them the trouble: an AI co-authored a lot of this software with me.

Here is your stamp. You can stop reading here if that is a dealbreaker, the derivations will not change their hashes for you.

Nixpkgs is a TypeScript program now. Go set a breakpoint in stdenv.

Justin Cormack, who worked on MirageOS and Unikernel Systems back in the day, has been noticing what I've been noticing: people are discovering (or rediscovering) unikernels again. He's running a series of conversations on the topic for his newsletter, and I was first up. He emailed me, and five minutes later I was talking to him from the pub with a stein of beer in hand. We went deep on Mirage, Orleans, Haskell, Nix, Cursed and a whole lot more.

Here's the gist, written up properly, with chapter links at the bottom if you want to jump to a specific part of the conversation. Justin's edited transcript is over at Ignore Previous Directions.

Unikernels were hard. Key word: were. Now we have AI.

what a unikernel is, and why they were hard

I first ran into unikernels around 2015. I'd staffed up a team of Haskellers and gone pretty deep on functional programming. Where there are Haskellers, there are OCaml programmers, and from there you find MirageOS. Great idea. I played with it back then.

A unikernel is the idea that your application is the operating system. There's no userland. If you want a web server, DNS, or to send an email, there's nothing you can fork or spawn. You have to write those things as libraries in your application.

That was the friction. Justin remembers it well: when they were building Mirage, they had a TCP stack and an HTTPS stack, but there was almost nothing for storage. They were pulling drivers out of NetBSD because they could run them in userspace. It was hard back then.

There's a lot of dogma in our industry. Nix is hard. Bazel is hard. Unikernels are hard. Yes, they were. These hard concepts are now in the model weights. All you've got to do is prompt for them and, cognitively, get rid of the dogma that they're hard.

the operating system is design debt

Every application that isn't a unikernel was built on the assumption that there's an application, and then there's an operating system underneath. Why do we even have an operating system? Because forty years ago there was a human operator. I've done IBM 5250, AIX, Solaris, and mainframes. The multi-user operating system exists because a person sat in front of it, and then we put the application on top.

I consider that design debt, and here's why it matters right now. Applications get popped. They were getting popped before AI. Someone pops the userland application and gets a shell. That shell is a VIP butler service for exfiltration.

With a unikernel, the attack surface is much smaller. If the functionality isn't in the application (the operating system), then the attacker is screwed. There's no next hop.

Justin pushed back here, and fairly. Attack-surface reduction is something people are very fuzzy about. You can remove the shell from a Linux container, but almost every Linux environment still has something that's effectively an interpreter. You can execute a new program without a writable filesystem. You've still got memory safety and gadgets to worry about.

All true. But look at what we've been doing for twenty-six years. The earliest adage I remember from the SunOS and cgi-bin days was "don't put the compiler on production." Then came build containers and production containers. Then Chainguard. We keep chipping away at the attack surface, instead of going in the opposite direction and ensuring there is no attack surface.

And this is the bit people miss: if there's no shell and no interpreter, there's nothing in the model weights that knows what to do next. That turns a drive-by (pick your framework's RCE of the week and you've got a shell, and the model weights know what to do with shell access) into a targeted attack that needs your source code.

you can just port the missing libraries now

The classic objection: your unikernel needs to talk to Stripe, and OCaml doesn't have a Stripe library. Before AI you'd sigh and write it. Now? Run a loop to port the Go library to OCaml. Here you go: Stripe in a unikernel.

Justin had a great example of the same thing. He'd been building minimal Linux OS images for appliances, which is halfway to a unikernel anyway, since you're only running one application as PID 1. He needed to make an XFS filesystem. Rather than drag in xfsprogs and everything it brings with it, he sat down with an agent and had it write mkfs.xfs in Rust, producing byte-for-byte identical output, with every flag interpreted. It reverse-engineered the on-disk formats one by one, with tests across block sizes. It took a few hours.

That works because the original tool is a golden oracle. Generate filesystems at different sizes with both implementations and diff them. Port the tests across. Automate it.

Storage was the other big gap. Most workloads these days are cloud-shaped, even on-prem, so take the turbopuffer approach: S3 as your primary, infinitely growable storage, with a local NVMe block cache and an LRU (or whatever caching algorithm you like) for the hot bits. Justin is a massive "S3 for everything" fan too. As long as latency isn't the constraint, you get infinite storage with multi-user access, and you can build everything on it.

nix machine tests and overlays

Justin had been experimenting with Nix too, and was surprised that the first time he got an agent to prototype an OS, it built all the tests into flakes.

Nix the language sucks. Nixpkgs is great. NixOS machine tests are the bee's knees: you write a test that spins up a fleet of machines and exercises the interaction between your network rules and your application. It's the thing people don't know about.

And when something upstream is broken, or there's a supply chain problem in your dependencies, that's just an overlay.

If you have to tool-call a human, also known as "Dear Maintainer", who might be on holiday or might have abandoned the project, and wait a day, two days, or even five minutes, that's not AGI.

We're building recursive products here. Agents need the ability to modify the world as first-party source, not as third-party bundled binaries. We're going back to the contrib folder and Unix patches.

if you care about security, you have two choices

Justin asked what unikernels still need for people to discover them. Honestly, it's this. We've raised two generations of developers who don't even know they're a thing.

When Ralph was published last year, I sat on it. People who have been reading this blog for a while know this. I first came across the idea of loop engineering in early January last year, sat on it for four months, and then started writing a lot, begging people to pay attention.

When Ralph and Loop Engineering went viral earlier this year, almost a year later, it really weighed heavily on me. Life's been pretty good lately, but it wasn't always historically, so I took some time off. I spent six months traveling around the world with no income, and I delivered the talk below almost 17 times in various cities.

Our profession is at a crossroads, and this is the last time I'll say it.

Usage of AI in the profession of software engineeering is no longer optional.

If your employer has banned the use of AI, you should quit and find a company that is already going through an AI transformation.

If you're at a company going through an AI transformation, I understand you'll have mixed feelings about this. You might not like it. But please understand that this is the new normal. Artisanal handwritten code is only possible for retired tech executives and as a hobby at home, but it's no longer a way to make money.

You really have two choices as an employee.

You can push against the grain and refuse to use AI, but that's just self-harm. You trade time and skill for money. The tastes and desires in the employment market have moved on.

From my position, I have a clear view of what's going on.

Every morning, I wake up to either a WhatsApp message or a LinkedIn DM from people who are going somewhere, sending their thanks. Like, don't thank me; thank yourself. You invested in yourself, and because you did, you just got promoted to a vice president role. You did all the work.

Hi Geoffrey. I wanted to drop you a message to just share what a profound impact Ralph’ing had had on me and the company i work for. I first heard of it back in Jan 2026 on social media and thought it was an outrageous concept, the idea of letting this clumsy persona rip through my codebase, so i just ignored it. My biggest frustration with AI has always been it never bought exactly what I wanted to life, it would get the first parts right, then it starts cutting corners, and I end up with a real mess at the end. That was until when I watched sonnet 5 loop through and churn out a product in about 6 hours, so for whatever reason I decided to come to your blog and read about Ralph directly - and I’d totally misundertood it. It took quite a bit of experimenting, however im now at a place where I’m literally delivering paid for dev worth 100K + for 3K in sonnet tokens in brownfield projects. Im re-platforming entire stacks in about 2 weeks that we used to pay contractors half a million for - the game has totally changed, and the inplications are both amazing and terrifying at the same time. Anyway, just wanted to say thank you, because at least for me and the company I work for, you publishing the Ralph paper has forever changed the game, and I hope that one day I’ll get to buy you a drink to say thank you. All the best.

So it's time to address the elephant in the room: people managers.

If your team is too busy doing their new job to experiment with AI, you're preparing them to be replaced. Maybe you're okay with this, but hopefully you're not, because it should weigh heavily on you if you truly care about your team.

Sometime in the next six months, you're going to have to start having some hard conversations about moving people on from your company. You should already be putting your employees on a vitality curve and ranking them. Naturally, competent employees who have been using AI will be ahead of competent engineers who haven't.

Here's something I've been meaning to write up for a while. It's kind of late, but I'm going to lean into the whole MCP vs. CLI argument because I have a perspective others might find interesting.

What's the actual difference between an MCP server and a regular old CLI program with a `--help` flag? Feels to me like a CLI program written with agents in mind solves the same problems without inventing anything new?
https://x.com/adamwathan/status/1968738637358522629

Developer tooling companies can be sorted into two tiers: S-tier (winning) and F-tier (losing). The Rubik for wherever a company falls into the S tier or F tier has got everything to do with the model weights.

An S-tier company is in the training weights. Let's take GitHub, for example. Now, we can yak about problems with GitHub till the cow comes home, but one thing is true: these models know how to interface with GitHub because that behavior is in the model weights themselves. They don't need any skills or an MCP to get outcomes. You can just use the CLI.

You are S-tier once you can do a prompt like this without any skills or MCP:

prompt: Use the GH cli to create a repository, commit the changes, push to origin, set the reposistory to public and update the repository description.

Now, if you're a brand new developer tooling company and you're not in the model weights yet, you've got a couple of choices. First, you'll have to ask users to download and install some sort of skill pack, or maybe even a CLI, which adds a lot of friction for users compared with the DevX of an S-tier company.

So, what do you do? It's simple.

Start by developing a really good CLI, and on that CLI, work on and optimize your --help section. What you want to do is measure whether a model can achieve outcomes by walking the help verb across all the subverbs, then optimize the language used by choosing whatever works best, so the model can achieve the outcomes with the fewest tool calls. This can be automated through a loop or an auto-research loop.

Once you have this bare-bones foundation, the next step is to publish documentation for this CLI online and then focus on ASEO (Agentic Search Engine Optimization) for agents web_search_tool. Again, from this point forward, the goal is to look at the journey a single agent takes and optimize it so the agent can get all the information it needs in a single web search tool. If it has to walk through your documentation, you're not doing a good enough job. Keep optimizing.

To take this to the next level, you should enter into a business contract with the labs. Perhaps you've already got a contract with the labs for inferencing. You want to lean on them to include your CLI documentation in their next training run.

If you nail these things, then you're well on the path to becoming an S-tier developer tooling company. The best part is that it doesn't require users to download or install a skill pack, configure an MCP, or deal with any of that junk.

ps. socials

I don't know if you've noticed yet, but JetBrains reported a net loss for the first time in its recorded history.

Before we go into this, let's rewind time to January 2025, nearly a year and a half ago, when I penned these words.

So, I've been wondering what the next generation of IDE looks like in this post-LLM world. Why do I need to have multiple IDEs open? What if we ripped out the "collaborate with humans" functionality from Zed and inverted it so it's "collaborate with LLMs," and an unbounded number of LLMs could be connected to the IDE. Even then, that might not be radical enough of a UX/design overhaul. Maybe we don't even need an IDE? Maybe Devin is onto something...

I remember writing these words as a tech lead at Canva, with coworkers calling me mad and saying, "Geoff, what do you mean?" I love JetBrains. I love my IDE. There's no way in heck this is changing; JetBrains will figure it out."

I'm not salty about it; it's just a fact. You get used to these things when you're early. If you're curious, you get used to the friction from people who aren't used to early, informed insights. To me, it was plain and clear that IDEs were dead and we needed better tools.

When I wrote the above, I was in the background enumerating JetBrains at all levels through our customer channels about what their roadmap would be, how they would counter this risk in business, and it's ultimately really sad to see that nothing was done other than jack the price of their subscriptions whilst simultaneously delivering a low-quality product (ala "private equity/broadcom subscription revenue extraction phase")

They essentially Kodaked themselves.

JetBrains could only see the world through the lens of the IDE, and they hoped that AST exposed by a language server would be a source of alpha even though everyone else in the agentic coding market was getting ripping results without symbolic context injection. Their experience and domain expertise became their curse, keeping them from changing and adapting to the market.

And that is the deepest lesson. From this point forward, I'm not going to mention JetBrains, but I'm going to focus on the lesson. Every company is blinded by its domain expertise and how it makes revenue, and in times like this with AI, it's important to stay highly neuroplastic and check your blind spots.

A good example of this is the labs. The labs deeply believe that a future version of their model will solve software verification because tokens shape their worldview and perspective. This belief blinds them to the fact that a better model will not solve the problem of verifying that an application will behave correctly in production under suboptimal conditions (i.e., flaky network connections)

My strongest advice to all business leaders right now is to look at your own dogma, how you view the world, and make sure you don't Kodak yourself in this super weird moment of hyper transformation.

ps. socials