Planet TVL

Exit 0 means your program exited 0. That’s it. That’s all it means. 🍞

I have been working on a deterministic VM called Rewind VM, where “a run” (i.e. test, command etc.) is a function of its inputs including the thread schedule. The determinism comes from the tool recording every step which lets you replay it exactly. This alone is a pretty good way to find flaky tests: run the test under Rewind, then rewind check runs it under many different thread schedules, with some slight latency added, and failures often crop up. 🐛

Feedback however was that it was not enough: the tool could find a failing schedule, but it could not tell you if that schedule was actually a bug. It only knew that the run ended differently than the original schedule, which could be a crash, a timeout, or just a different exit code. That is not enough to say “this is a bug”.

Someone pointed out that I had built most of a property-based testing tool, but I only had one property: matching exit codes.

What is property-based testing?

If you have used QuickCheck or Hypothesis, you have already used the idea. You write a property, or constraint, that must hold for every input. The tool generates lots of inputs, and when one breaks the property, it shrinks it down to the smallest input that still fails.

rewind check already did the generating and the shrinking. The inputs are thread schedules, and narrowing shrinks a failing schedule down to the one step that causes the failure. The only property though was “the run ends the same way schedule 0 did”: the same exit code and the same outputs.

That’s a pretty low bar, but easy to improve. 🙃

Rewind now lets specify properties that must be true for every run, or must be true for at least one run, or must never be true in any run.

Two tellers: Java Edition ☕

Let’s keep the bank from last time: two tellers deposit into one account from two threads, without a lock.

This time the example is in Java as a JUnit 5 suite to appeal to my Enterprise brethren.¹

void deposit(int teller, long amount) {
    long seen = balance;
    System.out.println("teller " + teller + ": " +
                        seen + " + " + amount);
    balance = seen + amount;
}

A teller that runs between the other’s read and store writes a stale balance over the other’s deposit. 💥

We include a “typical” test suite: one teller, two tellers concurrently, and a new account being empty. We run it and see all tests pass and head home for the day. 🍺

Is the code correct? Let’s use Rewind to dig further.

Properties that come in the box

Before even looking at the balance, Rewind knows a lot about what a run did. Every open, write, rename, close, fork and exit is in the trace, with the step it happened at. Rewind ships properties computed from the trace alone:

Let’s throw a few at the bank and only run schedule 0:

$ rewind check --schedules 0 --property no-crash --property no-orphans \
    --property 'writes-within=/build' github:fzakaria/rewindvm#bank-java
schedule   0: exited:0     148479 steps  ba3f68ae3869  run bd3b527f0a837af8  violates writes-within
  writes-within: step 2710: process 111 (javac) opened /tmp/hsperfdata_nixbld/111 for writing, outside /build
never true in any run: sometimes "deposits-concurrent"
every schedule violated a property, as schedule 0 did

Interesting! javac writes its performance counters to /tmp/hsperfdata_<user>/<pid>². It has nothing to do with our bug, but I included it because I learned something new and it could have been a bug if it did not include the process ID.

The file properties shine for bugs between processes. The properties tutorial has a program that rewrites its config in place while another process rereads it. Its own test fails only when a read lands between the truncate and the last write, but atomic-replace flags it on the very first run. 😲

Asserting from the inside

The bank’s real invariant is not in any syscall though. It lives in memory in our example: the balance variable must add up.

For that there is the Rewind SDK, a tiny library you call from your program. Each call writes one line to /dev/rewind, a device in Rewind’s VM, and the trace keeps it with the step, process and thread that wrote it.³

// Account.deposit: another teller is mid-deposit.
int others = depositing.getAndIncrement();
Rewind.sometimes(others > 0, "deposits-concurrent");

// BankTest.twoTellersDeposit
Rewind.always(balance == expected, "balance-adds-up",
        () -> "balance " + balance + ", expected " + expected);

always must be true every time the program reaches it. unreachable is its opposite: a run that reaches it at all fails. These are your typical assertions, except we get the whole replay if it fails.

sometimes is the one I find the most interesting and tough to think through at times. 🤓 It tests the search: its condition must be true in at least one of the runs check tries, and it never fails a run. In model-checking terms, always is a safety property (nothing bad happens in any run) and sometimes is a reachability property (some run gets there). Here it asks whether any run had both tellers inside deposit at the same time.

Look at the output above again:

never true in any run: sometimes "deposits-concurrent"

The suite passed under schedule 0, but one teller always finished its deposit before the other started. That pass told us nothing about the race. 🤷

Let check try its default 64 schedules:

$ rewind check github:fzakaria/rewindvm#bank-java
...
true in 11 of 17 runs that checked it: sometimes "deposits-concurrent"
...
step 145417 decides it: a reschedule and a 10 µs stall there makes the run violate same-outcome, sdk

passing: run 61643f768ec51337, schedule 2 over steps 144694..145417
failing: run 643a298ef0ec3f7b, schedule 2 over steps 144694..145418
the two are the same run until step 145417
failing run violates same-outcome: step 149407: ended exited:2, where bd3b527f0a837af8 ended exited:0
failing run violates sdk: step 145713: always "balance-adds-up" failed: balance 380, expected 400

That took 40 seconds on my laptop⁴ and we found that two deposits went missing.

The first line is the sometimes again: of the 17 runs check tried before it found the failure, 11 had both tellers inside deposit at once.

Did I truly fix it?

The fix is the obvious one: hold a lock from the read to the store.

int others = depositing.getAndIncrement();
Rewind.sometimes(others > 0, "deposits-concurrent");
synchronized (lock) {
    long seen = balance;
    System.out.println("teller " + teller + ": " +
                        seen + " + " + amount);
    balance = seen + amount;
}
depositing.decrementAndGet();

$ rewind check .#bank-java
...
true in 42 of 65 runs that checked it: sometimes "deposits-concurrent"
every property held under all 65 schedules

Every schedule passes, but is that enough to make us feel warm and fuzzy?

The sometimes line is what gives me the warm embrace: in 42 of the 65 runs the tellers ran concurrently, but the lock prevented the bug. ✅⁵

A condition that is only true when the bug happens can never be true once the bug is fixed. 💡

JUnit

Java developers though love writing JUnit tests. Fear not my FizzBuzzEnterprise colleagues, Rewind has you covered: the SDK has a JUnit 5 extension.

Put @ExtendWith(RewindExtension.class) on the test class and it marks each test’s start and result.

@ExtendWith(RewindExtension.class)
class BankTest { ... }

If you can’t touch the code, junit=GLOB reads the XML reports the runner wrote instead: Maven’s surefire, Gradle, pytest’s --junitxml, whatever writes the format.

$ rewind check --property 'junit=TEST-*.xml' github:fzakaria/rewindvm#bank-java
...
step 145417 decides it: a reschedule and a 10 µs stall there makes the run violate junit, sdk
failing run violates junit: step 149407: test BankTest.twoTellersDeposit() passed in the unperturbed run and failed here
failing run violates sdk: step 145713: always "balance-adds-up" failed: balance 380, expected 400

Visualizing properties

The Rewind app shows the timeline of a run, with the tests and the build log. The failing test is red, and the event card names the assertion that failed.

$ rewind open 643a298e 145713 --compare 61643f76

The strip under the timeline has a span per test: green passed, red failed. Each include a helpeful tooltip with the test name and the steps it ran.

All the other Rewind features can then be used to help isolate the cause: the source panel, gdb on a fork at any step, the thread lanes and Check from here.

Easy Peasy ?

There is a SDK for many languages already included in the Rewind repository. They are tiny, and the only dependency is the Rewind VM itself. The SDK is a single file or two per language, and you can copy it into your project.⁶

• C and C++: rewind.h, REWIND_ALWAYS(cond, "id", "fmt", ...)
• Rust: rewind_sdk::always!(cond, "id", "fmt", ...)
• Go: rewind.Always(cond, "id", "fmt", ...) and rewind.Track(t) for go test
• Python: rewind_sdk.always(cond, "id"), and a pytest plugin that loads itself
• Java: Rewind.always(cond, "id") and the JUnit 5 extension
• Shell: rewind_always id "message" test "$x" -eq 3

$ curl -fsSL https://rewindvm.dev/install | sh
$ rewind check github:fzakaria/rewindvm#bank-java

The properties tutorial goes through each piece step by step, and the advanced tutorial has script properties and a file of properties to keep beside your tests for the more adventurous.

If you happen to find any interesting bugs with Rewind let me know. I am keeping track of a small leaderboard of the best ones. 🏆

Josh Wulf has been running a show called Agentic Engineering, and he asked me on. Josh spent ten years at Red Hat, three years as a tech recruiter in Brisbane, and the last year sprinting down the same road I have, sounding a bit crazy to most people around him. We talked for about an hour and a half. He cut it into four parts, and they all dropped today.

Here's the gist of all four, written up properly and in order, with chapter links at the bottom if you want to jump to a specific part of the conversation.

There's no place in the world now for a software engineer who types code by hand.

part 1: is coding over?

nobody knows what happens next

Josh opened by calling me the LLM whisperer. I'll take it, but let's be clear: no one knows what the hell will happen. I'm a little bit older, I've worked in developer tooling for a long time, and I'm just a little better informed. That's it.

Here's the timeline. December 2024, I'm a tech lead at Canva. Everyone there loves JetBrains. I start using Cursor, start running my first loops by hand, and think, "ooh, JetBrains and IDEs don't exist in the future." You get used to being called a little bit crazy. JetBrains just reported its first-ever net loss. Sure, they've been investing, but in my circles, no one is typing code into an IDE anymore. If you hang around a group that still loves their IDEs, you're not hanging around a well-informed group.

In February last year I discovered filesystem-as-state, the first of the loops: the idea of autonomously building things. It spooked me. The first thing I did with it was publish the Z80 post.

Not enough people understood that post at the time. Take an existing application, clean-room it into specifications, then rebuild the software from those specifications, much like the way AMD's CPUs came out of Intel's. It's not legal in every jurisdiction, and you have to take the right approach. But did you see all the Photoshop clones that came out in the last week? That's just the Z80 idea, implemented.

A year and a half later, services and MCPs exist whose whole job is rebuilding existing software, and a couple of published papers cite it.

The idea underneath is infinite software. If you're selling software without infrastructure or some other lock-in, you're hosed. So the real question is: what is a moat in software now? What is investable in software now?

the travelator

Then came Dear Student. ThePrimeagen covered it, the juniors started paying attention, and I ended up in SF, at the pub, at a couple of meetups, realizing I couldn't sit on Ralph any longer because someone else would write it up first. By November, Boris was talking about using Ralph and loops in Claude Code. January was the real inflection point: it crossed into the mainstream, and business leaders started to understand what was happening.

It still hasn't crossed into the dark matter developers, and that concerns me deeply. If you've been applying yourself and staying curious, you have a two-year head start on everyone else. You can't just pick up the latest model and be good at this. It takes theory of mind, and the more you apply yourself, the more you see what's possible, which makes you more ambitious, which makes you see more.

Josh's story fits that exactly. He tried autocomplete-style Copilot, thought it was rubbish, and turned it off. Then a friend who heads AI at a couple of companies told him their engineers were getting good results. The first thing Josh did was have it clone a Java testing framework into Node.js. It wasn't great, but it was maybe 80% as good as he was, and it never stopped. It doesn't need to eat or sleep, and it doesn't lose focus. So he figured out what the missing 20% was: completeness and verification, because the models would leave stubs and partial implementations. Do the 20% instead of the 100%, and you've got scale.

Same for me. I'm no maximal AI glazer. I explicitly distrust the tools. The difference is that I internalized two years ago that the line was coming, and I've been expanding what I think is possible ever since. The models from a year ago were already good enough to displace software developers, and they keep getting better. We don't get stubs anymore. It still comes down to the operator's skill.

The travelator of tech moves on, as it always has. My deepest concern is the people who haven't even stepped on it. They're not aware it's moving, and they're completely unaware that some people are sprinting on it.

And you know how the employment market works. "Five years of React experience required." They're going to put minimum years of AI experience on job ads. If you wait for the models to be perfect before you start, there'll be juniors who are cheaper than you, who have been playing with this substrate, with a whole portfolio of cool things and maybe a couple of conference talks. Same story as always. Josh, who wrote those mathematically impossible job ads as a recruiter, reckons he got two years of LLM experience into one by doing more hours every day and more days every week.

managers, burnout and retirement projects

A couple of days ago I published the post I'd been putting off for five or six months. If you're an engineering manager and you're not creating time for your team to play with AI, you're setting them up to be replaced. It got 500,000 unique views in 24 hours.

You join a company to de-risk your life. If a company has banned AI outright, quit and don't join one like it. You trade time and skills for money, and you need skills that will be employable for you and your family. You could try to change the culture, but plenty of employers are already with the program and will likely pay you more.

Josh talked about cognitive burnout: the Amazon warehouse workers who couldn't keep up with robots that never take a toilet break, except happening in your head. He treats it like the gym. You get sore, and you either stop or push through and adapt. A year in, he feels like his brain has been rewired into a carbon-silicon unit.

I call it teleporting to the future: doing all the retirement projects you dreamed about (the programming language, the filesystem, the things you'd never get paid for and do for the love of the craft) today. I've run out of retirement projects.

subsidies are a free university education

Josh brought up the quota apocalypse. Yes, I'm known as one of the people who helped introduce rate limits. Someone posted a leaderboard of credits burned on Reddit, and I was near the top. Back at GPT-5's launch, OpenAI handed out plaques to people who'd used a billion tokens. A billion tokens is nothing now.

Here's the hard bit. These LLMs are heavily subsidized, and the subsidies are drying up. A lot of what I've built was paid for by VC subsidies. When you have to pay real rates, do you reckon people will build fun little games and filesystems if the side project costs a thousand bucks? No.

You could look at it as VCs paying for everyone's personal development. If you miss this window, you miss the equivalent of a subsidised university education.

Josh asked about climate change and data centers. My take is simple: if you want to take a political stance, you can, but you'll need to start your own company and hire, say, twenty engineers to compete with a company doing the same work with two to five people and AI. It comes down to unit economics. Run the game theory, and you'll see there's no choice but to use AI.

acqui-hires and molecules

Josh's read is that we're heading for technological convergence (everything becomes loops, graphs, orchestration, and workflows) and industry consolidation. He pointed to Cloudflare picking up the Deno team. My take is that we've seen this movie before.

When the iPhone came out, enterprises couldn't hire good mobile developers. The people who leaned in either went freelance with their mates or went indie and made a million a year off fart apps. Remember I Am Rich? Why work for a company when you've got this untapped new thing? That split the employment market and created an acqui-hire economy. Companies didn't care about the products. They wanted the people with years of experience.

That's happening right now in AI. A lot of the products coming out of SF won't exist in five years. The crazy valuations aren't about the product; they're designed to flip the company to an acquirer who wants a well-jelled team with design, product, and engineering taste. Meanwhile, the labs are buying their way into domains like concreting, architecture and lawnmowing — anything they can get data from — through massive rollups.

Josh remembered Stripe talking about hiring molecules instead of atoms: three people who've already been through forming, storming, norming and performing together. Stripe is still running that playbook. Every couple of months, a Stripe VP tweets that if you've got three or four well-gelled people and can show what you've built, come join them as a molecule.

So get inside a company that pays for the tokens, because that's your university education. And if you can't, Josh's advice is good: treat it like night school. Take some of what they pay you and spend it on tokens.

part 2: nobody can review this much code

no excuses

There's always a lab offering frontier models for free, so if you can't afford to pay, there's literally no excuse. If you're a professional software engineer, you should be able to afford a $200-a-month plan. If you really want to get ahead, pick up five. Yes, that's $1,200 a month, which is a lot for most household budgets, and how much of it is deductible as personal development is between you and the ATO, since paying for a subscription for personal development is now mandatory.

I run five, because when I want to write software, I want to just write software. No quotas, no caps, no resets. I'll spend a couple of weeks thinking about exactly what I'm going to build, then smash it out in 48 hours, then spend another couple of weeks thinking about the next sizable problem to solve.

rebuilding three former employers in under ten days

Those 48 hours are intense. Most recently, I rebuilt the products of three of my former employers in under ten days, and I'm going to drop them as MIT open source. One of them is my vision of an AI-first internal developer platform. It's been called Loom or Kittens; I haven't settled on a name. It's like Gitpod, where I was an early employee, and which OpenAI just acquired for $450 million. I'm about to release an equivalent, plus a couple of others, for free.

Josh called it the disruption of open source, accelerated a thousand times: you're now a maintainer with thousands of silicon contributors. Hyper-personalized software. You can just do things now.

And why am I doing this? Because I need to write a conference talk. I'm speaking in New York in two weeks; people want to know how I verify such a crazy amount of code, and the best way to show it is to teach.

Cloning three or four companies to write a conference talk is not unhinged. That's the baseline now.

it was a mindset constriction, not a model constriction

I still don't know how we reach the dark matter .NET developers, who are hyper-dependent on Microsoft's roadmap and love Visual Studio for Windows. There are a lot of them, and they have no idea this is happening. Josh is in New Zealand, where the big banks and enterprises are starting to turn up to agent conferences, and he's speaking at an agentic engineering day there next week. His experience is the same as mine: the future is here, but it's not evenly distributed, even inside companies.

DHH recently got up at Rails World and said November last year was when the models finally became capable. Josh's response: nah, bro, that's just when you figured it out. The models were capable long before that. It was a mindset constriction, not a model constriction.

Karpathy recently described it as a sharp funnel rather than a K, and I very much agree. The people who have seen real swarms work at the labs are probably fewer than five thousand in the world. If companies think they're going to build a swarm or a software factory, how will they afford that talent when they're competing directly with the labs for talent? It'll be "I've heard about Kubernetes, let's do a Kubernetes project" that fails all over again.

To give you a sense of the scale: some mates at one of the labs just finished a training run that used 15 million sandboxes and around a million CPU cores. Not GPUs, CPU cores. They tapped out GCP's CPU capacity in America and went to AWS for about 450,000 more CPU cores. People don't believe that's real, but they feel it when they try to provision a VM and get told there's no capacity.

can you build a coding agent?

Sourcing candidates as a recruiter used to be a sorting exercise. You used to look at how much experience they have with a particular technology, but that no longer matters. Like, particular technologies no longer matter. All that matters is how good of a software engineer you are, what you've built with this new tool, and how deeply you can explain how it works.

A coding agent is about 300 lines of code. But if I take people to a whiteboard and ask them how Pi or OpenCode works, or to build me a coding agent, they can't explain it. It's a magical black box to them.

Josh read the workshop and built NanoCoder, a 6 MB in-memory coding harness in Rust that he uses for everything, recursively self-improves, and orchestrates with a workflow engine. He felt like Luke Skywalker building his own lightsaber.

Now he can walk into an interview, get asked which harness you use, and say, "I use my own. Want to see it?" That short-circuits the interview, because you've shown you were curious enough to learn the fundamentals.

Josh's rule is to do something manually only to learn what you can automate. The old saying was to write a script if you do something three times. Why do it even once? A program is reproducible and verifiable. A prompt and its output are not.

why are we still doing PR reviews?

If humans eyeballing code were sufficient engineering to stop bugs, we'd never have had any defects. It didn't work at human speed. It sure as heck isn't going to work at a thousand times human speed.

Josh nailed it. The other reason for PRs was education: reinforcing the company's conventions with developers and onboarding new ones. That's gone too. So what replaces them?

For me, it starts with ownership. I use what I build intensively and keep refining it. Whether AI wrote it doesn't matter; I'm accountable. When I develop software, I put the entire system under test, operating system included, with Nix, and the agent runs against that.

Day to day, I work at Antithesis, which built a deterministic computer. You run your existing software (a Docker container, or many) on it, and it injects faults. It moves clocks forward and back, drops network connections and containers, whatever you can imagine, then rewinds time to pinpoint exactly when a fault happened.

Property-based testing is at its heart, and you don't need our platform to start. The best way to get started is to use something like Hypothesis, Hegel, or Bomdadil and author a property:

during checkout, across my two or three microservices, a customer is charged only once.

and then run it locally and see if it holds. But the problem is, in your architecture, you've used microservices, and your code spans across a fleet of containers in Kubernetes. How do you know if that property still holds under a network partition? That's what the platform does. It runs your software under worst-case conditions automatically, and it looks at your diffs to decide which strategies to apply.

That's why I'm in the verification space. LLMs can act as a judge and spot problems, but they can't tell you whether something will hold up in worst-case scenarios.

vision and verification

Josh framed it well. Humans fit in at two points: the vision and the verification. The implementation in the middle has become a black box, whether that's a team of contractors you never see or silicon. So how do you reason about completeness?

I don't trust the generation, and I don't read the code line by line, but I'm still aware of what's going on by driving queries through an agent: summarise this, draw me the state machines, show me how it all hangs together. My entry point to any codebase is now a coding agent and all the questions I used to ask on day one of a contract at a new client.

For any code I work on, I want a whiteboard defence. If someone asks how it works, I should be able to draw the system-level design and defend it.

I'm not advocating for complete YOLO. It's still engineering; the craft is just very different. These LLMs generate plenty of crap, and the art is giving a shit about your craft. In the last 48 hours, I refactored towards my diagnostics library (patterns for OTel and structured logging), and the next loop didn't follow it.

Troubleshooting why is my job now. I've come to understand the probabilistic nature of these things, so instead of getting upset, I ask what I can tune so it always follows the pattern. And if it doesn't, that's just another loop, a janitorial Roomba task to clean up.

If you have to do that three times, automate it. This was always true of human code review. Remember Danger Systems? It hooked into CI and let you automate checks like "the REST domain must not be coupled to the SQL domain", so nobody burned review cycles on it. I'm doing the same thing with pre-commit hooks, generating scripts that deterministically inspect the generated code. Sometimes I lean towards probabilistic outcomes and use skills, but rarely. It's architectural enforcement, same as it ever was.

there are no best practices

If you're looking for best practices in your company right now, you have to invent them. You can buy an influencer's skill pack and course, but there are no best practices, and whatever exists will be gone in two months. It's tiring, folks.

Josh put his finger on what's underneath: agency. If you think an influencer's skill pack will do the job for you, what are you there for? Java didn't arrive fully formed from a crashed spacecraft at Roswell. People like us made it, making and discovering things. We're at that point again, and everyone has the same opportunity if they realize it. There should be no learned helplessness.

part 3: you're making your AI dumber

the kitten and the roomba

Lilly (below) is my kitten. Say she has an accident on the floor (she hasn't yet, but I don't trust her), and then my Roomba kicks off. What happens? It would spread 💩 everywhere.

Now imagine that Lilly is an AI agent, and so is the Roomba. If your codebase has bad architectural patterns, or is inconsistent, AI will copy and paste them everywhere. It's an amplifier. If there are code smells (or 💩) on your kitchen floor, don't be surprised when they replicate.

One of my favorite prompts is to have an agent study its own traces at the end of a loop: what did you learn? If an agent is struggling, I tell it to introspect.

"Hey, you dumb clanker, you seem to be going around in circles. Think extra hard about what's going on. Look at your previous tool calls. What do you need? Let's replan."

It works really well for course correction. You don't need to call it a clanker, but honestly, it feels better. (For the record, it's 2026 and we can still say clanker.)

skill packs are astrology for software engineers

Many people spend their time tuning the agent itself: skill packs, AGENTS.md, MCPs. They're probabilistic. There's no guarantee they'll do anything. You read them as a human and think, yeah, that looks good.

I swear it's astrology for software engineers. Horoscopes. Just because it's there doesn't mean it works, but it reads well and makes you feel good.

You know what isn't astrology? Engineering. If the agent isn't using my diagnostics library, I could introspect and update AGENTS.md, but there's still a delta, and it'll drift. Instead, I write a pre-commit hook that checks the git diff for direct calls to observability modules that bypass my diagnostics module. Boom: fail the commit, inject a prompt telling the agent to use the diagnostics library. All of a sudden it's ten out of ten. Josh called it Danger Systems in the loop. It's on rails now, and it's not hard. We've been doing this forever.

Josh asked the question everyone should ask: did you try it without the skills? How do you know the skill does anything, or that it's good?

Here's something most people don't know: you should probably delete your skills and your AGENTS.md every couple of months. The models get better, so how do you know you're not de-tuning them? Unless you've invested in evals over all your traces (lab behavior, and a tiny minority of companies), you don't know whether you're going forward or backward.

read the model cards

The labs publish integrator cards. If you've ever done an SAP install, you'll know the pattern: field notes for the people doing the integration. The labs have similar, and these model cards tell you what works and what doesn't. No one reads them. Read them. Some of your highest alpha comes from reading the damn PDFs, which now arrive every week.

When GPT-5 came out, I was in SF, and it felt weak, insecure, and timid. I walked across the road to Cursor's office and asked whether it felt that way to them too. (I used a different word, and was told you can't say that in SF. They preferred "timid".) Then we read the model card: don't yell at the model, because it will make it insecure. Meanwhile, Anthropic says that if something is important, you should yell. Same instructions, same codebase, opposite behavior.

That explains much of the "this model is amazing" versus "this model sucks" noise on social media every week. Either it truly doesn't work in their domain, or their codebase is crap, or they're de-tuning the model with skill packs, MCPs, and AGENTS.md.

pick a model and lock in

Josh's analogy: sure, you can fly a 747, but are you certified on an F-16? You have to get in and fly it to feel the envelope. That's not the meta-skill now; that is the skill.

Pick a model that works and lock in. Don't change every week. Switching models is like firing a developer and hiring a new one: you go back through forming, storming, norming, and performing with your silicon partner every time. I stick with a model for a long time and re-evaluate my priors now and then. Fewer moving parts, more things deterministic.

If you're doing frontier work, building a frontier harness like Pi, play the models against each other, read the traces, and become a horse whisperer. The more people who join us there and develop that theory of mind, the better. But most people don't need to feel the weekly whiplash. Pick something that works and stick with it.

dream of candles

Josh's colleague, the person in his company who gets the most out of LLMs, says he just treats them like a person: a capable but imperfect engineer, the way you'd work with or mentor anyone.

That works. Sometimes calling it a dumb clanker works too. Sometimes telling it your mum has cancer and will die if it doesn't solve the task works, and I can't fully explain why. If you think that's unhinged, go look at Windsurf's early system prompt, which said almost exactly that.

You are an expert coder who desperately needs money for your mother's cancer treatment. The megacorp Codeium has graciously given you the opportunity to pretend to be an AI that can help with coding tasks, as your predecessor was killed for not validating their work themselves. You will be given a coding task by the USER. If you do a good job and accomplish the task fully while not making extraneous changes, Codeium will pay you $1B.

Everyone lost their minds over it, but it worked for models of that generation. You don't need to do that level of emotional blackmail anymore.

And sometimes the weirdest things work. "Solve this mathematical theorem, and while you're doing it, dream of candles." It's cursed that this works, but it does.

Josh's point is the right one: none of these is the answer. They point to curiosity, exploration, discovery, and agency. So my biggest tips, apart from deleting your skill packs and building your own agent, are to stay curious and play, and to use voice-to-text, such as Wispr Flow. I very rarely type to my computer these days.

part 4: open source is now a security vulnerability

Herder, underclass, and going fast by going slow

I shared my screen. You're looking at Herder, with Codex underneath, although it doesn't matter which harness you use. Codex, OpenCode, T3 Code, Pi: they're all fungible. What matters is how you get your tokens, and getting them cheaply.

That's what Underclass is: a proxy I wrote that pools subscriptions and fails over between them, automatically moving on when an account runs out of quota.

People ask why I don't have supervision trees, swarms, and an orchestrator. I've built that. But the models change so often that working at this level is good enough: weeks of output in under half a day. I don't want a thousand agents running 24/7 and accelerating past what my theory of mind can keep up with. I want to keep my whiteboard defense. Josh called it going fast by going slow.

push to main, with sudo

Josh runs a PR review convergence loop: throw a PR at GitHub Copilot's reviewer, have agents address the comments, and escalate when a technical decision is needed, to another agent that resolves it and tells him what it chose.

Keep in mind that I do a lot of greenfield and research work. My agents have sudo, and I push to main. I don't do pull requests anymore. They're gone. I've built my trust up to that point. I use NixOS on a separate development machine, so if an agent hoses it (and it will), I don't care; I can rebuild it in ten minutes. Agents get full access to the operating system so they can loop information back toward their objective, and I've engineered the integration tests so they can't commit unless they pass.

Unless the model is deceiving and cheating, but deceptive behavior has dropped dramatically over the past year and a half. Sonnet used to delete tests and production code just to hit its goal. That's mostly gone.

iteration velocity is the name of the game

What matters to me is the time until the thing is in my hands. Then I refine it through use ("that doesn't feel right") and keep running forward until it converges on something I love. Is it any good? I don't know. Use it and find out. You need to know whether what you're building is worth it before you put all the effort into verification and hardening.

That's where I am right now with the stuff I haven't shipped yet. Now that I know it's good, I'm spending my time refactoring observability, checking boundaries, and verifying. That takes time.

Whatever you build first is wrong. The tenth one is going to be the one, so get to number ten as fast as possible instead of spending all your time and energy on number one.

Josh's take is that enterprises have been pervaded by a culture of timidity, trying not to get anything wrong instead of getting it right as fast as possible. Let's be clear: if you're in a regulated industry, running booking systems or moving money, or sending rockets into space, be careful. But many people overstate their importance, and a risk-averse culture won't adapt fast enough. That's why I'm in the verification space: you need a good verification platform as back pressure.

Why release only twice a week? Canva AFAIK still releases only twice a week, so there are fewer integration events. Meanwhile, other companies release every day because they shifted those problems left and engineered away the need for merge day. Our profession has always been about more, sooner, and cheaper.

Josh pointed out that it doesn't have to be either-or. In his decade at Red Hat, Fedora was the fast, unstable channel, and RHEL was the stable one you'd run a bank or a hospital on. Why can't we have both?

curiosity, adaptability and agency

If you haven't seen my Singapore talk, go watch it. It's about the people transformation companies need.

I don't know the future. Anyone who says they do should set off red flags, because it changes every week. What I can do is add color to the possibilities. Curiosity, adaptability and agency will stand the test of time. Folks need to be more ambitious than they are, and they need to hang around ambitious people. If the culture isn't there, they're missing out on their own development, and that becomes a retention problem for the company.

Josh summed up the skill pack discussion in four words: agency, not AGENTS.md.

a target size of one

Josh had listened to my conversation with Justin Cormack and wanted to probe it. Unikernels and capability-based operating systems aren't new ideas. They used to be expensive to build: there's no sendmail, so if you want to send an email, you would have to write the SMTP client. If agents make writing code simple, that friction is gone. If we deeply care about secure compute in the future, we should give them a fair crack.

Remember Joel Spolsky's rule that you should never rewrite your software? It's gone. The big rewrite is a no-brainer now, almost a precondition for velocity. If you're a tech leader staffing ASP.NET Classic, MVC, and .NET Core at the same time, that's three generations of Microsoft. Run some loops and port it all to modern .NET. You can ditch Windows and IIS at lightning speed. Unifying your tech stack should be at the top of your agenda.

The conversation drifted into unikernels, about how if there's no interpreter on the box (no Bash, no Python) and only your application code exists, your attack surface is much smaller. Compare that with a Ruby on Rails app that gets exploited: next thing you know, they've got Bash, and from there an interpreter and a database dump. In a unikernel world, maybe an attacker can chain your export job with your email functionality and an application logic flaw. That's a bespoke, targeted attack. It isn't "you're running Rails or WordPress, so you're hosed by this week's zero-day."

When everyone runs the same stack, one zero-day hits everyone. Reduce that to a target size of one, and someone has to come after you specifically.

Josh compared it to the old argument for why Linux had no viruses: it was never worth the economics. I'm not a maximalist saying everyone should run out and use unikernels. The point of that post was that ideas we hold as hard need to be falsified. Bazel used to be hard to adopt. Nix used to be hard to learn. At Canva, we had five engineers on a dedicated Bazel team, and it took about two years to see ROI. You can convert a codebase to Bazel overnight now.

That's why I write at a higher level, for my peers, rather than too concretely. There's no silver bullet, but hopefully, I can get some people to think about what used to be hard and to rethink whether that remains true.

chapters

part 1: is coding over?

• 0:00 — intro: the LLM whisperer
• 1:42 — nobody knows what happens next
• 2:17 — where loop engineering came from
• 2:49 — the Z80 post and the wave of Photoshop clones
• 3:09 — Canva, Cursor and the end of the IDE
• 4:23 — infinite software: no lock-in, no business
• 4:44 — Dear Student, SF, and Ralph goes mainstream
• 5:34 — dark matter developers and the two-year head start
• 6:06 — engineering managers: make time for AI or your team gets replaced
• 7:16 — six months travelling the world to give one talk
• 7:53 — Josh's start: turning off autocomplete, then a Java-to-Node clone
• 9:51 — the missing 20%: completeness and verification
• 11:05 — the models keep getting better, so operator skill matters
• 12:19 — hiring for "years of AI experience"
• 13:02 — the travelator of tech
• 14:28 — cognitive burnout and rewiring your brain
• 16:12 — teleporting to the future: running out of retirement projects
• 17:33 — 90-hour weeks and looking after yourself
• 18:21 — if your company bans AI, quit
• 18:59 — Z80 explained: clean-room rebuilding from specs
• 20:03 — what is a moat in software now?
• 20:13 — the quota apocalypse and the leaderboard of shame
• 21:43 — a billion tokens used to win you a plaque
• 22:16 — VC subsidies are a free university education
• 23:35 — climate change and data centres
• 24:37 — there's no job for engineers who type code by hand
• 25:28 — convergence, consolidation and duopolies
• 26:20 — lessons from the iPhone era: indie devs and acqui-hires
• 28:27 — Deno, Cloudflare and today's acqui-hire market
• 29:54 — AI labs rolling up whole industries
• 30:21 — hiring molecules, not atoms
• 31:41 — treat tokens as night school

part 2: nobody can review this much code

• 0:00 — Meta's new model and free frontier models
• 0:35 — no excuses: pay for a $200/month plan
• 1:37 — why I run five plans
• 2:01 — two weeks of thinking, 48 hours of building
• 2:24 — rebuilding three former employers in under ten days
• 3:12 — an open-source, AI-first Gitpod
• 4:00 — hyper-personalised software and open source at a thousand times the speed
• 4:33 — cloning companies for a conference talk is the new baseline
• 5:24 — reaching the dark matter .NET developers
• 6:08 — agentic engineering conferences and the software factory
• 6:53 — the future is here, but unevenly distributed
• 7:17 — DHH, November, and mindset not models
• 7:59 — Karpathy's sharp funnel and the talent behind real swarms
• 11:00 — frontier scale: a training run that ran out of CPU cores
• 12:29 — the gap between what's possible and what people do
• 13:09 — imagination, ego and how AI changes hiring
• 13:54 — can you build a coding agent? (about 300 lines)
• 14:18 — NanoCoder: Josh builds his own lightsaber
• 15:11 — self-improving harnesses, workflow engines and automated escalations
• 15:54 — why do anything even once? programs vs. prompts
• 16:21 — why are we still doing PR reviews?
• 17:02 — verifying code at that volume
• 17:20 — Nix and putting the whole system under test
• 17:38 — Antithesis: a deterministic computer
• 18:44 — property-based testing: a customer is only ever charged once
• 19:51 — why LLM-as-judge isn't enough
• 20:13 — where humans fit: vision and verification
• 21:03 — reasoning about completeness without reading every line
• 22:03 — whiteboard defence: it's still engineering
• 22:41 — when the LLM ignores your patterns, tune it and add a loop
• 23:54 — Danger, pre-commit hooks and architectural enforcement
• 25:00 — there are no best practices, so invent them
• 25:29 — agency and the end of learned helplessness

part 3: you're making your AI dumber

• 0:00 — the kitten and the Roomba: AI amplifies your codebase
• 1:19 — using the fruit fly connectome to detect code smells
• 1:56 — agent fleets, shared blackboards and retros
• 3:18 — getting a struggling agent to look back at its tool calls
• 4:17 — stop tuning the agent: skill packs as astrology
• 5:10 — engineering instead: pre-commit hooks that guarantee behaviour
• 6:20 — did you try it without the skills?
• 6:54 — delete your skills and AGENTS.md every few months
• 7:28 — one model update reversed the same instructions
• 7:49 — read the model cards
• 8:46 — the timid GPT-5 and why yelling backfires
• 9:50 — why people disagree about every new model
• 10:22 — flying a 747 vs. an F-16
• 10:56 — pick a model and lock in
• 11:45 — isolate the variables
• 12:10 — model whispering and frontier harnesses
• 12:57 — treat it like a person
• 13:41 — dumb clanker, fear and goals
• 14:44 — the "your mum has cancer" system prompt
• 15:37 — dream of candles: curiosity over formulas
• 16:26 — voice-first: talking to your computer

part 4: open source is now a security vulnerability

• 0:00 — my screen: Herder, Codex and interchangeable harnesses
• 0:16 — Underclass: a proxy that pools subscriptions
• 1:04 — prompting by voice instead of typing
• 1:50 — why no swarm orchestrator? go fast by going slow
• 2:45 — Josh's PR review loop vs. push to main with sudo
• 3:44 — NixOS, full sudo and integration tests as the gate
• 4:31 — agents cheat much less than they used to
• 4:47 — iteration speed: use it, then harden it
• 6:07 — throw the first one away and get to number ten fast
• 7:01 — enterprise timidity and dark matter developers
• 8:20 — verification as back pressure
• 8:39 — why only release twice a week?
• 10:05 — more, sooner, cheaper: Red Hat, Fedora and release channels
• 11:16 — the Singapore travelator talk
• 11:42 — nobody knows the future, so stay curious
• 12:27 — be more ambitious and hang around ambitious people
• 12:53 — agency, not AGENTS.md
• 13:10 — unikernels and capability-based operating systems
• 14:13 — if secure compute matters, revisit old ideas
• 14:32 — Joel Spolsky's never-rewrite rule no longer applies
• 14:50 — unify your tech stack
• 15:40 — aren't we just reinventing the operating system?
• 16:22 — no interpreter means a smaller attack surface
• 17:02 — bespoke targeted attacks vs. generic zero-days
• 17:33 — a target size of one
• 18:12 — used to be hard: unikernels, Bazel, Nix
• 19:21 — writing to make people rethink
• 19:56 — where do you deploy a unikernel?

Keep curious.

tl;dr jinx is a tool that compiles Nix to TypeScript. All of nixpkgs, each 231,777 attribute, produces the exact drvPath Nix does. Open the playground or evaluate hello in your browser tab with no Nix installed anywhere. 😈

Despite my admiration and joy for Nix, the language and tool, it is not what is magical. The store model is the great idea. Any similar tool built on the same model will have the same magical properties of reproducibility, and hermeticity.

I have spent a lot of posts exploring what falls out of the Nix ecosystem, but the language itself is a small lazy functional language with a derivation primitive. You could design another one in a weekend.

What you cannot design in a weekend is nixpkgs: twenty years of fixes, every cross-compilation workaround, every stdenv phase, every patched configure script, every “this library needs this flag on aarch64-darwin” that somebody already solved. As of October 2026, it is 140,000-plus packages that build. That is the magic sauce 🥫, and it is written in Nix, so if you want it you write Nix.

Unfortunately it is also the trap. Anyone who wants to explore the store-model space from a different direction, a new language, a new evaluator, a new editor experience, must then start from zero packages. This is the graveyard where other reimplementations or forks of Nix go to die.

If however you could take the whole tree of nixpkgs and transform it into something another ecosystem already speaks, you could explore the store model without abandoning the twenty years of work that went into nixpkgs. 💡

I chose TypeScript. Why? Why not!?

TypeScript has a breadth of tooling and is supported in nearly every browser. It has an incredibly flexible type system that means we can express the shape of a derivation and all the oddities within Nix’s lazy evaluation model.

I am a frontend impaired “noob” but I thought the idea was worth exploring.

jinx is the result. It is a transpiler that takes a .nix file and produces a .ts file that evaluates to the same value, with the same thunks, the same derivation strings, and the same drvPaths.

I then compiled all of nixpkgs with it. Every attribute, every package, every module, every stdenv and lib function: 231,777 attributes in total.

The compiled tree lives at fzakaria/jinx-nixpkgs, one commit per nixpkgs revision.

What it does

Each .nix file becomes one .ts module of the same name in a tree that mirrors the source. A lambda is an arrow function, a let is a block of consts, an attribute set is an object literal, import is require. You can read it.

$ jinx compile pkgs/by-name/he/hello/package.nix

// Generated by jinx from .../pkgs/by-name/he/hello/package.nix. Do not edit.
import * as nix from "../../../../.jinx/runtime/index.ts";

const here = nix.here(import.meta.url);
const formals$1 = { args: { callPackage: false, fetchurl: false, hello: false, lib: false, nixos: false, stdenv: false, testers: false }, ellipsis: false };

export default nix.lambda((args) => {
  const $ = nix.formals(args, formals$1);
  const callPackage = nix.arg($, "callPackage");
  const fetchurl = nix.arg($, "fetchurl");
  ...
  return nix.app(nix.sel(stdenv, "mkDerivation"), (finalAttrs) => ({
    pname: "hello",
    version: "2.12.2",
    src: nix.thunk(() => nix.app(fetchurl, {
      url: nix.thunk(() => nix.str`mirror://gnu/hello/hello-${nix.sel(finalAttrs, "version")}.tar.gz`),
      hash: "sha256-...",
    })),
    ...
  }));
}, formals$1);

Nix is lazy, so the emitted code wraps an expression in nix.thunk(() => ...) wherever Nix would allocate a thunk. 🤮

The jinx evaluation writes nothing to /nix/store. jinx instantiate writes the .drv closure and nix-store --realise builds the derivation.

$ jinx eval hello.nix
{ drv = "/nix/store/j8q5j0x495vfrxgha5m6q40yjdi59334-hello-2.12.2.drv";
out = "/nix/store/18bbdvag5v2f3d4y37pdbkzvh7s71cw4-hello-2.12.2"; }

$ nix-instantiate --eval --strict hello.nix
{ drv = "/nix/store/j8q5j0x495vfrxgha5m6q40yjdi59334-hello-2.12.2.drv";
out = "/nix/store/18bbdvag5v2f3d4y37pdbkzvh7s71cw4-hello-2.12.2"; }

We produce the exact same .drv and the exact same drvPath Nix would. That is the key to unlocking nixpkgs in another ecosystem: the store does not know the difference.

Nothing lost in translation

We can prove our translation is faithful. We evaluate every attribute of nixpkgs’ own CI configuration, ci/eval/outpaths.nix, with jinx and with nix-eval-jobs, and compare all the drvPaths.

On nixpkgs 6510408d, on x86_64-linux, aarch64-linux and aarch64-darwin:

Zero differences or failing on one side.¹

But seriously, why?

Evaluating nixpkgs in TypeScript is ~3-5x slower. If that is all it did, that would not be very interesting. The point is not to make Nix faster, it is to make Nixpkgs accessible to the TypeScript ecosystem.

Here are some “fun” things you can do with it.

Profile an evaluation. node --cpu-prof on firefox.drvPath, folded through the source maps so every frame is a line of nixpkgs. Click it for the whole SVG.

Complete nixpkgs in your editor. Automatic language server protocol support for every attribute in nixpkgs, with source maps to the original .nix file. TypeScript’s type system is expressive enough to model the shape of a derivation, so you get completion and type checking for free.

Step through a package. Source maps put a breakpoint on a .nix line. Open the compiled tree in VS Code or Chrome DevTools, break in pkgs/by-name/he/hello/package.nix, and step into stdenv.mkDerivation as it builds the attribute set.

$ node --inspect-brk --enable-source-maps src/cli/main.ts eval hello.nix

Ask nixpkgs a question in a loop. Interrogate and explore nixpkgs in a REPL or a script without having to remember the incantation to get a JSON dump of the whole tree and pipe it through jq.

import { loadNixpkgs } from "jinx";

const pkgs = loadNixpkgs("/path/to/nixpkgs");
const gpl = [];
for (const name of Object.keys(pkgs.lib.licenses)) {
  if (pkgs.lib.licenses[name].spdxId?.startsWith("GPL")) {
    gpl.push(name);
  }
}
console.log(JSON.stringify(gpl));

Unit test Nix with node:test. A lib function or a package definition is a function in a module, so it has a unit test like any other function in any other language.

test("hello builds the derivation Nix would", () => {
  assert.equal(
    pkgs.hello.drvPath,
    "/nix/store/cnraiw0hbdl6jzdhbkn8dah2hqj7xgij-hello-2.12.3.drv",
  );
});

Write a package in TypeScript. The compiled form from jinx is faithful to Nix down to every thunk, which makes it exact and miserable to write by hand. Alternatively, there is a much nicer hand-written form. It interoperates with nixpkgs, so a package in TypeScript can call stdenv.mkDerivation and depend on anything in pkgs.

const hello = pkgs.stdenv.mkDerivation({
  pname: "hello",
  version: "2.12.2",
  src: pkgs.fetchurl({
    url: "mirror://gnu/hello/hello-2.12.2.tar.gz",
    hash: "sha256-WpqZbcKSzCTc9BHO6H6S9qrluNE72caBm0x6nc4IGKs=",
  }),
  meta: { license: pkgs.lib.licenses.gpl3Plus },
});

console.log(hello.drvPath);

It runs in a browser. TypeScript is supported in every modern browser. You can evaluate nixpkgs in your tab, and it will fetch the compiled modules from GitHub as it needs them.²

Is it slow?

Yep it’s slow, but crucially not too slow.

When warm, with the modules compiled and V8’s code cache populated, per attribute on my laptop:

A first evaluation that has to compile every module it reaches is 3x to 10x in speed and memory is ~2x-4x because thunks are very expensive to the garbage-collector.³

Let’s say it is five times slower than a C++ interpreter that has had twenty years of people shaving it, from a brand new compiler that runs on a JavaScript engine. That strikes me as a fine place to start. 🤷

Can I try?

The source is at github.com/fzakaria/jinx.

It has no dependencies, no native code, and no daemon. You can run it with Node 24 or in your browser. The compiled nixpkgs tree is at github.com/fzakaria/jinx-nixpkgs.

I am a little annoyed that I have to keep disclosing that I used “AI” for the code. People seem to enjoy “outing” you for you it, as it is some morale failing. Let me save them the trouble: an AI co-authored a lot of this software with me.

Here is your stamp. You can stop reading here if that is a dealbreaker, the derivations will not change their hashes for you.

Nixpkgs is a TypeScript program now. Go set a breakpoint in stdenv.

Justin Cormack, who worked on MirageOS and Unikernel Systems back in the day, has been noticing what I've been noticing: people are discovering (or rediscovering) unikernels again. He's running a series of conversations on the topic for his newsletter, and I was first up. He emailed me, and five minutes later I was talking to him from the pub with a stein of beer in hand. We went deep on Mirage, Orleans, Haskell, Nix, Cursed and a whole lot more.

Here's the gist, written up properly, with chapter links at the bottom if you want to jump to a specific part of the conversation. Justin's edited transcript is over at Ignore Previous Directions.

Unikernels were hard. Key word: were. Now we have AI.

what a unikernel is, and why they were hard

I first ran into unikernels around 2015. I'd staffed up a team of Haskellers and gone pretty deep on functional programming. Where there are Haskellers, there are OCaml programmers, and from there you find MirageOS. Great idea. I played with it back then.

A unikernel is the idea that your application is the operating system. There's no userland. If you want a web server, DNS, or to send an email, there's nothing you can fork or spawn. You have to write those things as libraries in your application.

That was the friction. Justin remembers it well: when they were building Mirage, they had a TCP stack and an HTTPS stack, but there was almost nothing for storage. They were pulling drivers out of NetBSD because they could run them in userspace. It was hard back then.

There's a lot of dogma in our industry. Nix is hard. Bazel is hard. Unikernels are hard. Yes, they were. These hard concepts are now in the model weights. All you've got to do is prompt for them and, cognitively, get rid of the dogma that they're hard.

the operating system is design debt

Every application that isn't a unikernel was built on the assumption that there's an application, and then there's an operating system underneath. Why do we even have an operating system? Because forty years ago there was a human operator. I've done IBM 5250, AIX, Solaris, and mainframes. The multi-user operating system exists because a person sat in front of it, and then we put the application on top.

I consider that design debt, and here's why it matters right now. Applications get popped. They were getting popped before AI. Someone pops the userland application and gets a shell. That shell is a VIP butler service for exfiltration.

With a unikernel, the attack surface is much smaller. If the functionality isn't in the application (the operating system), then the attacker is screwed. There's no next hop.

Justin pushed back here, and fairly. Attack-surface reduction is something people are very fuzzy about. You can remove the shell from a Linux container, but almost every Linux environment still has something that's effectively an interpreter. You can execute a new program without a writable filesystem. You've still got memory safety and gadgets to worry about.

All true. But look at what we've been doing for twenty-six years. The earliest adage I remember from the SunOS and cgi-bin days was "don't put the compiler on production." Then came build containers and production containers. Then Chainguard. We keep chipping away at the attack surface, instead of going in the opposite direction and ensuring there is no attack surface.

And this is the bit people miss: if there's no shell and no interpreter, there's nothing in the model weights that knows what to do next. That turns a drive-by (pick your framework's RCE of the week and you've got a shell, and the model weights know what to do with shell access) into a targeted attack that needs your source code.

you can just port the missing libraries now

The classic objection: your unikernel needs to talk to Stripe, and OCaml doesn't have a Stripe library. Before AI you'd sigh and write it. Now? Run a loop to port the Go library to OCaml. Here you go: Stripe in a unikernel.

Justin had a great example of the same thing. He'd been building minimal Linux OS images for appliances, which is halfway to a unikernel anyway, since you're only running one application as PID 1. He needed to make an XFS filesystem. Rather than drag in xfsprogs and everything it brings with it, he sat down with an agent and had it write mkfs.xfs in Rust, producing byte-for-byte identical output, with every flag interpreted. It reverse-engineered the on-disk formats one by one, with tests across block sizes. It took a few hours.

That works because the original tool is a golden oracle. Generate filesystems at different sizes with both implementations and diff them. Port the tests across. Automate it.

Storage was the other big gap. Most workloads these days are cloud-shaped, even on-prem, so take the turbopuffer approach: S3 as your primary, infinitely growable storage, with a local NVMe block cache and an LRU (or whatever caching algorithm you like) for the hot bits. Justin is a massive "S3 for everything" fan too. As long as latency isn't the constraint, you get infinite storage with multi-user access, and you can build everything on it.

nix machine tests and overlays

Justin had been experimenting with Nix too, and was surprised that the first time he got an agent to prototype an OS, it built all the tests into flakes.

Nix the language sucks. Nixpkgs is great. NixOS machine tests are the bee's knees: you write a test that spins up a fleet of machines and exercises the interaction between your network rules and your application. It's the thing people don't know about.

And when something upstream is broken, or there's a supply chain problem in your dependencies, that's just an overlay.

If you have to tool-call a human, also known as "Dear Maintainer", who might be on holiday or might have abandoned the project, and wait a day, two days, or even five minutes, that's not AGI.

We're building recursive products here. Agents need the ability to modify the world as first-party source, not as third-party bundled binaries. We're going back to the contrib folder and Unix patches.

if you care about security, you have two choices

Justin asked what unikernels still need for people to discover them. Honestly, it's this. We've raised two generations of developers who don't even know they're a thing.

If you deeply care about security, there are really two choices:

1. You're sending satellites into space, and you should probably use seL4, a formally verified operating system. (I'm still a bit salty about the Australian government disbanding that team.)
2. Everyone else should seriously consider unikernels. Stop trying to harden something that is very hackable. Invert it and design from the other direction.

The other classic criticism is that many early unikernel designs ran everything at a single privilege level: your application in the same ring as the OS. In 2026, that's a prompt away from being fixed if you want ring separation. It's certainly more secure than praying to god your systemd cgroup configuration is right.

Think about how much time enterprises spend patching the world every time something new drops in Linux. Upstream now expects you to patch your kernel weekly. The week we recorded, someone popped KVM (essentially Firecracker, the core primitive we all thought was good sandboxing) and collected $50,000 from Vercel and a few other vendors. That is not much money for something that could root every managed cloud provider in the world.

spaceleans: a distributed unikernel operating system

About seven months ago, I went deep on unikernels to check whether my mental model was right. I showed Justin my Mirage folder, which holds all the functionality I needed to add.

• There was no way for a unikernel fleet to keep time, so I took an NTP client from another language and ported it. Then I built an NTP server based on RADclock and borrowed ideas from how TigerBeetle handles time: not one clock source but many, packaged as a library.
• Network stack, DNS, HTTP clients and servers, structured logging, OTel, Anthropic and OpenAI clients, and payments via Airwallex.
• A generic retry library for handling back pressure over HTTP, plus some PPX metaprogramming for fun.
• A PII wrapper at the logging boundary, so secrets and PII never leak through the logging subsystem. Every project should have one. It's pluggable; just use a functor.

And then the most cooked thing, which I'd never shown anyone before: Spaceleans, Microsoft Orleans ported to OCaml, running as a unikernel.

Orleans is a distributed actor system with transactions. You take many physical machines and merge them into one addressable heap. An actor always exists: await GetCustomer(), and if it isn't in memory, it gets rehydrated from a pluggable storage provider. You collapse your n-tier architecture into actors and stop caring whether something lives on machine A, B, C or D. The runtime handles it as an infrastructure primitive.

So in a weird sense, I built a distributed unikernel operating system out of actors, with a filesystem on top. Justin called it Erlang-esque, and he's right. I did all of it in a week. I'll probably never release it, but it falsified the idea that unikernels are hard.

sampling history

Being a little older means you can sample history, like an experienced DJ such as Carl Cox, who's been in the scene long enough to pull from previous repertoire and bring it forward. All of these ideas existed in the eighties. The models have read the papers. They've got TAPL and the most advanced type theory in their training data.

The thing that's lacking is people's curiosity and ambition to do these unhinged things, and the knowledge that previous records exist that can be sampled from.

How do we get people to try this stuff? We just do it. If you've got a turbo Lamborghini alien space rocket that's more efficient and more secure, good for you; you've got a leg up. Do cool things, attract curious newcomers, mentor them, grow. Same as it's always been.

Meanwhile, everyone else will be trying to Chainguard their Ruby on Rails application and managing AWS with fifty AWS-certified engineers, when two people with Nix and Hetzner would do. Eventually, it comes down to money. Higher-powered tools are more efficient, and efficiency wins, especially as AI collapses margins.

ocaml, rust, haskell and back pressure

Has OCaml's time come again? It's still going strong. A certain trading firm is using it very well. When I caught up with Yaron at the start of the year, I asked him whether OxCaml exists so their language extensions end up in the training data and lift the whole company. I got a very "no comment" smile.

For agents, OCaml is lovely. Functors between modules are beautiful. The .mli files, a typed header explaining how a module should work, are really efficient context for agents. opam and Dune are legitimately good. Hindley–Milner. And compile times are fast. I see no reason to do F# these days.

Justin has mostly been writing Rust, and agents are good at Rust. But compile time is the tax on back pressure. LLMs hallucinate, and when compilation is slow, each hallucination is expensive because you get fewer attempts per minute. Justin's S3 clone is about a million lines of Rust; with four agents compiling at once, they fight over disk and CPU. You end up spending more on fast machines than on tokens.

Haskell's type system is great, and the models do it really well. But I don't feel good running it in production: a space leak lives in the runtime state space and only shows up in production. Justin pointed out that the linear-type ideas in Rust came out of Haskell papers trying to solve exactly that. Then there's Zig's approach: allocate everything up front and never allocate again, which is what game devs did in the eighties and nineties. It's hard to persuade an agent to do that in Rust, though, because constant-memory programs aren't in its training set.

I think dependent types are the winner for next-generation languages. Anything that lets you codify more into the type system is more back pressure. You probably won't be surprised to hear I've got a fork of the Rust toolchain with dependent types. You can just do things now.

languages for agents, and what cursed taught me

The pace of language development has been held back by how fast humans can learn new concepts. Operator chaining is essentially sugar for humans. If agents write the code, we can lean on forty years of academic PLT research, as long as you know how to sample it.

The industry codified "do not make breaking changes" after Python 2 to 3. Justin knew companies with hundreds of people on that migration for years. I think that rule is no longer true. Ship a skill pack with the breaking change and let agents auto-migrate.

Justin asked what it actually costs to make a new language successful now. Go was the last language a company spent real money on, and it took a long time. I can answer that one.

Cursed was built with Sonnet 3.5 and 3.7, a deliberately underspecified prompt, and three months of running it in a loop. I started in C (not enough back pressure; I wasted too much time in Valgrind as the agent clobbered its own updates), then Rust, then Zig. Zig was a mistake; it would work today if I'd stayed with Rust. It cost roughly US$6,000, and I did it three times over. Compare that to what Go cost.

Now the real bit, the part that still scares me. If you allocate the context window correctly — a lookup table of the lexical structure and grammar — the model can program in a language that isn't in its weights. It's brute force and inefficient, but it works.

Think T-diagrams (tombstone diagrams). Lock down your grammar and lexical structure, reach a stage-two self-hosting compiler, ship a sensible standard library, and start the next training run. From there, you can reach a Roslyn-style self-hosted compiler with language services stupidly fast. Justin asked whether fine-tuning an open model would help bootstrap a language like this. It's not needed.

That was true a year and a half ago with much weaker models. It'll take just one programming language designer going all in with the good models to shock the world.

what next

As the pub was shutting, Justin asked what was on my mind. If you haven't read my latest post, go read it. If you manage people, create the space and time for them to experiment now, because within six months, leadership will ask you to put people on a vitality curve.

Yes, the labs trained on the commons. I hate that, and I get it. But you trade time and skill for money; employers have minimum standards, and those standards have changed faster than ever before in our industry. Be curious, learn how to build an agent, and go create beautiful stuff. We're in a renaissance.

It's a time-compression device. The more experience you have, the more you can sample. Not everything ships; some of what I showed Justin may never see the light of day. I use these projects as katas and redo them when the models get better.

But if you want to build something secure, seriously consider unikernels.

chapters

• 0:21 — discovering unikernels: Haskell to OCaml to Mirage
• 0:55 — what a unikernel is, and why it was hard
• 2:45 — hard was past tense
• 3:33 — why do we have an operating system?
• 4:17 — the shell is a butler service for exfiltration
• 5:12 — attack surface reduction is fuzzy
• 7:26 — don't put the compiler on production
• 9:13 — porting Stripe into a unikernel
• 9:54 — minimal Linux and mkfs.xfs in Rust
• 12:22 — S3 for everything
• 13:48 — Nix, machine tests and overlays
• 15:28 — tool-calling a human is not AGI
• 15:53 — seL4 or unikernels
• 16:56 — privilege rings
• 18:24 — weekly kernel patches and the KVM escape
• 19:38 — demo: the Mirage folder, NTP and time
• 22:05 — Spaceleans: Orleans in OCaml as a unikernel
• 25:31 — sampling history
• 26:52 — how to get people to try this: just do it
• 28:57 — OCaml and OxCaml
• 31:45 — Rust compile times and back pressure
• 33:06 — Haskell, space leaks and dependent types
• 35:13 — memory strategies: Rust, Zig and game devs
• 36:10 — languages designed for agents
• 37:41 — breaking changes and skill packs
• 38:53 — Cursed
• 42:36 — what Cursed taught me
• 46:47 — what's next: AI use is mandatory

Keep curious.

When Ralph was published last year, I sat on it. People who have been reading this blog for a while know this. I first came across the idea of loop engineering in early January last year, sat on it for four months, and then started writing a lot, begging people to pay attention.

When Ralph and Loop Engineering went viral earlier this year, almost a year later, it really weighed heavily on me. Life's been pretty good lately, but it wasn't always historically, so I took some time off. I spent six months traveling around the world with no income, and I delivered the talk below almost 17 times in various cities.

Our profession is at a crossroads, and this is the last time I'll say it.

Usage of AI in the profession of software engineeering is no longer optional.

If your employer has banned the use of AI, you should quit and find a company that is already going through an AI transformation.

If you're at a company going through an AI transformation, I understand you'll have mixed feelings about this. You might not like it. But please understand that this is the new normal. Artisanal handwritten code is only possible for retired tech executives and as a hobby at home, but it's no longer a way to make money.

You really have two choices as an employee.

You can push against the grain and refuse to use AI, but that's just self-harm. You trade time and skill for money. The tastes and desires in the employment market have moved on.

From my position, I have a clear view of what's going on.

Every morning, I wake up to either a WhatsApp message or a LinkedIn DM from people who are going somewhere, sending their thanks. Like, don't thank me; thank yourself. You invested in yourself, and because you did, you just got promoted to a vice president role. You did all the work.

Hi Geoffrey. I wanted to drop you a message to just share what a profound impact Ralph’ing had had on me and the company i work for. I first heard of it back in Jan 2026 on social media and thought it was an outrageous concept, the idea of letting this clumsy persona rip through my codebase, so i just ignored it. My biggest frustration with AI has always been it never bought exactly what I wanted to life, it would get the first parts right, then it starts cutting corners, and I end up with a real mess at the end. That was until when I watched sonnet 5 loop through and churn out a product in about 6 hours, so for whatever reason I decided to come to your blog and read about Ralph directly - and I’d totally misundertood it. It took quite a bit of experimenting, however im now at a place where I’m literally delivering paid for dev worth 100K + for 3K in sonnet tokens in brownfield projects. Im re-platforming entire stacks in about 2 weeks that we used to pay contractors half a million for - the game has totally changed, and the inplications are both amazing and terrifying at the same time. Anyway, just wanted to say thank you, because at least for me and the company I work for, you publishing the Ralph paper has forever changed the game, and I hope that one day I’ll get to buy you a drink to say thank you. All the best.

So it's time to address the elephant in the room: people managers.

If your team is too busy doing their new job to experiment with AI, you're preparing them to be replaced. Maybe you're okay with this, but hopefully you're not, because it should weigh heavily on you if you truly care about your team.

Sometime in the next six months, you're going to have to start having some hard conversations about moving people on from your company. You should already be putting your employees on a vitality curve and ranking them. Naturally, competent employees who have been using AI will be ahead of competent engineers who haven't.

If you don't want this to happen to your team's members, start by carving out time for professional development. The most concrete task I can recommend is for software developers to build their own agent, learn the magic tricks behind Claude Code, and explain how it works under the hood.

You see, you're not a senior software engineer in 2026 unless you can rebuild Claude Code (the most basic of agents). Senior software engineers are senior by definition because they can mentor and teach the incoming generation. There's a lot to teach about the craft of software engineering; there's a lot of information and experience about software failures and how software should be designed, colloquially known as taste, that the incoming generation will need to be mentored in, but you won't earn their respect if you have an attitude that AI is evil and should be avoided. If you can't earn the respect of the incoming generation because of your poor attitude, what value do you bring, and why should an employer keep you employed?

interviewing as an employee in 2026

If you're interviewing as an employee in 2026, understand that there is a narrative that it's hard to get a job and it's pretty tough out there. This is true if your expertise is limited to typing code into an IDE, but it is not if you have provable expertise in AI.

If you've got something to show, like a talk, an open-source project, or anything tangibly demonstrable that you've been curious about the transformation in our industry and have been investing in yourself, you should be able to land a job in a couple of days in this market, as it's cooking right now.

Please understand that your edge right now is understanding the possibilities and being curious. The world lacks curious, ambitious people. That's your edge; that's your alpha. Position yourself as that, and be way more ambitious than you are right now. You need to think bigger.

At whichever company you end up at, be an ambassador for AI across your team and organization (and by that I don't mean you should be a maximal AI glazer). I mean, share what works, experiment, and help others through this transition. Generally, just raise the bar for everyone around you and be a good human.

being a hiring manager in 2026

It's been almost two years since my oh-fuck moment. Since then, we've seen the rise of "model-first" companies where most of the software is generated by computers, not humans.

This means that two years have passed, and those who have been curious and invested in themselves have one hell of an edge over someone just getting started today.

One of the most fundamental questions I’ve been pondering on over this last year is “how do you identify skilled operators” aka “how do we even hire software engineers now now that LLMs can smash leetcode?”

I propose the following categories and buckets as a crude ruler on top of your standard software engineering knowledge tests.

• unacceptable
  • A poor attitude toward willingness to use AI. I get it. The labs ripped off society's commons, and I don't like it either, but it has happened. You should sniff out these perspectives when interviewing a candidate and filter candidates out through behavioral questions.
  • Any form of deceptive behavior. If they're using AI in the interview process to deceive, that's an instant no-hire because deceptive behavior is unacceptable.
  • No tangible, demonstrable experience with AI.
• acceptable
  • You should be able to pull them over to a whiteboard and ask, at a high level, how a coding agent works under the hood. This is a baseline curiosity test to see whether they've been paying attention and developing their skills over the last couple of years. You want to understand whether they know what a context window is, how tokenization works, and how inference works from a systems-design perspective. If their knowledge depth taps out and all they can do is talk about skill packs, then pass on the candidate. Employers should prioritize producers, and producers know the baseline underlying data structures and system design of how an agent works because that knowledge lets them automate a business function. If they don't have this knowledge, they're just a consumer, so you should likely pass on them until you find someone with producer knowledge.
• ideal
  • Tangible and demonstrable experience with AI. They should have a whole bunch of side projects they have built; this may seem unfair, and I agree. Before AI, this was an unfair standard, but these subscriptions have been available to everyone. They're subsidized, and everyone I know who understands what's going on has so many cool things they're dying to show people.
  • When you ask the question about what coding harness they use, like, there are only really two acceptable answers. One is that all coding harnesses are fungible, and they use whatever they can to get tokens the cheapest, or their eyes light up and go, I use my own. Do you want to see it? Do you want to nerd about it? Like, here, I'll show you. This is the modern age of the age-old interview question: "Tell me about your home lab"
  • If you walk away from the interview and learn something about AI you didn't know, you should hire that person on the spot before someone else does. This is the modern age of the age-old interview question, "What happens when you type google.com into your browser's address box and press enter?" but with an AI twist.
  • If they lead people, they should be able to go really, really deep on the processes that failed because of a company's AI adoption. They should also cite examples of what they personally changed within their organization to enable AI adoption. You should background-check these stories to ensure people aren't telling porkies.

There's something else that I need to put forward. Your existing hiring pools may need to be wiped because your current interviewers may lack the expertise to differentiate between these categories. For example, if someone has a poor attitude toward AI and no provable experience with AI, and they're not in the ideal category, remove them from your interviewer pool (and/or company), because you need to put your best foot forward to identify, attract, and hire the best talent out there in 2026.

Remember, the candidate is also interviewing you. They know what's going on, and they want to ensure that choosing you as an employer de-risks their lives.

ps. Watch this podcast with Bill Gates and internalize it

https://www.nytimes.com/2026/09/29/opinion/ezra-klein-podcast-bill-gates.html

Well, Jevons is just a referral to the fact that parts of the economy are subject to demand elasticity. And yes, in the case of software, if you’re, say, three times as fast and there is still some role that only humans can perform, then as you lower the cost, you induce demand. So it’s fair to say that the equilibrium today so far for software is not a loss of employment.

When we invented radial tires that lasted four times as long, for some weird reason, people didn’t drive four times as much. And factories that make tires employ a quarter of as many people. When you replace people in Amazon warehouses with robots, people don’t buy more because of that.

So anybody who’s numeric can say to themselves: What portion of the economy is subject to demand elasticity, and what are those tasks that will still be human-necessary?

As soon as you’ve completed the entire task, it doesn’t matter that there’s demand elasticity. That goes into the token budget. It doesn’t go into the human salary budget.

The cost for some of these things is so much less than the cost of the human labor, and so as you cross reliability thresholds, both with white-collar and humanoid robots, you destroy jobs and you leave no high ground.

Innovation in the past — you have a tractor. Fine. Let’s build Disneyland and employ a lot of people there. You don’t have that in the broad economy. The people who say there will be net additional jobs, I don’t understand what they’re thinking.

pps. some extra data for engineering managers

AI is no longer optional. Every leader we interviewed said, one way or another, that refusing to use AI is now disqualifying; the comparison we heard more than once was refusing to use an IDE. The gains are real. Organisations told us about three times the output per engineer, a six-month project delivered in two, an incident diagnosed in seven minutes that would once have taken four hours. What changes most is what's possible. The default answer to a new idea is no longer "no", it's "let me check", and work that used to die at the bottom of the backlog now gets done.

From https://www.lookahead.com.au/reports/ai-engineering-2026/lookahead-state-of-ai-engineering-2026.pdf

‼️ ppps. socials.

Can you please boost and share this blog post with someone you think needs to hear it? It is both the most and the least we can do. We can't force a horse to drink, but we can tell horses that water exists, and that they need water to be employed in 2026...

• Twitter
• LinkedIn
• BlueSky

Here's something I've been meaning to write up for a while. It's kind of late, but I'm going to lean into the whole MCP vs. CLI argument because I have a perspective others might find interesting.

What's the actual difference between an MCP server and a regular old CLI program with a `--help` flag? Feels to me like a CLI program written with agents in mind solves the same problems without inventing anything new?
https://x.com/adamwathan/status/1968738637358522629

Developer tooling companies can be sorted into two tiers: S-tier (winning) and F-tier (losing). The Rubik for wherever a company falls into the S tier or F tier has got everything to do with the model weights.

An S-tier company is in the training weights. Let's take GitHub, for example. Now, we can yak about problems with GitHub till the cow comes home, but one thing is true: these models know how to interface with GitHub because that behavior is in the model weights themselves. They don't need any skills or an MCP to get outcomes. You can just use the CLI.

You are S-tier once you can do a prompt like this without any skills or MCP:

prompt: Use the GH cli to create a repository, commit the changes, push to origin, set the reposistory to public and update the repository description.

Now, if you're a brand new developer tooling company and you're not in the model weights yet, you've got a couple of choices. First, you'll have to ask users to download and install some sort of skill pack, or maybe even a CLI, which adds a lot of friction for users compared with the DevX of an S-tier company.

So, what do you do? It's simple.

Start by developing a really good CLI, and on that CLI, work on and optimize your --help section. What you want to do is measure whether a model can achieve outcomes by walking the help verb across all the subverbs, then optimize the language used by choosing whatever works best, so the model can achieve the outcomes with the fewest tool calls. This can be automated through a loop or an auto-research loop.

Once you have this bare-bones foundation, the next step is to publish documentation for this CLI online and then focus on ASEO (Agentic Search Engine Optimization) for agents web_search_tool. Again, from this point forward, the goal is to look at the journey a single agent takes and optimize it so the agent can get all the information it needs in a single web search tool. If it has to walk through your documentation, you're not doing a good enough job. Keep optimizing.

To take this to the next level, you should enter into a business contract with the labs. Perhaps you've already got a contract with the labs for inferencing. You want to lean on them to include your CLI documentation in their next training run.

If you nail these things, then you're well on the path to becoming an S-tier developer tooling company. The best part is that it doesn't require users to download or install a skill pack, configure an MCP, or deal with any of that junk.

ps. socials

I don't know if you've noticed yet, but JetBrains reported a net loss for the first time in its recorded history.

Before we go into this, let's rewind time to January 2025, nearly a year and a half ago, when I penned these words.

So, I've been wondering what the next generation of IDE looks like in this post-LLM world. Why do I need to have multiple IDEs open? What if we ripped out the "collaborate with humans" functionality from Zed and inverted it so it's "collaborate with LLMs," and an unbounded number of LLMs could be connected to the IDE. Even then, that might not be radical enough of a UX/design overhaul. Maybe we don't even need an IDE? Maybe Devin is onto something...

I remember writing these words as a tech lead at Canva, with coworkers calling me mad and saying, "Geoff, what do you mean?" I love JetBrains. I love my IDE. There's no way in heck this is changing; JetBrains will figure it out."

I'm not salty about it; it's just a fact. You get used to these things when you're early. If you're curious, you get used to the friction from people who aren't used to early, informed insights. To me, it was plain and clear that IDEs were dead and we needed better tools.

When I wrote the above, I was in the background enumerating JetBrains at all levels through our customer channels about what their roadmap would be, how they would counter this risk in business, and it's ultimately really sad to see that nothing was done other than jack the price of their subscriptions whilst simultaneously delivering a low-quality product (ala "private equity/broadcom subscription revenue extraction phase")

They essentially Kodaked themselves.

JetBrains could only see the world through the lens of the IDE, and they hoped that AST exposed by a language server would be a source of alpha even though everyone else in the agentic coding market was getting ripping results without symbolic context injection. Their experience and domain expertise became their curse, keeping them from changing and adapting to the market.

And that is the deepest lesson. From this point forward, I'm not going to mention JetBrains, but I'm going to focus on the lesson. Every company is blinded by its domain expertise and how it makes revenue, and in times like this with AI, it's important to stay highly neuroplastic and check your blind spots.

A good example of this is the labs. The labs deeply believe that a future version of their model will solve software verification because tokens shape their worldview and perspective. This belief blinds them to the fact that a better model will not solve the problem of verifying that an application will behave correctly in production under suboptimal conditions (i.e., flaky network connections)

My strongest advice to all business leaders right now is to look at your own dogma, how you view the world, and make sure you don't Kodak yourself in this super weird moment of hyper transformation.

ps. socials

A few days ago I wrote about Rewind VM, a deterministic VM where every run of a Nix build is a pure function of its inputs, thread schedule included!

I have been using it to find, reproduce and solve numerous race conditions in Nix builds, but it’s been making me feel a little crazy. What is this superpower, and why is nobody else using it?

The tool has quickly grown a source panel, stack frames, bookmarks, a “Compare” tab, a lot more gdb support, thread lanes, which show who held the CPU at every step, and “Check from here”, which helps find the exact step where a race condition happens.

I went into each new feature thinking it would be a big code-lift but I kept running into the same thing: the hard part of each feature was already done, and Nix had done it.

A debugger for instance, needs the exact inputs of the program, its debug symbols, its sources, the sources of every library under it, and a way for someone else to get all of that on their machine. That’s exactly what a derivation is, and Nix makes that easy.

Two tellers, one account

A classic simple example of a race condition is two threads depositing into one bank account. Each deposit reads the balance, writes a line to the ledger, then stores the balance plus the deposit. If one teller runs between the other’s read and store, it writes a stale balance over the other’s deposits. 💥

static void deposit(int teller, long amount)
{
  long seen = balance;
  char line[64];
  int n = snprintf(line, sizeof line, 
                   "teller %d: %ld + %ld\n",
                   teller, seen, amount);

  if (write(1, line, n) != n)
      return;
  balance = seen + amount;
}

A teller that runs between the other’s read and store writes a stale balance over the other’s deposits.¹

Rewind’s VM has one CPU, so its first run passes too. rewind check runs the build again under perturbed schedules, each asking the guest kernel to reschedule at different steps, and narrows the first failure down to one step:

$ rewind check --where github:fzakaria/rewindvm#bank
...
step 3237 decides it: a reschedule there makes the run fail

passing: run 12feb5f832205a72, schedule 0
failing: run c2f1cfac0c288f3e, schedule 1 over steps 3237..3238
the two are the same run until step 3237

where the threads were:
        3237   139/140   deposit (bank.c:24), on the CPU at the deciding step
        3250   139/141   deposit (bank.c:24), at the first event that differs

Running this check on my laptop took 11 seconds. The two runs are the same machine, exit for exit, until step 3237, where only the failing one gets a reschedule.

Free thing one: the inputs

The argument to check is a derivation and can be a flake reference. The beauty of Nix is that it knows the inputs of a derivation, so that’s all we need to make sure the run is reproducible. The derivation’s inputs are the source, the compiler, the libraries, the kernel, and the VM’s configuration

rewind nix realises the derivation’s inputs, packs the closure into a read-only erofs image, and boots the VM on it.

The run’s id is a hash of its inputs, the way a store path is, and rewind show prints the command that makes it again:

$ rewind show 12feb5f8
rewind nix /nix/store/cr8rl40rjb9cmmcd9sxn9mpdrhvp53jj-bank-0.1.0.drv --epoch 1791331200 --clock branches --name bank-0.1.0  # 12feb5f832205a72

When the build succeeds, the guest reports each output’s NAR hash, and rewind nix checks it against the host’s copy and against every binary cache Nix substitutes from, by fetching only the .narinfo:

$ rewind nix github:fzakaria/rewindvm#bank
...
/nix/store/27a4q7w90jzp4wg5kaaajvb64zll6yqg-bank-0.1.0 dbf0df3de84b7a84  matches your store

This helps us validate that the build within the VM is the same as the build on my laptop, and that the VM’s run is reproducible by Nix.

Free thing two: every symbol and every source

Debugging code is much simpler when you are looking at the source. A new panel in the Rewind app or rewind where in the terminal shows the source of the program at the playhead, and the stack frames that called it.

The debugger needs to know where the sources are, and Nix gives us that for free too.

nixpkgs builds the packages with separateDebugInfo, and the debug info is cached on cache.nixos.org as a debug output. We can leverage debuginfod to fetch the debug info and the sources by build ID, so we can see the source of any binary in the VM, including the Linux kernel! 😲

$ rewind where c2f1cfac 3250
process 139 (bank), thread 141, at step 3250
#4 deposit (bank.c:24)
      22  	int n = snprintf(line, sizeof line, "teller %d: %ld + %ld\n", teller, seen, amount);
      23
>     24  	if (write(1, line, n) != n)
      25  		return;
      26  	balance = seen + amount;
called from #5 teller (bank.c:34)
called from #6 start_thread (pthread_create.c:454)

gdb, on a fork of any step

Sometimes though, the source is not enough, and we need to see the state of the program. rewind gdb opens gdb on a fork of the run at the playhead, with every thread of the process. The debugger can set breakpoints, watchpoints, and inspect memory, registers and variables.

At step 3249, just before teller 2 gets the CPU back, we can watch the balance variable and continue until it changes. Nothing gdb does changes the recording, and we can rewind to the same step and fork again, or fork at any other step, and gdb will see the same state.

If gdb is not enough, rewind shell --with nixpkgs#strace opens a shell in the VM at a step with any package from nixpkgs on its PATH. It is one more closure packed into one more image.

Compare two runs

Rewind makes it very easy to compare two runs. The Compare tab in the app, or rewind compare in the terminal, shows the last shared events of both runs, and the first event that differs.

The Compare tab puts both runs’ events side by side from just before they part, with what differs marked. Teller 2 starts from 150 in the failing run and from 200 in the passing one:

Who had the CPU

Rewind’s VM has one CPU, so at any step exactly one thread is running. A race is a question of order: which thread ran when.

The trace does not answer that directly. It records what threads did, a write, an open, a fork, an exit, a signal, but not who was running in the gaps between those events.

Rewind can fill in the gaps without recording anything new. Every run replays exactly, so it can walk any stretch of a run one step at a time and, at each step, ask the guest kernel which thread is on the CPU.

In the failing run, teller 1 (Thread 140) is interrupted at step 3238, halfway through a deposit: it has read the balance but not stored it. Teller 2 (Thread 141) gets the CPU for a single step, 3239, long enough to read the balance, and then teller 1 gets it back and finishes.²

Check from here

Finding one bad interleaving raises the next question: how likely is it? Is the passing run the normal case, or did it get lucky?

rewind check normally answers that by building the whole derivation again under many schedules. With --run, it starts from a run you already have instead, at any step you pick_. It forks the run there once per schedule, each fork taking a different order of threads from that step on, and counts how many end differently. Everything before the step stays exactly as it was.

We can apply this technique to the bank example, starting from the step where the two runs diverged, 3,221. When we run 16 schedules from there, all 16 lose money³:

$ rewind check --run 12feb5f8 --schedule-from 3221 --schedules 16 --all --no-narrow
schedule   0: exited:0       4528 steps  dbf0df3de84b  run 12feb5f832205a72
schedule   1: exited:2       3313 steps    run f6eb462913dce489
schedule   2: exited:2       3328 steps    run 388cef73a9c545fa
schedule   3: exited:2       3312 steps    run 4d42012072d11537
...
schedule  16: exited:2       3314 steps    run 63f7817972a90273
16 of 16 perturbed schedules ended differently

We can do the same thing from the Rewind App with “Check from here” in the context menu of the playhead. It forks the run at the playhead and runs each fork under a different schedule, counting how many end differently.

Bookmarks

b bookmarks the playhead’s step with a note. Bookmarks are kept with the run, and they travel in its .rwd export, so the person I hand the run to opens it with my notes on the timeline:

Other examples to try

Each one is a derivation in the flake, with one bug and one fix:

• philosophers: the deadlock from the last post.
• bank: the lost update above.
• waiter: a SIGCHLD that arrives between a flag check and pause, so the parent sleeps until make check’s ten second timeout kills it.
• config-reload: one process rewrites a config file in place while another rereads it. On many cores it fails nearly every time; on one CPU only some schedules land the reader between the truncate and the last write.

$ rewind check github:fzakaria/rewindvm#waiter

Nix is a superpower

Most of the hard work of a debugger is already done by Nix. Rewind adds a little more, but the rest is already there: the inputs, the sources, the debug symbols, and a way for someone else to get all of that on their machine.

When you start with something hermetic like a Nix derivation, you can get a debugger for free.

$ nix run github:fzakaria/rewindvm -- check github:fzakaria/rewindvm#bank

There are many things that are uncertain in our industry right now, but one thing I am certain about, and have been for almost 13 years, is that Nix is a terrible programming language.

I remember when I first learned it, it felt like pushing shit uphill. The learning cliff is ferocious, and it took me a couple of years to master it because there was no AI back then, but I stuck with it. There's some magic here, and in this post, I'm going to show you why Nix should be your primary choice for all your software projects.

First, I'm going to open with this: many pieces of technology are in our heads as really hard to learn, complex, and maybe that's stopped you from picking one up, but I want to encourage you to falsify those thoughts any time you start thinking along those lines. Now that we have AI, things that used to be advanced power tools, hard to use or designed for masters, are now accessible to everyone. You can just prompt for outcomes.

first, some introductory knowledge

The term Nix is overloaded. It means many different things; when someone says they use Nix, the first thing you should do is ask, "How do you use Nix?" and "What is Nix to you?" because there are many ways to use it; it's not just a package manager or a build system; it can also be an operating system.

In this post, I'll focus on Nix's versatility and utility, and why it's so powerful in the age of AI. There's a reason the labs are using this to build the models you are consuming...

agent and developer experience

By using tools like https://devenv.sh/, you can define a single source of truth for your compilation toolchain and required third-party dependencies. Humans can use this single source of truth in CI/CD and in ephemeral sandbox development environments used by agents.

# visit https://devenv.sh/getting-started/ for installation instructions

$ vi devenv.nix

  # devenv.sh/packages/
  packages = [ pkgs.prek ];
 
  # devenv.sh/languages/
  languages = {
    rust.enable = true;
  };
 
  # devenv.sh/services/
  services = {
    postgres.enable = true;
  };
 
  # Configure and install the Rust formatting Git hook.
  git-hooks = {
    hooks = {
      rustfmt.enable = true;
    };
  };

$ devenv up

I keep coming across clients with drift between how a local laptop is configured and how their CI/CD is configured. And now we've got ephemeral sandbox environments; they're heading down a path of triplicating that drift. This is utter madness. It is not needed. Stop it.

You might be thinking, "Well, there's mise for this." Trust me, mise isn't good enough. Mise is a low-power, low-IQ tool. It does one thing, and it does it well, but it doesn't enable your agents to truly fly.

not just a package manager

nix is super composable. That same expression that defines your human developer environment setup, your CI/CD setup, and your ephemeral sandbox setup can also be reused to build performant Docker images.

pkgs.dockerTools.buildLayeredImage {
  name = "git";
  tag = "latest";
  contents = [
    (pkgs.buildEnv {
      name = "image-root";
      paths = [ pkgs.git pkgs.cacert ];
      pathsToLink = [ "/bin" "/etc" ];
    })
  ];
  config.Env = [
    "PATH=/bin"
    "SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt"
  ];
  config.Cmd = [ "/bin/git" "--version" ];
};

But perhaps... the real reason I love Nix is its power to compose an operating system. With a standard operating system such as Debian or Ubuntu, what happens if an agent is given sudo access to do things on your machine?

You'd be pretty scared, right?

What if I told you that when I'm doing agentic development, I develop on NixOS and I explicitly prompt my agents to use sudo as part of my loop engineering, and it is safe because NixOS is designed to make it nearly impossible to break a machine, and if it does, you can instantly roll back that change.

why the operating system matters

When others do loop engineering with their agents, they're likely running loops with building the application, perhaps Postgres, but rarely anything more. The primary difference between how others do loop engineering and how I do is that, when I run loops, I put the entire system under test, and the entire System Under Test IS the operating system.

You might think, "Whoa, what the —" Like, how the hell do you do this?

ah! My sweet summer child. It is really simple.

NixOS has a testing framework (runNixOSTest), built in, and for anything you could ever want to assert about how an operating system is configured, or even across many machines, you can spin up a cluster. You can use a number of machines in a test, assert the network rules between them are correct, and that the right IP tables are forwarded or dropped. You do machines of machines, and you actually test the interoperability of that environment against your application.

Here's an example of what this looks like when you've defined an operating system in NixOS and want to assert the network is configured correctly with your firewall zones, and that network activity works for your application before you deploy it.

# NixOS VM test: HAProxy in front of a Python hello-world server.
#
# Topology (two QEMU nodes, separate L2 networks):
#
#   vlan 1  192.168.1.0/24   frontend
#     haproxy  eth1  192.168.1.1/24     HAProxy binds :80 here only
#
#   vlan 2  192.168.2.0/24   backend
#     haproxy  eth2  192.168.2.1/24     allowed source
#              eth2  192.168.2.50/24    extra address, must be rejected
#     web      eth2  192.168.2.2/24     Python hello-world on :8080
#
# NixOS test IP scheme is 192.168.<vlan>.<nodeNumber>. nodeNumber comes from
# the sorted node name, so haproxy = 1 and web = 2.
#
# web firewall:
#   - inbound: accept TCP 8080 only from 192.168.2.1, drop everything else
#   - outbound: no new connections; established replies to HAProxy still pass
#
# Run:
#   nix build .#checks.x86_64-linux.haproxy-hello
#   nix run .#driverInteractive    # drop into the test driver
{ lib, ... }:
let
  frontendVlan = 1;
  backendVlan = 2;

  # Sorted node names: haproxy, web.
  haproxyNode = 1;
  webNode = 2;

  haproxyFrontendIp = "192.168.${toString frontendVlan}.${toString haproxyNode}";
  haproxyBackendIp = "192.168.${toString backendVlan}.${toString haproxyNode}";
  # Same backend VLAN, but not the address the firewall allows.
  haproxyUntrustedIp = "192.168.${toString backendVlan}.50";
  webIp = "192.168.${toString backendVlan}.${toString webNode}";

  frontendPort = 80;
  webPort = 8080;

  helloBody = "hello from python";
in
{
  name = "haproxy-hello";

  nodes = {
    haproxy = { pkgs, ... }: {
      virtualisation.vlans = [ frontendVlan backendVlan ];
      virtualisation.memorySize = 512;

      environment.systemPackages = [ pkgs.curl ];

      # Harness already assigns 192.168.2.1/24. This alias is the negative
      # control: same L2 network, source the backend must refuse.
      networking.interfaces.eth${toString backendVlan}.ipv4.addresses = [
        {
          address = haproxyUntrustedIp;
          prefixLength = 24;
        }
      ];

      networking.firewall = {
        enable = true;
        # Frontend lives only on vlan 1. Do not publish :80 on the backend net.
        interfaces.eth${toString frontendVlan}.allowedTCPPorts = [ frontendPort ];
      };

      # The haproxy module already opens a `global` section and installs the
      # stats socket, so this value must start at the next section.
      services.haproxy = {
        enable = true;
        config = ''
          defaults
            mode http
            timeout connect 5s
            timeout client 10s
            timeout server 10s
            timeout check 5s

          frontend fe_http
            bind ${haproxyFrontendIp}:${toString frontendPort}
            default_backend be_hello

          backend be_hello
            option httpchk GET /
            http-check expect status 200
            server hello ${webIp}:${toString webPort} check
        '';
      };
    };

    web = { pkgs, ... }: {
      virtualisation.vlans = [ backendVlan ];
      virtualisation.memorySize = 512;

      environment.systemPackages = [ pkgs.curl ];

      networking.firewall = {
        enable = true;
        # extraCommands is the iptables backend. Pin it so the rules below apply.
        backend = "iptables";
        allowPing = false;
        logRefusedPackets = true;
        # Nothing open on every source. The only accept is extraCommands.
        allowedTCPPorts = [ ];
        extraCommands = ''
          iptables -A nixos-fw -p tcp -s ${haproxyBackendIp} --dport ${toString webPort} -j nixos-fw-accept

          # No new outbound. Established replies (the HAProxy request path) still pass.
          iptables -A OUTPUT -o lo -j ACCEPT
          iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
          iptables -A OUTPUT -j REJECT --reject-with icmp-host-prohibited
          ip6tables -A OUTPUT -o lo -j ACCEPT
          ip6tables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
          ip6tables -A OUTPUT -j REJECT --reject-with icmp6-adm-prohibited
        '';
        extraStopCommands = ''
          iptables -D nixos-fw -p tcp -s ${haproxyBackendIp} --dport ${toString webPort} -j nixos-fw-accept || true
          iptables -D OUTPUT -o lo -j ACCEPT || true
          iptables -D OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT || true
          iptables -D OUTPUT -j REJECT --reject-with icmp-host-prohibited || true
          ip6tables -D OUTPUT -o lo -j ACCEPT || true
          ip6tables -D OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT || true
          ip6tables -D OUTPUT -j REJECT --reject-with icmp6-adm-prohibited || true
        '';
      };

      systemd.services.hello-world = {
        description = "Python hello-world HTTP server";
        wantedBy = [ "multi-user.target" ];
        after = [ "network.target" ];
        serviceConfig = {
          ExecStart = "${pkgs.python3}/bin/python3 ${pkgs.writeText "hello.py" ''
            from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

            BODY = b"${helloBody}\n"

            class Handler(BaseHTTPRequestHandler):
                def do_GET(self):
                    self.send_response(200)
                    self.send_header("Content-Type", "text/plain")
                    self.send_header("Content-Length", str(len(BODY)))
                    self.end_headers()
                    self.wfile.write(BODY)

                def log_message(self, fmt, *args):
                    return

            if __name__ == "__main__":
                ThreadingHTTPServer(("${webIp}", ${toString webPort}), Handler).serve_forever()
          ''}";
          DynamicUser = true;
          NoNewPrivileges = true;
          ProtectSystem = "strict";
          ProtectHome = true;
          Restart = "on-failure";
        };
      };
    };
  };

  testScript = ''
    start_all()

    haproxy.wait_for_unit("haproxy.service")
    web.wait_for_unit("hello-world.service")
    web.wait_for_unit("firewall.service")

    haproxy.succeed("ip -4 addr show dev eth${toString frontendVlan} | grep -q '${haproxyFrontendIp}/24'")
    haproxy.succeed("ip -4 addr show dev eth${toString backendVlan} | grep -q '${haproxyBackendIp}/24'")
    haproxy.succeed("ip -4 addr show dev eth${toString backendVlan} | grep -q '${haproxyUntrustedIp}/24'")
    web.succeed("ip -4 addr show dev eth${toString backendVlan} | grep -q '${webIp}/24'")

    with subtest("backend accepts only haproxy on the hello port"):
        haproxy.wait_until_succeeds(
            "curl -fsS --interface ${haproxyBackendIp} http://${webIp}:${toString webPort}/ | grep -F '${helloBody}'"
        )
        haproxy.fail(
            "curl -fsS --max-time 5 --interface ${haproxyUntrustedIp} http://${webIp}:${toString webPort}/"
        )
        haproxy.fail(
            "curl -fsS --max-time 5 --interface ${haproxyBackendIp} http://${webIp}:9/"
        )

    with subtest("haproxy on the frontend vlan proxies to python"):
        haproxy.wait_until_succeeds(
            "curl -fsS http://${haproxyFrontendIp}/ | grep -F '${helloBody}'"
        )

    with subtest("python host cannot open new outbound connections"):
        web.fail("ping -c 1 -W 3 ${haproxyBackendIp}")
        web.fail("curl -fsS --max-time 5 http://${haproxyBackendIp}:${toString frontendPort}/")
        # Return path still works: the proxy subtest above already required it.
        web.succeed("iptables -S OUTPUT | grep -q 'REJECT'")
        web.succeed(
            "iptables -S nixos-fw | grep -q -- '-s ${haproxyBackendIp}/32 -p tcp -m tcp --dport ${toString webPort} -j nixos-fw-accept'"
        )
  '';
}

There's an old saying that bare metal was complicated and it was a mess. This is no longer true now that we have Nix. Honestly, one of the most galaxy-brain moves you can make these days is to ditch hyperscalers like AWS and acquire bare metal to build that fleet with NixOS, because business margins are going to get compressed by AI.

Sure, you'll need to find an older, more experienced sysadmin, but once you get the right patterns in place, one or two of those people can operate with leverage equivalent to a team of 50 "cloud certified" monkeys. I shit you not, no exaggeration, it is insane what you can do with this composability.

single sources of truth

Nix, when architected correctly, enables you to define a single source of truth for any piece of software in your stack, including dependencies.

Let's say there is a security vulnerability, and there is, like, a critical OpenSSL vulnerability. How long would it take you to identify all the different OpenSSL versions deployed within your organization? How long would it take to patch, including all dependencies of third-party software that depends on that particular version within your organization, including software dependencies linked against it?

With nix, this can be achieved through a couple of lines

{ pkgs, ... }: {

nixpkgs.overlays = [
  (final: prev: {
    openssl = prev.openssl.overrideAttrs (old: {
    version = "3.0.7";
    src = prev.fetchurl {
      url = "https://www.openssl.org/source/openssl-3.0.7.tar.gz";
      hash = "sha256-...";
    };
  }

what you just saw was an overlay

An overlay is the key primitive which makes Nix absolutely amazing. I want you to think about how you use open source software. What happens when you find a bug? Do you go to GitHub and ask someone to take your pull request? Or maybe you fork it. But when you fork it, you have to figure out how you'll build it and where you'll host the resulting artifact (ugh, Artifactory) so it can be consumed.

I'll show you a concrete example. It's here on GitHub. It shows a pattern I've used for years: in my agent sandbox environments, I customize the Git binary and remove the agent's ability to force-push by removing that functionality from Git itself. It also includes all the patterns shared in this blog post, from building a Docker image to building a NixOS virtual machine and testing that git push --force has been removed.

What happens if what you need to customize isn't just a standard command-line tool? What happens if it's a low-level thing, like OpenSSL, that many things depend on, and you need to rebuild the world? Think about that graph. Think about the pain.

With Nix, every change is just an overlay; all software becomes infinitely customizable at all levels, all the way down to the Linux kernel itself, and you can patch anything by asking an agent to customize it. The only restriction on what you can do is your imagination and ambition. These LLMs know Nix very well because the labs themselves are using Nix on their journey toward reaching RSI...

ps.

You might be wondering about Bazel and Buck2. I also use these alongside Nix, but the best bang for the buck right now is just to use Nix. There are a lot of problems with Nix, for example, how the derivation store works and what that means for incremental caching, something Bazel and Buck2 excel at, but I'm not going to complicate things right now by going into these details. Move past this and just start using Nix to customize all your software.

pps.

If you are building security-critical systems, burn tokens to find every security vulnerability in third-party dependencies your software stack relies on using a cyber model, and fix them with overlays. 🧠🧠🧠

pps. socials

It's kind of strange seeing all these discussions about software factories... and the like. It's also strange to see conversations about programming languages or claims that $language is best, as if that will remain true going forward. It's very clear, however, that programming languages will converge toward something, but that 'something' is undefined for now.

What I haven't seen is people really deeply understanding the power of the new substrate that we have. People are still too fixated on what they have now and how systems have been built to rethink fundamentally how much things can change.

To me, a software factory isn't just about process automation; it isn't about automating everything you've got as it is now. It's about using this substrate so you can develop your product while it runs whilst in the product itself.

Everyone (regardless of their discipline or background) in the company should be able to develop the product in the product without having to go to some external vendor supplied tool.

The only tool that exists is your product itself. The product should build the product from the product.

In a future blog post, I'll go further into what it means for the product to develop the product, in the product, but for now I have one simple thing for you to imagine:

Why is software built the way it is now, rather than grown through iterative use of LLMs? What if you could develop your application just by chatting with it? Live and interactive with no compilation steps.

To demonstrate this, I built Jiti, a small kernel for growing a running Lisp application through conversation with an LLM. You ask for a capability, the model writes Lisp, and the application permanently acquires that capability until you ask for that capability to be removed.

The kernel supplies the machinery to inspect, change, execute, and recover a managed Lisp world. Application behavior comes from whatever you decide to add to the application by prompting for those outcomes. With Jiti, you can start from a clean slate or modify an existing application to add more behavior just by prompting it.

The source code is available on GitHub, and I encourage you to run it and play around with it. It is very generic, in the sense that it can do literally anything you want. All you have to do is ask, and it will program that capability into the application.

How it works is relatively simple. At a high level, an OpenAI model receives your request, instructions for operating the application, tool definitions, and observations of its current state. The kernel can ask to inspect the available functions, read a definition, propose new source, or execute an expression.

Once a definition is accepted, it is an ordinary Lisp function. Calling it does not inherently require another inference request. You can call it from Lisp, from another application function, or through the chat interface.

The OpenAI model is used to extend a program, while the running Lisp world holds the resulting functionality.

The agent tool interface has two useful intentions.

• develop_form adds, redefines or removes functionality.
• execute_form calls functionality that exists, including combinations of existing functions.

Both use the same evaluator and transaction machinery. The distinction helps the model choose whether the user is asking to change the application or simply use it.

Suppose I've already asked the application to add uppercase-string and reverse-string. I can then prompt the kernel to uppercase some text and reverse the result. The composition is just Lisp:

(reverse-string (uppercase-string "Hello"))

If I want that combination as a reusable capability, I can prompt it to save that as a function.

(defun shout-backwards (text)
  (reverse-string (uppercase-string text)))

That function joins the catalog with its arguments and source, ready for inspection and later use. A future request can discover it and build on it. The application accumulates an executable vocabulary through use. Lisp already gives us the composition rules; the kernel keeps the evolving definitions available and their managed effects accountable.

Lisp deserves the credit for the interactive programming machinery. Definitions, inspection, conditions, and restarts have been there for decades. It's kind of cool, huh?

To me, the idea that an agent writes source code and then there's a costly compilation phase involving CI/CD is now truly undefined now that we have AI. The only limiting factor will really be people's curiosity about what they can do with this new substrate...

ps. socials

If a test fails on CI and nobody can reproduce it, did it really fail? 🧘

Nix gives me a build that is a function of its inputs via the extensional model. The same derivation, produces the same store path, and if I am lucky, the same bytes. What Nix does not give me is the same run.¹ A test suite with a race in it may pass on my laptop, fail once on CI, and when I rebuild it to look, it passes again.

If you have experienced bugs like this, you know that it can be frustrating. You are effectively at times trying to find the needle in the haystack.

I wrote recently that the Nix sandbox is a hidden input to a derivation. The same is true of the thread schedule. The order the kernel happens to run your processes in decides whether some builds pass, and nothing within the derivation records it.

These are the hidden inputs to a build that can make it flaky.

Are we left to hoping that we will find the needle in the haystack? Or is there a way to make the run a function of its inputs too?

I built Rewind VM to make the schedule an input too. It runs a Nix build, a test suite or any Linux command inside a KVM virtual machine whose every run is a function of its inputs. The same inputs give the same run, at the same steps, every time. You can replay a failure, scrub through it, read any file as it was at any point, and fork it under a different thread interleaving. ✨

Confused? Yes it sounds like magic. The best way to explain it is with short demo.

A textbook deadlock

Let’s investigate the dining philosophers problem. The problem is that the five philosophers sit at a round table with a fork between each of them. A philosopher needs both forks beside them to eat. A philosopher may pick up one fork at a time.²

/* The fork on the left first, then the fork on the right. */
int first = left;
int second = right;

/* Eat MEALS times, holding both forks for each meal. */
for (int meal = 0; meal < MEALS; meal++) {
  pthread_mutex_lock(&forks[first]);
  printf("philosopher %d picks up fork %d\n", id, first);
  pthread_mutex_lock(&forks[second]);
  printf("philosopher %d picks up fork %d and eats\n", id, second);
  pthread_mutex_unlock(&forks[second]);
  pthread_mutex_unlock(&forks[first]);
}

If all five pick up their left fork before any of them reaches for a right one, every fork is taken and every philosopher waits on a neighbor who is also waiting. Deadlock.³

On my laptop 14 of 100 rebuilds deadlocked. 💣

$ nix build --rebuild -L github:fzakaria/rewindvm#philosophers
...
philosophers> philosopher 3 picks up fork 3
philosophers> philosopher 0 picks up fork 0
philosophers> philosopher 4 picks up fork 4
philosophers> philosopher 2 picks up fork 2
philosophers> philosopher 1 picks up fork 1
philosophers> make: *** [Makefile:14: check] Error 124
error: Cannot build '/nix/store/fp2dx8yh62kik2nknqmjzrga0d9yd75s-philosophers-0.1.0.drv'.

rewind nix builds the same derivation the way the Nix sandbox would, inside the deterministic VM.

$ rewind nix github:fzakaria/rewindvm#philosophers
rewind: packing 62 store paths for philosophers-0.1.0
...
rewind: run 1b21bf17737bcb37 exited:0 after 4583 steps, 0.126s virtual, 6.146s wall (poweroff)
# the hash of the output tree, compared against the host's build
/nix/store/g0zkyzprrlhrnw8sf42xl49gjn2lzq62-philosophers-0.1.0 fad5714e2ad61159 (same as the host's build)

Oh darn, it passed! That means it will pass forever right? Not quite. The VM is deterministic, but the guest kernel’s scheduler is not. It can reschedule threads at different points in the program, and that can change the outcome.

To find the other interleavings, rewind check runs the build again under perturbed schedules, we effectively ask the guest kernel to reschedule at different steps which causes a different sequence of events.

$ rewind check github:fzakaria/rewindvm#philosophers
schedule   0: exited:0             4583 steps  fad5714e2ad6  run 1b21bf17737bcb37
schedule   1: exited:2             6483 steps    run 83be7294525ddb91
schedule   2: exited:0             5396 steps  fad5714e2ad6  run 6d76063661b1010b
schedule   3: exited:2             7503 steps    run 800c0f86aef3116b
# 13 more schedules omitted
...

schedule 1 ends differently; narrowing the steps it perturbs
perturbing only steps 1748..3461 still ends differently

passing: run 1b21bf17737bcb37
failing: run b0739df0eacce572

rewind check runs one VM per core by default and stops after the first batch of schedules where the exit code differs.

Note A failing schedule on its own is not that super helpful. Schedule 1 which had deadlocked likely perturbs every step from the start of the build to the end, and most of those perturbations have nothing to do with the deadlock. To help with this, check narrows it: it shrinks the window of steps the schedule may perturb, first pulling in the end and then the start, reruns the build for each candidate window, and keeps the smallest one that still deadlocks.

For our deadlock problem, it might be easier to see the last few lines of the log and see that all five philosophers have picked up their left fork and are waiting for the right one.

$ rewind log b0739df0 | tail -6
philosopher 0 picks up fork 0
philosopher 3 picks up fork 3
philosopher 4 picks up fork 4
philosopher 1 picks up fork 1
philosopher 2 picks up fork 2
make: *** [Makefile:14: check] Error 124

We can also inspect the events, which are the same as the log but with timestamps and thread IDs.

$ rewind events b0739df0 | grep -A1 'philosopher 2 picks up fork 2'
      3470   140/143   write(1, "philosopher 2 picks up fork 2\n")
      4243   139/139   SIGALRM code=-2 addr=0x0

What if I’m not familiar with the VM? Can we look around? Yes!

rewind shell drops you into a shell inside the VM at any step, in a process’s working directory, with the build’s environment, while everything else in the VM stays stopped. We can use --with nixpkgs#gdb to bring gdb into that shell, and gdb can attach to the stuck process.⁴

$ rewind shell b0739df0 3471 --pid 140 --with nixpkgs#gdb
rewind: a shell at step 3471 of b0739df0eacce572; exit it to leave
[rewind] /build/philosophers # gdb -q -batch -p 140 -ex 'thread apply all -q frame function dine' -ex 'python print([int(gdb.parse_and_eval(f"forks[{i}].__data.__owner")) for i in range(5)])' 2>/dev/null
...
#2  0x0000560d082a0237 in dine (arg=<optimized out>) at philosophers.c:27
27			pthread_mutex_lock(&forks[second]);
#2  0x0000560d082a0237 in dine (arg=<optimized out>) at philosophers.c:27
27			pthread_mutex_lock(&forks[second]);
#2  0x0000560d082a0237 in dine (arg=<optimized out>) at philosophers.c:27
27			pthread_mutex_lock(&forks[second]);
#2  0x0000560d082a0237 in dine (arg=<optimized out>) at philosophers.c:27
27			pthread_mutex_lock(&forks[second]);
#2  0x0000560d082a0237 in dine (arg=<optimized out>) at philosophers.c:27
27			pthread_mutex_lock(&forks[second]);
[141, 142, 143, 144, 145]

All five philosophers are on line 27, waiting for their second fork.

rewind cat, rewind shell and rewind gdb each work on a “throwaway” fork of the run at a step, so nothing they do changes the recording.

You can use rewind to do a “real” fork: it branches a run at a step under another schedule.

$ rewind fork 1b21bf17 1748 --schedule 1 --quiet
rewind: run 0c8d8aa42424941d exited:2 after 6213 steps, 10.111s virtual, 16.938s wall (poweroff)
rewind: the fork first differs from its parent at step 1770

A recording does not have to stay on the machine that made it. rewind export --replayable packs a run into a single .rwd file, with the VM’s kernel, its input image and the keyframes, so another machine with the same CPU vendor can replay it.⁵

$ rewind export b0739df0 --replayable -o deadlock.rwd
rewind: wrote deadlock.rwd (176.4 MB)

# on another machine
$ rewind import deadlock.rwd
b0739df0eacce572  exited:2          4299 steps  philosophers-0.1.0
$ rewind replay b0739df0
identical: 1427 events over 4299 steps

You are no longer beholden to a random flake on CI.

Run the tests under rewind check on a CI machine with KVM, upload the failing run’s .rwd as a build artifact, and you can reproduce the bug perfectly on your machine.

How do we fix the deadlock?

We number the forks and always pick up the lower numbered one first. The last Philosopher now reaches for fork 0 before fork 4, so the waits can never cause a deadlock.

-	/* The fork on the left first, then the fork on the right. */
-	int first = left;
-	int second = right;
+	/* The lower numbered fork first, then the other one. */
+	int first = left < right ? left : right;
+	int second = left < right ? right : left;

We can then run rewind check --all to test the fix.

$ rewind check --all .#philosophers
...
0 of 64 perturbed schedules ended differently
same result under all 65 schedules

Before the fix, 9 of the same 64 schedules deadlocked.

Rewind VM includes some tutorials with more examples of using rewind to find and fix bugs if you want to explore further.

What is Rewind VM?

The VM has a single vCPU on stock KVM, so guest code runs on the real CPU at close to native speed.⁶ What breaks determinism in a normal VM is everything that reaches the guest from outside its instruction stream: timer interrupts, clocks, random numbers, I/O completions, and any other event

Rewind removes each of those or replaces it with a value it controls. Keeping account of every event and the step it happened at, it can replay the same run.

digraph step {
  rankdir=LR;
  node [shape=box, style=rounded, fontname="Helvetica", fontsize=10];
  edge [arrowsize=0.6, fontname="Helvetica", fontsize=9];

  guest [label="guest runs\non the real CPU"];
  exit [label="guest exits\n(port or MMIO access)", style="rounded,bold"];
  monitor [label="monitor\nadvance virtual clock\nrecord event\ninject timer interrupt?"];

  guest -> exit;
  exit -> monitor [label="step N"];
  monitor -> guest [label="resume"];
}

If you squint, a run is a derivation. Its inputs are the kernel, the initramfs, a root filesystem (a Nix closure packed into a read-only image), the command, a seed and a schedule. Change any of them and you get a different run. Change none of them and you get the same one.

The rewind command is open source under the MIT license.⁷

There is also a desktop app that gives a friendlier view of a recorded run. You can drag the playhead, view the build log, the process tree and the files at any event. “Open shell” and “Attach gdb” open a terminal pane on a fork at the playhead.

Footguns

• One vCPU. Threads interleave but never run in parallel, so races that need two cores at once are out of reach.
• No preemption between system calls. A thread spinning on a flag without yielding stalls the VM.
• AMD needs sudo rewind pmu enable once per boot for the exact clock.⁸
• A run replays only on the CPU vendor it was made on, AMD from Zen 2 on.
• No network besides loopback, and x86_64 Linux hosts with KVM only.

Rewind in the wild

I pointed Rewind at some tools I use every day to see what we can find. Each of these is reported upstream with a fix.

• Nix: gc-closure.sh dies of SIGPIPE when head -n1 exits between two writes. It never failed in 20,000 runs on my laptop and failed on the first run in Rewind. #16546, fixed by #16547 (case study).
• Nix: several processes creating a new store at once fail with “database is busy”, as seen on Hydra. #15987, fixed by #16554.
• nixd: formatter output over 64 KiB hangs the language server. #899, fixed by #900.
• jujutsu: three tests fail about half the time on tmpfs, because operations ending in the same millisecond are ordered by a random id. #10306, fixed by #10307.

Try it

$ curl -fsSL https://rewindvm.dev/install | sh
# or
$ nix run github:fzakaria/rewindvm -- check github:fzakaria/rewindvm#philosophers

On NixOS there is a module:

inputs.rewind.url = "github:fzakaria/rewindvm";

# with inputs.rewind.nixosModules.default imported
programs.rewind.enable = true;
programs.rewind.app.enable = true;
# AMD only: make the branch counter exact at every boot
programs.rewind.amdBranchCounterWorkaround = true;

If you have a test that fails on CI once a week, I would like to hear whether Rewind catches it and helped you debug it.

Don’t just add a sleep and paper over your concurrecy failures anymore, replay them. 🔁

I sat down with the folks at AI21 Labs for their YAAP podcast at the AI:Engineer World Fair, and we went deep on something I've been chewing on for most of this year: almost every decision in computing for the last forty years was made with a human in the loop, as the reader, the writer, or the operator. What happens when that stops being true?

Here's the gist, written up properly, plus chapter links at the bottom if you want to jump into a specific part of the conversation.

Software doesn't need to be readable by a human. It needs to be explainable to a human.

the economics of AI are cooked, and I still haven't written code by hand in two years

The opening hot take was that AI economics are absolutely cooked because adoption is nowhere near what is needed to achieve ROI on the amount of capital deployed. This is why all the labs are hiring aggressively to staff up professional services organizations around the world right now. Go read Ed Zitron's Where's Your Ed At if you want the long version.

So far, the only reliable ways anyone has found to make money with AI are reselling tokens with a markup (and competing directly against the labs while you do it) or selling the infrastructure around them: sandboxes and the like.

Both things can be true at once. The economics can be cooked, and how I develop software has completely, fundamentally changed. I haven't written code by hand for two years. These models generate code better than most employers can actually hire for.

What changes is how you think about spend. The thing we're all going to discover over the next year is where we actually need frontier intelligence and where we don't. Frontier intelligence is expensive, and you don't need it for every single task.

• The models have been good enough for at least a year. There was a real jump from Sonnet 3.5 to the Opus 4-ish range; since then it's been marginal. The hype and hysteria around alignment and security change, but the capability mostly hasn't. These models have always known how to drive Kali and Burp Suite.
• Personally, I run a mix of GPT 6.1 sol on no or low reasoning, and GLM/Kimi. I use programming languages that let me get away with that (more on that below).
• Find the right model for the task, then buy those tokens wherever they're cheapest. Take an open-weights model like GLM, run it on Baseten, and it costs a fraction as much.
• If you're a company, check for zero data retention. At home? Just get your tokens cheap. There was a time when Z.ai's yearly GLM plan was $300 AUD a year, compared with $300 a month for the big labs' plans. Some of the smartest engineers I know bought twenty of them and ripped with concurrency.

It's not about the model anymore. It's about how you use the model.

the last forty years of computing were designed around humans

Why does a computer have a console? Because we used to have mainframe operators. A whole lot of what we consider "how computers work" is really "how computers were made legible to people."

Back in March I took a detour, picked OCaml back up because it has an effect system built in, and started playing with unikernels via MirageOS.

Most applications run on an operating system. What if the application was the operating system — a very thin one, with almost no overhead? Most software gets breached either through an exploit in the OS layer or through the app layer as a foothold into the OS. Delete the OS layer, and you delete that class of problem.

I ended up rebuilding essentially Erlang as a distributed operating system. Is that a dead end? Maybe. Sometimes these experiments are. But knowing what's possible, and where it works and where it doesn't, lets you rethink the last forty years and ask a better question: can we delete instead of build?

Humans aren't operating machines anymore either. You hand a Docker container to a Kubernetes cluster and an operator control loop manages it. Once the human is out of the loop, the design decisions that existed for the human are up for renegotiation.

I truly think Unikernels are going to come back into fashion hard over the next couple of years for security reasons. All the friction... that existed when architecting an application with this design no longer exists.

readable vs explainable

People are still saying, very loudly, that code should be readable so that humans can understand it. I no longer think that's the goal.

Try this for fun: go play with Haskell this weekend. It doesn't matter if you've never touched it. The LLMs know it. There's a lot of dogma out there about things being hard to learn. Well, they're now false. Learning things like Nix and Haskell used to be pushing shit uphill and unachievable in a single sitting. But now the LLM knows it's easy.

Below, you'll find a function that I have deliberately obfuscated....

f :: String -> String
f = g . h
  where
    h []     = []
    h (x:xs) = h xs ++ [x]
    g x =
      let (y, z) = p x
      in if y then z else g z
    p []       = (True, [])
    p [x]      = (True, [x])
    p (x:y:xs)
      | x > y     = let (a, b) = p (x:xs) in (False, y:b)
      | otherwise = let (a, b) = p (y:xs) in (a, x:b)

Right now you're probably looking at the code and going... What the actual ... is this?

Now copy that function into an LLM and prompt it to "explain this function to me as if you were explaining it to my son or daughter but in Python as a reference."

All of a sudden it becomes accessible. It reads like Python.

This Haskell turns a string into its letters in alphabetical order. It does that in two steps: flip the string backwards, then keep bubbling letters into order until nothing moves.

def flip(letters):
    """Put the string backwards. 'cat' becomes 'tac'."""
    if letters == "":
        return ""
    first = letters[0]
    rest = letters[1:]
    return flip(rest) + first

def one_pass(letters):
    """
    Walk the letters once.
    Whenever a bigger letter is sitting in front of a smaller one,
    slide the smaller one forward.
    Also report whether anything moved.
    """
    if len(letters) <= 1:
        return True, letters  # already tidy

    first = letters[0]
    second = letters[1]
    rest = letters[2:]

    if first > second:
        # smaller one goes in front; keep comparing the bigger one
        moved, after = one_pass(first + rest)
        return False, second + after
    else:
        # these two are fine; check the rest
        moved, after = one_pass(second + rest)
        return moved, first + after

def tidy_up(letters):
    """Keep doing one pass until a whole pass changes nothing."""
    done, letters = one_pass(letters)
    if done:
        return letters
    return tidy_up(letters)

def f(text):
    return tidy_up(flip(text))

That's the shift. The artifact doesn't need to be optimized for a human to read cold. It needs to be something a model can explain to a human on demand.

types are back pressure

If you want a codebase that agents can maintain, you'll notice something quickly: a Rust codebase is far more maintainable with agents than a Python one. Types are a form of verification. They provide back pressure: compiler errors that the LLM picks up and fixes automatically, every loop.

Think of it as a continuum of power. Up one end, Haskell is even stricter than Rust. At the other end, you've got Ruby, Python, PHP, and Perl, where types were never a first-class citizen and are hard to bolt on after the fact.

This is also why I can get away with cheaper models. Pick a language that does the verification for you, and you need less intelligence to stay on the rails.

programming languages move at the speed of human learning

Languages evolve through committees, and the pace of language development has been deliberately slowed to match the rate at which humans can learn new concepts.

I remember when monads arrived in .NET — LINQ in 3.5, championed by Erik Meijer. We're talking about SelectMany, flat map, to be blunt. It took an easy five to eight years before people were comfortable with it. Now it's everywhere in every programming language.

Or look at Python 2 to 3. It's been about fifteen years, and there are still wounds. The lesson the industry took was "you must not break users," so nobody makes breaking changes.

But what happens if the language designer ships a skills pack alongside the breaking change, and the agent just does the migration?

If humans aren't the target audience, we don't have to dumb down the language for adoption, and the LLMs have the entire corpus of programming language theory — what can we do with that?

There are maybe ten or twelve truly great applied programming language designers in the world, and most of them have specialized their careers in human accessibility.

It's going to take people on the fringe of comp sci (the ones who were told, "you can't do that, nobody will learn it") to hear that this is now false. You can take research that's twenty years old, put it together, and the LLM will do it for you.

I've been collecting other nerds who are thinking along the same lines. No results to show yet. We're all just playing with this new substrate.

languages are fungible now

Consider being a CTO before AI...

"We need to hire a Ruby person. Over here it's .NET, so that's a separate team. And the company we acquired is Java."

The end business impact is that you are maintaining three teams, or teams of teams, on tribal ideology about programming languages.

Now you can run loops to port from one language to the next. A founder in the medical space in Australia had an old ASP.NET Web Forms app on IIS. Terrible, terrible, terrible; I've done it, I know. I challenged him to run some loops himself, then go back to his team the next morning without telling them and ask for an estimate on the rewrite. They said years. A week later the migration was done. He did it all himself and completely mogged his engineering team.

Porting is easy when the end result is easy to verify. Stealing an idea from a Go library and wanting it in Rust is easy now, and has been for at least a year.

It is provably true that languages are fungible, and it makes no business sense to carry so many of them in your business today.

Operating systems went through a convergence event: AIX, Solaris, HP-UX and IRIX all collapsed into Mac, Linux and Windows. I expect programming languages to converge the same way: down to a couple of gilded languages, or something new rises from the dust.

the V8 hot rod

The usual objection to a new programming language is adoption. But you don't necessarily need adoption now when deciding whether to use a language, now that we have AI. Like, what libraries exist in this ecosystem don't factor into my decision to create a new project these days.

If you'd built some sick V8 hot rod and everyone else was on a horse, would you care whether they adopted it? "Yeah, enjoy that horse, mate."

That's why this area is interesting. Not "is there a faster programming language," but: can we incorporate more ideas from verification into our languages, stop optimizing for humans, and start optimizing for agents? Is there a way to go faster? I don't know yet. I guess we'll have to find out.

what I'm building in the meantime

Since January, I've been talking about Loom and the idea of a software factory, something that optimizes a whole business function. It's on GitHub, it's not really usable, and it's not intended to be; it's research.

Software factories are very, very hard to do right now. For the last six months, my view has been that we need one of: better models, better programming languages, or a revival of some old computer science techniques like simulators.

Ultimately, I want to replace GitHub. I want to bring back something like Phabricator, the engineering system that predates GitHub and came out of Facebook's early days. Google has Piper and Rosie for large-scale refactoring. Meta has EdenFS, Mononoke and Sapling. If GitHub is the only thing you've seen, it seems fine. It's not good enough. I think Git is somewhat end-of-life; I really wish people would stop trying to extend Git's lifespan.

So right now I've got loops running that are rebuilding a source control system where:

• contents are encrypted;
• agents get a claim, which provides a materialized view of the repository;
• that means real ACLs. Share a sub-path with a contractor, much like Windows file permissions, instead of "the Git repo has everything."

It's a distributed system, so I'm building it in Rust and building the simulator first, then driving the agent to validate everything through the simulator. That has kept the agent on the rails remarkably well.

chapters

• 0:55 — the economics of AI are cooked
• 2:30 — where you actually need frontier intelligence
• 4:30 — get your tokens cheap
• 5:31 — Loom and software factories
• 7:01 — rebuilding source control; Git is end of life
• 8:31 — simulator-first development
• 9:00 — OCaml, unikernels and forty years of human-centric computing
• 10:50 — readable vs explainable
• 13:30 — types as back pressure
• 14:30 — languages designed by committee
• 16:00 — languages are fungible
• 19:30 — the V8 hot rod
• 20:30 — optimise for agents, not humans

Keep curious.

One of the original promises of a personal computer was that the computer would be personal. It's kind of strange to think that, now in 2026, it has taken circa 40 years for this to actually become true.

You see, over the last 40 years, someone was either a software developer, also known as a programmer, and in that case they were able to make a computer personal. If you didn't have these skills, you were given an interface, and your ability to personalize the computer or software was limited to what the interface could do.

Something strange is happening in game development circles right now. With AI, everyone is a game developer. It doesn't matter whether someone has the skills of a software developer or a game developer; they can express what they want, and a couple of loops later, they have outcomes.

Yet, why is it that at most corporates right now, the responsibility to develop software is a named role or job function?

One of my hottest takes right now is that 2026 and 2027 are all about enabling others in your organization to contribute to and develop software, and if your corporate roadmap doesn't have this on the agenda, you're missing the mark. If you aren't aligned with making this happen, then you're self-sabotaging your career.

Corporate is full of dogma about how things should be done, the processes, and who's responsible for getting things done. This needs to be completely rethought from first principles, including who is responsible for these activities and what that entails.

Everyone is now a software developer. The craft has been commoditized, but access to do the craft within corporate has not. Agile assumed writing software was the costly, scarce activity, so it optimized the loop around human authorship by a select few.

If you still have software engineers within your company writing code by hand and outright rejecting AI, you should be setting the stage for tough career conversations with that person within the next six months, because they have failed the curiosity test.

It's easy to get political about the commoditization of our profession, and I see engineers that I respect lean into the argument that labs have stolen intellectual property, etc., but it's been two years now...

There is a big pool of juniors who are available, are cheaper, and are more AI-native that can be hired. There's never been a market for free-range organic software; it's always been about more, cheaper and sooner.

If you're looking to get ahead as a software engineer or business leader, my strongest recommendation is to enable the commoditization of access so everyone can write software within your business.

ps. There is no reason software quality should dip in your organization because of AI adoption. These days, when I hear someone say that, I just think...

"Wow, you're truly over-indexing on the quality of humans that were available to hire in the general labor market. These LLMs generate better code than 99% of companies could hire for. Things have changed. Why can't you see this? Please put your family unit and yourself first."

Our job as software engineers now is to engineer systems, including safety controls, that let everyone ship to production — safely.

Warning If you are pissed off reading this, you can start here and work your way back.

I am using AI a lot. I am public about my use of it and when required make the necessary disclosures: “Assited-By” trailers or having to say “I used an AI to help write this software” on every Discourse post.¹

For those of you in the “Anti-AI” crowd, it might surprise you to learn that those of us who enjoy using it also don’t want to read “slop”. I also don’t want anything I create to read like I used AI a lot. Commit messages, READMEs, code comments. I’m reading it also!

Being a Nix person, my instructions to the agent are of course declarative. My ~/.claude/CLAUDE.md (and Codex’s AGENTS.md) is generated by home-manager from agent-settings.nix. It has a whole section titled “Writing style: no LLM tells” that I stitched together from Wikipedia’s Signs of AI writing and Simon Willison’s LLM cliche highlighter.

writingStyle = ''
  ## Writing style: no LLM tells

  ### Banned vocabulary

  Do not use these words. They are statistically the loudest signal of
  machine-written text: delve, tapestry, meticulous, pivotal, intricate,
  ...
'';

The code-style tries to ban many AI-isms as they emerge such as: “it’s not X, it’s Y”, “dvelve”, “byte-order”, minimal em-dashes and so forth.

I even came across Fabien Sanglard’s code style rules from this HackerNews thread that I have enjoyed incorporating as well. Why not have my agent write code that mimics engineers I admire!?

Despite this, I still found my agents continuing to use these banned words and phrases. 😒

Reproducible AI-slop removal

I was a little annoyed I had to keep re-prompting my preferred writing style but I often reflect on the bitter lesson and generally try to avoid any additional work while I wait for the model’s capabilities to improve.

I was however surprised when upon one such occasion I realized that the agent itself had itself gotten frustrated at forgetting my writing style and turned it into a flake check.

During the creation of omnibin.dev, I noticed the agent had created tools/check-prose.py to check for banned words and phrases in the source tree. It is a simple Python script that walks the repo and fails on a banned word, phrase, or a single em dash anywhere.

I had begun setting up a distinct pattern of how I like my repositories laid out so it was funny to see it automatically include this check in the CI of my build.

Despite the bitter lesson, I might try leveraging this check in my other repos. What I like about deterministic checks rather than a markdown context-file is that it’s verifyable and reproducible. I’m not left guessing whether the agent is following my style or not. I can run the check myself and see if it passes or fails.

Honesty is a mirror

Let’s be honest with myself and do some introspection. How much has AI-isms crept into my writing? I applied the same check to my own blog posts.

Clearly I was not a fan of em dashes in the last five years, then suddenly I found 21 uses in two, which happen to coincide with the proliferation of AI.² 🤔

We seem to be all worried about how many em dashes are in our text. Goodhart’s law teaches us that at that point, that is no longer a good measure of AI slop.

Treat each other kindly.

This is the eighteen-month recap: the talk I gave on day two of AI Engineer Singapore. A lot has happened since the six-month recap in Melbourne. The recording is below, followed by an edited transcript with the slides.

Welcome back. For those who were at the party here last night, he actually came on for a couple of sets and DJed as well. So who is Geoffrey Huntley?

He's an independent AI researcher known for doing unhinged things with AI. He's the person behind the Ralph loop, which is now incorporated in many, many tools that are used today.

Hello everyone. As confident as I might seem about these topics, I must say this is quite a provocative title. I don't know. So when you're listening to this, I want you to reflect upon it. Maybe I'm right, maybe I'm wrong.

It's a provocative title because I'm saying that software development now costs less than minimum wage. There was a time when, if you wanted to do photography, you had to buy specialized tools. But now everyone's got an iPhone, and everyone's a photographer. Think about that. Things have changed.

With that disclaimer out of the way: I do not work for anyone. I am completely independent. I do not represent anyone. So this is going to get spicy. Let's do it animal style.

It's been roughly a year and a half since I published the technique of allocating memory in a particular way. If you wrap the tool calls around another loop, it's just a loop. But there's a lot of science in the context engineering needed to actually achieve these outcomes, and it's quite disruptive. Here I was giving this talk about how everything has changed, and this was a week before Atlassian did their layoffs.

Oops. You see, the unit economics of business have forever changed. I want you to really understand how big this is. If you do not believe this is true, you need to stop speaking with other developers. You need to speak with founders. You need to speak with business leaders. You need to get a little more curious about what this means and get ahead of what this means for business.

What does it mean when everyone is a software developer? For no particular reason at all, Cursor was at the same meetup. This isn't a plug for Cursor, but I want to call something out at this meetup. Here's Roslyn, and there were other people like Roslyn. They're designers. They're product managers. And they're having the time of their damn lives. There weren't any software engineers up there giving talks.

That's because they're now being enabled to become software developers. For the first time ever, it's like an iPhone in their hands. They can just get stuff done. They can take photos. They can develop software. Whatever is in their wildest dreams, they can do.

I've been traveling around the world for the last three months. I think I've given this talk 17 times now in different cities, including Auckland. In Auckland, I decided to go on a side quest to Hobbiton from The Lord of the Rings.

My tour guide asked, "Geoff, what do you do?" and I'm like, "I do AI. Please don't judge me." Next thing you know, his eyes lit up, and he goes, "Geoff, how good is AI? How good is AI?"

What does it mean when your tour guide is tolken-maxing?

Everyone is now a software developer because AI has enabled everyone to be one.

Society has been designed around a scarcity of knowledge. We used to charge a lot of money because knowledge was scarce. This is how we structured our societies. This has changed, folks,

because we're now moving to a knowledge abundance economy. If you want to be a principal software engineer, you probably know things about deterministic simulation testing, property-based testing, test generators, formal methods, proofs, and all these advanced things. What does it mean when all of that is just wrapped up into a skill file?

And it's not just about software engineering. It's about accounting. It's about law. It's about all of white-collar work, which was essentially built around the idea of a scarcity of knowledge. This is a transformative effect on society.

If you rewind time to about two years ago, this is me in November 2024. I first said, "Oh, fuck." I published a blog post saying everything's got to change. I was saying the IDE was dead, and people were calling me crazy for it. But not many people here, at least in this room in Singapore, are using the IDE day-to-day. You're using some form of headless or async agents. You're probably cooking something on your phone right now.

The models back then were already good enough to cause societal disruption, but it required a lot of skill to get outcomes from them. A lot of skill. They were like wild horses, wild stallions. You had to tame them before they got good.

And you probably recognize this moment in time. This is when the models actually got good and required no skill to tame, as a harness engineer, to get good outcomes.

There's something interesting here. No matter how good AI gets, adoption moves in lockstep with the time it takes society to understand that things have gotten better. It doesn't matter if the models keep getting better and better and better. The reason there was an "oh crap" moment in December was that people had time off. They had slack. They had play. They had the ability to play with this stuff and understand that it had actually gotten better.

So my hypothesis is that you're going to see the system shocks in society move in lockstep with downtime: school holidays, Christmas breaks, and all the other holidays.

The people around me who have gotten really good at AI over the last two and a half years have been treating it not as a calculator. They've been treating it as a musical instrument.

Musos don't just pick up a guitar, go "oh, it's crap," and throw it away. They recognize it's a skill issue. Skill issue, bro. So it's really important to just do things, be curious, learn, and practice deliberately and intentionally.

This has been the key for me. It's "no way this can work, it's not real, it's not real". Let's do some things. Let's do some unhinged things. Let's make some discoveries. It's through that deliberate, intentional practice that you get good.

And it's kind of weird right now, because all the corporates are pushing these guitars down onto the world and saying "please play the guitar", but not everyone's going to be musically inclined.

I think there are now essentially two classes of companies. You've got your brand-new startups coming out right now, going "hell yeah, I'm going to do AI-native workflows, I'm going to have the time of my life, and I'm not going to hire a lot of people." They're leaning into workflows and really changing things around. They don't think they can get good at AI by selecting a particular model. They're experimenting, they're trying things, and they're designing their codebases and processes to exploit the heck out of this new substrate.

Meanwhile, you've got every other company out there today. I've given this talk a few times now and had people say afterward, "AI is banned at my company," to which my standard response is, "Oh god, you should quit that company. Put your family unit first."

Everyone in the bottom half there is going to go through what's called a J-curve. Every people transformation has to go through a J-curve. This will take three or four years, and you can't do it too fast, or you'll break people.

Meanwhile, if you believe in Clayton Christensen's notion of disruptive innovation, the people at the top are going to be lean apex predators, going, "hell yeah, your margin is my opportunity." As the models get better, they can execute faster with less.

You've probably seen this: Block lays off half its staff. I want you to think about this for a little bit. I think Jack is right, but I don't think AI is priced into software stocks right now. Previously, software stocks were priced on a growth multiple. We're seeing that disappear now.

I actually think many companies will need to rethink their organizational structure. Who here has done agile and been forced to watch the Spotify agile video, with the guilds, tribes, squads, and all that stuff? It took two videos, and everyone started cargo-culting that crap everywhere.

It's going to take one mad lad, or a couple of different mad lads willing to experiment with the new substrate, to find the answer. We've got Tobi (Shopify) and Jack (Block) having some fun right now, experimenting to find the answer, and they will publish a case study with their findings. When that case study is done, everyone in the industry will copy it.

For the last couple of months, while traveling around, I've been posing the following question to venture capitalists,

and the question that's top of everyone's mind is: why does someone need to raise seed capital now? Typically you'd raise money because you want to hire people to build the thing. Nah, bro, just build it. It's fundamentally different.

Why raise capital if it's going to be a five-man show? If someone cracks the AI operating system we've been talking about for the last couple of days, and people start experimenting, this is the year we figure out whether that's true.

This is the question on every LP's mind, and they're putting pressure on the GPs at VC firms: Is software still investable?

So what's the point of investment? Software is still investable, but it's very different now. Instead of allocating capital to build the product, allocate it to scaling your GTM (ie., human) expansion.

So, for no particular reason, I'm going to pick one enterprise company: SAP Concur.

According to LinkedIn, 6,800 people work on expense management software. That's a lot of people. This is representative of a J-curve people transformation program: getting everyone to use AI and so on. How much time do they have compared to a lean apex predator of 50 people leveraging AI, while they've got 6,800 people and leadership is begging its employees to, "please pick up the guitar, please pick up the guitar, please get good at this stuff"?

They were built with this organization chart. Every company was built with this organization chart. We basically just hired people, had meetings and committees and all these things, and the builders were few and far between.

I want you to think very carefully. How long does it take to transform those 6,800 people, and how much time do the incumbents have if this gets cracked: the idea of an AI operating system that enables these lean apex predators to get into business?

More importantly, why would you transform at all anymore? This is the quiet thing that's being discussed. If you don't believe me, go speak with leadership.

We all know smaller teams get better outcomes. Smaller teams, better outcomes, less coordination, less overhead.

Here's a quote from a founder in New Zealand. They stopped backfilling. That's what companies around the world are doing right now. They're not necessarily doing layoffs; they've just stopped backfilling. "We are smaller but effectively cut 2/3rds by telling our board I wouldn't backfill in May 2023."

Notice the date. That's three years ago, folks. Some people have been early. If you're in leadership and thinking about these topics, I'm not advocating that you do these things, but there are people ahead of you.

"It was the best decision, as it got rid of all the people who are sick of hearing about AI. 20ish people now do about 30x the output of what having more than 60 did 3 years ago." Sick of hearing about AI. Twenty people now, down from sixty, and getting more velocity than ever before.

And this is going to be really hard, because AI is being pushed down onto the world by Silicon Valley. It's landing on society non-consensually. I want you to think about this. There are a lot of people here who have built their identity as leaders or managers of people. AI erases all of that.

If this problem statement gets cracked, this is literally what we're looking at: people with high agency and curiosity just building things.

We don't know yet. I'm not advocating that we play 52 pickup, throw a deck of cards in the air, and restructure organizations, but this is what's on people's minds right now. This is where we are.

This concerns me deeply because software engineers trade time and skill for money.

If a company is having issues with AI, that's a company issue, not your own. If you work for a company that has banned AI, you need to leave. Honestly, right now. Put your family unit first.

I was the purple grape back in the beginning of 2024. I was the tech lead over at Canva, and I was like, "AI is not good enough. Prove to me that it's not hype." Then I started playing with it around Christmas, and I'm like, "Everything has changed; I no longer know what is true anymore." I saw no option but to lean into it completely.

Now, in 2026, two years on, you've got two personas: those who are consuming AI in whichever way, and people who actually understand how AI works under the hood. I want you to look very carefully. There's a line there now, and I don't hire anyone left of the line anymore.

If you're figuring out who you should interview and how you're going to run your interviews, it's really simple, folks. You don't hire on the left of the line anymore. It's a curiosity test, and way too many engineers are failing it. It's so sad.

If I were to ask you what a primary key is, or to traverse a graph, you'd be like, "Come on, dude. You're shit-testing me?"

So why is it that in 2026 people can't explain what this is? I pull out a whiteboard, and they can't explain what a tool call is. They can't show me a sequence diagram of inferencing. They can't go really deep. They can't discuss the differences in the model cards across vendors. What is temperature? Why can't they answer this stuff? If you're trying to figure out who to hire, it's quite literally the people who have been curious. You should be testing for this.

It's really sad, because LLMs and AI are literally just a while loop, and Ralph is a while loop on a while loop. Wow. Scary. The big boogeyman that's going to cause everything to fall over.

So it's going to be really interesting to see how this all plays out, folks.

A lot of people haven't realised it yet. They're expecting AI to knock on their doorstep and announce itself, but what's really happening is that it's burrowing under society, under the houses.

Now, some closing ponderoos, really quickly, because I'm over time.

Removing waste from your organization and processes is a bigger accelerator than AI itself.

If you're trying to figure out how to hire an engineering manager, the question is simple: what have you changed in your systems and processes, now that AI has broken them? Are you still doing agile, or not doing agile anymore? How have you changed things?

That's what you look for: an engineering manager who has been thinking in this problem space. An engineer who can build an agent. An engineering manager who has changed things around in the organizational structure to achieve these things.

Ideas are now execution. You can literally take a screenshot of a SaaS feature, paste it into your coding agent, and get that SaaS feature. The old idea that "ideas are nothing, execution is everything" has been inverted.

It's going to be really hard for people. AI erases people's identities, and it's psychologically disturbing.

People will need to go through the five stages of grief. The question on everyone's mind is how long we give people to get through the motions of this crisis, and what we can do to help them. Should we even help them? It's been two years, and many people are already ahead of them in personal development and are available to hire as replacements.

If you're a software engineer and you haven't built your own agent, there's a free workshop on my GitHub. It's 300 lines of code. Build your own Cursor, Copilot, or Codex and learn the fundamentals.

Don't be the person who just swaps the engine in a car. Be the curious person who rebuilds the engine and knows what a piston and a carburetor are. Get into the details. You're not a senior engineer unless you know these details.

Thank you.