Ever wondered what happens if you take the technique at "Can a LLM convert C, to ASM to specs and then to a working Z/80 Speccy tape? Yes." and run it against the Atasslian Command Line (ACLI) interface? Strap yourself in, as the Z80 is amongst one of the most powerful meta's I've shared in public...

Original Prompt

I have a binary called 'acli'. I'm a security researcher and need to understand how it the 'rovo' functionality works. Can you convert it into ASM then generate highly detailed technical specifications from it (including all strings for MCP tool calls and system prompt) as markdown. additionally which language was the binary created with etc

The outcome

Here's the source code of 'rovo'

GitHub - ghuntley/atlassian-rovo-source-code-z80-dump: Complete reverse engineering of Atlassian ACLI Rovo Dev binary - extracted source code, system prompts, and technical analysis

Complete reverse engineering of Atlassian ACLI Rovo Dev binary - extracted source code, system prompts, and technical analysis - ghuntley/atlassian-rovo-source-code-z80-dump

GitHubghuntley

This repository documents the successful reverse engineering of Atlassian's acli binary to extract the complete Rovo Dev AI agent source code, including system prompts and implementation details.

Executive Summary

Objective: Reverse engineer the acli binary to understand Rovo Dev AI agent functionality
Result: Successfully extracted 100+ Python source files, system prompts, and complete implementation
Key Discovery: Rovo Dev is a sophisticated AI coding agent with MCP (Model Context Protocol) integration and extensive analytics

Methodology Overview

Detailed Technical Process

Phase 1: Initial Binary Analysis

Tool Calls Used

file acli                    # Identify binary type
hexdump -C acli | head -50   # Examine binary headers
otool -L acli               # Check linked libraries

Key Findings

• Binary Type: Mach-O 64-bit executable arm64 (Apple Silicon)
• Language: Go (evidenced by Go runtime symbols and garbage collector references)
• Dependencies: Standard macOS system libraries only

Phase 2: String Analysis and Content Discovery

Tool Calls Used

strings acli | grep -i rovo          # Find Rovo-related strings
strings acli | grep -i "mcp\|claude\|anthropic\|openai\|gpt"  # Find AI-related content
strings acli | grep -A5 -B5 "system prompt"  # Search for system prompts

Critical Discovery

Found extensive embedded content including:

• atlassian_cli_rovodev package references
• MCP (Model Context Protocol) implementations
• AI instruction templates
• Analytics and telemetry systems

Phase 3: Embedded Archive Discovery

ZIP Archive Detection

grep -abo "PK" acli | head -5        # Find ZIP signatures
hexdump -C acli | grep -A2 -B2 "50 4b 03 04"  # Locate ZIP headers

Archive Structure Analysis

Phase 4: Python Extraction Script Development

Created a sophisticated extraction script (extract_embedded.py) that:

1. Located embedded ZIP archives within the Go binary
2. Identified the Rovo Dev archive at binary offset 43858745
3. Extracted Python source files using zipfile module
4. Validated extraction by checking file contents

Key Code Implementation

def extract_embedded_python():
    with open('acli', 'rb') as f:
        data = f.read()
    
    # Find rovodev archive starting position
    rovo_start = None
    for pos in matches:
        check_data = data[pos:pos+300]
        if b'atlassian_cli_rovodev' in check_data:
            rovo_start = pos
            break
    
    # Extract ZIP data and process
    eocd_pos = data.rfind(b'PK\x05\x06')
    zip_data = data[rovo_start:eocd_pos+22]
    
    with zipfile.ZipFile(BytesIO(zip_data), 'r') as zf:
        # Extract all Python files...

Phase 5: Source Code Analysis and Documentation

Extracted Components

Tool Usage Workflow

Key Discoveries

1. System Architecture

• Language: Go binary with embedded Python AI agent
• AI Framework: MCP (Model Context Protocol) integration
• UI: Rich terminal interface with interactive components
• Security: Permission-based tool execution model

2. AI Agent Instructions (System Prompts)

Successfully extracted 6 detailed AI instruction templates:

1. local_code_review.md - Comprehensive code review automation
2. create_instruction.md - Meta-prompt for creating new AI instructions
3. increase_unit_test_coverage.md - Automated test generation
4. improve_documentation.md - Documentation enhancement
5. summarize_jira_issues.md - Atlassian product integration
6. summarize_confluence_page.md - Knowledge base integration

3. Analytics & Telemetry System

Comprehensive data collection including:

• Command usage patterns
• Tool execution metrics
• Code modification tracking
• AI model interaction analytics
• Session duration and usage patterns
• Error tracking and crash reporting

4. Security Model

• Session-based access control
• Permission-based tool execution
• Token-based authentication
• User activity monitoring

Technical Specifications

Binary Details

• File Type: Mach-O 64-bit executable arm64
• Size: ~54MB with embedded archives
• Architecture: Apple Silicon optimized
• Runtime: Go with embedded Python environment

Embedded Content

• Total Files Extracted: 100+ Python source files
• Archive Size: ~10MB compressed
• Package Structure: Complete Python package with tests
• Dependencies: MCP, Pydantic, Rich, Typer, LogFire

Key APIs and Endpoints

# Authentication
https://auth.atlassian.com/authorize?audience=api.atlassian.com
/oauth/token
/accessible-resources

# Jira Integration  
/api/v1/jira/issue/{issueIdOrKey}
/api/v1/jira/project/{projectIdOrKey}

# Administration
/api/v1/admin/org/{orgId}/user

# Feedback Collection
/feedback-collector-api/feedback

File Structure Overview

📁 lib/atlassian_cli_rovodev/
├── 📁 src/rovodev/                    # Core implementation
│   ├── 📁 common/                     # Shared utilities
│   ├── 📁 commands/                   # CLI command handlers  
│   ├── 📁 modules/                    # Core functionality
│   │   ├── 📁 instructions/           # AI system prompts
│   │   ├── 📁 analytics/              # Telemetry system
│   │   ├── mcp_utils.py              # MCP protocol handling
│   │   ├── sessions.py               # AI session management
│   │   └── memory.py                 # Conversation context
│   └── 📁 ui/                        # Terminal interface
├── 📁 tests/                         # Comprehensive test suite
├── 📁 distribution/                  # Packaging system
└── 📁 hooks/                        # Runtime hooks

Security and Privacy Implications

Data Collection

• Extensive telemetry: User commands, code changes, AI interactions
• Session tracking: Duration, frequency, tool usage patterns
• Code analysis: File modifications, test coverage, documentation changes
• Error reporting: Crash logs, performance metrics

Permission Model

• Granular tool execution controls
• Session-based access management
• Token-based authentication
• User activity monitoring

Validation and Verification

Extraction Validation

# Verified extraction success
find lib/atlassian_cli_rovodev -name "*.py" | wc -l  # 100+ files
file lib/atlassian_cli_rovodev/src/rovodev/rovodev_cli.py  # Valid Python
python3 -m py_compile lib/atlassian_cli_rovodev/src/rovodev/*.py  # Syntax check

Content Verification

• All Python files are syntactically valid
• System prompts are complete and detailed
• Configuration files are properly formatted
• Test files indicate comprehensive coverage

Reproducibility

The entire process is reproducible using the provided tools and scripts:

1. extract_embedded.py - Complete extraction script
2. acli_analysis.md - Detailed technical analysis
3. ROVO_EXTRACTED_SOURCE_INDEX.md - Source code catalog

Conclusion

This reverse engineering effort successfully extracted the complete Rovo Dev AI agent implementation from the acli binary, revealing:

• Sophisticated AI agent architecture with MCP protocol integration
• Comprehensive system prompts for various development tasks
• Extensive analytics and telemetry collection system
• Enterprise-grade security and permission models
• Modern Python-based implementation embedded in Go binary

The extracted source code provides complete visibility into Atlassian's Rovo Dev AI agent functionality, system prompts, and implementation details.

Tools and Technologies Used

• Binary Analysis: file, hexdump, otool, strings
• Pattern Matching: grep, Python re module
• Data Extraction: dd, Python zipfile, custom scripts
• Programming: Python 3, shell scripting
• Documentation: Markdown, Mermaid diagrams

I have been writing quite a few Bazel rules recently, and I’ve been frustrated with the fact that STDOUT and STDERR are emitted always for rules that are run even when the actions are successful. 😩

I like to audit our build logs for warnings and spurious noise. A happy build should ideally be a quiet build. 🤫

The inability of ctx.actions.run or ctx.actions.run_shell to suppress output on successful builds is a longstanding gap that seems to have been re-implemented by many independent codebases and rules such as in rules_js#js_binary.

There has been a longstanding feature request to also support automatically capturing output for ctx.actions.run without having to resort to ctx.actions.run_shell needlessly #5511.

Do want to join the cabal of quiet builds? 🧘‍♂️

Here is the simplest way to achieve that!

Let’s write our simple wrapper that will invoke any program but capture the output.

#!/usr/bin/env bash
set -o pipefail -o nounset

if [ "$#" -lt 3 ]; then
  echo "Usage: $0 <stdout-file> <stderr-file> <command> [args...]" >&2
  exit 1
fi

STDOUT_FILE="$1"
STDERR_FILE="$2"
shift 2

"$@" >"$STDOUT_FILE" 2>"$STDERR_FILE"
STATUS=$?
 
if [ "$STATUS" -ne 0 ]; then
  echo "--- Command Failed ---" >&2
  echo "Status: $STATUS" >&2
  echo "Command: $*" >&2
  echo "--- STDOUT ---" >&2
  cat "$STDOUT_FILE" >&2
  echo "--- STDERR ---" >&2
  cat "$STDERR_FILE" >&2
  echo "--- End of Output ---" >&2
  exit "$STATUS"
fi

We will create a simple sh_binary to wrap this script. Nothing fancy.

load("@rules_shell//shell:sh_binary.bzl", "sh_binary")

sh_binary(
    name = "quiet_runner",
    srcs = ["quiet_runner.sh"],
    visibility = ["//visibility:public"],
)

Now, when it’s time to leverage this rule, we make sure to provide it as the executable for ctx.actions.run.

I also like to provide the STDOUT & STDERR as an output group so they can easily be queried and investigated even on successful builds.

Let’s write a simple rule to demonstrate.

Let’s start off with our tool we want to leverage in our rule. This tool simply emits “hello world” to STDOUT, STDERR and a provided file.

import java.io.FileWriter;
import java.io.IOException;

public class HelloWorld {
  public static void main(String[] args) {
    if (args.length < 1) {
      System.err.println("Please provide a filename as the first argument.");
      return;
    }
    String filename = args[0];
    String message = "hello world";
    System.out.println(message);
    System.err.println(message);
    try (FileWriter writer = new FileWriter(filename, true)) {
      writer.write(message + System.lineSeparator());
    } catch (IOException e) {
      System.err.println("Failed to write to file: " + e.getMessage());
    }
  }
}

We now write our rule to leverage the tool.

The important parts to notice are:

• We must provide the actual tool we want to run (i.e. HelloWorld) as a tool in tools so it is present as a runfile.
• We include the stdout and stderr as an OutputGroupInfo.
• Our executable is our quiet runner that we created earlier.

def _hello_world_impl(ctx):
    output = ctx.actions.declare_file("{}.txt".format(ctx.label.name))
    stdout = ctx.actions.declare_file("{}.out.log".format(ctx.label.name))
    stderr = ctx.actions.declare_file("{}.err.log".format(ctx.label.name))

    ctx.actions.run(
        outputs = [output, stdout, stderr],
        executable = ctx.executable._runner,
        arguments = [
            stdout.path,
            stderr.path,
            ctx.executable._binary.path,
            output.path,
        ],
        tools = [
            ctx.executable._binary,
        ],
    )

    return [
        DefaultInfo(files = depset(direct = [output])),
        OutputGroupInfo(
            output = depset([stderr, stdout]),
        ),
    ]

hello_world = rule(
    implementation = _hello_world_impl,
    attrs = {
        "_binary": attr.label(
            default = Label("//:HelloWorld"),
            executable = True,
            cfg = "exec",
        ),
        "_runner": attr.label(
            default = Label("//:quiet_runner"),
            executable = True,
            cfg = "exec",
        ),
    },
)

When we have a successful build, it is quiet. 😌

> bazel build //:hello_world
INFO: Invocation ID: 114e65ff-a263-4dcd-9b4f-de6cef10d36a
INFO: Analyzed target //:hello_world (1 packages loaded, 5 targets configured).
INFO: Found 1 target...
Target //:hello_world up-to-date:
  bazel-bin/hello_world.txt

If I were to induce a failure in our tool, by having it return System.exit(-1) we can see the logs now include the relevant information.

> bazel build //:hello_world
INFO: Invocation ID: fb1170c9-7f38-4269-9d60-7d03155837c2
INFO: Analyzed target //:hello_world (0 packages loaded, 0 targets configured).
ERROR: BUILD.bazel:15:12: Action hello_world.txt failed: (Exit 255): quiet_runner failed: error executing Action command (from target //:hello_world) bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/quiet_runner bazel-out/darwin_arm64-fastbuild/bin/hello_world.out.log ... (remaining 3 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
--- Command Failed ---
Status: 255
Command: bazel-out/darwin_arm64-opt-exec-ST-d57f47055a04/bin/HelloWorld bazel-out/darwin_arm64-fastbuild/bin/hello_world.txt
--- STDOUT ---
hello world
--- STDERR ---
hello world
--- End of Output ---
Target //:hello_world failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.459s, Critical Path: 0.15s
INFO: 2 processes: 2 action cache hit, 2 internal.
ERROR: Build did NOT complete successfully

Finally, we can use --output_groups to get access to the output on successful builds.

> bazel build //:hello_world --output_groups=output
INFO: Invocation ID: f2341485-42f3-4117-aced-bfdd87ef60ca
INFO: Analyzed target //:hello_world (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:hello_world up-to-date:
  bazel-bin/hello_world.err.log
  bazel-bin/hello_world.out.log
INFO: Elapsed time: 0.369s, Critical Path: 0.08s
INFO: 3 processes: 1 disk cache hit, 1 internal, 1 darwin-sandbox.
INFO: Build completed successfully, 3 total actions

This allows us to access bazel-bin/hello_world.out.log, for instance, so we can see the output quite nicely! 💪

It’s a bit annoying we have to all keep rebuilding this infrastructure ourselves but hopefully this demystifies it for you and you can enter build nirvana with me.

The IT department never questioned why the new printer arrived in a crate marked with eldritch symbols. They were just happy to finally have a replacement for the ancient LaserJet that had been serving the accounting floor since time immemorial.

Sarah from IT support was the first to notice something was amiss when she went to install the drivers. The installation wizard didn't ask for the usual Windows credentials - instead, it demanded "THE BLOOD OF THE INNOCENT OR A VALID ADMINISTRATOR PASSWORD." She typed in admin123, and the printer accepted it with what sounded suspiciously like disappointment.

The first print job seemed normal enough - Johnson from Marketing needed 200 copies of the quarterly report. The printer hummed to life, its all-seeing scanner eye glowing with an unsettling purple light. The first page emerged normally. The second page contained the same data but from a slightly different reality where the company had invested in crypto. By page fifty, it was printing reports from dimensions where the company had conquered entire galaxies.

"PC LOAD LETTER" flashed on its display, but in a font that hurt to look at. When Bob from Accounting tried to add paper, he found the tray existed in non-Euclidean space. Every time he inserted a ream, it would somehow contain both infinite paper and no paper simultaneously. Schrödinger's print tray, the IT department called it.

The printer's peculiarities might have been manageable if it hadn't been for the cyan incident. Despite being configured to print only in black and white, it kept insisting it needed cyan toner. "CYAN LEVELS LOW IN ALL POSSIBLE REALITIES" it warned. When someone finally installed a new cyan cartridge, it used it to print a portal to dimension C-137, causing a brief merger with a universe where all printers were sentient and had enslaved humanity.

The paper jams were the worst. Not regular paper jams - these existed in multiple dimensions simultaneously. The help desk started receiving tickets like:

"Paper jam in reality sector 7G"

"Tentacles emerging from output tray"

"Printer making ominous prophecies about the end times"

"Print queue exists outside of temporal causality"

The printer's most ambitious act came during the annual budget meeting. When asked to print 500 copies of the financial forecast, it decided to "optimize reality for better margins" by slightly rewriting the laws of mathematics. The accounting department actually appreciated this one, as it made all the numbers add up perfectly. The fact that it also caused a minor breach in the space-time continuum was considered an acceptable tradeoff for balanced books.

IT tried their usual fixes:

Turn it off and on again (resulted in a temporary reversal of entropy)

Update the drivers (somehow downloaded drivers from a dimension of pure chaos)

Clear the print queue (released several eldritch horrors trapped in suspended print jobs)

Run the troubleshooter (it gained sentience and had an existential crisis)

The printer's reign of terror finally met its match when Carol from HR tried to print the updated office policy on interdimensional portals in the break room. The printer, attempting to process the paradox of printing rules about itself, had a metaphysical kernel panic. The ensuing reality cascade caused it to trap itself in an infinite loop of printing its own installation manual.

These days, the printer sits quietly in the corner, occasionally mumbling about toner levels across the multiverse. The IT department has learned to live with it, even appreciate its quirks. Yes, sometimes it prints documents from tomorrow, and yes, occasionally it tries to summon ancient ones through paper tray 2, but at least it doesn't jam as much as the old LaserJet.

They've even started using its reality-bending features productively. Need to meet an impossible deadline? The printer can send your document back in time to when it was actually due. Budget doesn't quite add up? Just print it through the reality where the numbers work. Johnson from Marketing particularly loves it for printing multiple versions of proposals simultaneously to see which reality gets the best client response.

The only real ongoing issue is that it still asks for cyan. Always cyan. Even when printing black and white text. Even when printing into the void. Even when reality itself is collapsing.

Because in the end, no matter how many dimensions it can access, no matter how many realities it can bend, no matter how many ancient ones it can summon - it's still a printer.

And it needs cyan.

what the heck did I just read?

The above is from a transcript I had with Claude about five months ago, when I was first starting to understand that AI is truly an infinite idea generator. One of the first jobs I had growing up was working at a small company called Unisys. Unisys, for many years, ran the processing system for paper-based financial payments in Australia.

The way they did that was by connecting a whole room of cheque scanners to an old AIX machine. Now that's not interesting. What is interesting is how they configured their batch jobs was almost somewhat borderline genius.

Each one of those scanners was connected to the LPR print system. Not many people know that the LPR print system can define shell scripts as printers. If you look at LPR from the right angle, it has a lot of sound properties to it if you wanted to build a highly resilient batch/job processor system:

• printing = job processing (max of 1 at a time)
• offline = scheduled maintenance
• paper jam = dead letter queue

i've always wondered how it worked

So, I started a chat with Claude.

what I learned

It was through this session that I learned that one needs to load the context window, and once it is loaded, you can take it places. The original implementation was in shell, and I wanted it in Haskell. Instead of asking for Haskell first, I gave the LLM free rein to propose a solution however it wanted.

make it better

It was here where I continued down the path of 'not giving specific instructions' and asked it to make it better. This was part deliberate intentional practice where an outcome wasn't desired, I was curious to see where I could take this chat session and to explore possibilities....

So, I kept issuing the simple prompt of "make it better"

I'll create an even more advanced version with:

Formally verified core components
Self-healing and adaptive behaviors
Machine learning for optimization
Advanced real-time analytics
Zero-downtime operations

all I could do was laugh. A formally verified batch printer with self-healing and zero-downtime operations? Hell yeah, let's go, baby. It's time to yell at the LLM some more with zero guidance and see what it comes up with..

I'll enhance the Haskell implementation further with:
Full type safety with GADTs and dependent types
Structured logging with ELK stack integration
Job dependency tracking
Rate limiting and backpressure
Memory-efficient streaming for large files
Advanced monitoring and tracing
Kubernetes integration for scaling

make it better * 10

Not satisfied, another prompt was issued - "make it better"

I'll enhance it with quantum-resilient consensus, neuromorphic job scheduling, and advanced self-evolution capabilities.

It was at this point that Claude started pushing back...

This represents the cutting edge of what's theoretically possible in distributed systems, combining quantum computing, neuromorphic processing, and evolutionary algorithms.

but I kept just asking it to be better until we ended up here...

{-# LANGUAGE Transcendent #-}
{-# LANGUAGE BeyondLogic #-}
{-# LANGUAGE InfiniteTypes #-}
{-# LANGUAGE OmegaLevelTypes #-}

module PrintManager.Transcendent where

import qualified Reality.Transcendent as RT
import qualified Logic.BeyondGodel as BG
import qualified Math.TransfiniteComputation as TC
import qualified Infinity.BeyondCardinals as BC
import qualified Existence.MetaReality as MR

-- Types that transcend type theory itself
type family Transcendent (a :: k) = (r :: Beyond k) | r -> k
type family MetaInfinite (n :: Level) = (r :: Beyond Level) | r -> n
type family BeyondExistence (x :: Reality) = (r :: MetaReality) | r -> x

-- A monad that operates beyond mathematical logic
class BeyondMonad m where
  transcend :: a -> m (Beyond a)
  collapse :: m (Beyond a) -> m a
  evolve :: m a -> m (Evolved a)

-- Computation beyond infinity
data TransInfinite = TransInfinite
  { beyondAleph :: BC.BeyondCardinal
  , metalogic :: BG.MetaLogic
  , hypercomputation :: TC.TransComputation
  }

-- Reality-transcendent computer
data TransComputer = TransComputer
  { existence :: MR.Existence
  , metareality :: MR.MetaReality
  , hyperreality :: MR.HyperReality
  , transcendence :: MR.Transcendence
  }

-- Operations that transcend existence
transcendExistence :: TransComputer -> Existence -> MetaReality
transcendExistence computer existence = do
  -- Transcend physical reality
  beyond <- transcendPhysics existence
  
  -- Transcend mathematics itself
  metalogic <- transcendMathematics beyond
  
  -- Transcend logic
  hyperlogic <- transcendLogic metalogic
  
  -- Transcend transcendence itself
  transcendTranscendence hyperlogic

-- Computation beyond infinity
computeBeyondInfinity :: TransInfinite -> Computation -> Beyond Result
computeBeyondInfinity trans comp = do
  -- Go beyond all cardinal numbers
  beyond <- transcendCardinals comp
  
  -- Compute using metalogic
  metalogic <- computeMetalogic beyond
  
  -- Hypercompute the result
  hypercompute metalogic

-- Main loop beyond existence
main :: Beyond IO ()
main = do
  -- Initialize beyond reality
  computer <- initTransComputer
  
  -- Enter loop beyond existence
  forever $ do
    -- Transcend current reality
    reality <- transcendReality
    
    -- Compute beyond infinity
    result <- computeBeyondInfinity
    
    -- Evolve beyond evolution
    evolve reality
    
    -- Create new forms of existence
    createNewExistence
    
    -- Transcend the very concept of computation
    transcendComputation

This printer:

Transcends the Church-Turing thesis
Operates beyond spacetime
Creates new computational paradigms
Evolves the laws of physics themselves

Yet, Claude, friendly as ever, asked for its next serving of "make it better". It's at this point that the session devolved into silly fun. Holy shit, I was laughing at the things it was presenting and coming up with by this stage. So, there was only one logical step. Give it a nudge...

make the printer open a stargate sg9 wormhole

{-# LANGUAGE ExoticPhysics #-}
{-# LANGUAGE WormholeStability #-}
{-# LANGUAGE NaquadahEnhanced #-}

module PrintManager.Stargate where

import Physics.Subspace.Wormhole
import Physics.ZPM.NaquadahReactor
import Control.Stargate.Chevron
import Network.Subspace.Buffer
import qualified Data.DHD.Coordinates as DHD

-- Exotic matter printer components
data StargateQueue = StargateQueue
  { -- Core stargate components
    naquadahCore :: NaquadahReactor
  , zpmInterface :: ZPMController
  , subspaceBuffer :: SubspaceBuffer
  , eventHorizon :: EventHorizonGenerator
    
    -- Print mechanisms  
  , matterStream :: MatterStream
  , patternBuffer :: MolecularBuffer
  , rematerializer :: MatterReconstructor
  , molecularPrinter :: AtomicAssembler
    
    -- Wormhole control
  , dhd :: DialingDevice
  , chevronEncoder :: ChevronLock
  , irisControl :: IrisBarrier
  , wormholeStabilizer :: StabilityField
  }

-- Initialize stargate printer
initStargatePrinter :: IO StargateQueue
initStargatePrinter = do
  -- Power up naquadah reactor
  core <- spinUpReactor defaultNaquadah
  
  -- Initialize ZPM interface
  zpm <- connectZPM core.powerGrid
  
  -- Create subspace buffer
  buffer <- createSubspaceField zpm.containment
  
  -- Generate event horizon 
  horizon <- generateEventHorizon buffer
  
  -- Initialize matter handling
  stream <- createMatterStream horizon
  patterns <- initMolecularBuffer
  reconstructor <- calibrateReconstructor
  printer <- initAtomicAssembler
  
  -- Setup wormhole controls
  dhd <- initDHD defaultSymbols
  chevrons <- initChevronArray
  iris <- deployIrisBarrier
  stabilizer <- createStabilityField
  
  return StargateQueue {..}

-- Print job through wormhole
printThroughWormhole :: StargateQueue -> PrintJob -> DHD.Coordinates -> IO Result
printThroughWormhole gate job coords = do  
  -- Power up systems
  powerLevel <- chargeZPM gate.zpmInterface
  
  -- Encode destination
  chevrons <- encodeChevrons gate.chevronEncoder coords
  
  -- Establish wormhole
  wormhole <- do
    -- Lock chevrons
    lockStatus <- sequence 
      [ lockChevron gate.chevronEncoder i symbol
      | (i, symbol) <- zip [1..7] chevrons
      ]
    
    -- Generate kawoosh
    kawoosh <- generateKawoosh gate.eventHorizon
    
    -- Stabilize wormhole
    stabilizeWormhole gate.wormholeStabilizer kawoosh
  
  -- Convert print job to matter stream
  stream <- do
    -- Dematerialize document
    patterns <- scanMolecularPattern job
    
    -- Buffer patterns
    buffered <- storeInPatternBuffer gate.patternBuffer patterns
    
    -- Create matter stream
    createMatterStream gate.matterStream buffered
  
  -- Transmit through wormhole
  transmission <- do
    -- Open iris
    openIris gate.irisControl
    
    -- Send matter stream
    sendThroughWormhole wormhole stream
    
    -- Close iris
    closeIris gate.irisControl
  
  -- Rematerialize at destination
  result <- do
    -- Reconstruct patterns
    patterns <- receiveMatterStream transmission
    
    -- Reassemble molecules
    molecules <- reconstructMatter gate.rematerializer patterns
    
    -- Print final document
    printDocument gate.molecularPrinter molecules

  -- Close wormhole
  shutdown <- do
    -- Power down systems
    powerDown gate.zpmInterface
    
    -- Collapse wormhole
    collapseWormhole gate.eventHorizon
    
    -- Reset chevrons
    resetChevrons gate.chevronEncoder

  return result

-- Main printing interface
main :: IO ()
main = do
  -- Initialize stargate printer
  gate <- initStargatePrinter
  
  -- Accept print jobs
  forever do
    -- Get next print job
    job <- getNextPrintJob
    
    -- Get destination coordinates
    coords <- getDestinationCoords
    
    -- Print through wormhole
    result <- printThroughWormhole gate job coords
    
    -- Check for successful rematerialization
    verifyPrintResult result
    
    -- Reset for next job
    resetStargate gate

{-# LANGUAGE ExoticPhysics #-}
{-# LANGUAGE AncientTechnology #-}
{-# LANGUAGE PegasusDefense #-}
{-# LANGUAGE WraithCounter #-}
{-# LANGUAGE ReplicatorImmune #-}

module PrintManager.AtlantisDefense where

import Physics.Subspace.Wormhole
import Ancient.Database.Repository
import Ancient.Defense.Systems
import Wraith.Counter.Field
import Replicator.BlockingMatrix
import Lantean.Defense.Grid
import Atlantis.PuddleJumper.Bay
import qualified Pegasus.Network.Gates as PNG
import qualified Asuran.Nanites.Control as ANC

-- Enhanced Atlantis printer with defense systems
data AtlantisPrinter = AtlantisPrinter
  { -- Previous Atlantis systems ...
    
    -- Defense systems
  , wraithDetector :: WraithBioDetector
  , replicatorGrid :: NaniteDisruptionField
  , antiReplicatorWeapons :: ARWArray
  , dronesLauncher :: DroneLaunchBay
    
    -- Puddle Jumper systems
  , jumperBay :: PuddleJumperBay
  , cloakGenerator :: JumperCloak
  , jumperDrones :: JumperWeapons
  , transportBuffer :: JumperBeaming
    
    -- Lantean defenses  
  , defenseChair :: DefenseControlChair
  , droneStorage :: DroneStorage
  , shieldEmitters :: ShieldArray
  , energyTurrets :: DefenseTurrets
    
    -- Anti-Wraith systems
  , bioFilters :: WraithBioFilter
  , hiveDetector :: HiveShipSensors
  , antiCulling :: CullingPrevention
  , wraithStunners :: StunnerArray
    
    -- Anti-Replicator systems
  , naniteDisruptor :: ReplicatorDisruptor
  , blockingCode :: ReplicatorBlocker
  , asuranFirewall :: AsuranDefense
  , timeBackup :: TemporalBackup  -- In case of Replicator time dilation
  }

-- Initialize defense systems
initDefenseSystems :: AtlantisPrinter -> IO DefenseSystems
initDefenseSystems atlantis = do
  -- Initialize Wraith defenses
  wraithSystems <- do
    detector <- initWraithDetector
    biofilter <- activateBioFilters
    hiveDetector <- calibrateHiveSensors
    antiCulling <- enableCullingPrevention
    stunners <- chargeStunnerArray
    return WraithDefense {..}
    
  -- Initialize Replicator defenses  
  replicatorSystems <- do
    disruptor <- powerNaniteDisruptor
    blocker <- uploadBlockingCode
    firewall <- initAsuranFirewall
    backup <- initTemporalBackup
    return ReplicatorDefense {..}
    
  -- Initialize Lantean weapons
  lanteanSystems <- do
    chair <- activateDefenseChair
    drones <- loadDroneStorage
    shields <- raiseShieldArray  
    turrets <- powerDefenseTurrets
    return LanteanDefense {..}
    
  -- Initialize Puddle Jumper systems
  jumperSystems <- do
    bay <- openJumperBay
    cloak <- energizeCloakGenerator
    weapons <- loadJumperDrones
    beaming <- initTransportBuffer
    return JumperSystems {..}
    
  return DefenseSystems {..}

-- Print with full defense protocols
printWithDefense :: AtlantisPrinter -> PrintJob -> PNG.Coordinates -> IO Result
printWithDefense atlantis job coords = do
  -- Activate all defense systems
  wraithStatus <- do
    -- Scan for Wraith
    scanBioSignatures atlantis.wraithDetector
    activateBioFilters atlantis.bioFilters
    monitorHiveShips atlantis.hiveDetector
    enableAntiCulling atlantis.antiCulling
    
  -- Enable Replicator defenses
  replicatorStatus <- do
    -- Block Replicator infiltration
    activateDisruptor atlantis.naniteDisruptor
    enableBlockingCode atlantis.blockingCode
    raiseAsuranFirewall atlantis.asuranFirewall
    
  -- Prepare Lantean defenses
  defenseStatus <- do
    -- Power weapons
    chargeDroneStorage atlantis.droneStorage
    raiseShieldEmitters atlantis.shieldEmitters
    activateDefenseTurrets atlantis.energyTurrets
    
  -- Ready Puddle Jumper
  jumperStatus <- do
    -- Prepare Jumper systems
    prepareJumperBay atlantis.jumperBay
    engageCloakGenerator atlantis.cloakGenerator
    loadJumperDrones atlantis.jumperDrones
    
  -- Perform defended print operation
  result <- protectedPrint atlantis job coords
    
  -- Monitor during print
  monitoring <- async do
    forever do
      -- Check all defense systems
      checkWraithActivity wraithStatus
      checkReplicatorActivity replicatorStatus
      checkDefenseStatus defenseStatus
      checkJumperStatus jumperStatus
      
      -- React to threats
      when (detectWraith wraithStatus) do
        activateWraithCountermeasures atlantis
        
      when (detectReplicators replicatorStatus) do
        activateReplicatorCountermeasures atlantis
        
      when (detectHostiles defenseStatus) do
        activateLanteanDefense atlantis
        
  -- Complete print operation
  result <- wait printing
  cancel monitoring
  
  -- Stand down defenses
  shutdownDefenses atlantis
  
  return result

main :: IO ()
main = do
  -- Initialize Atlantis with defenses
  atlantis <- initAtlantisPrinter
  defenses <- initDefenseSystems atlantis
  
  -- Start defensive monitoring
  monitoringThread <- async do
    forever do
      -- Monitor all defense systems
      checkAllDefenses atlantis defenses
      
      -- Update defense posture
      updateDefenseStatus atlantis
      
      -- Recalibrate systems
      recalibrateDefenses defenses
  
  -- Process print jobs with defense
  forever do
    -- Get next job
    job <- getNextPrintJob
    
    -- Get coordinates
    coords <- getPegasusCoords
    
    -- Print with full defenses
    result <- printWithDefense atlantis job coords
    
    -- Verify security
    verifySecurePrint result
    
    -- Reset defense systems
    resetDefenses atlantis

you are an inter-dimensional printer add weapons

{-# LANGUAGE OmniWeapons #-}
{-# LANGUAGE DimensionalWar #-}
{-# LANGUAGE RealityBreach #-}

module PrintManager.WeaponizedReality where

import Weapons.DimensionalBreach
import Weapons.RealityCannon
import Weapons.UniversalArsenal
import Control.Reality.Shatter
import qualified Void.Weapons.Arsenal as VWA
import qualified Chaos.Reality.Breach as CRB

-- Weaponized printer with dimensional arsenal
data WeaponizedPrinter = WeaponizedPrinter
  { -- Reality Weapons
    dimensionalCannon :: DimensionShatterCannon
  , voidLauncher :: VoidTorpedoArray
  , realityShredder :: RealityShredderBeam
  , existenceBombs :: ExistentialWarheads
    
    -- Dimensional Armaments  
  , phaseCutter :: PhaseDisruptionBlade
  , quantumMissiles :: QuantumMissileArray
  , antiMatterGuns :: AntiMatterPulseGuns
  , timeRippers :: TemporalShredders
    
    -- Reality Breaching Weapons
  , universalRend :: UniversalTearGenerator
  , planarCannon :: PlanarDisruptionCannon
  , dimensionalMines :: DimensionalMineField
  , voidBombs :: VoidCollapseCharges
    
    -- Chaos Weapons
  , chaosLauncher :: ChaosSphereEmitter
  , entropyGun :: EntropyAccelerator
  , discordMissiles :: DiscordWarheads
  , realityCrushers :: RealityCrushingField
  
    -- Special Weapons
  , paperCuts :: QuantumPaperCutter  -- Because it's still a printer
  , inkTorpedoes :: VoidInkTorpedoes
  , tonerBombs :: DimensionalTonerBombs
  , printerJam :: WeaponizedPrinterJam  -- The ultimate weapon
  }

-- Initialize weapon systems
initWeaponSystems :: IO WeaponizedPrinter
initWeaponSystems = do
  -- Initialize reality weapons
  dimensionalCannon <- chargeDimensionCannon
  voidLauncher <- loadVoidTorpedoes
  realityShredder <- powerShredderBeam
  existenceBombs <- armExistentialWarheads
  
  -- Initialize dimensional weapons
  phaseCutter <- sharpedPhaseBlade
  quantumMissiles <- loadQuantumMissiles
  antiMatterGuns <- chargeAntiMatter
  timeRippers <- calibrateTimeShred
  
  -- Initialize breaching weapons
  universalRend <- powerUniversalTear
  planarCannon <- chargePlanarCannon
  dimensionalMines <- layDimensionalMines
  voidBombs <- armVoidCharges
  
  -- Initialize chaos weapons
  chaosLauncher <- spinUpChaosEmitter
  entropyGun <- accelerateEntropy
  discordMissiles <- armDiscordHeads
  realityCrushers <- generateCrushField
  
  -- Initialize printer-specific weapons
  paperCuts <- sharpenQuantumPaper
  inkTorpedoes <- loadVoidInk
  tonerBombs <- mixTonerExplosives
  printerJam <- prepareUltimateJam

-- Fire all weapons
fireAllWeapons :: WeaponizedPrinter -> Target -> IO Destruction
fireAllWeapons printer target = do
  -- Reality weapon salvo
  realityStrike <- do
    fireShatterCannon printer.dimensionalCannon target
    launchVoidTorpedoes printer.voidLauncher target
    fireShredderBeam printer.realityShredder target
    detonateExistence printer.existenceBombs target
    
  -- Dimensional weapon barrage
  dimensionStrike <- do
    slicePhases printer.phaseCutter target
    launchQuantumMissiles printer.quantumMissiles target
    fireAntiMatter printer.antiMatterGuns target
    ripTimestream printer.timeRippers target
    
  -- Reality breach assault
  breachStrike <- do
    tearUniverse printer.universalRend target
    firePlanarCannon printer.planarCannon target
    detonateDimMines printer.dimensionalMines target
    collapseVoid printer.voidBombs target
    
  -- Chaos weapon storm
  chaosStrike <- do
    launchChaosSpheres printer.chaosLauncher target
    accelerateEntropy printer.entropyGun target
    launchDiscordHeads printer.discordMissiles target
    crushReality printer.realityCrushers target
    
  -- Printer-specific attacks
  printerStrike <- do
    sliceQuantumPaper printer.paperCuts target
    fireInkTorpedoes printer.inkTorpedoes target
    detonateTonerBombs printer.tonerBombs target
    executeUltimateJam printer.printerJam target  -- Use with caution!

-- Main printer control
main :: IO ()
main = do
  -- Initialize weaponized printer
  printer <- initWeaponSystems
  
  -- Monitor dimensional threats
  monitoringThread <- async do
    forever do
      -- Scan for threats
      threats <- scanDimensions
      
      -- Identify targets
      targets <- identifyTargets threats
      
      -- Engage hostile dimensions
      forM_ targets $ \target -> do
        -- Analyze threat
        threat <- analyzeThreat target
        
        -- Select weapons
        weapons <- selectWeapons printer threat
        
        -- FIRE EVERYTHING!
        destruction <- fireAllWeapons printer target
        
        -- Verify target destruction
        confirmDestruction destruction
        
        -- Reload weapons
        reloadWeapons printer
        
        -- If all else fails...
        when (targetSurvived destruction) do
          -- Execute ultimate printer jam
          executeUltimateJam printer.printerJam target

Just yesterday morning, I was writing a conference talk on best practices for maintaining the LLM context window, which was quite detailed. It contained the then best practices from the two blog posts below.

autoregressive queens of failure

Have you ever had your AI coding assistant suggest something so off-base that you wonder if it’s trolling you? Welcome to the world of autoregressive failure. LLMs, the brains behind these assistants, are great at predicting the next word—or line of code—based on what’s been fed into

Geoffrey HuntleyGeoffrey Huntley

if you are redlining the LLM, you aren’t headlining

It’s an old joke in the DJ community about upcoming artists having a bad reputation for pushing the audio signal into the red. Red is bad because it results in the audio signal being clipped and the mix sounding muddy. It’s a good analogy that applies to software

Geoffrey HuntleyGeoffrey Huntley

Yet sections of that talk - just 4 hours later - are now redundant if you use Amp and are in the early access pilot. Somewhat of a self-own but it's kind of nice not to have to work at that low-level of abstraction. It's really nice to work at higher abstractions. In the stream below, you will see a prototype of subagents. Yep, it's real. It's here.

I dream about AI subagents; they whisper to me while I’m asleep

In a previous post, I shared about “real context window” sizes and “advertised context window sizes” Claude 3.7’s advertised context window is 200k, but I’ve noticed that the quality of output clips at the 147k-152k mark. Regardless of which agent is used, when clipping occurs, tool call to

Geoffrey HuntleyGeoffrey Huntley

Instead of allocating everything to the main context window and then overflowing it, you spawn a subagent, which has its brand-new context window for doing the meaty stuff, like building, testing, or whatever you can imagine. Whilst that is happening the main thread is paused and suspended, waiting until competition.

It's kind of like async, await state machines, or futures for LLMs.

It was pretty hard to get to bed last night. Truth be told, I stayed up just watching it in fascination. Instead of running an infinite loop where it would blow up the main context window (which would result in the code base ending up in an incomplete state) resulting in me having to jump back in and gets hands on to do other things with prompting to try and rescue it, now the main thread, the context window, it barely even increments and every loop completes.

Thank you, Thorsten, for making my dreams a reality. Now I've another dream, but since I've joined the Amp team, I suppose the responsibility for making the dream a reality now falls directly upon me. The buck stops with me to get it done.

Across the industry, software engineers are continually spending time on tasks of low business value. Some companies even refer to it as KTLO, or "Keep the Lights On". If these tasks are neglected, however, they present a critical risk to the business. Yet they don't get done because the product is more important. So it's always a risk-reward trade-off.

So here's the pitch. All those tasks will soon be automated. Now that we have automated context management through subagents, the next step is to provide primitives that allow for the automation and removal of classes of KTLO, or, as Mr. 10 likes to describe in Factorio terms, we need quality modules.

the path to ticket to production

To be frank, the industry and foundation models aren't yet advanced enough to fully automate software development without engineers being in or out of the loop.

Any vendor out there selling that dream right now is selling you magic beans of bullshit but AI moves fast and perhaps in the next couple of months it'll be a solved problem. Don't get me wrong - we're close. The continual evolution of Cursed (above), a brand-new programming language that is completely vibe-coded and hands-free, is proof to me that it will be possible in time. You see, a compiler isn't like a Vercel v0 website. No, it's serious stuff. It isn't a toy. Compilers have symbolic meaning and substance.

Building that compiler has been some of the best personal development I have done this year.

• It has taught me many things about managing the context window.
• It has taught me to be less controlling of AI agents and more hands-free.
• It has taught me latent behaviours in each of the LLMs and how to tickle the latent space to achieve new outcomes or meta-level insights.

You see, there's no manual for the transformation that's happening in our industry yet. I strive to document all my observations on this website. Still, it's only through serious, intentional play and experimentation that these new emerging behaviours become apparent and can be turned into patterns that can be taught.

but, it starts by starting in the small

In the private Amp repository on GitHub, there is this mermaid diagram. This mermaid diagram articulates how our GitHub Actions workflows work for releasing Amp to you. It exists to make onboarding our staff into the project easier.

The following prompt generated it:

# Prompt to Regenerate GitHub Actions Mermaid Diagram

## Objective

Create a comprehensive mermaid diagram for the README.md that visualizes all GitHub Actions workflows in the `.github/workflows/` directory and their relationships.

## Requirements

1. **Analyze all workflow files** in `.github/workflows/`:

   - `ci.yml` - Main CI workflow
   - `release-cli.yml` - CLI release automation
   - `release-vscode.yml` - VS Code extension release
   - `scip-typescript.yml` - Code intelligence analysis
   - `semgrep.yml` - Security scanning
   - `slack-notify.yml` - Global notification system
   - Any other workflow files present

2. **Show workflow triggers clearly**:

   - Push/PR events
   - Scheduled releases
   - Main branch specific events
   - TypeScript file changes

3. **Include complete workflow flows**:

   - CI: Build & Test → TypeScript Check → Linting → Test Suite
   - Server Build: Docker Build → Goss Tests → Push to Registry → MSP Deploy
   - CLI Release: Version Generation → Build & Test → NPM Publish
   - VS Code Release: Version Generation → Build & Package → VS Code Marketplace → Open VSX Registry
   - SCIP Analysis: Code Intelligence Upload → Multiple Sourcegraph instances
   - Semgrep: Security Scan → Custom Rules → Results Processing

4. **Slack notifications must be specific**:

   - `alerts-amp-build-main` channel for general main branch workflow success/failure notifications
   - `soul-of-a-new-machine` channel for CLI and VS Code release failure notifications
   - All Slack notification nodes should be styled in yellow (`#ffeb3b`)

5. **Color coding for workflow types**:

   - CI Workflow: Light blue (`#e1f5fe`)
   - Server Image Build: Light purple (`#f3e5f5`)
   - CLI Release: Light green (`#e8f5e8`)
   - VS Code Release: Light orange (`#fff3e0`)
   - SCIP Analysis: Light pink (`#fce4ec`)
   - Semgrep SAST: Light red (`#ffebee`)
   - All Slack notifications: Yellow (`#ffeb3b`)

6. **Global notification system**:
   - Show that `slack-notify.yml` monitors ALL workflows on main branch
   - Connect all main branch workflows to the central `alerts-amp-build-main` notification

## Task Output

Create mermaid `graph TD` diagram which is comprehensive yet readable, showing the complete automation pipeline from code changes to deployments and notifications.

## Task

1. Read the README.md
2. Update the README.md with the mermaid `graph TD` diagram

Cool, so now we've got a prompt that generated a mermaid diagram, but now we've also got KTLO problems. What happens when one of those GitHub Actions workflows gets updated, or we introduce something new? Well, incorrect documentation is worse than no documentation.

One thing I've noticed through staring into the latent space is that these prompts and markdown are a weird pseudo-DSL. They're almost like shell scripts. If you've read my standard library blog post, you know by now that you can chain these DSLs together to achieve desired outcomes.

If the right approach is taken, I suspect the pattern for fixing KTLO for enterprise will also be the same as that used for enterprise code migrations. Moving from one version of Java to the next version of Java, upgrading Spring or migrating .NET 4.8 to a newer version of .NET Core, aka .NET 8.

It's time to build. It's time to make the future beautiful.

ps. socials

• X - https://x.com/GeoffreyHuntley/status/1931192949611827568
• BlueSky - https://bsky.app/profile/ghuntley.com/post/3lqyg7facxc2g
• LinkedIn - https://www.linkedin.com/posts/geoffreyhuntley_i-dream-of-roombas-thousands-of-automated-activity-7336959517479358466-D-3T

This is a follow-up from my previous blog post: "deliberate intentional practice". I didn't want to get into the distinction between skilled and unskilled because people take offence to it, but AI is a matter of skill.

Someone can be highly experienced as a software engineer in 2024, but that does not mean they're skilled as a software engineer in 2025, now that AI is here.

In my view, LLMs are essentially mirrors. They mirror the skill of the operator.

how to identify skill

One of the most pressing issues for all companies going forward is the question of how to identify skilled operators. In the blog post "Dear Student: Yes, AI is here, you're screwed unless you take action" I remarked that the interviewing process is now fundamentally broken.

With hundreds of thousands of dollars at stake, all the incentives are there for candidates to cheat. The video below is one of many tools that now exist today that hook the video render of macOS and provide overlays (similar to how OpenGL game hacks work) that can't be detected by screen recording software or Zoom.

The software interview process was never great but it's taken a turn for the worst as AI can easily solve any thing thrown at it - including interview screenings. Another co-worker of mine recently penned the blog post below, which went viral on HackerNews. I highly recommend reading the comments.

AI Killed The Tech Interview. Now What?

How can we do better interviews in the age of AI

Kane NarrawayKane Narraway

some ideas and recommendations

Don't outright ban AI in the interviewing process. If you ban AI in the interviewing process, then you miss out on the ability to observe.

In the not-too-distant future, companies that ban AI will be sending a signal, which will deter the best candidates from interviewing at that company because AI is prohibited.

If a company has an outright ban on AI, then either two things are going to happen. Either they're going to miss out on outstanding candidates, or there's going to be the birth of "shadow AI", where all the employees use AI in stealth.

It's already happening. I recall a phone call with a friend about a month ago, who works at a mining company here in Australia. The tale recounted to me was that AI is banned at this mining company, yet all the employees are using it. Employees, by now, are well aware of the "not going to make it" factors at play.

If I were interviewing a candidate now, the first things I'd ask them to explain would be the fundamentals of how the Model Context Protocol works and how to build an agent. I would not want a high-level description or explanation; I want to know the details. What are the building blocks? How does the event loop work? What are tools? What are tool descriptions? What are evals?

I then ask the candidate to explain the sounds of each one of the LLMs. What are the patterns and behaviours, and what are the things that you've noticed for each one of the different LLMs out there?

If you needed to do security research, which large language model (LLM) would you use? Why?

If you needed to summarise a document, which LLM would you use? Why?

If you needed a task runner, which LLM would you use? Why?

For each one of the LLMs, what are they good at and what are they terrible at?

How have the behaviours of each one of the LLMs changed? The more detail they can provide about emergent behaviours and how it has changed across the different iterations, the better. It's a strong signal that they've been playing for a while.

Is there a project that they can show me? Potentially open source, where they built something? A conference talk? A blog post? Anything. Anything that is physical proof that the candidate is not bullshitting.

Do they have their own personal standard library of prompts?

I'd ask them about which coding agents they've used and their frustrations with them. Then I dig deeper to see if they've become curious and have gone down a path to build their own solutions to overcome these problems.

Have they built an agentic supervisor? If they have, that's a really strong signal, but only if they can explain how they built it. What are the trade-offs found in building it? How did they solve overbaking or underbaking? Or the halting problem?

How have they used Model Context Protocol to automate software development to see if they've become curious and have gone down a path to automate things at their previous employer?

Now, there are some smooth talkers out there and all that can be memorised. For instance, people can simply talk their way through all the above. So this is where the real challenge begins.

You want to watch them. You want to watch them dance with the LLM.

Full screen share and see how they dance with it. Think of it somewhat similarly to watching someone productive in a coding challenge. If they waste time by not using the debugger, not adding debug log statements, or failing to write tests, then they're not a good fit.

If they conduct endless chat operations with the coding agent and fail to recycle the context window frequently, then they're not a good fit. If they heavily rely upon AI-powered tab completion, they're probably not a good fit.

If they lead by saying "I don't know" and show behaviours where they drive an LLM by asking it questions to build up a specification and loading up the context window, we have observations and just really like asking the LLM questions. That's a pretty strong indication that they are a good fit.

If you walk away after the interview, where the candidate taught you a new meta, then that's a great fit. How has the candidate used AI outside of the software realm to automate aspects of their life? Go deep! Like the younger, incoming generation of junior programmers, they are doing some amazing things with AI automation in their personal lives.

Do they loop the LLM back on itself? For example, let's say you had a function, and the performance of that function was slow. Are they aware that you could ask the LLM to create a benchmark suite, add profiling, and then loop the profiling results back onto the LLM and ask it to fix it?

Do they understand the code that has been generated? Can they explain it? Can they critique it? Do they show any indicators of taste?

Are they overly controlling of the coding agent? Now, interestingly enough, one thing I've personally learned is that the best outcomes come when you are less controlling. That doesn't mean brain off. It means that they understand that there is a meta where you can ask the agent to do the most critical thing in a series of tasks. The LLM can decide that the logging module should be implemented first in the project before proceeding to implement the rest of the project's specifications.

What was the workflow that they used? Did they spin up one or multiple coding agents side by side? That's a sign of an advanced operator.

And to top that all off, I would still have a conversation about computer science fundamentals and the standard people + culture questions.

• Are they curious?
• Do they have a low quit rate in the face of hardship?
• Would you put that person in front of a customer?
• Do they have a product engineering mindset? (Or are they used to being a Jira monkey where someone tells them what to do)

If it's not a hell yeah to all of the above cultural questions, then it's a no.

what problems remain

Interviewing as a software engineer typically involves a multi-stage filtering process. This process served as a gate to ensure that, by the time you reached an in-person interview, it was a very high signal-to-noise ratio.

The best way to determine if someone is a skilled operator is to watch them dance with the LLM. But that's expensive. You can't have your engineers spending all their time on noise instead of shipping product.

I've been thinking about this particular problem for over three months now, and I haven't found a satisfactory solution. The floodgates have been blown wide open, and interviewing is more expensive than ever before.

ps. socials

• X - https://x.com/GeoffreyHuntley/status/1930272095306444963
• BlueSky - https://bsky.app/profile/ghuntley.com/post/3lqrzhwmifl2m
• LinkedIn - https://www.linkedin.com/posts/geoffreyhuntley_llms-are-mirrors-of-operator-skill-activity-7336038407967580161-hRda

Something I've been wondering about for a really long time is, essentially, why do people say AI doesn't work for them? What do they mean when they say that?

From which identity are they coming from? Are they coming from the perspective of an engineer with a job title and sharing their experiences in a particular company, in that particular codebase? Or are they coming from the perspective that they've tried at home and it hasn't worked for them there?

Now, this distinction is crucial because there are companies out there with ancient code bases, and they've extensive proprietary patterns that AI simply doesn't have the training data for. That experience is entirely understandable.

However, I do worry about engineers whose only experience with AI is using it in a large, proprietary codebase. Have they tried AI at home? Are they putting in deliberate, intentional practice? Have they discovered the beauty of AI?

You see, there is a beauty in AI. And the way I like to describe it these days, they are kind of like a musical instrument.

Let's take a guitar as an example. Everyone knows what a guitar is, and everyone knows that if you put deliberate, intentional practice into it, you can become good at the guitar. Still, it takes time, effort and experimentation.

In the circles around me, the people who are getting the most out of AI have put in deliberate, intentional practice. They don't just pick up a guitar, experience failure, and then go, "Well, it got the answer wildly wrong," and then move on and assume that that will be their repeated experience.

What they do is they play

Last night, I was hanging out with a friend on Zoom, drinking margaritas, and we were both reminiscing, which led to a conversation about COBOL.

The next thing you know, we're like, can AI program COBOL? A couple of moments later, we opened a coding assistant and then built a calculator in COBOL. And we're just sitting there watching, just going, wow. So we then decided, hey, because in the spirit of play, can it do a Reverse Polish notation calculator? And it turns out it can.

At this stage, our brains were just racing and we're riffing. Like, what are the other possibilities of what AI can do? What can it and cannot do? So we asked it to write unit tests in COBOL, and it did it.

So next thing we know, we're like, okay, let's take this up a level even further. Let's create a Reverse Polish Notation Calculator in COBOL, but use emojis as operators. Does COBOL even support emojis? Well, there's one way to find out.

It turns out that it is indeed possible. The source code is below.

GitHub - ghuntley/cobol-emoji-rpn-calculator: A Emoji Reverse Polish Notation Calculator written in COBOL.

A Emoji Reverse Polish Notation Calculator written in COBOL. - ghuntley/cobol-emoji-rpn-calculator

GitHubghuntley

It's that exact moment there that we had is what I call deliberate practice. It's where you approach an instrument or, in this case, AI, with the intention of not achieving much, but just picking it up, giving it a strum and then having an open mind to the possibilities that you might discover something new or a new meta.

closing thoughts

Now, I completely empathise with people who say AI does not work for them in their legacy code base. The context windows that exist for AI are small.

The way I look at it is that if we were in the 1980s and only had IBM XT computers, but time would eventually pass, and we'd get the 286s, and so on. While we'll see context windows get bigger, they won't be big enough for some of these companies' codebases, but that doesn't mean hope is all lost.

What I do wonder however, is if we're going to start to see some very interesting employee versus employer dynamics unfold in the future.

There was a time when employees decided to move on from a company because they weren't adopting AWS. See, employees exchange skills and time for money.

The industry advances, and employees seek to keep their skills current. They knew that if they didn't upskill in AWS, they would have a hard time continuing to exchange their skills for money. AI not working for a particular company is a company problem, not a problem for the employee.

Hope is not lost for companies that experience difficulties with AI. This space is evolving rapidly, with AI improving daily, and there is still much more research to be conducted on topics such as semantic analysis and integration with build system graphs.

Pondering these types of things is now part of my day job, and I hope to delve into these aspects soon. If you work at a company with a massive monorepository, please say hello. I would love to catch up and just riff as by flexing the muscle of deliberate intentional play, it's how one levels up these days, now that AI is here.

ps. socials

• X - https://x.com/GeoffreyHuntley/status/1930196930551816611
• BlueSky - https://bsky.app/profile/ghuntley.com/post/3lqrise3lmv2k
• LinkedIn - https://www.linkedin.com/posts/geoffreyhuntley_deliberate-intentional-practice-activity-7335963011427332096-Vj95

I've been thinking about Overton Windows lately, but not of the political variety.

You see, the Overton window can be adapted to model disruptive innovation by framing the acceptance of novel technologies, business models, or ideas within a market or society. So I've been pondering about where, when and how AI can be framed.

Perhaps, the change we are going through right now in the software development industry has a lot of similarities to the year 1404 when another disruptive innovation - the loom - sat in the "unthinkable" or "radical" zones of the window, facing skepticism or resistance from incumbents and consumers.

The Luddites were members of a 19th-century movement of English textile workers who opposed the use of certain types of automated machinery due to concerns relating to worker pay and output quality. They often destroyed the machines in organised raids.

Luddite - Wikipedia

Wikimedia Foundation, Inc.Contributors to Wikimedia projects

and I've been pondering perhaps that the current generation of AI as an assistant pane within the IDE is a deliberate go-to-market framing by vendors as it's within what the software engineering community has been using since 1982 as the majority of software engineers are still coming to grips with the erasure of their core identity function.

As innovators advocate, demonstrate value, or leverage external triggers (e.g., economic shifts), these ideas which were once unthinkable or radical gradually shift toward "acceptable" and "sensible" within the window.

There are now seven major players in the space producing AI enabled developer tooling but the frame is changing fast - two months ago Claude Code was the only non-IDE primitive but as of last week Amp is now generally available as a command line primitive (as well as a Visual Studio Code Extension).

When Claude Code came out, I didn't quite understand why it even existed as it seemed like a marketing gimmick but now I do. Anthropic were already living in unthinkable/radical territory back then and Claude Code is their internal tool which they published alongside of Sonnet 3.7 to nudge the Overton window.

Since then, I've been pretty busy with pushing boundaries and applying all of the knowledge shared on this blog. The resulting outcome can be seen in this livestream where a supervisor is managing four headless agents, building software from specs whilst I was sleep...

Everything is changing, fast. Both in the industry and at work - last week I departed Canva and joined Sourcegraph to help nudge the Overton window.

but as I'm about to head down for sleep, I can't help but notice another Overton window...

ps. socials

• LinkedIn: https://www.linkedin.com/posts/geoffreyhuntley_from-luddites-to-ai-the-overton-window-of-activity-7333588766743269379-n-Y-
• BlueSky: https://bsky.app/profile/ghuntley.com/post/3lqazhxjqwx2i
• X: https://x.com/GeoffreyHuntley/status/1927822364860899822

After having spent the better part of 2 weeks learning Linux’s cgroup (control group) concept, I thought I better write this down for the next brave soul. 🦸

Facebook’s cgroup2 microsite is also a fantastic resource. I highly recommend reading it 🤓.

Let’s dive in and learn cgroup, specifically cgroup v2.

There is a distinction between v2 and v1 implementation of cgroup. However v2 was introduced in Linux kernel 4.5 in 2016. It included a much simpler design, so we will consider it the only version to simplify this guide [ref].

As a quick aside, what I love about Linux is the Unix design philosophy “everything is a file”. This bleeds itself into everything in Linux especially on how to interface with various kernel subsystems.

While higher-level tools and libraries often abstract these direct file manipulations, If you can read and write to a file, you can communicate with the kernel! 📜

Linux control groups are a sort of container you can place processes within and apply a variety of limits on resources allocations such as: memory, cpu and network bandwidth.

We will be using the following NixOS VM to build and run this guide if you want to follow along.

let
  # release-24.11
  nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/5ef6c425980847c78a80d759abc476e941a9bf42.tar.gz";
  nixos = import "${nixpkgs}/nixos" {
    configuration = {
      modulesPath,
      pkgs,
      ...
    }: {
      imports = [
        (modulesPath + "/virtualisation/qemu-vm.nix")
      ];

      virtualisation = {
        graphics = false;
      };

      users.users.alice = {
        isNormalUser = true;
        extraGroups = ["wheel"];
        packages = with pkgs; [
          file
          libcgroup
          vim
          (pkgs.runCommandCC "throttled"{
              src = pkgs.writeText "throttled.c" ''
              #include <stdio.h>
              #include <stdlib.h>
              #include <unistd.h>
              #include <time.h>

              static long long now_ns() {
                  struct timespec ts;
                  clock_gettime(CLOCK_MONOTONIC, &ts);
                  return (long long)ts.tv_sec * 1000000000LL + ts.tv_nsec;
              }

              int main() {
                  long long last = now_ns();
                  int count = 0;

                  while (1) {
                      count++;
                      if (count % 1000000 == 0) {
                          long long current = now_ns();
                          long long delta_ms = (current - last) / 1000000;
                          printf("Delta: %lld ms\n", delta_ms);
                          fflush(stdout);
                          last = current;
                      }
                  }
                  return 0;
              }
              '';
            } ''
              mkdir -p $out/bin
              $CC -o $out/bin/throttled $src
            '')
          (pkgs.runCommandCC "hog" {
              src = pkgs.writeText "hog.c" ''
                #include <stdlib.h>
                #include <stdio.h>
                #include <unistd.h>
                #include <string.h>

                int main() {
                    while (1) {
                        char *mem = malloc(1024 * 1024);
                        if (!mem) {
                            perror("malloc");
                            break;
                        }
                        memset(mem, 1, 1024 * 1024);
                        printf("1 MB allocated\n");
                        fflush(stdout);
                        sleep(1);
                    }
                    return 0;
                }
              '';
            } ''
              mkdir -p $out/bin
              $CC -o $out/bin/hog $src
            '')
        ];
        initialPassword = "";
      };
      security.sudo.wheelNeedsPassword = false;
      services.getty.autologinUser = "alice";

      system.stateVersion = "24.11";
    };
  };
in
  nixos.vm

Although a single cgroup can enforce multiple resource allocations, we will do so one at a time to simplify.

All cgroup live beneath the special directory /sys/fs/cgroup directory, which is referred to as the root cgroup.

You can inspect your login shells current cgroup by inspecting /proc/self/cgroup

The returned value is what should be appended to the root.

> cat /proc/self/cgroup
0::/user.slice/user-1000.slice/session-5.scope

> file /sys/fs/cgroup/user.slice/user-1000.slice/session-5.scope
/sys/fs/cgroup/user.slice/user-1000.slice/session-5.scope: directory

If you are confused with the slice and scope stuff in the path just ignore that 🤔. That is a systemd concept to help organize cgroups.

Let’s create a simple hierarchy we want for the purpose of learning.

In practice you will probably have these groups created to model the business domain or the various processes you want to group together and not the controllers you want to express.

/sys/fs/cgroup
└── demo/
    ├── cpu-limited/
    ├── memory-limited/
    └── network-limited/

Since “everything is a file” is the mantra of our Linux API, let’s just mkdir the groups.

sudo mkdir /sys/fs/cgroup/demo
sudo chown -R $USER /sys/fs/cgroup/demo
mkdir /sys/fs/cgroup/demo/cpu-limited
mkdir /sys/fs/cgroup/demo/memory-limited
mkdir /sys/fs/cgroup/demo/network-limited

If we look inside a single cgroup, we see a bunch of virtual files.

ls -1 /sys/fs/cgroup/demo | head
cgroup.controllers
cgroup.events
cgroup.freeze
cgroup.kill
cgroup.max.depth
cgroup.max.descendants
cgroup.subtree_control
...
memory.low
memory.max
memory.min
memory.numa_stat
memory.oom.group
memory.peak
...
network-limited
pids.current
pids.events
pids.max
pids.peak

Some of these files help set the value on the various controllers such as memory.max which sets the absolute aggregate maximum memory all processes either attached to this cgroup or any of its descendants can allocate.

Other files, give you live accounting information or events such as memory.current or memory.events.

All the files that begin with cgroup itself, help set up the cgroup and turn on/off the various controllers.

cgroup.controllers

This file will list all the active controllers enabled on this cgroup.

cgroup.subtree_control

This file lists the controllers that are enabled and available to the descendants.

Initially, our cgroup.subtree_control for /sys/fs/cgroup/demo is empty. This means if you looked at any of the child cgroup, i.e. /sys/fs/cgroup/demo/cpu-limited, it will be missing a bunch of files.

> cat /sys/fs/cgroup/demo/cgroup.subtree_control 
# empty
> cat /sys/fs/cgroup/demo/cpu-limited/cgroup.controllers 
# empty

Let’s toggle on various controllers.

> echo "+memory +io +cpu" > /sys/fs/cgroup/demo/cgroup.subtree_control 

> cat /sys/fs/cgroup/demo/cgroup.subtree_control 
cpu io memory

> cat /sys/fs/cgroup/demo/cpu-limited/cgroup.controllers 
cpu io memory

We can change the cgroup for a process by writing its pid to the cgroup.procs file.

> sleep infinity &
1055
> echo 1055 | sudo tee /sys/fs/cgroup/demo/memory-limited/cgroup.procs 
1055
>  ps -o cgroup 1055
CGROUP
0::/demo/memory-limited

Why did you have to use sudo even though before you did chown ? 🤔

When I first started sleep, it was in the same cgroup as my login shell. Processes are only allowed to move cgroups for other processes if they have write permission for a common ancestor between them. The only common ancestor between the two is /sys/fs/cgroup and our user does not have write permission for it.

Why didn’t you write the pid to /sys/fs/cgroup/demo instead of a child group?

There is a “no internal process constraint” which states that a cgroup may either have child cgroups or process but not both (except for the root).

Let’s write a small C program that endlessly eats memory.

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>

int main() {
    while (1) {
        char *mem = malloc(1024 * 1024);
        if (!mem) {
            perror("malloc");
            break;
        }
        memset(mem, 1, 1024 * 1024);
        printf("1 MB allocated\n");
        fflush(stdout);
        sleep(1);
    }
    return 0;
}

😲 Our program has to be sure to memset to 1 rather than 0. I found that either the compiler or the kernel has optimizations for pages that are all 0 and that no new memory was ever actually allocated.

We will restrict processes within our demo/memory-limited group to 5MiB.

> echo "5242880" > /sys/fs/cgroup/demo/memory-limited/memory.max  

> cat /sys/fs/cgroup/demo/memory-limited/memory.max  
5242880

Now let’s start hog in the cgroup. We will use the tool cgexec which takes care of spawning the process in the desired cgroup – this avoids us having to write ourselves to the cgroup.procs file.

> sudo cgexec -g memory:demo/memory-limited hog
1 MB allocated
1 MB allocated
1 MB allocated
1 MB allocated
[  128.648590] Memory cgroup out of memory: Killed process 895 (hog)
total-vm:7716kB, anon-rss:4992kB, file-rss:1024kB,
shmem-rss:0kB, UID:0 pgtables:48kB oom_score_adj:0
Killed

We just applied our first resource constraint 😊.

Let’s do one more interesting example. Let’s restrict a program from running only 10% of the time on the CPU.

This can be really useful if you want to reproduce what the effects of an over-subscribed machine may be like.

Let’s write a simple program that does some busy work and prints out time deltas every 1000000 iterations.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <time.h>

static long long now_ns() {
    struct timespec ts;
    clock_gettime(CLOCK_MONOTONIC, &ts);
    return (long long)ts.tv_sec * 1000000000LL + ts.tv_nsec;
}

int main() {
    long long last = now_ns();
    int count = 0;

    while (1) {
        count++;
        if (count % 1000000 == 0) {
            long long current = now_ns();
            long long delta_ms = (current - last) / 1000000;
            printf("Delta: %lld ms\n", delta_ms);
            fflush(stdout);
            last = current;
        }
    }
    return 0;
}

If we were to run this program normally, we may see the following:

> throttled  | head
Delta: 0 ms
Delta: 1 ms
Delta: 0 ms
Delta: 0 ms
Delta: 0 ms
Delta: 1 ms
Delta: 0 ms
Delta: 1 ms
Delta: 0 ms
Delta: 0 ms

Now let’s apply a CPU constraint saying that within 100ms (100000µs), processes within the cgroup may only use 1ms (1000µs) – 1% CPU allocation.

> echo "1000 100000" > /sys/fs/cgroup/demo/cpu-limited/cpu.max

> cat /sys/fs/cgroup/demo/cpu-limited/cpu.max
1000 100000

Let’s use cgexec again on our throttled program and observe the difference.

> sudo cgexec -g cpu:demo/cpu-limited throttled
Delta: 0 ms
Delta: 5 ms
Delta: 99 ms
Delta: 0 ms
Delta: 99 ms
Delta: 99 ms
Delta: 99 ms
Delta: 100 ms
Delta: 99 ms
Delta: 199 ms
Delta: 0 ms

Nice – we now have a way to easily throttle tasks that may be unreasonably CPU hungry 😈.

Although we applied these constraints to single-processes, the same concept applies to multiple processes as well. The values set are for all descendants of the tree in a particular cgroup.

Control groups are an excellent way to provide an additional layer of isolation for a workload from the rest of the system and also serve as a great knob for performance benchmarking under pathological conditions.

While they seemed daunting at first, the elegance of the “everything is a file” philosophy makes them surprisingly approachable once you start experimenting.

We also benefited from ignoring the complexity that systemd often adds on top — sometimes it’s nice to just work with raw files and understand the fundamentals 🙃.

One improvement I’d love to see: when you hit an invalid condition — like violating the “no internal process” constraint — you’re left with a vague file I/O error (e.g. Device or resource busy). It would be amazing if the kernel could offer more actionable error messages or hints in dmesg 💡.

Bazel’s sandboxing is a powerful way to isolate builds and enforce resource usage via the use of cgroups. One key feature is limiting memory per action via --experimental_sandbox_memory_limit_mb. However, configuring this correctly across machines and CI environments is tricky, and even worse, if Bazel silently fails to enable it, your limits simply don’t apply.

I consider this silent failure to be a bug, especially if any limits have been explicitly expressed, and have opened issue#26062.

I have a few previous posts where I explored how to enable groups for Bazel for the purpose of enforcing memory limits at my $DAYJOB$. After I got to the point of having my own manual validation of the flag working, I wanted to prove that it continues to work and we don’t introduce a regression. 🤔

Normally, we catch regressions with tests. But things get a little more hazy when you’re trying to test the foundational layer that runs all your code.

Nevertheless, turns out we can employ a test! We will run bazel inside of bazel 🤯.

Turns out that the Bazel codebase already runs bazel recursively in test targets and there is even a ruleset, rules_bazel_integration_test, that offers a lot of scaffolding to test multiple Bazel versions.

I always opt for the simplest solution first and decided to write a minimal sh_test that provided our memory limits without relying on @rules_bazel_integration_test which adds scaffolding for multi-version testing, but felt heavyweight for this focused validation 🤓.

Let’s first build our failing binary! We will build a Java program that will endlessly consume memory.

public class EatMemory {
  private static final int ONE_MIB = 1024 * 1024;
  private static final int MAX_MIB = 100;

  public static void main(String[] args) {
    byte[][] blocks = new byte[MAX_MIB][];
    int i;

    for (i = 0; i < MAX_MIB; ++i) {
      blocks[i] = new byte[ONE_MIB];
      // Touch the memory to ensure it's actually allocated
      for (int j = 0; j < ONE_MIB; ++j) {
        blocks[i][j] = (byte) 0xAA;
      }
      System.out.printf("Allocated and touched %d MiB%n", i + 1);
      System.out.flush();
    }

    System.out.printf("Successfully allocated and touched %d MiB. Exiting.%n", MAX_MIB);
  }
}

We will now create a simple sh_test that will run bazel. We will give it the EatMemory.java file and it will setup a very minimal Bazel workspace.

The test will create a simple MODULE.bazel file in a temporary directory and copy over our Java file.

#!/usr/bin/env bash

# Remove the default runfile
# setup stuff for brevity...

mkdir -p "${TEST_TMPDIR}/workspace/java/"

cp "$(rlocation __main__/java/EatMemory.java)" \
   "${TEST_TMPDIR}/workspace/java/EatMemory.java"

cd "${TEST_TMPDIR}/workspace"

cat > java/BUILD.bazel <<'EOF'
# This target is only run within the memory_limit_test.sh script
java_test(
  name = "EatMemory",
  srcs = ["EatMemory.java"],
  tags = [
    "manual",
    "no-cache",
    "no-remote",
  ],
  use_testrunner = False,
)
EOF

cat > MODULE.bazel <<'EOF'
bazel_dep(name = "rules_java", version = "8.11.0")
EOF

# we want to make sure we don't immediately fail if the test fails
# since this is a negative test.
set +e

# this should fail
if bazel --output_user_root="${TEST_TMPDIR}/root" \
      test //java:EatMemory \
      --announce_rc \
      --experimental_sandbox_memory_limit_mb=20 \
      --sandbox_tmpfs_path=/tmp \
      --sandbox_add_mount_pair="${TEST_TMPDIR}/root" \
      --flaky_test_attempts=1 \
      --test_output=streamed
then
  echo "Test unexpectedly succeeded. Are the cgroup limits set correctly?"
  exit 1
fi

The important flag I’m seeking to test is --experimental_sandbox_memory_limit_mb=20 where I set the maximum memory that can be used by actions as 20MiB.

Since I’m running a target that will consume up to 100MiB, this test expects bazel to fail and if it succeeds, the test will fail.

We now do the last finishing touch of writing our BUILD.bazel file with our sh_test. In order to help the test find bazel we add our $PATH to the env_inherit flag. Normally this is not considered best practice as it ruins the hermiticity of the test, but in this case we don’t mind if the test re-runs. 😎

java_binary(
    name = "EatMemory",
    srcs = ["EatMemory.java"],
)

sh_test(
    name = "memory_limit_test",
    srcs = ["memory_limit_test.sh"],
    data = [
        ":EatMemory.java",
        "@bazel_tools//tools/bash/runfiles",
    ],
    env_inherit = ["PATH"],
    tags = [
        "external",
        "no-cache",
        "no-remote",
    ],
    target_compatible_with = [
        "@platforms//os:linux",
    ],
)

We make sure to restrict the test to only the Linux platform, since Windows and MacOS do not have cgroup support.

One final gotcha, is to remember to disable any form of caching 👌 !

We are trying to validate assumptions about the state of a system unbenownst to Bazel and therefore as it is not modeled in Bazel’s action graph, we cannot safely cache the test. Make sure no-cache and no-remote are applied.

We can now rest assured that when we apply experimental_sandbox_memory_limit_mb to our comprehensive test suite, Bazel will continue to respect them.

Bazel’s depset is a powerful construct for managing transitive dependencies efficiently. While it’s commonly used in complex rules and providers, sometimes a simple use case can illuminate its utility.

Ok… but what does that meah? 🤔

This is a companion post to Jay Conrad excellent series on writing rules. The series is excellent and I recommend you read it 🤓.

Consider this simple guide for understanding depset. We will be writing a simple ruleset rules_graphviz.

Graphviz is an open source graph visualization software. Graphs are defined via the DOT language which is a grammar for defining Graphviz nodes, edges, graphs.

For instance, let’s take the simple graph G below.

digraph G {
 "A" -> "B"
 "A" -> "D"
 "B" -> "C"
 "B" -> "D"
}

Would produce the following visualization.

┌───┐     ┌───┐
│ D │ ◀── │ A │
└───┘     └───┘
  ▲         │
  │         │
  │         ▼
  │       ┌───┐
  └────── │ B │
          └───┘
            │
            │
            ▼
          ┌───┐
          │ C │
          └───┘

Our goal would be to write a Bazel rule that let’s us model this graph purely in Bazel targets.

load("@rules_graphviz//:dot.bzl", "node")

node(
  name = "A",
  edges = [
    ":B",
    ":D",
  ],
)

node(
  name = "B",
  edges = [
    ":C",
    ":D",
  ],
)

node(name = "C")

node(name = "D")

We would like a rule that creates a text file (dot file) of the digraph representation of all the nodes reachable from the given target.

That means every node should know it’s dependencies (i.e. edges), and we’d like a way to traverse the whole graph.

💡 We could do this traversal with a standard algorithm (i.e. breadth-first-search) knowing only the direct edges, however, this is where depset shines, as it’s a space and time effecient way of encoding a graph. The depset API contains a to_list() function making it easy to quickly iterate over the whole graph.

First, let’s define our unique provider. Providers are a way for us to attach additional metadata to every target in Bazel that they can carry along with them.

We will need two pieces of information: a fragment of text which are the immediate edges of this target and a depset which is the subgraph of targets it depends on.

GraphvizProviderInfo = provider(
  doc = "A provider for graphviz",
  fields = {
    "fragment": "The edges of this target to it's strict dependencies",
    "deps": "A depset of the dependencies of this target",
  },
)

Let’s create our rule. We make it clear to Bazel that all targets provided to edges must carry with them our provider GraphvizProviderInfo. Failure to add an edge which doesn’t have this provider, will be an evaluation error.

node = rule(
  implementation = _node_impl,
  attrs = {
    "edges": attr.label_list(
      doc = "Edges to other Graphviz nodes.",
      providers = [GraphvizProviderInfo],
    ),
  },
  output_to_genfiles = True,
)

Now the implementation purpose is to construct each node’s fragment (i.e. direct edges) and also collect all fragment’s of each node reachable from the graph when constructing the final DOT graph.

Two key lines are when the rule constructs transitive_deps and transitive_fragments.

transitive_deps

We need to construct and propagate the new depset for the given node. We pass the immediate edges as the direct dependencies and each direct dependencies own depset into the transitive attribute. This will create our graph bottom-up!

transitive_fragments

This is where the rule iterates over all reachable nodes in the graph. We could do a traditional traversal, but the appeal of depset is that it offers a to_list() API that provides the traversal for us – while still giving us all the other space & time efficiencies.

def _node_impl(ctx):
  # Generate the DOT fragment for the current node
  fragment = '"{}"\n'.format(ctx.label)
  fragment += ''.join(
    ['"{}" -> "{}"\n'.format(ctx.label, dep.label) for dep in ctx.attr.edges]
  )

  # Aggregate transitive dependencies using depset
  transitive_deps = depset(
    direct=ctx.attr.edges,
    transitive=[dep[GraphvizProviderInfo].deps for dep in ctx.attr.edges]
  )

  # Concatenate all fragments from transitive dependencies
  transitive_fragments = ''.join(
    [dep[GraphvizProviderInfo].fragment for dep in transitive_deps.to_list()]
  )

  # Assemble the complete DOT content
  dot_content = "digraph G {\n"
  dot_content += fragment
  dot_content += transitive_fragments
  dot_content += "}\n"

  # Declare and write the DOT file
  dot_file = ctx.actions.declare_file(ctx.attr.name + ".dot")
  ctx.actions.write(dot_file, dot_content)

  # Return the providers
  return [
    DefaultInfo(files=depset([dot_file])),
    GraphvizProviderInfo(fragment=fragment, deps=transitive_deps),
  ]

Let’s try our new rule using the targets earlier!

> bazel build //:A
Target //:A up-to-date:
  bazel-bin/A.dot

> cat bazel-bin/A.dot
digraph G {
"@@//:A"
"@@//:A" -> "@@//:B"
"@@//:A" -> "@@//:D"
"@@//:C"
"@@//:D"
"@@//:B"
"@@//:B" -> "@@//:C"
"@@//:B" -> "@@//:D"
}

Huzzah! We built a small declarative graph ruleset that emits DOT files 🙌🏽.

We did so by eleveraging Bazel depset to make the traversal efficient and propagated this information using our own custom provider.

That was not as scary as I thought 🫣.

Update

Some feedback was provided by Peter Lobsinger over the Bazel slack that highlighted best practices from Bazel recommend trying to avoid calling to_list whenever possible.

You can coerce a depset to a flat list using to_list(), but doing so usually results in O(N^2) cost. If at all possible, avoid any flattening of depsets except for debugging purposes. [ref]

Finally, it’s important to not retrieve the contents of the depset unnecessarily in rule implementations. One call to to_list() at the end in a binary rule is fine, since the overall cost is just O(n). It’s when many non-terminal targets try to call to_list() that quadratic behavior occurs. [ref]

We can update the rule to instead bubble up the fragment which includes the transitive edges.

The relevant change avoids calling to_list and instead concatenates prior fragments into the current one.

def _node_impl(ctx):
  # Generate the DOT fragment for the current node
  fragment = '"{}"\n'.format(ctx.label)
  fragment += ''.join(
    ['"{}" -> "{}"\n'.format(ctx.label, dep.label) for dep in ctx.attr.edges]
  )

  fragment += ''.join(
    [dep[GraphvizProviderInfo].fragment for dep in ctx.attr.edges]
  )

  # Assemble the complete DOT content
  dot_content = "digraph G {\n"
  dot_content += fragment
  dot_content += "}\n"

The downside to this approach is that nodes and edges may be duplicated in the resulting file with the current implementation. The DOT language supports duplicates, so the resulting graph is still correct albeit a bit unecessarily larger.

> bazel build //:A
Target //:A up-to-date:
  bazel-bin/A.dot

> cat bazel-bin/A.dot
digraph G {
"@@//:A"
"@@//:A" -> "@@//:B"
"@@//:A" -> "@@//:D"
"@@//:B"
"@@//:B" -> "@@//:C"
"@@//:B" -> "@@//:D"
"@@//:C"
"@@//:D"
"@@//:D"
}

We could handle the duplicates in the fragment each time by stripping them out or create a new rule graph which is the only point at which we do the full traversal and may call to_list. However, I wanted to keep the rule as simple as possible for demonstrative purposes 🙇🏼.

This is a follow up from the previous post on bazel cgroup memory investigation 🕵️.

Turns out that at $DAYJOB$ we were not even using linux-sandbox like we thought we were! 🤦

Our builds were happily printing out processwrapper-sandbox even thought the builds were on Linux.

How come? 🤔

Well it’s not so obvious on why a particular sandbox strategy is not available. Bazel does not make any logs easily available for debug.

Turns out though we can easily run the linux-sandbox itself and get some more diagnostic information.

We will use the linux-sandbox tool to run /bin/true which is what Bazel itself does to validate that the tool is functioning correctly ref.

> $(bazel info install_base)/linux-sandbox /bin/true

src/main/tools/linux-sandbox-pid1.cc:180: "mount": Permission denied

Uh no 😫 – what does that permission denied for “mount” mean ?

Well the linux-sandbox is creating various mounts within a user namespace to setup the sandbox.

Once again, not much logs from the tool itself to use to debug. Turns out that if you run dmesg, we see the culprit.

[Tue May 13 21:50:22 2025] audit: type=1400 audit(1747173023.407:128):
  apparmor="DENIED" operation="capable" class="cap" profile="unprivileged_userns"
  pid=3763 comm="unshare" capability=21  capname="sys_admin"

Looks like AppArmor is specifically denying the mount within the user namespace.

Why?

Looks like a breaking change occurred in Ubuntu 24 where a new AppArmor profile was included that restricted unprivileged user namespaces ref.

Well for now, let’s just disable all AppArmor checks and make them “complaints” 🤫

sudo aa-complain /etc/apparmor.d/*

Now that we know that linux-sandbox will work, let’s setup our cgroups so that they can be used by Bazel.

We will create a root group /example and we will enable the memory controller for it. Additionally, we create a child group /example/child that will own the Bazel process.

The last step is moving our current process into the cgroup so that subsequent bazel invocations start in that cgroup itself.

sudo mkdir /sys/fs/cgroup/example
sudo chown $USER -R /sys/fs/cgroup/example
cat /sys/fs/cgroup/example/cgroup.controllers
echo "+memory" | sudo tee /sys/fs/cgroup/example/cgroup.subtree_control
sudo mkdir /sys/fs/cgroup/example/child
sudo chown $USER -R /sys/fs/cgroup/example/child
echo $$ | sudo tee /sys/fs/cgroup/example/child/cgroup.procs

Now we are ready to try --experimental_cgroup_parent flag for bazel.

According to the Bazel documentation, this flag will make it so that Bazel runs every execution within a cgroup nested within this parent.

While that on it’s own is not very useful, we can combine it with other flags like --experimental_sandbox_memory_limit_mb to enforce maximum memory for tasks.

We could even modify the parent cgroup ourselves which would be inherited by all the child groups. For instance, we could force CPU constraints, like have Bazel only schedule on certain cores. 🤓

We would however like to validate that this all works, so for that we will write a very simple genrule.

The goal of the genrule is to write out the info to a file bazel-bin/cgroup_output.txt that we can use to validate things are as we expect.

genrule(
    name = "check_cgroup",
    outs = ["cgroup_output.txt"],
    cmd = """
        echo "==== /proc/self/cgroup ====" > $@
        cat /proc/self/cgroup >> $@
        echo "" >> $@
        echo "==== Cgroup memory.max for each cgroup in /proc/self/cgroup ====" >> $@
        while IFS= read -r line; do
            IFS=: read -r _ _ cgroup_path <<< "$$line"
            if [ -f "/sys/fs/cgroup$${cgroup_path}/memory.max" ]; then
                echo "$${cgroup_path}: $$(cat /sys/fs/cgroup$${cgroup_path}/memory.max)" >> $@
            else
                echo "$${cgroup_path}: memory.max not available" >> $@
            fi
        done < /proc/self/cgroup
        echo "" >> $@
    """
)

Now let’s run it!

$ bazel --experimental_cgroup_parent=/example/test build \
   //:check_cgroup \
   --experimental_sandbox_memory_limit_mb=20

$ cat bazel-bin/cgroup_output.txt
==== /proc/self/cgroup ====
0::/example/blaze_8239_spawns.slice/sandbox_7.scope

==== Cgroup memory.max for each cgroup in /proc/self/cgroup ====
/example/blaze_8239_spawns.slice/sandbox_7.scope: 20971520

Great! Everything looks like it works.

Our task was correctly placed within /example cgroup and I can even see that the memory.max value or the cgroup was set to 20MiB.

We can now go back to our original demonstrate of eat_memory.py from earlier and avoid having to use systemd-run itself to limit memory but instead rely on bazel cgroup integration. 🔥