The Last Programming Language Might Not Be for Humans Unknown — Sat, 11 Apr 2026 00:00:00 +0000 This morning I was standing at my desk, drinking watered-down instant coffee, doing what I do every morning after triaging the high-alert emails and notifications: thirty minutes of HackerNews. It is a ritual I time-box and never skip. I go to the office every day, and whether I am at that desk or at my home desk, the morning is the same. Coffee, posture, front page.</p> HackerNews remains one of the best ways to keep a finger on the pulse of the Bay Area, of tech, of science, of whatever intellectually stimulating thought surfaced overnight. I follow a handful of curated newsletters too, but I have noticed over the years that HN covers most of their content anyway if you know how to filter high signal from low signal. I could write something about a different link every single day. Most mornings I resist. This morning I did not.</p>A link caught my eye. Vera</a>, a new programming language “designed for machines to write, not humans.” Statically typed, purely functional, compiles to WebAssembly, uses Microsoft’s Z3 solver for contract verification. It has a ferret mascot. I like animal mascots for tech projects. Ferris the crab for Rust, the gopher for Go, the Shisa guardian dog for Zentinel</a>. The ferret is a good choice.</p> But the mascot is not why I stopped scrolling. I stopped because somebody else had arrived at the same conclusion I had reached back in December: that conventional programming languages are not adapted to the capabilities of this new technology, and that something has to change.</p> The Christmas realization</h2> I had been a paying Claude Code subscriber since May 2025, when Anthropic first launched it. The CLI orientation made sense to me immediately, even though the early rate limits and model quality left me wanting more. By December 2025, I had upgraded to the Max subscription, I was on vacation, and Anthropic had made the daily limits generous that month. I was burning through every idea I had accumulated over the years. Some were good. Some were terrible. All of them were finally testable in a way they had not been before, because I could pair-program with a model that kept up. I wrote about that shift more fully in How I Work These Days</a>, the short version being that late 2025 was when the relationship between ambition and execution fundamentally changed for me. The dam broke. Ideas that had been sitting in notebooks for years started becoming real software in days.</p> It was during one of those late-night sessions, deep in a Claude Code conversation about compiler design, that a thought crystallized. I had been building hx</a> and thinking about how AI would change the way people write Haskell, when I realized the question was bigger than Haskell. Agent-assisted software development is approaching a point where the output language itself, the intermediate layer we use to express how information should be processed, is going to change fundamentally.</p> This is not an abstract observation. The history of software is a history of abstraction layers accumulating, each one letting the next generation of programmers ignore what the previous generation had to master.</p> It started with machine code. Raw opcodes, different for every CPU architecture. If you wanted to write software, you memorized instruction sets and wrote them by hand. People published thick reference manuals, and there were engineers who could hold entire instruction set architectures in their heads. Some of them are still around, and some of them still swear by that methodology.</p> Then assemblers gave those opcodes human-readable names. MOV AX, BX</code> instead of 89 D8</code>. You were still writing for a specific architecture, still thinking in registers, but now you could read what you wrote. The first abstraction was not a new capability. It was legibility.</p> Then C arrived and gave us a portable abstraction over hardware. You stopped thinking in registers and started thinking in functions and pointers. C compiled down to architecture-specific assembly, but you did not have to care which architecture. One language, many targets. The reference manuals shifted from instruction sets to language specifications.</p> Then interpreters and virtual machines added another layer. The JVM, Python, Perl. You stopped thinking about memory layout and started thinking about objects, iterators, garbage collection. The abstraction was thicker, the feedback loop was faster, and the audience expanded from hardware engineers to anyone who could write a script.</p> Then IDEs changed how you interacted with the language itself. Syntax highlighting, autocomplete, integrated debuggers, refactoring tools. You stopped holding the entire API surface in your head because the editor held it for you. The language did not change, but the cognitive cost of using it dropped. IntelliSense was not a language feature. It was an abstraction over the programmer’s memory.</p> Then the internet changed how code moved. Open source repositories, package managers, shared libraries. You stopped writing everything from scratch and started composing from parts other people had built. The reference material moved from printed books to web documentation, wikis, tutorials.</p> Then StackOverflow changed how people learned. Instead of reading manuals cover to cover, you searched for the specific problem you had and found someone who had already solved it. The knowledge layer itself became an abstraction. You did not need to understand the full system. You needed to find the right answer and adapt it to your context. StackOverflow never compiled a line of code, but it was an intermediate layer between human confusion and working software, and it was arguably more important to the average developer’s productivity than any language feature shipped in the same decade.</p> And now StackOverflow is receding. Not because the answers got worse. Because the next abstraction layer arrived. AI coding agents do what StackOverflow did, finding known solutions to known problems, but they also do what StackOverflow never could: synthesize novel solutions, hold project context across files, and generate working code from intent descriptions. The pattern is the same as every previous transition. Each layer makes the previous one less essential, not by replacing it but by absorbing it into a higher-level abstraction.</p> Programming languages have always been shaped by who writes them. Assembly was shaped by hardware engineers who thought in registers and opcodes. C was shaped by systems programmers who needed portable abstractions over memory. Python was shaped by people who wanted to get things done without fighting the syntax. COBOL was shaped by business analysts who wanted code that read like English. Every language carries the fingerprints of its intended author. If AI becomes the primary author of code, it follows that the language should adapt to that new author’s strengths and weaknesses. This is not a break from the pattern. It is the pattern, doing what it has always done.</p> I kept coming back to three concrete possibilities. Three ways the intermediate layer could evolve. Not competing visions, exactly. More like three points on a timeline.</p> Make the language explicit enough for machines</h2> Anyone who has spent real time with coding agents has seen the failure mode. You ask an LLM to write a Python function. The output looks plausible. It passes linting. The variable names are reasonable. The structure follows common patterns. Then it fails at runtime because of a subtle implicit behavior the model did not track. A default mutable argument that gets shared across calls. A generator that gets silently exhausted on second iteration. A method that returns None</code> instead of raising an exception because some library author decided that was more “Pythonic.” The model was not wrong about the algorithm. It was wrong about the language’s hidden behaviors.</p> This is the problem Vera is trying to solve, and the first approach I had been contemplating. Take the simplicity and explicitness of Go, push it further, and design a language where every instruction, every method, every design pattern is as unambiguous as possible. No implicit behaviors. No naming ambiguity. No style choices. One canonical way to write everything, so that an LLM does not have to waste inference tokens reasoning about which of seventeen valid approaches to take.</p> The language would need excellent compiler diagnostics. Not just “type mismatch on line 47,” but structured feedback that a model can parse, understand, and act on immediately. Rust and Elixir already do this well for humans. Do it even better, and do it for machines.</p> The pipeline looks like this:</p> +-------+ prompt +-------+ explicit +-----------+ | Human | -------------> | LLM | -------------> | Verifying | +-------+ +-------+ source | Compiler | ^ +-----------+ | | | structured error | | + suggested fix | +---------------------------+ | verified | v [correct program]</code></pre> The key insight is the feedback loop. The compiler does not just reject bad code. It explains what is wrong in terms the model can act on, with a concrete fix suggestion. The model re-generates. The compiler re-checks. You converge on correct code through iteration, and the tightness of that loop depends on how unambiguous the language is and how actionable the errors are.</p> Vera is exactly this idea, executed with conviction. Here is what a function looks like:</p> public fn safe_divide(@Int, @Int -> @Int) requires(@Int.1 != 0) ensures(@Int.result == @Int.0 / @Int.1) effects(pure) { @Int.0 / @Int.1 }</code></pre> No variable names at all. Parameters are referenced by type and positional index using De Bruijn slot notation. @Int.0</code> is the most recently bound integer, @Int.1</code> is the one before that. Every function must declare its preconditions (requires</code>), postconditions (ensures</code>), and side effects (effects</code>). The compiler verifies contracts statically using Z3 where possible and falls back to runtime checks for what it cannot decide at compile time.</p> The design principle is sharp: the model does not need to be right, it needs to be checkable. The language constrains the space of valid programs so tightly that the compiler catches mistakes before execution and explains them in natural language. Division by zero is not a runtime exception. It is a contract violation caught during compilation.</p> Think of it like this. Traditional languages are an open field. You can walk in any direction, and you might end up somewhere useful or you might walk off a cliff. Vera is a guided path with guardrails. You can only go certain directions, and every time you try to step off the path, a sign tells you exactly where to step instead. An LLM on an open field will wander. An LLM on a guided path will converge.</p> The early benchmark results are interesting, if mixed. Kimi K2.5 apparently writes perfect Vera code, scoring 100% on VeraBench and beating its own Python and TypeScript scores. Other models do not fare as well. Claude Opus 4 scores 88% in Vera versus 96% in Python. The Vera team is honest about this variance, which I appreciate. The HackerNews discussion</a> was thin, twelve points and three skeptical comments, which tells you how early this conversation still is. The broader developer community has not engaged with this idea seriously yet. That will change.</p> But there is something that bothers me about optimizing the language for how machines iterate on solutions. Look at the pipeline diagram again. The LLM is still generating step-by-step instructions. It is still describing HOW to do things, just with the ambiguity stripped out and the contracts made explicit. The machine still reasons through the process sequentially. You have reduced the noise in the feedback loop, but you have not changed the nature of the signal. The model is still writing recipes. They are just more precise recipes.</p> What if you stopped writing recipes entirely?</p> Describe what, not how</h2> The second idea starts from a different premise. Instead of making procedural code easier for AI to write correctly, change what you ask the AI to express.</p> Let me make this concrete. Say you need to find the ten most recent server errors in a log. Here is how you would describe that process in a procedural language:</p> errors = [] for entry in log_entries: if entry.status >= 500: errors.append({ "time": entry.timestamp, "path": entry.path, "code": entry.status }) errors.sort(key=lambda e: e["time"], reverse=True) return errors[:10] </code></pre> You are telling the machine: create an empty list. Walk through each entry. Check a condition. If it matches, build a dictionary and append it. Then sort the accumulated list by a key. Then take the first ten elements. Every step is an instruction. The machine has to track the mutable list, the iteration state, the sort, the slice. And if you get any step wrong, the others might still succeed, producing output that looks correct but is subtly broken. A missing reverse=True</code> and you silently get the oldest errors instead of the most recent.</p> Here is the same thing in Haskell:</p> recentErrors :: [LogEntry] -> [ErrorSummary] recentErrors = take 10 . sortBy (flip compare `on` time) . map toSummary . filter isServerError </code></pre> Read it bottom to top: filter server errors, transform each into a summary, sort by time descending, take ten. There is no mutable accumulator. No loop variable. No intermediate state. You are not describing a process. You are describing a relationship between the input and the output. The function says what the result IS, not how to compute it step by step.</p> The type signature at the top, [LogEntry] -> [ErrorSummary]</code>, is a contract the compiler enforces. If toSummary</code> returns the wrong type, if isServerError</code> does not take a LogEntry</code>, if you accidentally compose functions in an order that does not type-check, the compiler rejects the program before it runs. Not with a vague “object has no attribute” at runtime. With a precise type error at compile time that tells you exactly which piece does not fit.</p> This distinction matters enormously for AI. Think about what an LLM actually has to track in each case:</p> Procedural (Vera, Python, Go) Declarative (Haskell) ================================ ================================ - mutable variables and their - input type current state at each step - output type - loop iteration progress - which transformations to compose - conditional branching outcomes - whether types align - order of side effects - implicit language behaviors - names and what they refer to</code></pre> The procedural model asks the AI to simulate execution in its head. The declarative model asks the AI to describe a transformation and let the compiler verify it. One plays to an LLM’s weakness (tracking state across many steps). The other plays to its strength (recognizing and generating patterns that satisfy formal constraints).</p> The pipeline changes fundamentally:</p> +-------+ prompt +-------+ type sigs +-----------+ | Human | -------------> | LLM | -------------> | Compiler | +-------+ +-------+ pure exprs +-----------+ | types align? / \ yes no | | v v [proven correct] [precise type error: "Expected LogEntry, got String at ...]</code></pre> No feedback loop needed in the happy path. If the types align, the program is correct by construction for the properties the type system tracks. The compiler is not iterating with the model. It is checking a proof.</p> This is the approach I bet on. It is the reason this blog exists.</p> Raskell, the name behind this site, is a portmanteau of Rascal (as in raccoon, which is my mascot) and Haskell. The language I have loved for years because of how elegantly it describes things close to mathematical proofs. QEDs, not TODOs. When you write a well-typed Haskell function, you are not just writing code. You are writing a proof that a certain transformation is valid, and the compiler is the proof checker.</p> But loving Haskell and shipping Haskell in production are different experiences, and the gap between them is mostly tooling. The ecosystem is fragmented in a way that has frustrated people for over a decade. cabal</code>, stack</code>, ghcup</code>. Three tools that do overlapping jobs with different opinions about how dependencies should work. If you come from Python, imagine if pip</code>, poetry</code>, and pyenv</code> were all developed independently, with different lockfile formats, different resolver algorithms, and occasional incompatibilities. That was Haskell. Build times were slow. Error messages ranged from helpful to cryptic. The runtime assumed one performance profile fits every use case. If you wanted Haskell for edge functions, for embedded systems, for GPU-accelerated numerics, you spent as much time fighting the toolchain as writing the actual code.</p> The language was right. The surrounding infrastructure was not.</p> So I started building arcanist.sh</a>, taking the same approach that astral.sh</a> brought to Python tooling. When astral.sh released uv</code> and ruff</code>, it showed that you could take a mature ecosystem with entrenched tooling, rebuild the developer experience from scratch in Rust, and make everything dramatically faster and more coherent. I wanted to do the same for Haskell. arcanist.sh houses two projects.</p> hx</a> is a fast, opinionated, next-gen toolchain for Haskell, built in Rust. One tool that replaces the fragmented stack. Managed compiler versions pinned per-project. Deterministic TOML lockfiles with fingerprint verification. 5.6x faster cold builds than cabal. 7.8x faster incremental rebuilds.</p> curl -fsSL https://arcanist.sh/install.sh | sh hx new my-app && cd my-app hx run </code></pre> No ghcup. No stack. No cabal-install. Just hx</code>.</p> BHC</a>, the Basel Haskell Compiler, goes further. It is a clean-slate Haskell compiler written in Rust, not a GHC fork, targeting the Haskell 2026 Platform specification. It uses LLVM for native code generation and offers six runtime profiles that you select at compile time:</p> Profile</th> Designed for</th> Key trait</th></tr></thead> default</td> General applications</td> Lazy evaluation, GC-managed, GHC-compatible semantics</td></tr> server</td> Backend services</td> Structured concurrency, automatic cancellation, observability hooks</td></tr> numeric</td> ML and scientific compute</td> Strict numerics, tensor lowering, SIMD, GPU backends (CUDA/ROCm)</td></tr> edge</td> WASM and CDN workers</td> Minimal footprint, direct WASM emission without LLVM</td></tr> realtime</td> Games, audio, robotics</td> Bounded GC pauses under 1ms, arena allocation</td></tr> embedded</td> Bare metal, microcontrollers</td> No GC at all, static allocation, targets like ARM Cortex-M</td></tr> </tbody></table> The same Haskell source, different runtime contracts depending on what you are building. Your security policy engine compiles with the server profile and gets structured concurrency with tracing. Your tensor pipeline compiles with the numeric profile and gets GPU acceleration. Your edge function compiles to WASM and runs on Cloudflare Workers. Same language. Same type safety. Different performance envelopes.</p> The conviction behind this work is specific. When AI writes the code, the language that survives is not the one optimized for procedural explicitness. It is the one that brings consistency and purity by describing what something is, not how to compute it. AI is extraordinarily good at generating expressions that satisfy formal constraints. And source code that reads like a proof can survive time. It can survive maintainer burnout. It can survive the fact that three years from now, nobody remembers why a function was written the way it was. The types remember. The proof is self-documenting in a way that procedural code never is, because the types encode the intent.</p> Vera and arcanist.sh accept the same premise: the intermediate layer is changing. They disagree about which direction it should change in. Vera optimizes for reducing errors in the generation loop. Valuable. But hx and BHC optimize for making the generated code correct by construction, because the language itself constrains what valid programs look like at a structural level.</p> Skip the language entirely</h2> The third possibility is the one I think about late at night and do not have a concrete project for. Not yet. It is also the one I find most fascinating and most unsettling.</p> What if AI stops writing source code at all?</p> To understand why this is plausible, it helps to look at what source code actually is. It is not the final product. It never was. Source code is a set of instructions that a compiler transforms into machine code. Machine code is what the hardware executes. Source code exists because humans needed an abstraction layer between their intentions and the silicon. We think in concepts like “sort this list” or “reject unauthorized requests.” The CPU thinks in register moves, memory loads, and conditional jumps. Source code bridges that gap.</p> But that bridge was built for human authors. If the author is no longer human, the bridge serves a different purpose. It becomes an audit trail. A way for humans to read and verify what the AI produced. Not an authoring medium, but a transparency layer.</p> I see this playing out in two distinct phases.</p> Phase one: AI targets existing machine code</h3> The first phase is closer than most people think, and it is conceptually straightforward. We already train models on source code. What happens when we also train them extensively on compiled artifacts? On binaries, object files, intermediate representations, the actual output of compilers?</p> Consider what a compiler does. It takes source code and transforms it into machine instructions following well-defined, deterministic rules. There is a mapping between source patterns and output patterns. A for</code> loop in C becomes a specific sequence of compare, branch, and increment instructions on x86. A function call follows a specific calling convention. Memory allocation follows specific system call patterns. These mappings are learnable. They are patterns, and pattern recognition is exactly what LLMs excel at.</p> Today: human intent → prompt → LLM → source code → compiler → x86/ARM → CPU Phase one: human intent → prompt → LLM → x86/ARM directly → CPU (trained on source + compiled artifact pairs)</code></pre> In this phase, the model skips the source code layer and generates machine code directly. Not by “compiling” in the traditional sense. By having learned the patterns well enough to produce valid executables from intent descriptions. The way a fluent translator does not parse grammar rules consciously but produces correct sentences from meaning directly.</p> This sounds radical until you remember that we already trust compilers we do not read the output of. When was the last time you inspected the assembly output of gcc -O3</code> to verify it correctly compiled your C program? You trust the compiler. You test the behavior of the resulting binary. You do not audit the intermediate representation. If an AI can produce binaries that pass the same behavioral tests, the practical difference between “AI-generated machine code” and “compiler-generated machine code” becomes a question of trust calibration, not fundamental possibility.</p> The analogy I keep returning to is aviation. Early pilots flew by hand and understood every mechanical system in the aircraft. Fly-by-wire changed that. The pilot communicates intent (climb, turn, maintain altitude). The computer translates that into control surface movements. The pilot does not manually adjust ailerons and elevators for every gust of wind. They trust the system. They verify outcomes (altitude, heading, airspeed), not intermediate steps. Phase one of post-language programming is fly-by-wire for software.</p> If this sounds speculative, consider what happened this week. Anthropic announced Project Glasswing</a>, a coalition including AWS, Apple, Google, Microsoft, NVIDIA, CrowdStrike, Palo Alto Networks, the Linux Foundation, and others, formed to secure the world’s most critical software using AI. Dario Amodei, Anthropic’s CEO, put it plainly:</p> “AI models have reached a level of coding capability where they can surpass all but the most skilled humans at finding and exploiting software vulnerabilities.”</p> </blockquote> The proof point he offered: “For OpenBSD, we found a bug that’s been present for 27 years.”</p> Think about what that means. OpenBSD is one of the most carefully audited codebases in the world. Decades of security-focused human review by some of the most meticulous systems programmers alive. And an AI model found something that every human reviewer missed for twenty-seven years. If AI can understand existing code deeply enough to find vulnerabilities that humans cannot, it can understand code deeply enough to generate it without human-readable source as an intermediate step. The question is no longer whether AI comprehends code at a structural level. That question was answered this week. The question is what it does with that comprehension next.</p> Phase two: a new kind of machine</h3> The second phase is further out and more speculative. But I think it is where things ultimately go.</p> If AI is generating code for machines to execute and no human needs to read it, there is no reason that code needs to target instruction sets designed for human comprehension. x86 and ARM were designed with the assumption that someone, at least occasionally, would look at the instructions. They have mnemonics. They follow conventions that make disassembly feasible. They are organized into instructions that map, loosely, to operations humans understand.</p> But what if the execution target was designed from scratch for AI-generated code? A virtual machine or runtime that consumes a new kind of bytecode. Not optimized for human readability. Not optimized for hand-authored assembly. Optimized purely for execution density and machine generation.</p> Phase two: human intent → prompt → AI → dense symbolic bytecode → AI-native VM (opaque to humans, | optimized for machine v generation + execution) [result]</code></pre> I keep thinking about information density. Chinese characters encode meaning in individual symbols that carry far more semantic weight than Latin alphabet words. A single character can represent a concept that takes an entire English phrase to express. When a system is designed for readers who can process dense symbols natively, the representation compresses. It becomes more efficient at the cost of being less accessible to readers who were not part of the design audience.</p> AI-native bytecode could follow the same principle. Each instruction could encode complex composite operations that would take dozens of conventional instructions to express. The bytecode would be dense in ways that make current machine code look verbose. Entirely opaque when decompiled or analyzed. Not obfuscated on purpose. Just natively incomprehensible to human cognition, the same way a trained neural network’s weights are incomprehensible even though they encode real, functional knowledge.</p> The virtual machine running this bytecode would itself be a different kind of system. Not a stack machine or a register machine in the traditional sense. Possibly something closer to a dataflow engine, where the bytecode describes transformation graphs rather than sequential instructions. Think of it as the difference between giving someone turn-by-turn driving directions (go north, turn left, continue for two miles) versus handing them a map with the destination marked. The bytecode is the map. The VM figures out the route.</p> I want to be honest about where we are. We are not at phase one. Not in April 2026. Current models still need the intermediate layer. They produce better code when they can reason through it step by step. They benefit from type systems and contracts and explicit error messages. The first and second approaches are not just viable, they are necessary right now.</p> But the trajectory is visible. Models are getting better at generating correct programs with every generation. Formal verification is becoming more practical. Hardware is getting cheaper. The gap between “prompt that describes intent” and “correct executable output” is shrinking. Phase one will arrive when models trained on enough source-plus-binary pairs can reliably produce correct executables. Phase two will arrive when someone asks: if the model is already generating the binary, why are we targeting an instruction set that was designed for a species that is no longer doing the writing?</p> At some point the intermediate layer becomes optional. And optional things, given enough time, become vestigial.</p> What this means</h2> I do not think these three approaches are in competition. They are three points on a timeline, and the timeline is the story of the intermediate layer contracting.</p> Near term Medium term Long term (now) (2-5 years) (5-15 years) Explicit Declarative Post-language languages languages (AI-native targets) (Vera) (Haskell + BHC) Reduce noise --> Change the signal --> Remove the layer in the loop entirely entirely HOW, but WHAT, verified Intent to unambiguous by types execution</code></pre> In the near term, explicit languages like Vera make AI-generated code more reliable by constraining the generation space and providing machine-readable diagnostics. This is useful today. If you are building an AI coding pipeline right now and need to ship next quarter, this approach works.</p> In the medium term, declarative languages like Haskell, especially with modern tooling and modern runtimes, make AI-generated code correct by construction. The type system does the heavy verification work at a fundamental level. The code that survives is the code that describes invariants, not procedures. This is the era I am building for with arcanist.sh.</p> In the long term, the language disappears. First into existing machine code generated directly by AI. Then into new execution formats designed for AI generation from the ground up. The intermediate layer that has defined software engineering for seventy years becomes an implementation detail.</p> That last possibility makes some people uncomfortable. It made me uncomfortable when I first thought about it in December. It means that the craft of programming as we know it, the fluency in syntax, the mastery of idioms, the instinct for an elegant implementation, becomes less like a core skill and more like knowing how to operate a manual lathe. Valuable in the right context. Still needed for the hard cases. But no longer the primary way most software gets built.</p> This has happened before. There was a time when every programmer understood assembly. Then C abstracted it away, and most programmers stopped reading machine code. Then Python and JavaScript abstracted C away, and most programmers stopped thinking about memory management. Each time, the previous layer did not disappear. It became the domain of specialists who maintained the infrastructure everyone else stood on. The same thing will happen to source code. It will not vanish. It will specialize.</p> I still write code every day. I still think in types. I am still building hx and BHC because I believe the medium-term future is both real and long, and Haskell’s strengths are exactly what that future demands. Pure functions, strong types, provable correctness. These are not luxuries in a world where AI writes the implementation. They are the load-bearing structure. But I do it with one eye on the horizon, knowing that the intermediate layer I am investing in is exactly that. Intermediate.</p> The person who built Vera saw the same thing I saw, probably around the same time. They chose the first approach. I chose the second. Somebody, eventually, will build the third. And then we will all need to figure out what we mean by “programming” when nobody writes programs anymore.</p> I am already curious about the answer.</p> Notes from RSAC 2026 Unknown — Thu, 09 Apr 2026 00:00:00 +0000 I am writing this about two weeks after RSAC 2026 closed. In between, my coworkers and I drove down the West Coast, through Big Sur to LA, then out to Las Vegas for Easter weekend. That was deliberate. Not the route, exactly, but the space. The conference gave me a lot to process, and I have learned over the years that I process better when I am moving, when I am not sitting at a desk trying to force conclusions out of raw impressions.</p> So here is what I took away. Not a session-by-session recap. Not a vendor roundup. Just the things that are still with me now that the noise has faded and the Pacific Coast Highway is behind me.</p> The talk</h2> Milan Duric and I presented “Self-Learning WAF: Using Generative AI to Tame ModSecurity False Positives” on Wednesday morning, March 25, in Moscone West 3020. We had an 8:30 AM slot, which is either a curse or a blessing depending on your audience. It turned out to be a blessing. The room was full of people who had chosen to be there at that hour, which means they cared about the topic, and the energy reflected that.</p> The talk went flawlessly. If you have ever presented at a conference of that scale, you know that “flawless” is not something you take for granted. There is always the moment before you start where you wonder if the demo will work, if the projector will behave, if your timing will hold. All of it held. Milan and I had rehearsed enough that the talk felt natural rather than performed, which is the line you want to hit.</p> I will write a separate post about the content of the talk itself, what we built, what we learned, how the audience responded to specific ideas. This post is about everything else.</p> Third time, first time</h2> This was my third RSA Conference. I attended in 2024, 2025, and now 2026. But it was my first time as a speaker, and that changed the experience in ways I did not fully expect.</p> When you attend as a participant, you are a consumer of the conference. You pick sessions, you walk the expo floor, you absorb. When you are a speaker, even for just one session, you become part of the fabric. People approach you after the talk. They reference something you said in a hallway conversation two days later. You are on the other side of the dynamic, and it gives you a different relationship with the event.</p> I have grown to genuinely appreciate the sheer volume and quality of the whole thing. RSAC is not one conference. It is several conferences layered on top of each other. There are deeply technical sessions where people walk you through real implementations, real code, real incident response timelines. There are strategic talks where CISOs and policy architects work through the organizational and regulatory implications of what is changing. And then there are the keynotes, the big voices, people who have shaped the field for decades, sharing what they see on the horizon.</p> Because it happens in San Francisco, in the heart of the Bay Area, the reach is different from any other security event. You are not just at a conference. You are at the geographic center of the industry that is driving the transformation everyone is trying to understand. The density of talent, capital, and ambition in that city during RSAC week is difficult to describe if you have not experienced it. The only comparable events I can think of are BlackHat and DefCon, but even those have a different energy. RSAC pulls in a wider spectrum of the industry, from the deeply technical to the deeply strategic, from startup founders to government officials, and puts them all in the same building for a week. That range is what makes it valuable.</p> The year of agents</h2> Ever since I started attending in 2024, AI has been a substantial part of the conference. That makes sense. The developments in AI over the past few years have had a prominent cybersecurity dimension from the start, and the industry has been working through what that means, both as a threat to defend against and as a capability to harness.</p> But this year felt qualitatively different from the previous two. In 2024 and 2025, the AI conversation was broad and somewhat exploratory. What can large language models do for security? How do we detect AI-generated phishing? What does the threat landscape look like when attackers have access to the same models we do? Important questions, but still in the “what is possible” phase.</p> 2026 was past that. The conversation had narrowed and deepened. It was specifically about agents. Not AI in general, not language models as a capability, but autonomous agents as a new category of infrastructure. Enterprise-grade agentic systems. Agentic orchestration patterns. Agent-native architectures. The shift from “can we use AI?” to “how do we architect our systems around autonomous agents that are already here?” was palpable in almost every session I attended.</p> The concept that kept coming up was NHI: non-human identities. This term has existed in the identity and access management world for a while, but at RSAC 2026 it had taken on a new meaning. The old NHI conversation was about service accounts, API keys, machine certificates. The new NHI conversation is about LLM inference backends that operate as something fundamentally different from traditional automated systems. These are entities that do not just execute a fixed pipeline. They reason, they make judgment calls, they interact with systems and data in ways that look more like what a human analyst does than what a cron job does. But they operate at machine speed, they do not sleep, and they do not have the contextual judgment or accountability that comes with a human in the seat.</p> The trust problem this creates is real, and it is not just a theoretical concern. Human employees were already risk factors before AI entered the picture. Insider threats, social engineering, credential compromise, accidental misconfiguration. These are well-understood attack surfaces. Now add entities that move faster than any human, that can touch more systems in a minute than a human employee touches in a day, and that are harder to audit because their reasoning process is opaque. The attack surface did not just grow. It changed shape in ways that existing security architectures were not designed for.</p> The human-in-the-loop debate</h2> This was the most interesting tension I observed across the conference, and I think it is going to be one of the defining questions for cybersecurity in the next few years.</p> A small number of talks presented approaches that kept a human firmly in the loop. AI assists, human decides. AI flags, human acts. AI generates recommendations, human approves or rejects. These were careful, measured presentations, and some of them were good. The argument is intuitive and appeals to anyone who has been burned by automation gone wrong: keep a human in the critical path because humans have judgment that machines do not.</p> But the majority of the conference had reached a different consensus, and it was stated with increasing confidence as the week went on: the human in the loop is a bottleneck. Not philosophically. Operationally. In terms of the speed at which threats materialize and the speed at which defenses need to respond.</p> The argument is straightforward once you lay it out. Adversaries and threat actors are already leveraging AI to accelerate their operations. They are scanning for vulnerabilities at machine speed. They are generating novel attack variations faster than any human analyst can write detection rules. They are using AI to identify and exploit zero-day vulnerabilities in timeframes that make traditional patch cycles look like geological processes. If your defensive response depends on a human reading an alert, understanding the context, making a judgment call, and clicking a button before a countermeasure activates, you have introduced a rate limiter into your defense that your attacker does not have. You are playing at human speed against an adversary operating at machine speed.</p> I agree with this assessment, and I want to be precise about what I mean by that. I do not think humans are irrelevant to security. They are not. Human judgment, human understanding of organizational context, human ability to reason about novel situations, these remain essential. But I think the role of the human needs to shift fundamentally. The human should be a supervisor, not a gatekeeper. The human should set policy, define constraints, establish acceptable parameters, review outcomes, and intervene when something goes wrong. But the human should not be the bottleneck whose reaction time determines how fast sophisticated defense measures can respond to a threat that is moving at inference speed.</p> This is an engineering problem, not a philosophical one. How do you design agent-native architectures where the human is still in control, still has full visibility, still sets the rules, but is not the limiting factor in the response loop? That is the challenge of 2026. I did not hear anyone at the conference claim to have fully solved it. But I heard a lot of people working on it seriously, and the framing had matured beyond the naive “just automate everything” takes that dominated the early AI-security conversation.</p> This is also, incidentally, part of why I built Zentinel</a> the way I did. A reverse proxy that sits at the edge, where policy enforcement happens at wire speed, is exactly the kind of system that needs to operate autonomously within human-defined constraints. The agent architecture in Zentinel, where security logic runs in isolated processes with bounded resources and explicit failure modes, is my answer to the question of how you let autonomous systems make real-time decisions while keeping the human in the position of supervisor rather than bottleneck. I wrote about this in more detail in What Zentinel Is Really Optimizing For</a>.</p> The four factors</h2> I attended a panel with four former NSA directors and US Cyber Command commanders. Both positions are held by the same person at any given time, so these were people who had sat at the intersection of signals intelligence and military cyber operations at the highest level the United States has. Regardless of how you feel about the NSA or US foreign policy, the caliber of strategic thinking in that room was extraordinary.</p> Paul Nakasone said something that has stayed with me since. He laid out what he considers the four most important factors when assessing the strategic potentiality of a nation state in this era. Not military strength. Not GDP. Four specific things: chips, data, talent, and energy.</p> Chips, meaning silicon, meaning raw compute power. How many advanced GPUs can you deploy? How advanced are they relative to the frontier? And critically: can you manufacture them domestically, or are you dependent on someone else’s fabrication capacity? Right now, the entire world depends on TSMC in Taiwan for leading-edge chip fabrication, and the geopolitical implications of that single point of dependency are staggering.</p> Data, meaning access to the raw material that AI systems learn from. Who has it, how much of it, how diverse is it, and under what legal and political constraints can it be used for training and inference?</p> Talent, meaning the human capital that knows how to build, train, deploy, secure, and govern these systems. Where that talent lives, where it wants to live, and what it takes to attract and retain it. This is not just about researchers at frontier labs. It is about the entire pipeline: the engineers who build the infrastructure, the operators who keep it running, the security professionals who defend it, the policy people who regulate it.</p> Energy, meaning access to cheap, abundant, reliable power. Because the compute demands of frontier AI are measured in gigawatts now, not megawatts. A single frontier training run can consume more electricity than a small city. The question of whether you can physically power your AI ambitions is no longer abstract.</p> I could not stop thinking about AI 2027</a> while listening to Nakasone. The scenario work by Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, and Romeo Dean, which I first wrote about in How I Work These Days</a>, makes strikingly similar assessments from a technology trajectory perspective rather than a national security one. Their scenario tracks the distribution of global AI compute (the US holding roughly 70% of frontier capacity through its companies, China around 12%), the geopolitical competition for chip manufacturing, the energy infrastructure required to sustain frontier operations (their projection of global AI datacenter spending reaching the trillion-dollar range by 2026 no longer reads like speculation), and the talent concentration in a handful of US-based labs.</p> What makes AI 2027 feel prophetic, and I do not use that word casually, is that its core thesis keeps holding up month after month. The idea that automating AI research itself creates a self-reinforcing feedback loop, that the cycle between capability and capability-building is compressing, that the timeline for transformative change is shorter than most institutional planning horizons assume. The specific dates and milestones may shift. The authors themselves have revised some timelines. But the directional assessment, the shape of the curve, still looks right to me as of April 2026. The scenario’s detailed treatment of compute distribution, espionage risks, and the escalatory dynamics between nation states competing for AI dominance maps remarkably well onto what I heard discussed in more guarded terms on the RSAC floor.</p> Listening to Nakasone lay out those four factors, I kept thinking about Europe. And about Switzerland specifically, since that is where I live and work. Europe is behind on all four. The continent does not manufacture frontier chips. It does not host the leading AI labs. Its regulatory environment, while well-intentioned, has optimized more for constraint than for capability. Its energy infrastructure is in the middle of a complex transition. And its talent pipeline, while strong in research, struggles to retain builders who can turn research into deployed systems at scale, because many of them leave for the Bay Area, London, Singapore, the Gulf states, or other places where the ecosystem is more supportive of what they want to build.</p> This is part of why I co-founded Die Zukunft</a>, a new Swiss political party focused on structural transformation. The name means “The Future” in German. The party exists because I believe the political infrastructure in Switzerland, and in most European countries, is not designed to respond to the kind of structural shift that Nakasone was describing. The decisions being made right now about compute sovereignty, energy policy, talent retention, and regulatory frameworks will determine whether Europe is a participant in the next decade or a consumer of other people’s technology. Die Zukunft’s platform addresses these questions directly: digital sovereignty defined in infrastructure terms, faster permitting for critical energy and compute projects, open standards as a hard requirement for government systems, and immigration policy designed to attract the talent that builds these systems. It is not a technology party in the narrow sense. It is a party built on the recognition that the structural transformation AI is driving is too consequential to be left to the current pace of European political response.</p> And this is also why I built Archipelag</a>. If Europe wants digital sovereignty, it needs sovereign compute infrastructure. Not just policy positions about data residency, but actual physical capacity to run AI workloads within European jurisdictions, at competitive cost, without depending on American hyperscalers. Archipelag is a decentralized AI compute network that routes inference jobs to community-operated nodes with jurisdiction-aware routing baked into the infrastructure layer. It is designed so that a European company can run AI workloads with cryptographic guarantees about where their data is processed, using idle GPU capacity that already exists across the continent. It is my direct answer to the infrastructure gap that Nakasone’s four factors expose so clearly for Europe.</p> The enterprise problem</h2> This connects to something broader that I have been thinking about since well before RSAC, but that the conference brought into sharper focus.</p> I work for a large Swiss financial institution. My perspective is shaped by what I see inside that kind of organization every day. But I believe the dynamic applies to enterprises of all sizes and in all sectors, even if the scale and specifics differ.</p> The biggest threat I see right now is not a specific vulnerability, not a particular attack vector, not a novel exploit technique. It is inertia. It is the widening gap between what is happening at the frontier of AI capability and what most organizations are actually doing about it. Too many companies still think they can outsource their way through this transition. Buy an AI-powered security product from a vendor. Subscribe to a managed detection service that mentions “AI” somewhere in its marketing materials. Check the compliance box and move on to the next quarter.</p> I do not think that is going to work. Not because those vendors are bad. Some of them are genuinely good at what they do. But because the organizations that will remain competitive and secure over the next few years are the ones that build internal AI capability, not just consume external AI services. That means investing in your own GPU compute. That means building the internal expertise to deploy, fine-tune, and operate models on your own infrastructure. That means treating AI as a core organizational competency, not a procurement line item.</p> This is not a popular opinion in many boardrooms. It is expensive. It is hard. It requires talent that is difficult to hire and even harder to retain when they can work at a frontier lab or a well-funded startup instead. But the alternative, waiting and relying on service providers to package AI innovation for you at their pace and under their terms, means you are always operating one step behind. You are consuming someone else’s capability with someone else’s priorities, under someone else’s constraints. In a landscape that is moving as fast as this one, that delay compounds.</p> The companies that understand this and invest now are going to have a structural advantage that widens over time. The ones that wait are going to find themselves trying to close a gap that gets larger with every quarter of inaction. I saw enough at RSAC to believe that some organizations have internalized this. And I saw enough to believe that many more have not.</p> The vendor floor</h2> I want to be fair about this, because I know how the previous section sounds, and I do not want to come across as someone who dismisses the entire vendor ecosystem. I am not easily swayed by big tech vendors and their keynote promises, and I think anyone who works in security should maintain a healthy skepticism toward product demos and polished presentations. That is just professional hygiene.</p> That said, I genuinely enjoyed many of the keynotes this year. Some of these companies have real long-term vision. They see the shape of what is coming, and their best presenters can communicate that vision with clarity and conviction. I respect that, even when I disagree with their specific approach or their business model.</p> But enjoying a keynote and trusting that buying a product will translate into sustainable cybersecurity for your organization are very different things. Actual security is hands-on work. It is understanding your own systems, your own architecture, your own threat model, your own failure modes. It is the boring, unglamorous work of knowing what runs where, what talks to what, what happens when something fails, and what your actual attack surface looks like on a Tuesday afternoon. That work cannot be fully offloaded to a vendor. It cannot be outsourced to a dashboard, no matter how sophisticated the analytics behind it.</p> The vendors that impressed me most this year were the ones that acknowledged this honestly. The ones that positioned their tools as force multipliers for competent teams rather than replacements for the need to have competent teams in the first place. That distinction matters, and the vendors who understand it tend to build better products because they are designing for operators, not for procurement committees.</p> Closing night</h2> The last session of the conference was Hugh Jackman in conversation with Hugh Thompson. I was not sure what to expect from a Hollywood actor closing out a cybersecurity conference, and I suspect a lot of people in the audience had the same reservation going in. But it worked. Jackman is funny, self-aware, and surprisingly thoughtful about creativity, discipline, and the craft of doing hard things well. He talked about preparation, about the difference between performing and connecting, about the years of work that go into making something look effortless.</p> At one point he taught the audience that if you say “raise up lights” in an American accent, you are saying “razor blades” in Australian. The room loved it. It was one of those moments where several thousand cybersecurity professionals all became delighted seven-year-olds for about ten seconds, and it was a good reminder that conferences are also about shared human moments, not just information transfer.</p> It was the right way to end a dense week. Light enough to let people exhale after five days of intense content, but substantive enough in its own way that it did not feel like filler.</p> The road after</h2> After the conference closed, a few of us stayed on. We rented a car and drove south from San Francisco, down Highway 1 through Big Sur. If you have not done that drive, I do not know how to describe it adequately except to say that it recalibrates your sense of scale. The Pacific is very large and very indifferent, and spending a few hours winding along cliff-edge roads with that water stretching out to the horizon below you is a useful counterweight to a week of thinking about the future of everything.</p> We spent time in LA. Santa Monica and Venice Beach, walking the boardwalk, eating food that was too expensive and not caring. The kind of aimless, unstructured time that my brain needed after five days of absorbing information at high density. I find that the most useful thinking often happens when you are not trying to think. When you are just watching the ocean or walking on a beach and letting your subconscious do whatever it does with the raw material you fed it.</p> Then Las Vegas for Easter weekend. Spring break crowds, desert heat starting to build, the particular surreality of the Strip. It was not productive time in any conventional sense, and it was not meant to be. It was decompression. Space for the conference to settle from a collection of impressions into something more like understanding.</p> What stays with me</h2> Two weeks out, here is what I think is different.</p> I went to RSAC 2026 with a set of convictions about where things are heading. Agents are going to be the primary operating model for security infrastructure. Human-in-the-loop is going to shift to human-as-supervisor. Organizations that do not build their own AI infrastructure are going to fall behind structurally. Europe needs to wake up to the compute sovereignty problem before it becomes irreversible. These were things I already believed before I got on the plane to San Francisco.</p> What the conference did was sharpen them. Hearing Nakasone frame national potentiality in terms of chips, data, talent, and energy gave me a cleaner lens for thinking about the geostrategic dimension. Seeing the breadth and depth of the agentic conversation on the conference floor confirmed that this is not a niche position or an edge case anymore. It is the emerging consensus of the industry. And presenting our own work, standing in front of a room and showing what we actually built, made it more real in a way that writing code in a terminal at midnight does not.</p> Conferences like RSAC have always had this effect on me. They compress a year’s worth of signals into a week, and then you spend the following weeks unpacking what you heard and figuring out what it means for what you are building. After RSAC 2024, I started thinking seriously about edge security architectures, which eventually became Zentinel. After RSAC 2025, the urgency around AI-native infrastructure solidified into the work that became Archipelag. This year, I expect the sharpened understanding of agentic systems and the geostrategic landscape to feed directly into what I build next.</p> I also came away with a renewed sense of urgency about the gap between what the frontier looks like and what most organizations are doing about it. That gap, between the leading edge and the institutional mean, is the real risk. Not any single threat actor, not any specific vulnerability class. The systemic inability of large institutions to move at the pace that the situation demands. That is what keeps me up at night, and that is what I am trying to address in my own work, whether it is building infrastructure, writing about it, or working on the political dimension through Die Zukunft.</p> I am going to write a separate post about the talk itself, about what Milan and I built with self-learning WAFs and what we learned along the way. That is coming soon. For now, these are the notes I wanted to capture while the impressions are still sharp enough to be useful.</p> I am already looking forward to RSAC 2027.</p> References and further reading</h2> RSAC 2026</a> - The conference itself</li> AI 2027</a> - Scenario work by Kokotajlo, Alexander, Larsen, Lifland, and Dean</li> How I Work These Days</a> - Where I first wrote about AI 2027 and the shift in how I build</li> What Zentinel Is Really Optimizing For</a> - The design philosophy behind Zentinel and why agent isolation matters</li> Zentinel</a> - The security-first reverse proxy I built on Pingora</li> Archipelag</a> - Decentralized, sovereignty-first AI compute network</li> Die Zukunft</a> - Swiss political party for structural transformation</li> </ul> What Zentinel Is Really Optimizing For Unknown — Sun, 22 Mar 2026 00:00:00 +0000 The clearest way I can describe the motivation behind Zentinel is this: I got tired of not trusting the thing that stood between my users and the internet.</p> Not because the proxies I ran were bad. They were not. I have genuine respect for the engineering in Nginx, HAProxy, Envoy. I learned a lot from operating them, and I mean that sincerely. But over years of running these systems in production, a pattern kept repeating, and it was always some version of the same story: the proxy did something I could not predict from reading its configuration.</p> A WAF module gets slow under load, and because it runs inside the proxy process, the entire data path backs up. A retry storm starts because the default retry policy is implicit rather than explicit. A configuration reload takes effect partially because there is no atomic swap. An unbounded queue grows until memory runs out and the OOM killer takes the proxy down along with everything else on the box.</p> None of these are exotic. If you have operated proxies at any real scale, you have seen most of them. And every time, the root cause pointed at the same structural issue: the proxy was optimized for something other than what I actually needed from it.</p> That is not a criticism. It is a statement about time.</p> Every proxy was built for its era</h2> HAProxy was born in 2000. Willy Tarreau built it to solve a specific, urgent problem: distributing TCP connections across a pool of backend servers. The internet was scaling fast. Sites needed load balancing. HAProxy did that one job with extraordinary precision, and twenty-five years later it still does. It is one of the most reliable pieces of infrastructure software ever written. But it was built as a load balancer, and when you need it to do security enforcement, you are extending a load balancer.</p> Nginx arrived in 2004. Igor Sysoev was solving the C10K problem: how do you handle 10,000 concurrent connections without the process-per-connection model falling over? Nginx was built as a web server. An event-driven architecture that served static files and handled connections with remarkable efficiency. The reverse proxy capability came later, almost as a side effect of how well it handled connections, and eventually became one of its most important use cases. But the core was always a web server. The assumptions about configuration, about reload behavior, about how modules interact with the request path, those assumptions come from web serving.</p> Varnish showed up in 2006. Poul-Henning Kamp built it because dynamic web pages were slow and caching was the answer. Varnish sat in front of your web server, cached responses in memory, and served them fast. That was the whole job. A caching proxy, and a beautiful one.</p> Envoy was born at Lyft around 2016. Microservices had created a new problem: how do you route, observe, and control traffic between hundreds of internal services that come and go? The service mesh was the answer, and Envoy was the data plane. It brought observability, retries, circuit breaking, and policy enforcement to a world where the network topology was no longer something you could draw on a whiteboard and expect to remain accurate for more than a week.</p> Traefik arrived in the same era, optimized for automatic service discovery in container environments. Services appear and disappear. The proxy figures out the routing on its own.</p> Every single one of these was the right tool at the right time. They solved the problem that mattered most when they were built, and they solved it well. I used most of them. I admired most of them. I am not here to argue that any of them were wrong.</p> But I am here to say that their time shaped what they became, and so did the economics of how software was built.</p> The generality trap</h2> Here is something I noticed over years of operating these tools: they all converge.</p> Nginx adds load balancing. HAProxy adds HTTP/2 and Lua scripting. Envoy adds caching, WAF capabilities, ext_proc for external processing. Traefik adds middleware chains. Every proxy, over time, becomes a Swiss army knife.</p> This is not a design failure. It is an economic inevitability.</p> Building a production-grade reverse proxy takes a team, sometimes a large one, working over many years. Nginx took Igor years before it was production-ready. Envoy is maintained by hundreds of contributors across multiple organizations. HAProxy has been continuously refined for a quarter of a century by one of the most skilled systems programmers alive.</p> When the cost of building software is that high, you need the result to serve a broad market. You cannot afford to optimize for one narrow concern. You need your proxy to be useful to web servers and API gateways and service meshes and CDN edges and everything in between. The economic pressure pushes relentlessly toward generality. Toward one more feature. Toward covering one more use case. Toward becoming the tool that everyone can use, even if nobody uses it for exactly the thing they wish it was designed for.</p> This creates a particular kind of friction. You adopt a proxy for its core strength, and then you spend years working around its assumptions in every other area. You chose Nginx because it handles connections well, but now you are fighting its reload model and its embedded Lua modules that share a fate with the worker process. You chose Envoy because it observes service traffic brilliantly, but now you are wrestling with an xDS configuration surface that could fill a textbook, and a C++ codebase where memory safety is a matter of programmer discipline rather than compiler guarantee.</p> I lived in that friction for a long time. I knew what I wanted. I could describe it in detail to anyone who would listen. But wanting a purpose-built proxy and building one are different things when building one requires a team and years of sustained effort.</p> What I kept wishing for</h2> After enough 3 AM incidents and enough post-mortems, the shape of what I was looking for became concrete enough to write down.</p> I wanted a reverse proxy where the operator can reason about what will happen under any condition. Including conditions they did not anticipate. That is the whole thesis. Everything else follows from it.</p> When you are on call, “reason about what will happen” is not philosophical. It means concrete things.</p> It means every queue has a maximum depth. Every timeout is explicit and declared. Every connection pool has a ceiling. No unbounded allocations anywhere. If you set a body size limit of 10 MB, that is a hard limit, not a suggestion. If a security agent’s concurrency is capped at 100, the 101st request gets the configured failure mode, not a silent queue that grows until the box dies.</p> It means every route declares what happens when a security agent is unreachable. Not a global toggle. Per route, per agent. Your API fails closed when the WAF is down (deny everything, because your API handles sensitive data and you do not want it exposed without inspection). Your marketing site fails open (allow traffic, log the gap, because a few minutes of unfiltered marketing pages is better than a full outage). You decide. You write it down. The system enforces it.</p> agents { agent "waf" { transport { unix-socket "/var/run/zentinel-waf.sock" } events "request-headers" "request-body" timeout-ms 50 max-concurrent-calls 100 failure-mode "closed" circuit-breaker { failure-threshold 5 success-threshold 3 timeout-seconds 30 } } } routes { route "api" { priority 100 matches { path-prefix "/api/" } upstream "backend" filters "waf" failure-mode "closed" } route "marketing" { priority 50 matches { path-prefix "/" } upstream "static-backend" filters "waf" failure-mode "open" } } </code></pre> It means every security decision gets a trace ID and ends up in structured logs. When someone asks “why was this request blocked at 3:47 AM?”, you can answer with a correlation ID and a full trace: which agent decided, which rule matched, how long it took. Not “the WAF blocked it, probably.” The actual chain of events.</p> It means configuration reloads are atomic. You send SIGHUP, the new configuration is parsed, validated, and swapped in. In-flight requests finish on the old config. New requests pick up the new one. No window where half the routes are on the old version and half on the new.</p> I could describe all of this clearly. I had been able to for years. Describing what you want and having the means to build it are different things.</p> The insight about isolation</h2> The single biggest realization I had was about failure domains.</p> In every proxy I had operated, extension logic ran inside the proxy process. Nginx has embedded Lua via OpenResty. HAProxy has SPOE and Lua. Envoy has Wasm filters and ext_proc. They all share the same structural problem: the extension and the proxy share a fate.</p> If your Lua WAF script enters an infinite loop in nginx, the nginx worker is stuck. If your Wasm filter in Envoy allocates too much memory, the Envoy process pays for it. A slow SPOE agent in HAProxy backs pressure into the proxy’s request handling. I have been on the receiving end of all three patterns, and they all end the same way: you are awake at 3 AM trying to figure out why your entire proxy fleet is degraded because one security module is having a bad day.</p> This is the classic shared-fate problem. When everything runs in one process, everything fails together. A slow WAF does not just slow down WAF-protected routes. It consumes worker resources that affect all routes. A memory leak in an auth filter does not just take down auth. It takes down the process.</p> The answer I kept coming back to was process isolation. Not as a compromise or a workaround, but as the foundational design principle. Security and policy logic should live in separate processes, each with its own memory, its own concurrency limits, and its own circuit breaker.</p> ┌─────────────────────┐ │ Proxy Core │ │ (Rust / Pingora) │ └──────────┬──────────┘ │ ┌─────────────┼─────────────┐ │ │ │ ┌────────▼───────┐ ┌──▼──────────┐ ┌▼───────────────┐ │ WAF Agent │ │ Auth Agent │ │ Custom Agent │ │ semaphore: 100│ │ semaphore:50│ │ semaphore: 25 │ │ timeout: 50ms │ │ timeout:30ms│ │ timeout: 200ms │ │ fail: closed │ │ fail: closed│ │ fail: open │ └────────────────┘ └─────────────┘ └────────────────┘ Each agent: own process, own memory, own circuit breaker. Slow WAF ≠ slow auth. Crashed agent ≠ crashed proxy.</code></pre> If the WAF agent gets slow, the WAF agent’s semaphore fills up. The auth agent keeps running on its own semaphore. The proxy core keeps routing. Nobody shares a failure domain unless you explicitly configure them to.</p> This is not just about crash isolation, though that matters. The deeper point is queue isolation. In a shared-process model, a slow filter creates backpressure that affects all traffic. With process isolation, a slow agent only affects the routes that use it, and only up to the concurrency limit you configured. The blast radius is bounded and declared. You can look at the config and know the worst case.</p> The agents communicate over Unix domain sockets or gRPC, and they can be written in any language. There are SDKs for Rust, Go, Python, TypeScript, Elixir, Kotlin, and Haskell. The protocol is simple: 4-byte length, 1-byte type, JSON or MessagePack payload.</p> The operational consequence is what I care about most: you can deploy, restart, and update agents independently. Roll out a new WAF rule set without touching the proxy. Restart a misbehaving auth agent without dropping a single connection. When you are trying to fix one thing at 3 AM without breaking three others, that independence matters more than any benchmark number.</p> A product of this moment</h2> Every piece of software is shaped by when it was built.</p> The proxies I described earlier were products of their time not just in what problems they solved, but in how they could be built. Nginx needed Igor working for years. Envoy needed Google, Lyft, and a large open source community. HAProxy needed Willy Tarreau’s decades of sustained refinement. That was the only way to build infrastructure of that quality. One person, or even a small team, could not realistically build a production-grade reverse proxy with a novel architecture and ship it in months. The economics did not allow it.</p> That structural reality is changing, and I think the change matters more than most people in infrastructure have absorbed yet.</p> I wrote about the broader shift in How I Work These Days</a>. The short version: I had been using Claude Code since May 2025, but it was not until Christmas 2025, working with Opus 4.5, that something fundamentally clicked. The constraint I had lived with for years, the gap between knowing exactly what I wanted and having the bandwidth to build it, narrowed in a way that I still find hard to fully describe. Not because the model wrote the code for me. But because the feedback loop between design intent and working implementation compressed from weeks to hours.</p> When Cloudflare open-sourced Pingora in 2024, I had paid attention immediately. A proxy framework written in Rust, battle-tested at over a trillion requests per day in Cloudflare’s own network. The TCP listener, the HTTP parser, the TLS termination, the connection pooling, the async runtime. All the low-level machinery that you do not want to write from scratch. I had watched River, the community Pingora-based reverse proxy, hoping it would become the thing I could reach for and trust. It never got there.</p> So I stopped waiting and started building. Pingora as a foundation. Rust for memory safety at the boundary. An agentic workflow that let one person move at the pace of a small team.</p> What came out was not a general-purpose proxy. It was not a Swiss army knife. It was a purpose-built tool, tailored from the start to one specific problem: safe, observable, operatable edge traffic enforcement. No more, no less.</p> This is the part I think matters beyond Zentinel itself: when the cost of building serious software drops dramatically, you can afford to be specialized. You do not need to serve a broad market to justify the investment. You can build exactly the thing that solves exactly your problem, with exactly the tradeoffs you want. No feature creep driven by needing to justify a twenty-person team. No compromises driven by needing to appeal to every possible use case.</p> I think of it as bespoke infrastructure. Software that is tailored to a specific problem by someone who deeply understands that problem, made viable by tools that compress the gap between “I know exactly what this should be” and “it exists.” The same way that a load balancer was the right thing to build in 2000, and a service mesh data plane was the right thing to build in 2016, a purpose-built safe reverse proxy is the right thing to build in 2026. Not because the world suddenly needs one more proxy, but because the world can finally have proxies that are designed for specific jobs instead of being general enough to justify their development cost.</p> Zentinel is a post-agentic reverse proxy. Not because it uses AI internally (it does not, unless you count the inference-aware rate limiting for LLM traffic). But because it could only exist in a world where agentic development made bespoke infrastructure viable. One person. Three months. A clear vision that had been accumulating for years. That is not a story that was possible to tell in 2020.</p> Glass-box infrastructure</h2> There is a conviction behind Zentinel that is not technical, and I want to be honest about it.</p> I believe critical web infrastructure should be open. Not “open core” with the important parts behind a license. Not “source available” with restrictions on how you run it. Open in the way that matters: you can read it, fork it, modify it, run it on your own hardware, never call anyone for permission.</p> Zentinel is Apache-2.0-licensed. Every agent is open source. The configuration format is documented. The protocol is specified. There is no hidden control plane, no phone-home telemetry, no vendor dependency.</p> But open source alone is not what I mean by transparent. Plenty of projects publish their source and remain effectively opaque. The code is there, technically, but understanding what a particular configuration will actually do still requires reading thousands of lines of parser logic, or just deploying it and hoping for the best.</p> This is where the Rust decision pays off in a way I did not fully anticipate when I started.</p> Because Zentinel is written in Rust, the core crates compile to WebAssembly. Not as a side project or a reimplementation. The same zentinel-config</code> crate that parses and validates your KDL configuration in production compiles to a Wasm module that runs in a browser tab.</p> This means the config playground</a> on the Zentinel website is not a JavaScript approximation of what the parser does. It is the parser. The actual Rust code, compiled to Wasm, running against your configuration in real time. When it says your config is valid, that is the same validation logic that will run when Zentinel starts up on your server. When it flags an error, that is the same error you would see in production.</p> The same applies to the config converter</a>. If you are migrating from nginx, HAProxy, or Traefik, the conversion tool runs the actual Zentinel config crate in your browser. You paste your existing config, you get KDL output, and you can validate the result on the spot. No round-trip to a server. No “upload your infrastructure config to our cloud service.” It runs locally, in your browser, using the production code.</p> This matters more than it might sound. When you can run the same code that your proxy runs in production, right in your browser, you can reason about the proxy’s internal behavior directly. You are not trusting documentation about how the parser interprets a particular KDL construct. You are running the parser. You are not guessing what happens when two routes have the same priority. You are watching the actual matching logic evaluate your routes.</p> The proxy becomes glass-like. Not transparent in the “we published the source, good luck reading it” sense. Transparent in the sense that you can interact with its internals, poke at its logic, verify its behavior before it ever touches production traffic. The same Rust, the same types, the same validation rules, running wherever you need them: on the server, in CI, in your browser.</p> That is what I mean by trustworthy infrastructure. Not “trust us, it works.” Trust it because you can verify it yourself, using the same code, without asking anyone for permission.</p> Where this leaves me</h2> Every generation of proxies solved the problem of its era. HAProxy solved load balancing. Nginx solved web serving. Envoy solved service mesh routing. Varnish solved caching. Each was the right answer at the right time, and each was shaped by what was possible when it was built.</p> The problem I kept running into was different. Not throughput. Not feature count. Not service discovery. Just: can I understand what this system will do at 3 AM when something I did not plan for happens? Can I reason about its failure modes from reading its configuration? Can I trust it enough to sleep?</p> No existing proxy was designed from scratch for that question, because no existing proxy could afford to be that specialized. The economics of building infrastructure software pushed everything toward generality.</p> What changed is that the economics changed. One person, with the right foundation and the right tools, can now build purpose-built infrastructure that would have required a team and a multi-year roadmap before.</p> Zentinel is the proxy I needed and could not find, built in the specific window of time when building it became possible. It is a product of this moment, in the same way that every proxy before it was a product of its own.</p> And maybe that is what this era of software is really about. Not that AI writes code for you. That framing misses the point entirely. It is that the gap between knowing what should exist and making it exist got smaller, and the things people build when that gap closes are going to be very specific, very opinionated, and very good at exactly one thing.</p> References and further reading</h2> Zentinel</a> - The source code, Apache-2.0-licensed</li> Zentinel documentation</a> - Architecture, configuration reference, agent protocol</li> Zentinel playground</a> - Browser-based config validation using the actual Rust crate compiled to Wasm</li> Zentinel config converter</a> - Migrate from nginx, HAProxy, or Traefik configs to KDL</li> Zentinel manifesto</a> - The design philosophy in full</li> Pingora</a> - Cloudflare’s open source proxy framework that Zentinel builds on</li> How I Work These Days</a> - The broader shift in how I build software, and where Zentinel fits in that story</li> HAProxy</a> - Willy Tarreau’s load balancer, still one of the best pieces of infrastructure software ever written</li> Nginx</a> - Igor Sysoev’s web server that became the internet’s default reverse proxy</li> Envoy</a> - The service mesh data plane that brought observability to distributed systems</li> Varnish</a> - Poul-Henning Kamp’s caching proxy</li> </ul> How I Work These Days Unknown — Sun, 15 Mar 2026 00:00:00 +0000 If you had asked me a few years ago what kind of shift would truly change software again, I would probably have said something vague about machine learning becoming more useful, more accessible, more integrated into normal tooling. I would not have said that within a few years I would be spending large parts of my day in conversation with models, building products at a pace that used to feel unrealistic for one person.</p> But that is where I am now, and the path here did not start with ChatGPT. It started earlier.</p> Before the shock</h2> I had been on Kaggle and Hugging Face since 2020. I was already paying attention. I had a decent understanding of machine learning, enough to know that something important was happening. I was not looking at this space as an outsider who suddenly discovered AI in a news cycle. I had been around it long enough to see that the ingredients were there.</p> Still, understanding a field and feeling a historical shift are not the same thing. When OpenAI released ChatGPT 3.5 in November 2022, something in me changed almost immediately. I do not mean that in a mystical way. I mean I recognized, very quickly, that this was not just another incremental product launch. It felt like a boundary marker, one of those moments where you can see a new layer of the technology stack forming in front of you.</p> At the time, I thought: this is going to be enormous. Bigger than most people realize. Bigger, maybe, than the web itself in terms of how deeply it will alter the shape of work, software, and the distribution of capability. That sounds exaggerated when people say it too casually. I know that. But that was honestly my reaction back then. Not hype. Recognition.</p> I was one of the early people willing to pay OpenAI. That mattered to me. I wanted access, and I wanted to stay close to the frontier as it moved. I did not want to be reading second-hand summaries while something this consequential was taking shape in real time.</p> The part I still underestimated</h2> Even then, I still underestimated one thing: the speed. I understood generative AI was around the corner. I did not understand just how fast it would become operationally useful for actual software creation.</p> I had read AI 2027, the scenario work by Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, and Romeo Dean. It stayed with me, and it still does now in March 2026. I still think the basic direction it sketches is largely correct, even if the path is turning out a bit differently in practice than any single forecast can capture. But even with that framing in my head, I did not fully expect that less than two years after ChatGPT 3.5, coding agents would already start to feel like a real category rather than a novelty.</p> That part came faster than I thought.</p> Paying attention, waiting for the right moment</h2> In 2025 I was trying different tools seriously. I was paying for Claude Code from May 2025 onward, but I was not especially impressed at first. I liked the CLI orientation. I liked that it felt coder-friendly. That part made sense to me immediately. But the model quality at that moment, and the rate limiting, left me cold. It was interesting. It was not yet transformative for my own workflow.</p> So I mostly leaned on Zed’s offering. I liked the editor experience, and I still do. I still reach for Zed when I want to inspect, edit, or move through files outside of Vim in the terminal. It fit me better in that phase.</p> Then November 2025 arrived, three years after ChatGPT 3.5, and then December came with Christmas break. I gave Claude Code another real try, this time with Opus 4.5, and that was the moment it really landed for me. Not politely. Not academically. It hit me hard.</p> I remember the feeling very clearly: this is it. This is the first time the whole thing feels like more than an assistant and less than a gimmick. This is a tool I can genuinely build with.</p> Zentinel was the proof</h2> Zentinel had been sitting in the back of my mind for a long time. The idea was not new. The frustration behind it was not new either. At my day job, we had been dealing with unreliable reverse proxies for long enough that the pain was familiar. I had always wanted River, the Pingora-based reverse proxy, to succeed. I wanted that project to become the thing I could reach for and trust. But it never got there.</p> So I did what this new moment suddenly made possible: I stopped waiting for somebody else to build the thing I wanted to exist.</p> I built Zentinel with Opus 4.5, and it worked.</p> That is the part that still feels a little surreal when I say it plainly. Three months later it is up, it is real, and people are using it. Not as a demo. Not as an abandoned prototype. As actual software in the hands of actual users.</p> That changed something fundamental in how I think about work. Once one long-held idea made it through that bottleneck, a lot of others started moving too. It was not just that I had a new tool. It was that the relationship between ambition and execution had changed. The old constraint, the one that said “yes, this could exist, but not with your current time and current bandwidth,” had weakened.</p> Then everything else started moving</h2> In the time since, I have built a whole range of things that had been accumulating in my head for years: Cyanea, Archipelag, Humankind, Arcanist and its Rust-based hx</code> Haskell toolchain, the new Basel Haskell Compiler, and other pieces besides.</p> It has been a wild ride, but not in the shallow “everything is crazy” sense people often write about. More in the sense that an internal dam broke. For a long time I had more ideas than I had time, more design clarity than I had execution bandwidth, and more conviction than I had manpower. That is a frustrating place to live in for years. You learn to carry around a quiet backlog of unrealized things. Some of them stay alive as notes. Some become recurring thoughts during commutes or late at night. Some start to hurt a little because you know they are viable, but you also know you are not going to get to them with the tools and energy available to you at that point in your life.</p> Now the world is different. I can finally push through the backlog that used to exist only in notebooks, mental sketches, half-written design docs, and conversations with myself.</p> What my days look like now</h2> My daily routine changed dramatically.</p> I still have a day job, and then I have the rest of my work. Together it often adds up to something close to sixteen hours a day. That would sound bleak if I were forcing it. It does not feel bleak. It feels like release.</p> My workspace reflects that change. These days I use mostly Apple hardware, which is funny if you know how much of a Linux person I am, and how much of an OpenBSD person I still am. That part of me has not gone anywhere. I still love those systems. I still think they matter deeply. But if I am being honest about the practical question of where I am most productive right now, Apple has become the answer.</p> I work across a MacBook Pro M4, an iMac, and an Apple Vision Pro. I spend time talking ideas out in real time with ChatGPT’s voice mode, using it less like a search engine and more like a sparring partner. Sometimes intimate, sometimes brutally honest, sometimes audacious in exactly the way a good thinking partner should be. I push an idea, it pushes back, I sharpen it, it sharpens me, and we keep going until something solid emerges.</p> </p> Then there is the terminal, where much of the actual implementation happens in Ghostty, usually with four panes open, often with multiple Claude Code agents running in parallel on the Max plan. Two hundred dollars a month for that level of leverage is, for me, one of the clearest trades I have ever made.</p> This is not a lifestyle performance. It is just the current shape of my work: ideas moving between speech, terminal, editor, design notes, code, back to speech, then back to code again.</p> Beast mode, for lack of a better term</h2> There is a part of this that feels almost embarrassingly direct to say, but it would be dishonest to leave it out: I am the happiest I have ever been.</p> Not because everything is easy. It is not. Not because every project succeeds. They will not. Not because the industry suddenly became sane. It did not.</p> I am happy because the mismatch that used to define so much of my working life has narrowed. For years I had to live with the feeling that my ideas were outrunning my available hours and my available hands. Now, for the first time, it feels like I can actually meet myself where my ambition has been waiting.</p> There is a phrase people use, “beast mode,” and usually I would avoid it because it sounds like posturing. But I do not really have a cleaner shorthand for the intensity of this period. I am working hard, very hard, but with a degree of joy and clarity that makes the effort feel proportionate.</p> I am in conquest mode.</p> Not conquest in the empty startup sense. Not domination, not vanity metrics, not growth for its own sake. I mean conquest over the inertia that used to keep good ideas trapped inside my head. Conquest over backlog. Conquest over hesitation. Conquest over the old excuses about lacking time, lacking team, lacking the right moment.</p> And yes, some of it is for me. To feel better. To feel more whole. To stop carrying around years of deferred execution. But some of it is also because I genuinely want to make useful things. I want to build software that improves the texture of work, that makes systems more reliable, that gives people better tools, that opens up possibilities that were previously too expensive or too cumbersome to pursue.</p> That still matters to me. Probably more than ever.</p> The real change</h2> So when I say that the way I work these days has changed, I do not just mean that I use different tools.</p> I mean that the relation between thought and execution changed. The lag collapsed. The emotional burden of unrealized ideas shrank. The number of things that are now viable to attempt expanded dramatically. That is the real story.</p> I was already paying attention in 2020. I recognized the significance of ChatGPT in 2022. I underestimated the speed anyway. Then late 2025 arrived, the tools crossed a threshold, and my daily life reorganized itself around that fact.</p> Three years is not a long time. It feels longer when you live through a real transition.</p> And I suspect we are still early.</p> References and further reading</h2> AI 2027</a> - Scenario work that influenced how I thought about the trajectory of this space</li> Zed</a> - Editor I still use alongside Vim</li> Ghostty</a> - Terminal I use for most of my agent-heavy coding sessions</li> Zentinel</a> - Reverse proxy project that became the first real proof point for this workflow</li> Cyanea</a> - One of the projects that came to life during this period</li> Archipelag</a> - Another product that moved from idea to reality in this new working mode</li> </ul> Archipelag.io Is in Open Beta: Here's Why I Built It Unknown — Fri, 13 Mar 2026 00:00:00 +0000 There is an abandoned factory building in Glarus, a small town wedged between mountains in eastern Switzerland. In 2016, the building was loud. Not machinery-loud, fan-loud. Rows of bare motherboards bolted to open-air frames, each bristling with GPUs and daisy-chained power supplies. The air tasted like warm dust and ozone. Cables ran everywhere, held in place by zip ties and optimism. This was an Ethereum mining operation, and I was standing in the middle of it, watching people I knew convert their gaming rigs, hardware they loved, into money-printing machines.</p> I was there because Vitalik Buterin had decided to visit. He had flown in on a private jet to Geneva, driven up in a black limousine with tinted windows, and walked into this dusty, chaotic space to see what Swiss miners were building. It was surreal. The creator of Ethereum, stepping over power cables in an industrial ruin, nodding at rack after rack of GPUs humming away at proof-of-work hashes. I do not think he was impressed by the elegance of the setup. Nobody was. But something about that scene stuck with me.</p> People were willing to sacrifice their gaming entertainment, their leisure hardware</em>, to chase the dream of sovereign financial independence using fundamentally nerdy equipment: PCs, internet connections, blockchain protocols, and GPU graphics cards. They were converting consumer-grade technology into economic infrastructure, and they were doing it themselves. No data center leases. No vendor contracts. No permission from anyone. Just people, hardware, and a protocol that made it worth their while.</p> I had skin in the game too. I invested (gambled, honestly) in crypto during that era. I watched the charts, rode the swings, felt the dopamine spikes and the stomach-drops. The financial side was wild and ultimately unsustainable for most people. But the infrastructure</em> side, the part where ordinary humans turned their homes into compute nodes and got paid for it: that part was real, and that part stayed with me long after the crypto hype faded and the rigs went quiet.</p> This is the story of how that factory visit turned into Archipelag.io</a>, a distributed compute network that entered open beta today. It has been ten years of thinking, one year of building, and a lot of being wrong about the right things at the wrong time.</p> How AI Makes Bare Metal Viable Again Unknown — Sun, 08 Mar 2026 00:00:00 +0000 I was paying over two hundred dollars a month to run two apps that had zero paying users.</p> Not because the apps were complex. Not because they needed high availability across regions. Because I was running Kubernetes on DigitalOcean, and Kubernetes has opinions about how much infrastructure you need. A control plane. Worker nodes. Load balancers. Persistent volumes. Managed databases. Each line item modest on its own, adding up to a bill that felt absurd for two Phoenix applications in their bootstrapping phase.</p> The apps are archipelag.io</a> and cyanea.bio</a>. Both are Elixir/Phoenix projects. Archipelag uses PostgreSQL and NATS for its messaging layer. Cyanea uses SQLite. Neither gets meaningful traffic yet. Both are real products I am actively building, not side projects I will abandon next month. But they are pre-revenue, and every dollar I spend on infrastructure is a dollar I am betting against future income that does not exist yet.</p> Something had to change.</p> The Kubernetes trap</h2> Here is the thing about Kubernetes: it solves problems you might not have. If you are running fifty microservices across three regions with autoscaling requirements and a platform team to manage it, Kubernetes earns its keep. If you are running two BEAM applications that each consume less than 512 MB of memory, you are paying a complexity tax for infrastructure capabilities you will never touch.</p> My K8s setup on DigitalOcean looked like this: a managed cluster with two worker nodes (the minimum for any reasonable availability), a managed PostgreSQL instance for Archipelag, a load balancer for ingress, persistent volumes for Cyanea’s SQLite database. Each component had its own monthly cost. The cluster management fee alone was more than what I would eventually pay for an entire bare metal server.</p> The operational overhead was worse than the cost. Helm charts. Ingress controllers. Certificate managers. Pod disruption budgets. Every time I wanted to deploy a new version, I was wrangling YAML files that described infrastructure concerns my apps did not care about. A Phoenix release does not need a pod spec. It needs a port, an environment, and someone to restart it if it crashes.</p> And the YAML, my God, the YAML. A simple Phoenix app that listens on a port and serves HTTP needs, at minimum, a Deployment manifest, a Service manifest, and an Ingress manifest. Add a ConfigMap for environment variables, a Secret for credentials, a PersistentVolumeClaim if you need disk, a HorizontalPodAutoscaler if you want autoscaling. For Cyanea alone, I had six Kubernetes manifests totaling a few hundred lines of YAML, all to describe an application that boils down to: run this binary, give it a port, point a domain at it.</p> The cognitive load compounds. You learn the Kubernetes resource model, then the DigitalOcean-specific annotations for their load balancer, then the cert-manager CRDs for TLS, then the quirks of persistent volumes on managed K8s (spoiler: they are not as persistent as you think if you do not get the reclaim policy right). Each layer has its own documentation, its own failure modes, its own upgrade cycle. I spent more time debugging infrastructure than building product.</p> The irony is not lost on me. Kubernetes was designed for teams running hundreds of services at Google-scale. I was running two apps. The orchestrator had more moving parts than the things it was orchestrating. It was like hiring a logistics fleet to deliver two packages across town.</p> I knew I was over-engineered. But the alternative, at the time, seemed like a step backward.</p> The Nomad detour</h2> I should mention that Kubernetes was never the only orchestrator I considered. For the past five years, while the industry went all-in on K8s, I had been quietly admiring HashiCorp’s Nomad</a>. Where Kubernetes is a sprawling ecosystem of CRDs, operators, and control loops, Nomad is refreshingly minimal. A single binary. A simple job spec. No opinions about networking, no built-in service mesh, no mandatory etcd cluster. You tell it what to run, it runs it.</p> That minimalism appealed to me. Nomad treats workload scheduling as the core problem and stays out of everything else. No built-in networking layer means you bring your own, which sounds like a drawback until you realize it means you are not locked into someone else’s networking model.</p> And I happened to have my own networking layer already. I had been building Zentinel</a> in parallel, a security-first reverse proxy built on Cloudflare’s Pingora</a> framework in Rust. Zentinel handles TLS termination, WAF inspection, rate limiting, domain-based routing, all the edge concerns I care about. It also supports sleepable ops, where backend instances can be suspended and woken on demand, which is perfect for apps that do not need to be running 24/7.</p> So I tried pairing them. Nomad for workload scheduling, Zentinel for the network layer. And it worked. The combination gave me a lightweight orchestrator that did not try to own every concern, paired with a reverse proxy that handled edge traffic the way I wanted. Two focused tools, each doing one thing well.</p> But then IBM acquired HashiCorp, and the calculus changed.</p> The acquisition itself was not the problem. Companies get acquired. It happens. The problem was the trajectory. HashiCorp had already re-licensed Terraform from MPL to BSL (Business Source License) in 2023, a move that fractured the community and spawned the OpenTofu</a> fork. The pattern was familiar: open-source project gains adoption, company monetizes through enterprise features, company gets acquired, new owner tightens the screws. I had watched it happen with Redis, with Elasticsearch, with MongoDB. Each time the community forks, there is a period of uncertainty, split maintenance effort, and feature divergence.</p> I did not want to build my infrastructure on a foundation where the governance could shift at any time. Nomad is still open source today. But “still open source” and “will remain open source” are different statements, and after the Terraform situation, I was not confident in the latter. The BSL license change had been a signal, and IBM’s acquisition amplified it. I did not need to go down that road with another HashiCorp product.</p> The Nomad experiment did teach me something valuable, though. It confirmed that the KISS approach to deployment was right. You do not need the full Kubernetes machinery. A scheduler that starts processes, checks their health, and restarts them when they crash is sufficient for a wide range of workloads. And a dedicated reverse proxy that handles TLS and routing is cleaner than bundling networking into the orchestrator.</p> That insight, Nomad’s minimalism plus Zentinel’s Pingora-based proxy architecture, became the design seed for what I would eventually build.</p> The fly.io middle ground</h2> With Nomad off the table as a long-term bet, I migrated to fly.io</a> in late 2025. It was genuinely better than K8s for my use case. Fly understands BEAM applications at a fundamental level. The BEAM runtime is designed for the kind of lightweight, long-lived processes that Fly’s infrastructure optimizes for. You push a release, it runs it. No YAML. No ingress controllers. No cluster management.</p> Fly also made the service dependencies painless. Managed Postgres with a few commands. NATS was straightforward to set up. Tigris (Fly’s S3-compatible object storage) handled blob storage for Cyanea’s file uploads. The developer experience was genuinely excellent, and I mean that without reservation. The Fly team has built something thoughtful.</p> The cost dropped meaningfully. No cluster management fee. No minimum node count. Pay-per-VM pricing that scales down to fractions of a shared CPU. Fly’s model is honest about what small applications actually need, and the pricing reflects that. I went from over two hundred dollars a month on DigitalOcean K8s to roughly a quarter of that.</p> For a while, it was the right answer. And if I had been scaling horizontally, adding regions, needing the kind of elastic compute that cloud-native platforms excel at, I would have stayed. If my apps suddenly got traction and I needed instances in Tokyo, Frankfurt, and Virginia, Fly would be the obvious choice. The multi-region story is one of Fly’s genuine strengths. You deploy once, it runs everywhere. That is hard to replicate.</p> But I was not scaling horizontally. I was running two apps in one location. On a good day, they handled maybe a few hundred requests. The compute they needed was trivial, a fraction of a shared CPU core. And I was still paying for a platform designed to scale to thousands of instances across dozens of regions, even though I needed exactly one instance of each app, in exactly one place, doing very little work.</p> There is also a subtler cost that managed platforms carry: the abstraction tax. When something goes wrong on Fly (and it did, occasionally, things like deployment timeouts or the odd networking hiccup), you are debugging at the platform level, not the system level. You file a support ticket or check the status page. You do not SSH in and look at processes, because there are no processes you can see. The platform is the intermediary, and the intermediary has its own failure modes that you cannot inspect or fix.</p> The cloud-native model, even the lean version that Fly offers, has a floor. You are always paying for the platform’s capabilities, not just your usage of them. When your usage is “two small apps, one location, no scale,” that floor matters. And when the platform sits between you and your processes, you lose the ability to debug at the level where the answers actually live.</p> The bare metal math</h2> I started looking at dedicated servers. Not VPS instances, not cloud VMs. Actual hardware you can SSH into, where your processes run on real cores and your data sits on real disks.</p> Hetzner runs a server auction</a> where they sell refurbished dedicated machines at steep discounts. These are servers that have been running in Hetzner’s data centers, got rotated out of customer contracts, and are resold at prices that make cloud compute look like a luxury good. The hardware is used but maintained, and Hetzner’s data centers are well-run, proper cooling, redundant power, good network connectivity.</p> I found a box with a multi-core Intel CPU, 128 GB of DDR4 RAM, and two 1 TB NVMe drives that I configured in RAID 1 for redundancy. EUR 38 a month. About forty-two dollars. Fixed price. No bandwidth metering (Hetzner includes 20 TB of traffic on dedicated servers, which for my workload might as well be unlimited). No surprises on the bill.</p> Let that sink in for a moment. For less than what I was paying for managed Postgres alone on either platform, I could have an entire server with more RAM than I know what to do with, fast NVMe storage with mirror redundancy, and enough compute headroom to run not two but twenty applications without breaking a sweat. The two NVMe drives alone, if bought retail, would cost more than a year of hosting.</p> I ran the numbers on capacity. My two Phoenix apps, even under load, would use maybe 1-2 GB of RAM combined. PostgreSQL with a modest dataset, another gig or two. NATS, negligible. That leaves well over 120 GB of RAM sitting idle. The CPU tells a similar story. Phoenix on the BEAM is remarkably efficient with CPU resources, the scheduler does its own preemptive multitasking across lightweight processes, and my workloads are I/O-bound, not compute-bound. I could run my entire current stack and barely register on a load graph.</p> The headroom is the point. On a cloud platform, headroom costs money. More RAM, higher tier. More CPU, higher tier. On bare metal, the headroom is already paid for. Growing from two apps to ten does not change my monthly bill. Adding a staging environment does not change my monthly bill. Running background workers, a metrics stack, a CI runner, none of it changes my monthly bill. The marginal cost of additional workloads on existing hardware is zero.</p> The math was obvious. The problem was everything else.</p> Why bare metal was hard</h2> Bare metal has always been cheap. That was never the issue. The issue was everything you had to build and maintain yourself.</p> On a managed platform, you get deployment pipelines, TLS certificate management, process supervision, reverse proxying, log aggregation, health checks, and rollback mechanisms out of the box. On bare metal, you get a Linux login prompt and a blinking cursor.</p> Historically, going bare metal for web applications meant weeks of setup:</p> Install and configure nginx or HAProxy as a reverse proxy</li> Set up Certbot or acme.sh for Let’s Encrypt certificates, and hope the renewal cron does not silently break</li> Write deployment scripts (rsync, symlinks, restart commands) and debug them for months</li> Configure systemd services for each app, with the right restart policies and environment files</li> Build a process supervision layer that handles crashes, port allocation, and graceful shutdowns</li> Figure out zero-downtime deploys (which means running two instances, health checking the new one, swapping traffic, draining the old one)</li> Set up log rotation, monitoring, backups</li> Harden the server (firewall, SSH config, automatic security updates)</li> </ul> Each of these is a solved problem in isolation. There are blog posts and Stack Overflow answers for every one of them. But stitching them together into a coherent, reliable deployment system is a full-time job for a week or two, and maintaining it is an ongoing tax on your attention.</p> This is why the cloud won. Not because bare metal is expensive. Because the operational cost of doing it yourself was prohibitive for small teams. The cloud sold you a package deal: we handle the infrastructure, you handle the application. Worth it, even at a premium.</p> But what if that operational cost dropped to near zero?</p> The AI shift</h2> I had Claude Code with Opus 4.6 available. I had spent months working with it on other projects. Compilers, CRDT engines, reverse proxies. I knew what it could do with a clear spec and a well-defined problem domain.</p> And deploying web applications to bare metal is a well-defined problem domain.</p> The core requirements are straightforward: upload an artifact, start it on a port, check that it is healthy, route traffic to it, stop the old one. Everything else, TLS, process supervision, rollback, log capture, is layered on top of that core loop. The problem space is wide but shallow. Lots of features, few genuinely novel algorithms.</p> This is exactly the kind of work where AI shines. Not because it writes perfect code on the first try. But because it can iterate through a feature list at a pace that would take a solo developer weeks, producing working implementations in hours. The feedback loop is tight: describe what you want, get code, test it, refine. The domain knowledge exists in a thousand deployment tools that came before. The AI has seen all of them.</p> So I decided to build my own deployment tool. From scratch. With AI as my co-engineer.</p> Building Vela</h2> Vela</a> is what came out of that process. A single Rust binary that handles everything I listed above: reverse proxy, auto-TLS, process supervision, zero-downtime deploys, health checks, secret management, log streaming, rollbacks. No containers. No Docker. No YAML.</p> The design draws from both of its ancestors. From Nomad, the suckless philosophy: a single binary, minimal configuration, no opinions about things that are not its problem. From Zentinel, the Pingora-inspired proxy architecture: hyper-based reverse proxy with TLS termination, domain-based routing, and WebSocket support baked into the same process. Vela is what happens when you take the best ideas from tools you admire and combine them into something purpose-built for your exact workload.</p> The design philosophy is blunt: one binary, two modes.</p> ┌─────────────────────────────────────────────┐ │ Your server │ │ │ │ Vela daemon │ │ ├── Reverse proxy (:80/:443, auto-TLS) │ │ ├── Process manager (start, health, swap) │ │ └── IPC socket │ │ │ │ Apps │ │ ├── cyanea.bio → :10001 │ │ └── archipelag.io → :10002 │ └─────────────────────────────────────────────┘ ┌─────────────────────────────────────────────┐ │ Your laptop │ │ │ │ vela deploy → scp + ssh → server │ └─────────────────────────────────────────────┘ </code></pre> vela serve</code> runs on the server. It is the reverse proxy, the process manager, and the IPC daemon, all in one process. vela deploy</code> runs on your laptop. It reads a manifest, uploads your artifact over SSH, and tells the server to activate it.</p> SSH is the control plane. No tokens, no API keys, no custom authentication layer. If you can SSH into the server, you can deploy. This is a deliberate choice. SSH key management is a solved problem. Every developer already has it configured. Every server already has it running. Building a custom auth system on top would be adding complexity for no practical gain.</p> The manifest</h3> Each app gets a Vela.toml</code> in its project root:</p> [app] name = "cyanea" domain = "app.cyanea.bio" [deploy] server = "deploy@my-server" type = "beam" binary = "server" health = "/health" strategy = "sequential" pre_start = "bin/cyanea eval 'Cyanea.Release.migrate()'" [env] DATABASE_PATH = "${data_dir}/cyanea.db" SECRET_KEY_BASE = "${secret:SECRET_KEY_BASE}" </code></pre> That is the entire deploy configuration. The app type tells Vela how to start it (beam</code> runs Elixir releases, binary</code> runs compiled executables). The health path tells it where to check. The strategy tells it how to swap traffic. The pre_start</code> hook runs database migrations before the new instance starts, and if migrations fail, the deploy aborts and the old instance keeps running.</p> Environment variables support two substitution patterns: ${data_dir}</code> expands to the app’s persistent data directory (which survives deploys), and ${secret:KEY}</code> pulls from the server-side secret store. Secrets never live in your repo.</p> Deploying looks like this:</p> MIX_ENV=prod mix release vela deploy ./_build/prod/rel/cyanea </code></pre> Two commands. The artifact goes up, the health check passes, traffic swaps, done.</p> Zero-downtime deploys</h3> Vela supports two deploy strategies, and the choice matters.</p> Blue-green</strong> is the default. The new instance starts alongside the old one on a fresh port. Vela runs a health check against it (30 retries, one per second, five-second timeout per attempt). Once the health check passes, the reverse proxy atomically swaps the route table entry for that domain to point at the new port. The old instance gets a configurable drain period to finish in-flight requests, then receives SIGTERM. If it does not exit within the drain window, SIGKILL.</p> Time ──────────────────────────────────────────► Old instance ████████████████████░░░░ (draining) New instance ░░░░████████████████████ ▲ ▲ start │ │ health passes, swap </code></pre> Zero downtime. The user never sees a blip. This works for stateless apps and apps backed by PostgreSQL (where both instances can connect to the same database simultaneously).</p> Sequential</strong> is for SQLite apps. You cannot have two processes writing to the same SQLite database (WAL mode helps, but concurrent writers from separate instances is asking for trouble). So Vela stops the old instance first, starts the new one, health checks it, and activates it. Sub-second blip. Acceptable for apps where the alternative is write contention.</p> Time ──────────────────────────────────────────► Old instance ████████████████████ New instance ░░░░████████████████████ ▲ ▲ stop │ │ start + health check </code></pre> The decision is per-app, configured in the manifest. Cyanea uses sequential (SQLite). Archipelag uses blue-green (PostgreSQL).</p> Process supervision</h3> Vela does not just start your app and walk away. It supervises it. If a process crashes, Vela detects the exit (via non-blocking try_wait</code> on the child process handle), logs it, and restarts from the stored launch configuration:</p> pub async fn check_and_restart(&mut self) -> Vec<String> { let mut to_restart = Vec::new(); for (key, process) in &mut self.running { match process.child.try_wait() { Ok(Some(status)) if !status.success() => { // Process exited unexpectedly to_restart.push(( key.clone(), process.launch_config.clone(), )); } _ => {} } } for (key, config) in to_restart { // Restart on same port if available, allocate new otherwise self.restart_from_config(&key, &config).await; } } </code></pre> Each app’s LaunchConfig</code> (release directory, binary name, app type, environment variables, data directory) is stored so that restarts use the exact same configuration. The daemon also persists app state to disk, so if Vela itself restarts (server reboot, daemon upgrade), it restores all running apps from their saved configurations.</p> This is the kind of feature that would take a day to specify and a week to implement if you were writing it from scratch. With Claude, it took about an hour of iteration, including the edge cases around port reallocation and the pending/active state split during deploys.</p> Built-in services</h3> Both of my apps have service dependencies. Archipelag needs PostgreSQL and NATS. Rather than managing these separately, Vela handles service provisioning directly:</p> [services.postgres] version = "17" databases = ["archipelag_prod"] [services.nats] version = "2.10" jetstream = true </code></pre> On first deploy, Vela installs PostgreSQL (via apt), creates the database with a generated password, and injects DATABASE_URL</code> into the app’s environment. For NATS, it downloads the binary, generates a config, and starts it as a supervised child process with NATS_URL</code> injected. Service credentials persist across deploys and daemon restarts.</p> This was one of those features where the AI really earned its keep. The NATS lifecycle management alone, downloading the right binary for the platform, generating config, supervising the process, health-checking the monitoring endpoint, persisting credentials, involved touching six or seven modules. Claude handled the plumbing while I focused on the design decisions.</p> The reverse proxy</h3> Vela embeds its own reverse proxy built on hyper. It handles TLS termination (auto-provisioned via Let’s Encrypt ACME HTTP-01, or static certificates for Cloudflare setups), domain-based routing, WebSocket upgrades, and HTTP-to-HTTPS redirects.</p> The routing model is simple. A thread-safe hash map from domain to port:</p> pub struct RouteTable { routes: Arc<RwLock<HashMap<String, u16>>>, } </code></pre> When a request arrives, Vela extracts the Host header, looks up the port, and forwards the request to localhost:{port}</code>. When a deploy swaps traffic, it is a single write-lock on the hash map to update the port number. Atomic. No configuration reload. No proxy restart.</p> For WebSocket connections (which both Phoenix apps use for LiveView), Vela detects the Upgrade: websocket</code> header and switches to raw TCP tunneling with bidirectional I/O. This was important for my use case, Phoenix LiveView is WebSocket-native, and if the proxy does not handle upgrades correctly, the entire UI breaks.</p> From empty box to production in a day</h2> Here is the timeline of the actual migration. I bought the Hetzner server and within about 48 hours, both apps were running in production with HTTPS, process supervision, automated backups, and daily health reports.</p> The sequence went roughly like this:</p> Hardware validation</strong>: Check NVMe drive health, run memory tests, verify RAID configuration. The drives had about 25,000 power-on hours (these are auction servers, they have been used), but SMART health passed and wear levels were well within acceptable range.</p> </li> OS provisioning</strong>: Debian, RAID 1 across both NVMe drives. Straightforward.</p> </li> Server hardening</strong>: Firewall rules, SSH hardening (key-only auth, non-default port, rate limiting), automatic security updates, intrusion detection. This is the part I am deliberately vague about. If you are running a public-facing server, hardening is non-negotiable, but I am not going to publish my exact firewall configuration.</p> </li> Vela installation</strong>: Download the binary, create a config file, install the systemd service. Five minutes.</p> </li> First app deployed (Cyanea)</strong>: Built the Elixir release on the server, set secrets, ran migrations, deployed. The entire build-and-deploy cycle for a Phoenix app with a Rust NIF took about fifteen minutes, most of which was compilation.</p> </li> Second app deployed (Archipelag)</strong>: Same flow, plus provisioning PostgreSQL and restoring a database dump from Fly, plus setting up NATS. About thirty minutes.</p> </li> TLS certificates</strong>: Updated DNS records, Let’s Encrypt certificates issued automatically. Vela handles the ACME challenge internally, no Certbot, no cron job, no manual cert management.</p> </li> Monitoring</strong>: A daily health report script that checks system metrics, service status, and app health, then emails a summary. Simple but effective.</p> </li> </ol> The most time-consuming part was not the tooling. It was migrating the PostgreSQL data from Fly and verifying that both apps behaved correctly in their new environment. The infrastructure setup itself, the part that would have taken weeks without Vela, took hours.</p> The broader thesis</h2> Here is what I think is happening, and I think it is bigger than my personal infrastructure bill.</p> The cloud won because it sold a bundle: compute, networking, storage, deployment, monitoring, scaling, security, all integrated, all managed. The alternative was building each piece yourself, and the labor cost made that prohibitive for small teams. Managed infrastructure was cheaper than an ops engineer.</p> AI changes that equation. Not by making the cloud cheaper, but by making bespoke tooling economically viable.</p> Consider what I got with Vela. A deployment tool that does exactly what I need and nothing more. No container orchestration, because I do not use containers. No multi-region routing, because I run in one location. No autoscaling, because two apps do not need to autoscale. Every feature exists because I needed it. Every feature works with my specific stack (Elixir/BEAM, Rust, SQLite, PostgreSQL, NATS). The tool is tailored to my workload the way a bespoke suit is tailored to a body.</p> This kind of custom tooling used to be a luxury. You needed either a platform team that could invest weeks of engineering time, or the rare individual who was both a skilled systems programmer and willing to spend their evenings writing deployment tools instead of building products. The economics did not make sense for a solo founder or a two-person team.</p> With AI, the cost of building bespoke tooling drops by an order of magnitude. Not to zero, you still need to know what you want, you still need to test and iterate, you still need to understand the domain well enough to evaluate the output. But the gap between “I know what I need” and “I have a working implementation” shrinks from weeks to hours.</p> And when bespoke tooling is cheap, the cloud’s bundle becomes less compelling. You do not need the managed Kubernetes service if you can build a deployment tool that fits your exact needs. You do not need the managed database service if you can install PostgreSQL yourself and the AI helps you set up backups, monitoring, and failover. You do not need the managed TLS service if your deployment tool handles ACME natively.</p> What you are left paying for is compute and bandwidth. And for compute and bandwidth, bare metal is drastically cheaper than the cloud.</p> The API is bespoke</h2> There is a subtlety here that I think is worth calling out. When people talk about the cloud’s advantages, they often point to the API-driven experience. Infrastructure as code. Declarative configuration. Programmable everything. And that is real. The cloud’s API layer is genuinely valuable.</p> But the API does not have to come from a cloud provider. It can come from your own tooling.</p> Vela gives me an API-driven experience. I declare my app’s configuration in a TOML manifest. I run a single command to deploy. I can check status, stream logs, manage secrets, trigger backups, and roll back releases, all from my laptop, all through a CLI that speaks SSH to a daemon on the server. The experience is not worse than Fly or Heroku. In some ways it is better, because the tool does exactly what I need and nothing else, and when something goes wrong, I can read the source code.</p> The difference is that my “API” is a 5,000-line Rust binary instead of a multi-billion-dollar cloud platform. And that is fine. I do not need the platform. I need the interface. AI lets me build the interface.</p> This is, I think, the pattern that will play out more broadly. The cloud’s value was never just compute. It was the operational layer on top of compute, the tooling that made raw hardware usable. AI makes it possible to build that operational layer yourself, tailored to your needs, at a fraction of the cost. The cloud becomes optional. The server becomes a commodity. The differentiator is the tooling, and the tooling is something AI can help you build.</p> What the numbers look like</h2> Let me be concrete about costs, because this is ultimately an economic argument.</p> Kubernetes on DigitalOcean</strong> (my original setup):</p> Managed K8s cluster: ~$12/month (control plane fee)</li> Worker nodes (2x smallest): ~$24/month</li> Managed PostgreSQL: ~$15/month</li> Load balancer: ~$12/month</li> Persistent volumes, bandwidth, extras: ~$15/month</li> Total: ~$78-80/month</strong> (and this was after I trimmed it)</li> </ul> Fly.io</strong> (the middle ground):</p> Two Phoenix apps (shared-cpu-1x, 256MB each): ~$14/month</li> Managed Postgres: ~$25/month</li> Managed NATS: ~$20/month</li> Bandwidth, extras: ~$10/month</li> Total: ~$70/month</strong></li> </ul> The managed services were the killer. Fly’s compute pricing is fair, but managed Postgres and managed NATS added up fast. And that was at near-zero traffic. Egress pricing on Fly is metered, so if either app had started getting real user load, the bandwidth bill alone would have pushed the total well past a hundred dollars a month.</p> Hetzner bare metal</strong> (current):</p> Dedicated server (auction): EUR 38/month (~$42)</li> That is it. PostgreSQL, NATS, TLS, everything runs on the box.</li> Total: ~$42/month</strong></li> </ul> The Hetzner box is cheaper than Fly right now, and the gap only widens as usage grows. But the raw dollar comparison understates the difference. Look at what I am getting. 128 GB of RAM versus 512 MB. Multi-core CPU versus shared fractional cores. Two terabytes of NVMe storage versus a few gigs. Bandwidth that is essentially unlimited (Hetzner includes 20 TB of traffic) versus metered egress that scales with every user you add.</p> The capacity gap is the real story. On Fly, scaling from two apps to ten means linearly increasing costs, more VMs, more managed database instances, more bandwidth charges. On my Hetzner box, scaling from two apps to ten means… nothing. The resources are already there. I paid for them. PostgreSQL, NATS, any other service I want to run, it all fits on the same box with room to spare.</p> And there is no surprise bill. No bandwidth overage. No “your database exceeded the row limit” fee. No managed service add-on creep. Thirty-eight euros a month, every month, regardless of what I run on it.</p> When this does not apply</h2> I would be dishonest if I pretended bare metal is the right answer for everyone. It is not.</p> If you need multi-region presence</strong>, the cloud still wins. Running your own hardware in three continents is a different kind of problem. Edge computing, CDN-native architectures (which I wrote about previously</a>), and platforms like Fly or Cloudflare Workers are the right tools for workloads that need to be close to users worldwide.</p> If you need elastic scaling</strong>, bare metal does not flex. A server has fixed resources. If your traffic spikes 10x for an hour, you cannot add capacity on demand. You can over-provision (and at these prices, generous over-provisioning is affordable), but it is not the same as true elasticity.</p> If you do not understand the operational basics</strong>, bare metal will bite you. Server hardening, backup strategies, disk monitoring, security patching, these are your responsibility. The cloud abstracts them away. On bare metal, a missed security update is your problem. A full disk is your problem. A failed drive (RAID helps, but is not magic) is your problem.</p> If your team is large and needs guardrails</strong>, managed platforms provide consistency and governance that bare metal does not. Kubernetes is complex, but it is complex in a standardized way. Everyone knows how to deploy to K8s. Everyone knows how to debug a pod. Your custom Vela setup is legible to exactly the people who built it.</p> The sweet spot for bare metal, especially AI-assisted bare metal, is small teams building products that need reliability but not scale, performance but not elasticity, control but not standardization. Solo founders. Two-person startups. Side projects that might become real businesses.</p> What I learned</h2> The migration took about 48 hours from “I have an empty server” to “both apps are in production with HTTPS, monitoring, and automated backups.” Most of that time was data migration and validation, not infrastructure setup.</p> Vela is now at version 0.5.0 with a feature list I am genuinely proud of: blue-green and sequential deploys, process supervision with auto-restart, built-in reverse proxy with auto-TLS, service dependency management (Postgres and NATS), secret management, log streaming, rollbacks, remote builds, scheduled backups, deploy hooks, and machine-readable status output for monitoring integration.</p> I built most of it in a few focused sessions with Claude Code. Not because the code is trivial, it is about 4,000 lines of Rust with async IPC, Unix socket communication, ACME certificate management, process lifecycle handling, and a reverse proxy with WebSocket support. But because the problem domain is well-understood, the requirements were clear, and AI is remarkably good at turning clear requirements into working implementations.</p> The thing I keep coming back to: the cloud was never selling compute. It was selling convenience. And convenience used to require a company with thousands of engineers to build platforms that abstracted away the hard parts. Now, a developer with a clear idea of what they need and an AI that can write systems code can build a fit-for-purpose operational layer in a weekend.</p> That does not make the cloud irrelevant. It makes the cloud optional for a much larger class of workloads than it was before.</p> Buy a server. Build your tools. Ship your product. The infrastructure should be boring. With AI, it finally can be.</p> References and further reading</h2> Tools and platforms</h3> Vela</a> - The bare-metal deployment tool built in this article</li> Hetzner Server Auction</a> - Refurbished dedicated servers at steep discounts</li> fly.io</a> - The managed platform I migrated from</li> Nomad</a> - HashiCorp’s minimal workload orchestrator</li> OpenTofu</a> - Community fork of Terraform after the BSL relicense</li> Pingora</a> - Cloudflare’s Rust framework for building programmable proxies</li> hyper</a> - Rust HTTP library powering Vela’s reverse proxy</li> Let’s Encrypt</a> - Free TLS certificates, automated via ACME in Vela</li> NATS</a> - Lightweight messaging system used by Archipelag</li> </ul> Frameworks and runtimes</h3> Phoenix Framework</a> - Elixir web framework powering both apps</li> Erlang/OTP</a> - The BEAM virtual machine that runs Phoenix and Elixir</li> Rust</a> - Systems language Vela and Zentinel are written in</li> </ul> Projects referenced</h3> Zentinel</a> - Security-first reverse proxy built on Pingora</li> Archipelag</a> - Distributed compute platform</li> Cyanea</a> - Bioinformatics platform</li> Claude Code</a> - AI coding tool used to build Vela</li> </ul> Edge Systems Are the New Backend Unknown — Wed, 11 Feb 2026 00:00:00 +0000 A request arrives at your system. In the next 50 milliseconds, before any application code runs, this happens: TLS termination, route matching, WAF inspection against 285 detection rules, JWT validation, rate limit evaluation, request body validation against a JSON schema, and trace context generation. The request either dies at the edge or arrives at your backend pre-authenticated, pre-validated, and pre-authorized.</p> Five years ago, your backend did all of this. Every service validated its own tokens, enforced its own rate limits, ran its own security checks. Today, the backend might not even exist in the form you expect. It might be a static site served from edge nodes, a thin persistence API, or a headless CMS that publishes content to a CDN and never handles a user request directly.</p> Something shifted. Not just at the edge. On both ends.</p> The three-tier past</h2> The architecture most of us learned was simple. Client, backend, database. The browser rendered HTML, maybe ran some jQuery. The backend did everything: authentication, authorization, business logic, rendering, validation, rate limiting, session management. The database stored state. Clean separation, one direction, easy to reason about.</p> This model worked because the browser was dumb. It could render markup and submit forms. Any real computation had to happen on the server. The backend was fat by necessity, not by design.</p> Microservices made it worse. Consider a typical setup: a user service, an order service, a payment service, a notification service, an inventory service. Each one needs to validate JWTs. Each one needs to enforce rate limits. Each one needs input validation, request logging, and error handling. That is five services times six concerns. Thirty implementations of logic that should exist exactly once.</p> Now multiply. Real organizations have 15, 50, 200 services. Each team implements auth slightly differently. One uses a shared library, one copied the code two years ago, one rolled their own because the library did not support their token format. The rate limiting configurations drift. The logging formats diverge. A security patch to the JWT validation logic means PRs across every repository, coordinated deployments, and someone asking “did we get all of them?”</p> ┌──────────┬──────────┬──────────┐ │ Users │ Orders │ Payments │ │ Service │ Service │ Service │ ├──────────┼──────────┼──────────┤ Auth │ ✓ (v2.1) │ ✓ (v1.9) │ ✓ (v2.0)│ Rate limiting │ ✓ (lib) │ ✓ (copy) │ ✗ (none)│ Validation │ ✓ │ ✓ │ ✓ │ WAF/Security │ ✗ │ ✗ │ ✗ │ Logging │ JSON │ text │ JSON │ Tracing │ ✓ │ ✗ │ ✓ │ └──────────┴──────────┴──────────┘ </code></pre> Libraries helped. Service meshes helped more. But the complexity was still distributed across every service, in every team’s codebase, in every deployment pipeline. The mesh moved networking concerns to a sidecar. It did not move application-level concerns like auth, validation, or security inspection.</p> The edge was an afterthought. A reverse proxy. TLS termination. Maybe Varnish for caching. Maybe a CDN for static assets. It was infrastructure plumbing, not a place where decisions happened.</p> That model is over.</p> Two migrations, one hollowing</h2> Here is the thing I keep coming back to: business logic is migrating in two directions simultaneously.</p> Upward, to the edge. Infrastructure concerns like auth, WAF, and rate limiting now execute at the edge layer, before requests reach any backend. But it goes further than that. Edge Workers run actual application code. Containers deploy at the edge. Server-side rendering happens at edge nodes 50ms from the user, not in a data center 200ms away.</p> Downward, to the client. The browser is no longer dumb. WebAssembly runs near-native code. WebGPU puts the GPU to work on ML inference and image processing. Web Workers handle background computation. Service Workers intercept network requests and serve cached responses offline. CRDTs let the client own its data and sync when it feels like it.</p> The backend is caught in the middle. Squeezed from both sides. And what remains is not a “backend” in any traditional sense. It is a persistence layer. A place where data rests and syncs. The interesting work happens elsewhere.</p> What moved to the edge</h2> Infrastructure concerns</h3> The first wave was obvious. Cross-cutting concerns that every service needed are better handled once, at the point of entry.</p> Authentication.</strong> Validating a JWT does not require application context. The token is self-contained: a signature, an issuer, an expiry, a set of claims. Parse it, verify the signature against a JWKS endpoint, check the expiry, extract the claims, attach them as headers. Done. The backend receives X-User-Id: alice</code> and X-User-Role: admin</code> instead of a raw Bearer token it has to decode itself.</p> This is not hypothetical. Here is what this looks like in practice:</p> agent "auth" { type "auth" grpc address="http://localhost:50051" events "request_headers" timeout-ms 100 failure-mode "closed" max-concurrent-calls 100 } </code></pre> That agent handles JWT, OIDC, SAML, mTLS, and API key validation. Every route behind it gets authentication for free. Every backend service trusts the edge to have done the work. The auth agent crashes? Failure mode is “closed”. Requests stop, but the proxy stays up.</p> Rate limiting.</strong> Token bucket algorithms with per-client keys. The edge layer sees every request before the backend does. It is the natural place to enforce rate limits because it can reject bad traffic before it consumes backend resources. A rejected request at the edge costs microseconds. A rejected request at the backend costs a database query, a connection slot, and whatever work happened before the check.</p> There are two flavors. Local rate limiting uses in-process token buckets. Fast, no network hops, but each edge node tracks its own counters. If you have 10 edge nodes and a limit of 100 requests per second, each node allows 100, so the effective limit is 1,000. For most use cases, this is fine. Abuse does not distribute itself evenly across your infrastructure.</p> Distributed rate limiting uses a shared store (Redis, typically). Accurate across nodes, but adds a network hop per request. The tradeoff is latency versus precision. I default to local rate limiting and switch to distributed only when the use case demands exact global limits, like API billing or token budgets for LLM inference.</p> Security inspection.</strong> WAFs used to be appliances. Expensive, opaque, binary. A request was either blocked or allowed. Modern WAFs use anomaly scoring. Each rule contributes a score, and the total determines the action:</p> Score 0-9: Allow Score 10-24: Log (warning, investigate later) Score 25+: Block </code></pre> This is a fundamentally different model than binary block/allow. It lets you tune aggressively without breaking legitimate traffic. I run 285 detection rules at the edge and process 912K requests per second on clean traffic. That is 30x faster than ModSecurity’s C implementation. The performance gap matters because it means WAF inspection can happen on every request, not just suspicious ones.</p> API validation.</strong> If your API has a JSON Schema, why validate request bodies in your application code? Validate at the edge. Reject malformed requests before they consume a connection, a goroutine, a database transaction. The backend receives only structurally valid payloads.</p> Observability.</strong> Trace context should originate at the edge, not at the application. The edge is where the request enters your system. It is where you assign a trace ID, start the clock, and record the first span. If you originate traces in your application, you miss everything that happened before: TLS negotiation time, WAF processing time, the fact that the request sat in a rate limit queue for 50ms. Starting traces at the edge gives you the full picture.</p> The isolation problem</h3> You cannot put all of this in a monolithic proxy. That is how you end up with nginx and 47 modules where nobody understands the interaction effects. A WAF bug should not take down your routing. A slow auth provider should not block rate limit checks.</p> The answer is process isolation. Thin dataplane, crash-isolated external agents. Each agent runs as a separate process with its own failure domain:</p> ┌──────────────────────────────────────────┐ │ Edge Proxy (thin dataplane) │ │ Routing │ TLS │ Caching │ Load Balancing │ └─────┬──────────┬──────────┬──────────────┘ │ │ │ ▼ ▼ ▼ [WAF] [Auth] [Rate Limit] process process process </code></pre> Each agent gets its own concurrency semaphore. A slow WAF cannot starve auth. Each agent has a circuit breaker. Three failures in 30 seconds and the circuit opens. Each agent has a configurable failure mode, and this is where the design gets interesting:</p> agent "waf" { type "waf" timeout-ms 100 failure-mode "closed" max-concurrent-calls 50 circuit-breaker { failure-threshold 5 success-threshold 2 timeout-seconds 30 } } agent "rate-limit" { type "rate-limit" timeout-ms 50 failure-mode "open" max-concurrent-calls 200 } </code></pre> The WAF fails closed. If it crashes or times out, requests are blocked. You lose availability to preserve security. Rate limiting fails open. If it crashes, requests are allowed. You lose rate enforcement to preserve availability. These are explicit choices per agent, not global defaults. The operator decides which tradeoff to make for each concern, and the decision is visible in the config, not buried in code.</p> Agents return decisions. The proxy merges them. A blocking decision from any agent wins. Otherwise, header mutations accumulate. The model is simple: agents advise, the proxy decides. No agent can override another agent’s block. No agent can force a request through. The proxy owns the final call.</p> This is not a workaround. It is the fundamental design choice. Complex logic lives outside the core, behind process boundaries. The proxy stays small, fast, and boring. The agents handle the interesting work in isolation. A bug in a Lua scripting agent does not corrupt the routing table. A memory leak in the WAF agent does not exhaust the proxy’s memory. The process boundary is the blast radius.</p> Edge Workers: business logic at the edge</h3> Infrastructure concerns were the first wave. The second wave is actual business logic.</p> Cloudflare Workers, Deno Deploy, Fastly Compute, Vercel Edge Functions. These are not just “serverless at the CDN.” They are full compute environments running at edge nodes around the world. V8 isolates spin up in under 5ms. Cold starts are measured in single-digit milliseconds, not seconds. Your code runs 50ms from the user instead of 200ms away in us-east-1.</p> The constraints matter, because they shape what belongs here. Typical Edge Worker limits: 10-50ms CPU time per request (not wall time, actual CPU), 128MB memory, no raw TCP sockets, no persistent file system. You get a request, key-value storage, and the ability to make sub-requests to origins. That is it. These constraints are not bugs. They are what makes sub-millisecond cold starts possible. V8 isolates are cheap because they are small and short-lived.</p> What fits within these constraints is surprisingly broad:</p> API routing and transformation.</strong> A request comes in for /api/v2/users</code>. The edge Worker rewrites it, fans out to two backend services (user profiles from one, preferences from another), merges the responses, and returns a single payload. The backend services are simple data sources. The edge Worker is the API layer.</li> A/B testing and feature flags.</strong> Read the experiment cookie, hash the user ID, assign a variant, route to the right origin or rewrite the response. No round trip to a feature flag service. The decision happens in microseconds at the node closest to the user.</li> Personalization.</strong> Look up the user’s segment in KV storage, inject the right content block, set cache headers accordingly. The backend generated all variants at build time. The edge picks the right one per request.</li> Server-side rendering.</strong> Render HTML at the edge node closest to the user. Frameworks like Next.js and Remix already support this. React Server Components run at the edge. The “server” in server-side rendering is not your server. It is an edge node in 300 locations.</li> Authentication and session management.</strong> Validate tokens, refresh sessions, set secure cookies. The auth flow never touches your origin. Cloudflare Workers KV or Durable Objects store session state at the edge.</li> </ul> The pattern: compute that depends on request context but not on deep application state moves to the edge. If you can do it with a request, a key-value lookup, and a response, it probably belongs here. If it needs a complex database query or a multi-step transaction, it does not.</p> Containers at the edge</h3> Edge Workers hit a ceiling when you need persistent connections, large memory, or long-running processes. For those workloads, containers at the edge.</p> Fly.io, Railway, and Lambda@Edge deploy containers or full processes to edge locations worldwide. Your application runs with real file systems, TCP connections, and whatever runtime you need. But it runs close to users, not in a centralized data center. Latency drops from 200ms to 20ms.</p> The interesting problem is data gravity. Compute is easy to distribute. Data is not. If your container runs in Tokyo but your database is in Frankfurt, you have not solved the latency problem. You have moved it from the user-to-server hop to the server-to-database hop. The solutions are still maturing: read replicas at the edge (Turso, Neon), embedded databases that sync (LiteFS, libSQL), and eventually-consistent stores designed for multi-region (DynamoDB Global Tables, CockroachDB).</p> This model makes sense when compute and data can be co-located:</p> Regional APIs</strong> that comply with data residency requirements. Run the container and the database replica in the same region. GDPR data stays in the EU. Japanese user data stays in Japan.</li> Real-time applications</strong> where 200ms round trips kill the experience. Collaborative editing, multiplayer, live dashboards. A WebSocket server 20ms away feels instant. One 200ms away feels sluggish.</li> Stateful edge compute</strong> where you need more than a request/response cycle. Background processing, scheduled jobs, long-running connections.</li> </ul> The line between “edge” and “origin” blurs. If your container runs in 30 regions and handles requests locally with a local database replica, is that an edge deployment or a distributed backend? The distinction stops mattering. What matters is that the compute and the data are close to the user.</p> What moved to the client</h2> The other half of the migration goes downward. The browser is not the thin client it used to be.</p> WebAssembly</h3> WASM runs at near-native speed in every modern browser. Not “fast for JavaScript.” Actually fast. Compiled from Rust, C++, Go, or any language with an LLVM backend. Sandboxed, portable, deterministic.</p> What this enables:</p> Image and video processing</strong> in the browser. No upload to a server, no round trip, no privacy concern. The pixels never leave the device.</li> Document parsing and transformation.</strong> PDF rendering, spreadsheet computation, file format conversion. Libraries compiled to WASM and running client-side.</li> Cryptographic operations.</strong> End-to-end encryption where the server never sees plaintext. Key derivation, signing, verification, all in the browser.</li> Compute offloading from the backend.</strong> This is the one that changes how you think about server sizing.</li> Full relational databases in the browser.</strong> This is the one that changes architectures.</li> </ul> I build on this pattern directly. Cyanea</a>, a bioinformatics platform, uses WASM to offload computation from the backend to the client’s browser. Sequence analysis, structure visualization, dataset filtering, these are CPU-intensive operations that traditionally require beefy server infrastructure. Instead, the computation runs right there on the researcher’s device. The backend stays thin: it stores datasets and coordinates collaboration, but the heavy lifting happens in the browser. This means I can run the platform on a modest server and still deliver real computational capability, because the “compute fleet” is the users’ own machines.</p> Archipelag</a> takes the same idea in a different direction. It is a distributed compute platform where users contribute their browser’s idle compute power via WASM. The workloads compile to WASM modules, ship to participating browsers, execute in the sandbox, and return results. The browser is not just a client consuming a service, it is a compute node in a distributed system. The WASM sandbox is what makes this safe: untrusted code runs in a constrained environment with no access to the host file system, network, or memory beyond what is explicitly granted.</p> SQLite compiled to WASM (via projects like sql.js, wa-sqlite, or the official SQLite WASM build) gives the browser a real relational database. Not a key-value store. Not IndexedDB’s awkward object store API. Actual SQL with joins, indexes, transactions, and triggers. Backed by the Origin Private File System (OPFS) for persistence, it survives page reloads and browser restarts.</p> The implications are significant. Your application can run complex queries locally. Filter, sort, aggregate, full-text search. All instant, all offline. The server becomes a sync endpoint. It ships a database snapshot down and accepts change sets back. The client does the querying. The server does the storing.</p> This pattern scales down elegantly. A note-taking app with SQLite-in-WASM needs no backend API for reads. A project management tool can filter and search 10,000 tasks without a network request. A CMS authoring interface can work fully offline and sync when the author reconnects. The read path is local. The write path syncs eventually.</p> WASI (WebAssembly System Interface) extends this further. It gives WASM modules controlled access to file systems, clocks, and network sockets outside the browser. WASM becomes a universal runtime: the same binary runs in the browser, at the edge (Cloudflare Workers use WASM under the hood), and on bare metal. Write once, deploy to every layer of the stack.</p> The pattern: anything that is CPU-bound, privacy-sensitive, or latency-sensitive is a candidate for client-side WASM. If the computation does not need server-side state, it should not round-trip to a server.</p> WebGPU</h3> WebGPU landed in Chrome in 2023, in Firefox and Safari shortly after, and it changes the math on what the client can compute. This is not WebGL with a new name. WebGL exposes a graphics pipeline. WebGPU exposes compute shaders. Direct, general-purpose GPU computation from JavaScript or WASM.</p> The immediate application is ML inference. Run a language model, an image classifier, or a recommendation engine on the user’s GPU. No server call, no API cost per token, no latency. The model weights download once (cached by the browser) and run locally. Privacy by default, because the data never leaves the device.</p> This is not theoretical. Stable Diffusion generates images in the browser via WebGPU. Small language models (Phi-2, Gemma 2B, Llama 3.2 1B) run at usable speeds on consumer hardware. MediaPipe runs pose detection, face tracking, and hand gesture recognition in real time. The trajectory is clear: models get smaller through distillation and quantization, consumer GPUs get faster, and the gap between “cloud inference” and “local inference” narrows every quarter.</p> Both Cyanea and Archipelag use WebGPU alongside WASM. In Cyanea, WebGPU accelerates molecular visualization and large-scale dataset operations, the kind of parallel computation that bioinformatics demands but that would be prohibitively expensive to run server-side for every user session. In Archipelag, WebGPU-capable nodes can take on GPU-accelerated workloads from the compute pool, turning a user’s idle GPU into a productive resource. The combination of WASM for general compute and WebGPU for parallel workloads gives the browser a compute profile that would have required dedicated server hardware five years ago.</p> But inference is not the only use case. WebGPU handles any parallel computation: physics simulations for games, signal processing for audio applications, particle systems for data visualization, and large-scale matrix operations. Anything you would reach for CUDA or Metal for on native can now run in the browser. The compute budget of the client just increased by orders of magnitude.</p> Web Workers and Service Workers</h3> Web Workers give you background threads. Heavy computation does not block the UI. Parse a large file, run a simulation, index a search corpus. All off the main thread, all without janking the interface.</p> Service Workers sit between the browser and the network. They intercept every fetch request and decide what to do: serve from cache, go to network, do both and race them. This enables:</p> Offline-first applications.</strong> The app works without a network connection. Data syncs when connectivity returns.</li> Background sync.</strong> Queue mutations while offline, replay them when online.</li> Push notifications.</strong> Wake the app without the user having it open.</li> Intelligent caching.</strong> Cache API responses, serve stale data while revalidating, pre-fetch resources the user is likely to need.</li> </ul> The Service Worker is the client-side equivalent of the edge proxy. It intercepts, caches, validates, and routes. It makes the client self-sufficient.</p> Local-first and CRDTs</h3> Here is where it gets interesting. If the client has compute (WASM, WebGPU, Web Workers) and storage (IndexedDB, OPFS) and offline capability (Service Workers), why does it need a server at all?</p> CRDTs (Conflict-free Replicated Data Types) answer the consistency question. Multiple clients can edit the same data independently, offline, with no coordination. When they reconnect, their changes merge automatically without conflicts. No server-mediated locking. No “last write wins” data loss. Mathematical guarantees that concurrent edits converge to the same state.</p> The architecture:</p> Client A (offline) Client B (offline) │ │ ├── Local edits ├── Local edits │ (CRDT ops) │ (CRDT ops) │ │ └──────┐ ┌────────┘ ▼ ▼ ┌──────────────┐ │ Sync service │ (thin, stateless) │ (persistence │ │ + relay) │ └──────────────┘ </code></pre> The sync service is not a backend. It stores operations and relays them between clients. It does not run business logic. It does not validate (the CRDT handles consistency). It does not transform (the merge function is built into the data type). It is a persistence layer with a WebSocket attached.</p> I build systems like this. The concrete model: a document is a flat HashMap<EntityId, Entity></code> where each entity holds CRDT-typed fields. The field types determine how concurrent edits merge:</p> CRDT type</th> Merge behavior</th> Use case</th></tr></thead> LwwRegister<T></td> Last writer wins (by timestamp)</td> Simple values: name, status, URL</td></tr> GrowOnlySet<T></td> Union of both sides</td> Tags, labels, immutable references</td></tr> ObservedRemoveSet<T></td> Add wins over concurrent remove</td> Collaborator lists, mutable collections</td></tr> MaxRegister</td> Higher value wins</td> Version counters, progress indicators</td></tr> MinRegister</td> Lower value wins</td> Earliest timestamps, priority values</td></tr> </tbody></table> Each field carries a hybrid logical clock (HLC) timestamp. The HLC combines physical time with a logical counter, so causality is preserved even when wall clocks drift. Two clients edit the same field at the “same” time? The HLC ordering is deterministic. Both clients converge to the same value without coordination.</p> The merge function has three properties that make this work: it is associative (grouping does not matter), commutative (order does not matter), and idempotent (applying the same operation twice has no additional effect). These are not implementation details. They are the mathematical foundation that makes server-free consistency possible. You can sync operations in any order, from any number of clients, through any number of intermediate relays, and every replica converges to the same state.</p> The client owns its data. The server is optional. When the server exists, it persists operations and relays them. It does not arbitrate, transform, or validate beyond authentication.</p> This is not a niche pattern for collaborative text editors. Any application where users create and modify data can benefit. Notes, task managers, project planning tools, CMS authoring, form builders. The question is not “should this be local-first?” The question is “does this need a server, and if so, for what?”</p> What the backend becomes</h2> If the edge handles infrastructure concerns and business logic that depends on request context, and the client handles computation, rendering, and local state, what is left for the backend?</p> A persistence layer.</p> The backend becomes the place where data rests between sessions and syncs between devices. Not an application server. A persistence layer.</p> Consider the spectrum of what “backend” looks like now:</p> Static sites.</strong> This site is an example. raskell.io is built with Zola. Markdown files compile to HTML at build time and deploy to edge CDN nodes. No application server. No database. No runtime process. The “backend” is a git repository and a CI pipeline. Content lives as files. Serving happens at the edge. The total monthly infrastructure cost is the price of a domain name.</p> This is not limited to blogs. Documentation sites, marketing pages, product landing pages, e-commerce storefronts with pre-rendered product pages. Any content that changes at author-time rather than request-time can be static. The headless CMS (Contentful, Sanity, Strapi, or just a git repo) publishes content. The static site generator builds HTML. The CDN serves it. The “backend” runs at build time, not at request time.</p> I take this further than most. All of my projects are CDN-first, even the ones with dedicated backends. The principle: if the backend goes down, the user should still see something useful. The static layer is the safety net.</p> Kurumi</a>, a local-first second brain app, is the purest expression of this. It is a Progressive Web App served entirely from CDN edge nodes with Service Workers handling offline capability. There is no backend server. Notes sync between devices through CRDTs when connectivity exists, but the app works fully offline. The entire “infrastructure” is a static deployment and an optional sync relay.</p> But the CDN-first pattern also applies to applications that have real backends. Cyanea has a Phoenix/Elixir backend that manages datasets, user accounts, and collaboration. But the public-facing surface, the landing pages, category pages, trending spaces and protocols and datasets, is statically generated. The backend exports JSON snapshots of its database objects on a timed interval. A static site generator picks up those snapshots and rebuilds the public pages: what labs are active, which protocols are trending, which datasets were recently published. The result is a set of HTML pages sitting on a CDN that stay current without depending on the backend being up at the moment a visitor arrives.</p> ┌──────────────────┐ JSON export ┌──────────────┐ │ Cyanea Backend │ ──── (interval) ────> │ Static Site │ │ (Phoenix/BEAM) │ │ Generator │ │ │ │ (Zola) │ │ - datasets │ │ │ │ - protocols │ │ → CDN edge │ │ - labs │ │ nodes │ │ - spaces │ │ │ └──────────────────┘ └──────────────┘ │ │ │ dynamic app static pages ▼ ▼ app.cyanea.bio cyanea.bio (logged-in users, (public, always up, real-time features) fast, no backend dependency) </code></pre> This is a genuinely resilient architecture. The backend can be down for maintenance, mid-deploy, or experiencing load, and the public site keeps serving. The static pages are never stale by more than one generation interval. For a site where “trending this week” is sufficient freshness, that interval can be hours. The CDN handles traffic spikes that would overwhelm a backend. The backend handles the dynamic work that requires real-time data.</p> Thin persistence APIs.</strong> For applications with dynamic data, the backend shrinks to a database with an API in front of it. Accept writes, serve reads, enforce schema constraints. GraphQL or REST over Postgres. No rendering. No business logic beyond data integrity. The API exists so that clients and edge workers have somewhere to store and retrieve state.</p> The interesting shift: even the persistence API is getting thinner. Services like Supabase, PlanetScale, and Turso expose the database directly over HTTP or WebSockets with built-in auth. Your “backend” becomes a hosted database with row-level security policies. No application code at all.</p> Sync relays.</strong> For local-first applications, the backend is even simpler. Accept CRDT operations from clients, persist them to durable storage, fan them out to other connected clients via WebSocket. No merge logic (the CRDT handles that). No transformation. No validation beyond authentication. The relay does not understand the data. It stores and forwards.</p> Event logs.</strong> Append-only storage. Clients sync by replaying events from their last known position. The log is the source of truth. Everything else (search indexes, analytics dashboards, recommendation models) is a materialized view built asynchronously. The hot path is the append. The read path is the replay. Both are simple.</p> Batch processors.</strong> The one place where traditional backend compute survives: jobs that require access to the full dataset. Analytics aggregation, report generation, search index building, ML model training. These run on schedules or triggers, not in the request path. They read from the event log or the database, compute, and write results back. The user never waits for them.</p> The common thread: the backend does not touch the hot path. User requests hit the edge and the client. The backend runs in the background, on its own schedule, when no one is waiting.</p> The architecture that makes this work</h2> Pushing logic to the edge and the client is not free. Both environments have constraints, and ignoring them is how you build fragile systems.</p> At the edge: bounded resources</h3> Every operation at the edge needs explicit limits. No open-ended computations, no unbounded queues, no surprise behavior. This is not just good practice. It is existential. The edge proxy sits between the internet and your infrastructure. If it behaves unpredictably, everything behind it suffers.</p> Concretely:</p> Resource</th> Bound</th> Why</th></tr></thead> Agent concurrency</td> Per-agent semaphore (default: 100)</td> Prevents noisy neighbor between agents</td></tr> Agent timeout</td> 100ms default</td> Prevents latency cascade</td></tr> Connection pool</td> Explicit max (default: 10K)</td> Prevents file descriptor exhaustion</td></tr> Request body</td> Streaming, not buffered</td> Prevents memory exhaustion</td></tr> Route cache</td> LRU with size limit</td> Prevents unbounded growth</td></tr> Rate limit queues</td> Bounded with max delay</td> Prevents request pile-up</td></tr> </tbody></table> If you cannot articulate the bound for every resource your edge system uses, you do not have an architecture. You have an accident waiting for load.</p> On the client: isolation and sandboxing</h3> The client has different constraints. Battery life, memory pressure, the user closing the tab at any moment.</p> WASM runs in a sandbox. No file system access, no network access, no shared memory (unless explicitly granted). This is the security model that makes client-side compute viable. Untrusted code (your own, running on someone else’s device) cannot escape the sandbox.</p> Web Workers run in separate threads with message-passing. No shared mutable state. No locks. No data races. The isolation is enforced by the runtime, not by programmer discipline.</p> Service Workers have a lifecycle managed by the browser. They can be terminated at any time to save resources. Your offline logic must handle graceful shutdown. This means: durable state in IndexedDB, idempotent sync operations, no in-memory state that cannot be reconstructed.</p> CRDTs provide consistency guarantees without coordination. But they are not magic. They consume memory (tombstones for deleted items, version vectors for causal ordering). They need garbage collection. They need careful schema design because not every data model maps cleanly to CRDT primitives. A counter works. A last-writer-wins register works. A rich text document with formatting, comments, and embedded media requires careful thought.</p> The trust boundary</h3> Here is the part most edge-computing articles skip: trust.</p> If the edge handles auth, the backend trusts the edge to have done auth correctly. If the client handles business logic, the server trusts the client to have computed correctly. These are real trust boundaries with real failure modes.</p> At the edge, trust is earned through:</p> Failure isolation.</strong> Agent crashes do not take down the proxy. Bad config is validated before activation.</li> Observability.</strong> Every decision is logged, metered, and traceable. If the WAF blocked a request, you can see exactly which rules fired and why.</li> Bounded behavior.</strong> No surprise modes. Every resource has explicit limits. Every failure mode is configured, not assumed.</li> </ul> On the client, trust is conditional:</p> Never trust the client for security decisions.</strong> Validate at the edge or the backend. Client-side checks are UX, not security.</li> Trust the client for its own data.</strong> If the user is editing their own document, the client is authoritative. CRDTs handle consistency. The server persists, it does not arbitrate.</li> Verify at the boundary.</strong> When client data syncs to the server, validate schema and authorization. Trust the merge, verify the input.</li> </ul> When not to do this</h2> Not everything belongs at the edge or on the client. Here is what stays in the backend:</p> Multi-service transactions.</strong> If an operation needs to read from three databases, check inventory, charge a payment, and send a notification, that is a backend workflow. Distributed transactions need coordination, and coordination needs a central authority.</p> Heavy data joins.</strong> If your query joins six tables with complex filters and aggregations, it runs next to the database, not at an edge node 200ms away from the data.</p> Regulatory requirements.</strong> Some industries mandate that data processing happens in specific locations, on specific infrastructure, with specific audit trails. Edge deployment may not satisfy these constraints.</p> Small teams with simple needs.</strong> If you have one backend, ten users, and no latency problems, this architecture is overhead. A Django app behind nginx is fine. Optimize when you have a reason to optimize, not before.</p> The edge handles cross-cutting concerns and request-context computation. The client handles local state and user-facing compute. The backend handles coordination, persistence, and anything that needs the full dataset. Know which is which.</p> Where this is going</h2> Five years ago, the stack was: browser (thin) renders server-generated HTML, backend (fat) runs everything, database stores state. The mental model was request/response, and the backend was the center of gravity.</p> The stack now:</p> ┌─────────────────────────────────────────────────────┐ │ Client │ │ WASM │ WebGPU │ Web Workers │ Service Workers │ CRDT │ │ (compute, render, offline, local state) │ └────────────────────┬────────────────────────────────┘ │ ┌────────────────────┴────────────────────────────────┐ │ Edge │ │ Proxy │ Workers │ Containers │ KV │ Durable Objects │ │ (auth, WAF, routing, SSR, API aggregation, policy) │ └────────────────────┬────────────────────────────────┘ │ ┌────────────────────┴────────────────────────────────┐ │ Backend │ │ Database │ Sync relay │ Event log │ Batch processing │ │ (persistence, coordination, async compute) │ └─────────────────────────────────────────────────────┘ </code></pre> The client is fat. The edge is fat. The backend is thin. The center of gravity moved to both ends simultaneously.</p> Every year, this accelerates. Models get smaller and run on consumer GPUs. WASM runtimes get faster and gain more system APIs through WASI. Edge platforms add durable storage, queues, and cron triggers. CRDTs mature from academic curiosities to production libraries. SQLite-in-the-browser goes from experiment to default architecture for offline-capable apps.</p> The backend will not disappear. Data needs to live somewhere durable, and cross-device sync needs a relay. Coordination problems need a central authority. Batch processing needs access to the full dataset. But the backend’s role is narrowing to exactly these things. It is becoming infrastructure, not application. Plumbing, not logic.</p> I find myself building systems where the most interesting engineering happens at the boundaries. A reverse proxy (Zentinel</a>) that inspects 912K requests per second through 285 WAF rules, authenticates with sub-millisecond latency, and routes with crash-isolated agents. A bioinformatics platform (Cyanea</a>) where the browser runs the computation and the backend exports JSON for statically generated pages. A distributed compute platform (Archipelag</a>) where users’ browsers are the compute fleet via WASM and WebGPU. A note-taking app (Kurumi</a>) that works fully offline with CRDTs and never touches a server for reads. Between all of them, a database, a sync relay, or just a CDN. Necessary and boring.</p> The backend is not dead. It is just not where the interesting work happens anymore.</p> References and further reading</h2> Edge platforms and proxies</h3> Cloudflare Workers</a> - V8 isolate-based edge compute</li> Deno Deploy</a> - Edge runtime built on the Deno JavaScript runtime</li> Fastly Compute</a> - Wasm-based edge compute platform</li> Vercel Edge Functions</a> - Edge compute integrated with Next.js</li> fly.io</a> - Container-based edge deployment platform</li> Pingora</a> - Cloudflare’s Rust framework for programmable proxies</li> Zentinel</a> - Security-first reverse proxy with crash-isolated agents</li> </ul> Client-side compute</h3> WebAssembly</a> - Portable binary instruction format for the web</li> WASI</a> - WebAssembly System Interface for running Wasm outside the browser</li> WebGPU specification</a> - W3C standard for GPU compute in the browser</li> MediaPipe</a> - ML inference framework running client-side via Wasm and WebGPU</li> SQLite Wasm</a> - Official SQLite build targeting WebAssembly</li> sql.js</a> - SQLite compiled to Wasm via Emscripten</li> Origin Private File System</a> - MDN reference for persistent browser storage</li> Service Worker API</a> - MDN reference for offline-capable web apps</li> Web Workers API</a> - MDN reference for background threads in the browser</li> </ul> CRDTs and local-first</h3> CRDTs: The Hard Parts</a> - Martin Kleppmann’s talk on practical CRDT challenges</li> Local-first software</a> - Ink and Switch research paper on local-first architectures</li> Automerge</a> - CRDT library for collaborative applications</li> Yjs</a> - High-performance CRDT framework for the web</li> A comprehensive study of CRDTs</a> - Shapiro et al., the foundational survey paper</li> </ul> Databases at the edge</h3> Turso</a> - SQLite-compatible database with edge replicas</li> Neon</a> - Serverless PostgreSQL with branching</li> LiteFS</a> - Distributed SQLite replication by Fly.io</li> CockroachDB</a> - Distributed SQL database designed for multi-region</li> Supabase</a> - Open-source Firebase alternative built on PostgreSQL</li> </ul> Projects referenced</h3> Cyanea</a> - Bioinformatics platform using Wasm and WebGPU for client-side compute</li> Archipelag</a> - Distributed compute platform with browser-based Wasm nodes</li> Kurumi</a> - Local-first second brain app with CRDT sync</li> Conflux</a> - Schema-aware CRDT engine for deterministic merge</li> </ul> Looking back on 2025 Unknown — Wed, 31 Dec 2025 00:00:00 +0000 I spent part of this year on the shores of Okinawa. The water there is something else entirely, this impossible azure that shifts to turquoise in the shallows, so clear you can see the coral formations from the surface. I found myself thinking about systems while I was there, the way you do when you’re floating in salt water with nothing pressing to attend to.</p> Between swims, I read Tim Berners-Lee’s “This is for everyone.” I’ve been building web software for over a decade now, and I thought I understood what the web was. But reading TBL’s words while watching that reef ecosystem do its thing, thousands of species in constant exchange, no central coordinator, just emergent complexity from simple rules, something shifted in how I saw it all.</p> The web TBL imagined was supposed to work like that reef. A commons. Many small nodes, each doing their own thing, connected through open protocols. Information flowing freely. The beauty of it wasn’t in any single node but in the connections between them, the way the whole became more than the sum of its parts. The same principle that makes a reef resilient makes a network powerful: diversity, redundancy, local adaptation.</p> What we built instead looks more like industrial aquaculture. Five platforms. Algorithmic monoculture. Content optimized for engagement metrics rather than usefulness. We took a system designed for decentralization and built the most centralized information infrastructure in human history.</p> I keep thinking about how that happened. The web itself never changed. HTTP still works the same way, HTML still does what it always did. What changed was the economics. Publishing became free, but being found</em> became expensive. The platforms positioned themselves as the gatekeepers of attention, and suddenly you couldn’t reach people without paying the toll, whether in ad spend or in algorithmic compliance or in the slow erosion of doing whatever it took to game SEO.</p> The thing about monocultures is they’re efficient right up until they’re not. A reef can lose a species and adapt. A monoculture gets one disease and collapses. We’ve been watching the web’s monoculture show stress fractures for years. The enshittification of platforms, the SEO content farms drowning out signal with noise, the way social media stopped being social and started being a feed of engagement-optimized content from strangers.</p> Then 2025 happened, and AI started breaking things in interesting ways.</p> The obvious take is that AI makes the content problem worse. And superficially, that’s true. If you thought SEO spam was bad before, wait until you see what happens when generating ten thousand pages of plausible-sounding garbage costs essentially nothing. The content farms went into overdrive. Social platforms filled with synthetic engagement.</p> But here’s the thing I keep coming back to: maybe that’s the fever that breaks the infection.</p> The old economics of the web depended on a particular scarcity. Human attention is finite, and the platforms controlled access to it. You wanted eyeballs, you played their game. SEO worked because Google was the gateway and you could optimize for what Google wanted. Platform distribution mattered because that’s where the people were.</p> AI disrupts this in ways that I think are genuinely interesting. When an AI assistant can synthesize information from across the web and deliver it directly to the user, the value of ranking first on Google diminishes. Why click through to a content farm when the answer is already in front of you? When AI agents can find and surface relevant content directly, you don’t need to be on the platform where the eyeballs gather. The middleman’s leverage starts to evaporate.</p> And crucially: when everyone can generate infinite content at zero marginal cost, content quantity becomes worthless. What matters is provenance. Accuracy. Usefulness. The things that are actually hard. The things that require a human perspective, or at least require being right</em> in ways that matter.</p> I find myself unexpectedly optimistic about what comes next.</p> If AI breaks the distribution stranglehold that platforms have, the economics of the web could flip in interesting directions. The old model needed scale because reaching people was expensive. But if AI handles discovery, finding relevant content and bringing it to users, then maybe you don’t need scale anymore. Maybe small becomes viable again.</p> Think about what this means concretely. A static site costs nearly nothing to run. No databases to scale, no servers to babysit, just files sitting on edge nodes around the world. If you don’t need to capture user data for ad-driven personalization, you don’t need the complexity of the surveillance stack. If you don’t need platform distribution, you don’t need to play platform games.</p> There’s another piece to this that I think most people are missing: edge computing changes what personalization can mean. The conventional wisdom is that personalization requires surveillance, that you need to know everything about a user to show them relevant content. But that’s only true if personalization happens in a centralized database somewhere. If personalization happens at the edge, at the moment of request, you can adapt content to context without ever needing to know who the user is. The edge function doesn’t need a profile. It just needs to know what was asked for and what context it’s being asked in.</p> This is the architecture I keep thinking about: static content at the origin, edge functions that adapt it anonymously, AI agents that find and surface it based on actual relevance rather than SEO gaming. No surveillance required. No platform dependency. No scaling costs that force you into growth-at-all-costs mode.</p> It looks more like a reef than a fish farm.</p> I don’t want to oversell this. The transition, if it happens, won’t be clean. The platforms aren’t going to quietly cede control. The incentives that built the current web are still operating. And AI itself could go in directions that make things worse rather than better. There are plenty of dystopian paths from here.</p> But when I think about what I want to build toward, it’s that reef model. Many small, specialized nodes. Interconnected through open protocols. Resilient because distributed. Sustainable because the economics work at small scale.</p> This site is part of that bet. Static content, no tracking, no platform dependencies. The tools I’m working on (Zentinel, Sango, Ushio) are all about making edge infrastructure more accessible, making it easier to build and operate systems that are distributed and independent.</p> 2025 was the year AI started breaking the old model. I don’t know exactly what grows in its place. But floating in that Okinawan water, watching the reef do what reefs do, I got a sense of what healthy systems look like. Diverse. Interconnected. Resilient. Not optimized for any single metric, but somehow working anyway.</p> That’s what I’m betting on.</p> References and further reading</h2> Tim Berners-Lee</a> - Creator of the World Wide Web and author of “This is for everyone”</li> Enshittification</a> - Cory Doctorow’s term for the pattern of platform decay</li> IndieWeb</a> - Community building the independent web with open standards</li> Zola</a> - Static site generator used to build this site</li> Zentinel</a> - Security-first reverse proxy for the open web</li> Sango</a> - Edge diagnostics CLI for TLS, HTTP, and security header analysis</li> Ushio</a> - Deterministic edge traffic replay tool</li> </ul> Mise ate my Makefile Unknown — Sun, 14 Dec 2025 00:00:00 +0000 I maintain around forty repositories across four GitHub organizations. Zentinel</a> alone accounts for over thirty: the core proxy, a Rust SDK, and a growing collection of agents for WAF inspection, auth, rate limiting, GraphQL security, and a dozen other edge concerns. Archipelag</a> spans an Elixir coordinator, a Rust node agent, Python and TypeScript SDKs, mobile agents in Kotlin and Swift, and infrastructure-as-code repos. Cyanea</a> is Elixir with Rust NIFs and a separate Rust bioinformatics library. Then there are the standalone tools: Conflux</a> (Rust CRDT engine), Sango</a> (Rust edge diagnostics CLI), Shiioo</a> (Rust agentic orchestrator), Vela</a> (Rust bare-metal deployment), Refrakt</a> (Gleam web framework), Kurumi</a> (Svelte local-first app), and this site you are reading (Zola).</p> The languages span Rust, Elixir, Gleam, Python, TypeScript, Kotlin, Swift, and whatever shell scripts accumulated over the years. Every project needs a toolchain. Most need task automation. All of them need to be approachable for a contributor who clones the repo for the first time.</p> The Makefile approach was breaking down. So was everything else I tried.</p> What was failing</h2> The standard setup for most of my Rust projects was a Makefile with targets for build</code>, test</code>, clippy</code>, fmt</code>, and release</code>. Simple enough for one repo. The problem surfaces when you maintain thirty of them.</p> GNU Make and BSD Make disagree on syntax in ways that cause silent failures. A Makefile that works on my Linux CI runner breaks on a contributor’s macOS laptop because of a conditional or a shell invocation difference. The fix is always “use GNU make,” but that means documenting it, adding a check, and fielding issues from people who forget.</p> Worse, Makefiles cannot declare tool dependencies. A Rust project needs a specific Rust version, maybe protoc</code> for gRPC, maybe cargo-watch</code> for development convenience. The Makefile assumes these tools exist. When they do not, the developer gets a cryptic error five minutes into their first build.</p> So projects accumulated scaffolding:</p> .rust-version .tool-versions Makefile scripts/setup.sh scripts/ci.sh scripts/release.sh .envrc </code></pre> Six files to express what amounts to: “this project uses Rust 1.83, needs protoc, and has five things you can run.” Multiply that by forty repos and you have a maintenance surface that nobody wants to touch. The scripts/</code> folder in particular had a way of growing silently. Someone adds a helper. Someone else copies it from another project with modifications. Six months later you have three slightly different versions of the same release script across three orgs.</p> The Elixir projects had it worse. Elixir needs Erlang/OTP at a specific version, then Elixir itself at a matching version, then Node for asset compilation in Phoenix, then possibly Rust for NIFs (Cyanea compiles Rust bioinformatics code into the BEAM release). Four tool dependencies before you write a line of application code. asdf</code> handled the version management, but slowly and without task automation, so you still needed a Makefile on top.</p> Why not Nix</h2> I gave Nix a serious try. The promise is appealing: declare your entire development environment in a single file, get reproducible builds, never worry about system state. The Nix shell concept is genuinely elegant.</p> In practice, the cost was too high for my use case. Nix’s learning curve is steep even for experienced engineers. The language is its own thing. The documentation assumes you already understand the Nix store model. When something breaks, the error messages point at derivation hashes, not at the thing you actually did wrong.</p> The bigger issue was onboarding. If a contributor wants to fix a typo in a Zentinel agent’s README, asking them to install Nix and understand flakes is a non-starter. The tool that manages your development environment should not itself become a project you have to learn. Nix solves a harder problem than I have. I do not need bit-for-bit reproducible builds across machines. I need “install Rust 1.83 and run the tests.”</p> Why not asdf</h2> asdf was my default for years. It handled the version management problem well enough. The plugin system meant I could manage Rust, Elixir, Erlang, Node, and Python versions with a single .tool-versions</code> file.</p> Three things pushed me away.</p> First, speed. asdf is shell scripts. Every invocation pays the cost of sourcing plugins, resolving versions, and shimming binaries. On a fast machine you barely notice. On CI, where you run asdf install</code> in a fresh environment, the overhead adds up. Mise is a compiled Rust binary. It is meaningfully faster at both installation and version resolution.</p> Second, no task automation. asdf manages tool versions. That is all it does. You still need Make or a scripts folder for project tasks. That means two tools, two configuration surfaces, two things to document.</p> Third, plugin quality varied. The core plugins for Node and Ruby were solid. Plugins for less mainstream tools could be stale, broken, or missing. Mise started as an asdf-compatible rewrite and inherited the plugin ecosystem, but its built-in backends for common tools (Rust, Node, Python, Go, Erlang, Elixir) are faster and more reliable than shelling out to plugins.</p> What mise actually does</h2> Mise</a> is a single Rust binary that combines tool version management and task running into one configuration file per project. It does asdf’s job and Make’s job in a single tool.</p> Here is this site’s configuration. The entire thing:</p> # mise.toml (raskell.io) [tools] zola = "0.19" [env] _.file = ".env" [tasks.serve] description = "Start the Zola development server" run = "zola serve" [tasks.build] description = "Build the site for production" run = "zola build" [tasks.check] description = "Check the site for errors without building" run = "zola check" [tasks.new] description = "Create a new article" run = """ #!/usr/bin/env bash if [ -z "$1" ]; then echo "Usage: mise run new <article-slug>" exit 1 fi SLUG="$1" DATE=$(date +%Y-%m-%d) FILE="content/articles/${SLUG}.md" cat > "$FILE" << ARTICLE +++ title = "" date = ${DATE} description = "" [taxonomies] tags = [] categories = [] [extra] author = "Raffael" +++ ARTICLE echo "Created $FILE" """ </code></pre> One file. Declares the tool (Zola 0.19), loads environment variables, and defines every task a contributor needs. mise install</code> sets up the toolchain. mise tasks</code> shows what is available. mise run serve</code> starts the dev server. No Makefile. No scripts folder. No documentation page explaining how to get Zola at the right version.</p> For a Rust project like Shiioo</a> (the agentic orchestrator), the configuration is larger but follows the same pattern:</p> # .mise.toml (shiioo) [tools] rust = "latest" [env] RUST_LOG = "info" RUST_BACKTRACE = "1" _.path = ["./target/release", "./target/debug"] [tasks.build] description = "Build all crates in release mode" run = "cargo build --release" [tasks.test] description = "Run all tests" run = "cargo test" [tasks.clippy] description = "Run clippy lints" run = "cargo clippy --all-targets -- -D warnings" [tasks.fmt] description = "Format code with rustfmt" run = "cargo fmt --all" [tasks.ci] description = "CI pipeline: format check, clippy, test" depends = ["fmt-check", "clippy", "test"] [tasks.dev] description = "Full development build and run" depends = ["fmt", "check", "test"] run = "cargo run -p shiioo-server" </code></pre> The depends</code> key is where mise replaces the one thing Make was genuinely good at: task dependency ordering. mise run ci</code> runs format checking, then clippy, then tests, in sequence. If clippy fails, tests do not run. It is not as expressive as Make’s file-based dependency graph, but for project automation tasks (as opposed to build tasks, which cargo or mix handle), it covers what I actually need.</p> For a multi-language project like Cyanea, the value is even clearer. The Elixir app needs Erlang, Elixir, Node, and Rust. One [tools]</code> section pins all four. One mise install</code> gets a contributor from zero to a working environment. Without mise, that setup involved installing asdf, adding four plugins, running asdf install</code>, then installing direnv for environment variables, then reading the Makefile to figure out how to run things. With mise, it is two commands: mise install</code> and mise run dev</code>.</p> The cross-project pattern</h2> The real payoff is not in any single project. It is the consistency across all of them.</p> Every repo in every org follows the same contract:</p> Clone the repo</li> Run mise install</code></li> Run mise tasks</code> to see what is available</li> Run mise run dev</code> or mise run test</code></li> </ol> That is it. Whether the project is a Rust reverse proxy with thirty modules, an Elixir Phoenix application with LiveView and a NATS integration, a Gleam web framework, or a static site built with Zola, the entry point is identical. The person cloning the repo does not need to know which build system the project uses internally. They do not need to read a CONTRIBUTING.md to find out whether it is make test</code> or cargo test</code> or mix test</code>. It is always mise run test</code>.</p> This matters more than it sounds. When you maintain projects across four orgs and multiple languages, the cognitive overhead per context switch is the actual bottleneck. I work on Zentinel (Rust) in the morning, switch to Archipelag (Elixir) after lunch, then fix something on this site (Zola) in the evening. Without a consistent interface, each switch means recalling which project uses which conventions. With mise, the interface is always the same. The implementation behind mise run test</code> differs (cargo, mix, zola check), but I do not care about that. I type the same command and the right thing happens.</p> For new contributors, the effect is more pronounced. Zentinel’s agent ecosystem has over twenty Rust repos. A contributor who submits a PR to the WAF agent and then wants to help with the auth agent does not need to learn a new setup process. Same structure, same task names, same workflow. The consistency compounds.</p> What mise handles that Make does not</h2> Environment variables.</strong> Mise loads environment from the config file or from .env</code> files, scoped to the project directory. When I cd</code> into a project, the right environment is active. When I leave, it deactivates. No direnv, no .envrc</code>, no source .env</code> in every shell session.</p> Tool installation.</strong> mise install</code> in a fresh clone gets every tool the project needs at the exact specified version. Make cannot do this. Make assumes the tools exist. That assumption breaks on new machines, in CI, and for every new contributor.</p> Task discovery.</strong> mise tasks</code> lists every available task with its description. Make has make help</code> patterns, but those are conventions, not built-in features. With mise, discoverability is the default.</p> File-based tasks.</strong> Any executable file in .mise/tasks/</code> becomes a task automatically. No registration, no config entry needed. For tasks that outgrow a one-liner in TOML but do not warrant a standalone script in scripts/</code>, this is the right middle ground. The task is discoverable through mise tasks</code> but lives as a normal shell script you can test independently.</p> What breaks</h2> Mise is not perfect. Honest assessment after running it across forty repos:</p> Dynamic dependencies.</strong> Make can express “rebuild this if that file changed.” Mise tasks are imperative: they run or they do not. If you need file-level dependency tracking, you still need a build system (cargo, mix, webpack). Mise orchestrates tasks. It does not replace the build tool.</p> Ecosystem maturity.</strong> Mise is younger than Make and asdf. The documentation is good but not exhaustive. Some features (like hooks and watch mode) are recent additions. The pace of development is fast, which means features arrive quickly but occasionally change between minor versions.</p> Team familiarity.</strong> Make is universal. Every engineer has encountered a Makefile. Mise is still relatively unknown. Introducing it to a team requires a short pitch, but the pitch is easy: “it is Make plus asdf in one tool, configured in TOML.”</p> Complex shell tasks.</strong> When a task grows beyond a few lines, the inline TOML string syntax gets awkward. The workaround is file-based tasks in .mise/tasks/</code>, which works well but means the task definition lives in two places (TOML for metadata and task list, shell file for implementation).</p> The migration</h2> If you are moving an existing project, here is the approach I settled on after migrating across all four orgs:</p> Add a mise.toml</code> (or .mise.toml</code>) at the project root. Start with just [tools]</code> to declare the required versions.</li> Move the most-used Make targets to [tasks]</code> one at a time. Keep the Makefile around until everything is ported.</li> Add [env]</code> entries to replace .envrc</code> or .env.example</code> files.</li> Move standalone scripts from scripts/</code> to .mise/tasks/</code> as file-based tasks.</li> Delete the Makefile last.</li> </ol> Do not try to migrate everything at once. Start with the three tasks developers use daily (usually dev</code>, test</code>, and build</code>). The rest can move incrementally. I also settled on a few naming conventions that help across projects: use clear verb-noun prefixes like db-reset</code>, cache-clear</code>, test-unit</code>. Consistent naming makes task discovery predictable even before you run mise tasks</code>.</p> The bottom line</h2> Mise is not a revolutionary tool. It does not do anything that was previously impossible. You could always install the right Rust version, write a Makefile, set up direnv, and maintain a scripts folder. What mise does is collapse all of that into a single file that is readable, portable, and consistent.</p> The compound effect is what matters. Forty repositories, four organizations, six languages, one pattern. Clone, install, run. No guessing which build system this particular project uses. No debugging a Makefile that works on Linux but breaks on macOS. No explaining to a contributor that they need asdf plus three plugins plus direnv plus GNU make before they can run the tests.</p> Every new project starts with a mise.toml</code>. Setup takes two commands instead of a page of instructions. Contributors do not message me asking how to run things. They run mise tasks</code> and figure it out.</p> That is the tool working.</p> References and further reading</h2> mise</a> - Official documentation and installation guide</li> mise source code</a> - GitHub repository and issue tracker</li> asdf</a> - The version manager mise was originally inspired by</li> Nix</a> - Reproducible builds and development environments</li> GNU Make</a> - The build tool mise replaces for task automation</li> TOML specification</a> - The configuration format mise uses</li> direnv</a> - Environment variable manager that mise’s [env]</code> section replaces</li> Shiioo</a> - Real-world mise configuration referenced in this article</li> mise-hx</a> - Example of a custom mise plugin (for the hx Haskell toolchain)</li> </ul> Disk space maintenance on Void Linux Unknown — Wed, 01 May 2024 00:00:00 +0000 Monday morning surprise</h2> As I spent most time doing stuff with my computer rather than configuring my beloved Linux distribution, Void Linux, I have developed the tendency to not really bother about Void at all until something crucial becomes unusable. After almost two years of having switched from Arch to Void, I have actually never encountered any major problem and felt I had made the right decision.</p> I checked my disk usage out of curiosity if the 250GB solid-state disk would be enough. And there came the surprise:</p> $ df -H Filesystem Size Used Avail Use% Mounted on devtmpfs 8.4G 0 8.4G 0% /dev tmpfs 8.4G 1.9M 8.4G 1% /dev/shm tmpfs 8.4G 1.4M 8.4G 1% /run /dev/nvme0n1p3 138G 117G 21G 85% / efivarfs 158k 85k 69k 56% /sys/firmware/efi/efivars cgroup 8.4G 0 8.4G 0% /sys/fs/cgroup /dev/nvme0n1p4 366G 34G 332G 10% /home /dev/nvme0n1p1 536M 152k 536M 1% /boot/efi tmpfs 8.4G 25k 8.4G 1% /tmp </code></pre> My root partition was full, way too full in my opinion. Did I miss something? Is Void not what I was looking for after all? I don’t enjoy baby sitting my OS du jour</em>.</p> The painless solution</h2> After a quick Brave search, I ended up finding what I was looking for. Some kind fellow software engineer from China didn’t shy away to make a blog post about his journey when he faced the very same problem. Out of annoyance of having to deal with that, I copy-pasted as quickly as possible, not minding what kind of side-effects I might run into, these three commands.</p> 1. Cleaning the package cache</h3> All the knowledge I was lacking was to be found with the man page of xbps-remove</code>.</p> # xbps-remove -yO </code></pre> The man page</a> of xbps-remove</code> tells us the -O</code> parameter takes care of cleaning the cache directory removing obsolete binary packages.</em> Obsolete binary packages? Good riddance! I was surprised this to learn that solely this step freed up almost half of my used root partition disk space.</p> 2. Removing orphaned packages</h3> # xbps-remove -yo </code></pre> Here the same man page</a> tells us that the -o</code> parameter takes care of removing installed package orphans that were installed automatically (as dependencies) and are not currently dependencies of any installed package.</em> As before, good riddance!</p> 3. Purging old, unused kernels</h3> This one is interesting. While I knew about the circumstance that the people behind Void had developed their own package management ecosystem, I hadn’t fully realized there were other utilities that came along with the upstream Void installation which were there for me to manage my beloved OS. So, apparently, one of these is a shell script</a> name vkpurge</code>, I must assume as a short name for Void's Kernel purging</code> tool. I like this type of naming heavily implying its functionality.</p> # vkpurge rm all </code></pre> It performed as expected. Old kernel files (and modules?) were indeed purged and freed up even more disk space. I should add that this step is optional as it is always useful to have some old kernels at hand when things hit the fan (which for me, they haven’t in a very, very long time).</p> Result</h2> I couldn’t be happier.</p> $ df -H Filesystem Size Used Avail Use% Mounted on ... /dev/nvme0n1p3 138G 45G 93G 33% / ... </code></pre> Renewal of faith</h2> Overall, why am I even writing this if some other fellow engineer already figured this out? Simply, because I would therefore be able to explain why I have enjoyed my journey with Void as my go-to Linux distribution. It keeps things simple. Some well-documented utilities. As simple that a simple Brave search suffices to find the answer to my problems.</p> This very aspect of Void is worthwhile highlighing. I remember more arcane Linux distributions that had me in their grip in figuring things out. Many Googles searches were necessary and even more trial and errors attempts to get simple things fixed.</p> Now back to my Monday morning.</p> References and further reading</h2> Void Linux</a> - Official project site</li> Void Linux Handbook: XBPS</a> - Official documentation for the XBPS package manager</li> xbps-remove(1) man page</a> - Manual page for package removal and cache cleaning</li> vkpurge source</a> - Shell script for purging old kernels on Void Linux</li> Void Linux FAQ</a> - Common questions about running and maintaining Void</li> </ul> ^{1</sup> Painting in header image is “Seaside” by Aleksandr Deyneka</p> </div>} All beginning is Haskell Unknown — Mon, 06 Mar 2023 00:00:00 +0000 This site is called raskell.io. That is not an accident.</p> I started learning Haskell because I liked mathematics and someone told me there was a programming language built on top of it. Not “inspired by” in the loose way that every language claims some mathematical foundation. Actually built on lambda calculus, category theory, and type theory, in a way where the math is not decoration but structure.</p> What I did not expect was how thoroughly it would rewire the way I think about building software. Not because Haskell is the best language for every task. It is not, and I write far more Rust than Haskell these days. But because Haskell teaches you to think about programs in a way that makes you better at everything else.</p> What Haskell actually teaches you</h2> Most introductions to Haskell talk about pure functions, immutability, and monads. They are not wrong, but they miss the point. The point is not any single feature. It is how those features combine into a way of thinking about programs as compositions of well-typed transformations.</p> In an imperative language, you think about sequences of steps. Do this, then that, then check a condition, then loop. The program is a recipe. In Haskell, you think about transformations. What goes in, what comes out, what shape does the data have at each stage. The program is a pipeline.</p> This sounds abstract until you see it in practice. Suppose you need to process a list of user records: filter out inactive users, extract their email addresses, and normalize them to lowercase.</p> In an imperative style, you write a loop with conditions and mutations. In Haskell:</p> activeEmails :: [User] -> [Email] activeEmails = map (normalize . email) . filter isActive </code></pre> One line. Read it right to left: filter active users, then map over the result, extracting and normalizing emails. The type signature tells you what goes in ([User]</code>) and what comes out ([Email]</code>). No mutation. No intermediate variables. No place for off-by-one errors or null pointer exceptions.</p> The type signature is not just documentation. It is a contract enforced by the compiler. If isActive</code> expects a User</code> and you pass it a String</code>, the program will not compile. If normalize</code> returns an Email</code> but you try to use it as a String</code>, the program will not compile. The compiler is your first reviewer, and it is tireless.</p> Types as design tools</h2> The deeper lesson is that types are not just error catchers. They are design tools.</p> When I design a system in Haskell, I start with the types. What are the entities? What are the relationships? What transformations are valid? The type system forces you to be precise about these questions before you write any logic. This precision surfaces design problems early, when they are cheap to fix.</p> Consider modeling a document that can be in one of several states:</p> data Document = Draft { content :: Text, author :: UserId } | UnderReview { content :: Text, author :: UserId, reviewer :: UserId } | Published { content :: Text, author :: UserId, publishedAt :: UTCTime } </code></pre> This is an algebraic data type. Each variant carries exactly the data that makes sense for that state. A Draft</code> has no reviewer. A Published</code> document has a timestamp. You cannot accidentally access a reviewer on a draft because the type system will not let you. The invalid state is unrepresentable.</p> This pattern, making illegal states unrepresentable, is perhaps the most valuable idea I took from Haskell. I use it in Rust constantly. Rust’s enum</code> with associated data is directly descended from Haskell’s algebraic data types, and the same design principle applies: encode your invariants in the type system and let the compiler enforce them.</p> The monad is not the point</h2> Every Haskell introduction eventually gets to monads, usually with a metaphor involving burritos or boxes. I will skip the metaphor.</p> A monad is a pattern for sequencing computations that carry some context. The IO</code> monad carries the context of interacting with the outside world. The Maybe</code> monad carries the context of possible failure. The State</code> monad carries the context of mutable state. The pattern is the same in each case: take a value in a context, apply a function that produces a new value in a context, get back a combined context.</p> lookupUser :: UserId -> IO (Maybe User) lookupUser uid = do conn <- getConnection result <- query conn "SELECT * FROM users WHERE id = ?" [uid] return (listToMaybe result) </code></pre> The IO</code> monad here sequences database operations. The Maybe</code> handles the case where no user is found. The types tell you both things at a glance: this function does I/O and might not return a result.</p> The point of monads is not that they are clever. The point is that they make effects explicit and composable. In most languages, a function can do I/O, throw exceptions, mutate global state, or launch missiles, and you cannot tell from its signature. In Haskell, the type signature tells you exactly what effects a function can have. Int -> Int</code> is pure. Int -> IO Int</code> does I/O. Int -> Maybe Int</code> can fail. The information is right there, enforced by the compiler.</p> This discipline, making effects explicit, changed how I design APIs even in languages that do not enforce it. When I write a Rust function that returns Result<T, E></code>, I am using the same pattern: making failure explicit in the type rather than hiding it behind an exception. Rust learned this from Haskell, and so did I.</p> Why I am still building Haskell tooling</h2> If Haskell taught me so much, why do I mostly write Rust?</p> The honest answer: Haskell’s ecosystem has gaps. The language itself is excellent. GHC is one of the most sophisticated compilers ever built. The type system is unmatched in its expressiveness among production languages. But the surrounding infrastructure, the package management, the build tooling, the deployment story, has not kept pace.</p> Dependency management in Haskell is fragmented. Cabal and Stack coexist with overlapping but incompatible approaches. Build times are long. Cross-compilation is painful. Setting up a Haskell development environment from scratch still involves more friction than it should in 2026.</p> This is why hx exists. hx is a Haskell toolchain CLI that I am building in Rust. The choice of implementation language is deliberate. Haskell’s tooling problems are partly caused by tooling that is itself written in Haskell, creating bootstrap problems and long compile times for the tools themselves. A Rust binary starts instantly, compiles to a single static executable, and cross-compiles trivially. The tool should not have the same dependencies as the thing it manages.</p> hx is distributed through mise</a> (naturally), as well as through AUR, Homebrew, Scoop, and Chocolatey. The goal is that setting up a Haskell project should be as frictionless as setting up a Rust project: one command to install the toolchain, one command to build.</p> On the other end of the spectrum, bhc (the Basel Haskell Compiler) is an experiment in taking Haskell in a direction GHC was never designed for: compiling Haskell for low-latency runtimes without a garbage collector. The target is workloads like tensor pipelines and real-time systems where GC pauses are not acceptable. bhc is early and ambitious, but it comes from the same conviction: Haskell’s ideas deserve better infrastructure than they currently have.</p> The Haskell in my Rust</h2> I write Rust the way Haskell taught me to think.</p> Rust’s ownership model is not the same as Haskell’s purity, but it serves a similar purpose: it forces you to think about data flow explicitly. In Haskell, you cannot mutate a value because the language will not let you. In Rust, you can mutate, but the borrow checker forces you to be explicit about who owns the data and who can see it. Both languages make you think before you write.</p> Conflux</a>, my CRDT engine, uses algebraic data types for its merge semantics. Each CRDT field type (LwwRegister, GrowOnlySet, ObservedRemoveSet) is an enum variant with associated data, exactly the pattern I described above. The merge function is associative, commutative, and idempotent. These are mathematical properties that I learned to care about from Haskell, where such properties are often expressed as type class laws.</p> Zentinel</a>, the reverse proxy, uses Rust’s type system to enforce that WAF decisions are handled in the correct pipeline stage. An AgentDecision</code> is either Allow</code>, Block</code>, or Modify</code>, and the proxy’s merge logic ensures that a Block</code> from any agent cannot be overridden. The pattern is a monoid (decisions combine associatively with Block</code> as the absorbing element), though nobody would call it that in the Rust codebase. The concept came from Haskell. The implementation is pure Rust.</p> Even Shiioo</a>, the agentic orchestrator, uses Haskell-influenced patterns. DAG workflows are compositions of typed transformations. Events are algebraic data types with exhaustive pattern matching. The event-sourcing model treats state as a fold over an event stream. foldl</code> in Haskell, Iterator::fold</code> in Rust. Same idea, different syntax.</p> Why “raskell”</h2> The name is a portmanteau. Raffael plus Haskell. I chose it because Haskell is where my engineering thinking started to take its current shape. Not the first language I learned, but the first one that changed how I think about all the others.</p> I do not believe you need to write Haskell to benefit from Haskell. But I believe that learning it, really learning it, not just reading about monads but building something real with algebraic data types and type classes and higher-order functions, will make you a better engineer in whatever language you actually use.</p> All beginning is Haskell. The rest is implementation.</p> References and further reading</h2> Learning Haskell</h3> Haskell Language</a> - Official site with documentation and community links</li> Learn You a Haskell for Great Good!</a> - Approachable illustrated introduction</li> Real World Haskell</a> - Practical Haskell for production use</li> Haskell Wiki</a> - Community-maintained reference and tutorials</li> Typeclassopedia</a> - Comprehensive guide to Haskell’s type class hierarchy</li> </ul> Type systems and theory</h3> Haskell Curry</a> - The logician the language is named after</li> Lambda calculus</a> - Alonzo Church’s formal system underlying Haskell</li> Hindley-Milner type system</a> - The type inference algorithm at the core of Haskell and ML</li> Making illegal states unrepresentable</a> - Yaron Minsky’s influential talk on using types for correctness</li> Algebraic data types</a> - Haskell wiki reference on sum and product types</li> </ul> Monads and effects</h3> Philip Wadler, “Monads for functional programming”</a> - The foundational paper on monads in programming</li> All About Monads</a> - Haskell wiki guide to monadic programming</li> </ul> Haskell tooling</h3> GHC</a> - The Glasgow Haskell Compiler</li> Cabal</a> - Haskell’s build and package system</li> Stack</a> - Alternative build tool with curated package sets</li> mise-hx</a> - mise plugin for the hx Haskell toolchain CLI</li> </ul> Projects referenced</h3> Conflux</a> - CRDT engine using algebraic data types for merge semantics</li> Zentinel</a> - Reverse proxy with monoid-based decision merging</li> Shiioo</a> - Agentic orchestrator using event-sourced state folds</li> </ul> My OpenBSD journey: Getting it virtualized with libvirt (1) Unknown — Mon, 06 Feb 2023 00:00:00 +0000 Void Linux as my daily driver</h2> Around six months ago, I decided to ditch my long in the tooth Arch-based setup on my belovest Thinkpad X1 Carbon. I’ve been very loyal over the years, and almost came to belive that Arch will be a constant in my adult life. While I kept up with upcoming technologies, I somehow lost track of the ever so diversifying landscape of Linux distributions. It took me a while of constantly coming across a generically named reference to what seemed to be yet another Linux distribution. That outwardly generic sounding name, Void Linux</a>, kept poking my curiosity by supposedly feeling like Arch Linux in the old days, while sharing some substantial DNA with the BSD operating systems. Yet, that’s another story I might tell another day, but to remain brief, the BSDs, in particular the infamous OpenBSD with its quite infamous lead developer Theo De Raadt, always were what I considered the endgame. The holy grail of Unix operating systems, so did I think over the decades, FreeBSD, NetBSD and OpenBSD, have always been on my personal radar and I felt I had to earn the intellectual capacity to be able to properly put them at use one day. Last year, when I made the (almost painless) switch from Arch Linux to Void Linux, the simplicity and especially the barebone experience of Void reignited the fascination and the admiration I always had for the BSD operating systems and their philosophy.</p> While I could write (and definitely will in the near future) about how my journey onto Void Linux and how it has been so far, I preferred to write down, in some like diary, and document every step on how one can approach and ultimately use OpenBSD</a> in 2023. Big disclaimer, I’ve yet to install OpenBSD on some baremetal server I ordered some days ago, but dabbled around in the meantime with OS-level virtualization in order to get it running. That’s what brings me to libvirt</em> and my surprise to learn that I wouldn’t need some full-fledged virtualization solution like the ones offered by VirtualBox or VMWare to efficiently run a virtualized OpenBSD machine. So, ok, let’s recap, so far we got the following bill of materials:</p> Void Linux as the host system</li> OpenBSD to be virtualized on that host system</li> libvirt as the glue that makes virtualization feel like black magic</li> </ul> From Void to OpenBSD</h3> Before I tell you more about the history of how things went down while setting up OpenBSD, let me give you some basic notions about both Void Linux, being one out of many Linux distributions for the sake of simplicity representing them all as it ended up being my distribution of choice, and OpenBSD. As already mentioned earlier, Void Linux and OpenBSD are both Unix-like operating systems, but they feature enough differences to make them noteworthy. Here are a few similarities and differences between the two:</p> What is similar:</p> Both are free and open-source operating systems.</li> Both use a package manager for software management. Void Linux uses XBPS, while OpenBSD uses pkg_add.</li> Both prioritize security and stability in their development and design.</li> Both feature a version control based package repository, meaning that changes in build definition are managed by pull requests from users.</li> </ul> What is different:</p> License: Void Linux is licensed under the MIT License, while OpenBSD is licensed under the ISC License.</li> Philosophy: OpenBSD prioritizes security and privacy, while Void Linux prioritizes simplicity and modularity.</li> Package Management: Void Linux uses the XBPS binary package manager, while OpenBSD uses pkg_add (also binary). OpenBSD additionally has a ports system for building from source.</li> Package Repository: Void Linux has a large and diverse repository, while OpenBSD has a smaller and more curated repository.</li> Init System: Void Linux uses runit as its init system, while OpenBSD uses rc. None uses the infamous systemd init system.</li> </ul> What is OpenBSD in a nutshell</h2> OpenBSD is a free and open-source operating system that focuses on security, standardization, and robustness. It is based on the Berkeley Software Distribution (BSD) Unix operating system and is developed by a global community of volunteers. OpenBSD aims to provide a secure platform for both personal and enterprise use by implementing strong security features, including access control mechanisms, encryption, and auditing.</p> Theo de Raadt</a> is the founder and lead developer of OpenBSD. His main objective with OpenBSD is to create a secure operating system that is free from backdoors, vulnerabilities, and other security weaknesses. He is committed to auditing the source code of the operating system and third-party software included with it, to identify and remove any potential security risks. De Raadt is also dedicated to improving the overall quality of the codebase and ensuring compatibility with a wide range of hardware and software.</p> What makes OpenBSD really special and stand out is that is developed a suite of tools that got adopted by other OSs like Linux, macOS or even Windows. One of the most famous instances of such adoption is the now de facto standard openssh suite. It actually emerged from within the development circle of the OpenBSD project. OpenBSD also implemented a wide range of OS features that are by now considered staples among other OSs, things like Linux-based OS-level containerization done via the means of cgroups, something that OpenBSD already pioneered and solved with a different spin many years before Linux with pledge and unveil. Go check out Why OpenBSD rocks</a> to get a feel what makes OpenBSD so unique and interesting.</p> Virtualization with libvirt</h3> So now, let’s get back to our virtualization endeavour where we would like to virtualize OpenBSD on a Void Linux installation. If you happen to be using another Linux distribution, most of the individual steps would be very similar. That brings me to the next technology we should explain a bit more here, and that is libvirt.</p> libvirt</a> is an open-source virtualization management library that provides a simple and unified API for managing virtualization technologies, including KVM, QEMU, Xen, and others. It aims to simplify the process of creating, managing, and migrating virtual machines, storage, and networks, and to make it easier for administrators to manage virtual environments.</p> To virtualize an operating system like OpenBSD with libvirt, you need to follow these steps:</p> Install libvirt and the virtualization technology you want to use, such as KVM.</li> Download the OpenBSD iso file and place it in a location accessible by libvirt.</li> Create a new virtual machine in libvirt with the OpenBSD ISO as the installation media. This can be done through the command line or using a graphical user interface such as virt-manager.</li> Configure the virtual machine, including the amount of memory, CPU, and disk space, to meet the requirements of OpenBSD.</li> Start the virtual machine and install OpenBSD as you would on a physical machine.</li> Once the installation is complete, you can configure the virtual network, storage, and other settings as required.</li> Finally, you can use the libvirt API or the command line to manage and control the virtual machine, including starting, stopping, migrating, and snapshotting.</li> </ol> Step-by-step guide</h3> Let’s first install the libvirt</code> package and some related packages which we need in order to connect via VNC. The VNC will provide us with the possibility to use the graphical interface of the running OpenBSD instance.</p> $ sudo xbps-install -S dbus qemu libvirt virt-manager virt-viewer tigervnc </code></pre> Now, we need to add our user, in my case raskell</code>, to the libvirt</code> group which got simultanously created with the installation of libvirt.</p> $ sudo usermod -aG libvirt raskell </code></pre> OpenBSD features a release cycle of six months. We would need to update our system every six month to keep up with the latest packages. During a given release, only security and bug fix patches are applied to the curated packages maintained by pkg_add. Therefore, in February 2023, we’re using the OpenBSD 7.2</a> release version. As I’m living in Switzerland, I chose to pull the iso image from a Swiss mirror, in this case from mirror.ungleich.ch/pub/OpenBSD</code></a> (check what mirror is closest to you to get the best download rate).</p> # cd /var/lib/libvirt/boot/ # sudo wget https://mirror.ungleich.ch/pub/OpenBSD/7.2/amd64/install72.iso --2023-01-12 20:48:15-- https://mirror.ungleich.ch/pub/OpenBSD/7.2/amd64/install72.iso Resolving mirror.ungleich.ch (mirror.ungleich.ch)... 2a0a:e5c0:2:2:400:c8ff:fe68:bef3, 185.203.114.135 Connecting to mirror.ungleich.ch (mirror.ungleich.ch)|2a0a:e5c0:2:2:400:c8ff:fe68:bef3|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 583352320 (556M) [application/octet-stream] Saving to: ‘install72.iso.1’ install72.iso.1 100%[====================================================================>] 556.33M 7.11MB/s in 77s 2023-01-12 20:49:33 (7.19 MB/s) - ‘install72.iso.1’ saved [583352320/583352320] sudo wget https://mirror.ungleich.ch/pub/OpenBSD/7.2/amd64/SHA256 --2023-01-12 20:47:38-- https://mirror.ungleich.ch/pub/OpenBSD/7.2/amd64/SHA256 Resolving mirror.ungleich.ch (mirror.ungleich.ch)... 2a0a:e5c0:2:2:400:c8ff:fe68:bef3, 185.203.114.135 Connecting to mirror.ungleich.ch (mirror.ungleich.ch)|2a0a:e5c0:2:2:400:c8ff:fe68:bef3|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1992 (1.9K) [application/octet-stream] Saving to: ‘SHA256.1’ SHA256.1 100%[====================================================================>] 1.95K --.-KB/s in 0s 2023-01-12 20:47:39 (742 MB/s) - ‘SHA256.1’ saved [1992/1992] # grep install63.iso SHA256 > /tmp/x # sha256sum -c /tmp/x # rm /tmp/x </code></pre> Before we can start the virtualization server and get running our OpenBSD instance, we need to define the configuraiton on how to virtualize and ultimately boot the system with. This is done with virt-install</code>. Noteworthy here is that we QEMU</a> as our emulation solution of choice, we allocate up to 4GB of RAM and 4 CPU cores to the machine.</p> $ sudo virt-install \ --name=openbsd \ --virt-type=qemu \ --memory=2048,maxmemory=4096 \ --vcpus=2,maxvcpus=4 \ --cpu host \ --os-variant=openbsd7.0 \ --cdrom=/var/lib/libvirt/boot/install72.iso \ --network=bridge=virbr0,model=virtio \ --graphics=vnc \ --disk path=/var/lib/libvirt/images/openbsd.qcow2,size=40,bus=virtio,format=qcow2 </code></pre> Once, it is up and running, we can use a vnc solution to connect to the running machine. In this case, I chose virsh</a> to do the job. virsh is a command-line interface tool for managing virtualization environments created with libvirt. It allows us to manage virtual machines, storage pools, and network interfaces, as well as other virtualization components, from the command line.</p> To establish a VNC connection with a running libvirt virtualized OpenBSD instance, you can use the following steps:</p> Start the virtual machine in libvirt: You can start the virtual machine using the virsh command virsh start <vm-name></code></strong>, where <vm-name></code></strong> is the name of the virtual machine you want to start.</li> Find the VNC display: Once the virtual machine is running, you can find the VNC display number for the virtual machine using the virsh command virsh vncdisplay <vm-name></code></strong>.</li> Connect to the VNC display: You can connect to the VNC display using a VNC client, such as vncviewer</code></strong>, and specify the IP address of the host running the virtual machine and the VNC display number. For example, if the host’s IP address is 192.168.0.100</code></strong> and the VNC display number is :0</code></strong>, the command to connect would be vncviewer 192.168.0.100:0</code></strong>.</li> Authenticate to the VNC server: You may need to enter a password to authenticate to the VNC server. The password is set when the virtual machine is created in libvirt.</li> </ol> With these steps, you can establish a VNC connection with a running libvirt virtualized OpenBSD instance and interact with the virtual machine’s graphical user interface.</p> $ virsh dumpxml openbsd | grep vnc <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'> </code></pre> Like I did, most of you would like to interact with a graphical interface such as with X11. For that, we yet another tool, a so-called VNC viewer. A very simple implementation of such a vnc viewer is tigervnc</a> (simply install it with $ sudo xbps-install -S tigervnc</code>).</p> $ sudo virsh --connect qemu:///system start openbsd Domain 'openbsd' started </code></pre> References and further reading</h2> OpenBSD</h3> OpenBSD</a> - Official project site</li> OpenBSD FAQ</a> - Comprehensive installation and usage guide</li> Why OpenBSD Rocks</a> - Collection of OpenBSD innovations and features</li> OpenSSH</a> - The SSH suite that originated from the OpenBSD project</li> Theo de Raadt</a> - OpenBSD founder and lead developer</li> pledge(2)</a> - OpenBSD’s system call for restricting process capabilities</li> unveil(2)</a> - OpenBSD’s system call for restricting filesystem visibility</li> </ul> Void Linux</h3> Void Linux</a> - Official project site</li> Void Linux Handbook</a> - Official documentation</li> XBPS package manager</a> - Void’s binary package management system</li> </ul> Virtualization</h3> libvirt</a> - Virtualization management library and API</li> QEMU</a> - Open-source machine emulator and virtualizer</li> virsh(1)</a> - Command-line interface for managing libvirt guests</li> TigerVNC</a> - VNC implementation for remote desktop access</li> </ul> Guides</h3> [Guide] How to setup QEMU/KVM emulation on Void Linux</a> - Community guide on r/voidlinux</li> KVM virt-install: Install OpenBSD as Guest Operating System</a> - Step-by-step KVM installation guide</li> Auto-install OpenBSD on QEMU</a> - Automated OpenBSD installation on QEMU</li> </ul> Hello and outlook Unknown — Tue, 20 Sep 2022 00:00:00 +0000 Hello world</h2> This is the first post on raskell.io. My name is Raffael. I build software, mostly in the space of platform automation, edge infrastructure, and applied security. I also have a long-running interest in open-source software, operating systems, and the kind of hardware you find in recycling bins and give a second life to.</p> This site exists to document real work and share patterns that other engineers can use. Not thought leadership. Not personal branding. Just notes from the workshop.</p> What to expect</h2> At the time of writing, I planned to cover:</p> Linux and BSD systems, particularly Void Linux</a> and OpenBSD</a></li> Open-source tooling and infrastructure</li> Platform automation and operability</li> Whatever hard problem I happened to be stuck on</li> </ul> Looking back from 2026, the scope grew to include edge systems, AI-assisted engineering, and a few deep dives I did not expect to write. The thread that held it together was always the same: systems under pressure, and the tools that keep them running.</p> References</h2> Void Linux</a></li> OpenBSD</a></li> raskell.io source</a></li> </ul>

Raskell

Why We Built a Haskell Package Manager in Rust

The Last Programming Language Might Not Be for Humans

Notes from RSAC 2026

What Zentinel Is Really Optimizing For

How I Work These Days

Archipelag.io Is in Open Beta: Here's Why I Built It

How AI Makes Bare Metal Viable Again

Process supervision</h3> Vela does not just start your app and walk away. It supervises it. If a process crashes, Vela detects the exit (via non-blocking try_wait</code> on the child process handle), logs it, and restarts from the stored launch configuration:</p>

References and further reading</h2>

Edge Systems Are the New Backend

What moved to the edge</h2>

What moved to the client</h2> The other half of the migration goes downward. The browser is not the thin client it used to be.</p>

Looking back on 2025

Mise ate my Makefile

The cross-project pattern</h2> The real payoff is not in any single project. It is the consistency across all of them.</p> Every repo in every org follows the same contract:</p> Clone the repo</li> Run mise install</code></li> Run mise tasks</code> to see what is available</li>

Disk space maintenance on Void Linux

1. Cleaning the package cache</h3> All the knowledge I was lacking was to be found with the man page of xbps-remove</code>.</p>

Result</h2> I couldn’t be happier.</p>

All beginning is Haskell

References and further reading</h2>

My OpenBSD journey: Getting it virtualized with libvirt (1)

Step-by-step guide</h3> Let’s first install the libvirt</code> package and some related packages which we need in order to connect via VNC. The VNC will provide us with the possibility to use the graphical interface of the running OpenBSD instance.</p>

References and further reading</h2>

What moved to the client</h2>
The other half of the migration goes downward. The browser is not the thin client it used to be.</p>