Pausing Work on Playbooks AI
Pausing Work on Playbooks: Lessons from Building a Natural Language Programming System for AI Agents
By Amol Kelkar | April 2026
I'm pausing active development on Playbooks, the open-source semantic programming system for AI agents I've been building since mid-2022.
The most useful lesson from this project is also the most ironic one. Playbooks argued that agent specifications should be forward compatible, meaning old programs should get better on their own when the underlying model improves. I still think that principle is correct, and it's exactly why a simpler approach took the ecosystem. A harness with less scaffolding benefits more from model improvement than one with more. Playbooks built the most elaborate natural language execution infrastructure of any agent framework, then articulated the principle explaining why much of that infrastructure would eventually be unnecessary.
So I'm stepping back. Not deleting, not walking away from the ideas. Several of them still have no equivalent anywhere in the ecosystem, and I expect to come back to some of them.
The Origin
Playbooks began in June 2022, when I started experimenting with GPT-3 to build customer support agents that needed structured behavior and human-like flexibility at the same time. Imperative code couldn't handle the nuance. Neither could visual workflow builders like DialogFlow CX or story-based systems like Rasa. I got stuck on a question:
What if English itself could be the programming language, and LLMs could be the CPUs that execute it?
That became Playbooks: a structured markdown language for specifying agent behavior, a semantic intermediate representation (PBAsm) that compiled those specs into executable instructions, and a runtime that managed execution with the determinism of traditional software and the flexibility of LLMs. The first iteration went open source in early 2023. Playbooks was then rebuilt from scratch four times in total, the last of those starting in September 2024, and that fourth codebase shipped 16 releases through v0.7.4 in February 2026.
Why Pause Now?
The generalist agent plus skills approach has won the ecosystem. The paradigm that won is the one Playbooks helped pioneer: natural language specifications that define agent behavior, interpreted by an LLM at runtime. What changed is the representation. Playbooks as a specific format has been supplanted by skills, Anthropic's simpler, free-form alternative that is now the de facto standard.
Skills are free-form natural language specifications that a harness loads and an LLM follows. No compiler, no assembly language, no execution validation. A skill can encode knowledge, process, caveats, or edge case guidance. Playbooks encodes process: steps, control flow, triggers. Both are markdown, and the structural difference between them is small. But skills' free-form expressiveness, arriving alongside capable coding agents like Claude Code, Codex, Cursor, and Windsurf, won the ecosystem.
Anthropic announced skills on October 16, 2025. From that point the question stopped being which specification method is better and became which one has momentum.
Key Technical Ideas
Several ideas Playbooks explored have since appeared in the broader ecosystem, sometimes in different form, sometimes independently invented. They build on a long history of work in multi-agent systems, conversational AI, and business rule engines, but their combination in the LLM agent context was, as far as I can tell, new. These are the ones I want to keep on the record.
Natural Language as a Programming Language (June 2022 onward)
This is a real programming language, not prompt engineering. It has variables, control flow, function calls, return values, and a type system, all expressed in structured natural language and compiled to a semantic intermediate representation.
When I started in June 2022, the industry's idea of "agent development" was chaining API calls in Python. LangGraph, CrewAI, and AutoGen didn't exist yet; all three launched between May 2023 and early 2024. OpenAI function calling wouldn't arrive until June 2023, and Anthropic tool use wouldn't enter beta until April 2024. By the time I wrote Realizing the Dream of Natural Language Programming in October 2025, the idea that you could write $order_id:str = Ask user for their order ID and have it compile to typed, executable instructions with yield points was still new in the LLM agent context. A 29-line Playbooks program replaced 272+ lines of LangGraph for an equivalent customer support agent, though take line-count comparisons across paradigms with a grain of salt, since the natural language spec pushes complexity into the runtime and the LLM.
Anthropic's skills achieve something similar, natural language that agents follow, without the compilation step or the formal semantics. They trade verifiability for expressiveness and simplicity. For most use cases that's a good trade, especially as LLMs keep getting better at following instructions.
Software Engineering Paradigm for Agents
Playbooks brought ordinary software engineering concepts to agent development. Agents are classes. Playbooks are methods. Public playbooks are public methods, exported playbooks are mixins, agent state is instance variables, triggers are events specified with decorators. The call stack is a real call stack.
The industry standardized on a different abstraction: tools. An LLM has a bag of tools it can call. That works for simple agents, but it's flat. No encapsulation, no hierarchy, no lifecycle management, no scoping. Class-based modeling of agents goes back decades in multi-agent systems research (JADE, SPADE, and others), but applying it to LLM agents with natural language methods was, as far as I'm aware, unique to Playbooks.
Anthropic's December 2024 "Building Effective Agents" post emphasized workflows and orchestration patterns, but the underlying primitive stayed the tool call. The industry has slowly come around to seeing tools as an arbitrary construct, and as of April 2026 "code mode" agents are becoming the norm, using JIT coding instead of tool calling.
VSCode Debugging of Natural Language Programs (May 2025)
Playbooks shipped a full VSCode debugger integration in May 2025 (v0.4.0). You can set breakpoints on natural language steps, inspect the call stack, examine variables, and step through execution of a natural language program.
{
"type": "playbooks",
"request": "launch",
"name": "Launch Playbook",
"program": "${file}",
"stopOnEntry": true
}
LangGraph Studio (August 2024) offered graph visualization and state editing, but that debugs a graph, not natural language. Microsoft's AI Toolkit Agent Inspector (February 2026) brought F5 debugging to code-based agent workflows, useful but aimed at a different problem. Stepping through human-readable specifications has history in business rule management systems; applying it to LLM-executed natural language programs required a runtime that could map natural language steps to debuggable execution points. PBAsm's hierarchical line numbering (01, 01.01, 01.01.01) is what made that possible.
The skills ecosystem still has no equivalent. When a skill doesn't do what you expect, you read the markdown and guess at what the LLM misunderstood. For simple skills that's fine. For complex multi-step workflows it's a real gap, one that may close as models improve or may need new tooling.
Triggers as Event-Driven Interrupts (May 2025)
Playbooks has a trigger system that works like CPU interrupts:
### Triggers
- When user provides $email
- After calling ProcessPayment
- When $attempts > 3
- If user is extremely frustrated
The LLM evaluates trigger conditions continuously during execution. When one fires, the runtime saves state, invokes the handler, and resumes. Temporal, state-based, sentiment-based, and cross-agent triggers all work this way.
AutoGen v0.4 (January 2025) adopted an extensible event-driven architecture, but its events are system-level: message arrival, task completion, not semantic conditions the LLM evaluates mid-execution. Claude Code hooks fire on lifecycle events (PreToolUse, PostToolUse, Stop). Few frameworks have tried continuous LLM-monitored ambient conditions as first-class triggers. Agents today still mostly react to explicit tool calls or user messages, not to a condition like "is the user frustrated?" checked while running.
Semantic Intermediate Representation / PBAsm
PBAsm is a low-level instruction set for natural language programs. It standardizes explicit call stacks, yields, interrupts, scoped variables with lifetimes, and resumable execution boundaries.
The analogy is LLVM. Just as LLVM IR lets the same C program run on different CPU architectures, PBAsm lets the same natural language program run on different LLMs with consistent semantics. A natural language instruction like
Ask user for their name
compiles to:
01:QUE Say(user, Ask user for their $name:str); YLD for user
A semantic IR for natural language programs remains largely unexplored territory. PBAsm was designed for a world where LLMs needed structural guardrails to execute reliably, and as models improve the need for those guardrails shrinks for most applications. Whether it shrinks for enterprise workflows that require auditability and formal verification is still open. Formal methods and inherently ambiguous natural language sit in genuine tension, and nobody has resolved that yet.
Stack-Based Context Management
Traditional ReAct agents accumulate context linearly. Every step, tool result, and reasoning trace appends until the context window fills, and then something gets forcibly summarized.
Playbooks treats LLM context like a call stack instead. When a nested playbook completes, its detailed execution trace compresses into a compact return value, and the parent only sees the summary. That cascades upward, keeping context lean at every level.
This is architecturally out of reach for frameworks that orchestrate from Python without structural knowledge of what should persist in the context window. As of April 2026 most agents still accumulate linearly and lean on ever-expanding context windows. That works today with million-token windows, at a token cost stack-based management simply doesn't pay.
Other Notable Capabilities
Playbooks supports five playbook types, each tuned for a different need: Markdown Playbooks for structured workflows, ReAct Playbooks for dynamic reasoning, Raw Prompt Playbooks for single-shot LLM calls, Python Playbooks for deterministic logic, and External Playbooks for MCP integration. All five call each other seamlessly.
Natural language and Python share one call stack. A natural language playbook can call a Python function that calls another natural language playbook, all sharing execution context and variables. Skills can invoke tools including code execution, but the seamless interleaving on a shared stack is still unique to Playbooks.
Incremental code execution during LLM streaming landed in November 2025. As the LLM generated Python token by token, the runtime identified complete statements and executed them immediately, walking backward through partially-received code to find valid execution boundaries.
Adaptive waiting landed in December 2025. Instead of a binary timeout, the agent gets periodic check-ins carrying elapsed time, context, and any new messages, then decides whether to keep waiting, follow up, or escalate. OpenClaw's heartbeat (January 2026) is a similar pattern.
Multi-agent meetings shipped in July 2025 and improved in December 2025: multi-agent sessions with shared state, a structured communication protocol, and lifecycle management. Claude Code's Agent Teams (February 2026) tackles multi-agent coordination differently, with independent sessions and a team lead, seven months after the Playbooks implementation.
Forward compatibility (blogged December 2025) is the argument that traditional agent frameworks embed workarounds for current LLM limitations, and that when better models arrive those workarounds block the system from using new capabilities. Playbooks programs express pure intent, so the same program runs better on a stronger model without modification. The industry has since internalized this, and modern harnesses try to stay out of the model's way. Naming it as a deliberate property still has value: add too much scaffolding around the LLM and you lose it.
JIT coding (blogged November 2025) is the idea that the specification becomes the primary artifact while code gets generated on demand during execution and then discarded. The spec is the program; code is exhaust. Claude Code had been working this way since its February 2025 launch and Codex since May 2025. Playbooks named and framed the pattern. It differs from vibe coding, where generated code becomes a durable artifact humans maintain. In JIT coding the code is ephemeral.
A Research Direction: PlaybooksLM
One concept and prototype I never shipped publicly is worth writing down. PlaybooksLM is a model built specifically to execute natural language programs reliably, emit structured telemetry through special tokens, and provide model-side verifiability. Instead of hoping a general-purpose LLM follows your specification, you train a model whose job is to execute specifications, and it emits custom tokens the runtime uses for tracing and execution validation. Everyone in the industry is working on making general-purpose models better at following instructions. Models whose primary function is reliable execution of structured natural language programs remain under-explored.
Timeline
| Capability | Playbooks | Industry |
|---|---|---|
| NL as programming language | June 2022 (experiments); Sep 2024 (v0.1.0); Oct 2025 (blog) | Anthropic Skills: Oct 2025 |
| Agents as classes/methods | Feb 2025 (v0.2.0) | Not widely adopted in LLM agent frameworks |
| VSCode debugging of NL programs | May 2025 (v0.4.0) | MS Agent Inspector: Feb 2026 (code agents) |
| Event-driven triggers | May 2025 (v0.3.0) | AutoGen v0.4: Jan 2025 (system events); Claude Code hooks: early 2026 (lifecycle) |
| PBAsm / Semantic IR | June 2025 (v0.3.5) | Not attempted elsewhere |
| Stack-based context management | June 2025 | Not adopted elsewhere |
| Multi-agent meetings | July 2025 (v0.6.0) | Claude Code Agent Teams: Feb 2026 |
| Incremental streaming execution | Nov 2025 (v0.7.0) | Not adopted elsewhere |
| Adaptive waiting | Dec 2025 (v0.7.3) | OpenClaw heartbeat: Jan 2026 |
What the Simpler Approach Does Better
Where the skills and generalist agent approach genuinely beats Playbooks, it beats it clearly.
Expressiveness. A playbook encodes a process; a skill can encode knowledge, process, caveats, or unstructured thoughts. You can dump context, heuristics, and edge case guidance into a skill in whatever form makes sense, while Playbooks makes you think in steps and control flow. Skills pay for that expressiveness with less verifiability, and for most use cases it's worth it.
A simpler harness. A skills-supporting harness and the Playbooks runtime are both specialized runtimes in today's terms, but without a compilation step or heavy process management, a skills harness is far easier to build. That's why there are dozens of skills-capable harnesses and one Playbooks runtime.
Ecosystem momentum. Anthropic backing skills as a first-class concept, plus the broader move toward agentic patterns, created network effects a solo research project was never going to match. The best specification format is the one people actually use.
A simpler mental model. "Write whatever you want and the agent follows it" is immediately understandable. Authoring Playbooks is also simple, just markdown with minimal structure, but the compilation model and execution validation underneath add conceptual weight skills avoids entirely. The sophistication of the runtime leaked into how people perceived the authoring experience, even though it shouldn't have.
Graceful degradation. Because a skill is free-form guidance rather than a formal program, an imperfect skill degrades naturally. A Playbooks program that hits an execution validation error can halt instead. Strictness is a virtue for enterprise reliability and a liability for developer experience. Skills bet that model improvement would close the reliability gap; Playbooks engineered around it.
And forward compatibility, ironically. Less scaffolding means skills benefit more from model improvement, which is Playbooks' own principle working against it.
The Numbers
- 4 complete system rewrites
- 16 releases (v0.2.0 through v0.7.4, February 2025 to February 2026)
- 6 blog posts documenting novel concepts
- 79 GitHub stars, 12 forks
- ~44 months from first experiments (June 2022) to stepping back (April 2026)
Reflections
Building Playbooks taught me that being right about the direction isn't enough. You also have to be right about the timing and the level of abstraction.
Playbooks bet that enterprise use would demand reliable execution. It was built when LLMs were not reliable enough to execute arbitrary instructions with any consistency, so the design demanded compilation, execution validation, and formal semantics, because without them nothing worked well enough to trust. Skills started in a different era. LLMs were far more capable by late 2025, and the design constraints had moved.
The 79 stars tell their own story. I thought being technically right would matter more than it did. Even before skills existed, the compilation and runtime complexity made Playbooks hard to adopt. The ideas were sound; the packaging was too heavy. The industry wanted something you could drop into a markdown file and have it work, and Playbooks asked you to understand what was happening underneath. Most developers, reasonably, didn't want to.
I also thought the industry would reject an unreliable solution. Here we are in April 2026, and the reliable solution is the niche project with 79 stars. Skills bet, correctly, that by the time serious enterprise adoption arrived, LLMs would be reliable enough and the ecosystem would have built its own guardrails. The ceiling on automatable complexity keeps rising, and skills is riding that curve rather than engineering around it.
Playbooks is right that natural language should be the programming language for AI agents. It's right about context management, adaptive waiting, forward compatibility, and multi-agent coordination. But it put the complexity in the specification language and the runtime, when the simpler move was to push complexity into the model and keep the specification layer dead simple. As models get better, the skills approach gets better for free. That's the forward compatibility argument Playbooks championed, working against it.
The strongest argument for Playbooks' architecture turned out to be the argument for why a simpler approach would win. If you're building infrastructure around rapidly improving AI, every piece of scaffolding you add is a bet that the model won't get good enough to make it unnecessary. Sometimes that bet pays. Often it doesn't.
What Happens Next
This is a pause, not a shutdown. The repository stays on GitHub, and the documentation and blog posts at runplaybooks.ai stay up.
Look at the timeline table again and a few rows still read "not attempted elsewhere." A semantic IR for natural language programs. Stack-based context management that actually reclaims the window instead of growing it. A debugger for natural language execution. Ambient conditions the model evaluates while it runs. More than a year after Playbooks shipped them, the ecosystem has not built equivalents. Some of that is because the industry hasn't needed them yet. As agents run longer, coordinate more, and get audited harder, I think some of it starts to matter again.
So I'm leaving the door open. If I come back, it will likely be for the specific pieces that still have no equivalent rather than for the whole stack, and possibly inside a different system.
Thank You
Playbooks has been largely a solo project, but it didn't happen in isolation. Thank you to everyone who tried it and sent me feedback that changed the design. And to everyone who starred the repo: you kept me going. An open-source research project with no corporate backing runs on attention and curiosity, and you supplied both.
Research projects succeed by pushing ideas forward.
Playbooks is a semantic programming system for AI agents, with a natural language programming language, a semantic intermediate representation (PBAsm), and an execution runtime. First open-sourced in early 2023, rebuilt from scratch four times, and actively developed through February 2026.
GitHub | Documentation | Blog