T H E W A L K

Design Technologist,
AI Infrastructure,
Prototyping.

← Back to main

The Agent Hypervisor: Why Google Calls AX Orchestration

[ AUTHORIAL INTENT & AI DISCLOSURE ]

Drafted with Hermes Agent following a source-level architectural audit of google/ax at commit d8ed0fe.

Forensic Hygiene Active
View Policy Standard →

When AI developers talk about “agent orchestration,” they are usually describing a cognitive coordinator: LangGraph state graphs, AutoGen group chats, subagent delegation protocols, context window compaction, or multi-agent consensus loops. They are answering the question: Which model speaks next, and what tools does it receive?

When Google open-sourced AX (google/ax) as an “agentic orchestrator,” it caused immediate confusion. There are no prompt chains in AX. There are no subagent negotiation protocols, no memory vector stores, and no multi-agent consensus graphs.

That is because AX was designed by distributed systems and Kubernetes veterans—led by Jaana Dogan—who use the term “orchestration” in the classic Borg and Kubernetes sense: physical resource scheduling, sandboxing, network perimeter fencing, storage attachment, and cluster reconciliation.

AX is not the conductor of the cognitive choir. AX is the hypervisor for autonomous agents.

Why Agents Break Traditional Cluster Schedulers

To a systems engineer, an autonomous agent is an uncooperative, hostile workload. It does not fit the assumptions behind existing infrastructure:

Traditional Microservice (Kubernetes Pod):
  Stateless, predictable traffic, zero durable disk, runs indefinitely.

Traditional Batch Job (Kubernetes Job):
  Deterministic, runs to completion, exits 0 or 1, known resource bounds.

Autonomous Agent Workload:
  Transient, highly stateful, non-deterministic, long idle pauses,
  accumulates gigabytes of tool/workspace state, branches dynamically,
  and can burn thousands of dollars in a runaway tool loop if unfenced.

If an organization runs thousands of autonomous coding or research agents across a cluster, three systemic failures occur under standard Kubernetes:

  1. etcd Storage & Compaction Churn: Agents generate high-frequency lifecycle transitions (initializing, tool-calling, idling, retrying, failing). Storing millions of short-lived tasks as Kubernetes Custom Resource Definitions (CRDs) pushes etcd past its single-digit gigabyte storage bounds, causing write-rate bottlenecks and control plane degradation.
  2. Untrusted Code Execution: An agent that can execute shell commands, clone arbitrary Git repositories, and run package managers cannot safely share a host kernel with production services.
  3. Runaway Resource & Network Bleed: A prompt-injected or malfunctioning agent can execute port scans, flood internal APIs, or exfiltrate private tokens unless the physical network perimeter blocks it.

The Architecture: Redis Over etcd

In commit dc4f36c, AX discarded an earlier interactive CLI prototype and restructured around three distributed binaries:

   [Developer / External Agent]
                 │
          `ax apply -f task.yaml`
                 │
                 ▼
          ┌───────────────┐
          │   ax-server   │ (Stateless gRPC API on port 8080)
          └───────┬───────┘
                  │ 1. Validate & write Task Hash
                  │ 2. XADD ax:stream:tasks
                  ▼
          ┌───────────────┐
          │     Redis     │
          │   (Streams)   │
          └───────┬───────┘
                  │ XREADGROUP (Consumer Group: ax-controllers)
                  ▼
          ┌───────────────┐
          │ ax-controller │ (Horizontally scaled worker pool)
          └───────┬───────┘
                  │ gRPC Control API (Port 443)
                  ▼
       ┌─────────────────────┐
       │   Agent Substrate   │
       │  • gVisor Sandbox   │
       │  • Network Egress   │
       │  • GCS Snapshots    │
       └─────────────────────┘

The core architectural decision is decoupling the agent task lifecycle from Kubernetes etcd.

AX keeps its task definitions, state hashes, and live update subscriptions in Redis, using Redis Streams as the distributed work queue. The API server (ax-server) accepts typed protobuf requests over gRPC, writes the state into Redis, and queues a reconciliation message. A horizontally scaled pool of reconciler workers (ax-controller) consumes work using XREADGROUP, driving each task toward its desired state on Agent Substrate.

Developers interact with AX through a kubectl-style CLI (ax apply, ax get, ax watch, ax ssh, ax suspend), but the cluster control plane remains completely shielded from task churn.

The 4 Declarative Primitives

AX models agent execution using four declarative resources in the ax.io/v1alpha1 schema:

PrimitiveProto DefinitionRole
Taskpkg/apis/v1alpha1/ax.proto:67The unit of sandboxed execution. Defines container image, command, CPU/memory limits, workspace mounts, gateway reference, and debug flag.
Workspacepkg/apis/v1alpha1/ax.proto:186Filesystem and tool environment. Configures Git repos to pre-clone, MCP servers, skills directories, and an optional natural-language goal.
Gatewaypkg/apis/v1alpha1/ax.proto:153Network perimeter. Defines inbound listeners and an explicit outbound host/port allowlist enforced at the sandbox boundary.
Modelpkg/apis/v1alpha1/ax.proto:240Named model configuration. Decouples provider settings and Kubernetes API key secrets from individual task specs.

Here is how they compose into a single manifest:

apiVersion: ax.io/v1alpha1
kind: Workspace
metadata:
  name: codebase
spec:
  git:
    - repo: "https://github.com/golang/go.git"
      branch: "master"
---
apiVersion: ax.io/v1alpha1
kind: Gateway
metadata:
  name: secure-egress
spec:
  egress:
    allowlist:
      hosts:
        - host: "api.anthropic.com"
          port: 443
        - host: "github.com"
          port: 443
---
apiVersion: ax.io/v1alpha1
kind: Task
metadata:
  name: test-runner
spec:
  workspaces:
    - name: codebase
      goal: "Build toolchain from source and run smoke tests"
  gateway:
    name: secure-egress
  command: ["go", "test", "./..."]
  debug: true

Sandboxing, Durability, and the Runner

1. gVisor Kernel Isolation

In AX, containers do not run on shared host namespaces. The controller configures Agent Substrate with SandboxClass: SANDBOX_CLASS_GVISOR (internal/substrate/client.go:273). Even if an agent executes untrusted code or suffers a prompt-injection exploit, it operates inside a virtualized user-space kernel.

2. Volume Snapshots & Suspend/Resume

Agents spend substantial time idle—waiting on external tools, model rate limits, or human approvals. Keeping an active container running wastes cluster resources.

AX mounts the agent’s work directory under /workspace backed by persistent storage. When ax suspend task <id> is called:

  1. The reconciler sends SIGTERM to the container.
  2. Agent Substrate snapshots the /workspace volume and pushes it to a Google Cloud Storage bucket (internal/substrate/client.go:264-271).
  3. The physical container is destroyed, freeing compute on the worker node.

When ax resume task <id> runs, AX schedules a fresh container on any available node, restores the /workspace volume snapshot from GCS, and starts the runner. A marker file (/ax/initialized) ensures setup steps like Git cloning are skipped on resume, preserving the agent’s modified working state.

3. In-Container Supervision (ax-task-runner)

AX injects a dedicated Go binary as PID 1 inside the container (runner/runner.go). On port 80, the runner multiplexes HTTP/1.1 and unencrypted HTTP/2 (h2c):

  • /healthz and /readyz: Provide liveness and workspace readiness probes.
  • /metadata/v1alpha1/ax/task: Exposes cloud-style introspection metadata to the agent without requiring SDK dependencies.
  • Guest Daemon (ateenvv1alpha): When spec.debug: true, the runner exposes gRPC process and filesystem services. This powers ax ssh, allowing an engineer or parent agent to attach an interactive terminal into the live sandbox through the cluster’s network router.

Audit Findings & Current Limitations

A source-level audit of the repository reveals several early-stage implementation gaps:

  1. Fire-and-Forget Stream Acknowledgement (internal/controller/worker.go:101-104): Reconciler workers consume events from Redis Streams using XREADGROUP, but call sub.Ack() immediately even when processEvent() fails. There is no dead-letter queue or backoff retry. A transient network glitch to Agent Substrate leaves the task stuck in Pending or Failed.
  2. Pending Entries List (PEL) Leakage (internal/store/redis/store.go:768-774): Subscriptions only read new events with >. AX never inspects the Pending Entries List using XPENDING or XAUTOCLAIM. If a controller crashes mid-reconciliation, claimed messages remain unacknowledged in the PEL indefinitely.
  3. Execution on Failed Workspace Setup (runner/runner.go:157-187): If repository cloning or environment bootstrap fails, the runner flags ready = false (causing /readyz to return 503), but continues straight to executing spec.command. The agent process launches inside an incomplete workspace.
  4. Command Completion Invisibility (runner/runner.go:193-196): When spec.command exits, the runner keeps the container alive so logs and debug sessions remain accessible. However, the exit status is never reported back to the control plane. In Redis, the task phase remains Running indefinitely. The orchestrator cannot tell whether an agent completed its goal or crashed without an external reporter.
  5. Float64 Timestamp Precision in Pagination (internal/store/redis/store.go:164-170): Task sorting scores in Redis ZSets cast time.Now().UnixNano() (an int64) to float64. Nanosecond timestamps in 2026 exceed $1.78 \times 10^{18}$, while double-precision floats only preserve 53 bits of precision (~$9 \times 10^{15}$). The truncation discards the lowest 11 bits, risking score collisions under concurrent writes.

Systems Substrate vs. Cognitive Control Plane

The core takeaway from auditing google/ax is the clear boundary between two layers of the modern agent stack:

┌──────────────────────────────────────────────────────────────────┐
│ Cognitive Control Plane (Hermes, LangGraph, AutoGen)             │
│  • Plan decomposition & DAG execution                            │
│  • Memory, context caching, and session durability              │
│  • Multi-agent debate, review gates, and artifact synthesis      │
└────────────────────────────────┬─────────────────────────────────┘
                                 │ Dispatches isolated work
┌────────────────────────────────▼─────────────────────────────────┐
│ Systems Hypervisor (google/ax)                                   │
│  • Cluster scheduling & horizontal worker reconciliation         │
│  • gVisor kernel sandboxing & resource quotas                    │
│  • Physical network egress filtering & port routing              │
│  • Workspace pre-warming & GCS volume suspend/resume             │
└──────────────────────────────────────────────────────────────────┘

AX does not attempt to solve agent intelligence, planning, or collaboration. It solves the dirty, dangerous plumbing of running untrusted code in a shared cluster.

For developers building agent infrastructure, AX offers a proven model for how to build a scalable, sandboxed hypervisor layer—while leaving the cognitive orchestration where it belongs: in the agent application itself.

← Back to main