Reference manual · x86-64 · stable Rust · Limine

A minimal x86-64 kernel that runs isolated, unprivileged tasks which talk only by messages. Drivers, storage and attestation are ordinary tasks, and the system is fixed at build time.

Part one: guide

1Introduction#

vanir-os is a small kernel written in stable Rust and booted by Limine. Its design follows Oxide's Hubris: a fixed set of tasks is declared at build time, each runs in its own address space in ring 3, and they cooperate through synchronous send/receive/reply messages. Things a larger kernel does internally, such as device drivers, restarting failed programs, storage and talking to a host, are separate tasks here.

The kernel provides:

• serial output, GDT/TSS, IDT, and an ELF loader that enforces W^X
• a page-table tree per process, with SMEP, SMAP and NX
• a 100 Hz local APIC timer, round-robin preemption and CPU budgets
• synchronous IPC with a per-task allowlist of who may send to whom
• supervisor notifications and task restarts with generation-checked ids
• PCI device grants, MSI-X interrupts and DMA regions for driver tasks
• a kernel event log, signature checks on task images, and boot measurements into a TPM
• notifications a task posts to another without waiting, and a supervisor-only halt

On top of that, as tasks: virtio console, vsock, block and network drivers; telemetry streamed to a host agent; TPM remote attestation; encrypted per-task storage whose key the host releases only to a boot it has attested; and TCP services behind separate data and management network stacks, with routing and draining between instances.

2Quick start#

Install the dependencies (26), then:

make run                                   # build the ISO and boot it in QEMU
make test                                  # boot all 123 scenarios and check each one
make test ACCEL=kvm                        # the same under KVM
make run APP=tests/apps/two-spinners.toml  # boot a different set of tasks
make run-service ACCEL=kvm                 # one service on 127.0.0.1:8080 (15.8)

A normal boot prints the banner, the kernel's setup steps, then each task's output, and ends when nothing is left to run:

[kernel] GDT, TSS and IDT loaded
...
[kernel] pci 00:03.0 1af4:1043 virtio-console
[kernel] pid 1 (hello): 8704-byte ELF, signed version 1, entry 0x400000
[kernel] tpm: none found; the boot is not measured
[kernel] entering ring 3
hello world
[kernel] pid 1 (hello) exited with code 0 (called from CPL 3)
[kernel] no process can run:
[kernel]   pid 1 (hello): exited 0, 1 ticks
[kernel] frames in use: 3 (all task memory freed)

3The layers, from scratch#

An operating system is a stack of layers, each depending on the one below. This part walks up the stack once, saying in plain terms what each layer is for and how vanir-os does it. The reference in part two goes into detail.

 +---------------------------------------------------------------+
 | ring 3   tasks: hello, supervisor, drivers, storage, attest   |  unprivileged; one
 |          (each in its own address space)                      |  address space each
 +--------------------------- int 0x80 ---------------------------+
 | ring 0   vanir kernel: traps, page tables, scheduler, IPC,    |  privileged; small
 |          device grants, event log, measurements               |
 +---------------------------------------------------------------+
 | Limine   boot loader: loads the kernel and task images        |  runs once
 +---------------------------------------------------------------+
 | firmware SeaBIOS (BIOS) or OVMF (UEFI with Secure Boot)       |  runs once
 +---------------------------------------------------------------+
 | QEMU / KVM virtual machine: CPU, RAM, virtio devices, TPM     |  the "hardware"
 +---------------------------------------------------------------+

3.1The machine#

vanir-os runs in QEMU, which emulates the machine in software (TCG) or uses the host CPU's virtualization (KVM). The firmware is SeaBIOS or OVMF; the boot loader is Limine, which also loads every task's ELF file and signature as "modules" next to the kernel.

3.2Privilege#

The kernel sets this up with a GDT (segment descriptors for each ring), a TSS (which stack to switch to when ring 3 traps into ring 0), and CPU features: SMEP stops the kernel running user code, SMAP stops it reading or writing user memory except inside explicit copy routines, NX marks data pages non-executable, and CR0.WP makes read-only pages read-only for the kernel too.

3.3Virtual memory#

  virtual address (48 bits used)
  +---------+---------+---------+---------+--------------+
  | PML4 idx| PDPT idx|  PD idx |  PT idx | page offset  |
  |  9 bits |  9 bits |  9 bits |  9 bits |   12 bits    |
  +----+----+----+----+----+----+----+----+------+-------+
       |         |         |         |           |
  CR3 -+-> PML4 -+-> PDPT -+-> PD ---+-> PT -----+-> 4 KiB frame + offset
           (one tree per task; the kernel half is shared, supervisor-only)

3.4Traps: syscalls, faults, interrupts#

  ring 3 task                    ring 0 kernel
  -----------                    --------------------------------------------
  int 0x80  / fault / IRQ  ----> trap stub: push registers (TrapFrame)
                                 dispatch: syscall | fault | timer | device IRQ
                                 maybe switch task: swap TrapFrame, load CR3
  continues (same or other) <--- zero scratch registers, iretq

The timer interrupt fires 100 times a second. It's how the kernel takes the CPU back from a task that never gives it up (preemption) and how it counts CPU budgets.

3.5Tasks and messages#

vanir-os declares every task in an app manifest. The kernel never creates tasks at run time; it can only restart one, when the supervisor task asks.

3.6Devices#

Here a driver is an ordinary ring-3 task. The kernel maps only that device's registers into the driver, gives it a fixed DMA region, and turns the device's interrupts into messages. A crashing driver is just a crashing task.

3.7Trust: signatures, measurements, keys#

Part two: reference

4Boot#

Limine (base revision 3) loads the higher-half kernel at 0xffffffff80000000, plus one module per task image and one per signature, and jumps to kmain:

 kmain
  |-- serial::init, banner                 COM1 at 0x3f8: the boot log
  |-- cpu::enable_protections              SMEP, SMAP, NX, CR0.WP
  |-- gdt::init, interrupts::init          GDT + TSS, IDT (one trap path)
  |-- modules::init                        task images + .sig files from Limine
  |-- memory::init                         frame allocator over the memory map, HHDM
  |-- apic::init                           local APIC timer, 100 Hz (calibrated on the PIT)
  |-- pci::init                            discover devices, size BARs, set up MSI-X
  '-- process::start(kernel file)
        |-- verify each task image         Ed25519 signature, min_version
        |-- measure into PCR 12            kernel, manifest, each task (if a TPM exists)
        |-- load ELFs, build page tables   W^X per segment, stack, device + DMA mappings
        '-- enter ring 3                   first task runs; the scheduler takes over

5Memory and address spaces#

Physical memory is handed out in 4 KiB frames from Limine's memory map, with a free list so frames return when a task dies. The kernel reaches physical memory through the HHDM (Limine's direct map). Each task gets its own PML4; the kernel half is shared and supervisor-only.

 0xffff_ffff_8000_0000  +---------------------------+  kernel image (shared, supervisor-only)
                        |  ...                      |
 HHDM offset            +---------------------------+  all physical memory (supervisor-only)
                        :                           :
 0x0000_8000_0000_0000  +===========================+  USER_END: nothing user-visible above
 0x7fff_ffff_f000       +---------------------------+  user stack top (16 pages, 64 KiB, NX)
                        :                           :
 0x30_0000_0000         +---------------------------+  TPM CRB page (tpm = true task only)
 0x20_0000_0000         +---------------------------+  DMA region (dma_pages, pinned frames)
 0x10_0000_0000         +---------------------------+  device BARs, uncached
                        |                           |  (0x1000_0000 per device, 0x200_0000 per BAR)
 0x40_0000              +---------------------------+  ELF segments: code R-X, rodata R--, data RW-
 0x0                    +---------------------------+  unmapped (null page faults)

Two software bits in page-table entries record ownership: DEVICE (bit 9, MMIO, never freed) and PINNED (bit 10, DMA frames owned by the task slot, kept across restarts). Everything else a task maps is freed when it dies; the no-leak and restart-no-leak tests check the frame count returns to its baseline.

The kernel touches user memory only through copy_from_user and friends, which open a SMAP window (stac/clac) and recover from page faults through an exception table, so a bad pointer in a syscall returns an error instead of crashing the kernel.

6Tasks and scheduling#

• Static task table. The tasks are fixed at build time by an app manifest (7). Tasks get ids 1, 2, … in manifest order.
• One kernel stack. A trap saves the user's registers on it. A context switch stores that frame in the current task, copies the next task's saved frame over it, switches CR3, and returns through the normal iretq. Kernel code never waits in the middle of a syscall: a task that must wait is marked blocked with its registers saved, and whoever completes the operation writes the results into those saved registers.
• Round-robin preemption on the 100 Hz timer. A task with a budget is killed after that many ticks of CPU.
• Outcomes. A task ends by exit, by faulting, or by using up its budget; the others keep running. When nothing can run, the kernel prints a summary and halts with a status (20). Tasks waiting in recv count as idle servers; tasks stuck in send mean a deadlock.

              scheduled                   send / recv / sleep
   +-------+ ----------> +---------+ ----------------------------> +-----------------+
   | Ready |             | Running |                               | SendBlocked     |
   +-------+ <---------- +---------+ <---------------------------- | ReplyBlocked    |
       ^      timer tick     |  |       reply / message / wake-up  | RecvBlocked     |
       |                     |  |                                  | Sleeping        |
       |   restart(id)       |  +-- exit(code) -----> Exited(code) +-----------------+
       +---------------------+----- fault ----------> Faulted(vector)
         (supervisor only)   '----- budget spent ---> OverBudget

7App manifests#

An app manifest is a TOML file listing every task. kernel/build.rs turns it into the kernel's task table at build time. Pick one with make run APP=… (default apps/default.toml; tests use tests/apps/).

[[task]]
name = "client"        # unique task name; also the program, unless `program` is set
program = "hello"      # optional: build/<program>.elf (any program under user/)
budget = 50            # optional: CPU budget in 10 ms ticks (default: unlimited)
supervisor = true      # optional: told when other tasks die (at most one)
sends = ["echo"]       # optional: tasks this one may send IPC messages to
devices = ["virtio-console"]  # optional: PCI devices to drive (kernel/src/pci_kinds.rs)
dma_pages = 16         # optional: contiguous DMA region in 4 KiB pages (max 256)
events = true          # optional: may read the kernel event log and stats (at most one)
min_version = 2        # optional: refuse images signed with an older version
tpm = true             # optional: gets the TPM mapped, for attestation (at most one)
listen = [{ port = 80, max_connections = 64, max_per_source = 16 }]
                       # optional: TCP ports it may listen on (max_connections default 64,
                       # max_per_source default max_connections; at most 8 entries)
service = "web"        # optional: the network service it is an instance of (default: its name)
routes = true          # optional: may route, drain and blocklist in the network stacks
loads = true           # optional: may load service images into slots (at most one: the agent)
connects = true        # optional: may open outbound connections through a network stack

runtime_log = 1024     # optional, top level: run-time measurement log entries

[[slot_class]]         # optional: generic slots for services loaded at run time (15.1)
name = "small"
count = 2
memory_pages = 256
image_pages = 128

The build fails on unknown keys, duplicate names, unknown sends targets or device names, more than one supervisor, event reader, TPM task or loader, more than 64 tasks and slots, or a port declared by two different services (instances of one service may share it). Each devices entry claims the next unclaimed device of that kind; a missing device is logged and the task runs without it.

Names with meaning

8IPC#

Synchronous send/receive/reply, as in Hubris. A client's send waits until the server has received the message and replied. The kernel copies messages and replies between address spaces, up to 4 KiB each. A task may only send to the tasks listed in its manifest sends.

  client                      kernel                             server
     |                           |                                  |
     |- send(server, op, msg) -->| check sends allowlist            |
     |  [SendBlocked]            | queued until server recvs        |
     |                           |<-- recv(buf) --------------------|
     |  [ReplyBlocked]           |--------------------- copy msg -->| (sender, op, len)
     |                           |                                  |  ... does the work ...
     |                           |<-- reply(sender, code, data) ----|
     |<-- (code, reply_len) -----|                                  |  reply never blocks
     |                           |                                  |

Task ids carry a generation (slot + 1 | generation << 16) that changes on restart. A client holding an id from before a restart gets DEAD instead of silently reaching a fresh server that lost its state; it can lookup the name again and resend. Kernel notifications arrive as messages from sender 0.

9Supervisor#

One task may be marked supervisor = true (user/supervisor). The kernel reports every other task's death to it as a message from sender 0 with NOTIFY_DEATH and a death record: how it ended, exit code or fault vector, rip, cr2, error code and ticks used. The supervisor decides what to do and may call restart(id), which reloads the task's ELF into fresh memory with a new generation. The policy lives in the supervisor, not the kernel. If the supervisor itself dies, the kernel halts.

The policy (RFD 2, P5): a task that exits 0 is done. A task that faults, exits non-zero or uses up its CPU budget is always restarted, however often it crashes, so a request that crashes a service can't take it down for good. Restarts back off so a crash loop can't eat the CPU: 10 ms after a crash, doubling to at most 30 s, and back to 10 ms once the task has run 60 s since its last restart. The supervisor waits with recv's timeout, so a pending restart costs nothing. Each crash goes to telemetry as a crash event with the restart count and the backoff (crash-loop, backoff-reset). The agent asks it to stop a task (supervisor::OP_STOP: the kernel's stop, reported as death kind 5, never restarted) or to start a dead one again (OP_START, with its backoff reset), for deployments (15.4). Test features: give-up-after-3 and give-up-after-3000 stop restarting so a run made only of a crash loop can end, backoff-fast restarts at once, reset-after-2s shortens the reset window, and halt-on-exit stops the system when any task exits.

  task faults ---> kernel: free its memory, mark Faulted, log "fault" event
                     |
                     '--> NOTIFY_DEATH {kind, code, rip, cr2, ...} --> supervisor
                                                                          |
       exited 0: done   |   fault / non-zero exit / over budget: restart after a backoff
                                                                          |
  new generation <--- kernel: reload ELF, fresh page tables, id' = id + (1 << 16)
  clients holding the old id get DEAD; they lookup() the name again

10Drivers#

Device drivers are ordinary tasks. The kernel discovers PCI devices at boot and a manifest grants one to a task (devices = ["virtio-console"]). The driver gets:

• the device's register BARs mapped into its address space (uncached), except the BAR holding the MSI-X table: the kernel programs interrupt routing itself;
• read-only access to the device's PCI configuration space;
• a physically contiguous DMA region (dma_pages), the same physical pages for the life of the task slot and zeroed on every restart;
• the device's interrupts as notifications in recv (sender 0, NOTIFY_IRQ). A driver with work in flight passes RECV_EXPECT_IRQ so the kernel idles for it instead of ending the run.

     driver task (ring 3)                       device (virtio PCI)
  +-----------------------------+           +-----------------------------+
  | BAR window  0x10_0000_0000  |-- MMIO -->| common cfg, notify, ISR     |
  | DMA region  0x20_0000_0000  |           |                             |
  |   virtqueues + buffers      |<-- DMA ---| reads requests, writes data |
  | recv(): NOTIFY_IRQ <--+     |           +---------------+-------------+
  +-----------------------+-----+                           | MSI-X
                          '---- kernel irq.rs <-------------'

  clients ---- IPC (block / console protocol) ----> driver

Drivers: virtio-console, virtio-vsock (one stream to host CID 2, port 5000), virtio-blk and virtio-net (14), all on the shared user/virtio crate (virtio 1.x modern PCI, split virtqueues). A crashing driver is restarted by the supervisor; it resets the device and carries on, and its clients see DEAD and retry against the new id (console-restart, blk-restart, nic-restart).

A driver that must tell its client something without being asked (frames arrived) posts a notification with notify(client, bits) rather than sending: only the client sends, so the two can't deadlock. The client gets the bits in recv as sender 0, NOTIFY_POST.

11Telemetry#

The kernel logs events at its control points into a 128-entry ring: task start, exit, fault (vector, rip, cr2), CPU-budget kill, restart, refused IPC sends and rejected images. The task with events = true (user/telemetry) forwards them as newline-delimited JSON through the uplink driver, with a heartbeat every second, log lines from other tasks, storage events, and network events and counter snapshots (14.5).

  kernel event ring --read_events--+
  (128 entries, seq numbered)       |
  supervisor --OP_LOG---------------+--> telemetry --console write--> uplink driver --> host
  disk (storage) --OP_STORAGE-------+     (JSON lines,                (virtio-console     |
                                          heartbeat 1/s)               or virtio-vsock)    v
                                                                       tools/host-agent.py

{"seq":5,"t":1,"task":"crasher","id":"2","event":"fault","vector":14,"rip":"0x400182","cr2":"0x10"}
{"seq":6,"t":2,"task":"crasher","id":"2.1","event":"restart","previous":"2"}
{"t":2,"task":"supervisor","id":"1","event":"crash","of":"crasher","index":2,"kind":"fault","code":14,"restarts":1,"backoff_ms":20}
{"t":5,"event":"dropped","count":37}
{"t":102,"event":"heartbeat","n":1,"frames":89,"frames_free":28500,"irqs":3,"events":11,"tasks":[...]}

A heartbeat gives the frames in use and still free (4 KiB each) and each task's CPU ticks so far, so the CPU used over a second is the change in the tasks' ticks over the change in t (load tests do this, 20).

t is in 10 ms ticks since boot; ids are slot or slot.generation. Kernel events carry seq; if the ring wraps before telemetry reads it, a dropped record says how many were lost. The uplink is the virtio console by default (QEMU writes it to build/telemetry.log), or vsock with TRANSPORT=vsock and an uplink task running virtio-vsock.

tools/host-agent.py build/telemetry.log            # console uplink
tools/host-agent.py --listen-vsock 5000 &          # vsock: listen first, then boot
make run TRANSPORT=vsock ACCEL=kvm APP=tests/apps/telemetry-vsock.toml

host-agent.py checks that every line is JSON with an event, that sequence gaps are covered by dropped, and heartbeat spacing; --check turns problems into the exit status and --expect EVENT=N requires counts. --set-interval MS sets the counter snapshot interval once the guest connects (vsock), and --expect-interval TICKS checks that snapshots then arrive that far apart.

12Boot chain and attestation#

With BOOT=uefi-secure every link checks the next before running it:

  OVMF (Secure Boot on, snakeoil keys)
    | checks sbsign signature on BOOTX64.EFI ............. refuses: sb-unsigned-loader
    v                                       measures --> PCR 0-7
  Limine
    | checks its own config hash (enroll-config) ......... refuses: sb-edited-config
    | checks kernel BLAKE2b hash pinned in limine.conf .. refuses: sb-tampered-kernel
    v
  vanir kernel                              measures --> PCR 12: kernel, manifest, tasks
    | checks each task image: Ed25519 sig, name, version  refuses: sig-*
    v
  tasks (ring 3)
    attest: TPM quote of PCR 0,4,7,12 over host nonce --> host verifies --> releases key

Task images ship as boot modules beside the kernel, each with a .sig file (VNRSIG01, program name, version and SHA-256 of the ELF, signed with Ed25519). The kernel refuses an image that is unsigned, changed, signed for another program, signed by another key, or older than the manifest's min_version. A refused task never runs; the supervisor and telemetry are told why. Images can be updated without rebuilding the kernel, as long as they are signed.

Measured boot (TPM=1, swtpm): the firmware measures the boot loader and Secure Boot state into PCRs 0–7. Before any task runs, the kernel extends PCR 12 with SHA-256 of the kernel file, of the app manifest, and of each task's signed statement (zeros for a refused image). tools/expected-pcrs.py computes PCR 12 from the build output alone.

Attestation: the attest task (tpm = true) holds an attestation key in the TPM (restricted ECDSA P-256 under the endorsement hierarchy). When the host sends attest <nonce> over the vsock uplink, control asks attest, which returns a TPM quote of PCRs 0, 4, 7, 12 and 13 (13: images loaded into slots since boot, 15.3). tools/verify-quote.py checks it is genuine TPM output, over this nonce, correctly signed, consistent with the reported PCRs and the kernel's measurement log, and equal to an approved reference.

make image BOOT=uefi-secure APP=tests/apps/attest.toml USER_FEATURES=control/one-shot
tools/host-agent.py --listen-vsock 5000 --save build/telemetry.log --attest &
make run BOOT=uefi-secure ACCEL=kvm TPM=1 TRANSPORT=vsock APP=tests/apps/attest.toml USER_FEATURES=control/one-shot
tools/verify-quote.py --record build/attest-reference.json \
    --expected-pcr12 $(tools/expected-pcrs.py --app tests/apps/attest.toml)   # known-good boot
tools/verify-quote.py --reference build/attest-reference.json                  # later boots

13Storage#

DISK=1 attaches build/disk.img (64 MiB, kept between runs) as a virtio block device. Three tasks share the work:

  client-a ---+                                              +--> keys   (key server)
  client-b ---+--block IPC--> disk (storage) --OP_MASTER_KEY-+
              |  per-volume    |  encrypt / decrypt          '--< attest (OP_UNLOCK, after
              |  block numbers |  XChaCha20-Poly1305                 the host releases the key)
              |                v
              |           blockdev (virtio-blk) --DMA--> disk image (ciphertext only)
              x
  only "disk" may send to "blockdev" (storage-no-driver)

Clients send disk block requests (vanir_user::block: read, write, flush, info) with block numbers relative to their volume. There is no file system: a volume is 256 numbered 4 KiB blocks (1 MiB). The volume is chosen by the client's manifest name, which the storage task gets from the kernel (task_name of the sender), so a client can neither name nor reach another task's volume. A task's first request creates its volume.

13.1On-disk format#

 block 0        superblock: "VNRSTOR3", next free block                 plain text
 block 1        volume table: 64 x {name[32], start, size, kind}        plain text
 blocks 2..260  volume "client-a"
                  +-- header block:      {counter u64, tag[16]} in plain text, then sealed
                  |                      {"VNRVOL01", kind, name, size, epoch, lease}
                  +-- 2 metadata blocks: per data block {counter u64, tag[16]}
                  '-- 256 data blocks:   XChaCha20-Poly1305 ciphertext
 blocks 261..   volume "client-b" (same layout), in creation order ...

Block b of a volume is encrypted with the volume key, nonce = b ‖ counter, and associated data = volume name ‖ b ‖ counter. The counter goes up on every write, so a nonce is never reused. Ciphertext that was changed, moved to another block or volume, or paired with another counter fails to decrypt: the client gets INTEGRITY and telemetry an integrity_error. A block never written (counter 0) reads as zeros. A disk without the magic is formatted on first use, and so is a disk in an older format (VNRSTOR2), with a message saying so.

The header is sealed the same way, as block number 2⁶⁴−1, and holds the volume's kind, name and size, and its writer epoch and lease (RFD 3; recorded, not yet used). On a volume's first read or write after each unlock, disk opens the header and checks it against the table; if it doesn't open or doesn't match, every read and write of that volume gets INTEGRITY. Volumes are of two kinds: a fixed task's, and (from RFD 3) a service's. The kind is in the table, so a task and a service with the same name get different volumes, and in the key's salt (below), so their ciphertext never opens in each other's.

13.2Keys#

  master key (32 bytes; only in disk's memory, never on the disk)
     |
     +-- HKDF-SHA256(salt "vanir-storage-v1", info "client-a") --> volume key A
     +-- HKDF-SHA256(salt "vanir-storage-v1", info "client-b") --> volume key B
  service key (32 bytes, one per service; from the control plane, RFD 3, P2; 13.6)
     +-- HKDF-SHA256(salt "vanir-service-v2", info service)     --> the service's volume key
                                                                     |
                     per block: XChaCha20-Poly1305(key, nonce = block || counter,
                                                   aad = name || block || counter)

The storage task starts locked, answering LOCKED to every request for a task's volume until it has the master key, and to every request for a service's volume until it has that service's key (13.6). The master key comes from one of two places:

• The key server (keys, user/unlocker): a test stand-in for a local key store, with the key compiled in. If disk may send to keys, it asks for the key when it starts. keys gives it only to the task named disk.
• The host, after attestation (tests/apps/attest-storage.toml). The key lives on the host (keys/master.key, made by host-agent.py, never in the image).

13.3Key release#

  host-agent.py                       control         attest (TPM)                  disk
       | ----------- "attest <nonce>" --> |---- OP_QUOTE -->|                         |
       |                                  |                 | new X25519 pair         |
       |                                  |                 | quote(PCR 0,4,7,12,     |
       | <-- quote JSON + kx_pub -------- |<----------------|  SHA256(nonce||kx_pub)) |
       |                                  |                 |                         |
  verify quote vs reference;              |                 |                         |
  mismatch -> "key_refused", stop         |                 |                         |
  seal: X25519(eph, kx_pub)               |                 |                         |
    -> HKDF(salt = nonce,                 |                 |                         |
         "vanir key release")             |                 |                         |
    -> ChaCha20-Poly1305(                 |                 |                         |
         master, aad = nonce)             |                 |                         |
       | --- "key <eph_pub||ct||tag>" --> |--- OP_UNWRAP -->| open with kx secret     |
       |                                  |                 |--- OP_UNLOCK(master) -->| unlocked
       | <-- "unlocked" ----------------- |                 |                         |

A boot that measures differently (a changed kernel, manifest or task image) gets no key, and the sealed key is useless to anything but that boot's attest task. OP_UNLOCK is accepted only from tasks named keys or attest.

13.4Restarts and events#

The supervisor restarts disk and blockdev like any task. A restarted blockdev resets the device; disk sees DEAD, finds the new driver and resends, keeping its key (storage-blk-restart). A restarted disk starts locked and must get the key again: from keys it asks for it itself (storage-restart); clients see DEAD and resend.

If disk may send to telemetry, it reports:

{"t":6,"task":"disk","id":"3","event":"storage_locked"}
{"t":6,"task":"disk","id":"3","event":"storage_unlocked","by":"keys"}
{"t":7,"task":"disk","id":"3","event":"volume_created","volume":"client-a","start":2}
{"t":9,"task":"disk","id":"3","event":"integrity_error","volume":"client-a","block":0}
{"t":196,"task":"disk","id":"3","scope":"storage","event":"counters","blocks_read":5,"blocks_written":8,"flushes":3}

The counters (blocks read and written through the driver, and flushes, since it started) come once per snapshot interval while requests arrive. task comes from the kernel, so no other task can report storage events in disk's name.

make run DISK=1 DISK_RESET=1 APP=tests/apps/storage.toml   # fresh disk: two clients write and check
make run DISK=1 APP=tests/apps/storage.toml                # same disk: they find their data
make run DISK=1 DISK_CORRUPT=client-a:0 APP=...            # flip a byte of client-a's block 0 first

13.5Service volumes and writers#

A service image loaded into a slot (15) can declare a volume in its signed descriptor (volume = { blocks = 16 }, at most 65536). The kernel returns the size with the rest of the descriptor (task_listen), and disk gives an instance in a slot its service's volume, named after the service, whichever slot it runs in. An instance whose descriptor declares no volume gets NOT_ALLOWED. The version that makes the volume fixes its size; an instance that declares another size gets OUT_OF_RANGE and can't open it (RFD 3 leaves resizing to RFD 4).

A service volume has one writer (RFD 3, P1). Any instance of the service may read it; writes and flushes from any other get FENCED. The writer is named by an epoch, which the control plane assigns with each deployment and the agent hands to disk (block::OP_EPOCH: epoch, lease, writer). Only the task with loads = true may send it. disk records a higher epoch in the volume's header and flushes it before replying. The same epoch again is accepted only while there is no writer, or the writer is unchanged. Anything lower is refused, across reboots too, since the header is on the disk. The writer is a slot, kept in memory: a writer that crashes and is restarted in its slot stays the writer, and a restarted disk knows no writer until the agent names it again. The lease is recorded, not enforced (RFD 3, determination 8).

 agent: deploy counter v2 into small-1, epoch 2         counter v1 (small-0)   counter v2 (small-1)
   load, start ......................................... writes OK             reads OK, writes FENCED
   OP_HANDOFF(counter, small-1); route; drain small-0 .. drains                reads, writes wait
   small-0 drained: OP_EPOCH(2, lease, small-1) ........                       writer bit; held
                                                                               requests answered;
                                                                               writes OK
   stop small-0

Handoff (RFD 3, P3). Between the routing change and the old instance draining, the new instance's requests wait rather than fail. Just before routing to it, the agent names it the handoff's candidate (block::OP_HANDOFF: service, candidate). From then on disk holds the candidate's reads, writes and flushes without answering them; the kernel lets a server answer a blocked caller later. When OP_EPOCH makes the candidate the writer, disk posts it the writer bit (block::NOTIFY_WRITER, which every new writer gets) and then serves the held requests in order. Until an instance has seen the bit, it must not trust anything it cached while read-only; from the bit on, nobody else writes the volume. If the epoch is refused, the agent calls the handoff off (candidate 0) and the held requests are served as they are: reads succeed, writes get FENCED (volume-handoff-off). A client blocked in a call has one request outstanding, so disk holds at most one per slot (8 entries; BUSY, which the client retries, if ever full). A restarted disk loses what it held (its callers get DEAD), and the agent sends the handoff again with the epochs.

counter (user/counter, a service on the runtime, 15.8; tests/slots/counter-v*.toml) uses the volume its descriptor declares: /count reads a count from block 0, adds one and writes it back, or with the setting persist_ticks counts in memory and writes it every that many ticks and before it drains or exits. It opens the volume before it says it is ready, and never says so if that fails. A task named like a service still gets a volume of its own: the table records each volume's kind, and their keys use different salts (volume-kinds).

Events: epoch_changed (volume, epoch, writer slot), write_fenced (volume, slot, epoch; the first refusal per slot and epoch) and handoff (volume, requests held, longest wait in ticks, whether it was called off).

13.6Service keys#

Each service's volume has its own key (RFD 3, P2). The control plane makes it on first use, keeps it (build/cp-volume-keys/<service> for the mock), and sends it only to an attested node that runs the service. The agent forwards it but can't read it: the key is sealed to attest's per-boot X25519 key, which the handshake's quote binds along with the agent's own keys. The agent can dial out and load images, so a compromised agent shouldn't also be able to read every volume on the node.

  control plane                          agent                attest                    disk
       | <-- handshake: quote over SHA-256(e || s || seal_pub) ---|  (OP_QUOTE_FOR)          |
  check quote; for each service with a volume it runs here:      |                          |
  seal: X25519(fresh, seal_pub)                                  |                          |
    -> HKDF(salt = fresh pub, "vanir-volume-key-v1")             |                          |
    -> ChaCha20-Poly1305(key, aad = service name)                |                          |
       | -- K: service, fresh pub || ct || tag -->|               |                          |
       |                                          |-- OP_UNWRAP_VOLUME -->| open             |
       |                                          |               |-- OP_SERVICE_KEY -------->| service volume opens
       |                                          | keeps the blob for a restarted disk      |
       |    commit, no instance left here:        |-- OP_FORGET_KEY -------------------------->| locked again

• Only for that service. The service name is the seal's associated data, so a blob moved to another service doesn't open (agent-volume-key-swap); the agent reports key-refused <service>.
• Only to attested nodes. A node that said no-tpm gets no keys, so a service with a volume never gets ready there (agent-volume-unattested).
• Without its key a service's volume answers LOCKED; task volumes and other services are unaffected, and so is OP_EPOCH for other services. The instance never says it is ready, and the deployment fails (volume-no-key).
• Forgotten. Once a commit leaves no instance of a service on the node, the agent has disk forget its key and drops the blob (volume-forget). A node only holds the keys of what it runs.
• The stand-in. keys serves a test key per service as well (HKDF of a fixed seed and the name), and disk asks it when it needs one, so the one-node tests run without a TPM or a control plane. An app with keys accepts either source; the attested app (agent-volumes-tpm.toml) has none.

OP_SERVICE_KEY is accepted only from keys and attest, OP_FORGET_KEY only from the task with loads = true. A volume written under V1's keys (salt vanir-service-v1, from the master key) no longer opens.

14Networking#

Services serve TCP from tasks of their own, as RFD 1 (rfd/0001-network-services.md) proposes. Each network has a NIC driver task and a network stack task, and the management and data networks are separate: a flood or a compromise of the data stack can't reach the stack the control plane uses. Everything is declared in the manifest; nothing here loads at run time.

  host / clients
      | data network                      | management network
      v                                   v
  nic (virtio-net) --frames-- net     nic-mgmt (virtio-net) --frames-- mgmt-net
      DMA ring             smoltcp        DMA ring                  smoltcp
                    sockets, routing,                                  |
                    limits, counters                              mgmt-echo (stands in
                     |         |    \                              for the node agent)
                  web-1      web-2   '--counters, events--> telemetry
              (live)   (draining)
                     ^
               switcher (routes = true): route, drain, blocklist

14.1Drivers and stacks#

user/virtio-net moves Ethernet frames between a virtio-net device and one client, and knows nothing about IP (vanir_user::nic): OP_SEND_FRAME takes one frame, OP_RECV_FRAMES returns as many waiting frames as fit in 4 KiB and never waits, OP_INFO gives the MAC and the interface counters. It negotiates only VERSION_1 and MAC: no checksum offload, so smoltcp computes checksums. Sixteen 2 KiB receive buffers stay posted. The receive interrupt is on only after the client has emptied the ring; the driver then posts NOTIFY_RX to it with notify, turns the interrupt off, and the client polls until the ring is empty again (as Linux's NAPI does).

user/net runs smoltcp 0.12 (ARP, IPv4, TCP, DHCP) with no heap: 96 sockets with 4 KiB buffers each, in fixed arrays. Only net sends to the driver, and the driver only posts notifications back, so the two can't deadlock. The same program runs both stacks: the task named net drives nic, and mgmt-net drives nic-mgmt. Both use DHCP; the net/static-addr test feature gives the data stack 10.0.9.2/24 instead, for the dgram backend, which has no DHCP server.

net spends at most half of each tick on received frames. Past that, frames wait in the driver's ring until the next tick and the device drops what doesn't fit, so a flood can't take the CPU from the services or the management stack (mgmt-under-flood). The budget is measured with the TSC, so time net spends preempted counts too.

14.2Sockets#

A service declares its ports in the manifest (listen, 7). net asks the kernel which ports a client may use (task_listen), so a task can't claim ports it wasn't given (http-undeclared-port), and the build refuses two services declaring one port (build-http-port-clash). Instances of one service (service = "web") may share a port.

Events: EV_ACCEPT, EV_READ (data, or the peer closed), EV_WRITE, EV_CLOSED (reset), EV_DRAIN (handle 0). Only OP_WAIT waits, so a single-threaded service runs one event loop. Handles carry a generation, as task ids do, so a handle kept after its connection closed can't name a newer one. smoltcp has no accept queue (one socket holds one connection), so net keeps a few sockets listening on each port in use and hands each connected one to the live instance of the port's service.

user/http is a small HTTP/1.1 server on this interface: GET and HEAD, keep-alive, fixed buffers. / answers hello from <task> and /count the number of requests. When either side restarts, the other recovers: HTTP finds net again and listens anew (net-restart), and net forgets a restarted instance's connections (http-restart).

14.3Routing and draining#

Each service has one live instance, which gets its new connections; existing connections stay where they are. An instance becomes live only after it has said OP_READY. The first ready instance of a service becomes live by itself; after that, a task with routes = true changes it with OP_ROUTE (route-not-ready, route-not-allowed). In the RFD 1 tests that task is switcher; RFD 2's agent does it too (15.4), and asks an instance's status with OP_STATUS (starting, ready, live, draining, drained) instead of waiting for a drain's reply (DRAIN_NOWAIT).

  switcher: OP_ROUTE web-2      new connections now go to web-2
  switcher: OP_DRAIN web-1, deadline 10 s   (reply held)
     net --EV_DRAIN--> web-1    web-1 stops accepting; answers each connection's next
                                request with "Connection: close"; clients reconnect to web-2
     ...                        connections left at the deadline: reset, counted as drain_forced
     web-1 --OP_DONE--> net     no connections left: the OP_DRAIN reply (drain_forced) goes out

Connections the draining instance never accepted go to the live instance. HTTP leaves idle connections alone while draining: closing one just as its client sends a request would fail that request. With eight clients requesting without pause, no request fails and every client ends up on web-2 (swap-under-load); a connection held open without a request is reset at the deadline and counted (drain-deadline).

14.4Limits#

Before smoltcp sees a frame, net checks it:

• sources on the blocklist are dropped first (16 ranges with expiry, set by OP_BLOCK from a task with routes = true);
• IPv4 packets with a bad header checksum are dropped and counted;
• each new connection (a SYN for one of our ports) is held to the port's max_per_source and max_connections, and to 16 handshakes in progress per port. smoltcp has no SYN cookies, so at that cap a new SYN takes the place of the oldest handshake; handshakes also time out after 2 s.

HTTP limits each request: the header must fit in 4 KiB (else 431) and arrive within 5 s of its first byte (else 408), and bodies over 16 KiB are refused (413). Connections idle for 15 s between requests are closed by HTTP, and for 60 s by net. A 32-entry Space-Saving sketch tracks the heaviest sources; each snapshot interval net compares SYN and limit-drop rates and CPU budget use with fixed thresholds and reports attack_suspected with the top talkers.

14.5Counters and events#

Counters are cumulative and go to telemetry as snapshots (OP_COUNTERS), one per scope; rates are the host's to compute. Telemetry answers each snapshot with the interval for the next: 1 s by default, and the host may set 100 ms to 60 s (host-agent.py --set-interval MS, through the control task; net-interval).

{"t":210,"task":"net","id":"3","scope":"instance","event":"counters","instance":"web","service":"web","live":1,"accepted":1,"bytes_in":3533,"bytes_out":5806,...}
{"t":13,"task":"net","id":"3","event":"route_changed","service":"web","live":"web","previous":"none"}

Events (OP_NET): route_changed, drain_started, drained, listen_refused, limit_tripped, attack_suspected, blocklist_added. Records travel as name key=value ... text and telemetry turns them into JSON, adding the sender's name from the kernel. net-counters checks the counters against a known transfer, byte for byte (tools/check-net-counters.py). smoltcp doesn't count retransmits, so neither do we.

14.6Running it#

make run APP=tests/apps/http.toml NET=user          # then: curl http://127.0.0.1:8080/
make run APP=tests/apps/mgmt.toml NET=user NET_MGMT=user   # plus: nc 127.0.0.1 8081 (echo)
make run APP=tests/apps/http.toml NET=dgram USER_FEATURES=net/static-addr   # raw frames

NET adds the data NIC and NET_MGMT the management NIC, at PCI 00:10.0 and 00:11.0 (declare the data driver first in the manifest: each devices entry claims the next device of its kind). user is QEMU's NAT with DHCP, and forwards 127.0.0.1:NET_PORT (8080) to the guest's port 80 and NET_MGMT_PORT (8081) to port 7. dgram carries raw Ethernet frames in UDP datagrams between QEMU (NET_DGRAM_PORT, 18090) and tools/net-peer.py (the next port), which can send any frame it likes. Tests run net-peer.py next to QEMU (PEER=<scenario>); since servers never go idle, they build the supervisor with halt-on-exit and end the run by asking HTTP for /exit (http/exit-on-request).

15Deployment#

Services are deployed while a node runs, and move between nodes (RFD 2, rfd/0002-deployment.md; the plan is plans/step-10-deploy.md). A node's manifest can't name every service that might land on it, so it declares a pool of generic slots. The agent (user/agent, the only task with loads = true) loads signed service images into them at run time. Until there is a control plane, test scripts built into the agent stand in for it.

15.1Slots and descriptors#

[[slot_class]]
name = "small"           # slots are named small-0, small-1, ... after the tasks
count = 2
memory_pages = 256       # most frames a service may use: image, stack, page tables
image_pages = 128        # image area: the largest image a slot can hold (512 KiB)
budget = 20              # optional: CPU budget, as for tasks
sends = ["net"]          # optional: tasks its services may send to (by task name)

Each slot is a task table entry that starts empty. At boot the kernel sets aside its image area (contiguous, never freed, like a DMA region) and checks that free memory covers every slot at its class's limit. Tasks' memory is fixed once loaded (there is no syscall to grow it), so after that check a load can't fail for want of memory. The manifest decides the table's size: tasks plus slots, at most 64.

What a service may do comes with its image, in a descriptor signed with it (tests/slots/*.toml):

name = "web"             # the service
version = 2
min_version = 1          # oldest version that may still run
slot_class = "small"
listen = [{ port = 80, max_connections = 64 }]
drain = { default_secs = 30, max_secs = 300 }
volume = { blocks = 256 }  # optional: the service's volume (RFD 3, 13.5)
abi = "1.0"              # optional: the ABI it was built against (default: vanir-abi's)
format = 2               # optional: 1 writes the older 256-byte format (no abi, no config)

[config]                 # optional: up to 12 settings (RFD 9, P5)
persist_ticks = 1000     # keys: 1..16 of a-z, 0-9, _ (a letter first); integer values

tools/sign-image.py sign-slot KEY ELF DESC.toml OUT compiles it to the layout the kernel reads (OUT.desc; vanir_abi::descriptor: format 2, VNRDSC02, 512 bytes, or format 1, VNRDSC01, 256 bytes; the first 160 bytes are the same in both) and writes a VNRSIG02 signature file (OUT.sig, 176 bytes) over the name, version, and SHA-256 of both the image and the descriptor, with the boot images' key. The Makefile builds the test images (SLOT_IMAGES) into build/slots/, then the agent, which carries them. task_listen answers for a slot from its descriptor, so net lets a service listen only on the ports it was signed with; it also returns the version, and http in a slot calls itself e.g. web v1 on small-0.

The ABI version and settings (RFD 9). A format-2 descriptor records the ABI version the image was built against (sign-image.py reads ABI_MAJOR.ABI_MINOR from abi/src/lib.rs), and the kernel loads it only if the major version is its own and the minor is no newer (else REJECTED: "built for an ABI this kernel doesn't serve"; slot-abi-refused). A format-1 descriptor counts as 1.0 with no settings. The settings are signed with the image, checked at load (valid keys, each once, the unused entries zero), and handed to the instance through task_listen (ListenInfo::setting("persist_ticks")); http prints its slot's settings at start (slot-settings). New settings mean a new descriptor, so a new signed statement and a new image name: changing them is a deployment, never a change to a running instance.

15.2Loading an image#

The agent copies the image into an empty slot's image area in pieces (slot_stage, so it never needs a buffer the image's size), then calls slot_load with the signature file and descriptor. The kernel, in order:

1. checks the signature, and that it covers this image and this descriptor;
2. checks the version against the descriptor's min_version;
3. checks the descriptor names the slot's class;
4. checks the image was built for an ABI it serves (same major, minor no newer);
5. builds the address space, and refuses if it needs more frames than the class allows;
6. measures it (below), or fails with BUSY or LOG_FULL having changed nothing;
7. records the descriptor and starts it as a new generation of the slot.

A refused image fails with REJECTED and the reason, is logged as an image_rejected event, and is wiped from the image area. A crashed service is restarted from the image area like any task (9), without being measured again. slot_unload empties a slot whose service has ended: it zeroes the image area and measures the unload. A slot holding an image, running or not, can't be staged over.

15.3Measuring what runs#

Each load and unload is extended into PCR 13 and recorded in the kernel's run-time log: 80-byte entries (operation, slot, service name, version, SHA-256 of the signed statement), and PCR 13 is extended with each entry's SHA-256. The kernel keeps the log, as it does the boot log, so a crash of any task can't make the node unattestable. It has a fixed length (runtime_log in the manifest, default 1024 entries, about 80 KiB); when it is full, loads fail with LOG_FULL until the node reboots.

The attestation task sends TPM commands itself, so it takes a lock the kernel holds (tpm_lock/tpm_unlock) around each sequence of commands, and the kernel extends PCR 13 only while nobody holds it: a load meets BUSY instead of waiting. The kernel drops the lock if the task dies. Quotes cover PCRs 0, 4, 7, 12 and 13 and say how many run-time entries they cover; control then forwards those entries (attest::OP_RUNTIME_LOG, a JSON line of up to 20 at a time), and tools/verify-quote.py replays them to PCR 13 and lists what was loaded. tools/expected-pcrs.py --runtime load:small-0:web-v1 ... computes PCR 13 from the build. RFD 8 lists what this leaves open (rollback through an old descriptor, one key for boot and slot images, a task that holds the TPM lock forever).

15.4The lifecycle and the local swap#

The agent owns every slot's state (RFD 2, P4) and reports each transition as an instance event (slot, service, version, from, to) to telemetry:

 EMPTY --load--> STARTING --ready--> READY --promote--> LIVE
   ^               |                                     |
   |            timeout (10 s)                         drain
   |               v                                     v
   +------------ FAILED                              DRAINING
   '-- unload <-- STOPPED <-- stop <-- DRAINED <--------'

A local swap deploys a new version of a service into an empty slot while the old one serves: the agent loads it, waits until net says it is ready (net::OP_STATUS), makes it live (OP_ROUTE), and drains the old instance without waiting on net (OP_DRAIN with DRAIN_NOWAIT, then OP_STATUS until it is drained). The drain deadline is the descriptor's default, or what the deployment asks for capped at the descriptor's maximum. Once drained, the agent asks the supervisor to stop it (supervisor::OP_STOP): the kernel ends it (stop, death kind 5) and the supervisor never restarts a stopped task. Its slot keeps the image, so a rollback is the same flow with the roles reversed: the supervisor starts the stopped slot again (OP_START, backoff reset) and the newer version drains. Commit unloads the stopped slots of a service. A service that exits 0 by itself is done and isn't restarted. Nothing in the lifecycle blocks: each deployment advances a step at a time from the agent's loop. For a service with a volume, a deployment carries a writer epoch, which the agent hands to disk before routing to the new instance if there is no old one, and otherwise once the old one has drained and before it is stopped (13.5). If disk refuses it, the new instance stays live without it, and the agent reports the refusal.

Tests: deploy-under-load and rollback-under-load keep eight clients requesting through the swap (and back) with no failed request; deploy-not-ready leaves the old version live when the new one never says it is ready; drain-deadline-cap, deploy-commit, slot-exit-clean.

15.5The agent and the control plane#

The agent dials the control plane through the management stack (mgmt-net; it has connects = true, and net::OP_CONNECT opens the connection), at the address built in (CP_ADDR, by default 10.0.2.2:7000: the host, through slirp). It reconnects with backoff (100 ms doubling to 10 s) whenever the connection drops; deployments go on meanwhile (user/agent/src/control.rs).

  agent                                                  control plane (tools/control-plane.py)
    | -- e, es, s, ss + quote over SHA-256(e || s) ----->  pinned key? quote binds e and s?
    | <-- e, ee, se ------------------------------------   (Noise IK, X25519, ChaChaPoly, SHA-256)
    | -- R report: generation, each slot's state ------->
    | <-- S desired state, Ed25519-signed, generation N -   applied only if N is newer
    | -- G image <hash> into <slot> ------------------>
    | <-- C chunks (staged into the slot), E sig + desc -   then the kernel checks and loads it
    | -- T telemetry lines (pulled from telemetry) ----->
    | <-- K a service's volume key, sealed to attest ---   attested nodes only (13.6)

• Who it talks to. The control plane's Noise static key is pinned in the agent at build time (keys/cp-noise.pub), so the agent talks to no one else (agent-wrong-cp-key). The agent's static key is made once per boot and its ephemeral key for each connection (RDRAND). The static key is also how another node's agent knows this one when a volume moves (15.7), so it stays the same across reconnections.
• Who the node is. The first handshake message carries a TPM quote from attest (OP_QUOTE_FOR) over SHA-256(ephemeral key || static key || seal_pub), where seal_pub is attest's own per-boot key that service keys are sealed to (13.6): the keys of this handshake belong to a measured boot of a known image, and the private keys never leave the agent. The control plane checks it (agent-attested) and refuses a quote that binds another key (agent-unbound-key). A node without a TPM says no-tpm; --require-attestation refuses it.
• What it wants. Desired state is a text document signed with a second pinned key (keys/cp-sign.pub), with a generation the agent only lets rise, so a replayed or reordered command does nothing (agent-signed-state). Lines: live <service> <slot> <image hash> [drain <s>] [epoch <n> <lease ms>], stop <service> [drain <s>], commit <service> (and end, for tests). Images are named by the SHA-256 of their signed statement. For a service with a volume, epoch names its writer (13.5): the agent gives it to the new instance in a deployment, or to the live instance if only the epoch changed. The mock control plane raises it whenever the live instance changes.
• Reconciliation. The agent keeps moving towards the last state it applied: fetching an image it lacks straight into the named slot, deploying it, rolling back to a stopped slot that already holds the wanted image, draining, committing. It reports what it has after the handshake, on every change and every 5 s, so a dropped connection costs nothing but time (agent-reconnect, agent-deploy, agent-drain). A slot whose instance holds its volume's epoch reports epoch <n>, and epoch-refused <n> while disk refuses the one asked for (agent-epoch-stale). In agent-volume, four clients keep counting through a deployment of counter: every request is answered, and the counts are exactly 1 to N, none twice and none missing.
• Telemetry goes to the control plane through the agent. With no uplink driver, the telemetry task keeps its lines in a 16 KiB ring and the agent pulls them (telemetry::OP_PULL): if telemetry pushed to the agent while the agent sends to net and net to telemetry, the three could deadlock.

The Noise code (user/noise, no heap) and the host's (tools/vanir_noise.py, written separately from the spec) are both checked against the cacophony test vector in tests/vectors/noise-ik.json (noise-vectors, a host check). tools/control-plane.py gen-keys keys makes the dev keys on the first build; tests run it with CP=<scenario> (20).

15.6Moves, evacuation and updates#

A move takes a service to another node with the same pieces as a swap, driven by the control plane: it makes the service live on the new node, sends new connections there (its load balancer), then drains and stops the old instance and commits. Clients of the old node get Connection: close on their next request and reconnect through the load balancer to the new one (move-under-load: eight clients, no failed request).

A node's own image (kernel, drivers, net, the agent) only changes when it runs no instances (RFD 2, determination 1). The desired-state line update is refused by a node with instances, and its report says how many (agent-update-refused). The control plane evacuates it first, moving each instance away; an empty node then asks the supervisor to reboot it (supervisor::OP_REBOOT, the kernel's reboot: the 8042 reset line), and comes back with the image installed now. Its reports carry the node's name and its own signed version, so the control plane sees which image it rejoined with (evacuate-update: node a refuses, is evacuated to node b, reboots into an image signed as version 2, rejoins, and the service moves back, all under load).

tests/two-nodes.sh runs these cases: two images from one tree (agent built with NODE=a or b; with UPDATE=1 a second one for node a with SIGN_VERSION=2), two QEMUs (data ports 8080 and 8090), the control plane with its load balancer on port 8000, and the peer through it. QEMU runs with -no-reboot, so a reboot ends it with status 0, and the harness boots node a again from the updated image, standing in for an A/B update of its disk. Staging that image on a real disk is out of scope (RFD 2's plan). With DISK=1 each node gets its own copy of the disk the build made (build/disk-a.img, build/disk-b.img).

15.7Moving a volume#

A service with a volume takes its volume along when it moves (RFD 3, P4 and P5). The volume goes directly from the source node to the target node, not through the control plane, which only sets the transfer up (user/agent/src/transfer.rs).

                     control plane
     transfer counter send b <b's key>     transfer counter receive a 10.0.2.2:8081 <a's key>
        epoch 2 expires 60  \                /  epoch 2 expires 60
                             v              v
   node a (source) agent, listening on       node b (target) agent
   mgmt port 7 while it has the ticket  <--  dials; Noise IK, prologue "vanir-transfer 1",
                                             its first message names the service

• Tickets are lines in each node's signed desired state: transfer <service> send <peer> <peer key> epoch <n> expires <s> for the source, and transfer <service> receive <peer> <address:port> <peer key> epoch <n> expires <s> for the target. The peer's key is the static key of its agent's handshake with the control plane, which its quote binds; the expiry counts from when the agent applied the document. A ticket missing from a later document is withdrawn, and the connection closes. One transfer at a time per node.
• The target dials the source at the address in its ticket (the mock control plane is told each node's with --node-addr; in the tests, node a's management port 7 is the host's 8081, reached as 10.0.2.2). The source listens only while it holds a send ticket, and accepts one connection (listen = [{ port = 7 }] for the agent in agent-volumes.toml).
• Both check the other. The target knows the source's key from its ticket (Noise IK), and the source closes the connection unless the target's key and the service in its first message are the ones its ticket names (transfer-connect, transfer-wrong-peer). The control plane already holds the volume keys, so trusting its ticket for the peer adds no trust.
• Reports carry a line for the transfer: transfer <service> send|receive <phase> [detail], with phases waiting, connected, copying, mirroring, then handing-over and handed-over on the source and taken on the target, or refused-digest on both, or aborted (with why).

  source agent                         target agent                     target disk
    | -- H size, header (96 bytes) -----> | -- OP_IMPORT_HEADER -----------> | made or emptied; header must open
    | -- B b, counter, tag, ciphertext -> | -- OP_IMPORT_ENTRY, OP_IMPORT --> | must open with the service key
    |    ... every block written ...      |                                  |
    | -- R blocks in the round ---------> |                                  |
    |    rounds until one sends <= 2     |                                  |
  OP_MIRROR on; a last round; M -------> |                                  |
  a write by the writer: made, held      |                                  |
    | -- W the write -------------------> | -- OP_IMPORT ------------------> |
    | <-- A stored ---------------------- |                                  |
  OP_MIRROR_ACK: the writer is answered  |                                  |

• Blocks travel as they are on disk: ciphertext, write counter and tag (block::OP_EXPORT, OP_EXPORT_ENTRY, kept together when read). The seal binds only the volume name, the block number within the volume and the counter, so a block is valid on any disk with the service's key, and the target's disk checks each one opens before storing it (OP_IMPORT; transfer-corrupt: a bit changed on the way is refused, and the target drops its copy).
• Catching up. The source compares each block's counter (OP_EXPORT_META) with the one it last sent, and sends what changed, in rounds, until a round sends at most 2 blocks.
• Mirroring. The source's disk then makes each write by the writer and holds its reply (OP_MIRROR); the agent finds it each turn (OP_MIRROR_PENDING), sends it, and answers it (OP_MIRROR_ACK) once the target says it is stored. A last round catches what changed before mirroring began. A write is acknowledged only once both nodes have it (transfer-mirror: four clients counting, every count answered, none lost).
• Giving up. If the target doesn't store a mirrored write within 2 s, the source stops mirroring, answers the held write (it is on the source) and writes alone (transfer-stall). Either side's agent tells the other it is giving up (Q), and a ticket withdrawn or expired ends the transfer too: the source stops mirroring and the target drops what it received (OP_DROP: header and counters wiped, so every block reads as never written, and the space kept for the next volume of the same size).
• Limits. One transfer at a time per node; volumes of at most 4096 blocks; no rate limit (RFD 3, P5 question 1); a dropped transfer starts again from the beginning, with new tickets.

  control plane          node a (source)                         node b (target)
  live counter small-1 ............................................ starts counter only once
    epoch 2 (to b)                                                  mirrored; routed to it:
                                                                    OP_HANDOFF, requests wait
  load balancer -> b
  stop counter (to a) -> drains, stops its instance
                         OP_HAND_OVER: read-only, flushed
                         OP_DIGEST -- D digest ---------------->  OP_DIGEST: equal?
                                   <-- Y ----------------------    OP_EPOCH(2): writer bit,
                         handed-over                                held requests answered
  commit counter (to a) -> OP_DROP: its copy wiped, space freed

• The target's instance starts only once the volume is mirrored (the agent holds back a live line for a service whose volume is still coming). When it is routed to, its requests are held (13.5's handoff), and it doesn't get its epoch from the live line: the transfer gives it.
• The handoff starts on the source once no instance of the service runs there: the control plane has moved its load balancer to the target, then drained and stopped the source's instance, whose last writes were mirrored. The source's disk makes the volume read-only (OP_HAND_OVER: no writer, writes FENCED, the same epoch can't be asserted again; only a higher one makes it writable) and the agent sends its digest (OP_DIGEST: SHA-256 over the header and every counter and tag). If the target's is the same, its instance is offered the epoch its live line names (the ticket's), as any live instance is, and that answers what was held (move-volume-under-load: four clients counting through the move, every count answered, none lost, none twice).
• Different digests mean the target's copy isn't the source's: it says so, calls its handoff off (its instance's held writes get FENCED), and drops its copy when its ticket is withdrawn. The source's volume stays read-only until the control plane gives it a higher epoch, which the mock does by deploying the service there again (move-volume-digest).
• The commit on the source drops its copy (OP_DROP), then its key is forgotten (13.6). The ticket's expiry no longer matters once the handoff is decided.
• Not covered (RFD 3, determination 11): if the link fails after the source has handed over and before it hears the answer, neither node writes until the control plane decides; and nothing recovers a target that fails between the handoff and the commit. Both are for a follow-up RFD.

15.8Writing a service (vanir-rt)#

A service in a slot is written against vanir-rt (user/rt, RFD 9, P3), not against vanir_user: it implements Service, and the runtime owns the event loop. user/counter is the example:

#![cfg_attr(target_os = "none", no_std, no_main)]   // a slot's image, or host tests

use vanir_rt::{Ctx, Error, Request, Response, Service, Volume};

struct Counter { volume: Volume, persist_ticks: u64, /* ... */ }

impl Service for Counter {
    const CONFIG: &'static [(&'static str, u64)] = &[("persist_ticks", 0)];
    fn start(ctx: &mut Ctx) -> Result<Self, Error> {
        Ok(Counter { volume: ctx.volume()?, persist_ticks: ctx.config("persist_ticks") })
    }
    fn request(&mut self, ctx: &mut Ctx, req: &Request, res: &mut Response) { /* write!(res, ...) */ }
    fn tick(&mut self, ctx: &mut Ctx) -> Option<u64> { /* persist; ticks until the next call */ }
    fn drain(&mut self, ctx: &mut Ctx) { /* persist before done */ }
    fn fenced(&mut self, ctx: &mut Ctx) { /* drop the cached count */ }
}

vanir_rt::service!(Counter, connections = 64);

• The runtime owns the lifecycle. It finds net, listens on the port the descriptor declares, says it is ready once start returns Ok (never, if it returns Err: the agent fails the deployment), answers HTTP/1.1 with keep-alive and the header and idle deadlines, and sends http telemetry counters. On a drain it stops accepting, closes connections as their requests finish, calls drain(), then says it is done; a service can't skip a step or reorder them. It finds net again if the network stack restarts.
• Requests: request gets each GET or HEAD (the runtime answers anything else: 405, 413, 501, ...), and writes a status and body (at most 448 bytes).
• Settings (15.1): CONFIG lists the service's settings and defaults; ctx.config(key) reads one. A descriptor with a setting the service doesn't list, or allowing more connections than service! made room for, never becomes ready, and says why.
• The volume: ctx.volume() opens the one the descriptor declares (Error::NotDeclared without), waiting while storage is locked. Volume::read, write, flush return vanir_rt::Error (Fenced, Locked, Integrity, OutOfRange, ...). A write or flush refused as Fenced makes the runtime call fenced() once the callback returns; when the instance becomes the writer, the handle prints it.
• Sizes are fixed at build time: no heap; the connection table is a static of service!'s size (stable Rust can't size an array from a trait's constant).
• Escape hatch: vanir_rt::raw (feature unstable) gives the task's raw syscalls and IPC; counter's volume-probe test feature uses it to check it can't name itself the writer.

http stays the plain web server of the RFD 1 and 2 tests (web), with its own loop and test features; it shares the runtime's request parser (vanir_rt::http). The load tests (20) measure the runtime: moving counter from http to vanir-rt changed its throughput at 8 clients from 1687 to 1644 requests/s (a write per request) and from 14775 to 14376 (cached): 2–3%, about the spread between runs of the same build (1687 to 1742).

Running one (RFD 9, R4). make run-service SERVICE=counter boots a node for one service (apps/service.toml: the network stack, storage and the agent): the program SERVICE, signed with SERVICE_DESC (default apps/services/<program>.toml), is slot image service, and the agent (feature script-serve) deploys it into small-0, as its volume's writer (epoch 1) if it has one. The descriptor's first port is forwarded to 127.0.0.1:SERVICE_PORT (8080):

make run-service ACCEL=kvm         # ... agent: counter v1 is serving from small-0
curl http://127.0.0.1:8080/count   # 1, 2, 3, ... (Ctrl-C in the QEMU terminal ends it)

The volume lives in build/service-disk.img, kept across runs (the count carries on) and apart from the tests' build/disk.img; SERVICE_RESET=1 starts it from zeros. A new program must be in the workspace and USER_PROGS, with a build.rs like counter's, and its descriptor must name slot class small (256 frames, a 512 KiB image). The test run-service boots the same node and counts twice through the forwarded port.

Testing one on the host (RFD 9, open question 5). On any target but x86_64-unknown-none, vanir-rt is a test backend: vanir_rt::host::Host calls the service as the runtime in a slot does (start, then the first tick; tick when due; the same checks before request; fenced after a call whose write was refused; drain before an exit), with the clock moved by the test and the volume in memory. No network and no QEMU: requests are bytes.

let mut h = Host::<Counter>::new("counter").volume(16).setting("persist_ticks", 100);
h.start().unwrap();
assert_eq!(h.get("/count").body, "1\n");
h.get("/count");
h.advance(100);                       // the timer's tick writes the count
assert_eq!(h.block(0)[0], 2);
h.restart().unwrap();                 // a crash: no drain, the volume kept
h.fence(true);                        // another instance is the writer now

cargo test -p counter --target x86_64-unknown-linux-gnu runs counter's tests; the host check rt-host runs them with the backend's own (user/rt/tests/host.rs). Ctx, Response and the request and settings checks are one copy, used by both backends. The backend fakes only Fenced: not Locked, I/O errors, the network, or several instances at once.

16Syscalls#

int 0x80. Number in rax; arguments in rdi, rsi, rdx, r10, r8, r9; results in rax (then rdx, rsi). Simple calls return u64::MAX on error; IPC calls return the IPC error codes.

The numbers, error codes, notifications, flags, task states and the records the kernel writes into a task (Stats, ListenInfo, Event, DeathRecord) are defined once, in the vanir-abi crate (abi/src/lib.rs, RFD 9), which the kernel and vanir_user both use. It also holds the IPC protocols between tasks (vanir_abi::proto: block, net, nic, console, telemetry, supervisor, attest, keys), which the kernel doesn't read. vanir_user re-exports all of it under the paths programs use (vanir_user::err, vanir_user::block::OP_READ, ...). The crate is data only: constants, #[repr(C)] records, and safe functions that read them (accessors, names of codes); no syscalls, no unsafe, no dependencies. The host check abi-snapshot (abi/tests/snapshot.rs) pins every value and every record's size and field offsets, so a change shows up there before it shows up in a boot.

Versions. vanir_abi::ABI_MAJOR.ABI_MINOR is 1.0. A minor version may only add: syscalls, codes, flags, task states, rejection reasons, fields at the end of a record read by size, descriptor fields in reserved space. Anything else is a new major version. Slot images carry the version they were built for (15.1).

Records are read by size. stats, task_listen and the supervisor's death notice copy as much of their record as fits in the caller's buffer and return the record's whole size (the death notice is as long as what was copied), so a record can grow at its end: an older program gets the fields it knows, and a newer one on an older kernel finds the new fields zero. ListenInfo grew this way from 136 to 440 bytes (the ABI version and settings); abi-read-by-size reads it into 64, 136 and 600 bytes. Records read as arrays (Event, the measurement and run-time log entries) can't grow.

recv flags (in rdx): RECV_EXPECT_IRQ (1), RECV_TIMEOUT (2, ticks in r10; then sender 0, NOTIFY_TIMER; 0 ticks only takes what is already waiting). Kernel notifications come as sender 0 with op NOTIFY_DEATH (1, supervisor), NOTIFY_IRQ (2, drivers), NOTIFY_TIMER (3) or NOTIFY_POST (4, bits from notify).

17Protections#

18Limitations#

Stated plainly, in one place:

• No IOMMU. Driver tasks can point their devices at any memory, including the storage task's key. Planned for step 7.
• No KPTI. The kernel and the physical-memory map are mapped (supervisor-only) into every task. On Meltdown-affected CPUs (such as the development host's i7-7500U) user code could read them.
• The hypervisor sees everything. Keys and plain text are in guest memory, which the host can read. Protecting against the host needs AMD SEV-SNP or Intel TDX.
• Development keys and trust on first use in the boot chain (12).
• No storage rollback protection. An older data block written back with its older counter and tag still decrypts. Fixing it needs a root of freshness, e.g. a Merkle root in TPM NV storage.
• Plain-text volume table. Task names, volume positions and which blocks were ever written are visible; editing the table is a denial of service.
• Storage isn't crash-consistent. Data is written before its metadata; a crash in between leaves that block failing its integrity check until rewritten. Nothing is guaranteed durable before a flush.
• Volumes are only freed when they move away (a dropped copy), and the space is reused only by a volume of the same size. A fixed task's volume is always 256 blocks. A service's volume is sized by its descriptor, and that size can't change.
• Writer leases are recorded, not enforced. A move recovers from failures before its handoff only: a link lost during the handoff, or a target lost before the commit, needs the control plane and a follow-up RFD (15.7). A node's handed-over state is kept in memory, so a source that reboots before the commit has a writable copy again at the epoch it had (the control plane must not give it that epoch again).
• The control plane holds every service's key (RFD 8, vector 7), in plain files for the mock (build/cp-volume-keys/). Keys are never rotated.
• One key release per boot. In the attested flow the host says bye after releasing the key, so a disk restarted later stays locked until the next boot.
• One CPU, no IPC priorities, messages copied (no memory lending). Network bytes are copied twice through IPC.
• Network limits without attack tests. The connection caps, half-open cap and blocklist in net are implemented but the planned dos-* tests aren't written yet (14.4).
• TCP over IPv4 only, no TLS, no SYN cookies (smoltcp has none), 10 ms TCP timer resolution, and no retransmit counter.
• Restarting is the only crash policy. A service that crashes on every request is restarted forever, at most every 30 s; rolling back or blocking the client that triggers it is RFD 7.
• Known gaps in deployment (RFD 8): the version floor is each image's own descriptor, so the control plane or a compromised agent can load an old release with its old descriptor; boot and slot images are signed with one key, so a signature doesn't say which kind of image it is for; attest keeps its raw TPM mapping, so a compromised attest can hold the TPM lock forever and block every load; anything that can trigger loads can fill the run-time log, after which the node refuses loads until it reboots; a service that can be made to exit 0 stays down; the control plane's keys are in the image, so rotating them means updating every node.
• A mock control plane. tools/control-plane.py stands in for the real one (scheduling, load balancers, the fleet view are a later RFD). Nodes are named by a name built into the agent (NODE), not by their TPM's key, and the two-node tests boot an updated node from a new ISO rather than writing it to an A/B boot disk.
• Fixed slot classes, stateless services. Slots come in classes fixed by the manifest (sizing them per service is RFD 4).

Part three: working on vanir-os

19Agent quick reference#

Dense facts for coding agents (and people in a hurry). Every command runs from the repository root.

Where to change what

20Build, run, test#

make                                  # build build/vanir-os.iso
make run                              # boot in QEMU, serial output on the terminal
make test                             # boot every scenario in tests/run.sh; PASS/FAIL each
make test ACCEL=kvm                   # same, under KVM (needs /dev/kvm)
tests/run.sh --bench                  # also the load tests (BENCH=1 cases); or BENCH=1 make test
ONLY='^storage' make test             # only cases whose names match
make run APP=tests/apps/budget.toml   # choose the app manifest
make run USER_FEATURES=hello/poke-kernel   # turn on a test feature
make run-debug                        # also log interrupts and CPU resets to qemu.log
make clean                            # remove build outputs (distclean also removes limine/)

Make variables

Test-only settings in a case's variables column: TIMEOUT=<s> (default 60), ATTEST=1 (host agent challenges the guest), RELEASE=<reference> (host agent releases keys/master.key if the quote matches), INTERVAL=<ms> (host agent sets the snapshot interval), PEER=<scenario> (run tools/net-peer.py next to QEMU; its output follows a --- peer --- line), ATTEST_DELAY=<s> (wait before the challenge), CP=<scenario> (run tools/control-plane.py next to QEMU, with options from CP_ARGS=a,b,c; its output follows a --- control plane --- line and the node's telemetry lands in build/cp-telemetry.log), SLOW=1 (skipped with SLOW=0 make test), BENCH=1 (a load test: run only with tests/run.sh --bench or BENCH=1). A BUILD_FAILS list before the cases names manifests that must not build, with the error they must give, and HOST_CHECKS runs commands on the host alone (the Noise test vectors, and abi-snapshot: the ABI's values and layouts; descriptor-format: sign-image.py still writes the fixtures abi/tests/descriptor.rs reads).

Load tests

Load tests are skipped unless asked for, since they take a minute and report numbers rather than check behaviour (though a failed request, or a lost or repeated count, still fails them). counter-load boots tests/apps/counter-bench.toml: counter (http with a volume) in small-0, with telemetry on the virtio console. The peer (net-peer.py count-load) runs keep-alive clients asking for /count and, every second, samples the request rate, the number of clients, latency (p50, p99), and from the guest's telemetry its CPU (all tasks, and each task), memory and I/O (disk blocks read and written, network bytes). The samples go to build/bench/counter-load.csv, and tools/bench-chart.py draws them, time along the x axis, as build/bench/counter-load.svg.

counter-cached-load is the same with counter-v1-cached: the same program, signed with persist_ticks = 1000 (15.1), so counter keeps the count in memory once it has read (and written) it on the volume, and writes it back every 10 s and before it drains or exits. A crash loses up to 10 s of counts; a write that fails (FENCED) drops the cached count, so the next request goes to the volume. agent-volume-cached (the control plane with --image-suffix=-cached, deploying counter-v1-cached and counter-v2-cached) checks that a deployment loses no count with it: v1 writes its count before it says it has drained, and v2's first request, held until v2 is the writer, reads it. One run under KVM (60 s, 1 to 32 clients):

With a disk write per request, disk serves one request at a time (3 block reads and 2 writes each) and waits on the device, so more clients add only latency. Cached, the disk sees 2 block writes every 10 s and the CPU is the limit, mostly net.

BENCH=1 ACCEL=kvm ONLY=counter-load tests/run.sh             # 60 s, ramping 1, 2, 4, 8, 16, 32 clients
LOAD_CLIENTS=8 LOAD_SECONDS=30 BENCH=1 ACCEL=kvm ONLY=counter-load tests/run.sh   # 8 clients throughout
LOAD_PATH=/ BENCH=1 ACCEL=kvm ONLY=counter-load tests/run.sh # without the volume, for comparison

Guest CPU is sampled at the timer's 100 Hz (a tick is charged to the task it interrupts), so a second's figure is good to about 1%. Time the CPU spends halted, e.g. waiting for the disk, counts as idle.

Exit status

QEMU exits with (code << 1) | 1 through the isa-debug-exit device:

Test case format

Each row of CASES in tests/run.sh:

"name | app | user features | kernel features | make variables | expected status | expected text"

Expected text is a fixed string; with re:, a Perl regex over the whole output (grep -Pz, so [\s\S]* spans lines; use lookaheads for order-independent matches; avoid quotes); with cmd:, a shell command that reads the output on stdin and must succeed.

21Test catalogue#

141 cases (and two load tests with --bench), one build check and three host checks (noise-vectors, abi-snapshot, descriptor-format), run in this order. Order matters: later cases reuse the disk image, TPM state and attestation references earlier ones leave behind.

Cases needing vsock are skipped (printed as SKIP) when /dev/vhost-vsock isn't writable even through sg kvm.

22Repository map#

kernel/build.rs           generates the task table from the app manifest
kernel/linker.ld          higher half at 0xffffffff80000000, Limine request sections
kernel/src/main.rs        Limine requests, kernel entry, halt/panic, banner
kernel/src/cpu.rs         enables SMEP, SMAP, NX, CR0.WP
kernel/src/apic.rs        local APIC timer (calibrated against the PIT), PIC remap
kernel/src/serial.rs      COM1 serial output
kernel/src/gdt.rs         GDT + TSS (kernel stack for ring 3 -> 0 transitions)
kernel/src/interrupts.rs  IDT and the shared trap entry path
kernel/src/memory.rs      frame allocator, per-process page tables, kernel MMIO mappings
kernel/src/elf.rs         ELF loader (per-segment permissions, W^X)
kernel/src/usercopy.rs    copy_from_user with page-fault recovery
kernel/src/process.rs     task table, scheduler, budgets, IPC, supervisor and device calls
kernel/src/pci.rs         PCI discovery, BAR sizing, MSI-X (device kinds in pci_kinds.rs)
kernel/src/irq.rs         device interrupts to task notifications
kernel/src/events.rs      kernel event log
kernel/src/modules.rs     task images and signatures as Limine boot modules
kernel/src/verify.rs      Ed25519 task image signatures; slot images and their descriptors
kernel/src/tpm.rs         TPM 2.0 (CRB): PCR extend/read
kernel/src/measure.rs     measured boot into PCR 12; run-time log and PCR 13; the TPM lock
kernel/src/syscall.rs     int 0x80 dispatch
abi/                      vanir-abi: the ABI and IPC protocols, defined once (kernel and user); tests/snapshot.rs pins them
user/lib/                 shared runtime (vanir_user): entry!, syscall wrappers, println!, protocol clients; re-exports vanir-abi
user/hello/               greeting plus the attack tests (features)
user/spinner/ isolation/ runaway/ sleeper/     scheduling and isolation tests
user/echo/ client/ deadlock/                   IPC server, client, deadlock
user/supervisor/ crasher/                      restart policy; a task that faults at once
user/devprobe/ dmatest/                        device grant and DMA region tests
user/virtio/              virtio 1.x PCI transport and split virtqueues
user/virtio-console/      console driver; crash-after-50 test feature
user/virtio-vsock/        vsock driver: one stream to host CID 2 port 5000
user/virtio-blk/          block driver; crash-after-10 test feature
user/chatter/             writes 200 numbered lines through the console driver
user/telemetry/           event log + heartbeats + log/storage events -> JSON lines -> uplink
user/tpm2/                TPM 2.0 client library: attestation key, quote, PCR read
user/attest/              quotes; opens the released key and unlocks storage
user/control/             host commands from the uplink (attest, key, bye)
user/storage/             encrypted volume per task; crash-after-16 test feature
user/blktest/             block client: writes, reads back and checks a pattern
user/unlocker/            key server "keys" (test stand-in)
user/virtio-net/          NIC driver: frames in and out; crash-after-10-frames test feature
user/nictest/             answers ARP and ping by hand (driver test, no stack)
user/net/                 network stack (smoltcp): sockets, routing, limits, counters
user/http/                HTTP/1.1 server (web); exit-on-request, crash and listen-81 test features
user/rt/                  vanir-rt: the service runtime (Service trait, event loop, HTTP, volume, settings);
                          on the host, a test backend (src/host.rs)
user/counter/             counter: a service on vanir-rt with a volume; persist_ticks setting
user/mgmt-echo/           TCP echo on the management stack (stands in for the agent)
user/switcher/            routing scripts for tests (RFD 1)
user/agent/               the node agent: control plane connection, reconciliation, lifecycle, slot loading; test scripts (features)
user/noise/               Noise IK initiator (no heap), checked against tests/vectors/noise-ik.json
apps/                     app manifests and their format (apps/README.md)
tests/apps/               manifests the tests boot
tests/slots/              descriptors of the test slot images (built into build/slots/)
tests/two-nodes.sh        runs a case on two nodes, with the control plane's load balancer
tests/vectors/            the Noise IK test vector (cacophony)
tests/run.sh              the test harness
tools/host-agent.py       receives and checks telemetry; attests; releases the key
tools/bench-chart.py      draws a load test's per-second samples (CSV) as an SVG chart
tools/vanir_attest.py     quote checks and key sealing (shared)
tools/verify-quote.py     verifies a TPM quote against a reference
tools/expected-pcrs.py    computes PCR 12 (and PCR 13 for given loads) from the build output
tools/sign-image.py       makes the signing key; signs task images and slot images with descriptors
tools/check-backoff.py    checks the supervisor's restart backoff from a run's output
tools/control-plane.py    mock control plane: Noise responder, quote checks, signed desired state, images, scenarios
tools/vanir_noise.py      Noise IK (both sides) for the host tools
tools/with-swtpm.sh       runs QEMU with a software TPM
tools/corrupt-disk.py     flips a byte of a volume's block in the disk image
tools/check_lines.py      checks numbered lines arrived in order
tools/net-peer.py         host side of the network tests: HTTP, echo, raw frames, floods
tools/check-net-counters.py  compares network counters with a known transfer
plans/                    step plans: step-7-iommu.md, step-8-bench.md (not built), step-9-net.md, step-10-deploy.md
rfd/                      requests for discussion: network services, deployment, volumes, ...
keys/                     signing key and storage master key (generated, git-ignored)
docs/                     this manual (GitHub Pages) and llms.txt

23Recipes#

23.1Add a user program#

1. Copy user/hello/ to user/<name>/ (keep build.rs: it applies user/linker.ld). Rename the package and [[bin]].
2. Write src/main.rs:

   #![no_std]
   #![no_main]
   
   use vanir_user::println;
   
   vanir_user::entry!(main);
   
   fn main() -> u64 {
       println!("<name>: hello");
       0 // exit code
   }

3. Add "user/<name>" to members in the root Cargo.toml and <name> to USER_PROGS in the Makefile (that also makes it a boot module and signs it).
4. Reference it from a manifest: program = "<name>".

23.2Add a test case#

1. Add a manifest in tests/apps/ if no existing one fits.
2. Put test-only behaviour behind a cargo feature in the program's Cargo.toml, with a comment saying what it tests.
3. Add a row to CASES in tests/run.sh, after the cases whose state it depends on.
4. Run it alone (ONLY='^name$' make test), then break the protection on purpose and confirm the case fails, then restore and run both full suites.
5. Add it to 21 and, for a protection, 17.

23.3Add a syscall#

1. Pick an unused number; add SYS_<NAME> to vanir_abi::sys (abi/src/lib.rs) and to abi/tests/snapshot.rs, and a dispatch arm in kernel/src/syscall.rs.
2. Implement the handler (in kernel/src/process.rs if it touches the task table). Read arguments from the TrapFrame; write results to rax/rdx/rsi. Validate every user pointer with the address space's check_range and copy through usercopy. Never block inside the handler: mark the task blocked and switch.
3. Add a wrapper to user/lib/src/lib.rs (the constant is already there, through vanir-abi). A record the call fills goes in vanir-abi too, with its layout in the snapshot.
4. Restrict privileged calls to the right task (as restart is supervisor-only) and return NOT_ALLOWED otherwise.
5. Document it in 16.

23.4Add a kernel or storage event#

1. Kernel event: a constant in vanir_abi::events (document what a, b, c hold; add it to the snapshot), events::log(...) at the control point in the kernel, and a format arm in event_line in user/telemetry.
2. Storage event: a kind constant in vanir_abi::proto::telemetry, a format arm in storage_line, and self.event(kind, block, name) in user/storage.
3. Check it with tools/host-agent.py - --quiet --check --expect <event>=1 in a cmd: test.

23.5Add a device kind or driver#

1. Add (name, vendor, &[device ids]) to KINDS in kernel/src/pci_kinds.rs (shared with build.rs, which validates manifests).
2. Add the QEMU device to QEMU_FLAGS (behind a variable, as DISK is) with disable-legacy=on for virtio.
3. Write the driver on user/virtio; take requests by IPC; handle NOTIFY_IRQ; use RECV_EXPECT_IRQ while work is in flight.
4. Add a crash feature and a restart test, as virtio-blk/crash-after-10.

23.6Add a manifest key#

1. Add it to known in kernel/build.rs, parse and validate it, and emit it in the generated Task literal.
2. Add the field to Task in kernel/src/process.rs.
3. Document it in apps/README.md and 7.

24Invariants#

Rules the code relies on. Breaking one is a bug even when the tests pass.

1. Every commit passes the full suite under both TCG and KVM.
2. Stable Rust only. No nightly features or -Z flags.
3. The ABI is defined once, in vanir-abi. The kernel and user code never define their own copy of a syscall number, code, flag or shared record, and vanir-abi stays data only (no syscalls, no unsafe, no dependencies).
4. The kernel never blocks mid-syscall. Waiting means saving registers, marking the task blocked, and writing results into its saved frame later.
5. User pointers are untrusted. Range-checked and copied with fault recovery under SMAP; never dereferenced directly.
6. Identity comes from the kernel. Servers decide who a client is with task_name(sender), never from message contents.
7. W^X everywhere. No mapping is both writable and executable.
8. Mechanism in the kernel, policy in tasks. Restart decisions belong to the supervisor; key decisions to keys/attest/the host.
9. Default deny. No IPC without sends; no device without a grant; storage locked until keyed.
10. Secrets stay out of git and out of images. keys/ is ignored; the storage master key is never on the disk.
11. Crypto uses the portable backends. The --cfg flags in .cargo/config.toml select them; removing them makes rustc crash (SIGILL) on this target.
12. The measured set is the source of truth. Kernel, manifest and task statements go into PCR 12; tools/expected-pcrs.py must compute the same value.

25Troubleshooting#

26System dependencies#

sudo apt install make git gcc binutils xorriso qemu-system-x86
sudo apt install python3-cryptography ovmf sbsigntool swtpm swtpm-tools   # boot chain, TPM
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh             # if Rust isn't installed
rustup target add x86_64-unknown-none

Nightly Rust isn't required. Limine (branch v11.x-binary) is cloned into limine/ on the first build.

27Glossary#