Scroll to run itRead on: built figures are printed at their finished state
Part I
The idea
A phone writes the next system beside the running one, boots it, and keeps it only if it works. I wanted that for servers.
Chapter 1 How a phone updates
One
An Android phone runs from one slot and writes the update into the idle one.
A slot is a full copy of the system partitions. Virtual A/B fakes the second copy with a snapshot.
An A/B update runs while you keep using the phone. A background daemon writes the next system into the idle slot, which nothing running touches, then tells the bootloader to boot that slot next time. I start with phones because they solved safe updates first; this guide asks what a server looks like under the same rule.
Two
Each boot stage checks the next before handing over, and the system is checked block by block as it is read.
Verified Boot is the chain: the bootloader checks the kernel, and the kernel checks the slot with dm-verity, a hash tree with one hash per block under a root hash the vendor signed. A block that does not match is refused. Android and iPhone arrange the system differently and share one rule: check before you keep.
Three
The new slot is kept only if it boots. Boot it, mark it successful, or fall back.
Android says several attempts and names no number, so neither do I.
The bootloader tries slot B. Once the system is up, a user space daemon runs its health checks (is the system up and responding) and marks the slot successful, Android's word; until then it is only bootable. If several attempts never get there, the bootloader marks B unbootable and falls back to A, the system the phone had before. My servers did none of that.
write the idle slot
reboot into it
mark successful, or fall back
Fig. 1 Two system slots on a phone chassis. The update lands in idle B, is hashed and verified; the selector turns to B, B takes the GOOD stamp, A fades to hidden lines as the fallback.
Chapter 2 How servers used to update
One
A kickstart laid the host down once, from a recipe, and nothing read that recipe again.
A kickstart file is a text recipe: disks, packages, users and a script that runs when the install ends. Anaconda installs from it with no prompts (Debian calls its recipe preseed, Ubuntu autoinstall). Day zero was the only day my host and its recipe agreed. After the first reboot the file stayed in a repo; the host went on alone.
Two
Every patch window after that rewrote the running host in place, and hand edits landed beside the packages.
dnf history undoes package transactions, not edits in /etc. Restoring from backup was my practice.
dnf upgrade resolved the new package set and swapped files on the live filesystem while services kept running (a patch window is the scheduled hour I applied updates). A config line or a unit dropped into /etc sat beside those packages with no record beyond the shell history. I never exercised the result before it went live, and a window that stopped halfway left a host matching neither the old package set nor the new.
Three
What the host actually ran, I learned by scanning it, after the fact.
Configuration management calls the recipe a baseline; a compliance scan (OpenSCAP is the usual one) read packages and files back from the host and reported the distance. Every drift it reported was already live, and had been since some patch window I could date only from the logs. The phone in chapter 1 had four properties this host never had.
phone
server
tried first
idle slot
live host
atomic
one flip
file by file
rolls back
old slot
backup
verified
hash first
scan after
Fig. 2 One host as a slab of 24 package tiles laid down from a kickstart card; three swapped in place at patch windows, shown in ink; one under a detail circle; the rail dimension from the last patch to the scan reads found after the fact.
Chapter 3 The trade
One
The list of what a host is did not change: packages, kernel arguments, users, units, kernel modules.
Every host I ran was those five things, each landed on the machine: a package installed, a kernel argument edited, a user added, a systemd unit (a service definition) enabled, a kernel module compiled against whatever kernel was there. What changes is where it runs.
Two
The list left the host for Containerfiles, built by buildah in a GitLab CI pipeline.
A Containerfile is a build recipe whose image goes to a registry. CI runs a build on every push.
The host does one thing with this: it pulls the image and reboots into it.
What happens when the new image fails to boot, and how the host rolls back, is chapter 27.
The image holds a whole system, kernel included, tried in CI before any host sees it. bootc pulls it, stages it as the next system to boot, and the reboot switches, whole or not at all; boot checks come later. CI logs in only to throwaway test machines. The trade: I stopped changing hosts and started changing the list, so a host I lose costs nothing I had typed into it.
Fig. 3 Five labelled slabs on the host station. They lift into a Containerfile standing at the CI station, a build block beside it, and one image travels the wire back to a host that pulls and reboots.
Part II
Containers from first principles
A container is a process with its own view of the machine. Its filesystem arrives as a stack of layers, each named by its hash.
Chapter 4 A container is a process
One
A container is an ordinary process that the kernel has shown a narrower machine. There is no second kernel.
The narrowing is done by namespaces, eight kinds today: process ids, mounts, network, hostname, ipc, users, the cgroup root and clocks. Inside, the process is PID 1 and can see nothing it did not start. podman hands the container to an OCI runtime (Open Container Initiative, the shared container standard), which asks the kernel for the shells with clone, one flag per namespace.
Two
cgroups are the budget: how much CPU and memory a container may use, and how many processes it may start.
Cgroup v2 became official in Linux 4.5, in 2016. Fedora 31 made it the default in 2019.
A control group is a directory in a filesystem the kernel keeps. Three files hold the ceilings: cpu.max, memory.max and pids.max. A new group starts with max, which is no ceiling, and cpu.max also carries a period of 100000 microseconds: the quota is how much of each period the group may run. Every systemd service gets one too.
the runtime reads the container config
it writes the limits into the group
the process runs inside that group
Three
The root filesystem is the only part that travels. Everything else here belongs to the host kernel.
The mount namespace lets pivot_root turn one directory into /. That is the root filesystem, only files, and it is all an image ships: no running kernel, no process, no limits. In cairn it is the whole operating system, copied FROM scratch. Docker showed this combination in 2013; the kernel already had the pieces. Chapter 5 takes the slab apart.
Fig. 4 A process on a root filesystem slab inside three of the eight namespace shells, cut away at the front. cgroup ceilings arrive as dimensions; then the shells and the process lift away and the slab remains.
Chapter 5 An image is a stack
One
An image is an ordered list of layers. Each layer is a tar archive, named by the digest of its bytes.
sha256 over the bytes: same input, same name, on any machine.
The OCI image format is three kinds of blob. A layer is a tar of files holding what one build step added or changed, laid over the layers before it as a union mount. A config records how to run the result and the order of the layers. The manifest lists both by digest, and its digest is the image's identity.
blob
holds
manifest
layers and config, by digest
config
run parameters, layer order
layer
a tar of files
Two
Height is layer count. The base-zfs-nvidia flavor is 64 layers, and a new host needs all of it once.
A flavor is the base image, plain or with modules added; this one adds ZFS and NVIDIA drivers: a tall stack. A host files the layers it holds by digest. A pull starts with the manifest, and compares each layer digest against that store. Only the names it does not have are fetched; the registry serves blobs by name. The first pull onto a new machine is the whole stack; every later pull is a diff.
fetch the manifest
check each layer digest locally
fetch only the missing layers
Three
In a measured same-commit rebuild, five layers changed. The host pulled about 5% of the image.
Five of 64 layers; about 5% of the bytes.
Identical files with pinned timestamps make identical tars, and identical tars have identical digests. The 59 layers whose bytes did not change keep their names, so the host already has them and skips them. On that flavor this holds because chunkah re-layers the built image by package before it is pushed, so a change lands only in the layers of the part it touches. The rebuild pulled newer upstream packages; those are the five. Turns out the stack is the update mechanism.
Fig. 5 Sixty-four layers assemble into one image; the same commit rebuilt sends five of them to the host.
Chapter 6 Building and naming
One
A Containerfile is the recipe. Mine builds an operating system's root filesystem from nothing, one block at a time.
Each block says what it does, why, and which upstream document it follows.
A Containerfile lists build steps, and buildah runs each one in a fresh container and keeps what changed as a layer. Mine starts FROM scratch: an empty filesystem. A builder stage runs rpm-ostree compose, which lays a root filesystem down from a package list, and the rest of the file adjusts that tree. Every step sits under a cf: header; the sheet shows six. I wanted a reader to audit the whole system from one file.
What
the step, in a line
Why
the reason it exists
Upstream
the document it follows
Decision
my own ruling, if any
Two
A build yields a digest, the image's identity. A tag is a name that can be moved later.
:<short-sha> is the candidate: one tag per commit, written by the build.
Content addressing means the same bytes always give the same name, so a digest names exactly what was built. A tag is a mutable name in the registry that can point elsewhere tomorrow. When I need what was tested, I pull by digest. When I want whatever is current, I pull a tag and accept that it moves.
Three
Tags name the stages: :<short-sha> the candidate, :stable once it passes the gates, :stable-sealed for sealed hosts.
Tags named keep- sit outside registry cleanup, along with :latest and :stable, which promote moves.
Each commit gets :<short-sha>. Once its build boots under the gates (the pipeline's pass-or-fail checks), promote takes the digest the build recorded, never the tag, moves :stable to it. The sealed lane, a signed build taught in chapter 22, moves :stable-sealed on the same gates. A failed gate moves none; hosts keep the last good image.
re-pull the candidate by its digest
verify it across the registry
assert the OCI media types
retag the tested digest
Fig. 6 A Containerfile feeds a build station that yields one image, named by its digest. Tags point at digests on the registry shelf and move only after the gates pass. Digests are illustrative.
Part III
The container is the OS
bootc boots a host from an ordinary container image and keeps the previous one to fall back on.
Chapter 7 bootc: an image becomes a booted host
One
A bootc host keeps its root trees side by side on one filesystem, two at rest; the loader picks by entry order.
Cairn hosts boot composefs (Part V), the read-only root mounted from this store; the picture holds either way.
bootc boots a container image as the operating system and by default stores it as an OSTree deployment: a whole root tree as a directory, hard-linked into one repository. A is the running tree, B the next. Both share one stateroot with one /var. Each deployment gets one Boot Loader Specification entry; the bootloader, which starts the OS, takes them in order.
Two
bootc upgrade pulls the next image beside the running one and stages it; nothing running changes.
bootc upgrade fetches the image and checks it out as a new deployment; bootc status shows it staged. The /etc merge and the boot entries wait for shutdown, when OSTree's finalize service completes the staged deployment and writes the entries, B first. The reboot is the whole switch: the loader starts B, while A sits untouched. A host already holding a rollback carries three, booted, staged and rollback; finalize drops the oldest.
status
tree
boots
booted
A
now
staged
B
next, once finalised
rollback
A, once B boots
on bootc rollback
Three
bootc rollback reorders the two boot entries. Nothing is copied back, because A was never touched.
Chapter 27's opt-in health gate calls bootc rollback after repeated failed boots.
bootc rollback reorders the entries and creates no deployment, so A boots as it last ran and B becomes the rollback. OSTree keeps a writable /etc per deployment, filled by a three-way merge at finalise. Edits made while B ran stay in B; A's /etc returns as A left it. /var is shared and never moves.
Fig. 7 One disk cut away: trays A and B on one filesystem slab, a shared /var, two boot-entry tabs in front. The next image lands in B and the tabs put B first; rollback reorders them back and B fades to hidden lines.
Chapter 8 Base from scratch
One
The base image starts in a throwaway builder stage, where rpm-ostree composes a root filesystem from a treefile.
rpm-ostree's docs say its focus is moving to bootc and dnf; the compose is the part I use.
The last stage is FROM scratch: an empty image and a copy of the composed root. Then bootc lints it.
bootc container lint --fatal-warnings runs in the pipeline's build job, before the push.
There is no parent image. FROM scratch is an image with nothing in it; a COPY from the builder puts the composed root at /. Every file is one the build asked for: a treefile package, a repo overlay file, or a Containerfile step. Then bootc container lint checks the layout bootc expects, and any warning fails the job.
compose the root in the builder
COPY it onto FROM scratch
bootc container lint, warnings fatal
Three
One composed root, several flavors. The other four add what their names say. base-vm and base-zfs-nvidia earn their tags by booting; base-zfs and base-nvidia earn :stable on build and gates.
base-vm boots as a VM guest and earns :stable only, as the README states.
cairn/base publishes these flavors; flavors.yml names each one's parent and build directory. base is the root above. base-zfs adds source-built OpenZFS kernel modules, base-nvidia the GPU modules, base-zfs-nvidia both, base-vm VM-guest support (Chapter 9 builds the modules). base and base-zfs-nvidia are sealed rows: each earns :stable and :stable-sealed by booting its own sealed artifact, a signed, measured-boot build (Part V), under the gates of Chapter 6. base-zfs and base-nvidia earn :stable on build, lint and gates, and the instance layered on one seals and boots its own. base-vm earns :stable on a VM boot.
Fig. 8 A treefile feeds a hatched builder; the composed root lifts out, lands on the empty plate, clears the lint gate and roots a tree of flavors.
Chapter 9 Modules ahead of time
One
A kernel module is compiled against one kernel build and fits no other; the match is made somewhere.
The builder is Fedora's bootc image of the same release, with kernel-devel pinned to that version.
A loadable kernel module is code the kernel links in, built against one kernel's headers; the kernel refuses one built for another. DKMS keeps the source on each host and recompiles after every kernel update. I make the match once: the pipeline reads the kernel version out of the image it just composed and hands it to the module builder. When Fedora's kernel is newer than OpenZFS supports, the ZFS rows hold an older one and the modules are built for that.
Two
The modules are compiled, signed and checked against a committed file list in one pipeline, never on a host.
Decision D29: the modules sign with the boot image key, one signer to trust.
OpenZFS and the NVIDIA open modules live outside the kernel tree, and Fedora ships neither. The builder compiles them; sign-kmods.sh runs the kernel's own sign-file over each .ko and fails on one without the trailer, the appended signature block. A manifest gate diffs what each layer ships against its expected.txt; a path added or missing fails the job.
object
the module code the kernel links
signature
PKCS#7 over the object, appended
marker
~Module signature appended~
Three
At boot the kernel checks each module's signature against keys it trusts; the CI boot asserts that it does.
The GPU module is signed, its signature accepted, never exercised: the runner has no GPU.
Module signing checks against the kernel's builtin and secondary keyrings (its lists of trusted keys). Our signer is the boot image's signing key (chapter 23), and Fedora's kernel trusts the platform keyring, where the certificates in the firmware's db, its trusted-certificate list (Part V), land. With lockdown on and signatures enforced, an unsigned module does not load; assert-sealed.sh asserts both in the CI boot.
Fig. 9 A module milled to fit the kernel's keyed recess, dimensioned to the same build; stamped and gauged against the list; set into the OS layer while an unsigned ghost meets a closed gate.
Chapter 10 Instances
One
A host repo is tiny: one FROM line on a base flavor at :stable, plus one machine's config.
example: four lanes, all FROM base:stable; fork it, keep one.
The base README's fork and roll is the recipe: FROM a flavor, then a kernel argument file. An account or a unit goes in the same way. The instance component adds four jobs: lint, detect a base change, build, promote. It never boots the image: a fork rides a tag the base earned, so a green build is its promotion. The sealed-instance and variant components boot theirs, and can run a host's own script in the booted guest.
Two
whetstone, the image CI runners boot, bakes only what can be public; its SSH key arrives after first boot.
whetstone: the job user is in the image; its SSH key arrives as a credential.
whetstone bakes its file list: tmpfiles, sudoers, a package repo, units, an account. Signed modules and the signer's public half come with the base. A private host layers users, keys and services on the same shape and keeps that layer private; the seam is verification of a promoted digest. The module signing key lives in a masked CI variable (a secret CI holds and hides in logs) and never reaches a layer.
Three
Public instances share one shape. The flavor picks the base; the repo adds the rest.
The instance list is a protected CI variable; no public file names a private host.
The example's four lanes, plain, attended, edge and evidence, are build variations of base, each adding something different. whetstone and roon build on base-vm, and each bakes a qcow2 with the qcow2 component before promote, so a disk that fails its boot smoke holds :stable; whetstone's is the disk the CI runners boot. A promoted base rebuilds each registered instance; the cascade stops there, and no CI reaches a running host.
Fig. 10 A thin instance slab on a 40-unit base; a first-boot line splits what the build bakes from what arrives after. Then the host shrinks into a rack: four lanes, whetstone and roon dashed, private host hatched.
Part IV
CI is the factory
Every metal base image is built, sealed, booted and graded in GitLab CI before a host may pull it. An instance lane builds on that and promotes.
Chapter 11 GitLab CI primitives
One
A pipeline is one YAML file turned into jobs, in stages.
A runner is the machine a job lands on. Tags choose which one.
Lint and detect-changes run on GitLab-hosted runners; the component hardcodes that.
A runner polls GitLab, takes a job and runs its script. Tags name which runners may take it. The instance component asks for two: runner_tag for build, which must serve merge request (GitLab's pull request) pipelines, and promote_runner_tag for promote alone, a runner set to take jobs only from protected branches. Both of mine are self-hosted.
Three
The file writes no jobs. It includes a component at a major.
The example pins a major; the releases page says what that resolves to.
An include pulls YAML from elsewhere. A component is an include with declared inputs, released from a catalog on tagged versions. An instance’s file is includes and inputs, a base image among them. instance@N resolves at parse time to the newest release of major N, so a patch reaches every consumer with no commit in any of them.
Fig. 11 The CI file as a plan strip beside a rail of three stage plates, a job cube on each. Two runner stations wire to the build and promote jobs; a component cartridge seats in the rail and the jobs hatch as its output.
Chapter 12 The catalog
One
The pipeline is a catalog: a set of components, each a family of jobs a repo includes.
example includes three: supply-chain, instance and preset.
A CI/CD component is a pipeline fragment with declared inputs, published from a project GitLab lists in its catalog. cairn/pipeline publishes components that build and promote images (base-build-scratch, layer, promote, variant), seal and disk them (seal, sealed-instance, qcow2; chapters 14 and 22), and run an instance's pipeline (instance, supply-chain, and preset, which ships the shared stages and workflow rules). An instance repo includes the jobs it needs at a major version and names its base image.
Two
A version is an annotated tag. A consumer pins the major and takes every minor and patch for free.
Tags follow semantic versioning: patch for a fix, minor for an addition, major for anything that breaks a consumer. Each tag's message becomes a release. @N is a partial version: it resolves to the newest N.x.x release when the pipeline is parsed, so a minor reaches every consumer with no commit. Only a major asks them to move.
Three
A major is one transaction. Every consumer bumps together. A repo left behind has drifted.
Chapter 17 is how the bump gets proposed.
A major removes inputs the last minor before it had already stopped reading, and the changelog names the migration: stop passing them, bump the pin. The rule in CONTRIBUTING is one commit in every consumer, in one transaction. The public consumers, base, example, whetstone and roon, all sit on the current major.
Fig. 12 One sheet, three views: the core components as an exploded parts list, a release ladder with old and new major brackets, and four consumer slots that clear together as one major lands.
Chapter 13 The cascade
One
A base flavor promotes. One job rings every instance that builds from it.
Base rings the doorbell, never watches. The daily schedule builds nothing, so its ring carries nothing new.
Each flavor row gets a trigger-<image> job that waits for that row's promote jobs: :stable and :stable-sealed on a sealed row, :stable alone on the others, base-vm among them. It posts a pipeline trigger to each entry that builds from the flavor, with its own job token, the short-lived credential GitLab gives each running job; each instance lists base in its token allowlist. The token dies with the job, so there is no standing credential to rotate. Base never reads the pipeline it started.
Two
Each instance rebuilds in its own pipeline. Base never learns how it went.
A refused ring prints per entry, and the trigger job goes red once every entry has been tried.
The list is a CI variable, CASCADE_INSTANCES, not a file in the repo, so the fleet's membership and count stay private. Each entry names a project and the flavor it builds from. The instance component resolves that flavor's :stable to a digest, builds its thin FROM, and moves its own :stable on a green build. Every lane builds the same way.
Three
The cascade stops at a promoted image. No CI reaches a running host.
Running it, cairn docs: the host upgrades on its own timeline.
The instance's :stable waits in the registry. An operator runs bootc upgrade on the host; the host pulls the image, stages it beside the running one, and the reboot swaps them. The request starts at the host. Chapter 26 is the upgrade.
Fig. 13 A base press promotes, a bell rail rings one tap per instance station, and each station sets its own tag on the registry shelf. A hatched keep-out, dimensioned "no push wire", stands between the shelf and the host the operator pulls to.
Chapter 14 The loop
One
Every image-changing push to base's main builds a candidate named by its commit, then seals it.
Sealing is chapter 22, its keys chapter 23.
A GitLab pipeline builds the image and pushes it to the registry as :<short-sha>. The seal job signs a unified kernel image (UKI) with the root filesystem's digest written into it and pushes that as :<short-sha>-sealed. Nothing here touches a stable tag.
Two
The sealed candidate boots in a nested VM, then walks an upgrade and back.
The upgrade walk gates only the flagship flavor, base-zfs-nvidia.
sealed-boot.sh installs it to a disk, boots it under enforcing Secure Boot with custom keys and no shim, and checks what booted. upgrade-path.sh boots the published image, switches to the candidate, reboots, checks that a data mount is untouched, then rolls back and checks again.
Three
All green, promote re-reads the digest, verifies it again, and moves the tags.
:stable-sealed moves the same way, behind the same boot.
promote takes the digest its own build wrote to an artifact, never the :<short-sha> tag, which another pipeline could overwrite. It runs the image's own verify after the registry round trip, retags :latest and :stable, and reads each back to check the served digest. A red gate skips promote and the last good image keeps the tag.
Fig. 14 One pipeline as a belt with five stations here: build, seal, boot, upgrade path, promote. The candidate slab rides the belt to a gate that shuts on any red; past it, the registry rack moves :stable to the candidate.
Chapter 15 Booting inside CI
One
Booting the image is the test, on a runner that is itself a virtual machine built from the image.
Nested KVM: a virtual machine running inside another virtual machine.
Every job gets a fresh runner VM. Its disk is a whetstone image, built from base, with QEMU, swtpm and a Secure Boot build of OVMF baked into the image, so the job installs nothing before it boots the candidate. Under the runner is a lab host, drawn here as a slab. This gate earns the sealed tag.
Two
Inside the job, a second machine: enforcing Secure Boot, a software TPM.
On merge requests the harness writes PK, KEK and db (the key databases) itself, and only db authorizes a boot. On protected refs, the ones that earn tags, the firmware starts in Setup Mode and systemd-boot enrolls the set a real host gets.
The harness mints a throwaway PK and KEK on merge requests, puts the UKI's signing certificate in db and writes the firmware variables with virt-fw-vars. Secure Boot is enforcing, with no shim in the chain. The TPM is swtpm on a socket. The candidate is installed to a blank disk with bootc and boots.
Three
The gate reads the serial console: login, then assert over SSH.
The login wait polls for up to thirteen minutes, because nested KVM boots slowly.
It waits for login: or a firmware refusal on the serial line, then runs assert-sealed over SSH: Secure Boot on, the root's composefs digest equal to the one signed into the UKI, the module signer (the key modules are checked against) in the expected keyring, SELinux enforcing with a policy that still matches the checksums recorded in the image. The boot prints one verdict line; it must pass, or :stable and :stable-sealed freeze.
Fig. 15 A runner VM frame on a host slab; inside it the candidate on its disk beside firmware, variable store and software TPM, and a serial wire out to a readout of one boot and one verdict.
Chapter 16 A gate that never fails proves nothing
One
A red gate keeps the tag where it was, and I never retry a red boot.
base's default retry lists runner_system_failure, stuck_or_timeout_failure and api_failure. A failed script is not there by default.
A failing gate skips promote, and the last good digest keeps :stable. CI can retry a job; I let the boot gate retry only when the runner broke or the job stalled, never when the check failed. A flaky boot could pass on a lucky second try, and the green would mean nothing.
Two
Decide what a gate must prove, then break exactly that on purpose.
A probe the old code also survives is blind; the merge request waits until it can tell.
A mutation probe is a negative control: I make a check that always passes, and the suite has to go red or it proves nothing. base runs one in lint on every merge request: it rewrites the harness's check function, ck, so it always passes, runs the suite against the mutant, and fails if nothing failed.
Three
The health gate's probe runs in a lab VM, where I can break the host.
The health gate ships off: no systemd preset line, so it stays dark until a host enables it.
Enabled, the gate counts failed boots and calls bootc rollback at a threshold; with nothing to roll back to it holds. No CI job enables the gate, so I prove it by hand: a throwaway libvirt VM boots a bad candidate with bootc rollback swapped for true, and the health gate has to hold rather than loop. The gate itself is chapter 27.
Fig. 16 A boom gate on a rail between the candidate slab and the :stable plinth drops on a red run; the boom swings open and a hatched mutant slab lands on the rail past it; the probe moves to a lab bench, where the health gate holds.
Chapter 17 Keeping it moving
One
Renovate watches every pin in the stack, from one shared Renovate preset.
My own images skip the wait; kernel-module pins merge at every size.
Renovate is a bot that opens a merge request when a pin has a newer release. It reads each repo's renovate.json, which extends the estate preset. Third-party updates wait seven days; minor and patch bumps merge themselves behind a green pipeline; a catalog or runtime major waits for me. Regex managers read the pins no built-in manager knows: the ZFS release and the GPU driver tag.
Two
A catalog major is one transaction: one commit in every consumer.
The example includes its components at the catalog's current major.
The catalog is released on semver tags, and consumers include componentsby major, so patch and minor releases apply with no edit. A major is an edit in every consumer, which the catalog's contributing rules ask for in one transaction. The preset groups the major into one merge request per consumer.
Three
A Fedora release is capped until I raise it; then four pins land in one merge request.
The number lives in base's Containerfile; the preset caps what Renovate may propose.
Base pins the Fedora release twice, a variable in its CI config and an ARG in its Containerfile. The ZFS and GPU driver module builders default to fedora-bootc at the same number. The preset caps the release tag, since a new Fedora is a new kernel the ZFS and GPU modules must build against; I raise the cap when they do, then one rule groups every Fedora pin into one merge request.
Fig. 17 A base block and a row of consumers on one preset plate. A gang bar presses every consumer's register from one major to the next in one move; then a detail beside the base shows the Fedora release reaching two pins in base and one in each module builder, from one source.
Part V
Trust
Trust starts in silicon and firmware, long before Linux runs. Measuring the chain is how I know what booted.
Chapter 18 Before the OS
One
The first code a machine runs is burned into the silicon.
The trust anchor sits below the firmware: it runs before SEC, the firmware's first phase.
At power-on reset the processor fetches from a boot ROM in the silicon, a mask ROM fixed when the chip was made. Nothing on the board can rewrite it. Boards that verify their firmware check its signature from here, before the first instruction of it runs.
Two
The board's firmware sits in a flash chip wired to the processor.
Setup Mode: firmware with no Platform Key accepts new keys; delete it and systemd-boot enrolls mine.
UEFI firmware runs from that chip and brings the board up through POST. Its variables live in the same chip, the Secure Boot keys among them. A firmware update or flashrom rewrites it. Enrolling my own keys takes one step at the firmware console; the keys themselves are files I ship.
Three
A card can bring its own firmware, and it runs before any OS exists.
The BMC is a second computer with firmware of its own. One script fetches the vendor certificates.
A GPU's pre-boot driver is an option ROM, signed through Microsoft's CA, so the firmware's list of trusted certificates (its signature database, chapter 19) keeps those CAs beside my own certificate or the card's firmware stops loading. The OS takes none of this on trust. It gets an event log the firmware wrote about itself.
Fig. 18 Firmware from silicon to card, on one board, and the plane the OS sees, dimensioned "report only".
Chapter 19 A short history of Secure Boot
One
A BIOS ran whatever the first sector held. UEFI checks a signature before it runs anything.
A BIOS loads a disk's first sector and jumps into it; the sector can hold anything. UEFI firmware keeps four signature databases: PK names the owner, KEK says who may edit db and dbx, db says who may boot, dbx says who may not. A loader runs only if db allows it and dbx does not forbid it.
Two
Firmware made for Windows carries Microsoft's certificate, so Linux boots through shim.
Microsoft's UEFI CA signs shim, a small first-stage loader. Shim carries the distribution's own certificate and a machine-owner list, and checks the next loader and the kernel against those. The firmware trusts Microsoft, and shim trusts the distribution.
Three
A signed hole stays signed. Revoking it means a dbx entry for every binary that has it.
BootHole was a GRUB bug in 2020. BlackLotus in 2023 booted old, signed Windows loaders that nobody had revoked (CVE-2023-24932). A dbx hash entry forbids one binary, so shim added SBAT, a generation number that revokes a whole series at once. Microsoft's 2011 certificates begin expiring in June 2026; machines keep booting but stop getting boot-level fixes (Microsoft). cairn ships its own PK, KEK and db, systemd-boot enrolls them from Setup Mode at install, and there is no shim.
Fig. 19 Two loaders on one sheet, the UEFI one gated, with the four databases tabled. Then the chain from Microsoft's certificate through shim to the kernel, shim in detail. Then a dated line from 2011 to 2026, BootHole's 150 dbx hashes against one SBAT entry, and cairn's two-plate chain beneath.
Chapter 20 What a TPM is
One
A TPM is a small second computer on the board that keeps keys and registers; the host can only ask it.
A discrete chip, firmware beside the CPU, or software: the same commands.
A TPM has its own processor, a random number generator, a key engine, a little non-volatile memory and a bank of PCRs. Commands go in and answers come out; the host never sees inside. A discrete chip resists probing best, though the wire to it can be tapped. My CI boots against the software kind, swtpm.
Two
Its keys grow from seeds it holds: an endorsement root and a storage root, with everything else below them.
The specification calls them hierarchies; there are platform and null ones too.
The endorsement root identifies the chip, and its maker certifies it. The storage root wraps the keys I create, so they can sit on disk and only this chip can open them. Keys made under a root are not stored inside; they leave wrapped and come back to be loaded. Clear the chip and the storage root changes, which orphans everything under it.
Three
You hand it a secret and a rule; it gives the secret back only while the rule holds.
That is sealing. The rule is a policy over register values or over a signer, which chapter 21 draws. The chip checks the policy against its own registers, so a changed boot gets no secret and no argument. The same seal and release runs in CI against the software twin before it reaches a machine: sealing.
Fig. 20 A TPM with its lid lifted: processor, randomness, key engine, memory and PCR bank. Then a seed growing two roots, and a secret sealed in and released through a gate, with a software twin beneath.
Chapter 21 PCRs
One
A PCR is a register you can only extend: the new value is the hash of the old value and the measurement.
A hash chain: replay the log and the register comes out again.
The TPM is a small tamper-resistant chip on the board that keeps keys and these registers and releases a secret only while its policy holds. It holds a bank of them. Firmware and the boot loader feed measurements through extend, and the event log records what went in, in order. The final number depends on all of them.
Two
The PC Client map gives each register a job, so a policy can name what it cares about.
Rows 6 and 8 are the TCG PC Client profile's; systemd leaves them undefined.
Firmware measures its code into 0 and its configuration into 1, option ROMs into 2 and 3, the boot loader into 4 and the partition table into 5. The Secure Boot policy lands in 7. systemd-stub measures the UKI into 11; a command line passed in from outside lands in 12; the machine's identity goes in 15. systemd-cryptenroll takes registers per slot. LUKS is Linux's disk encryption; a slot is one way to unlock its key.
Three
A policy on a value breaks on every kernel update. A policy on a signer survives it.
Every new UKI measures to a new PCR 11, so a value-sealed slot needs re-enrolling per update. systemd-measure instead predicts, at build, the PCR 11 the new UKI will produce and signs it with a key I hold. The signature ships inside the UKI as a .pcrsig section, and the slot is enrolled to the public key. At boot the policy asks whether the measured value is one that key signed.
part
holds
UKI
.pcrsig: the signed expected value
LUKS slot
the signer's public key
TPM
PCR 11 as measured
Fig. 21 A register sheet: one measurement extends PCR 11, the PC Client map assigns the registers, and two policies meet a kernel update.
Chapter 22 Signing vs sealing
One
A signature says who built the image. It says nothing about what is on the disk now.
The firmware checks the loader and the UKI against our signing key in its db.
Secure Boot checks the signature on systemd-boot and on the UKI before either runs. The UKI is one PE file: kernel, initrd, command line and the policy sections, signed as a unit with our UKI signing key. The firmware's job ends there. Nothing it checked says what the root filesystem holds.
Two
The image seal pins the booted root to the signed image.
The seal recipe makes the image seal, bootc's term, not chapter 20's TPM sealing. It runs bootc's ukify over the root, puts the composefs digest of the whole tree on the command line, then signs the UKI. The root mounts with verity=require. The kernel checks every object it reads against the hash the digest names, so a changed byte fails to read. The digest is signed with the kernel.
Three
Whether the disk opens turns on who signed the boot.
The seal proves the bytes. The unlock asks who signed them.
The UKI also carries a PCR 11 policy signed by our policy key, so on a TPM-unlock host the LUKS2 slot opens for any boot that key signed, across kernel updates. A boot it did not sign leaves the slot shut and the host on the passphrase I set at install. In CI, sealed-boot boots the image under enforcing Secure Boot with no shim, and assert-sealed checks the digest on the running command line against the root object on disk.
Fig. 22 The sealed stack: a four-section UKI under a signature band, systemd-boot above it, the root digest dimensioned down to the composefs slab, one object refused by its fs-verity tree, a LUKS2 slot opened by the policy key.
Chapter 23 Your own keys
One
I wanted CI to sign everything, and the firmware to trust that signer directly.
The mint script picks RSA-4096 for PK, KEK and UKI, RSA-3072 for PCR 11, fifty years of validity.
Turns out one script does the whole mint. It is a key ceremony on an air-gapped box: it reaches no network and enrolls nothing. Out come up to five private keys, three signed payloads for the firmware, the blobs I paste into CI variables, and the public halves. Secure Boot firmware never checks certificate expiry, so the validity is a choice.
Two
Two keys own the firmware and stay offline. Three sign in CI.
I enroll db first, then KEK, PK last.
The Platform Key signs the payload that carries the Key Exchange Key; the Key Exchange Key signs the payload that carries the signature database, db. Both keys go to offline storage and never enter CI. The UKI signing key, the PCR 11 policy key and the cosign key ride as masked, protected CI variables: hidden from job logs, handed only to pipelines on protected branches or tags.
Three
One certificate covers systemd-boot, the UKI and the kernel modules.
A fork mints its own set and never reuses ours.
cairn ships no shim, so db holds one cairn certificate beside the vendor CAs, and the firmware trusts it directly (D29). Out-of-tree modules are signed with the same UKI key, and the kernel's module signing checks them against it. The PCR 11 key's public half rides inside every UKI as the .pcrpkey section; chapter 24 picks it up. The cosign key signs promoted image digests, opt-in per repo. Every public half, with its fingerprint, sits at /keys/ (the trust chain).
Fig. 23 Five keys lifted off one tray, sorted into an offline bay and a CI bay, then the three CI keys wired to what each signs.
Chapter 24 Measured boot in cairn
One
The root slot is enrolled to a public key. The TPM checks a signature, not a value.
In CI the seal recipe signs the expected PCR 11 value of each UKI with systemd-measure; the signature ships in the image's .pcrsig section, the public key in .pcrpkey. On the host systemd-cryptenroll binds the root slot to that key, and systemd-stub hands the signature over at boot. The slot opens for any boot state the key signed.
Two
A kernel update moves PCR 11. The same key signed the new value, so the slot still opens.
Chapter 21's value policy would shut here. D21 in base records why the trusted key is a committed file.
The key the slot trusts is the public half of the key that signed that image's policy. Base commits it as pcr11.pub.pem and promotion fails unless sealed-boot finds the image's .pcrpkey matching it. Enroll any other key and the TPM refuses; the host falls back to the recovery passphrase the install adds first.
Three
Then I dropped TPM unlock for root: the policy proves what booted, not where the box is.
A machine carried off boots the same image and unlocks itself. So root on my hosts now unlocks from a Tang server, a small server that hands a machine its key only while it is on my network (chapter 29), the path base ships as tang-initrd: a clevis binding on the LUKS2 slot, the passphrase slot kept, the TPM slot gone. The signed-policy TPM path stays the unattended option, the edge lane of chapter 34.
Fig. 24 An exploded host beside a PCR 11 register. The signature rides from the UKI's .pcrsig tab through the TPM to the root slot and opens it; a second image moves the register and the same key opens it again; then the host leaves its shelf, the TPM is hatched out and a wire leaves for Tang.
Chapter 25 Where it goes
One
Today every file is checked as it is read.
The root is ext4 because composefs needs fs-verity and XFS has none.
The seal signs a composefs digest into the UKI's measured command line. At boot the root mounts with verity=require, and the kernel's fs-verity checks every object it reads against the hash that digest's image records for it. A changed byte fails the read (sealing).
Two
The next step refuses to run what was never verified.
A script counts only when its interpreter asks; the IPE docs say so.
IPE, a Linux security module, checks every executable against a signed policy before it runs. A policy can trust code from a dm-verity volume or an fs-verity file and refuse the rest. It is not in what I run; composefs tracks the integration in issue 360.
Three
Red Hat ships the same chain, as a technology preview.
Their words: this is where OS management is heading.
Sealed images embed the composefs digest in the signed UKI at build time and verify every file against it at runtime, chained from Secure Boot with custom signatures: "the system refuses to load any OS file that does not match." They require bootc's composefs backend, which is what cairn runs.
Fig. 25 Four slabs on a ladder: build, boot, read, run. The checks reach read today; the run slab arrives as a dashed outline with a gate; a second stack rises beside ours with one dimension between them.
Part VI
Day two
An update is a pull and a reboot. A health gate decides whether the new deployment stays.
Chapter 26 Updating
One
The cascade ends at the registry. Nothing in CI touches a running host.
Two slots is usual. bootc keeps a list; a pinned deployment, held as a fixed rollback point, makes a third.
A promoted base rebuilds each instance in its own pipeline. What comes out is a tag in the registry, :stable-sealed, which a sealed row earns by booting its own sealed artifact under the gates. From there the host does the pulling, on its own timeline.
Two
I run bootc upgrade. The image lands in the spare slot, staged.
A same-commit rebuild sent about 5% of the image in chapter 5.
bootc upgrade downloads the layers the host lacks and queues the deployment; bootc status shows it as staged beside the booted one. Today the swap is made at shutdown, and the manual says that is likely to change so that only bootc upgrade --apply applies it. In the evidence lane the pull also checks a signature policy the lane baked in.
Three
Reboot. The bootloader boots the staged slot and the old one stays as the rollback.
I reboot by hand; --apply would do it for me.
bootc rollback changes the bootloader entry order: the previous deployment is queued for the next boot and the current one becomes the rollback. That is one command and nothing is copied, because both deployments were on disk all along (chapter 7). Whether the new boot is healthy enough to keep is chapter 27.
Fig. 26 A host tray on the bench, cut open to two deployment slabs. The promoted image sits in a registry block across the line where the cascade stops; a pull carries it into the spare slab, and a lever on the tray wall is the reboot.
Chapter 27 The health gate
One
A new image boots behind a gate. Every required check passes before systemd calls the boot complete.
Opt-in, built on systemd's automatic boot assessment. systemd runs a host as units, config files that each start one job; a target is a named milestone units order themselves around. A boot is good once boot-complete.target is reached. bootc writes no boot count for the loader to retry on, so the gate keeps its own (decision D40). The check unit runs every script in required.d before that target; wanted.d scripts are logged only (run-checks.sh). A host picks checks with symlinks; one required check ships.
Two
A failed check is a strike. At three, bootc rolls back to the previous deployment.
max-boot-attempts, default 3; the count lives in /var, shared by both deployments.
The check unit's OnFailure= starts the accountant, on-failure.sh. It adds one to the count and, under the threshold, reboots to retry. At the threshold it runs bootc rollback, which only reorders the boot entries, and reboots into the previous deployment. Nothing is copied back: the failed image stays on disk, and a breadcrumb records the rollback.
check fails: count +1
under 3: reboot and retry
at 3: bootc rollback, reboot
Three
A healthy boot zeroes the count. With nowhere left to roll back, the host stops and waits.
I tried rollback and hold in a lab VM, not on metal.
confirm.sh resets the count, and its unit hangs off Requisite= boot-complete.target, so a failed boot can never start it. Two cases hold instead of rebooting: a host with one deployment, and a failure that persists after a rollback, since rolling back again lands on the image that already failed. A host held at the console beats a boot loop.
Fig. 27 A boot rail with a lift gate: pins seat and the bar lifts; a failed check lifts a pin and fills the register; a confirmed boot zeroes it and a hold cuts the loop.
Part VII
Landing keystone with Tang
Disks unlock from a key server that never holds the key, and only while I say so.
Chapter 28 Disk encryption
One
One key encrypts every sector. The header keeps that key, wrapped.
On disk the volume key lives only in the header.
dm-crypt encrypts every sector of a volume under one volume key. LUKS2 stores that key in a header at the front of the device, wrapped under a secret, so the secret can change without re-encrypting the data. Opening the volume unwraps the key once.
Two
A keyslot wraps the same key under a different secret. Any one slot opens the volume.
LUKS2 holds up to 32 keyslots. Lose every one and the data is gone.
A passphrase slot runs the typed words through Argon2, slow on purpose. A token is a header record that says where a slot's secret comes from. clevis binds this way: it makes a random secret, adds it as a slot, and keeps that secret in the header, encrypted by the pin it was bound with. The Tang pin is chapter 29.
Three
Every volume I run keeps two slots: a recovery passphrase I type, and a clevis token bound to Tang.
The install leaves a TPM slot with no PCR policy: the machine unlocks itself for whoever boots it.
base ships tang-initrd in every flavor and changes nothing until a consumer image calls it; then root is clevis-bound, passphrase slot kept. I install in a fixed order: the passphrase goes in first and is proven at the console, clevis binds its token, and once a boot has unlocked from Tang I wipe the TPM slot. If keystone is unreachable the host waits at the passphrase prompt: losing the server costs a passphrase entry, never data.
Fig. 28 An encrypted disk cut open: hatched data slab, header tray with one wrapped key, three keyslots risen. The TPM slot is struck out; the clevis slot's wire leaves for Tang.
Chapter 29 Network-bound unlock
One
Tang publishes a public key. The client fetches it once, at enrol time.
GET /adv is the advertisement; POST /rec/{kid} is one answer; kid is the key's id.
Tang is a small HTTP service. GET /adv returns a signed set of public keys. One is the exchange key S; its private half s never leaves the server. Nothing is stored per host: the whole server state is its signing and exchange keys.
Two
Binding: a random key, one elliptic-curve step, and the private half is thrown away.
clevis picks a random scalar c (a large secret number) and its public point C = c·G, where G is the curve's fixed base point. One ECDH step with the server's key gives K = c·S, and K wraps the LUKS key. Then c and K are deleted. What stays on the disk is C, S and the wrapped blob; K needs c, which is gone, or s, which only the server has.
Three
Unlock: the client blinds C before it asks, so the server works on a point it never saw.
At boot the client picks a fresh random e and sends X = C + e·G. The server returns Y = s·X, which is s·C + e·s·G, which is K + e·S. The client subtracts e·S and has K back. The server learned nothing: X is random to it, and so is Y.
Four
The server never holds the LUKS key. Deleting s on the server is the revoke.
s alone unlocks nothing: K also needs C. C and the blob alone still need the server. keystone runs Tang as a Cloudflare Worker, a small program Cloudflare runs on request, with no server of mine to keep; a revoked key set answers 404.
where
holds
server
s
disk
C, S and the wrapped blob
nobody
c and K, until the next boot
Fig. 29 Client and server as a mechanism: the public key travels once, binding discards the secret, and the blinded exchange returns K.
Chapter 30 The initrd network
One
Two kernel arguments make the initrd bring up a network link before root exists.
base's script, run by a consumer image: ip=dhcp makes the generator write a catch-all .network, masked in the root.
The initrd is the small filesystem the kernel unpacks into memory and runs first, before the real disk is mounted; dracut builds it and reads rd.neednet=1, set with ip=dhcp as bootc kargs. It then runs systemd-networkd. wait-online is told --any, so one configured link is enough, and a DHCP lease on it is all I ask for.
Two
A throwaway stage builds the initramfs, so the final image carries no NetworkManager.
resolved takes DNS from the lease. That is the script's comment, not something I traced on the wire.
In base's script, clevis-dracut needs dracut-network, which needs NetworkManager. A builder stage installs them, runs dracut, and keeps only the initramfs. The modules are base's own list plus clevis and its pins, networkd, resolved, dbus-broker and the CA bundle libcurl reads. The Tang address is an https name.
Three
Root waits for Tang with no timeout, and the passphrase prompt stays up.
A drop-in sets JobTimeoutSec=infinity on the root device unit, in the initrd only. I set no rd.luks timeout, because that drops to the emergency shell. clevis POSTs to /rec over HTTPS; when the key comes back, root opens.
Fig. 30 The initramfs. A lease crosses the link, the lid lifts on the modules dracut packed, and a request leaves for Tang while the root volume waits.
Chapter 31 keystone: Tang on a Worker
One
A Tang server with no LAN behind it: one Cloudflare Worker, many named key sets.
A set is two private JWKs (JSON Web Keys) on one curve: a signing key and an exchange key.
Hosts with no tangd nearby can unlock anyway. keystone speaks the protocol of stock tangd from a Cloudflare Worker, for clevis to bind against. Each set lives in one Worker secret, TANG_KEY_<NAME>, and is served at /<name>/adv and /<name>/rec/<kid> (key recovery). A name with no secret behind it answers 404.
Two
Write-once: a name keeps its keys for life, and rotation is a new name.
P-521 by default, as tangd-keygen mints. P-256 fits the Free plan's CPU limit.
keyset mints, checks, deploys and revokes a set, and deploy and revoke only plan unless given --apply. deploy refuses different keys under a live name. Once written, the Cloudflare API cannot read a secret back, but any Worker version the account deploys can: whoever holds the account holds the keys.
mint and deploy a second name
bind the disk to it, reboot
unbind the old slot
revoke the old name
Three
Revoke means delete the secret. From then on, rec answers 404.
Its disks stop unlocking; a host already unlocked keeps its disk key. Worker versions carry their secrets, so a rollback past a revoke serves the set again, and wrangler secret list will not show it. You keep a marker for each revoked name in your offline key store, a record that its URL must answer 404, and keyset deploy checks it still does.
Fig. 31 A Worker slab with three secret bays. A rec packet arrives and a key set seats in bay one; a second seals beside it, write-once; then the first is pulled, a 404 leaves on the wire, and a ghost of it stays, held by an older version.
Chapter 32 The origin gate
One
A set names the networks it serves unattended. From inside one, rec is answered at once.
I pin both families, the IPv4 /32 and the delegated IPv6 prefix, since clevis may use either at boot.
allow lives in the set's secret beside its keys: a short list of prefixes, where I put the host's own egress so it boots with nobody watching. A set with neither allow nor approvers answers any client, as tangd does, so the gate is a choice I make per set in keystone's setup.
Two
A stranger is held. Each approver gets a link; typing the set's name there admits that network for a few minutes.
A set holds only a few strangers at once, and mail to approvers is rate-limited.
The Worker answers 403 and files the origin in one Durable Object, which mails each approver. Approved, the /24 or /48 is served for a short grant, and the held host unlocks soon after only because clevis-luks-askpass keeps retrying; a one-shot client just fails. A request nobody answers expires on its own.
Three
Everything else is refused before any private key is read.
It is not authentication: whoever reads an approver's mailbox holds that power until the request expires.
No approvers, a stranger past the hold limit, a denial, or egress through Cloudflare's own network: the same 403, decided in index.js before the key exchange runs, so a refused client never touches the key. A revoked set answers 404, and a host that already unlocked keeps its disk key.
Fig. 32 A keystone Worker with a keyhole port and a gate bar, a set block carrying allow and approvers tabs, an approver's mailbox, and a sheet whose ladder reads inside allow, no approvers, approved, with the exits A served, H held, R refused, X revoked.
Chapter 33 What a thief needs
One
A stolen disk is a LUKS2 volume with two slots, and neither opens on its own.
The initrd waits on Tang with the passphrase prompt still on the console.
The passphrase slot stays with me. The clevis slot's token names a keystone URL; the slot opens only while that Worker answers. base's Tang unlock ships with the passphrase slot kept. I keep no TPM slot on a Tang-bound host: a stolen machine would boot itself open.
Two
With the set gated, the thief needs the disk plus one more thing: an allowed address, an approver's mailbox, or the account.
The key store stays offline; sops keeps a second copy.
An address inside the set's allow list is served unattended. An approver is whoever gets the request email and can admit a network for ten minutes. Whoever reads an approver's mailbox or Workers Logs holds approval power until it expires. The keys live in Worker secrets; any version the account deploys reads them, so the Cloudflare account is guarded like the disks. Each is mine to pull.
Three
Delete the set and the disk stops unlocking; a host already open keeps its key.
Old keys open a copy of the disk taken before its slot was removed.
Revoked, the key-recovery request (rec) answers 404. Worker versions carry their secrets, so a rollback past the revoke serves the set again; keystone's hourly check mails when another version serves. A flood can spend the Free plan's daily quota and send bound hosts to their passphrase; a WAF rate limit eases it. Losing keystone costs a passphrase entry per disk per boot, never data.
Fig. 33 Two key slots, a hatched seat where a TPM slot would sit, the Worker's key block and three factors numbered into the thief's bill. Revoked, the key block lifts out, the wire reads 404, the open host keeps its key and an older version still holds the set.
Part VIII
Your lab
What it takes to run your own instance: a fork, one push, and one thing to supply at each rung.
Chapter 34 Your lab
One
A fork climbs seven rungs and supplies one thing at each. The first five need nothing sealed.
Rungs read bottom up.
Every example lane builds from the unsealed base :stable tag, not the sealed one, so the ladder splits at rung six. Up to five, a fork supplies a registry path, the catalog's includes, a base flavor, a TPM (the chip on the machine that holds the disk-unlock key) with a recovery key for when it will not release it, and a signing keypair. A sealed lane and rollback come after. The seven rungs are my ordering; no repo enforces it.
Two
Fork the example, keep one lane, delete the other three.
Each lane keeps its own decisions log.
cairn/example holds four lanes, each with its own include: block of the instance component. plain has no disk encryption, attended unlocks by passphrase, edge unlocks itself from a TPM policy, and evidence adds signature checks to any of them. Keep one (plain is simplest; the figure keeps edge), deleting the others with their include blocks and hand-written jobs. instance_image is built from ${CI_REGISTRY_IMAGE}, so a fork pushes to its own registry. The runner tags name GitLab's shared runners, so a fork builds as is.
Three
Edit the image definition (the lane's Containerfile under lanes/ in your fork), push to main, and CI builds and promotes it.
The instance component detects changes, builds, generates and checks an SBOM and, on main, tags the build :latest and :stable. It never boots the image; sealed-instance does. A fork rides a base tag the catalog already promoted, so an instance's promotion criterion is a green build against that tag.
Fig. 34 A stair of seven rungs, one thing to supply at each, clears for the example: four lanes, three deleted, one pushed through three stations to the fork's own registry, where it earns two tags, with three doors in.