cairn

Reference

Architecture

How cairn is built, signed, sealed, run and checked from outside.

01 Principles

1Public catalog

The whole build path is public: the image recipes and the catalog of CI components that build, gate and sign them.

Only the machines, and each host's own layer over a published image, stay private. No check depends on trusting either.

2One major

A repo pins a component's major version and takes every fix under it with no commit.

3Together

A breaking change reaches every consumer as one reviewable change.

The catalog owns how an image is built. Each consumer decides what verified means for its own images.

Fig. What is public, what is private, and how a release reaches each repo.
Principle What it means here Record
Verify before trust Keys are published with fingerprints, so a verifier never asks the registry to vouch for itself. Verify
The gate tests what production boots A sealed row's tag moves only after that exact artifact boots under enforcing Secure Boot. D27
Use the upstream's extension point Kernel arguments, initramfs policy and composefs switches go where their upstreams publish for them. D5
Declare every divergence Each step in the recipe carries a cf: block: what, why, and the upstream it follows. .deviations.yaml
One fact, one home The tree states a fact, this site how cairn works, a README how to run a repo, docs/decisions.md why. CONTRIBUTING

Guide, chapter 12, The catalog

02 The image

1Compose

The base image is composed from Fedora packages in a throwaway builder, with no parent image.

rpm-ostree compose rootfs runs against a treefile that allows no documentation and no weak dependencies, so one input composes to the same bytes.

2From scratch

The shipped image starts empty and takes the composed root in one copy.

Only a one-file Tang proof receipt joins it. bootc's container lint runs on the result, and a warning fails the build.

3Flavors

One composed root carries several flavors, and each adds what its name says.

base-zfs adds source-built OpenZFS. base and base-zfs-nvidia earn :stable and :stable-sealed by booting sealed; base-zfs and base-nvidia earn :stable on build, lint and manifest checks, and the unsealed base-vm earns it on a VM boot.

Fig. Two build stages, then the flavors and how each earned its tag.

What the image carries

Baked in Why
Secure Boot enrolment payloads Public certificates, copied to the ESP for systemd-boot to enrol (D19).
kmod-signer.der The certificate that signs systemd-boot and every UKI signs the modules too (D29). Its private half never reaches a layer.
nftables, disabled No firewall until an instance adds one. SSH is key-only.

Kernel modules

Modules compile from source against the kernel their row ships, never DKMS or akmods. The ZFS rows hold the newest stable kernel at or below the ceiling OpenZFS declares. cairn verify fails the build when shipped files differ from the builders' install records.

Adding a package

To add a package, end its RUN with /usr/libexec/cairn/selinux-policy-record. The first sealed boot fails when /etc/selinux differs from the recorded sums, on a relabel, and on any denial; CAIRN_POLICY_PIN=0 opts out a host that installs a module at first boot.

Guide, chapter 8, Base from scratch

03 The pipeline

1Build and seal

A push that changes an image builds a candidate named by its commit, then seals it.

base includes base-build-scratch, layer, seal and promote from the catalog and writes the rest itself: the gates and the drift check.

2Boot

The sealed candidate boots in a VM under enforcing Secure Boot, with no shim.

sealed-boot gates the two rows that seal, base and base-zfs-nvidia. The flagship adds upgrade-path and policy-diff.

3Promote

:stable moves only after every gate has passed.

promote pulls the candidate again, runs cairn verify and retags the tested digest.

4Red

A failing gate skips promote, and the last good image keeps the tag.

A boot gate retries only when the runner broke or the job stalled, never when a check failed.

5Probe

A gate that cannot fail proves nothing, so we break ours on purpose.

On every merge request, lint makes the harness's check always pass, runs the suite against that mutant, and fails if nothing failed.

6Lab

The health gate's probe runs in a lab VM, where a broken host costs nothing.

It boots a bad candidate with bootc rollback swapped for true, and the gate has to hold.

Fig. One row of the pipeline: the tag each job writes and the gates a promotion needs.

What starts a build

Pipeline source Result
A push touching only documentation, CI configuration or the scripts harness No build, no cascade.
Any other trigger: operator, API, downstream Builds.
The daily schedule detect-inputs compares each :stable label and the installed set against what the repos offer, and builds on a difference (D39).

The gates

Gate Proves
manifest-gate The module manifests match the committed goldens. No boot, so cheap.
sealed-boot The sealed candidate installs and boots under enforcing Secure Boot.
upgrade-path A running guest switches to the candidate, reboots, and rolls back.
policy-diff No SELinux label rule was removed or retyped.

Around the gates

An instance on an unsealed row seals and boots its own candidate, and may set host_assert to run its own script in the guest; the boot fails without its pass.

example and whetstone include supply-chain, instance and preset; whetstone adds qcow2.

Guide, chapter 14, The loop

04 The trust chain

1Ceremony

Everything we sign traces back to one key set, minted in a single offline run.

Two keys own the firmware, in offline escrow. Three sign what the pipeline builds, as masked, protected CI variables. The mint script enrols nothing.

2What each signs

The three CI keys sign the boot chain, the unlock policy and the image digest.

The UKI key signs systemd-boot, every unified kernel image and the kernel modules. The PCR 11 policy key signs each UKI's policy. The cosign key signs promoted digests in a repo that opts in.

3Public halves

One certificate covers the boot chain and the modules, and every public half is published.

We ship no shim, so the firmware trusts that certificate directly. The fingerprints are under Verify. A fork mints its own set.

Fig. Five keys by custody: what each signs and who checks it.

Trust arrives as signed payloads

Each certificate is self-signed; trust flows through the payloads the ceremony builds.

The firmware trusts Because
The Platform Key An operator enrolled it.
The Key Exchange Key The Platform Key signed its payload.
Our signing certificate The Key Exchange Key signed the signature database's payload.

Vendor certificates stay

The firmware checks option ROMs against the same database, and vendors sign a GPU's or a BMC's pre-boot code through Microsoft's UEFI CA. So the database holds the Microsoft and Windows CAs, both generations, beside ours. A fork keeps the CAs its hardware needs, or add-in card firmware stops loading.

Modules and images

With no shim, a module signer reaches the kernel through Fedora's platform keyring fallback (D29); sealed boot asserts it by Subject Key Identifier.

containers/image cannot verify GitLab's keyless identities at pull time (the evidence lane's record), so images are signed with our own cosign key, with no transparency log.

Guide, chapter 23, Your own keys

05 Sealing

1Signed

A signature says who built the image. It says nothing about what is on the disk.

The firmware checks systemd-boot and the UKI against our certificate before either runs. The UKI is one signed file: kernel, initrd, command line and policy. GRUB, shim and bootupd are absent.

2Sealed

The seal writes the root filesystem's digest onto the signed command line.

The seal recipe runs bootc container ukify over the root. The root mounts as composefs with verity=require, so the kernel checks every object it reads and a changed byte fails to read.

3Unlock

Whether the disk opens turns on who signed the boot.

Each UKI carries a PCR 11 policy signed by our policy key. A LUKS2 slot enrolled to that key opens for any boot the key signed, so the binding survives a kernel update. Any other boot falls back to the recovery passphrase.

Fig. One signed file pins the root and decides the unlock.

Unlock bound to the signer

systemd-cryptenroll --tpm2-public-key --tpm2-public-key-pcrs=11

Enrolment needs no signature. Unlock does, and sd-stub hands it over from the UKI, so root has no /etc/crypttab entry. The enrolled key must be the exact half of the key that signed that UKI's policy; tpm-rebind.sh derives it from the running UKI and fails closed (procedure).

Unlock from the network

Every flavor ships tang-initrd, inert until an instance calls it; then root unlocks from a Tang server and the passphrase slot stays. Our own hosts unlock this way (keystone, guide chapter 24).

Unsealed tags

Only sealed images install to disk; unsealed tags are substrates an instance layers on (D27).

Guide, chapter 22, Signing vs sealing

06 Running it

1Cascade stops

A promoted base rebuilds each instance, and the cascade stops at the registry.

Base posts a trigger and never waits (D30). The instance list is a protected CI variable, so no public file names a consumer. No CI reaches a running host.

2Pull

An operator runs bootc upgrade, and the image lands beside the running one, staged.

3Reboot

A reboot boots the staged image and keeps the old one as the rollback.

bootc rollback reorders the boot entries. Both deployments are on disk, so nothing is copied.

4Checks

An opt-in health gate holds a new boot until every required check has passed.

It ships disabled (D40). A host enables it and picks checks by symlink.

5Strikes

A failed check reboots to retry, and at the third failure the host rolls back.

The count lives in /var, which both deployments share. The failed image stays on disk.

6Hold

A healthy boot zeroes the count. With nowhere to roll back to, the host stops and waits.

It holds at the console, and so does a failure that outlives a rollback.

Fig. Where CI stops, what the host does, and the health gate on a new boot.

On the host

systemctl enable health-gate-check.service health-gate-confirm.service turns the health gate on.

An instance

An instance repo is one FROM line on a base flavor plus one machine's config. A plain instance rides a tag the base already earned, so a green build promotes it; one on sealed-instance or variant seals and boots its own candidate. Copy the method covers forking one.

whetstone, a real consumer

whetstone builds the image our CI runners boot. Its qcow2 is baked once from the candidate digest with bootc install to-disk --via-loopback, the installer that puts an instance on metal; a failed bake holds :stable.

A private host

A private host is the same shape, with real users, SSH keys and services layered on in private. Its digests check against the published key.

Guide, chapter 26, Updating

07 Verify

1Runner

Booting the image is the test, on a runner that is itself a VM built from the image.

Every job gets a fresh runner VM with QEMU, a software TPM and Secure Boot firmware already on its disk.

2Firmware

Inside the job a second machine boots the sealed candidate under enforcing Secure Boot.

On a protected ref the firmware starts in Setup Mode and systemd-boot enrols the key set a real host gets. A merge request's throwaway signer cannot enrol, so its boot starts with the keys already written.

3Verdict

The gate reads its receipts out of the running guest, and one verdict moves the tags.

Secure Boot is on, the root's digest equals the one signed into the UKI, the module signer sits in the expected keyring, and SELinux is enforcing. sealed-boot also checks the UKI's signer and policy key against the committed anchors published below.

4From outside

The same anchors are published here, so anyone can compare without trusting the registry.

Fig. The same anchors, checked in CI and from outside.

From outside

Claim Check A pass prints
An anchor is the one we published. The command on its card. The card's fingerprint.
An image digest carries our signature. cosign verify --key cosign.pub --insecure-ignore-tlog=true <image>@<digest> The signed payload naming the digest.
A module on a running host was signed by our key. modinfo -F signer zfs cairn UKI signing
A sealed image boots with the root it was signed for. No outside check; sealed-boot asserts it in CI.
base-zfs and base-nvidia boot. Unproven here. They promote without a boot.
The example lanes install to disk. Open. They carry no seal component.

With no transparency log, these fingerprints are the only anchor. The cards come from fingerprints.txt, and the build fails if it drifts from the repo's keys/.

PK · Platform Key

CN=cairn Platform Key, O=cairn.dunn.dev

B2:23:AE:9A:5A:7F:C8:5A36:ED:17:85:9F:ED:BD:E82F:45:E5:9F:80:37:DB:C332:91:68:9B:82:4B:7D:87

openssl x509 -in PK.crt -noout -subject -fingerprint -sha256

download PK.crt

KEK · Key Exchange Key

CN=cairn Key Exchange Key, O=cairn.dunn.dev

D4:E9:03:DA:CE:06:F3:D042:01:CA:B2:2C:74:18:FFC9:8A:56:EE:03:CC:A6:92F4:83:15:49:6C:94:20:9B

openssl x509 -in KEK.crt -noout -subject -fingerprint -sha256

download KEK.crt

UKI · UKI signing certificate

CN=cairn UKI signing, O=cairn.dunn.dev

A4:19:A8:25:C8:78:52:43C2:1F:23:74:C1:8B:49:8E4E:51:59:B4:83:2E:5B:ED3C:EF:46:88:A4:1E:F4:C6

openssl x509 -in UKI.crt -noout -subject -fingerprint -sha256

download UKI.crt

pcr11 · PCR 11 policy public key

RSA public key (no certificate)

0f0dbb4eecba169e5c036c4b45707a3371bece1f7f1795fdec6e9e1327aabe4f

openssl pkey -pubin -in pcr11.pub.pem -outform DER | sha256sum

download pcr11.pub.pem

cosign · Image signing public key

ECDSA public key (no certificate)

03bd99155ffaa044a97f2c6b2cff56aae83195df65f83a58abab5eda404d7ef7

sha256sum cosign.pub

download cosign.pub

Guide, chapter 15, Booting inside CI