testing harness, coredesktop-dev

skills/coredesktop-dev-sibling.md

@@ -77,7 +77,8 @@ coredesktop-dev/
   update.sh             run on every `update`
   bin/coredesktop-dev   the command (symlinked into ~/.local/bin)
   hooks/pre-push        git hook for the working copy (opt-in)
-  lib/                  shared functions, if bin/ grows large
+  lib/                  one file per subcommand, plus common.sh and vm.sh
+  vm/                   provision.sh and cloud-init user-data for the smoketest VM
   tests/                bats tests for this project
   Makefile              `make check`: shellcheck + bats for this repo
   .shellcheckrc         copy CoreDesktop's
@@ -95,6 +96,8 @@ added later reach existing machines on their next `update`. All are in Arch's
 | `shellcheck` | `make lint` |
 | `bats` | `make test` (CoreDesktop's tests use plain bats, no helper libraries) |
 | `tea` | Gitea's CLI, for pull requests on gitea.federated.computer |
+| `qemu-base` | The smoketest's virtual machines |
+| `libisoburn` | `xorriso`, to make the smoketest VM's cloud-init disk |
 
 `make`, `git` and `jq` already come with CoreDesktop. Don't add `shfmt` until
 CoreDesktop decides on a formatting style; it has no config for it today.
@@ -134,7 +137,7 @@ default answer.
 
 No prompts. It prints what it does and exits non-zero only on real failure:
 
-1. `install_packages shellcheck bats tea`
+1. `install_packages shellcheck bats tea qemu-base libisoburn`
 2. Make `~/.local/bin/coredesktop-dev` a symlink to `bin/coredesktop-dev` in
    this repo, so each pull updates the command without copying.
 3. If the pre-push hook is installed in the working copy, refresh it from
@@ -222,6 +225,26 @@ Installs `hooks/pre-push` into the working copy's `.git/hooks/pre-push`. The
 overwrite a pre-push hook that this project didn't write: write a marker
 comment into the hook and check for it.
 
+#### `coredesktop-dev smoketest`
+
+Runs CoreDesktop's end-to-end scenarios on QEMU virtual machines. The
+scenarios, the fingerprint and health scripts, and the helpers they use live
+in CoreDesktop's `smoketest/` folder, because they change with the code they
+test; read its README for the contract between them and this driver. This
+project only provides the VMs: `vm_exec`, `vm_put`, `vm_reboot`, `VM_GIT_SSH`
+and `VM_GIT_URL`, then calls `smoketest_prepare` and the scenario.
+
+- `list`, `run [--tier pr|all] [--ref REF] [--keep] [scenario...]`,
+  `images [base|installed [REF]]`, `shell [base|installed]`, `clean [--all]`.
+- Images, the shared pacman cache, the SSH key and run logs live in
+  `~/.cache/coredesktop-dev/smoketest`, never in either copy of CoreDesktop.
+- **base** is Arch's cloud image plus archinstall's GNOME desktop profile
+  (`vm/provision.sh`), with automatic login and passwordless sudo for `tester`.
+  **installed** is base after `install.sh` at the commit the branch started
+  from. Each scenario runs on a throwaway qcow2 overlay of one of them.
+- Commands in the VM run in an interactive bash with the GNOME session's
+  environment, as they would in GNOME Terminal.
+
 ### What not to do
 
 - Don't copy hooks into CoreDesktop's `updates.d/`: use `update.sh`, as the
@@ -278,9 +301,10 @@ Cases worth a test each:
 - [ ] The repo's README says to install with `sibling add <url>`
 - [ ] `install.sh` asks for the working copy, refuses the deployed copy,
       saves `~/.config/coredesktop-dev/config`, and is safe to re-run
-- [ ] `update.sh` installs shellcheck, bats and tea, keeps the
+- [ ] `update.sh` installs shellcheck, bats, tea, QEMU and xorriso, keeps the
       `~/.local/bin/coredesktop-dev` symlink, and never prompts
-- [ ] `doctor`, `check`, `try`, `deployed` and `hooks` work and have `--help`
+- [ ] `doctor`, `check`, `try`, `deployed`, `hooks` and `smoketest` work and
+      have `--help`
 - [ ] `check` lints untracked scripts in the working copy
 - [ ] `try` isolates `HOME`, `XDG_*` and `XDG_RUNTIME_DIR`, unsets the
       display and session variables, uses `dbus-run-session`, and warns that

smoketest/README.md

@@ -0,0 +1,111 @@
+# Smoketest
+
+End-to-end tests of `install.sh` and `update` on a real Arch + GNOME virtual
+machine. The bats tests in `tests/` check the logic with fake commands; these
+check what fakes can't: pacman, yay, Flatpak, systemd, ufw, Snapper on btrfs,
+GNOME settings, and a reboot.
+
+This folder holds what to test: the scenarios, and the scripts that describe a
+CoreDesktop machine. The VM itself (QEMU, images, the package cache) comes from
+the `coredesktop-dev` sibling project, which only contributors install:
+
+```
+sibling add <coredesktop-dev url>
+coredesktop-dev smoketest list             # scenarios and their tier
+coredesktop-dev smoketest run              # the quick (pr) scenarios
+coredesktop-dev smoketest run --tier all   # everything, including a fresh install
+coredesktop-dev smoketest run 40-failing-migration
+```
+
+It tests the working copy's `HEAD` commit, so commit first.
+
+## How it works
+
+Each scenario runs on a throwaway copy of one of two VM images:
+
+- **base**: Arch with GNOME, as archinstall's desktop/GNOME profile leaves it,
+  with a user `tester` who logs in to GNOME automatically.
+- **installed**: base after `install.sh`, at an older commit of CoreDesktop.
+  Before the scenario starts, `update` brings it to the commit under test.
+
+Inside the VM, a bare repo (`/srv/smoketest/coredesktop.git`) stands in for
+Gitea. git's `insteadOf` sends CoreDesktop's real URLs there, so `install.sh`
+and `update` run unchanged. The harness pushes the commit under test to it, and
+scenarios push further commits to simulate a new release: a new migration, a
+broken hook, a change to `update` itself.
+
+After a scenario, `fingerprint` describes the machine as text (packages,
+Flatpaks, services, firewall, CoreDesktop's files in /etc, GNOME settings...).
+`10-fresh-install` saves the fingerprint of a fresh install, and
+`30-upgrade-from-baseline` checks that installing an older commit from
+`baselines` and then running `update` gives the same machine. If they differ,
+an install.sh change is probably missing its migration.
+
+## Files
+
+| File | What it is |
+|------|------------|
+| `scenarios/NN-name.sh` | One scenario each (see below) |
+| `lib/scenario.sh` | Helpers and assertions the scenarios use |
+| `fingerprint` | Runs in the VM; prints the machine's fingerprint |
+| `health` | Runs in the VM; checks the machine works, exits 1 if not |
+| `baselines` | Older commits that `30-upgrade-from-baseline` installs first |
+
+## Writing a scenario
+
+A scenario is a bash file that sets a few variables and defines `scenario`:
+
+```bash
+DESCRIPTION="update runs a new migration once"
+IMAGE=installed      # base or installed
+TIER=pr              # pr (quick, uses the installed image) or nightly
+
+scenario() {
+  upstream_commit "Add a migration" 'echo "touch ~/migrated" > migrations/9999999999.sh'
+  cd_update
+  assert_status 0
+  assert_vm "the migration ran" 'test -f ~/migrated'
+}
+```
+
+`VARIANTS=(a b c)` runs the scenario once per value, each in its own VM, with
+`$VARIANT` set. `scenario` runs with `set -e`, so any failing command or
+assertion fails it. `skip "reason"` skips it.
+
+Helpers, from `lib/scenario.sh`:
+
+| Helper | |
+|--------|---|
+| `cd_install` | Run install.sh as `wget ... \| bash` would, from the fake upstream's main |
+| `cd_update [VAR=value...]` | Run `update` |
+| `run_in_vm NAME SCRIPT` | Run SCRIPT in the VM; sets `$OUTPUT` and `$STATUS`, logs to NAME.log |
+| `upstream_set REF` | Point the fake upstream's main at REF, a commit in this repo |
+| `upstream_commit MESSAGE SCRIPT` | Run SCRIPT in a clone of the upstream (in the VM), commit, push |
+| `fingerprint FILE` / `health` | Save the fingerprint / run the health checks |
+| `assert_status N`, `assert_output TEXT`, `refute_output TEXT` | Check the last `run_in_vm` |
+| `assert_vm DESCRIPTION SCRIPT` | SCRIPT must succeed in the VM |
+| `assert_same_fingerprint EXPECTED ACTUAL` | The two must match; saves the diff |
+
+## The contract with the driver
+
+`coredesktop-dev` boots the VM, defines these, sources `lib/scenario.sh`, calls
+`smoketest_prepare`, then the scenario's `scenario` function:
+
+| Name | |
+|------|---|
+| `vm_exec SCRIPT` | Run bash SCRIPT in the VM as `tester`, inside the GNOME session's environment, stdin from /dev/null. Returns its exit status. |
+| `vm_put LOCAL REMOTE` | Copy a file into the VM |
+| `vm_reboot` | Reboot and wait until `tester` is logged in to GNOME again |
+| `VM_GIT_SSH`, `VM_GIT_URL` | `GIT_SSH_COMMAND` and the `ssh://` prefix to push to a path in the VM |
+| `SMOKETEST_REF` | The commit under test |
+| `SMOKETEST_RESULTS` | This scenario's folder for logs and fingerprints |
+| `SMOKETEST_SHARED` | A folder shared by all scenarios in the run |
+| `IMAGE_REF` | For the installed image, the commit it was installed at |
+
+Anything that boots a VM can drive these scenarios, e.g. a CI runner with KVM.
+
+## Not covered
+
+NVIDIA drivers, firmware updates (`updates.d/90-firmware.sh`), and anything that
+needs a real Core, a Tailscale login or a Claude login (`configure`). Check
+those by hand.

smoketest/baselines

@@ -0,0 +1,12 @@
+# Older commits that 30-upgrade-from-baseline installs, then updates to the
+# commit under test, checking the result matches a fresh install. One commit
+# per line; # starts a comment. Each one takes a full install, so keep this to
+# a few points where machines in use were installed.
+
+ca658dc  # 2026-09-29 various fixes
+d8d93a4  # 2026-10-20 the last commit before migrations/
+0c0ddf5  # 2026-10-29 Add Federated Calendar
+
+# The first commit on Federated Gitea. Machines this old may well not reach
+# HEAD with update alone; uncomment to find out.
+# a48dc27  # 2026-04-29 fix coreupdate to keep fresh

smoketest/fingerprint

@@ -0,0 +1,82 @@
+#!/bin/bash
+#
+# Purpose : Describe this CoreDesktop machine as text, so two machines can be
+#           compared with diff (see README.md). Runs in the smoketest VM.
+#
+# Only what CoreDesktop decides goes in: names, not versions, and no times, so
+# a fresh install and an updated older install of the same commit match.
+
+COREDESKTOP="$HOME/.local/share/coredesktop"
+
+section() {
+  printf '\n## %s\n' "$1"
+}
+
+section "Explicitly installed packages"
+pacman -Qqe
+
+section "Packages not from the repos (AUR)"
+pacman -Qqm
+
+section "Flatpaks"
+flatpak list --app --columns=application | sort
+
+section "pipx apps"
+pipx list --short 2> /dev/null | cut -d' ' -f1 | sort
+
+section "Enabled system units"
+systemctl list-unit-files --state=enabled --no-legend | awk '{print $1}' | sort
+
+section "Enabled user units"
+systemctl --user list-unit-files --state=enabled --no-legend | awk '{print $1}' | sort
+
+section "Groups"
+id -nG | tr ' ' '\n' | sort
+
+section "Firewall"
+sudo ufw status verbose
+
+section "pacman repos"
+grep -E '^\[' /etc/pacman.conf
+
+section "SSH server"
+sudo sshd -T 2> /dev/null | sort
+
+section "CoreDesktop's files in /etc"
+sudo find /etc -path '*coredesktop*' -type f -print0 | sort -z | sudo xargs -0r sha256sum
+
+section "Snapper"
+if [ -f /etc/snapper/configs/root ]; then
+  sudo snapper -c root get-config | sort
+else
+  echo "no root config"
+fi
+
+section "Migrations run"
+ls ~/.local/state/coredesktop/migrations 2> /dev/null
+
+section "Commands in ~/.local/bin"
+for cmd in update refresh sibling version configure audit; do
+  if cmp -s "$HOME/.local/bin/$cmd" "$COREDESKTOP/$cmd"; then
+    echo "$cmd: current"
+  else
+    echo "$cmd: missing or out of date"
+  fi
+done
+
+section "Shell"
+grep -n coredesktop ~/.bashrc
+
+section "GNOME extensions"
+gsettings get org.gnome.shell enabled-extensions
+ls ~/.local/share/gnome-shell/extensions 2> /dev/null
+
+# The value of each key gnome-settings.dconf sets, which apply-gnome-settings.sh
+# loads under /org/gnome/shell/extensions/
+section "GNOME settings"
+awk '/^\[/ { dir = substr($0, 2, length($0) - 2); next }
+     /=/   { print dir "/" substr($0, 1, index($0, "=") - 1) }' \
+  "$COREDESKTOP/gnome/gnome-settings.dconf" \
+  | while read -r key; do
+      echo "$key = $(dconf read "/org/gnome/shell/extensions/$key")"
+    done

smoketest/health

@@ -0,0 +1,102 @@
+#!/bin/bash
+#
+# Purpose : Check that this CoreDesktop machine works: services, firewall,
+#           desktop, and CoreDesktop's own commands. Runs in the smoketest VM.
+#           Prints ok or FAIL for each check, and exits 1 if any failed.
+
+COREDESKTOP="$HOME/.local/share/coredesktop"
+
+# Units that may fail in the VM without anything being wrong with CoreDesktop
+EXPECTED_FAILED_UNITS=(
+  fwupd-refresh.service   # the VM's firmware has nothing for fwupd to check
+)
+
+FAILED=0
+
+check() {
+  local description="$1"
+  shift
+  if "$@" &> /dev/null; then
+    echo "ok   $description"
+  else
+    echo "FAIL $description"
+    FAILED=1
+  fi
+}
+
+# Failed units, apart from EXPECTED_FAILED_UNITS. $1 is --system or --user.
+failed_units() {
+  systemctl "$1" list-units --state=failed --no-legend --plain | awk '{print $1}' \
+    | grep -vxF -f <(printf '%s\n' "${EXPECTED_FAILED_UNITS[@]}")
+}
+
+# The unit is enabled, unless audit turned it off
+enabled_unless_off() {
+  grep -qsx "system $1" "$HOME/.config/coredesktop/services-off" && return 0
+  systemctl is-enabled "$1"
+}
+
+up_to_date_with_upstream() {
+  git -C "$COREDESKTOP" fetch --quiet \
+    && [ "$(git -C "$COREDESKTOP" rev-parse HEAD)" = "$(git -C "$COREDESKTOP" rev-parse '@{upstream}')" ]
+}
+
+no_local_changes() {
+  [ -z "$(git -C "$COREDESKTOP" status --porcelain --untracked-files=no)" ]
+}
+
+# GNOME Shell is running for this user on a graphical seat
+desktop_running() {
+  pgrep -u "$USER" -x gnome-shell \
+    && loginctl list-sessions --no-legend | awk -v u="$USER" '$3 == u && $4 == "seat0"' | grep -q .
+}
+
+firewall_active() {
+  sudo ufw status | grep -qx 'Status: active'
+}
+
+commands_current() {
+  local cmd
+  for cmd in update refresh sibling version configure audit; do
+    cmp -s "$HOME/.local/bin/$cmd" "$COREDESKTOP/$cmd" || return 1
+  done
+}
+
+version_ok() {
+  local out
+  out="$("$HOME/.local/bin/version")" || return 1
+  [[ "$out" == CoreDesktop* && "$out" != *modified* ]]
+}
+
+for unit in --system --user; do
+  failed="$(failed_units "$unit")"
+  if [ -z "$failed" ]; then
+    echo "ok   no failed ${unit#--} units"
+  else
+    echo "FAIL failed ${unit#--} units: $(echo "$failed" | tr '\n' ' ')"
+    FAILED=1
+  fi
+done
+
+# packages.conf lists the services install.sh enables
+source "$COREDESKTOP/packages.conf"
+for service in "${SERVICES[@]}" "${START_SERVICES[@]}"; do
+  check "$service is enabled" enabled_unless_off "$service"
+done
+for service in "${START_SERVICES[@]}"; do
+  check "$service is running" systemctl is-active "$service"
+done
+
+check "GNOME is running" desktop_running
+check "the firewall is on" firewall_active
+check "the Federated repo is in pacman.conf" grep -q '^\[federated\]' /etc/pacman.conf
+check "Snapper is set up (if / is btrfs)" \
+  bash -c '[ "$(findmnt -no FSTYPE /)" != btrfs ] || [ -f /etc/snapper/configs/root ]'
+check "CoreDesktop is on upstream's latest commit" up_to_date_with_upstream
+check "CoreDesktop's checkout has no local changes" no_local_changes
+check "the commands in ~/.local/bin are current" commands_current
+check "version works" version_ok
+check "audit --list works" "$HOME/.local/bin/audit" --list
+check "the shell setup is in ~/.bashrc" grep -q coredesktop "$HOME/.bashrc"
+
+exit $FAILED

smoketest/lib/scenario.sh

@@ -0,0 +1,199 @@
+#!/bin/bash
+#
+# Purpose : Helpers and assertions for smoketest scenarios (see ../README.md).
+#           Sourced by the driver after it defines vm_exec, vm_put and
+#           vm_reboot.
+
+SMOKETEST_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+# The CoreDesktop repo whose commits are tested. Overridable for the tests.
+SMOKETEST_REPO="${SMOKETEST_REPO:-$(cd "$SMOKETEST_DIR/.." && pwd)}"
+
+# In the VM: the fake upstream, the scratch clone upstream_commit works in, and
+# where the fingerprint and health scripts are copied to
+SMOKETEST_VM_DIR="${SMOKETEST_VM_DIR:-/srv/smoketest}"
+UPSTREAM="$SMOKETEST_VM_DIR/coredesktop.git"
+WORK="$SMOKETEST_VM_DIR/work"
+
+# Every URL CoreDesktop has been cloned from, so install.sh at any baseline
+# clones the fake upstream instead
+COREDESKTOP_URLS=(
+  https://gitea.federated.computer/Federated_Computer/coredesktop.git
+  https://gitea.young.computer/david/coredesktop.git
+)
+
+# Exit status of a scenario that skipped itself
+SKIP_STATUS=3
+
+# Output, exit status, name and log file of the last run_in_vm
+OUTPUT=""
+STATUS=0
+LAST_RUN=""
+LAST_LOG=""
+RUN_COUNT=0
+
+q() {
+  printf '%q' "$1"
+}
+
+fail() {
+  echo "FAIL: $*" >&2
+  exit 1
+}
+
+skip() {
+  echo "SKIP: $*"
+  exit "$SKIP_STATUS"
+}
+
+# run_in_vm NAME SCRIPT: run SCRIPT in the VM, keeping its output in $OUTPUT,
+# its exit status in $STATUS, and both in NN-NAME.log. Doesn't fail on its own:
+# check with assert_status.
+run_in_vm() {
+  RUN_COUNT=$((RUN_COUNT + 1))
+  LAST_RUN="$1"
+  LAST_LOG="$SMOKETEST_RESULTS/$(printf '%02d' "$RUN_COUNT")-$1.log"
+  echo "== $1"
+  OUTPUT="$(vm_exec "$2" 2>&1)" && STATUS=0 || STATUS=$?
+  printf '%s\n' "$OUTPUT" > "$LAST_LOG"
+  echo "== $1 exited $STATUS (log: $LAST_LOG)"
+}
+
+# Called by the driver once the VM is up, before the scenario
+smoketest_prepare() {
+  local urls="" url
+  for url in "${COREDESKTOP_URLS[@]}"; do
+    urls+=" $(q "$url")"
+  done
+  vm_exec "set -e
+    [ -d $(q "$UPSTREAM") ] || git init -q --bare --initial-branch=main $(q "$UPSTREAM")
+    git config --global user.name 'CoreDesktop Smoketest'
+    git config --global user.email smoketest@localhost
+    git config --global --unset-all url.$(q "$UPSTREAM").insteadOf || true
+    for url in $urls; do
+      git config --global --add url.$(q "$UPSTREAM").insteadOf \"\$url\"
+    done" || fail "couldn't set up the fake upstream in the VM"
+
+  if [ "${IMAGE:-}" = installed ]; then
+    git -C "$SMOKETEST_REPO" merge-base --is-ancestor "$IMAGE_REF" "$SMOKETEST_REF" \
+      || fail "the installed image is at $IMAGE_REF, which isn't in the history of" \
+        "the commit under test. Rebuild it: coredesktop-dev smoketest images --installed"
+    upstream_set "$SMOKETEST_REF"
+    cd_update
+    assert_status 0
+  else
+    upstream_set "$SMOKETEST_REF"
+  fi
+}
+
+# Called by the driver to make the installed image from the base image
+smoketest_install_image() {
+  smoketest_prepare
+  upstream_set "$1"
+  cd_install
+  assert_status 0
+}
+
+# upstream_set REF: point the fake upstream's main at REF, a commit in this repo
+upstream_set() {
+  local sha
+  sha="$(git -C "$SMOKETEST_REPO" rev-parse --verify --quiet "$1^{commit}")" \
+    || fail "no such commit: $1"
+  GIT_SSH_COMMAND="${VM_GIT_SSH:-ssh}" git -C "$SMOKETEST_REPO" push --quiet --force \
+    "${VM_GIT_URL:-}$UPSTREAM" "$sha:refs/heads/main" || fail "couldn't push $1 to the VM"
+}
+
+# upstream_commit MESSAGE SCRIPT: release a new commit, made by running SCRIPT
+# (in the VM) in a clone of the fake upstream's main
+upstream_commit() {
+  run_in_vm commit "set -e
+    rm -rf $(q "$WORK")
+    git clone -q $(q "$UPSTREAM") $(q "$WORK")
+    cd $(q "$WORK")
+    $2
+    git add -A
+    git commit -qm $(q "$1")
+    git push -q origin HEAD:main"
+  assert_status 0
+}
+
+# Run install.sh the way the README's wget ... | bash does: main's install.sh,
+# with no terminal to ask questions on
+cd_install() {
+  run_in_vm install "git --git-dir=$(q "$UPSTREAM") show main:install.sh > ~/install.sh \
+    && cd && TERM=dumb bash ~/install.sh"
+}
+
+# cd_update [VAR=value...]: run update, with those variables set
+cd_update() {
+  local vars="" var
+  for var in "$@"; do
+    vars+=" $(q "$var")"
+  done
+  run_in_vm update "TERM=dumb env$vars ~/.local/bin/update"
+}
+
+# fingerprint FILE: save the machine's fingerprint to FILE
+fingerprint() {
+  vm_put "$SMOKETEST_DIR/fingerprint" "$SMOKETEST_VM_DIR/fingerprint" \
+    || fail "couldn't copy the fingerprint script to the VM"
+  vm_exec "bash $(q "$SMOKETEST_VM_DIR/fingerprint")" > "$1" || fail "fingerprint failed (see $1)"
+}
+
+# Where 10-fresh-install saves the fingerprint of a fresh install of the
+# commit under test
+reference_fingerprint() {
+  echo "$SMOKETEST_SHARED/fresh-install.fingerprint"
+}
+
+# Run the health checks; fail the scenario if any fails
+health() {
+  vm_put "$SMOKETEST_DIR/health" "$SMOKETEST_VM_DIR/health" \
+    || fail "couldn't copy the health script to the VM"
+  run_in_vm health "bash $(q "$SMOKETEST_VM_DIR/health")"
+  printf '%s\n' "$OUTPUT"
+  [ "$STATUS" -eq 0 ] || fail "the health checks failed"
+}
+
+assert_status() {
+  [ "$STATUS" -eq "$1" ] || fail "$LAST_RUN exited $STATUS, expected $1 (see $LAST_LOG)"
+}
+
+assert_output() {
+  [[ "$OUTPUT" == *"$1"* ]] || fail "$LAST_RUN's output doesn't contain \"$1\" (see $LAST_LOG)"
+}
+
+refute_output() {
+  [[ "$OUTPUT" != *"$1"* ]] || fail "$LAST_RUN's output contains \"$1\" (see $LAST_LOG)"
+}
+
+# assert_vm DESCRIPTION SCRIPT: SCRIPT must succeed in the VM
+assert_vm() {
+  local out
+  out="$(vm_exec "$2" 2>&1)" || { printf '%s\n' "$out"; fail "$1"; }
+  echo "ok: $1"
+}
+
+# refute_vm DESCRIPTION SCRIPT: SCRIPT must fail in the VM
+refute_vm() {
+  local out
+  if out="$(vm_exec "$2" 2>&1)"; then
+    printf '%s\n' "$out"
+    fail "$1"
+  fi
+  echo "ok: $1"
+}
+
+# vm_value SCRIPT: print SCRIPT's output, failing the scenario if SCRIPT fails
+vm_value() {
+  vm_exec "$1" || fail "couldn't run in the VM: $1"
+}
+
+# assert_same_fingerprint EXPECTED ACTUAL: the fingerprint files must match
+assert_same_fingerprint() {
+  local diff="$SMOKETEST_RESULTS/fingerprint.diff"
+  if ! diff -u --label expected --label actual "$1" "$2" > "$diff"; then
+    cat "$diff"
+    fail "the machine differs from $1 (diff: $diff)"
+  fi
+  echo "ok: same fingerprint as $1"
+}

smoketest/scenarios/10-fresh-install.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+# shellcheck disable=SC2034
+DESCRIPTION="install.sh on a fresh Arch + GNOME machine; saves the reference fingerprint"
+IMAGE=base
+TIER=nightly
+
+scenario() {
+  cd_install
+  assert_status 0
+  assert_vm "install.sh left no local changes in the checkout" \
+    '[ -z "$(git -C ~/.local/share/coredesktop status --porcelain --untracked-files=no)" ]'
+
+  # The README says to reboot after installing
+  vm_reboot
+  health
+
+  fingerprint "$SMOKETEST_RESULTS/installed.fingerprint"
+  cp "$SMOKETEST_RESULTS/installed.fingerprint" "$(reference_fingerprint)"
+}

smoketest/scenarios/11-refresh.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+# shellcheck disable=SC2034
+DESCRIPTION="refresh (re-running install.sh) on an installed machine changes nothing"
+IMAGE=installed
+TIER=pr
+
+scenario() {
+  fingerprint "$SMOKETEST_RESULTS/before.fingerprint"
+
+  run_in_vm refresh 'TERM=dumb ~/.local/bin/refresh'
+  assert_status 0
+  refute_output "Running migration"
+
+  fingerprint "$SMOKETEST_RESULTS/after.fingerprint"
+  assert_same_fingerprint "$SMOKETEST_RESULTS/before.fingerprint" "$SMOKETEST_RESULTS/after.fingerprint"
+  health
+}

smoketest/scenarios/20-noop-update.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+# shellcheck disable=SC2034
+DESCRIPTION="update with nothing new to do installs nothing and changes nothing"
+IMAGE=installed
+TIER=pr
+
+# smoketest_prepare has already updated the machine to the commit under test
+scenario() {
+  fingerprint "$SMOKETEST_RESULTS/before.fingerprint"
+
+  cd_update
+  assert_status 0
+  assert_output "Done."
+  refute_output "Running migration"
+  # install_packages prints this only when it installs something
+  refute_output "Installing:"
+
+  fingerprint "$SMOKETEST_RESULTS/after.fingerprint"
+  assert_same_fingerprint "$SMOKETEST_RESULTS/before.fingerprint" "$SMOKETEST_RESULTS/after.fingerprint"
+  health
+}

smoketest/scenarios/30-upgrade-from-baseline.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+# shellcheck disable=SC2034
+DESCRIPTION="install an older commit, update to the commit under test: same machine as a fresh install"
+IMAGE=base
+TIER=nightly
+mapfile -t VARIANTS < <(sed 's/#.*//' "$(dirname "${BASH_SOURCE[0]}")/../baselines" | awk 'NF { print $1 }')
+
+scenario() {
+  [ -f "$(reference_fingerprint)" ] \
+    || skip "needs the fingerprint 10-fresh-install saves; run both in the same run"
+
+  upstream_set "$VARIANT"
+  cd_install
+  assert_status 0
+  vm_reboot
+
+  upstream_set "$SMOKETEST_REF"
+  cd_update
+  assert_status 0
+  vm_reboot
+  health
+
+  fingerprint "$SMOKETEST_RESULTS/updated.fingerprint"
+  assert_same_fingerprint "$(reference_fingerprint)" "$SMOKETEST_RESULTS/updated.fingerprint"
+}

smoketest/scenarios/31-update-replaces-itself.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+# shellcheck disable=SC2034
+DESCRIPTION="a release that changes update and adds a hook: the new update and hook run in the same update"
+IMAGE=installed
+TIER=pr
+
+scenario() {
+  upstream_commit "Change update and add a hook" '
+    sed -i "s/^FAILED=()\$/echo \"smoketest: the new update ran\"\nFAILED=()/" update
+    grep -q "smoketest: the new update ran" update
+    printf "%s\n" "#!/bin/bash" "run_update() { touch ~/smoketest-hook-ran; }" > updates.d/50-smoketest.sh'
+
+  cd_update
+  assert_status 0
+  assert_output "smoketest: the new update ran"
+  assert_vm "the new hook ran" 'test -f ~/smoketest-hook-ran'
+  assert_vm "~/.local/bin/update is the new one" \
+    'cmp ~/.local/bin/update ~/.local/share/coredesktop/update'
+}

smoketest/scenarios/32-new-migration.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+# shellcheck disable=SC2034
+DESCRIPTION="a release with a new migration: it runs once, and is recorded"
+IMAGE=installed
+TIER=pr
+
+scenario() {
+  upstream_commit "Add a migration" '
+    echo "echo ran >> ~/smoketest-migration" > "migrations/$(date +%s).sh"'
+
+  cd_update
+  assert_status 0
+  assert_output "Running migration"
+  cd_update
+  assert_status 0
+  refute_output "Running migration"
+
+  assert_vm "the migration ran exactly once" '[ "$(cat ~/smoketest-migration)" = ran ]'
+  assert_vm "the migration is recorded" '
+    cd ~/.local/share/coredesktop/migrations
+    for m in *.sh; do test -f ~/.local/state/coredesktop/migrations/$m; done'
+}

smoketest/scenarios/40-failing-migration.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+# shellcheck disable=SC2034
+DESCRIPTION="a failing migration stops later ones, fails the update, and runs again next time"
+IMAGE=installed
+TIER=pr
+
+scenario() {
+  # The first fails while ~/smoketest-block exists; the second must wait for it
+  upstream_commit "Add two migrations" '
+    now=$(date +%s)
+    echo "[ ! -e ~/smoketest-block ] && echo first >> ~/smoketest-ran" > "migrations/$now.sh"
+    echo "echo second >> ~/smoketest-ran" > "migrations/$((now + 1)).sh"'
+  assert_vm "block the first migration" 'touch ~/smoketest-block'
+
+  cd_update
+  assert_status 1
+  assert_output "failed; it will run again on the next update"
+  assert_output "Done, with errors in: 12-migrations.sh"
+  refute_vm "neither migration ran" 'test -e ~/smoketest-ran'
+
+  assert_vm "unblock the first migration" 'rm ~/smoketest-block'
+  cd_update
+  assert_status 0
+  assert_vm "both ran, in order" '[ "$(cat ~/smoketest-ran)" = "$(printf "first\nsecond")" ]'
+
+  cd_update
+  assert_status 0
+  assert_vm "neither ran again" '[ "$(wc -l < ~/smoketest-ran)" -eq 2 ]'
+}

smoketest/scenarios/41-failing-hook.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+# shellcheck disable=SC2034
+DESCRIPTION="a failing update hook fails the update but not the hooks after it"
+IMAGE=installed
+TIER=pr
+
+scenario() {
+  upstream_commit "Add a failing hook" '
+    printf "%s\n" "#!/bin/bash" "run_update() { echo smoketest: failing; return 1; }" > updates.d/13-smoketest-fail.sh
+    printf "%s\n" "#!/bin/bash" "run_update() { touch ~/smoketest-after; }" > updates.d/96-smoketest-after.sh'
+
+  cd_update
+  assert_status 1
+  assert_output "smoketest: failing"
+  assert_output "Done, with errors in: 13-smoketest-fail.sh"
+  assert_vm "the hook after it ran" 'test -f ~/smoketest-after'
+}

smoketest/scenarios/42-upstream-unreachable.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+# shellcheck disable=SC2034
+DESCRIPTION="update when CoreDesktop's repo can't be reached runs the hooks on disk and reports it"
+IMAGE=installed
+TIER=pr
+
+scenario() {
+  assert_vm "take the upstream away" "sudo mv $UPSTREAM $UPSTREAM.away"
+  cd_update
+  assert_status 1
+  assert_output "could not pull the latest CoreDesktop"
+  assert_output "Done, with errors in: git pull"
+  assert_output "Updating Arch..."
+
+  assert_vm "bring it back" "sudo mv $UPSTREAM.away $UPSTREAM"
+  cd_update
+  assert_status 0
+}

smoketest/scenarios/43-local-edits.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+# shellcheck disable=SC2034
+DESCRIPTION="local edits to the checkout: kept if upstream doesn't touch the file, reported if it does"
+IMAGE=installed
+TIER=pr
+
+# Expanded in the VM, not here
+# shellcheck disable=SC2088
+CD="~/.local/share/coredesktop"
+
+scenario() {
+  assert_vm "edit README.md in the checkout" "echo '# local edit' >> $CD/README.md"
+  upstream_commit "Change packages.conf" 'echo "# upstream change" >> packages.conf'
+  cd_update
+  assert_status 0
+  assert_vm "the edit is kept" "tail -n 1 $CD/README.md | grep -qx '# local edit'"
+  assert_vm "the upstream change arrived" "tail -n 1 $CD/packages.conf | grep -qx '# upstream change'"
+
+  upstream_commit "Change README.md" 'echo "# upstream readme change" >> README.md'
+  cd_update
+  assert_status 1
+  assert_output "Done, with errors in: git pull"
+
+  assert_vm "throw the edit away" "git -C $CD checkout -- README.md"
+  cd_update
+  assert_status 0
+}

smoketest/scenarios/44-low-disk.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+# shellcheck disable=SC2034
+DESCRIPTION="update refuses to start with less than 10 GiB free, unless UPDATE_FORCE=1"
+IMAGE=installed
+TIER=pr
+
+scenario() {
+  # Leave 5 GiB free on /
+  assert_vm "fill /" '
+    avail=$(df -k --output=avail / | tail -n 1 | tr -d " ")
+    sudo fallocate -l $(( (avail - 5 * 1024 * 1024) * 1024 )) /smoketest-fill'
+
+  cd_update
+  assert_status 1
+  assert_output "updating needs at least 10 GiB"
+  refute_output "Updating Arch..."
+
+  cd_update UPDATE_FORCE=1
+  assert_status 0
+}

smoketest/scenarios/50-stale-system.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+# shellcheck disable=SC2034
+DESCRIPTION="a machine a month behind on a stale mirror: update picks fresh mirrors, upgrades the kernel, asks for a reboot"
+IMAGE=installed
+TIER=nightly
+
+scenario() {
+  local date kernel_before
+  date="$(date -d '30 days ago' +%Y/%m/%d)"
+
+  # Go back to Arch as it was a month ago, from the Arch Linux Archive, which
+  # is also a mirror that stopped syncing a month ago
+  run_in_vm downgrade "set -e
+    echo 'Server = https://archive.archlinux.org/repos/$date/\$repo/os/\$arch' \
+      | sudo tee /etc/pacman.d/mirrorlist
+    sudo pacman -Syyuu --noconfirm"
+  [ "$STATUS" -eq 0 ] || skip "couldn't downgrade to Arch as of $date (see $LAST_LOG)"
+  vm_reboot
+  kernel_before="$(vm_value 'uname -r')"
+
+  run_in_vm version '"$HOME/.local/bin/version"'
+  assert_output "last synced with Arch"
+
+  local snapshots_before
+  snapshots_before="$(vm_value 'sudo snapper -c root list 2> /dev/null | wc -l')"
+
+  cd_update
+  assert_status 0
+  assert_output "Please reboot when convenient"
+  refute_vm "the first mirror isn't the archive any more" \
+    'grep -m1 "^Server" /etc/pacman.d/mirrorlist | grep -q archive.archlinux.org'
+  assert_vm "update took a snapshot first (if / is btrfs)" "
+    [ \"\$(findmnt -no FSTYPE /)\" != btrfs ] \
+      || [ \"\$(sudo snapper -c root list | wc -l)\" -gt $snapshots_before ]"
+
+  vm_reboot
+  refute_vm "a newer kernel is running" "[ \"\$(uname -r)\" = $(q "$kernel_before") ]"
+  health
+}

smoketest/scenarios/51-skip-packages.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+# shellcheck disable=SC2034
+DESCRIPTION="update doesn't reinstall a package or Flatpak listed in skip-packages"
+IMAGE=installed
+TIER=pr
+
+scenario() {
+  assert_vm "remove Zoom and LibreOffice, and skip them" '
+    set -e
+    sudo flatpak uninstall --system --noninteractive us.zoom.Zoom
+    sudo pacman -Rns --noconfirm libreoffice-fresh
+    mkdir -p ~/.config/coredesktop
+    printf "%s\n" us.zoom.Zoom libreoffice-fresh >> ~/.config/coredesktop/skip-packages'
+
+  cd_update
+  assert_status 0
+  refute_vm "Zoom is still gone" 'flatpak info --system us.zoom.Zoom'
+  refute_vm "LibreOffice is still gone" 'pacman -Qi libreoffice-fresh'
+}

smoketest/scenarios/52-siblings.sh

@@ -0,0 +1,38 @@
+#!/bin/bash
+# shellcheck disable=SC2034
+DESCRIPTION="update pulls a sibling project and runs its update.sh; a failing one is reported"
+IMAGE=installed
+TIER=pr
+
+SIB="$SMOKETEST_VM_DIR/smoketest-sibling"
+
+scenario() {
+  assert_vm "make a sibling project" "set -e
+    git init -q $SIB
+    cd $SIB
+    echo 'touch ~/sibling-installed' > install.sh
+    echo 'echo v1 >> ~/sibling-updated' > update.sh
+    git add -A
+    git commit -qm v1"
+
+  run_in_vm sibling-add "\"\$HOME/.local/bin/sibling\" add $SIB"
+  assert_status 0
+  assert_vm "its install.sh ran" 'test -f ~/sibling-installed'
+
+  assert_vm "release v2" "set -e
+    cd $SIB
+    echo 'echo v2 >> ~/sibling-updated' > update.sh
+    git commit -qam v2"
+  cd_update
+  assert_status 0
+  assert_vm "v2's update.sh ran" 'tail -n 1 ~/sibling-updated | grep -qx v2'
+
+  assert_vm "release a v3 whose update.sh fails" "set -e
+    cd $SIB
+    echo 'exit 1' > update.sh
+    git commit -qam v3"
+  cd_update
+  assert_status 1
+  assert_output "smoketest-sibling's update.sh failed"
+  assert_output "Done, with errors in: 71-siblings.sh"
+}

tests/smoketest.bats

@@ -0,0 +1,209 @@
+#!/usr/bin/env bats
+
+load test_helper
+
+# smoketest/lib/scenario.sh with a "VM" that is just a local shell in the
+# test's HOME, and a small repo of its own standing in for CoreDesktop
+
+setup() {
+  common_setup
+  export GIT_AUTHOR_NAME=Test GIT_AUTHOR_EMAIL=test@localhost
+  export GIT_COMMITTER_NAME=Test GIT_COMMITTER_EMAIL=test@localhost
+
+  # Two commits: install.sh writes which one ran, and what it read from stdin
+  export SMOKETEST_REPO="$BATS_TEST_TMPDIR/repo"
+  git init -q "$SMOKETEST_REPO"
+  cd "$SMOKETEST_REPO"
+  printf '%s\n' 'echo v1 > ~/installed' 'read -r line; echo "read: $line" >> ~/installed' > install.sh
+  git add install.sh
+  git commit -qm v1
+  V1="$(git rev-parse HEAD)"
+  sed -i 's/v1/v2/' install.sh
+  git commit -qam v2
+  V2="$(git rev-parse HEAD)"
+  cd "$BATS_TEST_TMPDIR"
+
+  export SMOKETEST_VM_DIR="$BATS_TEST_TMPDIR/vm"
+  mkdir -p "$SMOKETEST_VM_DIR"
+  export SMOKETEST_RESULTS="$BATS_TEST_TMPDIR/results"
+  export SMOKETEST_SHARED="$BATS_TEST_TMPDIR/shared"
+  mkdir -p "$SMOKETEST_RESULTS" "$SMOKETEST_SHARED"
+  export SMOKETEST_REF="$V2"
+
+  vm_exec() { bash -c "$1" < /dev/null; }
+  vm_put() { cp "$1" "$2"; }
+  source "$REPO/smoketest/lib/scenario.sh"
+}
+
+upstream_main() {
+  git --git-dir="$UPSTREAM" rev-parse main
+}
+
+@test "prepare makes the fake upstream, at the commit under test" {
+  smoketest_prepare
+  [ "$(upstream_main)" = "$V2" ]
+}
+
+@test "prepare sends CoreDesktop's URLs to the fake upstream" {
+  smoketest_prepare
+  for url in "${COREDESKTOP_URLS[@]}"; do
+    git clone -q "$url" "$BATS_TEST_TMPDIR/clone"
+    [ "$(git -C "$BATS_TEST_TMPDIR/clone" rev-parse HEAD)" = "$V2" ]
+    rm -rf "$BATS_TEST_TMPDIR/clone"
+  done
+}
+
+@test "prepare can run twice" {
+  smoketest_prepare
+  smoketest_prepare
+  [ "$(git config --global --get-all "url.$UPSTREAM.insteadOf" | wc -l)" -eq "${#COREDESKTOP_URLS[@]}" ]
+}
+
+@test "prepare on the installed image updates it to the commit under test" {
+  export IMAGE=installed IMAGE_REF="$V1"
+  mkdir -p "$HOME/.local/bin"
+  printf '%s\n' '#!/bin/bash' 'echo "update ran with $(git --git-dir="$UPSTREAM" rev-parse main)"' \
+    > "$HOME/.local/bin/update"
+  chmod +x "$HOME/.local/bin/update"
+  export UPSTREAM
+
+  run smoketest_prepare
+  [ "$status" -eq 0 ]
+  grep -qx "update ran with $V2" "$SMOKETEST_RESULTS/01-update.log"
+}
+
+@test "prepare on the installed image fails if the image isn't behind the commit under test" {
+  export IMAGE=installed IMAGE_REF="$V2" SMOKETEST_REF="$V1"
+
+  run smoketest_prepare
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"isn't in the history of the commit under test"* ]]
+}
+
+@test "upstream_set moves main back and forth" {
+  smoketest_prepare
+  upstream_set "$V1"
+  [ "$(upstream_main)" = "$V1" ]
+  upstream_set "$V2"
+  [ "$(upstream_main)" = "$V2" ]
+}
+
+@test "upstream_set fails on an unknown commit" {
+  smoketest_prepare
+  run upstream_set nosuchcommit
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"no such commit: nosuchcommit"* ]]
+}
+
+@test "upstream_commit releases a commit made by its script" {
+  smoketest_prepare
+  upstream_commit "Add a file" 'echo hello > new-file'
+
+  [ "$(git --git-dir="$UPSTREAM" rev-parse main^)" = "$V2" ]
+  [ "$(git --git-dir="$UPSTREAM" show main:new-file)" = hello ]
+  [ "$(git --git-dir="$UPSTREAM" log -1 --format=%s main)" = "Add a file" ]
+}
+
+@test "upstream_commit fails the scenario if its script fails" {
+  smoketest_prepare
+  run upstream_commit "Broken" 'false'
+  [ "$status" -eq 1 ]
+  [ "$(upstream_main)" = "$V2" ]
+}
+
+@test "cd_install runs main's install.sh, with nothing on stdin" {
+  smoketest_prepare
+  upstream_set "$V1"
+  cd_install
+  [ "$STATUS" -eq 0 ]
+  [ "$(cat "$HOME/installed")" = "$(printf 'v1\nread: ')" ]
+}
+
+@test "run_in_vm keeps the output, status and a numbered log, and doesn't fail" {
+  run_in_vm first 'echo one; exit 3'
+  [ "$STATUS" -eq 3 ]
+  [ "$OUTPUT" = one ]
+  [ "$(cat "$SMOKETEST_RESULTS/01-first.log")" = one ]
+
+  run_in_vm second 'echo two >&2'
+  [ "$STATUS" -eq 0 ]
+  [ "$OUTPUT" = two ]
+  [ -f "$SMOKETEST_RESULTS/02-second.log" ]
+}
+
+@test "cd_update passes its variables to update" {
+  mkdir -p "$HOME/.local/bin"
+  printf '%s\n' '#!/bin/bash' 'echo "force=$UPDATE_FORCE"' > "$HOME/.local/bin/update"
+  chmod +x "$HOME/.local/bin/update"
+
+  cd_update UPDATE_FORCE=1
+  [ "$STATUS" -eq 0 ]
+  [ "$OUTPUT" = "force=1" ]
+}
+
+@test "assert_status and assert_output fail with the log's path" {
+  run_in_vm thing 'echo hello; exit 1'
+
+  run assert_status 0
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"thing exited 1, expected 0"*"01-thing.log"* ]]
+
+  run assert_output goodbye
+  [ "$status" -eq 1 ]
+  run assert_output hello
+  [ "$status" -eq 0 ]
+  run refute_output hello
+  [ "$status" -eq 1 ]
+}
+
+@test "assert_vm and refute_vm" {
+  run assert_vm "true passes" true
+  [ "$status" -eq 0 ]
+  run assert_vm "false fails" 'echo why; false'
+  [ "$status" -eq 1 ]
+  [[ "$output" == *why*"FAIL: false fails"* ]]
+  run refute_vm "false passes" false
+  [ "$status" -eq 0 ]
+  run refute_vm "true fails" true
+  [ "$status" -eq 1 ]
+}
+
+@test "assert_same_fingerprint passes on a match, and saves the diff when they differ" {
+  printf 'a\nb\n' > "$BATS_TEST_TMPDIR/one"
+  printf 'a\nb\n' > "$BATS_TEST_TMPDIR/two"
+  printf 'a\nc\n' > "$BATS_TEST_TMPDIR/three"
+
+  run assert_same_fingerprint "$BATS_TEST_TMPDIR/one" "$BATS_TEST_TMPDIR/two"
+  [ "$status" -eq 0 ]
+
+  run assert_same_fingerprint "$BATS_TEST_TMPDIR/one" "$BATS_TEST_TMPDIR/three"
+  [ "$status" -eq 1 ]
+  grep -qx -- '-b' "$SMOKETEST_RESULTS/fingerprint.diff"
+  grep -qx -- '+c' "$SMOKETEST_RESULTS/fingerprint.diff"
+}
+
+@test "skip exits with the skip status" {
+  run skip "not today"
+  [ "$status" -eq "$SKIP_STATUS" ]
+  [[ "$output" == *"SKIP: not today"* ]]
+}
+
+@test "every scenario sets a description, image and tier, and defines scenario" {
+  for file in "$REPO"/smoketest/scenarios/*.sh; do
+    (
+      unset DESCRIPTION IMAGE TIER
+      unset -f scenario
+      source "$file"
+      [ -n "$DESCRIPTION" ] || { echo "$file: no DESCRIPTION"; exit 1; }
+      [[ "$IMAGE" == base || "$IMAGE" == installed ]] || { echo "$file: IMAGE is '$IMAGE'"; exit 1; }
+      [[ "$TIER" == pr || "$TIER" == nightly ]] || { echo "$file: TIER is '$TIER'"; exit 1; }
+      declare -f scenario > /dev/null || { echo "$file: no scenario function"; exit 1; }
+    )
+  done
+}
+
+@test "every baseline is a commit in this repo" {
+  sed 's/#.*//' "$REPO/smoketest/baselines" | awk 'NF { print $1 }' | while read -r ref; do
+    git -C "$REPO" rev-parse --verify --quiet "$ref^{commit}" > /dev/null || { echo "unknown: $ref"; exit 1; }
+  done
+}