diff --git a/skills/coredesktop-dev-sibling.md b/skills/coredesktop-dev-sibling.md index a7dbc42..cd5510e 100644 --- a/skills/coredesktop-dev-sibling.md +++ b/skills/coredesktop-dev-sibling.md @@ -77,7 +77,8 @@ coredesktop-dev/ update.sh run on every `update` bin/coredesktop-dev the command (symlinked into ~/.local/bin) hooks/pre-push git hook for the working copy (opt-in) - lib/ shared functions, if bin/ grows large + lib/ one file per subcommand, plus common.sh and vm.sh + vm/ provision.sh and cloud-init user-data for the smoketest VM tests/ bats tests for this project Makefile `make check`: shellcheck + bats for this repo .shellcheckrc copy CoreDesktop's @@ -95,6 +96,8 @@ added later reach existing machines on their next `update`. All are in Arch's | `shellcheck` | `make lint` | | `bats` | `make test` (CoreDesktop's tests use plain bats, no helper libraries) | | `tea` | Gitea's CLI, for pull requests on gitea.federated.computer | +| `qemu-base` | The smoketest's virtual machines | +| `libisoburn` | `xorriso`, to make the smoketest VM's cloud-init disk | `make`, `git` and `jq` already come with CoreDesktop. Don't add `shfmt` until CoreDesktop decides on a formatting style; it has no config for it today. @@ -134,7 +137,7 @@ default answer. No prompts. It prints what it does and exits non-zero only on real failure: -1. `install_packages shellcheck bats tea` +1. `install_packages shellcheck bats tea qemu-base libisoburn` 2. Make `~/.local/bin/coredesktop-dev` a symlink to `bin/coredesktop-dev` in this repo, so each pull updates the command without copying. 3. If the pre-push hook is installed in the working copy, refresh it from @@ -222,6 +225,26 @@ Installs `hooks/pre-push` into the working copy's `.git/hooks/pre-push`. The overwrite a pre-push hook that this project didn't write: write a marker comment into the hook and check for it. +#### `coredesktop-dev smoketest` + +Runs CoreDesktop's end-to-end scenarios on QEMU virtual machines. The +scenarios, the fingerprint and health scripts, and the helpers they use live +in CoreDesktop's `smoketest/` folder, because they change with the code they +test; read its README for the contract between them and this driver. This +project only provides the VMs: `vm_exec`, `vm_put`, `vm_reboot`, `VM_GIT_SSH` +and `VM_GIT_URL`, then calls `smoketest_prepare` and the scenario. + +- `list`, `run [--tier pr|all] [--ref REF] [--keep] [scenario...]`, + `images [base|installed [REF]]`, `shell [base|installed]`, `clean [--all]`. +- Images, the shared pacman cache, the SSH key and run logs live in + `~/.cache/coredesktop-dev/smoketest`, never in either copy of CoreDesktop. +- **base** is Arch's cloud image plus archinstall's GNOME desktop profile + (`vm/provision.sh`), with automatic login and passwordless sudo for `tester`. + **installed** is base after `install.sh` at the commit the branch started + from. Each scenario runs on a throwaway qcow2 overlay of one of them. +- Commands in the VM run in an interactive bash with the GNOME session's + environment, as they would in GNOME Terminal. + ### What not to do - Don't copy hooks into CoreDesktop's `updates.d/`: use `update.sh`, as the @@ -278,9 +301,10 @@ Cases worth a test each: - [ ] The repo's README says to install with `sibling add ` - [ ] `install.sh` asks for the working copy, refuses the deployed copy, saves `~/.config/coredesktop-dev/config`, and is safe to re-run -- [ ] `update.sh` installs shellcheck, bats and tea, keeps the +- [ ] `update.sh` installs shellcheck, bats, tea, QEMU and xorriso, keeps the `~/.local/bin/coredesktop-dev` symlink, and never prompts -- [ ] `doctor`, `check`, `try`, `deployed` and `hooks` work and have `--help` +- [ ] `doctor`, `check`, `try`, `deployed`, `hooks` and `smoketest` work and + have `--help` - [ ] `check` lints untracked scripts in the working copy - [ ] `try` isolates `HOME`, `XDG_*` and `XDG_RUNTIME_DIR`, unsets the display and session variables, uses `dbus-run-session`, and warns that diff --git a/smoketest/README.md b/smoketest/README.md new file mode 100644 index 0000000..b5a4080 --- /dev/null +++ b/smoketest/README.md @@ -0,0 +1,111 @@ +# Smoketest + +End-to-end tests of `install.sh` and `update` on a real Arch + GNOME virtual +machine. The bats tests in `tests/` check the logic with fake commands; these +check what fakes can't: pacman, yay, Flatpak, systemd, ufw, Snapper on btrfs, +GNOME settings, and a reboot. + +This folder holds what to test: the scenarios, and the scripts that describe a +CoreDesktop machine. The VM itself (QEMU, images, the package cache) comes from +the `coredesktop-dev` sibling project, which only contributors install: + +``` +sibling add +coredesktop-dev smoketest list # scenarios and their tier +coredesktop-dev smoketest run # the quick (pr) scenarios +coredesktop-dev smoketest run --tier all # everything, including a fresh install +coredesktop-dev smoketest run 40-failing-migration +``` + +It tests the working copy's `HEAD` commit, so commit first. + +## How it works + +Each scenario runs on a throwaway copy of one of two VM images: + +- **base**: Arch with GNOME, as archinstall's desktop/GNOME profile leaves it, + with a user `tester` who logs in to GNOME automatically. +- **installed**: base after `install.sh`, at an older commit of CoreDesktop. + Before the scenario starts, `update` brings it to the commit under test. + +Inside the VM, a bare repo (`/srv/smoketest/coredesktop.git`) stands in for +Gitea. git's `insteadOf` sends CoreDesktop's real URLs there, so `install.sh` +and `update` run unchanged. The harness pushes the commit under test to it, and +scenarios push further commits to simulate a new release: a new migration, a +broken hook, a change to `update` itself. + +After a scenario, `fingerprint` describes the machine as text (packages, +Flatpaks, services, firewall, CoreDesktop's files in /etc, GNOME settings...). +`10-fresh-install` saves the fingerprint of a fresh install, and +`30-upgrade-from-baseline` checks that installing an older commit from +`baselines` and then running `update` gives the same machine. If they differ, +an install.sh change is probably missing its migration. + +## Files + +| File | What it is | +|------|------------| +| `scenarios/NN-name.sh` | One scenario each (see below) | +| `lib/scenario.sh` | Helpers and assertions the scenarios use | +| `fingerprint` | Runs in the VM; prints the machine's fingerprint | +| `health` | Runs in the VM; checks the machine works, exits 1 if not | +| `baselines` | Older commits that `30-upgrade-from-baseline` installs first | + +## Writing a scenario + +A scenario is a bash file that sets a few variables and defines `scenario`: + +```bash +DESCRIPTION="update runs a new migration once" +IMAGE=installed # base or installed +TIER=pr # pr (quick, uses the installed image) or nightly + +scenario() { + upstream_commit "Add a migration" 'echo "touch ~/migrated" > migrations/9999999999.sh' + cd_update + assert_status 0 + assert_vm "the migration ran" 'test -f ~/migrated' +} +``` + +`VARIANTS=(a b c)` runs the scenario once per value, each in its own VM, with +`$VARIANT` set. `scenario` runs with `set -e`, so any failing command or +assertion fails it. `skip "reason"` skips it. + +Helpers, from `lib/scenario.sh`: + +| Helper | | +|--------|---| +| `cd_install` | Run install.sh as `wget ... \| bash` would, from the fake upstream's main | +| `cd_update [VAR=value...]` | Run `update` | +| `run_in_vm NAME SCRIPT` | Run SCRIPT in the VM; sets `$OUTPUT` and `$STATUS`, logs to NAME.log | +| `upstream_set REF` | Point the fake upstream's main at REF, a commit in this repo | +| `upstream_commit MESSAGE SCRIPT` | Run SCRIPT in a clone of the upstream (in the VM), commit, push | +| `fingerprint FILE` / `health` | Save the fingerprint / run the health checks | +| `assert_status N`, `assert_output TEXT`, `refute_output TEXT` | Check the last `run_in_vm` | +| `assert_vm DESCRIPTION SCRIPT` | SCRIPT must succeed in the VM | +| `assert_same_fingerprint EXPECTED ACTUAL` | The two must match; saves the diff | + +## The contract with the driver + +`coredesktop-dev` boots the VM, defines these, sources `lib/scenario.sh`, calls +`smoketest_prepare`, then the scenario's `scenario` function: + +| Name | | +|------|---| +| `vm_exec SCRIPT` | Run bash SCRIPT in the VM as `tester`, inside the GNOME session's environment, stdin from /dev/null. Returns its exit status. | +| `vm_put LOCAL REMOTE` | Copy a file into the VM | +| `vm_reboot` | Reboot and wait until `tester` is logged in to GNOME again | +| `VM_GIT_SSH`, `VM_GIT_URL` | `GIT_SSH_COMMAND` and the `ssh://` prefix to push to a path in the VM | +| `SMOKETEST_REF` | The commit under test | +| `SMOKETEST_RESULTS` | This scenario's folder for logs and fingerprints | +| `SMOKETEST_SHARED` | A folder shared by all scenarios in the run | +| `IMAGE_REF` | For the installed image, the commit it was installed at | + +Anything that boots a VM can drive these scenarios, e.g. a CI runner with KVM. + +## Not covered + +NVIDIA drivers, firmware updates (`updates.d/90-firmware.sh`), and anything that +needs a real Core, a Tailscale login or a Claude login (`configure`). Check +those by hand. diff --git a/smoketest/baselines b/smoketest/baselines new file mode 100644 index 0000000..f350a09 --- /dev/null +++ b/smoketest/baselines @@ -0,0 +1,12 @@ +# Older commits that 30-upgrade-from-baseline installs, then updates to the +# commit under test, checking the result matches a fresh install. One commit +# per line; # starts a comment. Each one takes a full install, so keep this to +# a few points where machines in use were installed. + +ca658dc # 2026-09-29 various fixes +d8d93a4 # 2026-10-20 the last commit before migrations/ +0c0ddf5 # 2026-10-29 Add Federated Calendar + +# The first commit on Federated Gitea. Machines this old may well not reach +# HEAD with update alone; uncomment to find out. +# a48dc27 # 2026-04-29 fix coreupdate to keep fresh diff --git a/smoketest/fingerprint b/smoketest/fingerprint new file mode 100755 index 0000000..64a7f2a --- /dev/null +++ b/smoketest/fingerprint @@ -0,0 +1,82 @@ +#!/bin/bash +# +# Purpose : Describe this CoreDesktop machine as text, so two machines can be +# compared with diff (see README.md). Runs in the smoketest VM. +# +# Only what CoreDesktop decides goes in: names, not versions, and no times, so +# a fresh install and an updated older install of the same commit match. + +COREDESKTOP="$HOME/.local/share/coredesktop" + +section() { + printf '\n## %s\n' "$1" +} + +section "Explicitly installed packages" +pacman -Qqe + +section "Packages not from the repos (AUR)" +pacman -Qqm + +section "Flatpaks" +flatpak list --app --columns=application | sort + +section "pipx apps" +pipx list --short 2> /dev/null | cut -d' ' -f1 | sort + +section "Enabled system units" +systemctl list-unit-files --state=enabled --no-legend | awk '{print $1}' | sort + +section "Enabled user units" +systemctl --user list-unit-files --state=enabled --no-legend | awk '{print $1}' | sort + +section "Groups" +id -nG | tr ' ' '\n' | sort + +section "Firewall" +sudo ufw status verbose + +section "pacman repos" +grep -E '^\[' /etc/pacman.conf + +section "SSH server" +sudo sshd -T 2> /dev/null | sort + +section "CoreDesktop's files in /etc" +sudo find /etc -path '*coredesktop*' -type f -print0 | sort -z | sudo xargs -0r sha256sum + +section "Snapper" +if [ -f /etc/snapper/configs/root ]; then + sudo snapper -c root get-config | sort +else + echo "no root config" +fi + +section "Migrations run" +ls ~/.local/state/coredesktop/migrations 2> /dev/null + +section "Commands in ~/.local/bin" +for cmd in update refresh sibling version configure audit; do + if cmp -s "$HOME/.local/bin/$cmd" "$COREDESKTOP/$cmd"; then + echo "$cmd: current" + else + echo "$cmd: missing or out of date" + fi +done + +section "Shell" +grep -n coredesktop ~/.bashrc + +section "GNOME extensions" +gsettings get org.gnome.shell enabled-extensions +ls ~/.local/share/gnome-shell/extensions 2> /dev/null + +# The value of each key gnome-settings.dconf sets, which apply-gnome-settings.sh +# loads under /org/gnome/shell/extensions/ +section "GNOME settings" +awk '/^\[/ { dir = substr($0, 2, length($0) - 2); next } + /=/ { print dir "/" substr($0, 1, index($0, "=") - 1) }' \ + "$COREDESKTOP/gnome/gnome-settings.dconf" \ + | while read -r key; do + echo "$key = $(dconf read "/org/gnome/shell/extensions/$key")" + done diff --git a/smoketest/health b/smoketest/health new file mode 100755 index 0000000..8577af2 --- /dev/null +++ b/smoketest/health @@ -0,0 +1,102 @@ +#!/bin/bash +# +# Purpose : Check that this CoreDesktop machine works: services, firewall, +# desktop, and CoreDesktop's own commands. Runs in the smoketest VM. +# Prints ok or FAIL for each check, and exits 1 if any failed. + +COREDESKTOP="$HOME/.local/share/coredesktop" + +# Units that may fail in the VM without anything being wrong with CoreDesktop +EXPECTED_FAILED_UNITS=( + fwupd-refresh.service # the VM's firmware has nothing for fwupd to check +) + +FAILED=0 + +check() { + local description="$1" + shift + if "$@" &> /dev/null; then + echo "ok $description" + else + echo "FAIL $description" + FAILED=1 + fi +} + +# Failed units, apart from EXPECTED_FAILED_UNITS. $1 is --system or --user. +failed_units() { + systemctl "$1" list-units --state=failed --no-legend --plain | awk '{print $1}' \ + | grep -vxF -f <(printf '%s\n' "${EXPECTED_FAILED_UNITS[@]}") +} + +# The unit is enabled, unless audit turned it off +enabled_unless_off() { + grep -qsx "system $1" "$HOME/.config/coredesktop/services-off" && return 0 + systemctl is-enabled "$1" +} + +up_to_date_with_upstream() { + git -C "$COREDESKTOP" fetch --quiet \ + && [ "$(git -C "$COREDESKTOP" rev-parse HEAD)" = "$(git -C "$COREDESKTOP" rev-parse '@{upstream}')" ] +} + +no_local_changes() { + [ -z "$(git -C "$COREDESKTOP" status --porcelain --untracked-files=no)" ] +} + +# GNOME Shell is running for this user on a graphical seat +desktop_running() { + pgrep -u "$USER" -x gnome-shell \ + && loginctl list-sessions --no-legend | awk -v u="$USER" '$3 == u && $4 == "seat0"' | grep -q . +} + +firewall_active() { + sudo ufw status | grep -qx 'Status: active' +} + +commands_current() { + local cmd + for cmd in update refresh sibling version configure audit; do + cmp -s "$HOME/.local/bin/$cmd" "$COREDESKTOP/$cmd" || return 1 + done +} + +version_ok() { + local out + out="$("$HOME/.local/bin/version")" || return 1 + [[ "$out" == CoreDesktop* && "$out" != *modified* ]] +} + +for unit in --system --user; do + failed="$(failed_units "$unit")" + if [ -z "$failed" ]; then + echo "ok no failed ${unit#--} units" + else + echo "FAIL failed ${unit#--} units: $(echo "$failed" | tr '\n' ' ')" + FAILED=1 + fi +done + +# packages.conf lists the services install.sh enables +source "$COREDESKTOP/packages.conf" +for service in "${SERVICES[@]}" "${START_SERVICES[@]}"; do + check "$service is enabled" enabled_unless_off "$service" +done +for service in "${START_SERVICES[@]}"; do + check "$service is running" systemctl is-active "$service" +done + +check "GNOME is running" desktop_running +check "the firewall is on" firewall_active +check "the Federated repo is in pacman.conf" grep -q '^\[federated\]' /etc/pacman.conf +check "Snapper is set up (if / is btrfs)" \ + bash -c '[ "$(findmnt -no FSTYPE /)" != btrfs ] || [ -f /etc/snapper/configs/root ]' +check "CoreDesktop is on upstream's latest commit" up_to_date_with_upstream +check "CoreDesktop's checkout has no local changes" no_local_changes +check "the commands in ~/.local/bin are current" commands_current +check "version works" version_ok +check "audit --list works" "$HOME/.local/bin/audit" --list +check "the shell setup is in ~/.bashrc" grep -q coredesktop "$HOME/.bashrc" + +exit $FAILED diff --git a/smoketest/lib/scenario.sh b/smoketest/lib/scenario.sh new file mode 100644 index 0000000..759fe91 --- /dev/null +++ b/smoketest/lib/scenario.sh @@ -0,0 +1,199 @@ +#!/bin/bash +# +# Purpose : Helpers and assertions for smoketest scenarios (see ../README.md). +# Sourced by the driver after it defines vm_exec, vm_put and +# vm_reboot. + +SMOKETEST_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +# The CoreDesktop repo whose commits are tested. Overridable for the tests. +SMOKETEST_REPO="${SMOKETEST_REPO:-$(cd "$SMOKETEST_DIR/.." && pwd)}" + +# In the VM: the fake upstream, the scratch clone upstream_commit works in, and +# where the fingerprint and health scripts are copied to +SMOKETEST_VM_DIR="${SMOKETEST_VM_DIR:-/srv/smoketest}" +UPSTREAM="$SMOKETEST_VM_DIR/coredesktop.git" +WORK="$SMOKETEST_VM_DIR/work" + +# Every URL CoreDesktop has been cloned from, so install.sh at any baseline +# clones the fake upstream instead +COREDESKTOP_URLS=( + https://gitea.federated.computer/Federated_Computer/coredesktop.git + https://gitea.young.computer/david/coredesktop.git +) + +# Exit status of a scenario that skipped itself +SKIP_STATUS=3 + +# Output, exit status, name and log file of the last run_in_vm +OUTPUT="" +STATUS=0 +LAST_RUN="" +LAST_LOG="" +RUN_COUNT=0 + +q() { + printf '%q' "$1" +} + +fail() { + echo "FAIL: $*" >&2 + exit 1 +} + +skip() { + echo "SKIP: $*" + exit "$SKIP_STATUS" +} + +# run_in_vm NAME SCRIPT: run SCRIPT in the VM, keeping its output in $OUTPUT, +# its exit status in $STATUS, and both in NN-NAME.log. Doesn't fail on its own: +# check with assert_status. +run_in_vm() { + RUN_COUNT=$((RUN_COUNT + 1)) + LAST_RUN="$1" + LAST_LOG="$SMOKETEST_RESULTS/$(printf '%02d' "$RUN_COUNT")-$1.log" + echo "== $1" + OUTPUT="$(vm_exec "$2" 2>&1)" && STATUS=0 || STATUS=$? + printf '%s\n' "$OUTPUT" > "$LAST_LOG" + echo "== $1 exited $STATUS (log: $LAST_LOG)" +} + +# Called by the driver once the VM is up, before the scenario +smoketest_prepare() { + local urls="" url + for url in "${COREDESKTOP_URLS[@]}"; do + urls+=" $(q "$url")" + done + vm_exec "set -e + [ -d $(q "$UPSTREAM") ] || git init -q --bare --initial-branch=main $(q "$UPSTREAM") + git config --global user.name 'CoreDesktop Smoketest' + git config --global user.email smoketest@localhost + git config --global --unset-all url.$(q "$UPSTREAM").insteadOf || true + for url in $urls; do + git config --global --add url.$(q "$UPSTREAM").insteadOf \"\$url\" + done" || fail "couldn't set up the fake upstream in the VM" + + if [ "${IMAGE:-}" = installed ]; then + git -C "$SMOKETEST_REPO" merge-base --is-ancestor "$IMAGE_REF" "$SMOKETEST_REF" \ + || fail "the installed image is at $IMAGE_REF, which isn't in the history of" \ + "the commit under test. Rebuild it: coredesktop-dev smoketest images --installed" + upstream_set "$SMOKETEST_REF" + cd_update + assert_status 0 + else + upstream_set "$SMOKETEST_REF" + fi +} + +# Called by the driver to make the installed image from the base image +smoketest_install_image() { + smoketest_prepare + upstream_set "$1" + cd_install + assert_status 0 +} + +# upstream_set REF: point the fake upstream's main at REF, a commit in this repo +upstream_set() { + local sha + sha="$(git -C "$SMOKETEST_REPO" rev-parse --verify --quiet "$1^{commit}")" \ + || fail "no such commit: $1" + GIT_SSH_COMMAND="${VM_GIT_SSH:-ssh}" git -C "$SMOKETEST_REPO" push --quiet --force \ + "${VM_GIT_URL:-}$UPSTREAM" "$sha:refs/heads/main" || fail "couldn't push $1 to the VM" +} + +# upstream_commit MESSAGE SCRIPT: release a new commit, made by running SCRIPT +# (in the VM) in a clone of the fake upstream's main +upstream_commit() { + run_in_vm commit "set -e + rm -rf $(q "$WORK") + git clone -q $(q "$UPSTREAM") $(q "$WORK") + cd $(q "$WORK") + $2 + git add -A + git commit -qm $(q "$1") + git push -q origin HEAD:main" + assert_status 0 +} + +# Run install.sh the way the README's wget ... | bash does: main's install.sh, +# with no terminal to ask questions on +cd_install() { + run_in_vm install "git --git-dir=$(q "$UPSTREAM") show main:install.sh > ~/install.sh \ + && cd && TERM=dumb bash ~/install.sh" +} + +# cd_update [VAR=value...]: run update, with those variables set +cd_update() { + local vars="" var + for var in "$@"; do + vars+=" $(q "$var")" + done + run_in_vm update "TERM=dumb env$vars ~/.local/bin/update" +} + +# fingerprint FILE: save the machine's fingerprint to FILE +fingerprint() { + vm_put "$SMOKETEST_DIR/fingerprint" "$SMOKETEST_VM_DIR/fingerprint" \ + || fail "couldn't copy the fingerprint script to the VM" + vm_exec "bash $(q "$SMOKETEST_VM_DIR/fingerprint")" > "$1" || fail "fingerprint failed (see $1)" +} + +# Where 10-fresh-install saves the fingerprint of a fresh install of the +# commit under test +reference_fingerprint() { + echo "$SMOKETEST_SHARED/fresh-install.fingerprint" +} + +# Run the health checks; fail the scenario if any fails +health() { + vm_put "$SMOKETEST_DIR/health" "$SMOKETEST_VM_DIR/health" \ + || fail "couldn't copy the health script to the VM" + run_in_vm health "bash $(q "$SMOKETEST_VM_DIR/health")" + printf '%s\n' "$OUTPUT" + [ "$STATUS" -eq 0 ] || fail "the health checks failed" +} + +assert_status() { + [ "$STATUS" -eq "$1" ] || fail "$LAST_RUN exited $STATUS, expected $1 (see $LAST_LOG)" +} + +assert_output() { + [[ "$OUTPUT" == *"$1"* ]] || fail "$LAST_RUN's output doesn't contain \"$1\" (see $LAST_LOG)" +} + +refute_output() { + [[ "$OUTPUT" != *"$1"* ]] || fail "$LAST_RUN's output contains \"$1\" (see $LAST_LOG)" +} + +# assert_vm DESCRIPTION SCRIPT: SCRIPT must succeed in the VM +assert_vm() { + local out + out="$(vm_exec "$2" 2>&1)" || { printf '%s\n' "$out"; fail "$1"; } + echo "ok: $1" +} + +# refute_vm DESCRIPTION SCRIPT: SCRIPT must fail in the VM +refute_vm() { + local out + if out="$(vm_exec "$2" 2>&1)"; then + printf '%s\n' "$out" + fail "$1" + fi + echo "ok: $1" +} + +# vm_value SCRIPT: print SCRIPT's output, failing the scenario if SCRIPT fails +vm_value() { + vm_exec "$1" || fail "couldn't run in the VM: $1" +} + +# assert_same_fingerprint EXPECTED ACTUAL: the fingerprint files must match +assert_same_fingerprint() { + local diff="$SMOKETEST_RESULTS/fingerprint.diff" + if ! diff -u --label expected --label actual "$1" "$2" > "$diff"; then + cat "$diff" + fail "the machine differs from $1 (diff: $diff)" + fi + echo "ok: same fingerprint as $1" +} diff --git a/smoketest/scenarios/10-fresh-install.sh b/smoketest/scenarios/10-fresh-install.sh new file mode 100644 index 0000000..609224f --- /dev/null +++ b/smoketest/scenarios/10-fresh-install.sh @@ -0,0 +1,19 @@ +#!/bin/bash +# shellcheck disable=SC2034 +DESCRIPTION="install.sh on a fresh Arch + GNOME machine; saves the reference fingerprint" +IMAGE=base +TIER=nightly + +scenario() { + cd_install + assert_status 0 + assert_vm "install.sh left no local changes in the checkout" \ + '[ -z "$(git -C ~/.local/share/coredesktop status --porcelain --untracked-files=no)" ]' + + # The README says to reboot after installing + vm_reboot + health + + fingerprint "$SMOKETEST_RESULTS/installed.fingerprint" + cp "$SMOKETEST_RESULTS/installed.fingerprint" "$(reference_fingerprint)" +} diff --git a/smoketest/scenarios/11-refresh.sh b/smoketest/scenarios/11-refresh.sh new file mode 100644 index 0000000..f0c641e --- /dev/null +++ b/smoketest/scenarios/11-refresh.sh @@ -0,0 +1,17 @@ +#!/bin/bash +# shellcheck disable=SC2034 +DESCRIPTION="refresh (re-running install.sh) on an installed machine changes nothing" +IMAGE=installed +TIER=pr + +scenario() { + fingerprint "$SMOKETEST_RESULTS/before.fingerprint" + + run_in_vm refresh 'TERM=dumb ~/.local/bin/refresh' + assert_status 0 + refute_output "Running migration" + + fingerprint "$SMOKETEST_RESULTS/after.fingerprint" + assert_same_fingerprint "$SMOKETEST_RESULTS/before.fingerprint" "$SMOKETEST_RESULTS/after.fingerprint" + health +} diff --git a/smoketest/scenarios/20-noop-update.sh b/smoketest/scenarios/20-noop-update.sh new file mode 100644 index 0000000..dd849b3 --- /dev/null +++ b/smoketest/scenarios/20-noop-update.sh @@ -0,0 +1,21 @@ +#!/bin/bash +# shellcheck disable=SC2034 +DESCRIPTION="update with nothing new to do installs nothing and changes nothing" +IMAGE=installed +TIER=pr + +# smoketest_prepare has already updated the machine to the commit under test +scenario() { + fingerprint "$SMOKETEST_RESULTS/before.fingerprint" + + cd_update + assert_status 0 + assert_output "Done." + refute_output "Running migration" + # install_packages prints this only when it installs something + refute_output "Installing:" + + fingerprint "$SMOKETEST_RESULTS/after.fingerprint" + assert_same_fingerprint "$SMOKETEST_RESULTS/before.fingerprint" "$SMOKETEST_RESULTS/after.fingerprint" + health +} diff --git a/smoketest/scenarios/30-upgrade-from-baseline.sh b/smoketest/scenarios/30-upgrade-from-baseline.sh new file mode 100644 index 0000000..7010cad --- /dev/null +++ b/smoketest/scenarios/30-upgrade-from-baseline.sh @@ -0,0 +1,25 @@ +#!/bin/bash +# shellcheck disable=SC2034 +DESCRIPTION="install an older commit, update to the commit under test: same machine as a fresh install" +IMAGE=base +TIER=nightly +mapfile -t VARIANTS < <(sed 's/#.*//' "$(dirname "${BASH_SOURCE[0]}")/../baselines" | awk 'NF { print $1 }') + +scenario() { + [ -f "$(reference_fingerprint)" ] \ + || skip "needs the fingerprint 10-fresh-install saves; run both in the same run" + + upstream_set "$VARIANT" + cd_install + assert_status 0 + vm_reboot + + upstream_set "$SMOKETEST_REF" + cd_update + assert_status 0 + vm_reboot + health + + fingerprint "$SMOKETEST_RESULTS/updated.fingerprint" + assert_same_fingerprint "$(reference_fingerprint)" "$SMOKETEST_RESULTS/updated.fingerprint" +} diff --git a/smoketest/scenarios/31-update-replaces-itself.sh b/smoketest/scenarios/31-update-replaces-itself.sh new file mode 100644 index 0000000..821216d --- /dev/null +++ b/smoketest/scenarios/31-update-replaces-itself.sh @@ -0,0 +1,19 @@ +#!/bin/bash +# shellcheck disable=SC2034 +DESCRIPTION="a release that changes update and adds a hook: the new update and hook run in the same update" +IMAGE=installed +TIER=pr + +scenario() { + upstream_commit "Change update and add a hook" ' + sed -i "s/^FAILED=()\$/echo \"smoketest: the new update ran\"\nFAILED=()/" update + grep -q "smoketest: the new update ran" update + printf "%s\n" "#!/bin/bash" "run_update() { touch ~/smoketest-hook-ran; }" > updates.d/50-smoketest.sh' + + cd_update + assert_status 0 + assert_output "smoketest: the new update ran" + assert_vm "the new hook ran" 'test -f ~/smoketest-hook-ran' + assert_vm "~/.local/bin/update is the new one" \ + 'cmp ~/.local/bin/update ~/.local/share/coredesktop/update' +} diff --git a/smoketest/scenarios/32-new-migration.sh b/smoketest/scenarios/32-new-migration.sh new file mode 100644 index 0000000..2252bc7 --- /dev/null +++ b/smoketest/scenarios/32-new-migration.sh @@ -0,0 +1,22 @@ +#!/bin/bash +# shellcheck disable=SC2034 +DESCRIPTION="a release with a new migration: it runs once, and is recorded" +IMAGE=installed +TIER=pr + +scenario() { + upstream_commit "Add a migration" ' + echo "echo ran >> ~/smoketest-migration" > "migrations/$(date +%s).sh"' + + cd_update + assert_status 0 + assert_output "Running migration" + cd_update + assert_status 0 + refute_output "Running migration" + + assert_vm "the migration ran exactly once" '[ "$(cat ~/smoketest-migration)" = ran ]' + assert_vm "the migration is recorded" ' + cd ~/.local/share/coredesktop/migrations + for m in *.sh; do test -f ~/.local/state/coredesktop/migrations/$m; done' +} diff --git a/smoketest/scenarios/40-failing-migration.sh b/smoketest/scenarios/40-failing-migration.sh new file mode 100644 index 0000000..9ad9325 --- /dev/null +++ b/smoketest/scenarios/40-failing-migration.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# shellcheck disable=SC2034 +DESCRIPTION="a failing migration stops later ones, fails the update, and runs again next time" +IMAGE=installed +TIER=pr + +scenario() { + # The first fails while ~/smoketest-block exists; the second must wait for it + upstream_commit "Add two migrations" ' + now=$(date +%s) + echo "[ ! -e ~/smoketest-block ] && echo first >> ~/smoketest-ran" > "migrations/$now.sh" + echo "echo second >> ~/smoketest-ran" > "migrations/$((now + 1)).sh"' + assert_vm "block the first migration" 'touch ~/smoketest-block' + + cd_update + assert_status 1 + assert_output "failed; it will run again on the next update" + assert_output "Done, with errors in: 12-migrations.sh" + refute_vm "neither migration ran" 'test -e ~/smoketest-ran' + + assert_vm "unblock the first migration" 'rm ~/smoketest-block' + cd_update + assert_status 0 + assert_vm "both ran, in order" '[ "$(cat ~/smoketest-ran)" = "$(printf "first\nsecond")" ]' + + cd_update + assert_status 0 + assert_vm "neither ran again" '[ "$(wc -l < ~/smoketest-ran)" -eq 2 ]' +} diff --git a/smoketest/scenarios/41-failing-hook.sh b/smoketest/scenarios/41-failing-hook.sh new file mode 100644 index 0000000..32747ce --- /dev/null +++ b/smoketest/scenarios/41-failing-hook.sh @@ -0,0 +1,17 @@ +#!/bin/bash +# shellcheck disable=SC2034 +DESCRIPTION="a failing update hook fails the update but not the hooks after it" +IMAGE=installed +TIER=pr + +scenario() { + upstream_commit "Add a failing hook" ' + printf "%s\n" "#!/bin/bash" "run_update() { echo smoketest: failing; return 1; }" > updates.d/13-smoketest-fail.sh + printf "%s\n" "#!/bin/bash" "run_update() { touch ~/smoketest-after; }" > updates.d/96-smoketest-after.sh' + + cd_update + assert_status 1 + assert_output "smoketest: failing" + assert_output "Done, with errors in: 13-smoketest-fail.sh" + assert_vm "the hook after it ran" 'test -f ~/smoketest-after' +} diff --git a/smoketest/scenarios/42-upstream-unreachable.sh b/smoketest/scenarios/42-upstream-unreachable.sh new file mode 100644 index 0000000..db101b8 --- /dev/null +++ b/smoketest/scenarios/42-upstream-unreachable.sh @@ -0,0 +1,18 @@ +#!/bin/bash +# shellcheck disable=SC2034 +DESCRIPTION="update when CoreDesktop's repo can't be reached runs the hooks on disk and reports it" +IMAGE=installed +TIER=pr + +scenario() { + assert_vm "take the upstream away" "sudo mv $UPSTREAM $UPSTREAM.away" + cd_update + assert_status 1 + assert_output "could not pull the latest CoreDesktop" + assert_output "Done, with errors in: git pull" + assert_output "Updating Arch..." + + assert_vm "bring it back" "sudo mv $UPSTREAM.away $UPSTREAM" + cd_update + assert_status 0 +} diff --git a/smoketest/scenarios/43-local-edits.sh b/smoketest/scenarios/43-local-edits.sh new file mode 100644 index 0000000..165d32b --- /dev/null +++ b/smoketest/scenarios/43-local-edits.sh @@ -0,0 +1,27 @@ +#!/bin/bash +# shellcheck disable=SC2034 +DESCRIPTION="local edits to the checkout: kept if upstream doesn't touch the file, reported if it does" +IMAGE=installed +TIER=pr + +# Expanded in the VM, not here +# shellcheck disable=SC2088 +CD="~/.local/share/coredesktop" + +scenario() { + assert_vm "edit README.md in the checkout" "echo '# local edit' >> $CD/README.md" + upstream_commit "Change packages.conf" 'echo "# upstream change" >> packages.conf' + cd_update + assert_status 0 + assert_vm "the edit is kept" "tail -n 1 $CD/README.md | grep -qx '# local edit'" + assert_vm "the upstream change arrived" "tail -n 1 $CD/packages.conf | grep -qx '# upstream change'" + + upstream_commit "Change README.md" 'echo "# upstream readme change" >> README.md' + cd_update + assert_status 1 + assert_output "Done, with errors in: git pull" + + assert_vm "throw the edit away" "git -C $CD checkout -- README.md" + cd_update + assert_status 0 +} diff --git a/smoketest/scenarios/44-low-disk.sh b/smoketest/scenarios/44-low-disk.sh new file mode 100644 index 0000000..aa3a394 --- /dev/null +++ b/smoketest/scenarios/44-low-disk.sh @@ -0,0 +1,20 @@ +#!/bin/bash +# shellcheck disable=SC2034 +DESCRIPTION="update refuses to start with less than 10 GiB free, unless UPDATE_FORCE=1" +IMAGE=installed +TIER=pr + +scenario() { + # Leave 5 GiB free on / + assert_vm "fill /" ' + avail=$(df -k --output=avail / | tail -n 1 | tr -d " ") + sudo fallocate -l $(( (avail - 5 * 1024 * 1024) * 1024 )) /smoketest-fill' + + cd_update + assert_status 1 + assert_output "updating needs at least 10 GiB" + refute_output "Updating Arch..." + + cd_update UPDATE_FORCE=1 + assert_status 0 +} diff --git a/smoketest/scenarios/50-stale-system.sh b/smoketest/scenarios/50-stale-system.sh new file mode 100644 index 0000000..fb132f0 --- /dev/null +++ b/smoketest/scenarios/50-stale-system.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# shellcheck disable=SC2034 +DESCRIPTION="a machine a month behind on a stale mirror: update picks fresh mirrors, upgrades the kernel, asks for a reboot" +IMAGE=installed +TIER=nightly + +scenario() { + local date kernel_before + date="$(date -d '30 days ago' +%Y/%m/%d)" + + # Go back to Arch as it was a month ago, from the Arch Linux Archive, which + # is also a mirror that stopped syncing a month ago + run_in_vm downgrade "set -e + echo 'Server = https://archive.archlinux.org/repos/$date/\$repo/os/\$arch' \ + | sudo tee /etc/pacman.d/mirrorlist + sudo pacman -Syyuu --noconfirm" + [ "$STATUS" -eq 0 ] || skip "couldn't downgrade to Arch as of $date (see $LAST_LOG)" + vm_reboot + kernel_before="$(vm_value 'uname -r')" + + run_in_vm version '"$HOME/.local/bin/version"' + assert_output "last synced with Arch" + + local snapshots_before + snapshots_before="$(vm_value 'sudo snapper -c root list 2> /dev/null | wc -l')" + + cd_update + assert_status 0 + assert_output "Please reboot when convenient" + refute_vm "the first mirror isn't the archive any more" \ + 'grep -m1 "^Server" /etc/pacman.d/mirrorlist | grep -q archive.archlinux.org' + assert_vm "update took a snapshot first (if / is btrfs)" " + [ \"\$(findmnt -no FSTYPE /)\" != btrfs ] \ + || [ \"\$(sudo snapper -c root list | wc -l)\" -gt $snapshots_before ]" + + vm_reboot + refute_vm "a newer kernel is running" "[ \"\$(uname -r)\" = $(q "$kernel_before") ]" + health +} diff --git a/smoketest/scenarios/51-skip-packages.sh b/smoketest/scenarios/51-skip-packages.sh new file mode 100644 index 0000000..dd77a19 --- /dev/null +++ b/smoketest/scenarios/51-skip-packages.sh @@ -0,0 +1,19 @@ +#!/bin/bash +# shellcheck disable=SC2034 +DESCRIPTION="update doesn't reinstall a package or Flatpak listed in skip-packages" +IMAGE=installed +TIER=pr + +scenario() { + assert_vm "remove Zoom and LibreOffice, and skip them" ' + set -e + sudo flatpak uninstall --system --noninteractive us.zoom.Zoom + sudo pacman -Rns --noconfirm libreoffice-fresh + mkdir -p ~/.config/coredesktop + printf "%s\n" us.zoom.Zoom libreoffice-fresh >> ~/.config/coredesktop/skip-packages' + + cd_update + assert_status 0 + refute_vm "Zoom is still gone" 'flatpak info --system us.zoom.Zoom' + refute_vm "LibreOffice is still gone" 'pacman -Qi libreoffice-fresh' +} diff --git a/smoketest/scenarios/52-siblings.sh b/smoketest/scenarios/52-siblings.sh new file mode 100644 index 0000000..af26ffa --- /dev/null +++ b/smoketest/scenarios/52-siblings.sh @@ -0,0 +1,38 @@ +#!/bin/bash +# shellcheck disable=SC2034 +DESCRIPTION="update pulls a sibling project and runs its update.sh; a failing one is reported" +IMAGE=installed +TIER=pr + +SIB="$SMOKETEST_VM_DIR/smoketest-sibling" + +scenario() { + assert_vm "make a sibling project" "set -e + git init -q $SIB + cd $SIB + echo 'touch ~/sibling-installed' > install.sh + echo 'echo v1 >> ~/sibling-updated' > update.sh + git add -A + git commit -qm v1" + + run_in_vm sibling-add "\"\$HOME/.local/bin/sibling\" add $SIB" + assert_status 0 + assert_vm "its install.sh ran" 'test -f ~/sibling-installed' + + assert_vm "release v2" "set -e + cd $SIB + echo 'echo v2 >> ~/sibling-updated' > update.sh + git commit -qam v2" + cd_update + assert_status 0 + assert_vm "v2's update.sh ran" 'tail -n 1 ~/sibling-updated | grep -qx v2' + + assert_vm "release a v3 whose update.sh fails" "set -e + cd $SIB + echo 'exit 1' > update.sh + git commit -qam v3" + cd_update + assert_status 1 + assert_output "smoketest-sibling's update.sh failed" + assert_output "Done, with errors in: 71-siblings.sh" +} diff --git a/tests/smoketest.bats b/tests/smoketest.bats new file mode 100644 index 0000000..8bb3508 --- /dev/null +++ b/tests/smoketest.bats @@ -0,0 +1,209 @@ +#!/usr/bin/env bats + +load test_helper + +# smoketest/lib/scenario.sh with a "VM" that is just a local shell in the +# test's HOME, and a small repo of its own standing in for CoreDesktop + +setup() { + common_setup + export GIT_AUTHOR_NAME=Test GIT_AUTHOR_EMAIL=test@localhost + export GIT_COMMITTER_NAME=Test GIT_COMMITTER_EMAIL=test@localhost + + # Two commits: install.sh writes which one ran, and what it read from stdin + export SMOKETEST_REPO="$BATS_TEST_TMPDIR/repo" + git init -q "$SMOKETEST_REPO" + cd "$SMOKETEST_REPO" + printf '%s\n' 'echo v1 > ~/installed' 'read -r line; echo "read: $line" >> ~/installed' > install.sh + git add install.sh + git commit -qm v1 + V1="$(git rev-parse HEAD)" + sed -i 's/v1/v2/' install.sh + git commit -qam v2 + V2="$(git rev-parse HEAD)" + cd "$BATS_TEST_TMPDIR" + + export SMOKETEST_VM_DIR="$BATS_TEST_TMPDIR/vm" + mkdir -p "$SMOKETEST_VM_DIR" + export SMOKETEST_RESULTS="$BATS_TEST_TMPDIR/results" + export SMOKETEST_SHARED="$BATS_TEST_TMPDIR/shared" + mkdir -p "$SMOKETEST_RESULTS" "$SMOKETEST_SHARED" + export SMOKETEST_REF="$V2" + + vm_exec() { bash -c "$1" < /dev/null; } + vm_put() { cp "$1" "$2"; } + source "$REPO/smoketest/lib/scenario.sh" +} + +upstream_main() { + git --git-dir="$UPSTREAM" rev-parse main +} + +@test "prepare makes the fake upstream, at the commit under test" { + smoketest_prepare + [ "$(upstream_main)" = "$V2" ] +} + +@test "prepare sends CoreDesktop's URLs to the fake upstream" { + smoketest_prepare + for url in "${COREDESKTOP_URLS[@]}"; do + git clone -q "$url" "$BATS_TEST_TMPDIR/clone" + [ "$(git -C "$BATS_TEST_TMPDIR/clone" rev-parse HEAD)" = "$V2" ] + rm -rf "$BATS_TEST_TMPDIR/clone" + done +} + +@test "prepare can run twice" { + smoketest_prepare + smoketest_prepare + [ "$(git config --global --get-all "url.$UPSTREAM.insteadOf" | wc -l)" -eq "${#COREDESKTOP_URLS[@]}" ] +} + +@test "prepare on the installed image updates it to the commit under test" { + export IMAGE=installed IMAGE_REF="$V1" + mkdir -p "$HOME/.local/bin" + printf '%s\n' '#!/bin/bash' 'echo "update ran with $(git --git-dir="$UPSTREAM" rev-parse main)"' \ + > "$HOME/.local/bin/update" + chmod +x "$HOME/.local/bin/update" + export UPSTREAM + + run smoketest_prepare + [ "$status" -eq 0 ] + grep -qx "update ran with $V2" "$SMOKETEST_RESULTS/01-update.log" +} + +@test "prepare on the installed image fails if the image isn't behind the commit under test" { + export IMAGE=installed IMAGE_REF="$V2" SMOKETEST_REF="$V1" + + run smoketest_prepare + [ "$status" -eq 1 ] + [[ "$output" == *"isn't in the history of the commit under test"* ]] +} + +@test "upstream_set moves main back and forth" { + smoketest_prepare + upstream_set "$V1" + [ "$(upstream_main)" = "$V1" ] + upstream_set "$V2" + [ "$(upstream_main)" = "$V2" ] +} + +@test "upstream_set fails on an unknown commit" { + smoketest_prepare + run upstream_set nosuchcommit + [ "$status" -eq 1 ] + [[ "$output" == *"no such commit: nosuchcommit"* ]] +} + +@test "upstream_commit releases a commit made by its script" { + smoketest_prepare + upstream_commit "Add a file" 'echo hello > new-file' + + [ "$(git --git-dir="$UPSTREAM" rev-parse main^)" = "$V2" ] + [ "$(git --git-dir="$UPSTREAM" show main:new-file)" = hello ] + [ "$(git --git-dir="$UPSTREAM" log -1 --format=%s main)" = "Add a file" ] +} + +@test "upstream_commit fails the scenario if its script fails" { + smoketest_prepare + run upstream_commit "Broken" 'false' + [ "$status" -eq 1 ] + [ "$(upstream_main)" = "$V2" ] +} + +@test "cd_install runs main's install.sh, with nothing on stdin" { + smoketest_prepare + upstream_set "$V1" + cd_install + [ "$STATUS" -eq 0 ] + [ "$(cat "$HOME/installed")" = "$(printf 'v1\nread: ')" ] +} + +@test "run_in_vm keeps the output, status and a numbered log, and doesn't fail" { + run_in_vm first 'echo one; exit 3' + [ "$STATUS" -eq 3 ] + [ "$OUTPUT" = one ] + [ "$(cat "$SMOKETEST_RESULTS/01-first.log")" = one ] + + run_in_vm second 'echo two >&2' + [ "$STATUS" -eq 0 ] + [ "$OUTPUT" = two ] + [ -f "$SMOKETEST_RESULTS/02-second.log" ] +} + +@test "cd_update passes its variables to update" { + mkdir -p "$HOME/.local/bin" + printf '%s\n' '#!/bin/bash' 'echo "force=$UPDATE_FORCE"' > "$HOME/.local/bin/update" + chmod +x "$HOME/.local/bin/update" + + cd_update UPDATE_FORCE=1 + [ "$STATUS" -eq 0 ] + [ "$OUTPUT" = "force=1" ] +} + +@test "assert_status and assert_output fail with the log's path" { + run_in_vm thing 'echo hello; exit 1' + + run assert_status 0 + [ "$status" -eq 1 ] + [[ "$output" == *"thing exited 1, expected 0"*"01-thing.log"* ]] + + run assert_output goodbye + [ "$status" -eq 1 ] + run assert_output hello + [ "$status" -eq 0 ] + run refute_output hello + [ "$status" -eq 1 ] +} + +@test "assert_vm and refute_vm" { + run assert_vm "true passes" true + [ "$status" -eq 0 ] + run assert_vm "false fails" 'echo why; false' + [ "$status" -eq 1 ] + [[ "$output" == *why*"FAIL: false fails"* ]] + run refute_vm "false passes" false + [ "$status" -eq 0 ] + run refute_vm "true fails" true + [ "$status" -eq 1 ] +} + +@test "assert_same_fingerprint passes on a match, and saves the diff when they differ" { + printf 'a\nb\n' > "$BATS_TEST_TMPDIR/one" + printf 'a\nb\n' > "$BATS_TEST_TMPDIR/two" + printf 'a\nc\n' > "$BATS_TEST_TMPDIR/three" + + run assert_same_fingerprint "$BATS_TEST_TMPDIR/one" "$BATS_TEST_TMPDIR/two" + [ "$status" -eq 0 ] + + run assert_same_fingerprint "$BATS_TEST_TMPDIR/one" "$BATS_TEST_TMPDIR/three" + [ "$status" -eq 1 ] + grep -qx -- '-b' "$SMOKETEST_RESULTS/fingerprint.diff" + grep -qx -- '+c' "$SMOKETEST_RESULTS/fingerprint.diff" +} + +@test "skip exits with the skip status" { + run skip "not today" + [ "$status" -eq "$SKIP_STATUS" ] + [[ "$output" == *"SKIP: not today"* ]] +} + +@test "every scenario sets a description, image and tier, and defines scenario" { + for file in "$REPO"/smoketest/scenarios/*.sh; do + ( + unset DESCRIPTION IMAGE TIER + unset -f scenario + source "$file" + [ -n "$DESCRIPTION" ] || { echo "$file: no DESCRIPTION"; exit 1; } + [[ "$IMAGE" == base || "$IMAGE" == installed ]] || { echo "$file: IMAGE is '$IMAGE'"; exit 1; } + [[ "$TIER" == pr || "$TIER" == nightly ]] || { echo "$file: TIER is '$TIER'"; exit 1; } + declare -f scenario > /dev/null || { echo "$file: no scenario function"; exit 1; } + ) + done +} + +@test "every baseline is a commit in this repo" { + sed 's/#.*//' "$REPO/smoketest/baselines" | awk 'NF { print $1 }' | while read -r ref; do + git -C "$REPO" rev-parse --verify --quiet "$ref^{commit}" > /dev/null || { echo "unknown: $ref"; exit 1; } + done +}