112 lines
4.9 KiB
Markdown
112 lines
4.9 KiB
Markdown
# Smoketest
|
|
|
|
End-to-end tests of `install.sh` and `update` on a real Arch + GNOME virtual
|
|
machine. The bats tests in `tests/` check the logic with fake commands; these
|
|
check what fakes can't: pacman, yay, Flatpak, systemd, ufw, Snapper on btrfs,
|
|
GNOME settings, and a reboot.
|
|
|
|
This folder holds what to test: the scenarios, and the scripts that describe a
|
|
CoreDesktop machine. The VM itself (QEMU, images, the package cache) comes from
|
|
the `coredesktop-dev` sibling project, which only contributors install:
|
|
|
|
```
|
|
sibling add <coredesktop-dev url>
|
|
coredesktop-dev smoketest list # scenarios and their tier
|
|
coredesktop-dev smoketest run # the quick (pr) scenarios
|
|
coredesktop-dev smoketest run --tier all # everything, including a fresh install
|
|
coredesktop-dev smoketest run 40-failing-migration
|
|
```
|
|
|
|
It tests the working copy's `HEAD` commit, so commit first.
|
|
|
|
## How it works
|
|
|
|
Each scenario runs on a throwaway copy of one of two VM images:
|
|
|
|
- **base**: Arch with GNOME, as archinstall's desktop/GNOME profile leaves it,
|
|
with a user `tester` who logs in to GNOME automatically.
|
|
- **installed**: base after `install.sh`, at an older commit of CoreDesktop.
|
|
Before the scenario starts, `update` brings it to the commit under test.
|
|
|
|
Inside the VM, a bare repo (`/srv/smoketest/coredesktop.git`) stands in for
|
|
Gitea. git's `insteadOf` sends CoreDesktop's real URLs there, so `install.sh`
|
|
and `update` run unchanged. The harness pushes the commit under test to it, and
|
|
scenarios push further commits to simulate a new release: a new migration, a
|
|
broken hook, a change to `update` itself.
|
|
|
|
After a scenario, `fingerprint` describes the machine as text (packages,
|
|
Flatpaks, services, firewall, CoreDesktop's files in /etc, GNOME settings...).
|
|
`10-fresh-install` saves the fingerprint of a fresh install, and
|
|
`30-upgrade-from-baseline` checks that installing an older commit from
|
|
`baselines` and then running `update` gives the same machine. If they differ,
|
|
an install.sh change is probably missing its migration.
|
|
|
|
## Files
|
|
|
|
| File | What it is |
|
|
|------|------------|
|
|
| `scenarios/NN-name.sh` | One scenario each (see below) |
|
|
| `lib/scenario.sh` | Helpers and assertions the scenarios use |
|
|
| `fingerprint` | Runs in the VM; prints the machine's fingerprint |
|
|
| `health` | Runs in the VM; checks the machine works, exits 1 if not |
|
|
| `baselines` | Older commits that `30-upgrade-from-baseline` installs first |
|
|
|
|
## Writing a scenario
|
|
|
|
A scenario is a bash file that sets a few variables and defines `scenario`:
|
|
|
|
```bash
|
|
DESCRIPTION="update runs a new migration once"
|
|
IMAGE=installed # base or installed
|
|
TIER=pr # pr (quick, uses the installed image) or nightly
|
|
|
|
scenario() {
|
|
upstream_commit "Add a migration" 'echo "touch ~/migrated" > migrations/9999999999.sh'
|
|
cd_update
|
|
assert_status 0
|
|
assert_vm "the migration ran" 'test -f ~/migrated'
|
|
}
|
|
```
|
|
|
|
`VARIANTS=(a b c)` runs the scenario once per value, each in its own VM, with
|
|
`$VARIANT` set. `scenario` runs with `set -e`, so any failing command or
|
|
assertion fails it. `skip "reason"` skips it.
|
|
|
|
Helpers, from `lib/scenario.sh`:
|
|
|
|
| Helper | |
|
|
|--------|---|
|
|
| `cd_install` | Run install.sh as `wget ... \| bash` would, from the fake upstream's main |
|
|
| `cd_update [VAR=value...]` | Run `update` |
|
|
| `run_in_vm NAME SCRIPT` | Run SCRIPT in the VM; sets `$OUTPUT` and `$STATUS`, logs to NAME.log |
|
|
| `upstream_set REF` | Point the fake upstream's main at REF, a commit in this repo |
|
|
| `upstream_commit MESSAGE SCRIPT` | Run SCRIPT in a clone of the upstream (in the VM), commit, push |
|
|
| `fingerprint FILE` / `health` | Save the fingerprint / run the health checks |
|
|
| `assert_status N`, `assert_output TEXT`, `refute_output TEXT` | Check the last `run_in_vm` |
|
|
| `assert_vm DESCRIPTION SCRIPT` | SCRIPT must succeed in the VM |
|
|
| `assert_same_fingerprint EXPECTED ACTUAL` | The two must match; saves the diff |
|
|
|
|
## The contract with the driver
|
|
|
|
`coredesktop-dev` boots the VM, defines these, sources `lib/scenario.sh`, calls
|
|
`smoketest_prepare`, then the scenario's `scenario` function:
|
|
|
|
| Name | |
|
|
|------|---|
|
|
| `vm_exec SCRIPT` | Run bash SCRIPT in the VM as `tester`, inside the GNOME session's environment, stdin from /dev/null. Returns its exit status. |
|
|
| `vm_put LOCAL REMOTE` | Copy a file into the VM |
|
|
| `vm_reboot` | Reboot and wait until `tester` is logged in to GNOME again |
|
|
| `VM_GIT_SSH`, `VM_GIT_URL` | `GIT_SSH_COMMAND` and the `ssh://` prefix to push to a path in the VM |
|
|
| `SMOKETEST_REF` | The commit under test |
|
|
| `SMOKETEST_RESULTS` | This scenario's folder for logs and fingerprints |
|
|
| `SMOKETEST_SHARED` | A folder shared by all scenarios in the run |
|
|
| `IMAGE_REF` | For the installed image, the commit it was installed at |
|
|
|
|
Anything that boots a VM can drive these scenarios, e.g. a CI runner with KVM.
|
|
|
|
## Not covered
|
|
|
|
NVIDIA drivers, firmware updates (`updates.d/90-firmware.sh`), and anything that
|
|
needs a real Core, a Tailscale login or a Claude login (`configure`). Check
|
|
those by hand.
|