This commit is contained in:
@@ -0,0 +1,111 @@
|
||||
# Smoketest
|
||||
|
||||
End-to-end tests of `install.sh` and `update` on a real Arch + GNOME virtual
|
||||
machine. The bats tests in `tests/` check the logic with fake commands; these
|
||||
check what fakes can't: pacman, yay, Flatpak, systemd, ufw, Snapper on btrfs,
|
||||
GNOME settings, and a reboot.
|
||||
|
||||
This folder holds what to test: the scenarios, and the scripts that describe a
|
||||
CoreDesktop machine. The VM itself (QEMU, images, the package cache) comes from
|
||||
the `coredesktop-dev` sibling project, which only contributors install:
|
||||
|
||||
```
|
||||
sibling add <coredesktop-dev url>
|
||||
coredesktop-dev smoketest list # scenarios and their tier
|
||||
coredesktop-dev smoketest run # the quick (pr) scenarios
|
||||
coredesktop-dev smoketest run --tier all # everything, including a fresh install
|
||||
coredesktop-dev smoketest run 40-failing-migration
|
||||
```
|
||||
|
||||
It tests the working copy's `HEAD` commit, so commit first.
|
||||
|
||||
## How it works
|
||||
|
||||
Each scenario runs on a throwaway copy of one of two VM images:
|
||||
|
||||
- **base**: Arch with GNOME, as archinstall's desktop/GNOME profile leaves it,
|
||||
with a user `tester` who logs in to GNOME automatically.
|
||||
- **installed**: base after `install.sh`, at an older commit of CoreDesktop.
|
||||
Before the scenario starts, `update` brings it to the commit under test.
|
||||
|
||||
Inside the VM, a bare repo (`/srv/smoketest/coredesktop.git`) stands in for
|
||||
Gitea. git's `insteadOf` sends CoreDesktop's real URLs there, so `install.sh`
|
||||
and `update` run unchanged. The harness pushes the commit under test to it, and
|
||||
scenarios push further commits to simulate a new release: a new migration, a
|
||||
broken hook, a change to `update` itself.
|
||||
|
||||
After a scenario, `fingerprint` describes the machine as text (packages,
|
||||
Flatpaks, services, firewall, CoreDesktop's files in /etc, GNOME settings...).
|
||||
`10-fresh-install` saves the fingerprint of a fresh install, and
|
||||
`30-upgrade-from-baseline` checks that installing an older commit from
|
||||
`baselines` and then running `update` gives the same machine. If they differ,
|
||||
an install.sh change is probably missing its migration.
|
||||
|
||||
## Files
|
||||
|
||||
| File | What it is |
|
||||
|------|------------|
|
||||
| `scenarios/NN-name.sh` | One scenario each (see below) |
|
||||
| `lib/scenario.sh` | Helpers and assertions the scenarios use |
|
||||
| `fingerprint` | Runs in the VM; prints the machine's fingerprint |
|
||||
| `health` | Runs in the VM; checks the machine works, exits 1 if not |
|
||||
| `baselines` | Older commits that `30-upgrade-from-baseline` installs first |
|
||||
|
||||
## Writing a scenario
|
||||
|
||||
A scenario is a bash file that sets a few variables and defines `scenario`:
|
||||
|
||||
```bash
|
||||
DESCRIPTION="update runs a new migration once"
|
||||
IMAGE=installed # base or installed
|
||||
TIER=pr # pr (quick, uses the installed image) or nightly
|
||||
|
||||
scenario() {
|
||||
upstream_commit "Add a migration" 'echo "touch ~/migrated" > migrations/9999999999.sh'
|
||||
cd_update
|
||||
assert_status 0
|
||||
assert_vm "the migration ran" 'test -f ~/migrated'
|
||||
}
|
||||
```
|
||||
|
||||
`VARIANTS=(a b c)` runs the scenario once per value, each in its own VM, with
|
||||
`$VARIANT` set. `scenario` runs with `set -e`, so any failing command or
|
||||
assertion fails it. `skip "reason"` skips it.
|
||||
|
||||
Helpers, from `lib/scenario.sh`:
|
||||
|
||||
| Helper | |
|
||||
|--------|---|
|
||||
| `cd_install` | Run install.sh as `wget ... \| bash` would, from the fake upstream's main |
|
||||
| `cd_update [VAR=value...]` | Run `update` |
|
||||
| `run_in_vm NAME SCRIPT` | Run SCRIPT in the VM; sets `$OUTPUT` and `$STATUS`, logs to NAME.log |
|
||||
| `upstream_set REF` | Point the fake upstream's main at REF, a commit in this repo |
|
||||
| `upstream_commit MESSAGE SCRIPT` | Run SCRIPT in a clone of the upstream (in the VM), commit, push |
|
||||
| `fingerprint FILE` / `health` | Save the fingerprint / run the health checks |
|
||||
| `assert_status N`, `assert_output TEXT`, `refute_output TEXT` | Check the last `run_in_vm` |
|
||||
| `assert_vm DESCRIPTION SCRIPT` | SCRIPT must succeed in the VM |
|
||||
| `assert_same_fingerprint EXPECTED ACTUAL` | The two must match; saves the diff |
|
||||
|
||||
## The contract with the driver
|
||||
|
||||
`coredesktop-dev` boots the VM, defines these, sources `lib/scenario.sh`, calls
|
||||
`smoketest_prepare`, then the scenario's `scenario` function:
|
||||
|
||||
| Name | |
|
||||
|------|---|
|
||||
| `vm_exec SCRIPT` | Run bash SCRIPT in the VM as `tester`, inside the GNOME session's environment, stdin from /dev/null. Returns its exit status. |
|
||||
| `vm_put LOCAL REMOTE` | Copy a file into the VM |
|
||||
| `vm_reboot` | Reboot and wait until `tester` is logged in to GNOME again |
|
||||
| `VM_GIT_SSH`, `VM_GIT_URL` | `GIT_SSH_COMMAND` and the `ssh://` prefix to push to a path in the VM |
|
||||
| `SMOKETEST_REF` | The commit under test |
|
||||
| `SMOKETEST_RESULTS` | This scenario's folder for logs and fingerprints |
|
||||
| `SMOKETEST_SHARED` | A folder shared by all scenarios in the run |
|
||||
| `IMAGE_REF` | For the installed image, the commit it was installed at |
|
||||
|
||||
Anything that boots a VM can drive these scenarios, e.g. a CI runner with KVM.
|
||||
|
||||
## Not covered
|
||||
|
||||
NVIDIA drivers, firmware updates (`updates.d/90-firmware.sh`), and anything that
|
||||
needs a real Core, a Tailscale login or a Claude login (`configure`). Check
|
||||
those by hand.
|
||||
Reference in New Issue
Block a user