# Smoketest

End-to-end tests of `install.sh` and `update` on a real Arch + GNOME virtual
machine. The bats tests in `tests/` check the logic with fake commands; these
check what fakes can't: pacman, yay, Flatpak, systemd, ufw, Snapper on btrfs,
GNOME settings, and a reboot.

This folder holds what to test: the scenarios, and the scripts that describe a
CoreDesktop machine. The VM itself (QEMU, images, the package cache) comes from
the `coredesktop-dev` sibling project, which only contributors install:

```
sibling add <coredesktop-dev url>
coredesktop-dev smoketest list             # scenarios and their tier
coredesktop-dev smoketest run              # the quick (pr) scenarios
coredesktop-dev smoketest run --tier all   # everything, including a fresh install
coredesktop-dev smoketest run 40-failing-migration
```

It tests the working copy's `HEAD` commit, so commit first.

## How it works

Each scenario runs on a throwaway copy of one of two VM images:

- **base**: Arch with GNOME, as archinstall's desktop/GNOME profile leaves it,
  with a user `tester` who logs in to GNOME automatically.
- **installed**: base after `install.sh`, at an older commit of CoreDesktop.
  Before the scenario starts, `update` brings it to the commit under test.

Inside the VM, a bare repo (`/srv/smoketest/coredesktop.git`) stands in for
Gitea. git's `insteadOf` sends CoreDesktop's real URLs there, so `install.sh`
and `update` run unchanged. The harness pushes the commit under test to it, and
scenarios push further commits to simulate a new release: a new migration, a
broken hook, a change to `update` itself.

After a scenario, `fingerprint` describes the machine as text (packages,
Flatpaks, services, firewall, CoreDesktop's files in /etc, GNOME settings...).
`10-fresh-install` saves the fingerprint of a fresh install, and
`30-upgrade-from-baseline` checks that installing an older commit from
`baselines` and then running `update` gives the same machine. If they differ,
an install.sh change is probably missing its migration.

## Files

| File | What it is |
|------|------------|
| `scenarios/NN-name.sh` | One scenario each (see below) |
| `lib/scenario.sh` | Helpers and assertions the scenarios use |
| `fingerprint` | Runs in the VM; prints the machine's fingerprint |
| `health` | Runs in the VM; checks the machine works, exits 1 if not |
| `baselines` | Older commits that `30-upgrade-from-baseline` installs first |

## Writing a scenario

A scenario is a bash file that sets a few variables and defines `scenario`:

```bash
DESCRIPTION="update runs a new migration once"
IMAGE=installed      # base or installed
TIER=pr              # pr (quick, uses the installed image) or nightly

scenario() {
  upstream_commit "Add a migration" 'echo "touch ~/migrated" > migrations/9999999999.sh'
  cd_update
  assert_status 0
  assert_vm "the migration ran" 'test -f ~/migrated'
}
```

`VARIANTS=(a b c)` runs the scenario once per value, each in its own VM, with
`$VARIANT` set. `scenario` runs with `set -e`, so any failing command or
assertion fails it. `skip "reason"` skips it.

Helpers, from `lib/scenario.sh`:

| Helper | |
|--------|---|
| `cd_install` | Run install.sh as `wget ... \| bash` would, from the fake upstream's main |
| `cd_update [VAR=value...]` | Run `update` |
| `run_in_vm NAME SCRIPT` | Run SCRIPT in the VM; sets `$OUTPUT` and `$STATUS`, logs to NAME.log |
| `upstream_set REF` | Point the fake upstream's main at REF, a commit in this repo |
| `upstream_commit MESSAGE SCRIPT` | Run SCRIPT in a clone of the upstream (in the VM), commit, push |
| `fingerprint FILE` / `health` | Save the fingerprint / run the health checks |
| `assert_status N`, `assert_output TEXT`, `refute_output TEXT` | Check the last `run_in_vm` |
| `assert_vm DESCRIPTION SCRIPT` | SCRIPT must succeed in the VM |
| `assert_same_fingerprint EXPECTED ACTUAL` | The two must match; saves the diff |

## The contract with the driver

`coredesktop-dev` boots the VM, defines these, sources `lib/scenario.sh`, calls
`smoketest_prepare`, then the scenario's `scenario` function:

| Name | |
|------|---|
| `vm_exec SCRIPT` | Run bash SCRIPT in the VM as `tester`, inside the GNOME session's environment, stdin from /dev/null. Returns its exit status. |
| `vm_put LOCAL REMOTE` | Copy a file into the VM |
| `vm_reboot` | Reboot and wait until `tester` is logged in to GNOME again |
| `VM_GIT_SSH`, `VM_GIT_URL` | `GIT_SSH_COMMAND` and the `ssh://` prefix to push to a path in the VM |
| `SMOKETEST_REF` | The commit under test |
| `SMOKETEST_RESULTS` | This scenario's folder for logs and fingerprints |
| `SMOKETEST_SHARED` | A folder shared by all scenarios in the run |
| `IMAGE_REF` | For the installed image, the commit it was installed at |

Anything that boots a VM can drive these scenarios, e.g. a CI runner with KVM.

## Not covered

NVIDIA drivers, firmware updates (`updates.d/90-firmware.sh`), and anything that
needs a real Core, a Tailscale login or a Claude login (`configure`). Check
those by hand.