coredesktop/skills/coredesktop-dev-sibling.md

# Skill: Building coredesktop-dev, the CoreDesktop Development Sibling

## Overview

`coredesktop-dev` is a CoreDesktop sibling project for people who work on
CoreDesktop itself. Installing it with `sibling add <url>` gives a machine
the tools to change CoreDesktop safely: the linters and test runner, a
command that runs the checks the way CI should, a way to try a working copy
without touching the installed one, and a guard that keeps the installed copy
updatable.

Follow this document when creating the `coredesktop-dev` repository from
scratch. It covers what to build, why, and how to test it. It assumes
CoreDesktop is installed at `~/.local/share/coredesktop`. The rules for any
sibling are in that checkout's `skills/sibling-project-format.md`: read it
first. This document only adds what is specific to this project.

## Why a sibling and not part of CoreDesktop

- **Only contributors need it.** Every package in CoreDesktop's
  `packages.conf` lands on every user's machine. shellcheck, bats and the
  Gitea CLI don't belong there.
- **It changes on its own schedule.** Development tooling can change without
  a CoreDesktop release, and `update` still keeps it current on every
  developer's machine.

## Background: the two copies of CoreDesktop

Every problem this project solves comes from one fact. A developer's machine
has two copies of CoreDesktop:

| Copy | Path | What it is |
|------|------|------------|
| **Deployed** | `~/.local/share/coredesktop` | The machine's running CoreDesktop. A git clone that `update` pulls with `--ff-only`. `~/.local/bin` commands, `updates.d` hooks, `configure` items and sandbox configs all run from here. |
| **Working** | e.g. `~/Projects/coredesktop` | Where development happens: branches, uncommitted changes. Nothing runs from here unless you run it yourself. |

Lessons learned the hard way, which the tools below exist to enforce:

1. **Never edit the deployed copy.** Copying working files into it to "try
   them" leaves tracked files modified, and the next `update` fails to pull
   (`--ff-only` refuses). If it happens, `git -C ~/.local/share/coredesktop
   checkout -- .` restores it, but it's easy to forget.
2. **Scripts hard-code the deployed path.** Most CoreDesktop scripts set
   `COREDESKTOP="$HOME/.local/share/coredesktop"`. To run working-copy code
   as if it were deployed, point a throwaway `HOME` at the working copy with
   a symlink. Don't swap the real deployed copy.
3. **A throwaway `HOME` doesn't isolate the session.** `XDG_RUNTIME_DIR`
   (`/run/user/<uid>`) is shared with the live GNOME session. A test that ran
   `gnome-keyring-daemon --replace` with only `HOME` changed found the real
   keyring daemon through `/run/user/<uid>/keyring`, killed it, and left the
   user's keyring locked, which hung Bitwarden. Any test that starts session
   daemons needs its own `XDG_RUNTIME_DIR` (mode 700), no `DISPLAY` or
   `WAYLAND_DISPLAY`, and a `dbus-run-session`. Never use `--replace`.
4. **`sudo`, `yay`, `pacman` and `systemctl` ignore `HOME`.** A throwaway home
   protects your files, not your system. Code that changes the system has to
   be tested with fake commands (bats stubs) or on a machine or VM you don't
   mind changing.
5. **Editors can run in a Flatpak.** VSCodium from Flathub, for one, runs its
   terminal and any AI agent inside the Flatpak sandbox, where `pacman`,
   `systemctl`, `journalctl` and the host's `/tmp` don't exist. Commands
   there must go through `flatpak-spawn --host`. The tools should notice
   this and say so, rather than fail in confusing ways.
6. **`make lint` only lints tracked files.** CoreDesktop's Makefile builds its
   list from `git ls-files`, so a new script isn't linted until it's
   committed. The checks have to include untracked scripts.
7. **CI runs only shellcheck.** `.gitea/workflows/shellcheck.yml` runs
   `make lint`, not the bats tests, so `make check` has to run locally
   before pushing.

## What to build

### Repository layout

```
coredesktop-dev/
  install.sh            run once by `sibling add`
  update.sh             run on every `update`
  bin/coredesktop-dev   the command (symlinked into ~/.local/bin)
  hooks/pre-push        git hook for the working copy (opt-in)
  lib/                  one file per subcommand, plus common.sh and vm.sh
  vm/                   provision.sh and cloud-init user-data for the smoketest VM
  tests/                bats tests for this project
  Makefile              `make check`: shellcheck + bats for this repo
  .shellcheckrc         copy CoreDesktop's
  README.md             install with: sibling add <url>
```

### Packages

Installed by `update.sh` with CoreDesktop's `install_packages`, so packages
added later reach existing machines on their next `update`. All are in Arch's
`extra` repo:

| Package | Why |
|---------|-----|
| `shellcheck` | `make lint` |
| `bats` | `make test` (CoreDesktop's tests use plain bats, no helper libraries) |
| `tea` | Gitea's CLI, for pull requests on gitea.federated.computer |
| `qemu-base` | The smoketest's virtual machines |
| `qemu-hw-display-virtio-vga` | The VMs' `-vga virtio` (`qemu-base` leaves it out) |
| `libisoburn` | `xorriso`, to make the smoketest VM's cloud-init disk |

`make`, `git` and `jq` already come with CoreDesktop. Don't add `shfmt` until
CoreDesktop decides on a formatting style; it has no config for it today.

### Settings

`~/.config/coredesktop-dev/config`, a shell file `install.sh` writes and every
other script sources:

```bash
WORKING_COPY=/home/<user>/Projects/coredesktop
```

Write it outside the repo: siblings must never write tracked files.

### `install.sh`

1. Source `~/.local/share/coredesktop/utils.sh`. Stop with a clear message if
   it's missing (CoreDesktop isn't installed).
2. Ask where the working copy is, defaulting to `~/Projects/coredesktop`.
   - If that folder is a git clone of CoreDesktop, use it. Check that its
     `origin` points at a CoreDesktop repository.
   - If it doesn't exist, offer to clone it there from the deployed copy's
     `origin` URL (`git -C ~/.local/share/coredesktop remote get-url origin`).
   - Never use the deployed copy itself as the working copy. Refuse if the
     answer resolves to `~/.local/share/coredesktop`.
3. Save the answer in the settings file.
4. Run `update.sh`, which installs the packages and the command.
5. Offer to install the pre-push hook (see `hooks` below). Default no.
6. Run `coredesktop-dev doctor` and show the result.

It must be safe to re-run: re-running `sibling add` is how users repair an
install. If the settings file exists, show the current working copy as the
default answer.

### `update.sh`

No prompts. It prints what it does and exits non-zero only on real failure:

1. `install_packages shellcheck bats tea qemu-base qemu-hw-display-virtio-vga libisoburn`
2. Make `~/.local/bin/coredesktop-dev` a symlink to `bin/coredesktop-dev` in
   this repo, so each pull updates the command without copying.
3. If the pre-push hook is installed in the working copy, refresh it from
   `hooks/pre-push` (compare the files, copy only if different).

### The `coredesktop-dev` command

Subcommands, each with `--help`. With no subcommand, it prints usage.

#### `coredesktop-dev doctor`

Read-only. It reports each of these as ok or a problem, with the fix:

- shellcheck, bats and tea are installed.
- QEMU has the `virtio-vga` display (`qemu-system-x86_64 -device help`
  lists it). It's missing without `qemu-hw-display-virtio-vga`, and the
  smoketest's VMs then fail to start.
- The working copy from the settings file exists and is a CoreDesktop clone.
- **The deployed copy has no local changes** (`git status --porcelain
  --untracked-files=no` is empty). If it has some, list them and point to
  `coredesktop-dev deployed restore`.
- The deployed copy can fast-forward to its upstream, or is already there.
- Whether it is running inside a Flatpak (`/.flatpak-info` exists, or
  `$FLATPAK_ID` is set). If it is, say that host commands need
  `flatpak-spawn --host`. If the subcommand that was asked for needs the
  host (anything that runs `make check` with the host's tools), re-run
  itself on the host through `flatpak-spawn --host` instead of failing.
- Whether the pre-push hook is installed.

Exit 0 if everything is ok, 1 otherwise.

#### `coredesktop-dev check [--fast]`

Runs the checks on the working copy, from its root:

1. `make check` (shellcheck on tracked scripts, then bats).
2. **shellcheck on untracked scripts too**, with the same arguments the
   Makefile uses (`--severity=warning`, the repo's `.shellcheckrc`). The
   files are `git ls-files --others --exclude-standard` whose first line is a
   bash or sh shebang, plus untracked `*.sh` files. That closes the lint gap
   in lesson 6.
3. `--fast` runs only the shellcheck steps.

Exit non-zero if anything fails. Print a one-line summary at the end, e.g.
`shellcheck: ok (91 files, 6 untracked)  bats: ok (85 tests)`.

#### `coredesktop-dev try [command...]`

Runs working-copy code as if it were the deployed CoreDesktop, without
changing the deployed copy or the real home folder:

1. Make a temporary directory (`mktemp -d`) holding a `home/` and a
   `runtime/` folder (mode 700).
2. In the temporary home, symlink `.local/share/coredesktop` to the working
   copy and put `.local/bin` at the front of `PATH`. Link
   `.local/bin/{update,refresh,sibling,version,configure,audit}` to the
   working copy's scripts of the same names, as CoreDesktop's installer
   installs them.
3. Set `HOME` and the `XDG_*_HOME` variables to the temporary home, and
   `XDG_RUNTIME_DIR` to `runtime/`. Unset `DISPLAY`, `WAYLAND_DISPLAY`,
   `DBUS_SESSION_BUS_ADDRESS` and `SSH_AUTH_SOCK`.
4. Run the command, or an interactive `bash` if none was given, under
   `dbus-run-session`, so anything that needs a session bus gets a private
   one.
5. Before starting, print a warning: `sudo`, `yay`, `pacman` and `systemctl`
   still change this machine (lesson 4).
6. Remove the temporary directory afterwards, unless `--keep` was given;
   then print its path.

Example: `coredesktop-dev try configure --list` shows what the working copy's
`configure` would list.

#### `coredesktop-dev deployed [status|restore]`

- `status` (the default): the deployed copy's branch, commit, how far behind
  upstream it is, and any local changes.
- `restore`: show `git diff --stat` of the local changes, ask for
  confirmation, then `git checkout -- .` in the deployed copy. Untracked
  files are listed but left alone. Never `git clean` or `git reset --hard`
  there.

#### `coredesktop-dev hooks [install|remove]`

Installs `hooks/pre-push` into the working copy's `.git/hooks/pre-push`. The
`.git` folder isn't tracked, so this is allowed. The hook runs
`coredesktop-dev check` and blocks the push if it fails. `git push
--no-verify` skips it, and the hook's failure message says so. Refuse to
overwrite a pre-push hook that this project didn't write: write a marker
comment into the hook and check for it.

#### `coredesktop-dev smoketest`

Runs CoreDesktop's end-to-end scenarios on QEMU virtual machines. The
scenarios, the fingerprint and health scripts, and the helpers they use live
in CoreDesktop's `smoketest/` folder, because they change with the code they
test; read its README for the contract between them and this driver. This
project only provides the VMs: `vm_exec`, `vm_put`, `vm_reboot`, `VM_GIT_SSH`
and `VM_GIT_URL`, then calls `smoketest_prepare` and the scenario.

- `list`, `run [--tier pr|all] [--ref REF] [--keep] [scenario...]`,
  `images [base|installed [REF]]`, `shell [base|installed]`, `clean [--all]`.
- Images, the shared pacman cache, the SSH key and run logs live in
  `~/.cache/coredesktop-dev/smoketest`, never in either copy of CoreDesktop.
- **base** is Arch's cloud image plus archinstall's GNOME desktop profile
  (`vm/provision.sh`), with automatic login and passwordless sudo for `tester`.
  **installed** is base after `install.sh` at the commit the branch started
  from. Each scenario runs on a throwaway qcow2 overlay of one of them.
- Commands in the VM run in an interactive bash with the GNOME session's
  environment, as they would in GNOME Terminal.

### What not to do

- Don't copy hooks into CoreDesktop's `updates.d/`: use `update.sh`, as the
  sibling format says. Using both updates the project twice.
- Don't change CoreDesktop's files from this project, including the working
  copy's tracked files. Suggest such changes to CoreDesktop instead (below).
- Don't start, stop or `--replace` real session services (keyring, portals,
  GNOME Shell) in any command or test.

## Changes to suggest to CoreDesktop itself

These belong in CoreDesktop's repository, not in this project. Once they're
made, `coredesktop-dev check` still works, and step 2 above simply finds
nothing extra to lint.

1. **Lint untracked scripts in the Makefile.** Build `SCRIPTS` from
   `git ls-files --cached --others --exclude-standard`.
2. **Run the bats tests in CI.** Add a `make test` job to
   `.gitea/workflows/`, installing `bats` alongside `shellcheck`.

## Testing this project

Use the same habits as CoreDesktop's own tests:

- **bats with stubs.** Copy CoreDesktop's `tests/test_helper.bash` pattern: a
  fresh `HOME` and a `bin/` of fake commands at the front of `PATH` for each
  test. Fake `pacman`, `yay`, `sudo`, `flatpak-spawn` and `make`, so no test
  changes the machine.
- **Prove the tests catch bugs.** After the tests pass, break the code on
  purpose once per behavior (e.g. make `deployed restore` skip the
  confirmation, or make `check` skip untracked files) and confirm a test
  fails each time.
- **Test the sibling flow in a throwaway home**, as described in
  `sibling-project-format.md`. `update.sh` calls `install_packages`, which
  uses `yay`, so stub it there too, or run it only where installing
  packages is fine.

Cases worth a test each:

- `install.sh` refuses the deployed copy as the working copy, and is safe to
  re-run.
- `update.sh` doesn't prompt and doesn't re-copy an unchanged hook.
- `doctor` reports a dirty deployed copy, QEMU without `virtio-vga`, and
  Flatpak use.
- `check` lints an untracked script and fails on a shellcheck warning in it.
- `try` gives the command a private `HOME` and `XDG_RUNTIME_DIR`, no
  `DISPLAY`, and the working copy at `~/.local/share/coredesktop`, and
  cleans up afterwards.
- `deployed restore` asks first, restores tracked files, and leaves
  untracked ones alone.
- `hooks install` won't overwrite another hook.

## Checklist

- [ ] The repo's README says to install with `sibling add <url>`
- [ ] `install.sh` asks for the working copy, refuses the deployed copy,
      saves `~/.config/coredesktop-dev/config`, and is safe to re-run
- [ ] `update.sh` installs shellcheck, bats, tea, QEMU and xorriso, keeps the
      `~/.local/bin/coredesktop-dev` symlink, and never prompts
- [ ] `doctor`, `check`, `try`, `deployed`, `hooks` and `smoketest` work and
      have `--help`
- [ ] `check` lints untracked scripts in the working copy
- [ ] `try` isolates `HOME`, `XDG_*` and `XDG_RUNTIME_DIR`, unsets the
      display and session variables, uses `dbus-run-session`, and warns that
      sudo, yay, pacman and systemctl still reach the real system
- [ ] Nothing writes to the deployed copy except `deployed restore`, after
      confirmation
- [ ] The project's own `make check` passes: shellcheck plus bats tests
      with stubs, and the tests catch deliberately broken code
- [ ] The two suggested CoreDesktop changes are raised with CoreDesktop