From 6770150b111d8fe064113371227aa6100dd3b155 Mon Sep 17 00:00:00 2001 From: David Young Date: Sat, 31 Oct 2026 08:10:05 -0600 Subject: [PATCH] fix the fail2ban issues and other things --- .gitignore | 1 + README.md | 36 ++- configure.d/10-core.sh | 68 +++++- configure.d/lib/goa-account | 61 +++++ install-flatpaks.sh | 7 +- install.sh | 9 +- migrations/1793454799.sh | 29 +++ packages.conf | 3 +- shell/bashrc | 6 + skills/sibling-project-updates.md | 2 +- system/email-watch | 225 ++++++++++++++++++ system/email-watch.sh | 19 ++ system/run-in-terminal | 27 +++ .../systemd/coredesktop-email-watch.service | 14 ++ system/update-check | 20 +- tests/utils.bats | 39 +++ updates.d/17-email-watch.sh | 7 + updates.d/40-mise.sh | 33 ++- utils.sh | 25 +- 19 files changed, 580 insertions(+), 51 deletions(-) create mode 100644 migrations/1793454799.sh create mode 100755 system/email-watch create mode 100644 system/email-watch.sh create mode 100755 system/run-in-terminal create mode 100644 system/systemd/coredesktop-email-watch.service create mode 100644 updates.d/17-email-watch.sh diff --git a/.gitignore b/.gitignore index 484db71..8ccfe1e 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ gnome/.gnome-settings.md5 +__pycache__/ diff --git a/README.md b/README.md index 8a4d5b7..15abb33 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,7 @@ You will end up with a Gnome system with a three-tile set up (again, great for w The scripts are installed in ~/.local/share/coredesktop/ -The scripts install lots of useful software. There are (4) main ways software is installed: pacman, AUR, Flathub, and mise. +The scripts install lots of useful software. There are (3) main ways software is installed: pacman, AUR and Flathub. mise installs other languages if you want them (see Mise below). ### Pacman @@ -90,15 +90,27 @@ The file "install-flatpaks.sh" file for Flatpaks. Flatpaks installed include: Chrome, Chromium, Firefox, Brave, Zen, Vivaldi (browsers), Signal, Telegram, Discord, Revolt, FluffyChat (messaging), Thunderbird and Evolution (email), VS Code and VSCodium (VS Code without the Microsoft telemetry), Obsidian and Logseq (notes), Apostrophe (markdown writer), Kdenlive (video editing), OBS (video recording), Audacity (audio editing), Inkscape (vector graphics), Scribus (page layout), Spotify, Jellyfin (media client), Zoom, Remmina and RustDesk (remote desktop), Postman, DBeaver and Beekeeper Studio (database clients), Alpaca and LM Studio (local AI), LocalSend (send files to other devices), Boxes (virtual machines), Impression (write ISOs to USB sticks), Trayscale (Tailscale), Warehouse (manage Flatpaks), Extension Manager (GNOME extensions), RetroArch and more. See [install-flatpaks.sh](install-flatpaks.sh) for the full list. -If you want to install more software from Flathub, simple visit https://flathub.org. +If you want to install more software from Flathub, simply visit https://flathub.org. + +### Don't want something CoreDesktop installs? + +Uninstall it, then add its name to ~/.config/coredesktop/skip-packages, one per line: the pacman package name (e.g. `libreoffice-fresh`) or the Flatpak's ID (e.g. `us.zoom.Zoom`, shown by `flatpak list`). Otherwise `update` installs it again. For example: + +``` +sudo flatpak uninstall us.zoom.Zoom +echo us.zoom.Zoom >> ~/.config/coredesktop/skip-packages +``` ### Mise -Ruby and Rails are installed using Mise from the file "install-mise-ruby-rails.sh". +Go, Rust, Node.js, Python, Bun and Zig come from pacman, and Arch keeps them current. [mise](https://mise.jdx.dev) is installed for everything else: + +- **Other languages.** `configure languages` installs Ruby on Rails, or any tool mise knows (`mise registry` lists them), such as `java@21` or `deno@latest`. `update` keeps them up to date within the version you chose: `latest` moves to each new release, `21` stays on 21. To remove one, run `mise unuse -g `. +- **The version a project needs.** Put a mise.toml in the project, e.g. `mise use node@20` in its folder, and inside that folder `node` is version 20, while everywhere else it's pacman's. ### Development Tools -Lots of development tools including herdr, neovim, tmux, go, hugo, python, rust, rails, ruby, docker and docker compose, etc. Three terminals: console, kitty, ghostty. +Lots of development tools including herdr, neovim, tmux, go, hugo, python, rust, docker and docker compose, etc. Three terminals: console, kitty, ghostty. ### Printing and Networking @@ -164,7 +176,7 @@ Besides installing software, CoreDesktop changes these settings. They're listed - **Services turned on**: Avahi, Bluetooth, CUPS (printing), fwupd, GDM, NetworkManager, sshd, Docker, Tailscale, ufw and systemd-oomd. Your user is added to the docker group. - **Snapper** (btrfs only): a config for / in /etc/snapper/configs/root. - **NVIDIA** (only if it installs the driver): /etc/mkinitcpio.conf.d/nvidia.conf and NVIDIA's suspend services. -- **Your home folder**: lines in ~/.bashrc (CoreDesktop's shell setup, ~/.local/bin on PATH, the Bitwarden SSH agent), the commands in ~/.local/bin (update, refresh, sibling, version, configure, audit), GNOME extensions and their settings, Claude Code's ~/.claude/settings.json if you had none, and a user timer that checks for updates. +- **Your home folder**: lines in ~/.bashrc (CoreDesktop's shell setup, ~/.local/bin on PATH, the Bitwarden SSH agent), the commands in ~/.local/bin (update, refresh, sibling, version, configure, audit), GNOME extensions and their settings, Claude Code's ~/.claude/settings.json if you had none, a user timer that checks for updates, and a user service that pauses your Core's email if its password stops working (see Configure). - **configure** adds more, only for what you set up: your Nextcloud files mount in /etc/fstab, GNOME Online Accounts, passwords in your GNOME keyring, and the backup timer. ## Keep your System Up-to-date @@ -181,7 +193,7 @@ The script pulls the latest CoreDesktop, then runs each step in ~/.local/share/c - stops if there's less than 10 GiB free on / (run `UPDATE_FORCE=1 update` to skip the check) - snapshots / first, if Snapper is set up (see Snapshots above) -- updates the Arch keyring, then Arch and GNOME, then AUR packages, Flatpaks, Ruby, the pipx apps CoreDesktop installs, GNOME extensions and firmware +- updates the Arch keyring, then Arch and GNOME, then AUR packages, Flatpaks, the languages you installed with mise, the pipx apps CoreDesktop installs, GNOME extensions and firmware - warns you not to reboot if the initramfs failed to rebuild - runs any one-time migrations, the way existing machines pick up new CoreDesktop setup - lists packages that were installed only because something else needed them, and are no longer needed by anything, and asks before removing them. Packages CoreDesktop installs are never on this list. To keep one you use directly off the list, run `sudo pacman -D --asexplicit ` @@ -247,10 +259,20 @@ It covers: - **Your Federated Computer Core.** Give it your Superpanel's address (e.g. https://superpanel.example.com), your Core email, name and password, and it adds Superpanel to your applications, adds your Nextcloud (calendar, contacts and files) and email to GNOME's Online Accounts, and mounts your Nextcloud files at /mnt/example.com ("example.com" in the Files sidebar), and points Element at your Core's Matrix server. For Nextcloud and the files it creates a Nextcloud app password, which you can revoke in Nextcloud under Settings > Security. You can add more Cores: run `configure core` again and choose "Add a Core". Each gets its own Superpanel, Nextcloud, email and files; Element, Tailscale and backups use your main Core (the first one, unless you choose another). "Remove a Core" takes it off this computer and revokes its app password, and leaves everything on the Core itself as it is. If you change your Core password (e.g. in Superpanel), run `configure core` and choose "Your Core password changed": email keeps your Core password, so it saves the new one there, and it signs in to Nextcloud with it, which turns the app passwords for Nextcloud, files and backups back on (Nextcloud turns them off when your password changes), then checks each of them. To check that a Core still works from this computer, run `configure core` and choose "Test a Core": it signs in to Superpanel, Nextcloud, email, your files, Element's Matrix server and backups the way this computer does, says which work, and offers to fix those that don't (asking for your Core password if a fix needs it, e.g. to make a new app password for one that was revoked), then tests again. - **Your Core's Tailscale network.** Joins this computer to it through headscale.example.com, so you can reach your other machines on it. Set up your Core first. You sign in through a link it shows you; if this computer is on another Tailscale network, it moves it to your Core's. To leave your Core's network, run `configure tailscale` again. - **Daily backups of your home folder** to your Nextcloud files, in a .backup-{machine name} folder, with restic. Set up your Core first. The backup is encrypted with a passphrase that `configure` shows you once: save it somewhere other than this computer, such as Bitwarden. It keeps 7 daily, 4 weekly and 6 monthly backups, and leaves out caches, things that can be reinstalled, and WinBoat's Windows VM (~/winboat). To leave out more, list paths in ~/.config/coredesktop/backup-exclude, one per line. restic reaches Nextcloud directly through rclone with its own Nextcloud app password, so the backup doesn't depend on the /mnt mount. See "Restoring from a backup" below. To turn backups off, run `configure backup` again; it also offers to delete the backups from your Nextcloud files (they go to Nextcloud's trash bin). +- **Tally for Plausible**, if your main Core runs Plausible Analytics: it's pointed at plausible.example.com. - Signing in to Claude Code, opencode and Hermes Agent and choosing their model providers. +- **Languages** to install with mise, such as Ruby on Rails (see Mise above). To set up one thing, name it, e.g. `configure hermes`; `configure --list` shows the names. +### When your Core password changes + +Your email in Online Accounts keeps your Core password, so after you change it (e.g. in Superpanel), the mail server starts refusing the old one. GNOME tries it again each time your network changes, and Evolution each time it fetches mail, and after a few wrong passwords the mail server blocks this computer for a while. + +So CoreDesktop watches for it, without ever trying the password on the mail server. When your password changes, Nextcloud stops accepting the app passwords this computer uses, and Online Accounts notices. CoreDesktop then checks the password your email has with Nextcloud, once. If Nextcloud says it's wrong, CoreDesktop pauses your email: this computer stops trying the old password, and you get a notification. Click it, choose "Your Core password changed" and enter your new password; that ends the pause. Or remove the Core, or just the email account in Settings > Online Accounts. `configure core` > "Test a Core" shows a paused email too, and checks email's password with Nextcloud before trying it on the mail server. + +This needs your Core's Nextcloud in Online Accounts, which `configure core` sets up along with email. + ### Restoring from a backup To see the backups and get files back: @@ -312,7 +334,7 @@ CoreDesktop doesn't have an uninstaller, but you can undo it by hand. If you set Then remove CoreDesktop's own files: ``` -systemctl --user disable --now coredesktop-update-check.timer coredesktop-backup.timer +systemctl --user disable --now coredesktop-update-check.timer coredesktop-backup.timer coredesktop-email-watch.service rm -f ~/.config/systemd/user/coredesktop-* rm -f ~/.local/bin/{update,refresh,sibling,version,configure,audit} rm -rf ~/.local/share/coredesktop ~/.local/share/siblings ~/.local/state/coredesktop ~/.config/coredesktop diff --git a/configure.d/10-core.sh b/configure.d/10-core.sh index 1a3b442..457ef68 100755 --- a/configure.d/10-core.sh +++ b/configure.d/10-core.sh @@ -14,6 +14,7 @@ CORES_DIR="$HOME/.config/coredesktop/cores" TAILSCALE_CONF="$HOME/.config/coredesktop/tailscale.conf" BACKUP_CONF="$HOME/.config/coredesktop/backup.conf" GOA_ACCOUNT="$COREDESKTOP/configure.d/lib/goa-account" +EMAIL_WATCH="$COREDESKTOP/system/email-watch" source "$COREDESKTOP/configure.d/lib/nextcloud.sh" [ -f "$CORE_CONF" ] && source "$CORE_CONF" MAIN_DOMAIN="$CORE_DOMAIN" @@ -40,6 +41,7 @@ core_names() { SUPERPANEL_DESKTOP="$HOME/.local/share/applications/core-$CORE_DOMAIN-superpanel.desktop" MATRIX="https://matrix.$CORE_DOMAIN" ELEMENT_CONFIG="$HOME/.config/Element/config.json" + PLAUSIBLE="plausible.$CORE_DOMAIN" } superpanel_done() { @@ -70,6 +72,26 @@ element_done() { "$ELEMENT_CONFIG" &> /dev/null } +# Tally for Plausible, a Flatpak, shows one Plausible Analytics. It follows the +# main Core, if that runs Plausible. +TALLY=com.cassidyjames.plausible + +tally_installed() { + flatpak info "$TALLY" &> /dev/null +} + +tally_setting() { + flatpak run --command=gsettings "$TALLY" "$@" 2> /dev/null +} + +tally_done() { + [ "$(tally_setting get "$TALLY" domain)" = "'$PLAUSIBLE'" ] +} + +plausible_on_core() { + curl -sf -o /dev/null --max-time 15 "https://$PLAUSIBLE/api/health" +} + # Element only follows the main Core core_done() { superpanel_done && nextcloud_done && email_done && files_done \ @@ -293,6 +315,13 @@ setup_element() { echo "Element will use $MATRIX the next time it starts." } +# Tally opens the page it showed last, so forget that one too +setup_tally() { + tally_setting set "$TALLY" domain "$PLAUSIBLE" || return 1 + tally_setting set "$TALLY" current-url "" || return 1 + echo "Tally for Plausible will show https://$PLAUSIBLE the next time it starts. Sign in there." +} + # Run a step unless it's done; a failed step doesn't stop the ones after it step() { local name="$1" title="$2" @@ -359,7 +388,12 @@ setup_core() { step files "Nextcloud files at $MOUNT" [ "$CORE_DOMAIN" = "$MAIN_DOMAIN" ] && element_installed \ && step element "Element with your Core's Matrix server" + [ "$CORE_DOMAIN" = "$MAIN_DOMAIN" ] && tally_installed && plausible_on_core \ + && step tally "Tally for Plausible with your Core's Plausible" unset PASSWORD APP_PASSWORD + + # Pauses email if the mail server stops taking its password (system/email-watch) + systemctl --user start coredesktop-email-watch.service 2> /dev/null return $STATUS } @@ -399,10 +433,12 @@ change_password() { echo if email_done; then + local was_paused="" + "$GOA_ACCOUNT" paused imap_smtp "$CORE_EMAIL" 2> /dev/null && was_paused=", and isn't paused any more" jq -n --arg password "$PASSWORD" '{"imap-password": $password, "smtp-password": $password}' \ | "$GOA_ACCOUNT" update imap_smtp "$CORE_EMAIL" 2> /dev/null case $? in - 0) echo " Email: has your new password" ;; + 0) echo " Email: has your new password$was_paused" ;; 3) echo -e " \e[31mEmail: saved your new password, but the mail server didn't accept it.\e[0m" STATUS=1 ;; *) echo -e " \e[31mEmail: couldn't save your new password.\e[0m"; STATUS=1 ;; @@ -462,6 +498,10 @@ main_changed() { if element_installed && ! element_done && confirm_yes "Point Element at $MATRIX?"; then setup_element fi + if tally_installed && ! tally_done && plausible_on_core \ + && confirm_yes "Point Tally for Plausible at https://$PLAUSIBLE?"; then + setup_tally + fi [ -f "$TAILSCALE_CONF" ] && echo "To move this computer to its Tailscale network: configure tailscale" [ -f "$BACKUP_CONF" ] && echo "To back up to it instead: configure backup" return 0 @@ -547,13 +587,26 @@ failed() { # Test an Online Accounts account, and that none of what CoreDesktop set it up # for is turned off in Settings: # <provider> <identity> <setup> <fix> <enable> +# One Online Accounts already found failing isn't signed in to again: each +# wrong password counts towards the mail server blocking this computer. test_account() { local error off works=1 if ! "$GOA_ACCOUNT" exists "$2" "$3" 2> /dev/null; then failed "$1" "not set up" "$4" return fi - if ! error="$("$GOA_ACCOUNT" check "$2" "$3" 2>&1)"; then + if "$GOA_ACCOUNT" paused "$2" "$3" 2> /dev/null; then + failed "$1" "paused, as the server stopped taking its password" "$5" \ + "So this computer doesn't keep trying it. Did your Core password change?" + works=0 + elif "$GOA_ACCOUNT" attention "$2" "$3" 2> /dev/null; then + failed "$1" "the server doesn't take its password" "$5" "Did your Core password change?" + works=0 + # Check email's password with Nextcloud first, not the mail server + elif [ "$2" = imap_smtp ] && { "$EMAIL_WATCH" --check-password "$3"; [ $? -eq 2 ]; }; then + failed "$1" "has an old password: Nextcloud doesn't take it" "$5" "Did your Core password change?" + works=0 + elif ! error="$("$GOA_ACCOUNT" check "$2" "$3" 2>&1)"; then failed "$1" "can't sign in" "$5" "$error" works=0 fi @@ -592,6 +645,16 @@ test_element() { fi } +test_tally() { + local title="Tally for Plausible with your Core's Plausible" + plausible_on_core || return 0 + if tally_done; then + passed "$title" + else + failed "$title" "Tally shows another Plausible" setup_tally + fi +} + test_backups() { local webdav webdav="$(source "$BACKUP_CONF" && echo "$BACKUP_WEBDAV")" @@ -631,6 +694,7 @@ test_core() { test_account "Email in Online Accounts" imap_smtp "$CORE_EMAIL" setup_email fix_email enable_email test_files [ "$CORE_DOMAIN" = "$MAIN_DOMAIN" ] && element_installed && test_element + [ "$CORE_DOMAIN" = "$MAIN_DOMAIN" ] && tally_installed && test_tally backup_on "$CORE_DOMAIN" && test_backups return 0 } diff --git a/configure.d/lib/goa-account b/configure.d/lib/goa-account index f784660..27c50f0 100755 --- a/configure.d/lib/goa-account +++ b/configure.d/lib/goa-account @@ -15,6 +15,9 @@ Usage: goa-account check <provider> <presentation-identity> goa-account disabled <provider> <presentation-identity> goa-account enable <provider> <presentation-identity> + goa-account attention <provider> <presentation-identity> + goa-account pause <provider> <presentation-identity> + goa-account paused <provider> <presentation-identity> goa-account remove <provider> <presentation-identity> account.json: @@ -26,6 +29,18 @@ check signs in with what's saved, and fails if the server says no. update exits with 3 if it saved the password but the server didn't accept it. disabled prints the features CoreDesktop uses that are turned off in Settings, e.g. "Calendar, Files", and enable turns them back on. + +attention exits 0 if Online Accounts found that the server no longer takes +the saved password (it then shows the account as needing attention). It +doesn't sign in, so it doesn't add to the server's count of failed logins. +Online Accounts 3.58 only finds this for Nextcloud: a mail server refusing the +password is reported as a general failure (see system/email-watch). + +pause empties the account's saved password, keeping its place in the keyring. +Online Accounts signs in again whenever the network changes, and Evolution +whenever it fetches mail; with no password neither tries, so a mail server +with fail2ban doesn't block this computer for repeated wrong passwords. +update puts a password back, which ends the pause. paused exits 0 if paused. """ import json @@ -112,6 +127,45 @@ def store_credentials(provider, account_id, credentials): secret, None) +# The account's saved credentials, e.g. {"imap-password": "..."} +def lookup_credentials(provider, account_id): + items = Secret.password_search_sync( + None, {"xdg:schema": "org.gnome.OnlineAccounts"}, + Secret.SearchFlags.ALL | Secret.SearchFlags.UNLOCK | Secret.SearchFlags.LOAD_SECRETS, None) + for item in items: + identity = item.get_attributes().get("goa-identity", "") + if identity.startswith(f"{provider}:gen") and identity.endswith(f":{account_id}"): + return GLib.Variant.parse(None, item.get_secret().get_text(), None, None).unpack() + raise LookupError(f"there's no saved password for {account_id} in the keyring") + + +# The credential each kind of account signs in with +PASSWORD_KEYS = {"owncloud": "password", "imap_smtp": "imap-password"} + + +def attention(bus, provider, presentation_identity): + found = find_accounts(bus, provider, presentation_identity) + if not found: + raise LookupError(f"there's no {provider} account for {presentation_identity}") + return any(account.get("AttentionNeeded") for _, account in found) + + +def pause(bus, provider, presentation_identity): + found = find_accounts(bus, provider, presentation_identity) + if not found: + raise LookupError(f"there's no {provider} account for {presentation_identity}") + for _, account in found: + store_credentials(provider, account["Id"], {}) + + +def paused(bus, provider, presentation_identity): + found = find_accounts(bus, provider, presentation_identity) + if not found: + raise LookupError(f"there's no {provider} account for {presentation_identity}") + return any(PASSWORD_KEYS[provider] not in lookup_credentials(provider, account["Id"]) + for _, account in found) + + def update(bus, provider, presentation_identity, credentials): found = find_accounts(bus, provider, presentation_identity) if not found: @@ -180,6 +234,13 @@ def main(): if len(sys.argv) == 4 and sys.argv[1] == "enable": enable(bus, sys.argv[2], sys.argv[3]) return 0 + if len(sys.argv) == 4 and sys.argv[1] == "attention": + return 0 if attention(bus, sys.argv[2], sys.argv[3]) else 1 + if len(sys.argv) == 4 and sys.argv[1] == "pause": + pause(bus, sys.argv[2], sys.argv[3]) + return 0 + if len(sys.argv) == 4 and sys.argv[1] == "paused": + return 0 if paused(bus, sys.argv[2], sys.argv[3]) else 1 if len(sys.argv) == 4 and sys.argv[1] == "remove": remove(bus, sys.argv[2], sys.argv[3]) return 0 diff --git a/install-flatpaks.sh b/install-flatpaks.sh index 7a196d1..771e93f 100755 --- a/install-flatpaks.sh +++ b/install-flatpaks.sh @@ -38,16 +38,17 @@ FLATPAKS=( "org.gnome.Evolution" "com.github.xournalpp.xournalpp" "org.libretro.RetroArch" - "io.github.anil_e.Codd" "com.rustdesk.RustDesk" - "io.github.tanaybhomia.Whisp" "org.pipewire.Helvum" "com.jeffser.Alpaca" "ai.lmstudio.lm-studio" "com.cassidyjames.plausible" - "io.github.sitraorg.sitra" ) +# Leave out those listed in ~/.config/coredesktop/skip-packages +source "$HOME/.local/share/coredesktop/utils.sh" +mapfile -t FLATPAKS < <(without_skipped "${FLATPAKS[@]}") + for pak in "${FLATPAKS[@]}"; do if ! flatpak info "$pak" &> /dev/null; then echo "Installing Flatpak: $pak" diff --git a/install.sh b/install.sh index e67c4f1..de5c75c 100755 --- a/install.sh +++ b/install.sh @@ -150,11 +150,6 @@ echo "Installing flatpaks..." echo "Installing tmux plug-in manager..." . install-tpm.sh -# Install Mise version manager, then use it to set up Ruby and Rails -echo "Installing Mise, Ruby, Rails..." -cd ~/.local/share/coredesktop -. install-mise-ruby-rails.sh - # Install Kickstart Neovim config as the default Neovim configuration echo "Installing Kickstart Neovim..." . install-kickstart-nvim.sh @@ -213,6 +208,10 @@ bash ~/.local/share/coredesktop/shell/setup.sh echo "Configuring update notifications..." bash ~/.local/share/coredesktop/system/update-notify.sh +# Pause a Core's email when the mail server stops taking its password +echo "Configuring email password checks..." +bash ~/.local/share/coredesktop/system/email-watch.sh + # Apply GNOME dconf settings only if the settings file has changed since last run bash ~/.local/share/coredesktop/gnome/apply-gnome-settings.sh diff --git a/migrations/1793454799.sh b/migrations/1793454799.sh new file mode 100644 index 0000000..5975dd8 --- /dev/null +++ b/migrations/1793454799.sh @@ -0,0 +1,29 @@ +#!/bin/bash + +# CoreDesktop no longer installs nerdfetch (fastfetch does the same), Sitra +# (fonts), Whisp (notes) or Codd (PostgreSQL). Remove them once. Install one +# again yourself and update leaves it alone. Flatpaks' data stays in ~/.var/app. +DROPPED=(nerdfetch) +DROPPED_FLATPAKS=(io.github.sitraorg.sitra io.github.tanaybhomia.Whisp io.github.anil_e.Codd) + +for pkg in "${DROPPED[@]}"; do + # The exact name: pacman -Q also matches a package that only provides it + [ "$(pacman -Qq "$pkg" 2> /dev/null)" = "$pkg" ] || continue + + # Something else still needs it: leave it to go with that, as an orphan + if [ "$(LC_ALL=C pacman -Qi "$pkg" | awk -F' *: *' '/^Required By/ { print $2 }')" != "None" ]; then + sudo pacman -D --asdeps "$pkg" > /dev/null || exit 1 + continue + fi + + echo "Removing $pkg..." + sudo pacman -Rns --noconfirm "$pkg" || exit 1 +done + +for pak in "${DROPPED_FLATPAKS[@]}"; do + if flatpak info --system "$pak" &> /dev/null; then + echo "Removing $pak..." + sudo flatpak uninstall --system --noninteractive "$pak" || exit 1 + fi +done +exit 0 diff --git a/packages.conf b/packages.conf index db4b1a7..0abe7bd 100644 --- a/packages.conf +++ b/packages.conf @@ -34,7 +34,6 @@ SYSTEM_UTILS=( mc mdns-tui-browser mtr - nerdfetch net-tools networkmanager nmap @@ -79,6 +78,8 @@ DEV_TOOLS=( # The source-built herdr pins an exact zig version that Arch rarely ships herdr-bin hugo + # Installs other languages (configure languages), and the versions projects pin + mise neovim nodejs npm diff --git a/shell/bashrc b/shell/bashrc index 5db2404..0f31e0f 100644 --- a/shell/bashrc +++ b/shell/bashrc @@ -31,6 +31,12 @@ if [[ $TERM != "dumb" ]] && command -v starship &> /dev/null; then eval "$(starship init bash)" fi +# mise: the language versions a project's mise.toml asks for, and those you +# installed with configure languages, ahead of the ones from pacman +if command -v mise &> /dev/null; then + eval "$(mise activate bash)" +fi + # z <part of a path> jumps to a directory you've visited before if command -v zoxide &> /dev/null; then eval "$(zoxide init bash)" diff --git a/skills/sibling-project-updates.md b/skills/sibling-project-updates.md index 7970e68..dbb33f1 100644 --- a/skills/sibling-project-updates.md +++ b/skills/sibling-project-updates.md @@ -66,7 +66,7 @@ Existing coredesktop hooks use a numeric prefix to control execution order: | `15-services.sh` | Enable new systemd services | | `20-aur.sh` | AUR packages | | `30-flatpak.sh` | Flatpak apps | -| `40-ruby.sh` | mise, Ruby/gems | +| `40-mise.sh` | languages installed with mise, Rails | | `45-tmux.sh` | TPM and tmux plugins | | `46-neovim.sh` | Neovim plugins | | `50-bitwarden.sh` | Bitwarden SSH agent | diff --git a/system/email-watch b/system/email-watch new file mode 100755 index 0000000..d41005c --- /dev/null +++ b/system/email-watch @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +"""Pause a Core's email in Online Accounts when its password stops working. + +When your Core password changes (e.g. in Superpanel), email in Online Accounts +still has the old one. Online Accounts signs in with it each time the network +changes, and Evolution each time it fetches mail, and after a few wrong +passwords the mail server's fail2ban blocks this computer. + +Online Accounts doesn't say when a mail server refuses a password (3.58 reports +it as a general failure). It does say when Nextcloud refuses one, and Nextcloud +turns off its app passwords when your Core password changes. So when the +Core's Nextcloud in Online Accounts needs attention, this signs in to Nextcloud, +once, with the password email has (your Core password). If Nextcloud says it's +wrong, the mail server will too, so this pauses email +(configure.d/lib/goa-account pause): nothing signs in with the old password +again. It shows a notification; clicking it runs configure core, where "Your +Core password changed" saves the new password and ends the pause. It never +signs in to the mail server itself. + +Run by coredesktop-email-watch.service (see system/email-watch.sh). + +email-watch --check-password <email> checks the password email has with +Nextcloud, as above, for configure core's tests: it exits 0 if it works, 2 if +Nextcloud says it's wrong, 1 if it couldn't tell. +""" + +import base64 +import glob +import importlib.machinery +import importlib.util +import os +import subprocess +import sys +import threading +import urllib.error +import urllib.request + +import gi + +gi.require_version("Gio", "2.0") +from gi.repository import Gio, GLib # noqa: E402 + +COREDESKTOP = os.path.expanduser("~/.local/share/coredesktop") +RUN_IN_TERMINAL = os.path.join(COREDESKTOP, "system/run-in-terminal") +CORES = os.path.expanduser("~/.config/coredesktop/cores/*.conf") + + +def load_goa_account(): + # No __pycache__ in the CoreDesktop checkout, which update pulls into + sys.dont_write_bytecode = True + path = os.path.join(COREDESKTOP, "configure.d/lib/goa-account") + loader = importlib.machinery.SourceFileLoader("goa_account", path) + module = importlib.util.module_from_spec(importlib.util.spec_from_loader("goa_account", loader)) + loader.exec_module(module) + return module + + +goa = load_goa_account() + + +# Each Core's email and domain, from configure core's cores/<domain>.conf +def cores(): + found = {} + for conf in glob.glob(CORES): + result = subprocess.run( + ["bash", "-c", 'source "$1" && printf "%s\\n%s\\n" "$CORE_EMAIL" "$CORE_DOMAIN"', "-", conf], + capture_output=True, text=True) + lines = result.stdout.split("\n") + if result.returncode == 0 and len(lines) >= 2 and lines[0] and lines[1]: + found[lines[0]] = lines[1] + return found + + +def nextcloud_url(domain): + return f"https://nextcloud.{domain}" + + +# Sign in to the Core's Nextcloud with the password email has, the way +# configure core checks a password (configure.d/lib/nextcloud.sh). True if it +# works, False if Nextcloud says it's wrong, None if it couldn't tell. +def nextcloud_takes(email, domain, password): + user = email.rsplit("@", 1)[0] + auth = base64.b64encode(f"{user}:{password}".encode()).decode() + request = urllib.request.Request( + f"{nextcloud_url(domain)}/ocs/v2.php/cloud/user?format=json", + headers={"Authorization": f"Basic {auth}", "OCS-APIRequest": "true"}) + try: + with urllib.request.urlopen(request, timeout=15): + return True + except urllib.error.HTTPError as error: + return False if error.code == 401 else None + except OSError: + return None + + +def email_password(bus, email): + for _, account in goa.find_accounts(bus, "imap_smtp", email): + return goa.lookup_credentials("imap_smtp", account["Id"]).get("imap-password") + return None + + +def check_password(bus, email): + domain = cores().get(email) + if domain is None: + return None + password = email_password(bus, email) + if not password: + return None + return nextcloud_takes(email, domain, password) + + +# Waits for a click in its own thread, so the watching goes on meanwhile +def notify(email, domain, paused_now): + def run(): + if paused_now: + body = (f"Your Core password seems to have changed: {email} has the old one. " + "So the mail server doesn't block this computer for trying it, it's " + "paused. Click to enter your new password.") + else: + body = (f"{email} is paused because it has an old password. " + "Click to enter your new Core password.") + result = subprocess.run( + ["notify-send", "--wait", "--app-name=CoreDesktop", "--icon=mail-unread", + "--action=default=Fix", "--action=fix=Enter new password", + f"Email for {domain} is paused", body], + capture_output=True, text=True) + if result.stdout.strip() in ("default", "fix"): + subprocess.run([RUN_IN_TERMINAL, "configure core"]) + + threading.Thread(target=run, daemon=True).start() + + +# Emails already dealt with since they started failing, so each is checked and +# notified about once until it works again +handled = set() + + +def check(bus): + try: + accounts = {(account.get("ProviderType"), account.get("PresentationIdentity")): account + for _, account in goa.accounts(bus)} + except GLib.Error: + return + for email, domain in cores().items(): + mail = accounts.get(("imap_smtp", email)) + if mail is None: + continue + nextcloud = accounts.get(("owncloud", f"{email.rsplit('@', 1)[0]}@nextcloud.{domain}")) + mail_failing = bool(mail.get("AttentionNeeded")) + nextcloud_failing = bool(nextcloud and nextcloud.get("AttentionNeeded")) + if not mail_failing and not nextcloud_failing: + handled.discard(email) + continue + if email in handled: + continue + + try: + if goa.paused(bus, "imap_smtp", email): + handled.add(email) + notify(email, domain, False) + continue + # Nextcloud may only have had an app password revoked, and email may + # still be marked from a pause whose new password hasn't been + # checked yet, so check the password email has before pausing it + password = email_password(bus, email) + if not password: + continue # e.g. the keyring is locked; try again later + works = nextcloud_takes(email, domain, password) + if works is None: + continue # try again when the network changes + if works: + handled.add(email) + continue + goa.pause(bus, "imap_smtp", email) + except (GLib.Error, LookupError): + continue + handled.add(email) + notify(email, domain, True) + + +def watch(): + bus = Gio.bus_get_sync(Gio.BusType.SESSION, None) + pending = [] + + # Signals come in bursts; check once they've settled + def queue_check(*_): + if not pending: + pending.append(GLib.timeout_add_seconds(2, run_check)) + + def run_check(): + pending.clear() + check(bus) + return GLib.SOURCE_REMOVE + + account = "org.gnome.OnlineAccounts.Account" + bus.signal_subscribe(goa.BUS_NAME, "org.freedesktop.DBus.Properties", "PropertiesChanged", + None, account, Gio.DBusSignalFlags.NONE, queue_check) + bus.signal_subscribe(goa.BUS_NAME, "org.freedesktop.DBus.ObjectManager", "InterfacesAdded", + None, None, Gio.DBusSignalFlags.NONE, queue_check) + # Online Accounts checks every account when it starts, e.g. at login, and + # when the network changes + bus.signal_subscribe("org.freedesktop.DBus", "org.freedesktop.DBus", "NameOwnerChanged", + "/org/freedesktop/DBus", goa.BUS_NAME, Gio.DBusSignalFlags.NONE, queue_check) + Gio.NetworkMonitor.get_default().connect("network-changed", queue_check) + queue_check() + GLib.MainLoop().run() + + +def main(): + if len(sys.argv) == 3 and sys.argv[1] == "--check-password": + bus = Gio.bus_get_sync(Gio.BusType.SESSION, None) + try: + works = check_password(bus, sys.argv[2]) + except (GLib.Error, LookupError): + works = None + return {True: 0, False: 2, None: 1}[works] + if len(sys.argv) == 1: + watch() + return 0 + print(__doc__.strip().split("\n\n")[-1], file=sys.stderr) + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/system/email-watch.sh b/system/email-watch.sh new file mode 100644 index 0000000..48ed37d --- /dev/null +++ b/system/email-watch.sh @@ -0,0 +1,19 @@ +#!/bin/bash +# +# Purpose : Install and start the service that pauses a Core's email when the +# mail server stops taking its password (system/email-watch). +# Run by install.sh and on every update (updates.d/17-email-watch.sh), +# so changes to it reach existing machines. + +set -e + +COREDESKTOP="$HOME/.local/share/coredesktop" +UNITS="$HOME/.config/systemd/user" + +mkdir -p "$UNITS" +cp "$COREDESKTOP/system/systemd/coredesktop-email-watch.service" "$UNITS/" +systemctl --user daemon-reload +systemctl --user enable coredesktop-email-watch.service +# Restart, so it runs the version update just pulled. It doesn't start without +# a Core set up. +systemctl --user restart coredesktop-email-watch.service diff --git a/system/run-in-terminal b/system/run-in-terminal new file mode 100755 index 0000000..d6587b6 --- /dev/null +++ b/system/run-in-terminal @@ -0,0 +1,27 @@ +#!/bin/bash +# +# Purpose : Run a command in a new terminal window, for notifications that +# start something when clicked (system/update-check, system/email-watch). +# +# run-in-terminal <command>, e.g. run-in-terminal update +# +# bash -i reads ~/.bashrc, as if the user had typed the command themselves, and +# the window stays open to show how it went. + +run=(bash -ic "$1; echo; read -rp \"Press Enter to close this window.\"") +if command -v xdg-terminal-exec &> /dev/null; then + terminal=(xdg-terminal-exec) +elif command -v kgx &> /dev/null; then + terminal=(kgx --) +elif command -v ghostty &> /dev/null; then + terminal=(ghostty -e) +elif command -v kitty &> /dev/null; then + terminal=(kitty) +else + notify-send --app-name=CoreDesktop "No terminal found" "Open a terminal and run $1." + exit 1 +fi + +# Run the terminal as its own unit: when the service that called this exits, +# systemd stops everything still running in it, which would close the terminal +exec systemd-run --user --quiet --collect -- "${terminal[@]}" "${run[@]}" diff --git a/system/systemd/coredesktop-email-watch.service b/system/systemd/coredesktop-email-watch.service new file mode 100644 index 0000000..397d76f --- /dev/null +++ b/system/systemd/coredesktop-email-watch.service @@ -0,0 +1,14 @@ +[Unit] +Description=Pause a Core's email when the mail server stops taking its password +PartOf=graphical-session.target +After=graphical-session.target +# Only for those who've set up a Core (configure core starts it then) +ConditionPathExistsGlob=%h/.config/coredesktop/cores/*.conf + +[Service] +ExecStart=%h/.local/share/coredesktop/system/email-watch +Restart=on-failure +RestartSec=30 + +[Install] +WantedBy=graphical-session.target diff --git a/system/update-check b/system/update-check index 28d1a5f..d665b0f 100755 --- a/system/update-check +++ b/system/update-check @@ -30,23 +30,5 @@ echo "$latest" > "$NOTIFIED" [ "$action" = "default" ] || [ "$action" = "update" ] || exit 0 -# Open the first terminal found. bash -i reads ~/.bashrc, as if the user had -# typed update themselves, and the window stays open to show how it went. -run_update=(bash -ic 'update; echo; read -rp "Press Enter to close this window."') -if command -v xdg-terminal-exec &> /dev/null; then - terminal=(xdg-terminal-exec) -elif command -v kgx &> /dev/null; then - terminal=(kgx --) -elif command -v ghostty &> /dev/null; then - terminal=(ghostty -e) -elif command -v kitty &> /dev/null; then - terminal=(kitty) -else - notify-send --app-name=CoreDesktop "No terminal found" "Open a terminal and run update." - exit 1 -fi - -# Run the terminal as its own unit: when this service exits, systemd stops -# everything still running in it, which would close the terminal mid-update -exec systemd-run --user --quiet --collect -- "${terminal[@]}" "${run_update[@]}" +exec "$HOME/.local/share/coredesktop/system/run-in-terminal" update } diff --git a/tests/utils.bats b/tests/utils.bats index 0fc029e..acce5a3 100644 --- a/tests/utils.bats +++ b/tests/utils.bats @@ -117,6 +117,45 @@ esac' not_called yay } +@test "install_packages leaves out packages in skip-packages" { + export INSTALLED="" AS_DEPS="" + stub_pacman + mkdir -p "$HOME/.config/coredesktop" + printf '# Not on this machine\nlibreoffice-fresh\n\n gimp \n' > "$HOME/.config/coredesktop/skip-packages" + source "$REPO/utils.sh" + + run install_packages git libreoffice-fresh gimp tmux + [ "$status" -eq 0 ] + called yay -S --noconfirm --answerdiff=None --answeredit=None --answerclean=None git tmux +} + +@test "install_packages does nothing when every package is skipped" { + export INSTALLED="" AS_DEPS="" + stub_pacman + mkdir -p "$HOME/.config/coredesktop" + printf 'gimp\n' > "$HOME/.config/coredesktop/skip-packages" + source "$REPO/utils.sh" + + run install_packages gimp + [ "$status" -eq 0 ] + # pacman -Qdq with no names would mark every dependency explicit + not_called pacman + not_called yay +} + +@test "install-flatpaks.sh leaves out Flatpaks in skip-packages" { + mkdir -p "$HOME/.local/share" "$HOME/.config/coredesktop" + ln -s "$REPO" "$HOME/.local/share/coredesktop" + printf 'us.zoom.Zoom\n' > "$HOME/.config/coredesktop/skip-packages" + stub flatpak 'exit 1' + stub sudo + + run bash "$REPO/install-flatpaks.sh" + [ "$status" -eq 0 ] + called sudo flatpak install --system --noninteractive org.mozilla.firefox + ! grep -q 'us.zoom.Zoom' <(grep '^sudo' "$STUB_LOG") +} + @test "remove_packages removes only exact matches, not providers" { # herdr-bin provides herdr, so pacman -Qq herdr prints herdr-bin stub pacman ' diff --git a/updates.d/17-email-watch.sh b/updates.d/17-email-watch.sh new file mode 100644 index 0000000..170afcd --- /dev/null +++ b/updates.d/17-email-watch.sh @@ -0,0 +1,7 @@ +#!/bin/bash + +COREDESKTOP="$HOME/.local/share/coredesktop" + +run_update() { + bash "$COREDESKTOP/system/email-watch.sh" +} diff --git a/updates.d/40-mise.sh b/updates.d/40-mise.sh index baaa207..0764333 100644 --- a/updates.d/40-mise.sh +++ b/updates.d/40-mise.sh @@ -1,16 +1,25 @@ #!/bin/bash -# Answer yes to any mise prompt (e.g. trusting config files) -export MISE_YES=1 +# Upgrade the languages you installed with mise (configure languages), within +# the versions you chose: ruby@latest moves to the newest Ruby, java@21 stays 21. +# mise itself comes from pacman, which updates it. -echo "Updating mise..." -mise self-update -y +is_installed() { + command -v mise > /dev/null && [ -n "$(mise ls --global --quiet 2> /dev/null)" ] +} -echo "Updating Ruby and Rails..." -mise upgrade --bump ruby -mise reshim -# A new Ruby's bundled gems (e.g. minitest) come with executables RubyGems -# doesn't recognise as its own, so gem update stops to ask whether to overwrite -# them. Regenerating them first lets the update go through without asking. -gem pristine --all --only-executables > /dev/null -gem update +run_update() { + # Answer yes to any mise prompt (e.g. trusting config files) + export MISE_YES=1 + # From home, so a project's mise.toml in the current directory doesn't count + cd "$HOME" || return 1 + + echo "Updating languages installed with mise..." + mise upgrade || return 1 + + # A new Ruby gets Rails from ~/.default-gems; this updates it between Rubies + if mise exec -- gem list -i rails &> /dev/null; then + echo "Updating Rails..." + mise exec -- gem update rails bundler || return 1 + fi +} diff --git a/utils.sh b/utils.sh index d7f7961..2d18039 100755 --- a/utils.sh +++ b/utils.sh @@ -10,13 +10,36 @@ is_group_installed() { pacman -Qg "$1" &> /dev/null } +# Packages and Flatpaks you don't want, one name per line (# starts a comment). +# install_packages and install-flatpaks.sh leave them out, so update doesn't +# put back what you've removed. +SKIP_PACKAGES="$HOME/.config/coredesktop/skip-packages" + +# Print each argument that isn't in SKIP_PACKAGES, one per line +without_skipped() { + local -A skip=() + local name + if [ -f "$SKIP_PACKAGES" ]; then + while read -r name _; do + [[ -n "$name" && "$name" != \#* ]] && skip["$name"]=1 + done < "$SKIP_PACKAGES" + fi + for name in "$@"; do + [ -n "${skip[$name]}" ] || echo "$name" + done +} + # Function to install packages if not already installed install_packages() { - local packages=("$@") + local packages=() local to_install=() local as_deps=() + mapfile -t packages < <(without_skipped "$@") + # pacman -Qdq with no names would list every dependency on the system + [ ${#packages[@]} -ne 0 ] || return 0 + for pkg in "${packages[@]}"; do if ! is_installed "$pkg" && ! is_group_installed "$pkg"; then to_install+=("$pkg")