add deploy-stack skill; falco: fix hook-tamper condition, add three more post-incident rules

git-hook-tamper.yaml's condition used proc.exepath, which resolves to the script interpreter's path (e.g. /bin/busybox) for shebang scripts, not the script's own path - switched to proc.cmdline, which retains the originally invoked path. Confirmed via live-testing both ways. ssh-persistence/cloud-metadata-probe/db-spawned-process round out the post-incident hardening pass with a few more incubating-ruleset adaptations.
2026-08-16 14:33:56 +00:00
parent 9bf72f8e19
commit 2ed0565486
6 changed files with 172 additions and 1 deletions
@@ -0,0 +1,79 @@
+---
+name: deploy-stack
+description: Deploy a docker-infrastructure change - push to Gitea, then redeploy the stack via Portainer. Use whenever a service's docker-compose.yaml (or other tracked file) changed and needs to go live.
+---
+
+# Deploy a stack (Gitea → Portainer)
+
+This repo's services go live via GitOps: commit → push to Gitea (self-hosted, SSH
+remote) → tell Portainer to pull + redeploy. Always use `./portainer.sh` for the
+Portainer side; never raw `curl` unless `portainer.sh` can't do it.
+
+## Steps
+
+1. **Stage and commit** only the files that changed for this task (never `git add -A`).
+   Imperative mood, first line < 72 chars, explain *why* not *what*.
+
+2. **Sync with remote before pushing.** `git push origin main` fails non-fast-forward
+   if anyone (or the Gitea web UI) committed since your last pull. Prefer:
+   ```
+   git pull --no-edit origin main   # fetch + merge in one step
+   git push origin main
+   ```
+   The remote is SSH (`git@gitea.kolpacksoftware.com:...`). `GITEA_TOKEN` in
+   `.credentials` is NOT valid for HTTPS push (403) - don't try HTTPS for push.
+
+3. **If push/fetch fails with a low-level git protocol error** (`bad pack header`,
+   `unable to fork git-pack-objects`, `cannot exec '.../hooks/...'`) - this is NOT
+   a normal merge conflict, it means Gitea's git service itself is broken
+   (corrupted repo, poisoned global gitconfig, etc). Do not just retry blindly.
+   Stop and investigate server-side (`docker logs gitea`, check
+   `/data/gitea/home/.gitconfig` inside the container) before continuing - this
+   exact failure mode was an active RCE backdoor once (see
+   `gitea-log-poisoning-attack-2026-08.md` memory). HTTPS fetch with `GITEA_TOKEN`
+   can help bisect whether it's SSH-specific or instance-wide:
+   ```
+   source .credentials
+   git fetch "https://${GITEA_USER}:${GITEA_TOKEN}@gitea.kolpacksoftware.com/homelab/docker-infrastructure.git" main
+   ```
+
+4. **Find the stack name/ID** if you don't already know it:
+   ```
+   source .credentials && ./portainer.sh list
+   ```
+   Stack names in Portainer usually match the service directory name, but not
+   always (verify with `list`, don't assume).
+
+5. **Redeploy** (pulls latest git commit + recreates containers):
+   ```
+   source .credentials && ./portainer.sh redeploy <stack-name>
+   ```
+   This only works for stacks that are git-linked in Portainer. If it's not
+   git-linked, changes to `.env` values must go through
+   `./portainer.sh set-env <stack-name> KEY=VALUE` instead - a `.env` file at
+   the repo path is gitignored and NOT read by git-linked Portainer deploys.
+
+6. **Verify**: `docker ps --filter name=<container>` for status, `docker logs
+   <container> --tail 50` to confirm it actually came up clean, not just "Up".
+
+## Gotchas
+
+- **`.env` files are gitignored and invisible to git-linked stacks.** Portainer
+  reads env vars it has stored for the stack (set via UI or `portainer.sh
+  set-env`), not the local `.env` file. Check current values first with
+  `./portainer.sh get-env <stack-name>` before adding new ones, and set any new
+  var explicitly - editing the local `.env` alone does nothing for a deployed
+  git-linked stack.
+- **Not every service is a Portainer stack.** Some (e.g. `falco`) are run via
+  plain `docker run`/manual `docker compose`, bind-mounting config directly from
+  this repo checkout. For those, `portainer.sh redeploy` will fail with "stack
+  not found" - check `docker inspect <container> --format '{{json .Mounts}}'`
+  to see if it's bind-mounted from this repo (if so, a `git push` alone is
+  enough to update the *source* files; the container itself needs a manual
+  `docker restart` or recreate to pick up new volume mounts).
+- **No `docker compose` CLI on this host.** `docker compose up -d` fails
+  outright - always go through Portainer, or `docker run`/`docker restart`
+  directly for manually-managed services.
+- Portainer's `GET /api/stacks/<id>` can return Unauthorized for some stacks -
+  `portainer.sh` already works around this by listing + filtering; don't call
+  the raw API directly.