Found during live testing: netdata's cloud-provider auto-detection runs
curl --fail -s -m1 --noproxy * http://169.254.169.254 on every
startup/reconnect to check whether the host is in AWS/GCP/Azure - standard
monitoring-agent behavior, not a bug. Without this exception the rule
would have paged every time netdata restarts. Scoped to netdata's own
curl specifically so any other connection from that container still
alerts.
git-hook-tamper.yaml's condition used proc.exepath, which resolves to the
script interpreter's path (e.g. /bin/busybox) for shebang scripts, not the
script's own path - switched to proc.cmdline, which retains the originally
invoked path. Confirmed via live-testing both ways.
ssh-persistence/cloud-metadata-probe/db-spawned-process round out the
post-incident hardening pass with a few more incubating-ruleset adaptations.
Written after the 2026-08-10/11 gitea internal-API log-poisoning attack
that planted a malicious uploadpack.packObjectsHook backdoor. Catches
both the planting (unexpected exec from a hooks/ path) and the firing
(git-upload-pack/git-receive-pack spawning anything but its own
pack-objects binary), independent of how the hook config got written.
No repeat intrusion - investigated all Falco alerts since deployment
(2026-08-08 to 2026-08-16); the only crypto-miner rule hits were the
original test alerts. The daily self-resolving Telegram alerts the user
was seeing came from four known-benign sources: Obsidian's Electron
self-re-exec pattern and s6-init sudoers setup (fires ~daily on container
recreation), ttyd/Synchronet's telnet/pv/busybox-extras BBS tooling, and
ollama's nvidia-ctk ldconfig-refresh hook (memfd_create by design). Added
scoped exceptions for each via Falco's own rule extension points, same
pattern as the earlier pg_isready/gitea fixes.
Also fixed netdata's used_swap alarm, which recalculated from the
instantaneous raw sample every 10s with only a 30s notification debounce -
a brief swap spike was enough to page both Telegram and (via Netdata
Cloud) email. Widened the up-delay to 5m so only sustained swap pressure
notifies.
Verified end-to-end via a fresh fake-xmrig test after redeploying Falco
and reloading netdata: CRITICAL alarm fired and Telegram delivery
succeeded.
- 'Drop and execute new binary in container' fired on every git push/pull
to gitea over SSH (gitea's own binary re-executing itself for git-shell
hooks looks like container drift). Scoped exception via
known_drop_and_execute_activities to proc.name=gitea on that specific
image, not a blanket container whitelist, so other unexpected binaries
in that container still get caught.
- netdata alarm text was a static generic blurb requiring a manual
'docker logs falco' every time to find out what actually happened.
Now interpolates ${label:rule_name}/${label:priority} (exposed by
Falco's Prometheus metric) so the Telegram message names the specific
rule directly. Note: literal double-quotes in the info/summary text
broke netdata's alarm-notify.sh (silent delivery failure, exit 1) -
avoided.
- Added netdata/health.d and netdata/go.d to the repo for documentation;
netdata does not auto-deploy these from git, same as gitea's app.ini -
copy to /srv/netdata/config/ manually and restart the container.
Navidrome indexes the existing music library on nas_media for Subsonic
access. SUB/WAVE (byo-proxy variant, LAN-only) provides the Icecast
broadcast, Liquidsoap crossfade/ducking, and AI DJ controller, reusing
the existing Ollama instance over npm-network for the LLM and Piper
for TTS.
Detects unauthorized miners (and other runtime threats) via Falco's
syscall/eBPF monitoring, following the 2026-08-06 gitea/xmrig
compromise. Wired into netdata rather than a new notification channel:
Falco exposes a Prometheus /metrics endpoint, netdata scrapes it, and a
new health.d alarm pages through netdata's already-configured Telegram
bot - no new alerting infra needed.
Includes a custom process-name rule for known miner binaries (the stock
Stratum-protocol rule wouldn't have caught the actual gitea incident,
which used a bare host:port with no scheme prefix), an outbound
miner-pool-port rule as a second layer, and rule_matching: all in the
Falco config - without it, Falco silently drops all but the first
matching rule per event, which would have suppressed our custom rule
whenever a stock rule also matched the same process.
Two cryptominer incursions (inbox-zero, then gitea) went unnoticed until
found manually — there was no alerting on sustained host CPU. Wired
netdata's existing 10min_cpu_usage alarm to notify via the shared Telegram
bot (config is host-only, not committed). Also narrowed the dashboard port
from 0.0.0.0 to the host's LAN IP to reduce exposure.
Same category of risk as the gitea :nightly issue: floating dev/beta tags
let watchtower silently pull unvetted upstream builds. couchdb had no tag
at all (implicit :latest); filebrowser and statping only publish
beta/dev channels upstream so pinned to the exact version in use rather
than a moving target.
Pin image to digest instead of :nightly (DB schema already ahead of
stable 1.27, downgrading would break startup) and disable open
self-registration/OpenID signup, which attackers used to plant xmrig
via a git hook.
- quizmaster: removed entirely (test service, no longer used); also removed Portainer stack and /srv/quizmaster data
- leafweb: removed orphaned MSSQL compose file (no running container, no Portainer stack, no data on disk)
- authelia, gitea, couchdb, portainer, immich (server + ml): add com.centurylinklabs.watchtower.monitor-only=true so watchtower still notifies on new versions without auto-applying them
- ladder: restore restart: unless-stopped (was commented out)
- rackpeek: add missing TZ env var
- watchtower: move Telegram token/chat ID to .env instead of plaintext in compose
- plex: remove stale/dead PLEX_CLAIM token
- uptime-kuma: mount docker.sock read-only (only needs to read container state)
- home-assistant: drop privileged: true (no device passthrough configured)
- glances: drop privileged: true, bind dashboard port to 127.0.0.1
- couchdb, docker-registry, leafweb: bind ports to 127.0.0.1 (proxy/localhost-only access confirmed, no direct LAN clients)
immich-server auto-updated to v3.0.1 via watchtower but postgres stayed
pinned to the old pgvecto.rs image, causing a crash loop (missing vector
extension). Per official Immich migration guide.
open-webui was sending OAuth credentials via both client_secret_basic
(header) and client_secret_post (body) simultaneously, which Authelia
rejects. Setting OAUTH_TOKEN_ENDPOINT_AUTH_METHOD=client_secret_post
forces a single method and matches the updated Authelia client config.
WEBUI_SECRET_KEY is now a stable env var so watchtower image updates
no longer invalidate all user sessions.
These vhost.d entries were symlinks to a non-existent 'public' file.
Portainer refuses to clone repos containing symlinks, blocking all
git-based stack redeployments. Replace with empty regular files
(equivalent behavior: no extra nginx config = public access).
NFS named volumes go ESTALE when unRAID drives spin down and Docker cannot recover.
systemd automount handles NFS reconnection transparently; Docker just sees a bind mount.
poprhythm