Host rebooted 2026-09-16 17:47 UTC, recreating every container at once and
producing 70 alerts in a single burst - all traced to benign init-time
patterns that will recur on every future reboot: systemd-executor's PAM
reads during boot, firefly-iii/fidi's s6 healthcheck shell, and romm's own
loopback redis/app connection at container start.
Investigated recent Telegram alerts - no intrusion, both new hits trace to
benign one-offs: calibre's linuxserver.io s6-init NOPASSWD setup (same
pattern already whitelisted for obsidian) and lsyncd truncating its own
status log (not evidence-clearing).
"Redirect STDOUT/STDIN to Network Connection in Container" (a reverse-shell
detector) was firing legitimately: gitea's sshd/sshd-session dup2 the
accepted SSH socket onto stdio for every session (4x per connection), and
firefly-iii's wait-for-it.sh does the same TCP-readiness-check dance while
waiting for postgres. Both confirmed recurring via netdata's alert history,
not one-off. Scoped exceptions added to each's specific binary/cmdline,
not the whole container/image.
Verified: a gitea SSH login no longer alerts, while a real dup2-based
redirect (bash's /dev/tcp exec pattern) from an unrelated container still
fires - the exception is narrow, not a blanket disable.
"Run shell untrusted" was also flagged as noisy, but investigation showed
it fired exactly once, during my own rule-testing window, and never
before or since - left alone rather than building a permanent exception
for a self-caused test artifact.
No repeat intrusion - investigated all Falco alerts since deployment
(2026-08-08 to 2026-08-16); the only crypto-miner rule hits were the
original test alerts. The daily self-resolving Telegram alerts the user
was seeing came from four known-benign sources: Obsidian's Electron
self-re-exec pattern and s6-init sudoers setup (fires ~daily on container
recreation), ttyd/Synchronet's telnet/pv/busybox-extras BBS tooling, and
ollama's nvidia-ctk ldconfig-refresh hook (memfd_create by design). Added
scoped exceptions for each via Falco's own rule extension points, same
pattern as the earlier pg_isready/gitea fixes.
Also fixed netdata's used_swap alarm, which recalculated from the
instantaneous raw sample every 10s with only a 30s notification debounce -
a brief swap spike was enough to page both Telegram and (via Netdata
Cloud) email. Widened the up-delay to 5m so only sustained swap pressure
notifies.
Verified end-to-end via a fresh fake-xmrig test after redeploying Falco
and reloading netdata: CRITICAL alarm fired and Telegram delivery
succeeded.
- 'Drop and execute new binary in container' fired on every git push/pull
to gitea over SSH (gitea's own binary re-executing itself for git-shell
hooks looks like container drift). Scoped exception via
known_drop_and_execute_activities to proc.name=gitea on that specific
image, not a blanket container whitelist, so other unexpected binaries
in that container still get caught.
- netdata alarm text was a static generic blurb requiring a manual
'docker logs falco' every time to find out what actually happened.
Now interpolates ${label:rule_name}/${label:priority} (exposed by
Falco's Prometheus metric) so the Telegram message names the specific
rule directly. Note: literal double-quotes in the info/summary text
broke netdata's alarm-notify.sh (silent delivery failure, exit 1) -
avoided.
- Added netdata/health.d and netdata/go.d to the repo for documentation;
netdata does not auto-deploy these from git, same as gitea's app.ini -
copy to /srv/netdata/config/ manually and restart the container.
Detects unauthorized miners (and other runtime threats) via Falco's
syscall/eBPF monitoring, following the 2026-08-06 gitea/xmrig
compromise. Wired into netdata rather than a new notification channel:
Falco exposes a Prometheus /metrics endpoint, netdata scrapes it, and a
new health.d alarm pages through netdata's already-configured Telegram
bot - no new alerting infra needed.
Includes a custom process-name rule for known miner binaries (the stock
Stratum-protocol rule wouldn't have caught the actual gitea incident,
which used a bare host:port with no scheme prefix), an outbound
miner-pool-port rule as a second layer, and rule_matching: all in the
Falco config - without it, Falco silently drops all but the first
matching rule per event, which would have suppressed our custom rule
whenever a stock rule also matched the same process.
poprhythm