Host rebooted 2026-09-16 17:47 UTC, recreating every container at once and
producing 70 alerts in a single burst - all traced to benign init-time
patterns that will recur on every future reboot: systemd-executor's PAM
reads during boot, firefly-iii/fidi's s6 healthcheck shell, and romm's own
loopback redis/app connection at container start.
Confirmed 5 false positives over 2026-08-20 to 2026-08-25, all connecting
to ::1:8888 at ~21:3x daily - never a real mining-pool destination since
that's loopback, and every attempt failed instantly (nothing listens on
8888). Couldn't identify the source process (falco loses metadata for it
before it can be captured), but a real miner pool can never be localhost,
so excluding loopback fixes the false-positive class without weakening
the rule for actual remote pool connections.
Investigated recent Telegram alerts - no intrusion, both new hits trace to
benign one-offs: calibre's linuxserver.io s6-init NOPASSWD setup (same
pattern already whitelisted for obsidian) and lsyncd truncating its own
status log (not evidence-clearing).
The rule checked proc.name against gitea's managed hook names, but git's
hook dispatcher always execs these as `bash ./hooks/<hookname>.d/gitea` -
proc.name is "bash" (the interpreter), never the hook name. That check
could never match, so this fired CRITICAL 3x (pre-receive/update/
post-receive) on every single push since it was added - including two
notifications the user got moments ago from this repo's own commits.
Fixed to check the actual invoked script path in proc.cmdline instead.
This commit's own push is the live verification.
"Redirect STDOUT/STDIN to Network Connection in Container" (a reverse-shell
detector) was firing legitimately: gitea's sshd/sshd-session dup2 the
accepted SSH socket onto stdio for every session (4x per connection), and
firefly-iii's wait-for-it.sh does the same TCP-readiness-check dance while
waiting for postgres. Both confirmed recurring via netdata's alert history,
not one-off. Scoped exceptions added to each's specific binary/cmdline,
not the whole container/image.
Verified: a gitea SSH login no longer alerts, while a real dup2-based
redirect (bash's /dev/tcp exec pattern) from an unrelated container still
fires - the exception is narrow, not a blanket disable.
"Run shell untrusted" was also flagged as noisy, but investigation showed
it fired exactly once, during my own rule-testing window, and never
before or since - left alone rather than building a permanent exception
for a self-caused test artifact.
Found during live testing: netdata's cloud-provider auto-detection runs
curl --fail -s -m1 --noproxy * http://169.254.169.254 on every
startup/reconnect to check whether the host is in AWS/GCP/Azure - standard
monitoring-agent behavior, not a bug. Without this exception the rule
would have paged every time netdata restarts. Scoped to netdata's own
curl specifically so any other connection from that container still
alerts.
git-hook-tamper.yaml's condition used proc.exepath, which resolves to the
script interpreter's path (e.g. /bin/busybox) for shebang scripts, not the
script's own path - switched to proc.cmdline, which retains the originally
invoked path. Confirmed via live-testing both ways.
ssh-persistence/cloud-metadata-probe/db-spawned-process round out the
post-incident hardening pass with a few more incubating-ruleset adaptations.
Written after the 2026-08-10/11 gitea internal-API log-poisoning attack
that planted a malicious uploadpack.packObjectsHook backdoor. Catches
both the planting (unexpected exec from a hooks/ path) and the firing
(git-upload-pack/git-receive-pack spawning anything but its own
pack-objects binary), independent of how the hook config got written.
No repeat intrusion - investigated all Falco alerts since deployment
(2026-08-08 to 2026-08-16); the only crypto-miner rule hits were the
original test alerts. The daily self-resolving Telegram alerts the user
was seeing came from four known-benign sources: Obsidian's Electron
self-re-exec pattern and s6-init sudoers setup (fires ~daily on container
recreation), ttyd/Synchronet's telnet/pv/busybox-extras BBS tooling, and
ollama's nvidia-ctk ldconfig-refresh hook (memfd_create by design). Added
scoped exceptions for each via Falco's own rule extension points, same
pattern as the earlier pg_isready/gitea fixes.
Also fixed netdata's used_swap alarm, which recalculated from the
instantaneous raw sample every 10s with only a 30s notification debounce -
a brief swap spike was enough to page both Telegram and (via Netdata
Cloud) email. Widened the up-delay to 5m so only sustained swap pressure
notifies.
Verified end-to-end via a fresh fake-xmrig test after redeploying Falco
and reloading netdata: CRITICAL alarm fired and Telegram delivery
succeeded.
- 'Drop and execute new binary in container' fired on every git push/pull
to gitea over SSH (gitea's own binary re-executing itself for git-shell
hooks looks like container drift). Scoped exception via
known_drop_and_execute_activities to proc.name=gitea on that specific
image, not a blanket container whitelist, so other unexpected binaries
in that container still get caught.
- netdata alarm text was a static generic blurb requiring a manual
'docker logs falco' every time to find out what actually happened.
Now interpolates ${label:rule_name}/${label:priority} (exposed by
Falco's Prometheus metric) so the Telegram message names the specific
rule directly. Note: literal double-quotes in the info/summary text
broke netdata's alarm-notify.sh (silent delivery failure, exit 1) -
avoided.
- Added netdata/health.d and netdata/go.d to the repo for documentation;
netdata does not auto-deploy these from git, same as gitea's app.ini -
copy to /srv/netdata/config/ manually and restart the container.
Detects unauthorized miners (and other runtime threats) via Falco's
syscall/eBPF monitoring, following the 2026-08-06 gitea/xmrig
compromise. Wired into netdata rather than a new notification channel:
Falco exposes a Prometheus /metrics endpoint, netdata scrapes it, and a
new health.d alarm pages through netdata's already-configured Telegram
bot - no new alerting infra needed.
Includes a custom process-name rule for known miner binaries (the stock
Stratum-protocol rule wouldn't have caught the actual gitea incident,
which used a bare host:port with no scheme prefix), an outbound
miner-pool-port rule as a second layer, and rule_matching: all in the
Falco config - without it, Falco silently drops all but the first
matching rule per event, which would have suppressed our custom rule
whenever a stock rule also matched the same process.
poprhythm