From c23ff52563b29fde3af08f4e23d1e0956caec402 Mon Sep 17 00:00:00 2001
From: poprhythm <james.kolpack@gmail.com>
Date: Fri, 7 Aug 2026 00:12:18 +0000
Subject: [PATCH] netdata: alert on sustained high CPU via Telegram, restrict
 LAN exposure
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Two cryptominer incursions (inbox-zero, then gitea) went unnoticed until
found manually — there was no alerting on sustained host CPU. Wired
netdata's existing 10min_cpu_usage alarm to notify via the shared Telegram
bot (config is host-only, not committed). Also narrowed the dashboard port
from 0.0.0.0 to the host's LAN IP to reduce exposure.
---
 netdata/README.md           | 96 ++++++++++++++-----------------------
 netdata/docker-compose.yaml |  6 ++-
 2 files changed, 42 insertions(+), 60 deletions(-)

diff --git a/netdata/README.md b/netdata/README.md
index 96b6ab8..9a9c15a 100644
--- a/netdata/README.md
+++ b/netdata/README.md
@@ -24,76 +24,54 @@ Comprehensive real-time monitoring for Docker host and containers with alerting
 
 Once deployed, access Netdata at: `http://your-host-ip:19999`
 
-## Webhook Notifications Setup
+## Notifications: Telegram (configured)
 
-Netdata supports Discord, Slack, and many other notification methods.
+Netdata's notification config lives at `/srv/netdata/config/health_alarm_notify.conf`
+on the host (bind-mounted into the container at `/etc/netdata/health_alarm_notify.conf`).
+This file is **host-only and not tracked in git** since it holds the bot token —
+same pattern as Authelia's config. It's populated from the shared homelab
+Telegram bot credentials in `.credentials` (`TELEGRAM_BOT_TOKEN` /
+`TELEGRAM_CHAT_ID` — same bot used by Watchtower and Uptime Kuma):
 
-### Option 1: Via Web UI (Recommended)
-1. Access Netdata at `http://your-host-ip:19999`
-2. Click on "Alert Notifications" in the left sidebar
-3. Select your notification service (Discord, Slack, etc.)
-4. Enter your webhook URL
-5. Test and save
-
-### Option 2: Configuration File
-
-Create a custom alert configuration by editing the health_alarm_notify.conf file:
-
-1. After starting the container, exec into it:
-   ```bash
-   docker exec -it netdata bash
-   ```
-
-2. Edit the notification config:
-   ```bash
-   cd /etc/netdata
-   ./edit-config health_alarm_notify.conf
-   ```
-
-#### For Discord:
 ```conf
-SEND_DISCORD="YES"
-DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/YOUR_WEBHOOK_ID/YOUR_WEBHOOK_TOKEN"
-DEFAULT_RECIPIENT_DISCORD="alarms"
+SEND_TELEGRAM="YES"
+TELEGRAM_BOT_TOKEN="<TELEGRAM_BOT_TOKEN from .credentials>"
+DEFAULT_RECIPIENT_TELEGRAM="<TELEGRAM_CHAT_ID from .credentials>"
 ```
 
-#### For Slack:
-```conf
-SEND_SLACK="YES"
-SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
-DEFAULT_RECIPIENT_SLACK="alarms"
+Netdata reads this file fresh each time it sends a notification, so no
+container restart is needed after editing it — just re-run the test below.
+
+To recreate this file (e.g. after a host rebuild), source `.credentials` and:
+```bash
+set -a; source .credentials; set +a
+cat > /srv/netdata/config/health_alarm_notify.conf <<EOF
+SEND_TELEGRAM="YES"
+TELEGRAM_BOT_TOKEN="${TELEGRAM_BOT_TOKEN}"
+DEFAULT_RECIPIENT_TELEGRAM="${TELEGRAM_CHAT_ID}"
+EOF
 ```
 
-#### For Generic Webhook:
-```conf
-SEND_CUSTOM="YES"
-CUSTOM_SENDER="${status}"
-DEFAULT_RECIPIENT_CUSTOM="alarms"
+Test the notification pipeline any time with:
+```bash
+docker exec netdata /usr/libexec/netdata/plugins.d/alarm-notify.sh test
 ```
+(An email delivery error in the output is expected/harmless — Telegram is
+the only channel actually configured.)
 
-3. Test the notification:
-   ```bash
-   /usr/libexec/netdata/plugins.d/alarm-notify.sh test
-   ```
+### Host CPU/load alerting (cryptominer detection)
 
-4. Restart the container to apply changes:
-   ```bash
-   docker restart netdata
-   ```
+Netdata ships a built-in alarm, `10min_cpu_usage` (in
+`/usr/lib/netdata/conf.d/health.d/cpu.conf`), that watches the 10-minute
+rolling average of total host CPU utilization and warns at >85%, critical at
+>95%. It's core-count-agnostic (percentage-based) so it needs no tuning per
+host. This is now wired to notify via Telegram above — it's the primary
+signal for "something is pegging the CPU" (e.g. a cryptominer), and doesn't
+require any custom `health.d` override.
 
-### Getting Webhook URLs
-
-**Discord:**
-1. Go to Server Settings → Integrations → Webhooks
-2. Click "New Webhook"
-3. Copy the webhook URL
-
-**Slack:**
-1. Go to https://api.slack.com/apps
-2. Create a new app or select existing
-3. Go to "Incoming Webhooks"
-4. Add webhook to workspace
-5. Copy the webhook URL
+Related stock alarms `load_average_1/5/15` (in `load.conf`) exist but are
+intentionally routed `to: silent` upstream — they're informational only,
+not part of the notified alert path.
 
 ## Alert Configuration
 
diff --git a/netdata/docker-compose.yaml b/netdata/docker-compose.yaml
index 78e3056..746d373 100644
--- a/netdata/docker-compose.yaml
+++ b/netdata/docker-compose.yaml
@@ -7,7 +7,11 @@ services:
     restart: unless-stopped
     hostname: ${HOSTNAME:-docker-host}
     ports:
-      - 19999:19999
+      # Bound to the host's LAN IP (not 0.0.0.0) so the dashboard stays
+      # reachable from the LAN but isn't listening on every interface.
+      # NOTE: this host's LAN IP is DHCP-assigned — if it changes, update
+      # this binding to match.
+      - 192.168.1.67:19999:19999
     cap_add:
       - SYS_PTRACE
       - SYS_ADMIN