netdata: alert on sustained high CPU via Telegram, restrict LAN exposure
Two cryptominer incursions (inbox-zero, then gitea) went unnoticed until found manually — there was no alerting on sustained host CPU. Wired netdata's existing 10min_cpu_usage alarm to notify via the shared Telegram bot (config is host-only, not committed). Also narrowed the dashboard port from 0.0.0.0 to the host's LAN IP to reduce exposure.
This commit is contained in:
+37
-59
@@ -24,76 +24,54 @@ Comprehensive real-time monitoring for Docker host and containers with alerting
|
||||
|
||||
Once deployed, access Netdata at: `http://your-host-ip:19999`
|
||||
|
||||
## Webhook Notifications Setup
|
||||
## Notifications: Telegram (configured)
|
||||
|
||||
Netdata supports Discord, Slack, and many other notification methods.
|
||||
Netdata's notification config lives at `/srv/netdata/config/health_alarm_notify.conf`
|
||||
on the host (bind-mounted into the container at `/etc/netdata/health_alarm_notify.conf`).
|
||||
This file is **host-only and not tracked in git** since it holds the bot token —
|
||||
same pattern as Authelia's config. It's populated from the shared homelab
|
||||
Telegram bot credentials in `.credentials` (`TELEGRAM_BOT_TOKEN` /
|
||||
`TELEGRAM_CHAT_ID` — same bot used by Watchtower and Uptime Kuma):
|
||||
|
||||
### Option 1: Via Web UI (Recommended)
|
||||
1. Access Netdata at `http://your-host-ip:19999`
|
||||
2. Click on "Alert Notifications" in the left sidebar
|
||||
3. Select your notification service (Discord, Slack, etc.)
|
||||
4. Enter your webhook URL
|
||||
5. Test and save
|
||||
|
||||
### Option 2: Configuration File
|
||||
|
||||
Create a custom alert configuration by editing the health_alarm_notify.conf file:
|
||||
|
||||
1. After starting the container, exec into it:
|
||||
```bash
|
||||
docker exec -it netdata bash
|
||||
```
|
||||
|
||||
2. Edit the notification config:
|
||||
```bash
|
||||
cd /etc/netdata
|
||||
./edit-config health_alarm_notify.conf
|
||||
```
|
||||
|
||||
#### For Discord:
|
||||
```conf
|
||||
SEND_DISCORD="YES"
|
||||
DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/YOUR_WEBHOOK_ID/YOUR_WEBHOOK_TOKEN"
|
||||
DEFAULT_RECIPIENT_DISCORD="alarms"
|
||||
SEND_TELEGRAM="YES"
|
||||
TELEGRAM_BOT_TOKEN="<TELEGRAM_BOT_TOKEN from .credentials>"
|
||||
DEFAULT_RECIPIENT_TELEGRAM="<TELEGRAM_CHAT_ID from .credentials>"
|
||||
```
|
||||
|
||||
#### For Slack:
|
||||
```conf
|
||||
SEND_SLACK="YES"
|
||||
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
|
||||
DEFAULT_RECIPIENT_SLACK="alarms"
|
||||
Netdata reads this file fresh each time it sends a notification, so no
|
||||
container restart is needed after editing it — just re-run the test below.
|
||||
|
||||
To recreate this file (e.g. after a host rebuild), source `.credentials` and:
|
||||
```bash
|
||||
set -a; source .credentials; set +a
|
||||
cat > /srv/netdata/config/health_alarm_notify.conf <<EOF
|
||||
SEND_TELEGRAM="YES"
|
||||
TELEGRAM_BOT_TOKEN="${TELEGRAM_BOT_TOKEN}"
|
||||
DEFAULT_RECIPIENT_TELEGRAM="${TELEGRAM_CHAT_ID}"
|
||||
EOF
|
||||
```
|
||||
|
||||
#### For Generic Webhook:
|
||||
```conf
|
||||
SEND_CUSTOM="YES"
|
||||
CUSTOM_SENDER="${status}"
|
||||
DEFAULT_RECIPIENT_CUSTOM="alarms"
|
||||
Test the notification pipeline any time with:
|
||||
```bash
|
||||
docker exec netdata /usr/libexec/netdata/plugins.d/alarm-notify.sh test
|
||||
```
|
||||
(An email delivery error in the output is expected/harmless — Telegram is
|
||||
the only channel actually configured.)
|
||||
|
||||
3. Test the notification:
|
||||
```bash
|
||||
/usr/libexec/netdata/plugins.d/alarm-notify.sh test
|
||||
```
|
||||
### Host CPU/load alerting (cryptominer detection)
|
||||
|
||||
4. Restart the container to apply changes:
|
||||
```bash
|
||||
docker restart netdata
|
||||
```
|
||||
Netdata ships a built-in alarm, `10min_cpu_usage` (in
|
||||
`/usr/lib/netdata/conf.d/health.d/cpu.conf`), that watches the 10-minute
|
||||
rolling average of total host CPU utilization and warns at >85%, critical at
|
||||
>95%. It's core-count-agnostic (percentage-based) so it needs no tuning per
|
||||
host. This is now wired to notify via Telegram above — it's the primary
|
||||
signal for "something is pegging the CPU" (e.g. a cryptominer), and doesn't
|
||||
require any custom `health.d` override.
|
||||
|
||||
### Getting Webhook URLs
|
||||
|
||||
**Discord:**
|
||||
1. Go to Server Settings → Integrations → Webhooks
|
||||
2. Click "New Webhook"
|
||||
3. Copy the webhook URL
|
||||
|
||||
**Slack:**
|
||||
1. Go to https://api.slack.com/apps
|
||||
2. Create a new app or select existing
|
||||
3. Go to "Incoming Webhooks"
|
||||
4. Add webhook to workspace
|
||||
5. Copy the webhook URL
|
||||
Related stock alarms `load_average_1/5/15` (in `load.conf`) exist but are
|
||||
intentionally routed `to: silent` upstream — they're informational only,
|
||||
not part of the notified alert path.
|
||||
|
||||
## Alert Configuration
|
||||
|
||||
|
||||
@@ -7,7 +7,11 @@ services:
|
||||
restart: unless-stopped
|
||||
hostname: ${HOSTNAME:-docker-host}
|
||||
ports:
|
||||
- 19999:19999
|
||||
# Bound to the host's LAN IP (not 0.0.0.0) so the dashboard stays
|
||||
# reachable from the LAN but isn't listening on every interface.
|
||||
# NOTE: this host's LAN IP is DHCP-assigned — if it changes, update
|
||||
# this binding to match.
|
||||
- 192.168.1.67:19999:19999
|
||||
cap_add:
|
||||
- SYS_PTRACE
|
||||
- SYS_ADMIN
|
||||
|
||||
Reference in New Issue
Block a user