poprhythm
c23ff52563
Two cryptominer incursions (inbox-zero, then gitea) went unnoticed until found manually — there was no alerting on sustained host CPU. Wired netdata's existing 10min_cpu_usage alarm to notify via the shared Telegram bot (config is host-only, not committed). Also narrowed the dashboard port from 0.0.0.0 to the host's LAN IP to reduce exposure.
103 lines
3.7 KiB
Markdown
103 lines
3.7 KiB
Markdown
# Netdata - Docker Host Monitoring
|
|
|
|
Comprehensive real-time monitoring for Docker host and containers with alerting capabilities.
|
|
|
|
## Features
|
|
|
|
- **Container Monitoring**: Auto-detects and monitors all Docker containers
|
|
- CPU, memory, network, disk I/O per container
|
|
- Container state monitoring (running, stopped, unhealthy)
|
|
- Pre-configured alerts for resource usage
|
|
|
|
- **Host Monitoring**: Full system metrics
|
|
- CPU, memory, disk, network usage
|
|
- Process monitoring
|
|
- System health checks
|
|
|
|
- **Built-in Alerts**: Pre-configured alarms for:
|
|
- Container CPU and memory usage
|
|
- Container health status
|
|
- System resource exhaustion
|
|
- Disk space usage
|
|
|
|
## Access
|
|
|
|
Once deployed, access Netdata at: `http://your-host-ip:19999`
|
|
|
|
## Notifications: Telegram (configured)
|
|
|
|
Netdata's notification config lives at `/srv/netdata/config/health_alarm_notify.conf`
|
|
on the host (bind-mounted into the container at `/etc/netdata/health_alarm_notify.conf`).
|
|
This file is **host-only and not tracked in git** since it holds the bot token —
|
|
same pattern as Authelia's config. It's populated from the shared homelab
|
|
Telegram bot credentials in `.credentials` (`TELEGRAM_BOT_TOKEN` /
|
|
`TELEGRAM_CHAT_ID` — same bot used by Watchtower and Uptime Kuma):
|
|
|
|
```conf
|
|
SEND_TELEGRAM="YES"
|
|
TELEGRAM_BOT_TOKEN="<TELEGRAM_BOT_TOKEN from .credentials>"
|
|
DEFAULT_RECIPIENT_TELEGRAM="<TELEGRAM_CHAT_ID from .credentials>"
|
|
```
|
|
|
|
Netdata reads this file fresh each time it sends a notification, so no
|
|
container restart is needed after editing it — just re-run the test below.
|
|
|
|
To recreate this file (e.g. after a host rebuild), source `.credentials` and:
|
|
```bash
|
|
set -a; source .credentials; set +a
|
|
cat > /srv/netdata/config/health_alarm_notify.conf <<EOF
|
|
SEND_TELEGRAM="YES"
|
|
TELEGRAM_BOT_TOKEN="${TELEGRAM_BOT_TOKEN}"
|
|
DEFAULT_RECIPIENT_TELEGRAM="${TELEGRAM_CHAT_ID}"
|
|
EOF
|
|
```
|
|
|
|
Test the notification pipeline any time with:
|
|
```bash
|
|
docker exec netdata /usr/libexec/netdata/plugins.d/alarm-notify.sh test
|
|
```
|
|
(An email delivery error in the output is expected/harmless — Telegram is
|
|
the only channel actually configured.)
|
|
|
|
### Host CPU/load alerting (cryptominer detection)
|
|
|
|
Netdata ships a built-in alarm, `10min_cpu_usage` (in
|
|
`/usr/lib/netdata/conf.d/health.d/cpu.conf`), that watches the 10-minute
|
|
rolling average of total host CPU utilization and warns at >85%, critical at
|
|
>95%. It's core-count-agnostic (percentage-based) so it needs no tuning per
|
|
host. This is now wired to notify via Telegram above — it's the primary
|
|
signal for "something is pegging the CPU" (e.g. a cryptominer), and doesn't
|
|
require any custom `health.d` override.
|
|
|
|
Related stock alarms `load_average_1/5/15` (in `load.conf`) exist but are
|
|
intentionally routed `to: silent` upstream — they're informational only,
|
|
not part of the notified alert path.
|
|
|
|
## Alert Configuration
|
|
|
|
Netdata comes with sensible defaults, but you can customize:
|
|
|
|
- **Alert thresholds**: Edit individual alarm files in `/etc/netdata/health.d/`
|
|
- **Notification frequency**: Configure repeat intervals in `health_alarm_notify.conf`
|
|
- **Alert filtering**: Set up role-based notifications
|
|
|
|
## Monitoring Capabilities
|
|
|
|
- Real-time metrics with 1-second granularity
|
|
- Automatic anomaly detection
|
|
- Zero-configuration auto-detection of containers
|
|
- Historical data retention
|
|
- Beautiful web dashboard with interactive charts
|
|
|
|
## Resources
|
|
|
|
- Official docs: https://learn.netdata.cloud/
|
|
- Alert configuration: https://learn.netdata.cloud/docs/alerting/
|
|
- Notification setup: https://learn.netdata.cloud/docs/alerting/notifications/
|
|
|
|
## Notes
|
|
|
|
- The container requires privileged access and specific capabilities to monitor the host system
|
|
- Docker socket is mounted read-only for security
|
|
- Data persists across container restarts via Docker volumes
|