poprhythm
c23ff52563
Two cryptominer incursions (inbox-zero, then gitea) went unnoticed until found manually — there was no alerting on sustained host CPU. Wired netdata's existing 10min_cpu_usage alarm to notify via the shared Telegram bot (config is host-only, not committed). Also narrowed the dashboard port from 0.0.0.0 to the host's LAN IP to reduce exposure.
3.7 KiB
Netdata - Docker Host Monitoring
Comprehensive real-time monitoring for Docker host and containers with alerting capabilities.
Features
-
Container Monitoring: Auto-detects and monitors all Docker containers
- CPU, memory, network, disk I/O per container
- Container state monitoring (running, stopped, unhealthy)
- Pre-configured alerts for resource usage
-
Host Monitoring: Full system metrics
- CPU, memory, disk, network usage
- Process monitoring
- System health checks
-
Built-in Alerts: Pre-configured alarms for:
- Container CPU and memory usage
- Container health status
- System resource exhaustion
- Disk space usage
Access
Once deployed, access Netdata at: http://your-host-ip:19999
Notifications: Telegram (configured)
Netdata's notification config lives at /srv/netdata/config/health_alarm_notify.conf
on the host (bind-mounted into the container at /etc/netdata/health_alarm_notify.conf).
This file is host-only and not tracked in git since it holds the bot token —
same pattern as Authelia's config. It's populated from the shared homelab
Telegram bot credentials in .credentials (TELEGRAM_BOT_TOKEN /
TELEGRAM_CHAT_ID — same bot used by Watchtower and Uptime Kuma):
SEND_TELEGRAM="YES"
TELEGRAM_BOT_TOKEN="<TELEGRAM_BOT_TOKEN from .credentials>"
DEFAULT_RECIPIENT_TELEGRAM="<TELEGRAM_CHAT_ID from .credentials>"
Netdata reads this file fresh each time it sends a notification, so no container restart is needed after editing it — just re-run the test below.
To recreate this file (e.g. after a host rebuild), source .credentials and:
set -a; source .credentials; set +a
cat > /srv/netdata/config/health_alarm_notify.conf <<EOF
SEND_TELEGRAM="YES"
TELEGRAM_BOT_TOKEN="${TELEGRAM_BOT_TOKEN}"
DEFAULT_RECIPIENT_TELEGRAM="${TELEGRAM_CHAT_ID}"
EOF
Test the notification pipeline any time with:
docker exec netdata /usr/libexec/netdata/plugins.d/alarm-notify.sh test
(An email delivery error in the output is expected/harmless — Telegram is the only channel actually configured.)
Host CPU/load alerting (cryptominer detection)
Netdata ships a built-in alarm, 10min_cpu_usage (in
/usr/lib/netdata/conf.d/health.d/cpu.conf), that watches the 10-minute
rolling average of total host CPU utilization and warns at >85%, critical at
95%. It's core-count-agnostic (percentage-based) so it needs no tuning per host. This is now wired to notify via Telegram above — it's the primary signal for "something is pegging the CPU" (e.g. a cryptominer), and doesn't require any custom
health.doverride.
Related stock alarms load_average_1/5/15 (in load.conf) exist but are
intentionally routed to: silent upstream — they're informational only,
not part of the notified alert path.
Alert Configuration
Netdata comes with sensible defaults, but you can customize:
- Alert thresholds: Edit individual alarm files in
/etc/netdata/health.d/ - Notification frequency: Configure repeat intervals in
health_alarm_notify.conf - Alert filtering: Set up role-based notifications
Monitoring Capabilities
- Real-time metrics with 1-second granularity
- Automatic anomaly detection
- Zero-configuration auto-detection of containers
- Historical data retention
- Beautiful web dashboard with interactive charts
Resources
- Official docs: https://learn.netdata.cloud/
- Alert configuration: https://learn.netdata.cloud/docs/alerting/
- Notification setup: https://learn.netdata.cloud/docs/alerting/notifications/
Notes
- The container requires privileged access and specific capabilities to monitor the host system
- Docker socket is mounted read-only for security
- Data persists across container restarts via Docker volumes