Homelab monitoring, Part 7: An ISP report card with outages, ping and speed tests

pfSense tells you how much data went through the line. But it does not tell you if the line was down at 3 a.m., if the ping is bad every evening, or if you really get the speed you pay for. This part adds two small containers that answer these questions. They fill the ISP report card section of the dashboard from Part 6:

Two alerts come with it: Internet down and Internet slower than contract, sent to your phone like the alerts from Part 4.

How it works:

• blackbox_exporter pings Cloudflare (1.1.1.1), Google (8.8.8.8) and Quad9 (9.9.9.9) every 30 seconds, and pfSense too. The internet only counts as down when all three fail. If one provider has a bad minute, that is not a problem with your line.
• A speed-test exporter runs the official Ookla Speedtest CLI at fixed hours of the day and keeps the last result. Prometheus only reads the saved result, so it never starts a test itself.

Data use. A speed test downloads and uploads for about 10 seconds each, at full speed. On a 250 Mbit/s line that is about 400 MB per test. Three tests a day use about 35 GB a month. On an unlimited line that does not matter. If your data is limited, choose fewer hours in Your values. The tests also appear as short spikes in the WAN graphs of Part 6. This is a good way to check that both agree.

Step 1: The files for the two containers

From the Proxmox host shell, go into the monitoring container:

pct enter 140

Inside the container, create the ping settings and download the speed-test exporter. The exporter is a short Python script. If you want to read it first, open it in the browser.

cd /opt/monitoring
mkdir -p blackbox speedtest

cat > blackbox/blackbox.yml <<'EOF'
modules:
  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: false
EOF

curl -fsSL -o speedtest/Dockerfile https://www.zadran.net/downloads/grafana/speedtest/Dockerfile
curl -fsSL -o speedtest/speedtest_exporter.py https://www.zadran.net/downloads/grafana/speedtest/speedtest_exporter.py
ls -l blackbox speedtest

ls must list three files, none of them 0 bytes.

Step 2: Add the containers

The two services go into compose.override.yaml from Part 2, below the existing ones. The first line only creates the file if you skipped Part 2.

cd /opt/monitoring
mkdir -p backup
cp compose.override.yaml backup/compose.override.yaml.$(date +%Y%m%d-%H%M%S) 2>/dev/null
[ -f compose.override.yaml ] || echo "services:" > compose.override.yaml
cat >> compose.override.yaml <<'EOF'

  blackbox:
    image: prom/blackbox-exporter:latest
    container_name: blackbox
    restart: unless-stopped
    user: root
    cap_add:
      - NET_RAW
    command:
      - --config.file=/etc/blackbox/blackbox.yml
    volumes:
      - ./blackbox/blackbox.yml:/etc/blackbox/blackbox.yml:ro

  speedtest:
    build: ./speedtest
    image: local/speedtest-exporter
    container_name: speedtest
    restart: unless-stopped
    environment:
      - TZ=Europe/Berlin
      - SPEEDTEST_HOURS=8,14,20
EOF
docker compose config -q && docker compose up -d --build blackbox speedtest

Building the speed-test image takes about a minute. docker compose config -q prints nothing when the file is correct. If it shows an error, the spaces at the start of the pasted lines are wrong. There must be two spaces before blackbox: and speedtest:.

Step 3: The Prometheus jobs

P=/opt/monitoring/prometheus/prometheus.yml
cp $P /opt/monitoring/backup/prometheus.yml.$(date +%Y%m%d-%H%M%S)
cat >> $P <<'EOF'

  - job_name: isp-ping
    metrics_path: /probe
    params:
      module: [icmp]
    static_configs:
      - targets: ['1.1.1.1', '8.8.8.8', '9.9.9.9']
        labels:
          probe: internet
      - targets: ['192.168.1.1']
        labels:
          probe: pfsense
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115

  - job_name: speedtest
    scrape_interval: 60s
    static_configs:
      - targets: ['speedtest:9798']
        labels:
          host: speedtest
EOF
docker exec prometheus promtool check config /etc/prometheus/prometheus.yml && curl -fsS -X POST localhost:9090/-/reload && echo reloaded

The relabel_configs turn each address in the list into a request to blackbox: "ping 1.1.1.1 for me". The ping to pfSense is for comparison. If the internet is down but pfSense still answers, the problem is outside your house.

Wait one minute, then check both:

curl -s -G localhost:9090/api/v1/query --data-urlencode 'query=probe_success{job="isp-ping"}' | grep -o '"instance":"[^"]*"\|"value":\[[^]]*\]' | paste - -
docker logs speedtest 2>&1 | tail -2

Expected: four targets ending in "1", and a test line with your speeds.

"instance":"1.1.1.1"	"value":[1791302400.123,"1"]
"instance":"8.8.8.8"	"value":[1791302400.123,"1"]
"instance":"9.9.9.9"	"value":[1791302400.123,"1"]
"instance":"192.168.1.1"	"value":[1791302400.123,"1"]
speedtest exporter on :9798, tests at 8,14,20:00 (Europe/Berlin)
2026-10-06 15:42:11 ok: 243 / 48 Mbit/s, ping 11.2 ms, server Example ISP - Frankfurt

If the second line is missing, the first test is still running. Wait half a minute and run docker logs speedtest again.

Step 4: The alerts

The alert file uses the contact point and notification policy from Part 4. This command downloads it and writes your contract speed into it:

D=250
curl -fsSL https://www.zadran.net/downloads/grafana/alerting/isp.yaml | sed -e "s/777 Mbit/$D Mbit/g; s/388\.5\]/$((D/2))]/g" > /opt/monitoring/grafana/provisioning/alerting/isp.yaml
grep -n 'Mbit\|params: \[' /opt/monitoring/grafana/provisioning/alerting/isp.yaml
cd /opt/monitoring && docker compose restart grafana

grep shows your contract in the message and half of it as the threshold.

Step 5: Read the report card

Open Network → Internet & pfSense. The ISP report card row now fills. The first speed test appears right away. Availability and outages build up over the month.

• Availability this month counts from the 1st, or from the day you started. 99.9 % is about 43 minutes offline in a month.
• Outages are counted one by one, so ten short drops of one minute show as 10, not as one long break.
• In Speed tests (last 30 days) each step is one test. If the speed is low only in the evening, the provider's network is busy. If it is low all the time, the problem is the line or the modem.

Speeds above about 900 Mbit/s? The test runs inside the monitoring container, so it cannot be faster than the network port of the Proxmox host. On a 1 Gbit port, a 1 Gbit contract measures about 900–940 Mbit/s.

Troubleshooting

The Ookla CLI is free for personal, non-commercial use. The image accepts its licence for you when it runs.

In Part 8: backups. A dashboard and an alert tell you when a VM has no recent backup, before you need it.