Prometheus Alertmanager

Overview

The Alertmanager integration forwards a monitor’s status changes to a Prometheus Alertmanager instance. On each change OKStatus sends a POST to ‹your-url›/api/v2/alerts, and Alertmanager routes it to whatever receivers you’ve configured (email, Slack, PagerDuty, …).

Available on the Pro plan and above.

What OKStatus sends

One alert per status change, in the Alertmanager v2 format:

Field	Value
labels.alertname	OKStatusMonitorDown
labels.severity	critical (down) · warning (degraded)
labels.monitor	the monitor name
labels.instance	the monitored URL
labels.source	okstatus
annotations.summary	human-readable description
annotations.status	down · degraded · up · paused
annotations.region	the region that reported the change
annotations.target	the monitored URL

The labels name the monitor and nothing else, so the recovery closes the very alert the outage opened.

• Down fires the critical alert (and closes the warning one if the monitor was degraded first); degraded fires the warning alert. A firing alert carries an endsAt 7 days ahead: OKStatus sends once per status change, and without it Alertmanager would resolve the alert after its resolve_timeout (5 min) while the monitor is still down.
• Up and paused close both alerts (endsAt set to now).

Route on the source="okstatus" label to keep OKStatus alerts separate from the rest of your alerting.

Setup

1. In OKStatus, go to Alerting → Channels.
2. Click Add a channel → Alertmanager / Prometheus.
3. Enter the base URL of your Alertmanager — not the /api/v2/alerts path, OKStatus appends that itself.
4. (Optional) Add basic auth credentials if your Alertmanager is protected.
5. Click Test, then attach the channel to your monitors.

Self-hosted / in-cluster Alertmanager

You can point OKStatus at an Alertmanager running inside the same Kubernetes cluster. Use the Alertmanager Service DNS name, and mind these three things:

1. Use the DNS name, not the ClusterIP. For safety OKStatus refuses URLs that resolve to a literal private/reserved IP (e.g. 10.x, 192.168.x) — but an in-cluster DNS name such as *.svc.cluster.local is accepted.
2. Include the port — Alertmanager listens on 9093, not 80.
3. Enter the base URL only — OKStatus adds /api/v2/alerts.

Cross-namespace works with the fully-qualified name, for example:

http://alertmanager.monitoring.svc.cluster.local:9093

Point at a Service that has endpoints

A headless Service with no ready pods (0 endpoints) resolves to nothing, so the send fails with ENOTFOUND / “unreachable” even though the URL looks right. Confirm your target has endpoints:

kubectl get endpoints <alertmanager-service> -n <namespace>

If it lists no addresses, pick the Service that fronts the running Alertmanager pods instead.

Testing

The Test button sends a single firing OKStatusMonitorDown alert (labels above, monitor="Test Monitor", instance="https://example.com"), which expires after 5 minutes.

You can also verify reachability from inside the cluster, from the namespace OKStatus runs in:

kubectl run amtest --rm -i --restart=Never --image=curlimages/curl \
  -n <okstatus-namespace> -- \
  curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST http://alertmanager.monitoring.svc.cluster.local:9093/api/v2/alerts \
  -H 'Content-Type: application/json' -d '[]'

A 200 means Alertmanager accepted the payload.

How it appears

• In the Alertmanager UI → Alerts, the test shows up as a firing group alertname="OKStatusMonitorDown". It resolves itself after 5 minutes, so it’s only briefly visible — open the UI right after testing, or filter on source="okstatus".
• In your receivers (email, Slack, …) it arrives as a normal notification and stays there — the most reliable confirmation the pipeline works.
• A real outage stays firing in Alertmanager until the monitor recovers — at which point OKStatus sends the resolve and Alertmanager marks it resolved.

Troubleshooting

Symptom	Cause / fix
“unreachable” / ENOTFOUND	Wrong hostname, or the Service has 0 endpoints — check kubectl get endpoints.
“unreachable” / ECONNREFUSED	Wrong port — use :9093.
“private or reserved IP address”	You used a raw ClusterIP — use the Service DNS name instead.
200 but nothing in receivers	Alertmanager received it but no route matches — add a route on source="okstatus".
Test worked once, then nothing in the UI	Expected: the test alert resolves itself after 5 minutes.
Works locally but not from the cluster	A NetworkPolicy may block egress from the OKStatus namespace to Alertmanager.