Prometheus Alertmanager

Overview

The Alertmanager integration forwards a monitor’s status changes to a Prometheus Alertmanager instance. On each change OKStatus sends a POST to ‹your-url›/api/v2/alerts, and Alertmanager routes it to whatever receivers you’ve configured (email, Slack, PagerDuty, …).

Available on the Pro plan and above.

What OKStatus sends

One alert per status change, in the Alertmanager v2 format:

Field	Value
labels.alertname	OKStatusMonitorDown
labels.severity	critical (down) · warning (degraded) · info (up / paused)
labels.monitor	the monitor name
labels.region	the region the check ran from
labels.status	down · degraded · up · paused
labels.source	okstatus
annotations.summary	human-readable description
annotations.target	the monitored URL

• Firing alerts (down, degraded) are sent with no endsAt.
• Resolving alerts (up, paused) are sent with endsAt set to now, which closes the alert in Alertmanager.

Route on the source="okstatus" label to keep OKStatus alerts separate from the rest of your alerting.

Setup

1. In OKStatus, go to Alerting → Channels.
2. Click Add a channel → Alertmanager / Prometheus.
3. Enter the base URL of your Alertmanager — not the /api/v2/alerts path, OKStatus appends that itself.
4. (Optional) Add basic auth credentials if your Alertmanager is protected.
5. Click Test, then attach the channel to your monitors.

Self-hosted / in-cluster Alertmanager

You can point OKStatus at an Alertmanager running inside the same Kubernetes cluster. Use the Alertmanager Service DNS name, and mind these three things:

1. Use the DNS name, not the ClusterIP. For safety OKStatus refuses URLs that resolve to a literal private/reserved IP (e.g. 10.x, 192.168.x) — but an in-cluster DNS name such as *.svc.cluster.local is accepted.
2. Include the port — Alertmanager listens on 9093, not 80.
3. Enter the base URL only — OKStatus adds /api/v2/alerts.

Cross-namespace works with the fully-qualified name, for example:

http://alertmanager.monitoring.svc.cluster.local:9093

Point at a Service that has endpoints

A headless Service with no ready pods (0 endpoints) resolves to nothing, so the send fails with ENOTFOUND / “unreachable” even though the URL looks right. Confirm your target has endpoints:

kubectl get endpoints <alertmanager-service> -n <namespace>

If it lists no addresses, pick the Service that fronts the running Alertmanager pods instead.

Testing

The Test button sends a single firing OKStatusMonitorDown alert (labels above, monitor="Test Monitor", region="TEST").

You can also verify reachability from inside the cluster, from the namespace OKStatus runs in:

kubectl run amtest --rm -i --restart=Never --image=curlimages/curl \
  -n <okstatus-namespace> -- \
  curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST http://alertmanager.monitoring.svc.cluster.local:9093/api/v2/alerts \
  -H 'Content-Type: application/json' -d '[]'

A 200 means Alertmanager accepted the payload.

How it appears

• In the Alertmanager UI → Alerts, the test shows up as a firing group alertname="OKStatusMonitorDown". Because the test is sent once, Alertmanager auto-resolves it after its resolve_timeout (5 minutes by default), so it’s only briefly visible — open the UI right after testing, or filter on source="okstatus".
• In your receivers (email, Slack, …) it arrives as a normal notification and stays there — the most reliable confirmation the pipeline works.
• A real down monitor re-sends on every failed check, so its alert stays firing in Alertmanager continuously until the monitor recovers — at which point OKStatus sends the resolve and Alertmanager marks it resolved.

Troubleshooting

Symptom	Cause / fix
“unreachable” / ENOTFOUND	Wrong hostname, or the Service has 0 endpoints — check kubectl get endpoints.
“unreachable” / ECONNREFUSED	Wrong port — use :9093.
“private or reserved IP address”	You used a raw ClusterIP — use the Service DNS name instead.
200 but nothing in receivers	Alertmanager received it but no route matches — add a route on source="okstatus".
Test worked once, then nothing in the UI	Expected: the one-shot test alert auto-resolves after resolve_timeout.
Works locally but not from the cluster	A NetworkPolicy may block egress from the OKStatus namespace to Alertmanager.