Skip to content
OKStatusOKStatusDocs

Outside-in uptime and error budget in Grafana

Put OKStatus checks next to your application’s own metrics to see at a glance where an outage comes from, and alert on how fast you burn your error budget rather than on every failed check. The Prometheus & Grafana page stays the reference for the scrape, the ready-made dashboard and the metrics.

What you’ll build

Section titled “What you’ll build”
  • a panel that overlays the state seen by OKStatus probes and the error rate measured by your application;
  • three recording rules that compute the error budget burn rate;
  • two alerts: one that pages when the budget melts, one that opens a ticket when it wears down slowly.

Before you start

Section titled “Before you start”
  • The Prometheus export set up, available from the Pro plan (see Prometheus & Grafana).
  • An availability target per monitor (sloTarget, Max and Enterprise plans, see SLOs and error budget). Without one, step 3 shows how to use the organization’s SLA target instead.
  • A request metric on the application side, for example http_requests_total with a code label.

1. Pick the same service on both sides

Section titled “1. Pick the same service on both sides”

OKStatus labels every series with monitor_host: the host (and port) of the monitored service, never its path or credentials. Select your service with it, or with monitor_name:

okstatus_monitor_up{monitor_host="api.shop.example.com"}

2. The outside / inside panel

Section titled “2. The outside / inside panel”

In a Time series panel, two queries on the same time axis:

# A · seen from outside: 1 up, 0 down (majority of regions)
okstatus_monitor_up{monitor_host="api.shop.example.com"}


# B · seen from inside: share of 5xx responses
sum(rate(http_requests_total{service="api", code=~"5.."}[5m]))
  / sum(rate(http_requests_total{service="api"}[5m]))

Put B on a right axis from 0 to 1. Reading it comes down to four cases:

Seen from outsideSeen from insideWhere to look
down5xx errorsthe application or its dependencies
downnormal, or no trafficthe way in: DNS, CDN, certificate, load balancer
up5xx errorssome routes only: the check doesn’t cover the broken path
up, one region failingnormalthe network between that region and you; nobody was alerted

The last case has its own series: okstatus_region_up == 0 while okstatus_monitor_up == 1.

3. The error budget

Section titled “3. The error budget”

With a 99.9 % target over 30 days, the budget is 0.1 %: about 43 minutes of downtime. The burn rate tells how fast you spend it. At 1, it lasts exactly 30 days. At 3, it’s gone in 10 days.

okstatus-slo.yml
groups:
  - name: okstatus-slo
    rules:
      - record: okstatus:budget_burn:24h
        expr: |
          (1 - okstatus_monitor_availability_ratio{window="24h"})
            / on (monitor_id) (1 - okstatus_monitor_sla_target_ratio)
      - record: okstatus:budget_burn:7d
        expr: |
          (1 - okstatus_monitor_availability_ratio{window="7d"})
            / on (monitor_id) (1 - okstatus_monitor_sla_target_ratio)
      - record: okstatus:budget_left:30d
        expr: |
          1 - (1 - okstatus_monitor_availability_ratio{window="30d"})
                / on (monitor_id) (1 - okstatus_monitor_sla_target_ratio)

okstatus:budget_left:30d goes from 1 (budget intact) to 0 (spent), and below 0 once the target is missed. Show it in a gauge per monitor.

Without a per-monitor target, divide by the organization’s SLA target instead: replace each / on (monitor_id) (1 - okstatus_monitor_sla_target_ratio) with / on () group_left () (1 - okstatus_org_sla_target_ratio).

4. The alerts

Section titled “4. The alerts”

They read the recording rules of step 3, in a second file:

okstatus-slo-alerts.yml
groups:
  - name: okstatus-slo-alerts
    rules:
      - alert: OKStatusBudgetFastBurn
        expr: okstatus:budget_burn:24h > 3
        for: 15m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.monitor_name }}: 10 % of the monthly budget spent in a day"


      - alert: OKStatusBudgetSlowBurn
        expr: okstatus:budget_burn:7d > 1
        for: 1h
        labels:
          severity: ticket
        annotations:
          summary: "{{ $labels.monitor_name }}: at this rate the budget runs out before the month ends"

Load both files with rule_files: in prometheus.yml, or wrap each groups: in a PrometheusRule for the Prometheus Operator.

The limits

Section titled “The limits”
  • The export gives 24-hour, 7-day and 30-day windows. The classic 1-hour and 6-hour burn alerts aren’t possible with these metrics alone.
  • The exported availability is the one of the dashboard and the SLA reports, computed with the majority rule and refreshed every 5 minutes. Grafana and the PDF agree.
  • A paused monitor, or one in a maintenance window, doesn’t spend budget.

Going further

Section titled “Going further”