Escalation, Scheduled Actions and Auto-Fix: Recipes

Basic monitoring sends one kind of alert to one place. That works until an outage lasts longer than the person who received the alert can handle, or until the same chores (warming caches, triggering backups, pinging an internal service) keep getting forgotten. This article walks through three Uptime Monitor features that close those gaps: escalation by duration, scheduled actions, and auto-fix rules scoped to failure causes. Each section ends with concrete recipes you can copy.

How the three features fit together

A useful mental model: normal alerts tell the first responder, auto-fix tries the obvious fix while they are reaching for a laptop, and escalation pulls in more people if neither worked. Scheduled actions run on their own clock, outage or not.

Escalation by duration

Escalation answers the question "what if nobody fixes this?" You set a number of minutes and a webhook URL that is separate from your normal notification channels. Escalation rides on the normal alert cycle: the first down alert (or reminder) that goes out after the site has been down that long also sends a POST to that URL. The body looks like this:

{
  "event": "escalation",
  "site": {"id": 42, "url": "https://shop.example.com", "is_active": true},
  "downtime_minutes": 20,
  "escalation_after_minutes": 15,
  "notification": {
    "subject": "Escalation: https://shop.example.com",
    "message": "Website has been down for 20 minutes (escalation threshold: 15 min).",
    "severity": "critical",
    "timestamp": "2026-10-22T10:00:00+00:00"
  },
  "meta": {
    "status_code": 502,
    "response_time_ms": 312,
    "cause_code": "server_5xx",
    "cause_summary": "Server error (5xx)"
  }
}

Because the escalation URL is separate, you can send it somewhere different from day-to-day alerts: a pager, a team lead's channel or a ticketing system.

Choosing the threshold

Escalation should fire after the first responder has had a fair chance and any auto-fix has finished its retries. If your auto-fix fires at 3 minutes and retries twice five minutes apart, it is done by about minute 13; an escalation at 15 or 20 minutes fits behind it. For a business-hours internal tool, 30 to 60 minutes may be fine. For checkout or login, shorter is better. Whatever you pick, make sure the check interval is short enough to measure it: with a 10-minute interval, "down for 15 minutes" is only two data points. Keep the site's notification interval short as well, because escalation goes out with the next reminder after the threshold: with a 60-minute notification interval, a 15-minute escalation would not fire until about an hour into the outage.

Recipe: relay escalations to PagerDuty or another incident tool

Most incident tools expect their own JSON format, so a generic payload needs a small translator. This Python relay uses only the standard library: it receives the escalation and forwards it to the PagerDuty Events API v2. The dedup_key uses the site id, so repeated escalations for the same outage update one incident instead of opening several.

#!/usr/bin/env python3
import json, urllib.request
from http.server import BaseHTTPRequestHandler, HTTPServer

ROUTING_KEY = "your-pagerduty-integration-key"

class Relay(BaseHTTPRequestHandler):
    def do_POST(self):
        body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        if body.get("event") != "escalation":
            self.send_response(204); self.end_headers(); return
        event = {
            "routing_key": ROUTING_KEY,
            "event_action": "trigger",
            "dedup_key": "uptime-site-%s" % body["site"]["id"],
            "payload": {
                "summary": body["notification"]["message"],
                "source": body["site"]["url"],
                "severity": "critical",
                "custom_details": body.get("meta", {}),
            },
        }
        req = urllib.request.Request(
            "https://events.pagerduty.com/v2/enqueue",
            data=json.dumps(event).encode(),
            headers={"Content-Type": "application/json"},
        )
        urllib.request.urlopen(req, timeout=10)
        self.send_response(200); self.end_headers()

HTTPServer(("127.0.0.1", 8081), Relay).serve_forever()

Run it behind your existing nginx with HTTPS and put a long random path in the URL (for example /hooks/esc-8f3c2a91d7) so that strangers cannot fire fake incidents. The same shape works for Opsgenie, a Jira issue or a ticket in your help desk: read the fields you need, map them, forward.

Recipe: a louder chat channel

Slack and Discord incoming webhooks do not accept arbitrary JSON: Slack needs a text field, Discord a content field. So you cannot paste a Slack webhook URL into the escalation field directly. Add a handler to the relay above that posts {"text": "ESCALATION: " + message} to a dedicated #incidents channel webhook, separate from the noisier channel where routine alerts go. The guide to webhook alerts covers payload handling and security for receivers in more depth.

Scheduled actions

A scheduled action runs a webhook call or an SSH command daily at a set time, or weekly on a chosen day. Webhooks are sent as a POST with a small JSON body:

{
  "event": "scheduled",
  "site": {"id": 42, "url": "https://shop.example.com"},
  "action": {"id": 7, "name": "Nightly cache warm"},
  "timestamp": "2026-10-22T03:00:00+00:00"
}

Your receiver can switch on action.name or action.id to decide what to do.

When to use a scheduled action instead of cron

Cron on the server is simpler and has no network dependency, so keep using it for things that must run even when the outside world cannot reach the box (logrotate, local backups, certificate renewal timers). Scheduled actions are useful when:

• You are on hosting where you can run commands over SSH but cannot edit the crontab.
• You want a job to run from outside the server, so it fails visibly if the server is unreachable instead of silently not running.
• You want recurring jobs for several sites defined in one place, next to the monitoring for those sites.

Recipe: daily cache warmer (SSH)

After a nightly cache flush or deploy, the first visitors pay for cold caches. Warm the most important URLs from the sitemap at a quiet hour:

curl -s https://example.com/sitemap.xml \
  | grep -oP '(?<=<loc>)[^<]+' \
  | head -n 200 \
  | xargs -n 1 -P 4 curl -s -o /dev/null --max-time 20

This fetches up to 200 URLs, four at a time. Keep the parallelism low so the warmer does not become the traffic spike that slows the site down. If you use the restricted-key setup from the SSH auto-recovery guide, put this in a script and add a warm-cache verb to the dispatcher instead of allowing raw commands.

Recipe: weekly backup trigger (webhook)

Many backup services and CI systems can start a job from an HTTP call. Point a weekly scheduled webhook, for example Sunday at 02:00, at the endpoint that starts an off-site backup or a database dump pipeline. Have that job report its own success somewhere you look (a chat message with the backup size is enough), because a trigger that fires is not the same as a backup that worked.

Recipe: a synthetic deep check (webhook)

An HTTP check proves the homepage renders. It does not prove the password-reset email goes out or the queue drains. Build an internal endpoint that runs those checks (send a test email to a mailbox you control, push a job and wait for it, read a row from the replica) and have a daily scheduled webhook call it. The endpoint should report failures through your own alerting. Keep it out of the public URL space or protect it with a secret path.

Recipe: weekly housekeeping (SSH)

Some applications ship cleanup commands that nobody remembers to run: pruning expired sessions, failed-job tables or temporary uploads. A weekly SSH action on a quiet night is a reasonable home for them, for example php artisan queue:prune-failed --hours=168 in a Laravel app. Make the command idempotent and fast, and remember that anything destructive belongs in code review, not in a field in a monitoring dashboard.

Smarter auto-fix: one action per cause

An automated action can be limited to specific cause codes. That lets you stack several narrow actions on one site instead of one blunt one. A practical set for a PHP site on Let's Encrypt:

Leave dns_error, client_4xx, redirect problems and maintenance without actions; they need a person. The reasoning behind each row is in mapping failure causes to targeted fixes.

Plan limits apply: Starter includes 1 automated action per site, Plus 5 and Professional 10. If you have only one, make it the app restart limited to server_5xx and timeout, because that covers the most common outages that a restart can actually fix.

A complete timeline

Here is how the pieces play out for a site checked every minute with a fail threshold of 2, a 5-minute notification interval, the action set above and escalation at 15 minutes, when PHP-FPM hangs at 10:00:

1. 10:00–10:01 Two failed checks with timeout. The fail threshold is reached and the normal alert goes to Telegram and email.
2. 10:03 The restart action fires. If the hang was a one-off, the next check passes and a recovery notification goes out.
3. 10:08, 10:13 If the site is still down, the action retries, then stops.
4. 10:15–10:20 The first reminder after the 15-minute mark goes out, and the escalation webhook fires with it; the relay opens a PagerDuty incident and posts to #incidents.
5. Throughout, reminders repeat at the site's notification interval, and screenshots taken on the down and up transitions show what visitors saw.

Checklist

• Escalation threshold set later than the last auto-fix retry.
• Escalation URL goes through a relay that translates the payload and deduplicates by site id.
• Scheduled actions used for jobs that benefit from running from outside the server; cron kept for local essentials.
• Every scheduled job reports its own success or failure.
• Auto-fix actions limited to the causes they can fix; DNS, WAF and redirect problems notify only.

Escalation, scheduled actions and cause-scoped auto-fix are all configured per site in Uptime Monitor, alongside the normal email, Telegram, Slack, Discord and webhook alerts. You can start on the free plan with one automated action and move to Plus or Professional when you need more; the pricing page lists the limits for each plan.