Self-Healing Outages: Match Each Failure to the Right Fix

An automated fix that runs on every outage is a blunt instrument: restarting PHP-FPM does nothing for a DNS outage, and restarting the web server during an expired-certificate incident just adds a second outage on top of the first. Good self-healing starts from why a check failed and picks a response that matches. This article gives you a decision table from failure cause to automated action, the guardrails that keep automation from making things worse, and the warning signs that self-healing has started hiding a real bug.

Start from the cause, not from the outage

Every failed check in Uptime Monitor is tagged with a cause code, such as server_5xx, timeout or dns_error. Those codes split outages into groups that need very different responses:

• Your process is stuck or dead (most server_5xx, many timeout and connect_error). A restart usually helps, at least for now.
• Your last change broke something (keyword_missing, not_found_404 right after a deploy). Restarting is pointless; rolling back helps.
• A dependency outside the server is broken (dns_error, many ssl_error, network-level connect_error). Nothing you run on the box will fix it, and some actions make diagnosis harder.
• The site is deliberately unavailable or answering as configured (maintenance, client_4xx from a WAF, redirect_3xx). These need a human to change configuration, not an automated action.

If you are not sure what each code means in practice, the field guide to diagnosing why a site is down goes through them one by one.

The decision table

This is a sensible default for a typical PHP or Node application behind nginx. Adapt the commands to your stack, but keep the shape: act only when the cause makes the action likely to help.

Recipes for the actions worth automating

Restart the app tier on 5xx and timeouts

This is the bread-and-butter action. A PHP-FPM pool that has hit pm.max_children with every worker stuck on a slow external call will keep timing out until those workers die; a restart clears them in seconds. The command your auto-fix runs should be a single, narrow, pre-approved operation, ideally through a restricted SSH key and a one-line sudo rule, as described in the safe SSH auto-recovery setup guide:

sudo -n /usr/bin/systemctl restart php8.3-fpm

Before the restart, have the script capture what you will need later, because the restart destroys it:

{
  date -Is
  systemctl status php8.3-fpm --no-pager | head -20
  tail -n 50 /var/log/php8.3-fpm.log
} >> /var/lib/autofix/php-before-restart.log 2>&1

If the log shows server reached pm.max_children setting every time, you have a capacity or slow-dependency problem, not a crash.

Renew and reload on SSL errors

Most certificate outages on Let's Encrypt setups are a renewal that silently failed weeks earlier. One action can retry it:

sudo -n /usr/bin/certbot renew --quiet --deploy-hook "systemctl reload nginx"

certbot renew only renews certificates close to expiry and the deploy hook runs only when something was actually renewed, so the command is safe to repeat. It will not fix a broken chain or a vhost pointing at the wrong certificate. If it fails, the reason (a closed port 80, a changed DNS record, a webroot that moved) is in /var/log/letsencrypt/letsencrypt.log. Better still, enable SSL expiry alerts so you hear about it days before visitors do.

Roll back on keyword_missing via your CI pipeline

When the keyword check fails, the server is answering, so the problem is almost always what it is answering with. If your pipeline can redeploy the previous release, let the automated action call it. Automated actions run SSH commands, so put the call in a small script on your server and keep the trigger token there rather than in the monitoring settings. GitLab, for example, can start a pipeline from its trigger API:

#!/bin/sh
# /usr/local/bin/trigger-rollback
curl -fsS -X POST \
  -F token="$(cat /etc/rollback/gitlab-trigger-token)" \
  -F ref=main \
  -F "variables[ROLLBACK]=true" \
  https://gitlab.com/api/v4/projects/12345/trigger/pipeline

The rollback job must protect itself. Have it check when the current release was deployed and do nothing if that was more than, say, 30 minutes ago. A keyword that vanishes hours after a deploy is more likely an empty database table or a CMS edit than bad code, and rolling back a good release just adds confusion.

Guardrails: when, how often, how many times

A good action fires late enough to skip blips, early enough to beat a human, and never in a loop. Uptime Monitor's auto-fix has a downtime threshold (how long the site must be down before the action runs), a maximum number of attempts per outage and a delay between attempts, and it can be limited to specific causes. A worked example for a site checked every minute:

With these values the worst case is three restarts over roughly 13 minutes, after which the action stops and humans take over. The rules behind the numbers:

• The action threshold should be longer than your normal restart or deploy time. Otherwise the action fires during every deploy and restarts the service halfway through.
• The retry delay must be longer than the time the service needs to come back plus one or two check intervals, or you restart a service that was already recovering.
• Cap attempts. If two restarts did not help, the third will not either. Stop and escalate.
• One cause, one action. Two actions that both restart things on server_5xx will fight each other.
• Escalation still runs. Configure an escalation webhook for, say, 15 minutes of downtime so that a failed self-heal pages a human instead of failing quietly. Escalation and scheduled actions covers the setup.

When self-healing hides real bugs

Automatic restarts are a painkiller. They make the symptom go away while the cause keeps growing. Watch for these signs:

• The same action fires on a schedule. A restart every night around the same time usually means a memory leak, a cron job that starves the server, or a log that fills a disk. Count actions per week from your action log (journalctl -t autofix if you use the logging pattern from the SSH guide) and treat any repeat as a bug ticket.
• Restarts get more frequent. Traffic growth or data growth is hitting a capacity limit. Restarting more often does not add capacity.
• Short outages vanish from your reports. If the action fixes things within the fail threshold, nobody gets alerted. Uptime Monitor's unstable-site badge (several brief failures within a time window) is the place where these show up; the article on detecting flapping websites explains how to read it.
• The restart works but users still complain. A restart drops in-flight requests, sessions held in memory and warm caches. If it happens daily, users notice even when the monitor shows green.

A useful habit: every automated fix should create a note somewhere a human reads, such as a ticket, a chat message or a log line in a weekly review. Self-healing that nobody reviews turns into a hidden dependency.

FAQ

Should I auto-restart on every outage to be safe?

No. For DNS, certificate, WAF and redirect problems a restart cannot help and can add a second outage. Limit restart actions to the causes where the process itself is the likely problem.

Is it safe to auto-reboot a server?

Rarely. A reboot is a guaranteed multi-minute outage, and if the machine does not come back cleanly no one is there to see it. Restart the specific service instead.

How many automated actions do I need?

Most sites need one or two: a restart for server_5xx/timeout and, if you use Let's Encrypt, a renewal for ssl_error. Everything else should notify.

Key takeaways

• Map cause codes to actions; never run the same fix for every outage.
• Automate restarts for dead or stuck processes, renewals for certificates, rollbacks for recent bad deploys. Notify for everything else.
• Fire after a short delay, retry once or twice with a real gap, then stop and escalate.
• Log every automated fix and review how often each one fires. Recurring fixes are bugs.

Uptime Monitor records a cause code for every failed check and lets each automated action run only for the causes you choose, with its own downtime threshold, attempt limit and retry interval. The free Starter plan includes one action per site, and Plus and Professional include five and ten per site, which covers a restart, a renewal and a rollback. See the plan limits for details.