Safe SSH Auto-Recovery: Locked-Down Keys and Sudo

Letting a monitoring service SSH into your server and restart things at 3 a.m. is genuinely useful: the most common outages (a wedged PHP-FPM pool, a crashed queue worker, a leaking Node process) are fixed by exactly the restart you would have typed by hand. It is also a way to hand a third party a key to your machine. This guide shows how to set up SSH-based auto-recovery so that the key can do one or two harmless things and nothing else, even if it leaks.

The threat model in one paragraph

Assume the private key you paste into any external tool will eventually be exposed: a breach, a misconfigured backup, a former teammate with dashboard access. The goal is that whoever holds that key can, at worst, restart a service you were going to restart anyway. They must not get a shell, must not read files, must not forward ports into your private network and must not escalate to root. Every layer below enforces one part of that, and they are meant to be used together.

Step 1: create a dedicated, unprivileged user

Never reuse root, your personal account or your deploy user. Create a system account that exists only for automated recovery:

sudo adduser --system --group --shell /bin/bash --home /var/lib/autofix autofix
sudo mkdir -p /var/lib/autofix/.ssh
sudo chmod 700 /var/lib/autofix/.ssh
sudo chown -R autofix:autofix /var/lib/autofix

The account needs a real shell because OpenSSH runs forced commands through the user's login shell; /usr/sbin/nologin would block it. That is fine, because the forced command in Step 3 means the shell is never handed to the client. The account has no password, so it cannot log in any way other than with the key you are about to install. Some systems mark such accounts as locked (! in /etc/shadow), and if sshd runs without PAM it then refuses even key logins with "account is locked" in the log. If you see that, run sudo usermod -p '*' autofix: * means no usable password, but the account is not locked.

Step 2: generate a key used for nothing else

Generate the key pair on your workstation, not on the server, and give it a name that says what it is for:

ssh-keygen -t ed25519 -C "autofix@uptime-monitor" -f ~/.ssh/autofix_ed25519

One key per purpose means you can revoke it by deleting one line without touching anything else. If your monitoring tool supports an encrypted key with a passphrase, you can set one; it protects the file at rest, but the passphrase has to be stored next to it, so treat it as a minor extra rather than the main control. The real protection is what the server allows the key to do.

While you are on the server, confirm password logins are off globally in /etc/ssh/sshd_config (or a file in /etc/ssh/sshd_config.d/):

PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin no

Run sudo sshd -t to validate the config before reloading with sudo systemctl reload ssh (the unit is called sshd on RHEL-family systems). Keep your current session open until you have confirmed a fresh login still works.

Step 3: lock the key down in authorized_keys

This is the most important step. OpenSSH lets you prefix a public key with options that restrict what a session using that key can do. Put this in /var/lib/autofix/.ssh/authorized_keys as a single line:

restrict,command="/usr/local/sbin/autofix-dispatch",from="203.0.113.10" ssh-ed25519 AAAAC3Nza...rest-of-key autofix@uptime-monitor

What each option does:

• restrict (OpenSSH 7.2 and later) turns off every optional capability at once: port forwarding, agent forwarding, X11 forwarding, PTY allocation and execution of ~/.ssh/rc. It also covers capabilities added in future OpenSSH releases, which is why it is preferred over listing no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty one by one. On an older server without restrict, use that explicit list instead.
• command="..." forces this program to run no matter what the client asks for. Whatever command the client sent is placed in the environment variable SSH_ORIGINAL_COMMAND, so your script can inspect it and decide.
• from="..." accepts the key only from the listed addresses or patterns (comma-separated, e.g. from="203.0.113.10,198.51.100.0/24"). Use the address the monitoring connections actually come from. If you do not know it, make one test connection and read it from /var/log/auth.log (Debian/Ubuntu) or journalctl -u ssh.

File permissions matter; sshd silently ignores the file if it is writable by others:

sudo chmod 600 /var/lib/autofix/.ssh/authorized_keys
sudo chown autofix:autofix /var/lib/autofix/.ssh/authorized_keys

Step 4: a dispatcher that only accepts known verbs

The forced command is a tiny allowlist. The monitoring tool sends a short verb such as restart-php; the dispatcher maps it to a fixed script and refuses everything else. The client never supplies paths, flags or arguments that end up in a shell.

#!/bin/bash
# /usr/local/sbin/autofix-dispatch  (owner root, mode 755)
set -euo pipefail

cmd="${SSH_ORIGINAL_COMMAND:-}"
logger -t autofix "request from ${SSH_CLIENT%% *}: '${cmd}'"

case "$cmd" in
  restart-php)    exec /usr/local/sbin/autofix-restart-php ;;
  restart-queue)  exec /usr/local/sbin/autofix-restart-queue ;;
  status)         exec /usr/bin/systemctl is-active php8.3-fpm nginx ;;
  *)
    logger -t autofix "rejected: '${cmd}'"
    echo "unknown command" >&2
    exit 1
    ;;
esac

Make the dispatcher and the scripts it calls owned by root and not writable by autofix. If the restricted user could edit the scripts that sudo runs as root, the sudo rule in the next step would be a privilege escalation.

Step 5: a sudoers rule for exactly one command

Restarting a systemd service needs root. Grant it for one exact command line, not for systemctl in general. Find the binary path first, because sudoers needs an absolute path and it differs between distributions:

command -v systemctl
# /usr/bin/systemctl on current Debian, Ubuntu, Fedora and RHEL

Always edit sudoers through visudo, which refuses to save a file with a syntax error (a broken sudoers file can lock everyone out of sudo):

sudo visudo -f /etc/sudoers.d/autofix

Contents:

autofix ALL=(root) NOPASSWD: /usr/bin/systemctl restart php8.3-fpm, /usr/bin/systemctl restart laravel-worker.service

Then check the whole configuration and the permissions:

sudo chmod 0440 /etc/sudoers.d/autofix
sudo visudo -c
sudo -l -U autofix

Details that trip people up:

• When a command in sudoers is listed with arguments, sudo only allows that exact argument list. systemctl restart php8.3-fpm does not permit systemctl stop php8.3-fpm or systemctl restart ssh. That is what you want.
• Never write /usr/bin/systemctl * or a bare /usr/bin/systemctl. That lets the key stop your firewall or start arbitrary units.
• Files in /etc/sudoers.d/ whose names contain a dot or end in ~ are skipped on Debian-family systems. Name it autofix, not autofix.conf.
• Do not allow editors, pagers, interpreters, tee, cp or bash via sudo. Each of those can be turned into a root shell.

Step 6: make restart scripts idempotent and self-checking

A monitoring tool may retry an action, and two checks can overlap. The script must be safe to run twice in a row, must not stack up parallel restarts and should report whether the fix actually worked:

#!/bin/bash
# /usr/local/sbin/autofix-restart-php  (owner root, mode 755)
set -uo pipefail

exec 9>/var/lib/autofix/php.lock
if ! flock -n 9; then
  logger -t autofix "restart-php already running, skipping"
  echo "already running"
  exit 0
fi

stamp=/var/lib/autofix/last-php-restart
now=$(date +%s)
last=$(cat "$stamp" 2>/dev/null || echo 0)
if (( now - last < 300 )); then
  logger -t autofix "restart-php skipped: last restart $((now - last))s ago"
  echo "cooldown"
  exit 0
fi

logger -t autofix "restarting php8.3-fpm"
sudo -n /usr/bin/systemctl restart php8.3-fpm
echo "$now" > "$stamp"

sleep 3
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 http://127.0.0.1/health || true)
logger -t autofix "post-restart health check: HTTP ${code}"
echo "php8.3-fpm restarted, health=${code}"
[[ "$code" == "200" ]]

The lock file prevents overlapping runs; the timestamp gives a five-minute cooldown so a site that keeps dying does not get restarted in a tight loop; the final health probe makes the exit code meaningful. Adjust /health to a local URL that exercises PHP. The script runs as autofix, so the stamp directory must be writable by that user and the only privileged call is the one sudo allows.

Step 7: logging you can actually read later

Everything above writes through logger -t autofix, so one command shows every request, rejection and result:

journalctl -t autofix --since "24 hours ago"
grep autofix /var/log/syslog        # Debian/Ubuntu without persistent journal

sudo logs each command it runs as well (journalctl _COMM=sudo), and sshd logs each login with the key fingerprint. If you ever see a rejected command you did not configure, rotate the key immediately: delete its line from authorized_keys, generate a new pair and update the monitor.

Testing before you trust it

From the allowed host (or temporarily without from=), confirm each layer:

ssh -i ~/.ssh/autofix_ed25519 autofix@server restart-php    # should restart and print health
ssh -i ~/.ssh/autofix_ed25519 autofix@server "cat /etc/shadow"  # should print "unknown command"
ssh -i ~/.ssh/autofix_ed25519 autofix@server                  # no shell; dispatcher rejects empty command
ssh -i ~/.ssh/autofix_ed25519 -N -L 15432:127.0.0.1:5432 autofix@server  # then connect to localhost:15432

The tunnel in the last line should be refused: connecting to local port 15432 makes the ssh client print administratively prohibited. If the restart works and the other three fail, the setup is doing its job. The -n on sudo in the script makes it fail immediately instead of waiting for a password prompt if the sudoers rule ever stops matching.

What not to automate

Auto-recovery should only run actions that are safe to repeat and cheap to get wrong. Keep these manual:

Also remember that a restart erases evidence. Before restarting, have the script save something cheap and useful, such as systemctl status php8.3-fpm --no-pager or the last 100 lines of the error log, to a file. Then you can still work out why it hung once the site is back. Deciding which failure should trigger which action is its own topic, covered in mapping failure causes to targeted auto-fix actions.

FAQ

Is a password instead of a key acceptable for auto-recovery?

Avoid it. A password cannot carry command= or from= restrictions, so whoever has it gets a full shell. Disable password authentication for the account and use a restricted key.

Does the restricted user need to be in the sudo group?

No, and it should not be. The single rule in /etc/sudoers.d/autofix grants exactly what is needed; group membership would grant everything the group allows.

Can I use this with Docker instead of systemd?

Yes, but do not add the user to the docker group, which is equivalent to root. Grant a sudo rule for one exact command such as /usr/bin/docker restart app-php instead.

Checklist

• Dedicated autofix user, no password, not in sudo or docker groups.
• Dedicated ed25519 key used for nothing else.
• restrict,command="...",from="..." on the key line; authorized_keys mode 600.
• Dispatcher with a case allowlist; scripts owned by root.
• Sudoers rule with full paths and exact arguments, created with visudo -f, verified with visudo -c.
• Scripts with a lock, a cooldown, a health check and logger output.
• Reboots, databases, deploys and deletions kept out of automation.

Uptime Monitor's Auto-Fix can run an SSH command like restart-php when a site has been down for a set number of minutes, with configurable retries and delay, and it can be limited to specific failure causes so a DNS problem never triggers a server restart. Pair it with escalation webhooks and scheduled actions so a human gets pulled in if the restart does not help. The free Starter plan includes one automated action, enough to try this setup on a single site: create an account.