Monitoring Competitor Websites: An Ethical, Useful Approach

You probably know your own site's uptime and response time. You probably do not know whether that is good or bad compared with the businesses your customers also consider. Watching a few competitors' public pages with the same tool you use for your own site gives you a benchmark, a few useful surprises, and sometimes a sales argument. This article covers how to do it politely and legally, what the numbers can and cannot tell you, and how to turn them into decisions.

Ground rules: public pages, low frequency, no tricks

Competitor monitoring is only worth doing if you would be comfortable explaining exactly what you did to the competitor. In practice that means:

Only public pages. Monitor URLs any anonymous visitor can open: homepage, pricing page, public documentation, public status page, a public API health endpoint if they publish one. Never use credentials, trial accounts or session cookies to monitor pages behind a login.
Low frequency. A check every 10–30 minutes, or hourly, is plenty for benchmarking. One request every few minutes is far less than a single human browsing the site, but there is no reason to go faster: you are not on call for their outages.
A single, lightweight request. One HTTP GET of the HTML. Do not load-test, crawl the whole site, or request expensive search or checkout URLs.
No evasion. If a site blocks your checks, shows a CAPTCHA, or its terms of service prohibit automated access, stop. Do not rotate IPs or spoof browsers to get around it.
Respect robots.txt as a signal of intent, even though a single monitoring request is not a crawler in the usual sense.

This is a matter of ethics and of law, and laws differ by country. If you plan to publish comparisons, talk to a lawyer first; see the section on positioning below.

What you can actually learn

An external HTTP check sees what any visitor sees: whether the page answered, with what status code, how long it took, and what the response contained. Over weeks, that adds up to useful patterns.

Signal	What it tells you	How reliable
Availability of the homepage or app login page	How often they are fully down, and for how long	Good for full outages; misses partial ones
Recurring outage times	Maintenance habits, e.g. a weekly late-night deployment window	Good once you have several weeks of data
Response time and time to first byte	Rough server and caching performance, trends after redesigns or migrations	Relative only; one network path
Status codes and redirects	Domain moves, new URL structures, region redirects	Good
SSL certificate details	Issuer, renewal habits, near-misses with expiry	Good
Security headers	Whether they set HSTS, CSP and similar headers — a hint about engineering maturity	Good, but say nothing about real security
Presence of a specific text	When a price, plan name or "free trial" disappears from a public page	Good for one or two well-chosen phrases

The last row is a simple trick: set a required keyword such as their current entry-level price ("$29") on the public pricing page. When the text disappears, the check alerts you and you go look. It is not a full change-tracking tool, but for one or two facts it works.

What you cannot learn

Be honest about the limits, especially before you show the data to anyone:

Causes. You see that a site returned 502 for 40 minutes, not why.
Partial outages. Their homepage may be served from a CDN cache while the logged-in product is broken. A homepage check will not notice.
Regional experience. A check from one location measures one network path. A competitor hosted on another continent will look slower from your monitor than to their own customers.
Traffic, conversions or revenue. Uptime data says nothing about how many people use the site.
Their internal SLAs or how fast their team responds. You see the duration of an outage, not the effort behind it.
Short blips at a low check frequency. With hourly checks, a 10-minute outage between checks is invisible. That is the price of being polite, and it is fine for benchmarking.

A setup that produces fair comparisons

A benchmark is only useful if the same rules apply to everyone, including you. A reasonable setup:

Pick three to five competitors that customers actually compare you with, not the biggest names in your industry.
Choose equivalent pages. Homepage vs homepage, pricing vs pricing. If you compare their cached marketing homepage with your dynamic dashboard, you will draw the wrong conclusion.
Use the same monitor, interval and timeout for all, including your own equivalent page. A 15-minute interval for everyone is a sensible default.
Keep competitor alerts out of your on-call path. If your tool cannot mute alerts per site, use a separate account whose notifications go to a low-priority channel. You do not want to be woken up by someone else's outage.
Collect at least four weeks of data before drawing conclusions, so a single bad day does not dominate.
Export and compare monthly: uptime percentage, number of incidents, median and 95th percentile (P95) response time.

When reading uptime percentages, remember how much the check interval matters: with 15-minute checks each failed check represents up to 15 minutes, so two competitors at "99.8%" may differ by an hour of real downtime. Our article on what uptime percentages really mean explains the math, and how to read uptime reports covers interpreting the charts.

Turning the data into positioning

The point is not to collect trivia about other companies. It is to make better decisions about your own product and message.

If you are clearly better

Make reliability part of your story, but back it with your own data, not theirs. A public status page with your history is far more credible than a claim on a landing page, and it is evidence prospects can check themselves; see the benefits of public status pages. Avoid naming competitors' outages publicly: comparative advertising rules in many countries require claims to be accurate, verifiable and fair, and data from a single location over a few weeks may not meet that standard.

If you are clearly worse

That is the most valuable finding. You now have an external, neutral argument for investing in caching, hosting or deployment practices, and a target to measure progress against.

If everyone is similar

Reliability is table stakes in your market and not a differentiator. Spend your positioning effort elsewhere, and keep monitoring so you notice if that changes.

Use their incidents as a learning prompt

When a competitor has a long outage, ask "could the same thing happen to us?" If their site was down after an obvious migration, check your own migration runbook. If their certificate nearly expired, check your renewal automation. This costs nothing and does not involve saying anything public about them.

FAQ

Is it legal to monitor a competitor's website?

Requesting a public page occasionally, as any visitor would, is generally considered normal. Accessing non-public areas, bypassing technical blocks or violating terms of service is a different matter. Laws vary by jurisdiction, so get legal advice before publishing comparisons.

How often should I check a competitor's site?

Every 10 to 30 minutes, or hourly, is enough for benchmarking. Faster checks add load to their server without giving you information you would act on.

Will the competitor notice?

Your requests will appear in their access logs like any other visitor's. Assume they can see them, and behave accordingly.

Checklist

Public URLs only; no logins, no trial accounts.
Check every 10–30 minutes or hourly; one lightweight GET per check.
Stop if blocked or if their terms prohibit automated access.
Same interval, timeout and page type for every site, including yours.
Keep competitor alerts away from your on-call channel.
Compare monthly: uptime %, incident count, median and P95 response time.
Use the results internally; get legal advice before publishing comparisons.

Uptime Monitor can hold a small benchmark set: add your own page and a few competitors' public pages with the same check interval (15 or 30 minutes, or hourly, is plenty), use a required keyword to watch a pricing page, compare uptime on the dashboard, and work out median and P95 response time from a CSV export. Checks run from a single location, so treat results as relative rather than absolute. The Plus plan covers up to 10 websites.