500, 502, 503 and 504 Errors: What Googlebot Does and How to Fix Them

Direct answer: a 5xx status means the server side failed to answer the request. Google treats 500, 502, 503, and 429 responses, along with network timeouts, as a reason to slow crawling for the whole hostname, and URLs that keep failing are eventually dropped from the index. Find the hop that produced the code, fix that hop, and use 503 with a Retry-After header only for short, planned unavailability.

This guide covers server errors as Google sees them: which code points where, what Googlebot does with each, how to serve maintenance correctly, and how to confirm recovery. It does not cover soft 404s, where the server reports success for a page that is effectively missing, or retries inside an automation that calls someone else's API.

Which server error are you seeing?

The status code tells you which component gave up, not where the original bug lives. RFC 9110 defines the codes by role: 500 comes from the server that hit an unexpected condition, while 502 and 504 come from a gateway or proxy that could not get a usable answer from the server behind it.

CDNs add their own codes. Cloudflare, for example, uses 520 for an empty or unexpected origin response, 521 when the origin refuses the connection, 522 when connecting to the origin times out, and 524 when the origin connects but does not answer within the proxy read timeout. Cloudflare's documentation currently gives 125 seconds as the default for that timeout.

What Google does when your site returns 5xx

Google's crawling documentation is specific about the effect, and the effect reaches beyond the failing URLs.

• Crawling slows down across the hostname. Google says 5xx and 429 errors make its crawlers temporarily slow down. Its crawl-rate page says that a significant number of 500, 503, or 429 responses reduces the crawl rate for the whole hostname, including URLs that still return content. For 500, the slowdown is proportionate to the number of URLs returning errors.
• Recovery is gradual. Once the server returns 2xx again, Google gradually increases the crawl rate. Don't expect an immediate return to the previous pace.
• Persistent errors remove URLs. Already-indexed URLs are kept at first but eventually dropped. Google's crawl-rate page warns that seeing these codes on the same URL for multiple days can drop it from the index. Google gives no exact number of days.
• Timeouts count as server errors. Network timeouts, connection resets, and DNS errors are treated like 5xx errors. Google's network-errors page says unreachable indexed URLs are removed within days.
• A failing robots.txt can pause the whole site. If robots.txt returns a 5xx or 429, Google stops crawling the site for the first 12 hours and keeps retrying. For the next 30 days it uses the last good copy. After 30 days, if the site is generally available, Google behaves as if there is no robots.txt.

There is a practical limit on intentional 503 or 429 responses. Google's Crawl Stats help warns against returning them for more than two or three days, and its crawl-rate page against doing so for longer than one or two days. Treat a 503 as a way to cover a short window, not as a parking state for a site that is down for a week.

For the wider crawl sequence these errors interrupt, see how Google crawling works.

Reproduce the four errors locally

Definitions are easier to apply once you have watched each code appear. The script below starts a small application on port 8081 and a gateway on port 8080 that waits two seconds for the application, as a reverse proxy or CDN would. It needs Node.js 18 or later and no packages.

// 5xx-lab.mjs: reproduce 500, 502, 503 and 504 locally. Node.js 18+, no dependencies.
// A teaching demo, not a production proxy.
import http from "node:http";

const APP = 8081, GATEWAY = 8080, NOTHING = 8089; // nothing listens on 8089

// The application behind the gateway.
http.createServer((req, res) => {
  if (req.url === "/error") {
    try { throw new Error("database query failed"); }
    catch { res.writeHead(500, { "Content-Type": "text/plain" }); return res.end("error"); }
  }
  if (req.url === "/hang") return; // a bug: the handler never responds
  if (req.url === "/maintenance") {
    res.writeHead(503, { "Retry-After": "3600", "Cache-Control": "no-store", "Content-Type": "text/html" });
    return res.end("<h1>Down for maintenance</h1>");
  }
  res.end("ok");
}).listen(APP);

// A gateway that gives the application 2 seconds, like a reverse proxy or CDN.
http.createServer((req, res) => {
  const port = req.url === "/down" ? NOTHING : APP;
  let settled = false; // answer each request once: a timeout is followed by an error event
  const fail = (code, text) => {
    if (settled) return;
    settled = true;
    if (res.headersSent) return res.destroy(); // too late for a status code: drop the connection
    res.writeHead(code, { "Content-Type": "text/plain" }).end(text);
  };
  const upstream = http.get({ port, path: req.url, timeout: 2000 }, (up) => {
    settled = true;
    res.writeHead(up.statusCode, up.headers);
    up.pipe(res);
    up.on("error", () => res.destroy()); // the application died mid-response
  });
  upstream.on("timeout", () => { fail(504, "upstream timed out"); upstream.destroy(); });
  upstream.on("error", () => fail(502, "bad upstream"));
}).listen(GATEWAY, () => console.log(`gateway: http://localhost:${GATEWAY}`));

Run node 5xx-lab.mjs, then request each route through the gateway in a second terminal:

curl -sS -i -w "[time_total=%{time_total}s]\n" http://localhost:8080/error

The gateway settles each request once, because Node.js reports a timed-out upstream first as a timeout and then as an error. That is enough for a lab. A production proxy also has to forward request bodies, strip hop-by-hop headers, limit connections and log failures, so use nginx, a load balancer or your CDN for real traffic.

We ran it on November 2, 2026 (Philippine time) with Node.js 20.20.0 and curl 8.18.0. Abridged output, with the date, content-type and keep-alive headers removed:

GET /error        HTTP/1.1 500 Internal Server Error    time_total=0.005s
GET /maintenance  HTTP/1.1 503 Service Unavailable
                  retry-after: 3600
                  cache-control: no-store               time_total=0.002s
GET /hang         HTTP/1.1 504 Gateway Timeout          time_total=2.008s
GET /down         HTTP/1.1 502 Bad Gateway              time_total=0.028s

Three things in that output matter in production.

1. One bug, three possible codes. A failed database call becomes a 500 if the application catches it and answers, a 504 if the handler never answers, and a 502 if the process dies and the gateway cannot connect. The status points to the layer that reported the failure, so search the logs of the layer below it.
2. A 504 takes exactly as long as the gateway's timeout. The /hang request took 2.008 seconds because the gateway's limit was 2 seconds. When a production 504 always arrives after the same number of seconds, that number tells you which layer's timeout fired: 60 seconds is nginx's default proxy_read_timeout, and around 125 seconds suggests Cloudflare's 524.
3. The 503 carries its own instructions. Retry-After: 3600 tells clients to come back in an hour, and Cache-Control: no-store keeps the maintenance page out of caches.

Planned maintenance: return 503 with Retry-After

RFC 9110 describes 503 as a temporary state, such as scheduled maintenance, that will likely be resolved after some delay. Google's crawl-rate documentation, updated October 6, 2026, says you can include a Retry-After header with a 503 or 429 to tell its crawlers when to retry. The value is either a number of seconds (Retry-After: 3600) or an HTTP date in GMT (Retry-After: Fri, 30 Oct 2026 14:00:00 GMT).

A maintenance page that returns 200 tells Google the page's content is now the maintenance notice. A 404 tells Google the page is gone. A 503 is the only one of the three that says “come back later.”

nginx

server {
    # ...existing listen, server_name and TLS settings...
    error_page 503 /maintenance.html;

    location = /maintenance.html {
        root /var/www/maintenance;
        add_header Retry-After 3600 always;
        add_header Cache-Control "no-store" always;
        internal;
    }

    location / {
        return 503;
    }
}

The always flag is the detail that most often goes missing. nginx documentation lists the response codes add_header applies to by default (200, 201, 204, 206, 301, 302, 303, 304, 307 and 308), and 503 is not one of them. Without always, the header silently disappears from the maintenance response. Also, a block that defines any add_header stops inheriting add_header values from the outer level, so put the header where the final response is produced.

Apache

# Requires mod_rewrite and mod_headers
RewriteEngine On
RewriteCond %{REQUEST_URI} !=/maintenance.html
RewriteRule ^ - [R=503,L]
ErrorDocument 503 /maintenance.html
Header always set Retry-After "3600"
Header always set Cache-Control "no-store"

Apache's rewrite flags documentation allows a non-redirect code such as R=503. The substitution is then dropped and the response goes through the normal error path, including ErrorDocument. As with nginx, the header needs the always condition: Apache's default header table does not apply to locally generated non-2xx responses.

Verify before you rely on it

We have not run these two server blocks against your stack, and configurations vary. Test the live result, not the file:

curl -sS -I https://www.example.com/any-page/
# expect: HTTP/2 503, a retry-after header and cache-control: no-store

curl -sS -I https://www.example.com/robots.txt
# decide deliberately what this returns during the window

The robots.txt check is a decision, not a formality. If robots.txt also returns 503, Google stops crawling the site for up to 12 hours and then falls back to its last good copy. That can suit a short, whole-site outage. If you want Google to keep its normal robots rules during a partial maintenance, exclude /robots.txt from the maintenance rule.

Before a planned migration, plan the maintenance window together with the redirect map in the site migration SEO checklist.

Diagnose a 500 Internal Server Error

1. Get the exact time and URL. Use Search Console's example URLs, your crawler, or the uptime alert. A 500 without a timestamp is hard to match in logs.
2. Read the application log, not the access log. The access log shows that a 500 happened. The application or PHP error log shows the exception and the line that caused it.
3. Compare with the latest change. A deploy, plugin update, configuration edit or expired credential near the first error is the most likely suspect. Roll back or patch that change first.
4. Find the pattern. Errors on one template, one parameter or one content type point to code. Errors on every URL at once point to configuration, the database or the host.
5. Test as Googlebot sees it. If browsers work but Search Console still reports server errors, check the firewall, bot rules and rate limits for Google's crawler. Google's Page indexing help lists inadvertently blocking Google as a cause of server connectivity errors.

Do not hide a 500 by returning 200 with an error message. That swaps a crawl problem for an indexing problem. For a site-wide sweep, combine the crawl with Search Console data as in the technical SEO audit with Screaming Frog and Search Console.

Diagnose 502 and 504 errors

Both codes are reported by a layer in the middle, so the first job is to find which layer reported them and which layer it was waiting for.

• Read the timing. curl -sS -o /dev/null -w "%{http_code} %{time_starttransfer}s\n" URL shows how long the response took. A fixed duration matches a timeout setting: your lab gateway at 2 seconds, nginx's default proxy_read_timeout at 60 seconds, Cloudflare's 524 at about 125 seconds.
• Separate the CDN from the origin. In Cloudflare's HTTP Traffic analytics, compare the edge status code with the origin status code. If the origin returned 200 and the edge returned 5xx, the problem sits between them. Cloudflare's own troubleshooting notes that a Cloudflare-branded 502 or 504 page usually reports an error from the origin web server.
• For 502, check that the process is alive. A crashed or restarting application, a full process manager, or a wrong upstream port all produce 502. Look for restarts around the error time.
• For 504, find the slow call. Look for slow database queries, locks, and outbound API calls without a timeout. Raising the gateway's timeout hides the symptom. It doesn't make the page faster, and Googlebot still waits.

If the slow call is an external API in an automation, the fix is a timeout and a safe retry, as shown in retrying failed n8n API requests without duplicates. That guide covers the client side of the same 503 and 504 codes.

Find server errors in Search Console

Two cautions from Google's help pages. Server errors can be transient, so a live test can succeed even though Google's crawl failed. And Crawl Stats counts responses per request, not per URL, so one heavily crawled resource can dominate the chart. Crawl Stats is available only for root-level properties, such as a domain property.

A 5xx during a crawl can also leave a URL in Crawled - currently not indexed or other indexing states. Fix the server error first, then review the page itself.

Validate the fix and recover crawling

1. Confirm the fix outside Search Console: request a sample of the affected URLs and check that they return 200 with normal timing.
2. Run URL Inspection's live test on two or three representative URLs.
3. Click Validate fix on the Server error (5xx) reason. Google's help says validation typically takes up to about two weeks and sometimes much longer. Do not click it again until it has passed or failed.
4. Watch Crawl Stats. The server connectivity status should return to normal, and the share of server-error responses should fall. Expect crawl activity to recover gradually.
5. Request indexing only for important URLs that dropped out, after they return 200 reliably.

Record the fix date. Reports lag, and a correct fix does not need to be redone because one chart still shows last week's errors.

What not to do

• Return 200 for a maintenance or error page. Google may treat the notice as the page's content or as a soft 404.
• Keep a 503 for days. Google's documentation warns that multi-day errors on the same URL can drop it from the index.
• Throttle Googlebot with 403 or 404. Google names 500, 503 and 429 as the codes that reduce crawl rate. Other codes don't mean “slow down” and can remove pages instead.
• Block crawling in robots.txt to make the errors go away. Blocked URLs stop showing server errors because Google stops fetching them. That removes the evidence, not the problem.
• Add noindex to pages that fail. The page needs to load, not to leave the index.
• Raise every timeout. Longer timeouts turn fast 504 responses into slow ones and keep connections busy longer.

Monitor before Google notices

Search Console reports server errors after Google has already met them. An uptime check that requests a few key URLs every few minutes and alerts on repeated 5xx responses finds the same problem sooner. The website uptime monitoring build shows one way to do this with n8n: it alerts on repeated failures rather than a single error, and sends one recovery message. Add alerts on the share of 5xx responses in your server or CDN logs, and review Crawl Stats once a month for host-status warnings.

Practical questions

Does a short 503 hurt rankings?

Google describes the immediate effect as slower crawling, with indexed URLs kept at first. Risk grows with duration: Google warns that errors on the same URL over multiple days can drop it from the index. A planned window of an hour or two, served as a 503, is the intended use of the code.

Do errors on some pages affect the whole site?

For crawling, yes. Google reduces the crawl rate across the whole hostname when it sees a significant number of 500, 503 or 429 responses. Healthy pages are fetched less often until the errors stop.

Should I use 429 or 503 when Googlebot crawls too fast?

Both slow Google's crawlers. Google's crawl-rate page presents them as a short emergency measure for one or two days. For a lasting reduction, fix capacity or follow Google's documented process for reducing crawl rate.

Why does Search Console show errors my browser doesn't?

The error may have been brief, or it may affect only Google's requests. Check firewall and bot rules, rate limits, and CDN security settings for Google's crawler. If you use Cloudflare, see whether Cloudflare is blocking Google or AI crawlers.

Evidence basis

Official documentation was checked on October 30, 2026. The local lab output comes from FloxoLab's own run of the script above on loopback ports. It demonstrates status-code behavior and is not a measurement of any production site. The nginx and Apache blocks follow each server's documentation and were not executed in our lab.

• Google: how HTTP status codes affect Google's crawlers (last updated February 4, 2026)
• Google: reduce Googlebot's crawl rate, including Retry-After (last updated October 6, 2026)
• Google: DNS and network errors
• Google: robots.txt error handling
• Search Console Help: Crawl Stats report
• Search Console Help: Page indexing report and Validate fix
• Search Console Help: URL Inspection tool
• RFC 9110: server error status codes and Retry-After
• nginx: add_header and the always parameter
• Apache: the R flag with non-redirect codes and the Header always condition
• Cloudflare: 5xx error troubleshooting (520–524 and 502/504 pages)