Skip to content

Crawl, index and render

robots.txt decides what Googlebot may fetch; noindex, status codes and the rendered HTML decide what is indexed.

Predict first

Before you read: predict

This item runs on this lesson's Googlebot model, whose rules quote Google's pages. ExamHall's robots.txt holds one group: User-agent: * and Disallow: /attempts/. Attempt pages carry <meta name="robots" content="noindex"> in their HTML; proctor pages are sent with the header X-Robots-Tag: noindex; catalog pages carry neither. Every page answers 200. A public study forum links to three URLs: (1) /attempts/a-1, (2) /proctor/hall-b, (3) /exams/aws-saa.

Which of them can appear in Google's search results?

Pick one

The 30-second version

Google crawls, renders, then indexes. robots.txt controls crawling: a disallowed URL is never fetched, but when other pages link to it, it can still be listed, without a description. noindex, as a meta tag or an X-Robots-Tag header, controls indexing, and Googlebot sees it only on a page it may fetch. Script-added content arrives only at rendering.

robots.txt decides what Googlebot may fetch; noindex, status codes and the rendered HTML decide what is indexed.

How it works

Step 1 of 11

Three phases. Googlebot fetches a URL (crawl), runs the page's JavaScript in headless Chromium (render), and indexes the page from the rendered HTML.
In the kitchen
  • she walks the paths on her list, returns in spring to see what the season brings out, and the editor prints the guide from her notes → crawl, render, index (steps 1, 9)

Analogy: A surveyor walking the paths for a county walking guide

On the walking guide In Search
she walks the paths on her list, returns in spring to see what the season brings out, and the editor prints the guide from her notes crawl, render, index (steps 1, 9)
a "Private, no entry" gate: she never walks that path a robots.txt Disallow (step 2)
a path other maps mention still goes in as a dotted line, with no description a blocked URL listed without a description (step 5)
a landowner's sign halfway along, asking for the path to be left out: she sees it only once she walks it noindex (step 6)
a sign at the stile for a path that is closed, or a fingerpost still pointing to a footbridge that washed away how a gone page answers (steps 7, 8)
she follows fingerposts, never "ask at the farm for the way to the lake" <a href> links (step 11)

In real terms: each phase can fail on its own: the fetch, the render, or the decision to index.

Breaks down when: she re-walks a path when she chooses; Googlebot recrawls on its own schedule, and a page it rarely visits may wait months before a new noindex is seen.

Contrasting case: a signed-in attempt page needs no crawl rule to stay private: Googlebot has no session, so the page answers 401 (step 7), and noindex is a second guard, not the first.

Examples, run for real

4.1 ExamHall EXH-4610 → MR !4910: attempt URLs listed as bare links

ExamHall's robots.txt disallowed /attempts/, and attempt pages sent noindex. Attempt URLs pasted in a public forum still showed in Google as bare links (EXH-4610). MR !4910 deletes the Disallow line, so Googlebot may fetch the pages and see what they answer.

4.1 EXH-4610, Disallow: /attempts/ · /attempts/a-9 fetched: no · listed: URL only, no description (linked from the forum)
4.1 MR !4910, Disallow line deleted · /attempts/a-9 fetched: yes · status 401 · X-Robots-Tag noindex: yes · listed: no

Without the Disallow, Googlebot fetches the page, which answers 401 with its noindex header, and the URL leaves the results; the block had hidden both (steps 5–7).

4.2 Proof domain, ticketing: a draft line-up PDF

A ticketing site keeps draft line-ups for unannounced festivals as PDFs, and a promoter's newsletter links one. A PDF has no <head> for a meta tag, so the site sends the header.

4.2 /lineups/draft-summer.pdf, no header · status 200 · X-Robots-Tag noindex: no · listed: yes
4.2 /lineups/draft-summer.pdf, with the header · status 200 · X-Robots-Tag noindex: yes · listed: no

The same file, with the header, is dropped from the results; nothing in the PDF changed (step 6).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 6

This item runs on this lesson's Googlebot model, whose rules quote Google's pages. ExamHall's exam aws-saa-2024 ended last month. Its URL, /exams/aws-saa-2024, must still tell a visitor that the exam has ended, and must stop being a page Google keeps. Three engineers each built the response for that URL.

Which of the three is a soft 404?

Pick one

Question 2 of 6

This item runs on this lesson's Googlebot model, whose rules quote Google's pages. A ticketing site's home page, as its server sends it, holds these five elements:

<a href="/events/e-1">Neon Nights</a>
<button onclick="location.href='/events/e-2'">Harbour Jazz</button>
<a href="#/events/e-3">Comedy Cellar</a>
<span class="event" data-href="/events/e-4">Brass Week</span>
<ul id="more"></ul>

The page's script makes each .event element open its data-href on a click, and adds <li><a href="/events/e-5">Folk Fair</a></li> to #more. The page answers 200, and robots.txt allows every URL.

How many of the five event pages can Googlebot discover from this home page?

Pick one

Question 3 of 6

ExamHall's robots.txt reads:

User-agent: *
Disallow: /attempts/
Disallow: /proctor/

User-agent: Googlebot
Disallow: /drafts/

Googlebot asks for four URLs: /attempts/a-1/q-3, /proctor/hall-b, /drafts/aws-saa-v2 and /exams/aws-saa.

How many of them may it fetch?

Pick one

Question 4 of 6

This item runs on this lesson's Googlebot model, whose rules quote Google's pages. A ticketing site's event page /events/e-7 is an app shell. Its server sends <h1 id="title">Loading…</h1> and a script that requests /api/events/e-7, then writes the event's name and date into the heading, or, if the request fails, writes Details unavailable. The API answers { "name": "Neon Nights", "date": "12 March" }. The site's robots.txt holds User-agent: * and Disallow: /api/. The page answers 200.

In the version of the page Google indexes, what does the heading say?

Pick one

Question 5 of 6

This item runs on this lesson's Googlebot model, whose rules quote Google's pages. Two ExamHall catalog pages each got a fix; both answer 200 and robots.txt allows both. Page 1, /exams/gcp-ace, still ships <meta name="robots" content="noindex"> in the HTML its server sends, left over from staging; the team wants it in Google, so its fix is a script that removes that tag as soon as the page loads. Page 2, /exams/az-104, is a retired exam whose server sends no robots tag; the team wants it out of Google, so its fix is a script that adds <meta name="robots" content="noindex"> when the exam API reports it retired.

Which fix can the team rely on?

Pick one

Question 6 of 6

A ticketing site's robots.txt reads:

User-agent: *
Disallow: /events/
Allow: /events/*/tickets
Disallow: /*?sort=

Googlebot asks for four URLs: /events/e-1, /events/e-1/tickets, /events/e-1/tickets?sort=price and /search?sort=date.

How many of them may it fetch?

Pick one

Go deeper

Pro tip
  1. When asked whether Google will index a client-rendered page, walk the three phases before naming a fix: may Googlebot fetch it, does it answer 200 without noindex, and what does the rendered HTML hold once its requests have run. When: the interviewer asks whether Google will index a client-rendered React page. Status: SELF; not run.
Source

Valid as of 2026-10-16. Every rule the model adds is quoted from these pages in lib/googlebot.mjs and lib/robots.mjs.

  • Google Search Central, Understand the JavaScript SEO basics (named in the row): https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics · google-js-seo-basics.txt. "Google processes JavaScript web apps in three main phases"; "If it marks the URL as disallowed, then Googlebot skips making an HTTP request to this URL and skips the URL."; "Googlebot queues all pages with a 200 HTTP status code for rendering, unless a robots meta tag or header tells Google not to index the page."; "Google also uses the rendered HTML to index the page."; "Google Search won't render JavaScript from blocked files or on blocked pages."; "Google can only discover your links if they are <a> HTML elements with an href attribute." Supports steps 1, 2, 7–11; two more lines from this page sit in the assessment block below.
  • Google Search Central, Introduction to robots.txt (named in the row): https://developers.google.com/search/docs/crawling-indexing/robots/intro · google-robots-intro.txt. "it is not a mechanism for keeping a web page out of Google"; "If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result won't have a description." Supports steps 2, 5.
  • Google Search Central, Block Search indexing with noindex (named in the row): https://developers.google.com/search/docs/crawling-indexing/block-indexing · google-block-indexing.txt. "If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule"; "There are two ways to implement noindex: as a <meta> tag and as an HTTP response header."; "it may take months for Googlebot to revisit a page"; "some search engines might interpret the noindex rule differently".
  • RFC 9309, Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309.txt · rfc9309.txt. "Crawlers MUST use case-insensitive matching to find the group that matches the product token and then obey the rules of the group."; "The most specific match is the match that has the most octets."; "If no match is found amongst the rules in a group for a matching user-agent or there are no rules in the group, the URI is allowed." Its section 2.2.1 sends a crawler that no group names to the * group, and section 2.2.2 prefers the allow rule when an allow and a disallow rule are equivalent. Supports steps 3, 4; robots-parser 3.0.1 implements it and is the executed truth.

Related lessons

Checked against the tools and docs as of 2026-10-16.