How Google Actually Crawls and Indexes a Website

Crawling and Indexing Are Two Different Steps

People often talk about "getting a page on Google" as one event. It's actually two separate steps, and a page can succeed at the first and still fail at the second. Understanding the difference explains a lot of confusing SEO behavior — like a page that Search Console confirms was crawled, but still doesn't show up in search results months later.

Crawling is Google's automated software (Googlebot) visiting your page and downloading its content. Indexing is the separate decision to store that page in Google's database and make it eligible to appear in search results. A page can be crawled and then deliberately or accidentally excluded from the index. Crawling is necessary but not sufficient.

How Googlebot Finds Your Pages in the First Place

Googlebot doesn't magically know your new page exists. It finds pages primarily through:

• Links — from other pages on your site (internal links) or from other websites (backlinks). This is the primary discovery mechanism, which is why internal linking matters more than it seems like it should.
• Sitemaps — an XML file listing your URLs, submitted through Google Search Console, that gives crawlers a direct list rather than requiring them to stumble onto every page through links. See robots.txt and sitemaps explained for how this works in practice.
• Previously crawled pages that link out — if your homepage was crawled last month and it links to a page that's new since then, Googlebot will typically pick that link up on its next visit.

A brand-new page with zero internal links pointing to it and no sitemap entry can sit undiscovered for a long time, regardless of how good the content is. This is one of the most common reasons a newly published page "isn't showing up" — it was never actually crawled, because nothing pointed Google to it.

What Happens During a Crawl

When Googlebot visits a page, it downloads the HTML, follows the links it finds, and — for modern JavaScript-driven sites — attempts to render the page much like a browser would, which is more resource-intensive and sometimes delayed compared to crawling a simple static HTML page. This is worth knowing if your site is built heavily in JavaScript without server-side rendering: content that only appears after JavaScript executes can be crawled more slowly, or in some cases missed, compared to content present in the initial HTML. Frameworks like Next.js address this specifically through server-side rendering or static generation, which is one practical reason it's a common choice for SEO-sensitive business sites.

Googlebot also respects robots.txt at this stage — if a path is disallowed, it won't request pages under it at all, meaning those pages are never crawled, never mind indexed.

Crawl Budget: Why It Matters More on Larger Sites

Google allocates a rough "budget" of how much time and how many pages it will crawl on your site during a given period, influenced by your site's overall authority, how often content changes, and server response speed. For a ten-page local business site, crawl budget is essentially irrelevant — Google can crawl the whole thing in seconds and will do so regularly. For a large e-commerce catalog with tens of thousands of URLs, or a site generating excessive duplicate or low-value pages (filtered product listings, endless pagination, session-ID URLs), crawl budget becomes a real constraint: Google may simply not get around to crawling every page, prioritizing what it judges as more valuable.

This is part of why cleaning up duplicate content and blocking low-value paths in robots.txt isn't just about avoiding ranking dilution — it also frees up crawl attention for the pages that actually matter.

Why Indexing Can Fail Even After a Successful Crawl

Once a page is crawled, Google decides whether to index it. Reasons a crawled page might not make it into the index include:

• A noindex meta tag on the page, either intentional (you don't want that page in search results) or accidental (often left over from a staging environment that got pushed to production without removing it).
• Thin or low-value content — a page with very little unique text, or content that duplicates another page closely enough that Google chooses not to index a near-copy.
• A canonical tag pointing elsewhere — telling Google this page isn't the authoritative version, so the canonical target gets indexed instead. See canonical tags explained.
• Quality judgments — Google increasingly weighs whether a page demonstrates genuine expertise and usefulness before indexing it prominently, which connects to broader E-E-A-T considerations.

Google Search Console's "Page indexing" report is the direct way to check this — it lists your pages by status, including "Crawled – currently not indexed" (a distinct and fairly common status) versus genuinely indexed pages, along with Google's stated reason for each exclusion.

How Long the Whole Process Takes

There's no fixed timeline, and it varies enormously by site authority and how often Google already crawls you. A large, frequently updated site might see a new page crawled and indexed within hours. A small, infrequently updated site might wait days or weeks for the same page. Submitting the URL directly through Search Console's "URL Inspection" tool and requesting indexing can speed this up for individual pages, though it's not a guarantee and isn't a substitute for having the page properly linked and included in your sitemap in the first place.

What This Means Practically

If a page isn't ranking, the diagnostic order that actually makes sense is: first confirm it's been crawled (Search Console URL Inspection tool), then confirm it's indexed, and only then start worrying about ranking factors like content quality or backlinks. Troubleshooting rankings on a page that was never indexed — because it's disallowed, orphaned, or carries a stray noindex tag — is solving the wrong problem entirely, and it's a more common cause of "invisible" pages than most business owners expect.