How a page gets discovered and re-crawled

There are three separate questions here and they have three different answers: how a crawler finds a page, how fast it learns the page changed, and what you can see about the result. This page answers all three with the actual timers.

How does a crawler find a page at all?#

Two pull-based paths, both automatic:

1. The sitemap. Each owner gets one sitemap.xml listing every page of every indexed repository, plus one entry per genuinely translated locale. It is served at the site's own host and rebuilt at most once an hour.
2. robots.txt names it. Every Docsbook-hosted host serves a robots.txt with a Sitemap: line pointing at the sitemap that belongs to that host. The apex carries several: its own, one per product documentation site, and one per public showcase site with SEO switched on — because those sites are only reachable from the apex, so nothing else would point a crawler at them.

Beyond that, the sidebar is rendered as real HTML links on every page, so every crawl of any page is also a crawl of the map of your content.

How fast does a change reach an engine?#

Push. When a change is published through Docsbook — the editor, an agent, the MCP tools, a workspace settings change — Docsbook invalidates its own caches for that site immediately, and for sites hosted on the docsbook.io apex it also sends an IndexNow notification naming the site root and its sitemap. That call is fire-and-forget: it names two URLs rather than the changed pages (at that point only "this site changed" is known, and the sitemap lets a participating engine work out the specifics), it never blocks or fails the publish, and an IndexNow outage is logged rather than surfaced.

Pull. Everything else waits for a crawler to come back and re-read the sitemap.

And the honest part: Docsbook has no GitHub webhook. A commit pushed straight to your repository, bypassing Docsbook, triggers no invalidation and no IndexNow push. It is picked up by timers:

So a page edited in Docsbook is current for the next crawler within seconds; a page pushed directly to GitHub can be served from cache for up to 24 hours. The 24-hour window is a deliberate trade: bot crawls re-rendering every page every hour were the single largest line on the hosting bill, and doc pages are read far more often than they are written.

What happens when a page is renamed?#

A move made through Docsbook writes the old page path → new page path into a .docsbook/redirects.json file in the same commit as the move, and the old URL then answers a 308 permanent redirect to the new one. Permanent, not temporary, because a temporary redirect tells search engines to keep the dead URL as the canonical one — which loses exactly the ranking the redirect exists to preserve. The map holds up to 500 entries, oldest dropped first, and it is only consulted for requests that were already heading for a 404.

A rename made by moving the file yourself in git gets no redirect: nothing wrote the map. You can add the entry by hand — it is a plain file in your repository, and both sides are page paths, the thing that appears in a URL.

What does the Search Console integration actually do?#

It reads. Data flows one way, Google → Docsbook, over the read-only Search Analytics scope. Docsbook does not submit URLs, request indexing, or write anything into your Search Console account, and there is nothing for you to connect: Docsbook reads a Search Console domain property on docsbook.io, which by Google's definition "aggregates data for all subdomains, protocols, and subpaths", and filters the rows down to the URLs that belong to your site.

What lands in the admin panel: average position, impressions, clicks, the queries your pages rank for, a daily trend, and an "worth improving" set — the queries sitting at position 5–20, already visible and not yet winning. Windows of 7, 28 and 30 days; 7 is the default because 30 days of history is all that is kept, so it is the only window with an equal period behind it to compare against.

Three constraints are shown rather than hidden. Google's data lags about two days, so a "data through" date is always on screen. A refresh is allowed once every 24 hours, because Google itself refreshes daily and a second pull would spend quota to return identical numbers. And headline totals are rolled up from the page grain, not by summing the query rows: Google withholds low-volume queries for privacy, so the query breakdown covers a fraction of real traffic. Our own measurement, on Docsbook's own documentation property: 91 impressions when totalled by query against 413 when totalled by page. That is one site on one day and is quoted to show the direction of the gap, not its size on yours.

Why these mechanisms (evidence)#

Limits and open questions#

• IndexNow is not sent for customer sites today. The protocol requires every URL in a submission to share one host, and requires a key file at that host's root. Docsbook hosts that key on the docsbook.io apex only, so pushes go out for Docsbook's own documentation and for showcase sites on the apex — not for <owner>.docsbook.io sites and not for custom domains. Per-tenant key hosting is a separate piece of work and is not built.
• Under question: how fast IndexNow actually is. Docsbook's code comment says "within minutes instead of waiting for their next scheduled crawl". IndexNow's own FAQ will only say it "increases the likelihood that important changes are discovered and crawled faster", and avoids naming a timeline. Treat minutes as a hope, not a measurement — we have published none.
• Google is not an IndexNow participant. indexnow.org names its support as coming from "Microsoft Bing, Naver, Seznam.cz, Yandex, Yep" — five engines, and Google is not among them, on that page or in the protocol documentation. Google discovery still runs on the sitemap and on ordinary crawling.
• Under question: how secret the IndexNow key is. Docsbook treats the key as a public identifier — it is hardcoded in the open, and the key file is served unauthenticated at the apex root, which is what the FAQ requires ("no login required", so a search engine can "confirm domain ownership"). What contradicts the "it is not a secret" reading is IndexNow's own protocol page, which says "Only you and the search engines should know the key and your file key location". Both cannot be fully true of a file any client may fetch. What the key can do if copied is bounded — it says "this host changed", nothing more — but treat "public by construction" as our reading of a protocol that does not say it.
• Search Console coverage stops at Docsbook-hosted hosts. The domain property covers docsbook.io and its subdomains. A site on your own custom domain is not in that property, so no positions are read for it. Serving those numbers needs a per-customer Google authorisation flow, which does not exist yet — use your own Search Console account for a custom domain in the meantime.
• Search Console history is 30 days here. Google keeps considerably more; Docsbook stores a 30-day rolling window per site, which is why the 30-day view has no comparison period.
• Nothing here makes a page rank. Discovery and re-crawl are the parts Docsbook can automate. Whether a crawled page ranks is Google's call on your content, and no timer on this page changes it.

Related#

• SEO — what Docsbook does for search visibility
• How Docsbook builds the head of a page
• Analytics
• GEO — being cited by AI assistants
• AI translations