Skip to content
WebInsights

A sitemap that matches what you want indexed

You have a sitemap when the list is what you want in the index — not everything that can answer 200.

Sitemap.xml is the list of URLs you want in the index. If it holds noindex, staging and parameters — or forgets the pages you want found — you have asked Google to guess.

13 min read
Stylised browser window with a URL list where only the canonical, indexable pages are marked — not staging, parameters or noindex.

SEO & web

If you are searching for sitemap.xml, or for sitemap does not match index, you are typically in one place: Search Console says you submitted URLs that are not indexed, or that you have pages in the index you never put on the list. The canonical post is choosing one URL. The JSON-LD post is markup that mirrors HTML. The redirects post is chains that rot. The relaunch post is the cutover map. This one is the list. A sitemap is a promise: these URLs we want crawled as candidates for the index. A generator that dumps everything — tags, parameters, preview, noindex, the other host — is not a sitemap. It is a dump you called SEO.

Google treats the sitemap as a hint about what you yourself think is important, and as a way to discover URLs on a new or thinly linked site. It is not a command to index. If the list fights noindex, canonical, robots.txt, you have three votes. The crawl guesses. The guess is rarely the page you updated on Monday. You have not 'set a sitemap'. You have published a file that either matches what you want found — or contradicts it.

TL;DR

The main takeaways

  • Sitemap.xml is the list of canonical, indexable URLs you want found — not everything that answers 200.
  • Noindex, staging, UTM, the other host and thin archives on the list is asking for crawl you do not want in the index.
  • lastmod that is always 'now' is noise. An honest date beats a generator that lies on every build.
  • The list must agree with canonical, hreflang and robots.txt. Disagreement is three votes Google has to count.
  • The canonical post is the address. JSON-LD is the script. Redirect is the hop. Relaunch is cutover. This one is the list.
  • You have a sitemap when you can say: every URL here may be indexed. Not when the plugin is on.

Why this matters at all

A site that submits hundreds of URLs, a quarter of them tag archives, parameters and the other locale host, does not have 'good coverage'. It has crawl noise. On a new domain the budget is small. Every URL on the list is a request for attention. If the request is a thin tag page, a UTM copy or a noindex you forgot to drop from the generator, you have spent attention on something you yourself said no to. The coverage report becomes an argument you cannot explain.

Teams do it because the first generator 'worked'. The WordPress plugin turned XML sitemap on. Next.js `app/sitemap.ts` mapped every slug. Preview URLs came along because the origin was `VERCEL_URL`. lastmod was set to `new Date()` so 'Google sees it as fresh'. Nobody chose. Everybody added. Three months later you have submitted URLs that are noindex, URLs that canonicalise away, and you have forgotten the three landing pages you actually want found because they are not in the collection the generator knows.

The damage is concrete. 'Discovered – currently not indexed' on pages you never linked. 'Crawled – currently not indexed' on thin tags you sent yourself. A staging host in the index because the sitemap pointed there for a week. lastmod that screams 'changed' on pages you have not touched, so the crawl stops trusting the date. A list that is missing the canonical form and only knows the slash variant you redirect. The canonical post is the choice. This one is that the list must know the choice.

We write this because 'we have a sitemap' has become a checkbox you tick when `/sitemap.xml` answers 200. An honest sitemap is boring: only canonical, public, indexable URLs, honest lastmod, hreflang that matches the page, submitted in one place. Without that choice you have a file. You do not have a list Google can use.

What a sitemap actually is — and what a generator is not

A sitemap is a statement: these URLs are the canonical candidates we want discovered. Absolute URLs. The host you chose. The slash rule you chose. The locales you actually publish. Not preview. Not parameters. Not the page you noindexed because it is thin. The difference is not whether the file parses. The difference is whether every line is a URL you will stand behind in the index.

It is not the same as 'the plugin is on'. Yoast, Rank Math, a Next.js map over every dynamic route — they write what you feed them. If you feed them tag archives with one post, search-result URLs, sort parameters, they write that. The generator is the pen. You are the editor. The canonical post is choosing the address. This one is choosing who even makes the team.

It is also not the same as robots.txt. Robots says where the crawl must not go. The sitemap says where you would like it to go. A URL that is disallowed and also listed is a contradiction. A URL that is noindex and listed is a request to crawl a document you said should not be in the index. You can have reasons to crawl without indexing. Then the URL does not belong in the sitemap. It belongs in internal navigation, if you want humans to find it at all.

And it is not 'the more URLs, the better'. On a young site it is the opposite. Every thin URL competes for crawl with the page you actually wrote. The relaunch post is parity at cutover. The redirects post is the hop. This one is that the list after cutover must not hold the old URLs you just 301'd, and must not forget the new ones you just published.

What typically goes wrong when someone says the sitemap is set

The first failure is listing variants. www and apex. Slash and not. HTTP. UTM. Pagination you canonicalise to page 1. The canonical post is choosing one. A sitemap that lists the other three is sending the crawl into hops and duplicates. List the one you chose. Let 301 do the rest. A generator that takes `req.url` and writes it will write the preview host when you build in the wrong place.

The second failure is lastmod that lies. `new Date()` on every build. A CMS that sets 'now' when someone saves without changing content. The crawl learns to ignore the field. Then it does not help the day you actually fix the H1. Either lastmod is the date the content changed — and you mean it — or you omit the field. A lying date is worse than no date. The JSON-LD post is datePublished matching the page. Same honesty here.

The third failure is forgetting what you want found. Landing pages that are not in the collection the generator maps. A new insight that only joins when someone regenerates a static sitemap you committed three weeks ago. An image sitemap you never had, and a video sitemap you pasted in empty. Dynamic sitemap from the same data source as the routes. If the page can render as indexable, it can be listed. If it cannot, it must not. No manual XML you forget.

The fourth failure is hreflang in the sitemap that does not match the page. The DA row points at the EN sister in the sitemap, but the page points a third place — or has no hreflang. Two votes. Relaunch is cutover. Canonical is the address. This one is that sitemap alternates are the same URLs as `hreflang` in HTML. A generator that guesses locale by swapping `/da/` for `/en/` on a page that does not exist in EN is listing 404s as siblings.

An edited list vs. a dump you call coverage

The honest sitemap

Canonical, indexable URLs. Honest lastmod. Hreflang that matches HTML. Submitted in one place.

  • Every URL may go in the index — noindex and staging are not on the list
  • Only the host and slash form you chose as canonical
  • lastmod is the content date, or the field is omitted
  • Is boring to maintain because it comes from the same data source as the routes

The hoped-for dump

Plugin on, every tag, every parameter, lastmod = now, and the preview host for a week.

  • Thin archives and search URLs 'so they get found too'
  • Noindex pages and 301 sources on the list
  • lastmod that screams changed on every deploy
  • Is crawl noise you call SEO. The coverage report becomes an argument
sitemap.tsts
// Right: only canonical, indexable URLs from the same source as the routes
export default function sitemap(): MetadataRoute.Sitemap {
  return allPublicPages
    .filter((p) => p.indexable)
    .map((p) => ({
      url: canonicalAbsolute(p),
      lastModified: p.updatedAt, // real content date, not new Date()
      alternates: { languages: p.hreflang },
    }));
}

// Wrong: everything that can 200, plus 'now' as lastmod
// return [{ url: previewHost + path, lastModified: new Date() }]

How to decide what may stand on the list — before you generate the file

Without this order you get another dump you call a sitemap.

  1. 01

    Write who may go in the index, as a rule you can point at

    Public, canonical, not noindex, not a search result, not preview. If you cannot say the rule in one sentence, the generator will guess. The canonical post is the URL shape. This one is membership.

  2. 02

    Generate from the same data source as the routes, not from XML you committed

    Next.js `app/sitemap.ts`, a CMS field, a plugin you configured — the pen changes. If an indexable page can render, it should be listable without a manual pass. A static file you forget is how new insights land off the list.

  3. 03

    Make lastmod honest or omit it

    The content update. Not build time. Not 'someone pressed save'. The crawl trusts the field until you teach it not to. JSON-LD dateModified is the same honesty. A lie in one is a lie in the other.

  4. 04

    Check the list against canonical, robots and noindex before you submit

    Spot-check: open five URLs from the sitemap. Are they the canonical form? Are they indexable? Do they redirect? A URL that 301s does not belong. A URL that is disallowed does not belong. Submit one sitemap index, not three plugins each posting to Search Console.

  5. 05

    Read coverage as an argument you can explain, not as weather

    Submitted not indexed: why did you send them? Crawled not indexed: were they thin, or did you send noise? Missing in sitemap: why can the route render without the list knowing it? Relaunch is the cutover map. This one is the habit after cutover.

“You have a sitemap when the list is what you want in the index — not everything that can answer 200.”

Tags, images, locales and lastmod — same word, four jobs

Tag and category archives are the job the generator loves and the crawl on a young site hates. An archive with one post is a thin page you asked crawl for. Either the archive is a document you want found (so write it, and list it), or it is navigation (so do not list it, and let internal links carry). We keep thin tag pages out of the sitemap until they have substance. That is a choice. The plugin default is not that choice.

Image and video sitemaps are the job you paste on when someone has read that 'images should be in the sitemap'. An image sitemap is the images you want in Image Search, with the page they belong to. It is not dumping every `next/image` URL. Without real images you want found that way, the file is noise. The JSON-LD post is whether the script mirrors the page. This one is whether the list mirrors the images you actually show.

Locale pairs are the job hreflang already has on the page. Sitemap `alternates.languages` is the same truth, not another one. The DA URL is canonical for DA. The EN sister is in the map if it exists. Do not guess an EN path. The canonical post is that a locale does not canonicalise to the other locale. This one is that the list does not invent siblings.

lastmod is the job you either mean or drop. An honest date on a post you edited is a signal. A date that jumps on every deploy is shouting. The relaunch date on every URL at once is a cutover signal you can defend once — not a habit. After that: the content date, or nothing.

What you should typically do in the situation you are in

Left is what you typically say. Right is the next step we typically recommend.

  • Situation

    Coverage shows 'submitted but not indexed' on tags, parameters and old URLs

    Next step

    Take them off the list. Noindex and 301 sources do not belong in the sitemap. Crawl budget is not a dump

  • Situation

    The sitemap lists the preview host or both www and apex

    Next step

    Hardcode the canonical origin. The canonical post is the choice. The list knows only the choice

  • Situation

    lastmod is new Date() on every build

    Next step

    Use the content updatedAt, or omit the field. A lying date trains the crawl to ignore you

  • Situation

    New landing pages and insights are missing from the sitemap for weeks

    Next step

    Generate from the same data source as the routes. Committed XML is a list you forget

  • Situation

    Hreflang in the sitemap points at paths the page does not have

    Next step

    Only locales that actually render. Do not guess /en/ from /da/. Two 404 siblings are not a language pair

Checklist before you call it a sitemap

If you are missing several of these, the sentence we have a sitemap — the setup is a file you are hoping for.

  • Every URL on the list is canonical, public and indexable — you want it in the index
  • No staging, UTM, noindex, 301 sources or search results on the list
  • lastmod is the content date, or the field is omitted — not build time
  • Hreflang alternates match HTML, and only locales that actually exist
  • The list comes from the same data source as the routes, and you have sampled five URLs
  • Canonical is the address. JSON-LD is the script. Redirect is the hop. Relaunch is cutover. This one is the list

Questions we get again and again

  • Is this the same as canonical URL, JSON-LD or relaunching without losing SEO?

    No. The canonical post is choosing one address and 301 on the rest. JSON-LD is the script that must mirror visible HTML. The redirects post is chains that rot. Relaunch is the cutover map. This one is the list of URLs you want crawled as candidates. You can have one canonical host and still dump tags and parameters into the sitemap. You can have honest markup and a list that points at staging. Four jobs. One does not replace the others.

  • Should every page be in the sitemap, including the ones with noindex?

    No. The sitemap is 'please put this in the index'. Noindex is 'please do not'. Listing them is asking for crawl of a document you rejected. If you need the page to exist for humans, link to it. If you do not want it in the index, keep it off the list. The plugin default that takes 'all published' without reading robots meta is how coverage fills with 'submitted but not indexed'.

  • Is lastmod required?

    No. It is a signal the crawl can use when it trusts you. An honest content date is useful. A date that is 'now' on every deploy is noise that trains it to ignore the field. Prefer omitting it. When you then fix a page, and the date moves with the content, it means something. JSON-LD dateModified is the same rule: mirror the truth, or drop the field.

  • What about a WordPress plugin versus Next.js app/sitemap.ts?

    The job is the same: canonical origin, only indexable URLs, honest lastmod, hreflang that matches. The plugin or the function is the pen. If the pen writes every tag, every parameter and `home_url()` from the environment you build in, it writes it wrong. Configure include/exclude. Hardcode production origin. Do not let 'all published posts' be SEO without you having read what 'all' holds.

  • Does it help to submit the sitemap several times a day?

    No. Submit when you have a real list. Ping when you have published something the list now knows. Posting the same file in a loop is not crawl budget. It is noise in Search Console. On a young site, content and internal links matter more than how often you press submit. The list should be right. Frequency is secondary.

If the sitemap is a dump you call coverage

Let us separate a boring list from a file you are hoping for.

Only the URLs you want in the index beat a plugin that dumps everything.

Why we wrote this

This is how we think — and it's what we build.

Our insights are about the work we actually do. If this hit something you're working on, there's a concrete service that lines up.