A sitemap answers one question: which URLs do you consider canonical and want in the index? Everything that goes wrong with sitemaps comes from answering a different question by accident.
What belongs in it
Only URLs that return 200, are canonical to themselves, are not blocked by robots.txt, and are not set to noindex. Every URL that fails one of those tests is a contradiction: you are asking for indexing and refusing it in the same breath.
Google treats sitemap inclusion as a hint about canonicalisation. A sitemap full of non-canonical URLs actively confuses that signal rather than merely wasting space.
The common breakages
- Redirecting URLs. The old address of a page you moved. Common after a migration when the sitemap is generated from a stale database table.
- Parameter variants.
?sort=price,?utm_source=…and session IDs. Each is a separate URL to a crawler, and none of them is canonical. - Noindexed pages. Tag archives, thin filter pages, and paginated series that were excluded on purpose.
- Mixed protocol or host. Listing
http://URLs on an HTTPS site, or mixing www and non-www. - Fabricated lastmod. Setting every entry to today's date on every regeneration. Google learns to ignore the field, and you lose a genuinely useful signal on the pages that did change.
- Size overruns. A single sitemap is capped at 50,000 URLs or 50 MB uncompressed. Past either, split it and use a sitemap index.
Fields that matter, and fields that do not
loc is required and must be absolute. lastmod is used, and is worth getting right: it should reflect meaningful content change, not template edits or a nightly rebuild.
changefreq and priority are, for practical purposes, ignored by Google. They are valid, harmless, and not worth arguing about in a planning meeting. Set them consistently and move on.
Where to put it
At the root: /sitemap.xml. Reference it from robots.txt with a Sitemap: line, and submit it in Search Console. Robots.txt discovery works for every crawler; Search Console submission gives you the indexing report, which is the part you actually need.
Large sites and index files
Past a few thousand URLs, split by content type rather than arbitrarily: one for products, one for articles, one for category pages. The Search Console coverage report is broken down per sitemap, so a meaningful split turns “3,000 pages not indexed” into “2,900 of the not-indexed pages are products”, which points at a cause.
Checking yours
Run your live sitemap through the sitemap validator. It expands index files, counts URLs, and flags malformed entries and size overruns. If you do not have one yet, the XML sitemap generator crawls your site, respects robots.txt, and hands back a file you can upload. Then confirm nothing in it is blocked with the robots.txt tester.