1. robots.txt Controls Crawling; Meta Robots & Canonicals Control Indexing
A frequent technical SEO mistake is blocking a URL in robots.txt (Disallow: /private) while expecting Google to drop it from search results or read a <meta name="robots" content="noindex"> tag inside that page. If bots are disallowed from fetching the page, they never see the noindex directive.
Use robots.txt strictly to protect infinite faceted-search parameters, internal APIs, and admin endpoints while pointing crawlers to your authoritative Sitemap: index URL.
- Strip campaign noise (
utm_*,fbclid,gclid,ref) in<link rel="canonical"> - Keep trailing slash policy and HTTPS scheme 100% consistent across sitemaps and canonicals
- For downloadable PDFs without HTML
<head>, sendLink: <https://...>; rel="canonical"HTTP headers
2. Clean XML Sitemaps & Reciprocal Hreflang Annotations
Only list 200-OK, indexable canonical URLs inside sitemap.xml, avoiding 301 redirects, 404s, or non-canonical duplicates that erode crawl trust. Include accurate W3C ISO-8601 <lastmod> dates only when page content meaningfully changes.
For multi-language or multi-region sites (en-US, en-GB, fr-CA), every page in a localized set must declare both a self-referencing hreflang link and return links to every alternate version plus x-default fallback.
Technical References & Standards Basis
- Google Search Central — Robots Exclusion Protocol (RFC 9309) & Canonicalization
- Sitemaps.org XML Schema 0.9 & ISO 639-1 / ISO 3166-1 Hreflang Matrix Standards
