Crawling and indexing are different stages
A robots.txt Disallow limits fetching. Because the crawler cannot read page content or a noindex directive, it is the wrong tool when the sole goal is removal from search.
For public HTML that should not be indexed, allow crawling so the page can return noindex. Protect sensitive content with authentication, not either search directive.
Canonical is a strong hint, not a command
A canonical proposes the representative among duplicate or highly similar URLs. Signals conflict when redirects, internal links, and sitemaps point elsewhere.
Do not consolidate unrelated pages to control indexing. Search systems compare content and may ignore an inappropriate canonical.
Hreflang is a graph of equivalent alternatives
If the Korean page points to the English page, the English page must point back. Include the page itself and use supported language or region codes.
Each language page should generally canonicalize to itself. Canonicalizing Korean to English contradicts the language-alternative relationship.
- robots.txt: may this URL be fetched?
- noindex: may this response be indexed?
- canonical: which duplicate URL is representative?
- hreflang: which URL is the language or regional alternative?
Audit one page as a single signal line
Write fetch access → HTTP status → meta robots → canonical → hreflang → sitemap inclusion → internal links on one line. Conflicts become obvious; keep redirect target and final URL as separate fields.
On larger sites, test representative URLs per template and retain pre/post-deploy diffs. Individual inspection alone catches shared-layout regressions too late.
Pre-release checklist
- ✓ Are noindex pages still crawlable?
- ✓ Do canonical, internal links, and sitemap agree?
- ✓ Does each language page use a self-canonical?
- ✓ Does hreflang include self and reciprocal references?
- ✓ Are sensitive pages protected with authentication rather than search directives?
Frequently asked questions
Is Disallow plus noindex safer?
Usually the opposite. Disallow can prevent the crawler from seeing noindex. If removal from indexing is the goal, allow fetching and return noindex.
Is x-default required?
No, but it is useful for a language selector or default global page. With a deterministic default locale, it can point consistently to that route.