NNicheStackTutorials
HugoVitePressSvelteKitAstroFrontend ToolsStatic HostingSEO & Web VitalsAd Monetization
← All tutorials
SEO & Web Vitals

robots.txt for Ad-Monetized Static Blogs: What to Allow and Block

Blocked thin tag pages and ad test routes without hurting AdSense crawlers. Includes copy-paste templates per host.

·10 min read·SEO & Web Vitals
robots.txt for Ad-Monetized Static Blogs: What to Allow and Block cover

Mistake I made: Disallow: / during staging, forgot to update on prod launch. Took 3 weeks to recover rankings. Second mistake: Blocking /api/ broke nothing, but blocking /ads.txt nearly killed AdSense verification.

robots.txt is not security

It guides crawlers. Malicious bots ignore it. Never put secrets in disallowed paths thinking they're hidden.

Template for ad-monetized tutorial blog

User-agent: *
Allow: /

# Thin taxonomy pages (also set noindex in HTML)
Disallow: /tags/
Disallow: /page/

# Staging / preview (if accidentally deployed)
Disallow: /draft/
Disallow: /preview/

# Search result pages (if any client-side search creates URLs)
Disallow: /search?

# NEVER block these
# Allow: /ads.txt
# Allow: /app-ads.txt

Sitemap: https://example.com/sitemap.xml

ads.txt must be crawlable at root. AdSense checks it.

AdSense / ad network crawlers

Mediavine, Ezoic, AdSense use standard Google crawlers. No special User-agent: Mediapartners-Google rules needed for static sites.

If using Ad Manager with /ad/ paths for creatives, don't disallow /ad/.

Combine robots.txt with meta noindex

For tag pages you disallow:

<meta name="robots" content="noindex, follow">

Belt and suspenders. follow passes link equity to linked posts.

Hugo: auto robots.txt

layouts/robots.txt:

User-agent: *
Allow: /
Disallow: /tags/
Sitemap: {{ .Site.BaseURL }}sitemap.xml

Netlify / Cloudflare staging

Use X-Robots-Tag: noindex header on preview deploys instead of robots.txt:

# _headers
/*
  X-Robots-Tag: noindex

Only on deploy-preview-* branch — not production.

Crawl budget reality check

Small blogs (< 1000 pages) don't have crawl budget problems. Disallow is for:

  • Duplicate pagination (/page/2/)
  • Internal search URLs
  • Parameter faceting (?sort=date)
  • Admin/test routes

Canonical + robots alignment

If noindex on a page, don't include it in sitemap. GSC warns about this.

# Find sitemap URLs that have noindex
grep -l 'noindex' public/posts/**/*.html | while read f; do
  slug=$(echo $f | sed 's/public//;s/index.html//')
  grep -q "$slug" public/sitemap.xml && echo "CONFLICT: $slug"
done

Verification

curl -s https://example.com/robots.txt
curl -s https://example.com/ads.txt
curl -I https://example.com/ads.txt | grep HTTP

Search Console → Settings → robots.txt → verify Google can fetch.

Results

After blocking /tags/ + noindex:

Indexed pages: dropped 89 thin URLs
Avg position on money posts: +2.3 (less dilution)
AdSense revenue: unchanged (users still browse tags, just not from Google)

robots.txt is a scalpel. Block junk, never block money or ads.txt.

Tags: robots.txtCrawl BudgetAdSensenoindex