Skip to main content
This runbook defines the baseline that keeps DealDash discoverable in classic search and AI-assisted search while preventing private app routes from indexing.

​
Scope

Managed files:
  • /client/index.html
  • /client/public/robots.txt
  • /client/public/sitemap.xml
  • /client/public/llms.txt
  • /client/public/llms-full.txt
  • /client/src/components/SeoManager.tsx
  • /server/lib/agent-readiness.ts
  • /server/routes/agent-readiness.ts

​
What We Optimize

  1. Route-aware metadata for public pages
  2. Canonical URLs and Open Graph/Twitter tags
  3. Crawl boundaries for private/authenticated pages
  4. Machine-readable site map (sitemap.xml)
  5. AI-crawler context (llms.txt and llms-full.txt)
  6. Agent discovery headers and well-known endpoints

​
Public Indexable Routes

  • /welcome
  • /features
  • /ai-link-shortener
  • /pricing
  • /about
  • /contact
  • /terms
  • /privacy
  • /cookies
Everything else defaults to noindex, nofollow via SeoManager.

​
Why SeoManager Exists

DealDash is an SPA. Static head tags alone are not enough for route-level quality signals. SeoManager updates title, description, robots, Open Graph, Twitter, and canonical tags by route. Location:
  • /client/src/components/SeoManager.tsx

​
Crawl Policy

robots.txt allows public marketing pages and blocks private app surfaces:
  • /api/*
  • authenticated product routes (/dashboard, /deals, /contacts, etc.)
  • auth pages (/login, /auth/*)
  • share viewer route (/s/*)
Location:
  • /client/public/robots.txt
Robot groups explicitly include GPTBot, ClaudeBot, Claude-User, and PerplexityBot. The policy also publishes Content-Signal: ai-train=no, search=yes, ai-input=yes.

​
AI Search Baseline

llms.txt publishes concise product context and approved public URLs for AI systems. llms-full.txt adds setup details and troubleshooting. Location:
  • /client/public/llms.txt
  • /client/public/llms-full.txt
Policy:
  1. Keep claims factual and non-promotional.
  2. Link only public, stable routes.
  3. Link only public discovery endpoints, never private route examples.
  4. Include the current agent discovery endpoints.

​
Agent Discovery Baseline

Global responses include:
  • Link: </.well-known/api-catalog>; rel="api-catalog"; type="application/linkset+json"
  • Link: </docs/api>; rel="service-doc"
  • Link: </openapi.json>; rel="service-desc"; type="application/openapi+json"
  • Content-Signal: ai-train=no, search=yes, ai-input=yes
Public pages return Markdown when requests send Accept: text/markdown. Discovery endpoints:
  • /.well-known/api-catalog
  • /openapi.json
  • /.well-known/oauth-authorization-server
  • /.well-known/oauth-protected-resource
  • /.well-known/mcp/server-card.json
  • /.well-known/mcp/drdj-shortener/server-card.json
  • /.well-known/agent-card.json
  • /.well-known/agent-skills/index.json
  • /.well-known/agent-skills/dealdash-agent-bridge/SKILL.md
  • /.well-known/agent-skills/drdj-shortener/SKILL.md
  • /.well-known/http-message-signatures-directory
/.well-known/http-message-signatures-directory is Web Bot Auth-ready but only becomes a signed bot identity when production Ed25519 public/private key material is configured through WEB_BOT_AUTH_PUBLIC_JWK_JSON or WEB_BOT_AUTH_JWKS_JSON plus WEB_BOT_AUTH_PRIVATE_KEY_PEM or WEB_BOT_AUTH_PRIVATE_KEY_BASE64.

​
Verification Checklist

pnpm check
Manual probes: 
curl -I https://dealdash.neonoir.ai/robots.txt
curl -I https://dealdash.neonoir.ai/sitemap.xml
curl -I https://dealdash.neonoir.ai/llms.txt
curl -I https://dealdash.neonoir.ai/.well-known/api-catalog
curl -H 'Accept: text/markdown' https://dealdash.neonoir.ai/welcome
curl https://dealdash.neonoir.ai/.well-known/mcp/server-card.json
curl https://dealdash.neonoir.ai/.well-known/mcp/drdj-shortener/server-card.json
curl https://dealdash.neonoir.ai/.well-known/agent-card.json
curl https://dealdash.neonoir.ai/.well-known/agent-skills/dealdash-agent-bridge/SKILL.md
curl https://dealdash.neonoir.ai/.well-known/agent-skills/drdj-shortener/SKILL.md
curl -I https://dealdash.neonoir.ai/.well-known/http-message-signatures-directory
For the public shortener domain after deployment, repeat the shortener probes against https://drdj.me. Browser check:
  1. Open each public marketing page.
  2. Inspect <title>, canonical, and robots tags in DevTools.
  3. Confirm private routes resolve to noindex, nofollow.

​
Update Rule

If a public marketing route is added/removed, update all three:
  1. /client/src/components/SeoManager.tsx
  2. /client/public/sitemap.xml
  3. /client/public/llms.txt
  4. /client/public/llms-full.txt