> ## Documentation Index
> Fetch the complete documentation index at: https://docs.drdj.me/llms.txt
> Use this file to discover all available pages before exploring further.

# SEO and AI Search Optimization

> Canonical SEO baseline for DealDash web pages, crawler controls, and AI-discoverable content.

This runbook defines the baseline that keeps DealDash discoverable in classic search and AI-assisted search while preventing private app routes from indexing.

## Scope

Managed files:

* `/client/index.html`
* `/client/public/robots.txt`
* `/client/public/sitemap.xml`
* `/client/public/llms.txt`
* `/client/public/llms-full.txt`
* `/client/src/components/SeoManager.tsx`
* `/server/lib/agent-readiness.ts`
* `/server/routes/agent-readiness.ts`

## What We Optimize

1. Route-aware metadata for public pages
2. Canonical URLs and Open Graph/Twitter tags
3. Crawl boundaries for private/authenticated pages
4. Machine-readable site map (`sitemap.xml`)
5. AI-crawler context (`llms.txt` and `llms-full.txt`)
6. Agent discovery headers and well-known endpoints

## Public Indexable Routes

* `/welcome`
* `/features`
* `/ai-link-shortener`
* `/pricing`
* `/about`
* `/contact`
* `/terms`
* `/privacy`
* `/cookies`

Everything else defaults to `noindex, nofollow` via `SeoManager`.

## Why `SeoManager` Exists

DealDash is an SPA. Static head tags alone are not enough for route-level quality signals. `SeoManager` updates title, description, robots, Open Graph, Twitter, and canonical tags by route.

Location:

* `/client/src/components/SeoManager.tsx`

## Crawl Policy

`robots.txt` allows public marketing pages and blocks private app surfaces:

* `/api/*`
* authenticated product routes (`/dashboard`, `/deals`, `/contacts`, etc.)
* auth pages (`/login`, `/auth/*`)
* share viewer route (`/s/*`)

Location:

* `/client/public/robots.txt`

Robot groups explicitly include `GPTBot`, `ClaudeBot`, `Claude-User`, and
`PerplexityBot`. The policy also publishes
`Content-Signal: ai-train=no, search=yes, ai-input=yes`.

## AI Search Baseline

`llms.txt` publishes concise product context and approved public URLs for AI systems. `llms-full.txt` adds setup details and troubleshooting.

Location:

* `/client/public/llms.txt`
* `/client/public/llms-full.txt`

Policy:

1. Keep claims factual and non-promotional.
2. Link only public, stable routes.
3. Link only public discovery endpoints, never private route examples.
4. Include the current agent discovery endpoints.

## Agent Discovery Baseline

Global responses include:

* `Link: </.well-known/api-catalog>; rel="api-catalog"; type="application/linkset+json"`
* `Link: </docs/api>; rel="service-doc"`
* `Link: </openapi.json>; rel="service-desc"; type="application/openapi+json"`
* `Content-Signal: ai-train=no, search=yes, ai-input=yes`

Public pages return Markdown when requests send `Accept: text/markdown`.

Discovery endpoints:

* `/.well-known/api-catalog`
* `/openapi.json`
* `/.well-known/oauth-authorization-server`
* `/.well-known/oauth-protected-resource`
* `/.well-known/mcp/server-card.json`
* `/.well-known/mcp/drdj-shortener/server-card.json`
* `/.well-known/agent-card.json`
* `/.well-known/agent-skills/index.json`
* `/.well-known/agent-skills/dealdash-agent-bridge/SKILL.md`
* `/.well-known/agent-skills/drdj-shortener/SKILL.md`
* `/.well-known/http-message-signatures-directory`

`/.well-known/http-message-signatures-directory` is Web Bot Auth-ready but only
becomes a signed bot identity when production Ed25519 public/private key
material is configured through `WEB_BOT_AUTH_PUBLIC_JWK_JSON` or
`WEB_BOT_AUTH_JWKS_JSON` plus `WEB_BOT_AUTH_PRIVATE_KEY_PEM` or
`WEB_BOT_AUTH_PRIVATE_KEY_BASE64`.

## Verification Checklist

```bash theme={null}
pnpm check
```

Manual probes:

```bash theme={null}
curl -I https://dealdash.tech/robots.txt
curl -I https://dealdash.tech/sitemap.xml
curl -I https://dealdash.tech/llms.txt
curl -I https://dealdash.tech/.well-known/api-catalog
curl -H 'Accept: text/markdown' https://dealdash.tech/welcome
curl https://dealdash.tech/.well-known/mcp/server-card.json
curl https://drdj.me/.well-known/mcp/drdj-shortener/server-card.json
curl https://dealdash.tech/.well-known/agent-card.json
curl https://dealdash.tech/.well-known/agent-skills/dealdash-agent-bridge/SKILL.md
curl https://drdj.me/.well-known/agent-skills/drdj-shortener/SKILL.md
curl -I https://dealdash.tech/.well-known/http-message-signatures-directory
```

The standalone shortener probes use its canonical `https://drdj.me` host; the
full DealDash discovery and Agent Bridge probes use `https://dealdash.tech`.

Browser check:

1. Open each public marketing page.
2. Inspect `<title>`, canonical, and robots tags in DevTools.
3. Confirm private routes resolve to `noindex, nofollow`.

## Update Rule

If a public marketing route is added/removed, update all three:

1. `/client/src/components/SeoManager.tsx`
2. `/client/public/sitemap.xml`
3. `/client/public/llms.txt`
4. `/client/public/llms-full.txt`