buildmymcpserver/apps/web/app/robots.ts
Marco Sadjadi b843394d0f
Some checks failed
Deploy to Production / deploy (push) Failing after 46s
feat(web): full SEO stack — metadata, JSON-LD, sitemap, robots, OG image
Ported and adapted from the BuildMyDiscord SEO setup:

- lib/seo.ts — single source for site constants, the FAQ data (shared by
  the rendered FAQ and the FAQPage schema so they never drift) and JSON-LD
  builders.
- Rich root metadata: title template, keywords, Open Graph, Twitter card,
  robots directives, canonical.
- JSON-LD: Organization + WebSite + SoftwareApplication sitewide, FAQPage
  on the landing page. No AggregateRating — there are no real reviews yet.
- app/robots.ts — allow all, explicit allow-list for AI answer-engine
  crawlers (GPTBot, ClaudeBot, PerplexityBot, …), disallow private routes.
- app/sitemap.ts — every public marketing + docs route.
- app/opengraph-image.tsx — monochrome on-brand 1200x630 share card.
- app/manifest.ts + public/llms.txt.
- Per-page metadata for pricing, changelog, security, privacy, terms,
  docs, templates and status.
- opengraph-image + apple-icon pinned to the edge runtime — next/og
  crashes during a Node-runtime prerender.

Verified: next build passes; /robots.txt, /sitemap.xml,
/manifest.webmanifest and /opengraph-image all generate.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 19:16:40 +02:00

39 lines
978 B
TypeScript

import { SITE_URL } from '@/lib/seo';
import type { MetadataRoute } from 'next';
// AI search/answer crawlers — explicitly allowed so the product surfaces in
// ChatGPT, Claude, Perplexity, Google AI and similar answer engines.
const AI_CRAWLERS = [
'GPTBot',
'OAI-SearchBot',
'ChatGPT-User',
'ClaudeBot',
'anthropic-ai',
'Claude-Web',
'PerplexityBot',
'Perplexity-User',
'Google-Extended',
'CCBot',
'Applebot-Extended',
'Amazonbot',
'meta-externalagent',
'DuckAssistBot',
];
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{
userAgent: '*',
allow: '/',
disallow: ['/api/', '/admin', '/dashboard', '/login'],
},
// Aggressive scraper with no search value.
{ userAgent: 'Bytespider', disallow: '/' },
{ userAgent: AI_CRAWLERS, allow: '/', disallow: ['/api/', '/admin', '/dashboard'] },
],
sitemap: `${SITE_URL}/sitemap.xml`,
host: SITE_URL,
};
}