About
The Website Crawler skill provides a high-performance engine designed to map entire website architectures, discover internal/external links, and generate comprehensive site reports. By combining a TypeScript/Bun frontend with a concurrent Go backend, it efficiently handles large-scale crawls while offering advanced features like automatic sitemap discovery, token-bucket rate limiting, and persistent checkpointing to resume progress on long-running tasks. It is ideal for developers and SEO professionals needing a structured overview of a site's content and connectivity.
