SEO Crawler

About

The SEO Crawler is a self-contained, powerful website crawler and SEO analyzer designed to integrate seamlessly with AI assistants like Claude. Built on the MCP protocol SDK, it locally processes and stores comprehensive website data, including HTML, metadata, and link relationships, in an embedded SQLite database. Users can initiate crawls and then leverage natural language queries through their AI assistant for detailed SEO analysis, identifying critical issues, content quality problems, technical SEO pitfalls, security vulnerabilities, and optimization opportunities. For larger sites or background tasks, the tool also offers a robust command-line interface, allowing for flexible crawl execution and subsequent AI-powered data analysis.

Key Features

• Embedded Better-SQLite3 database stores every crawled page's HTML, metadata, headers, link relationships, and site structure.
• Seamless integration with AI assistants (e.g., Claude) for natural language interaction and deep analytical insights.
• Smart request scheduling, queue management, automatic retry logic, and error handling for robust crawling.
• Custom SQL analysis engine with 25+ specialized queries detecting content issues, technical SEO problems, security vulnerabilities, and optimization opportunities.
• 5 GitHub stars
• CLI mode for direct terminal usage, enabling background processing and batch crawls for large sites.

Use Cases

• Automate large-scale website crawls via the terminal for background processing, then use an AI assistant for intelligent analysis.
• Conduct security audits to check for missing security headers, unsafe external links, and other vulnerabilities.
• Perform complete technical SEO audits for websites, identifying critical errors and optimization opportunities.