Web Scraper

概要

Leveraging Google Chrome's headless APIs, this web scraping package efficiently extracts content from web pages, including those that heavily rely on JavaScript like single-page applications. It is specifically designed for integration with AI and LLM agents, offering dual support for direct use within Go applications via LangChainGo and deployment as an MCP server, ensuring broad applicability for data collection tasks.

主な機能

• Parses content from JavaScript-heavy sites and single-page applications (SPAs)
• Includes a fallback mode to Go's HTTP user agent if Chrome is not available
• 1 GitHub stars
• Scrapes web pages using Google Chrome's headless APIs
• Offers easy integration with Go applications via LangChainGo
• Can be deployed and run as an MCP server

ユースケース

• Integrating robust web scraping capabilities into Go-based applications through LangChainGo
• Collecting structured and unstructured data for AI and LLM agent training and inference
• Providing web content extraction services to MCP clients like Claude Desktop or Visual Studio Code