Conectar
Web Scraper icon

Web Scraper

bylmorg
•
1
Ciencia de Datos y ML
Web Scraping y Recopilación de Datos
Herramientas para Desarrolladores

Extracts web page content for AI and LLM agents, compatible with LangChainGo and MCP integration.

Acerca de

Leveraging Google Chrome's headless APIs, this web scraping package efficiently extracts content from web pages, including those that heavily rely on JavaScript like single-page applications. It is specifically designed for integration with AI and LLM agents, offering dual support for direct use within Go applications via LangChainGo and deployment as an MCP server, ensuring broad applicability for data collection tasks.

Características Principales

  • Parses content from JavaScript-heavy sites and single-page applications (SPAs)
  • Includes a fallback mode to Go's HTTP user agent if Chrome is not available
  • 1 GitHub stars
  • Scrapes web pages using Google Chrome's headless APIs
  • Offers easy integration with Go applications via LangChainGo
  • Can be deployed and run as an MCP server

Casos de Uso

  • Integrating robust web scraping capabilities into Go-based applications through LangChainGo
  • Collecting structured and unstructured data for AI and LLM agent training and inference
  • Providing web content extraction services to MCP clients like Claude Desktop or Visual Studio Code