연결
Web Scraper icon

Web Scraper

bylmorg
•
1
데이터 과학 및 ML
웹 스크래핑 및 데이터 수집
개발자 도구

Extracts web page content for AI and LLM agents, compatible with LangChainGo and MCP integration.

소개

Leveraging Google Chrome's headless APIs, this web scraping package efficiently extracts content from web pages, including those that heavily rely on JavaScript like single-page applications. It is specifically designed for integration with AI and LLM agents, offering dual support for direct use within Go applications via LangChainGo and deployment as an MCP server, ensuring broad applicability for data collection tasks.

주요 기능

  • Parses content from JavaScript-heavy sites and single-page applications (SPAs)
  • Includes a fallback mode to Go's HTTP user agent if Chrome is not available
  • 1 GitHub stars
  • Scrapes web pages using Google Chrome's headless APIs
  • Offers easy integration with Go applications via LangChainGo
  • Can be deployed and run as an MCP server

사용 사례

  • Integrating robust web scraping capabilities into Go-based applications through LangChainGo
  • Collecting structured and unstructured data for AI and LLM agent training and inference
  • Providing web content extraction services to MCP clients like Claude Desktop or Visual Studio Code