Fetchkit empowers AI agents with advanced web interaction capabilities, allowing them to autonomously fetch pages, extract structured data, and download content. It addresses the limitations of existing tools by offering realistic browser fingerprints, a comprehensive crawl-scrape-download pipeline backed by Postgres and MinIO, and deep integration with media downloaders like yt-dlp. Its MCP Server exposes 16 structured tools, enabling AI agents (such as Claude, LangChain, or LangGraph) to interact with the web directly, bypassing traditional scraping complexities and bot detection.
主要功能
01Multiple HTTP backends (httpx, aiohttp, curl_cffi, cloudscraper) with rate limiting and retry logic.
02Deep downloader integration for media (yt-dlp) and image galleries (gallery-dl) with progress hooks.
03Full event-driven crawl-scrape-download pipeline via Postgres job queues and MinIO object storage.
04MCP Server with 16 structured tools for AI agent integration and Pydantic outputs.
05Realistic browser identity with 11 profiles, consistent headers, TLS fingerprinting, and Cloudflare bypass.
060 GitHub stars
使用案例
01Extract structured data from any webpage using CSS selectors, table parsing, or article extraction.
02Autonomously research topics by enabling AI agents to fetch pages, extract content, and follow links.
03Audit websites by allowing agents to check metadata, robots.txt, sitemaps, and page structure.