Conectar

Fetchkit

pr1m8bypr1m8
•
2
•
Herramientas para Desarrolladores
Ciencia de Datos y ML
Web Scraping y Recopilación de Datos

Provides agentic web infrastructure for autonomous fetching, scraping, extraction, and downloading with realistic browser fingerprints and a full crawl-scrape-download pipeline.

Fetchkit empowers AI agents with advanced web interaction capabilities, allowing them to autonomously fetch pages, extract structured data, and download content. It addresses the limitations of existing tools by offering realistic browser fingerprints, a comprehensive crawl-scrape-download pipeline backed by Postgres and MinIO, and deep integration with media downloaders like yt-dlp. Its MCP Server exposes 16 structured tools, enabling AI agents (such as Claude, LangChain, or LangGraph) to interact with the web directly, bypassing traditional scraping complexities and bot detection.

Características Principales

01Multiple HTTP backends (httpx, aiohttp, curl_cffi, cloudscraper) with rate limiting and retry logic.
02Deep downloader integration for media (yt-dlp) and image galleries (gallery-dl) with progress hooks.
03Full event-driven crawl-scrape-download pipeline via Postgres job queues and MinIO object storage.
04MCP Server with 16 structured tools for AI agent integration and Pydantic outputs.
05Realistic browser identity with 11 profiles, consistent headers, TLS fingerprinting, and Cloudflare bypass.
060 GitHub stars

Casos de Uso

01Extract structured data from any webpage using CSS selectors, table parsing, or article extraction.
02Autonomously research topics by enabling AI agents to fetch pages, extract content, and follow links.
03Audit websites by allowing agents to check metadata, robots.txt, sitemaps, and page structure.