PyScrappy

•
265
•

Provides a Python web scraping toolkit and MCP server to deliver clean, structured web data to AI agents from any URL or specialized scrapers.

PyScrappy is an out-of-the-box Python toolkit designed for efficient web scraping, allowing users to extract structured data from any URL or leverage its extensive collection of built-in scrapers for popular platforms like Wikipedia, IMDB, and Yahoo Finance. It also functions as a Model Context Protocol (MCP) server, enabling AI agents to seamlessly access and utilize its scraping capabilities as dynamic tools, providing them with clean, organized web information for various analytical and generative tasks.

Key Features

01Generic web scraping for any URL with auto-extraction of text, links, images, tables, and metadata.
02Advanced scraping features including JavaScript rendering with Playwright, auto-pagination, and custom CSS selectors.
03Optional Model Context Protocol (MCP) server to expose scrapers as tools for AI agents (e.g., Claude, Cursor).
04Over 20 specialized built-in scrapers for platforms like Wikipedia, IMDB, Yahoo Finance, news feeds, and e-commerce sites.
05Robust configuration options with proxy support, scraping-API integration, concurrent scraping, rate-limiting, and caching.
0690 GitHub stars

Use Cases

01Provide AI agents with structured real-time web data for research, summarization, or content generation tasks.
02Gather data from e-commerce sites, news feeds, or social media for market analysis, competitive intelligence, or trend monitoring.
03Automate data extraction from various websites using custom CSS selectors or specialized built-in scrapers for specific platforms.