Media Crawler
Collects public data from popular self-media platforms like Xiaohongshu, Bilibili, Douyin, Kuaishou, Weibo, Tieba, and Zhihu, integrating with AI assistants.
概要
Media Crawler is a powerful multi-platform self-media data collection tool designed for personal efficiency. It provides a robust service architecture for scraping public information from major platforms such as Bilibili, Xiaohongshu, Douyin, Kuaishou, Weibo, Tieba, and Zhihu. Leveraging FastMCP, it enables AI assistants like Claude and ChatGPT to directly invoke its crawling capabilities. The tool emphasizes ease of use, a clear architecture, and features like persistent browser contexts, dynamic task-level configurations, and a streamlined login process, all while offering clear extension points for customization.
主な機能
- Native MCP support for direct AI assistant integration
- Persistent browser context for faster data retrieval
- Modular architecture with clear Endpoint, Service, and Crawler layers
- Task-level configuration using Pydantic Settings
- Decoupled login flow with real-time status and background polling
- 7 GitHub stars
ユースケース
- Automated data collection from multiple social media platforms for personal analysis
- Integrating AI assistants with web scraping capabilities for enhanced workflows
- Developing custom data collection services with an extensible framework
