Does this skill handle JavaScript-heavy websites?

Yes, it includes Playwright templates specifically designed for full browser rendering and interaction with dynamic, JavaScript-heavy content.

Can I use this for RAG pipelines?

Absolutely. The skill includes scripts that automatically convert scraped content into clean, structured Markdown, which is ideal for RAG knowledge bases and LLM context.

How does it handle rate limiting?

It features built-in rate limiting utilities and templates that respect robots.txt, implement configurable delays, and handle 429 'Too Many Requests' errors gracefully.

What frameworks are supported?

It supports three major Python libraries: Playwright for browser automation, BeautifulSoup for lightweight static parsing, and Scrapy for complex, large-scale crawling.

Web Scraping & Content Extraction Tools

Name: Web Scraping & Content Extraction Tools
Author: vanman2024

byvanman2024

ウェブスクレイピングとデータ収集

Automates documentation collection and structured data extraction using Playwright, BeautifulSoup, and Scrapy templates.

概要

This skill provides a comprehensive suite of production-ready scraping tools designed for building RAG knowledge bases and collecting technical documentation. It features a decision matrix for choosing between Playwright for dynamic sites, BeautifulSoup for static parsing, and Scrapy for large-scale crawls. With built-in rate limiting, error handling, and markdown conversion, it ensures ethical and efficient data harvesting while respecting website resources and robots.txt protocols.

主な機能

Multi-framework support for Playwright, BeautifulSoup, and Scrapy
Automated markdown conversion for RAG-ready knowledge bases
0 GitHub stars
Robust error handling with exponential backoff and request retries
Built-in ethical scraping patterns including rate limiting and jitter
Specialized scripts for technical documentation and article extraction

ユースケース

Building and updating RAG knowledge bases from online technical documentation
Aggregating blog posts or news articles for automated content analysis
Monitoring competitor websites for content updates and new releases