Do I need to provide CSS selectors manually?

No, the intelligent analysis engine automatically detects page structures and selectors, though it can store successful patterns to improve future performance on the same site.

What happens if a website uses infinite scrolling?

The agent identifies lazy-loading mechanisms and automatically scrolls the page until all content is loaded before starting the extraction process.

How does the self-learning system improve over time?

The skill maintains an experience database that records successful extraction strategies and CSS patterns for every domain it visits, ensuring faster and more reliable results on repeat tasks.

Can I run this on a headless server or VPS?

Absolutely. The skill includes a dedicated headless setup mode optimized for VPS environments using Playwright and Crawl4AI.

Does this skill handle websites that require logins?

Yes, by utilizing the Local Browser (CDP) mode, the scraper can leverage your existing browser profiles and active login sessions to access protected content.

Intelligent Web Scraper

Name: Intelligent Web Scraper
Author: Yrzhe

byYrzhe

•

Web Scraping & Data Collection

Automates complex web data extraction using self-learning algorithms to navigate pagination, bypass blocks, and analyze page structures autonomously.

This skill transforms Claude into a sophisticated web scraping agent capable of autonomous data harvesting through a self-learning system that records successful patterns for specific domains. It eliminates the need for manual selector configuration by automatically identifying data containers, handling infinite scrolls, and following detail links for comprehensive extraction. With support for both headless Playwright execution and local browser integration via CDP, it can effectively scrape dynamic content while preserving user login sessions and bypassing sophisticated anti-blocking mechanisms.

Key Features

01Intelligent detail-link following to extract full content from list-based pages

02Self-learning experience database that remembers domain-specific scraping patterns

03Automated handling of infinite scroll, 'load more' buttons, and multi-page pagination

049 GitHub stars

05Adaptive anti-blocking strategies including User-Agent rotation and variable delays

06Local browser (CDP) integration to leverage existing login sessions and cookies

Use Cases

01Automated archiving of entire article series or documentation sets from complex blogs

02Extracting comprehensive product catalogs from e-commerce sites with infinite scrolling

03Gathering structured data from authenticated web portals using existing browser profiles

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add yrzhe/claude-skills intelligent-web-scraper

For use in Claude.ai and ChatGPT

Download Skill