Web Scraping & Data Collection Agent Skills

Image Extraction

Extracts and correlates high-quality food images from delivery platforms and restaurant websites to populate digital catalogs and menus.

YouTube Video to Audio

Downloads and converts YouTube videos into high-quality audio files using yt-dlp and ffmpeg.

AI Research Collector

Aggregates and preprocesses high-quality AI programming tips and best practices from major developer platforms.

Ebook Text Extractor

Extracts clean, plain text from EPUB, MOBI, and PDF files for analysis and data processing.

Scanner Excel Extraction

Extracts, transforms, and structures data from complex Excel files into JSON or CSV formats.

Competition Scraper & GitHub Persister

Scrapes Australian creative writing competitions and automatically manages them as structured GitHub issues with intelligent duplicate detection.

YouTube Transcript Downloader

Extracts and converts YouTube transcripts or auto-generated captions into clean, readable text files.

Scanner PDF Analysis

Analyzes PDF documents to extract structured data, including tables, section headers, and metadata, while providing automated summaries.

Web Connectivity and Search

Searches the internet and converts live webpage content into markdown for real-time information retrieval and analysis.

Web Search

Performs structured web searches via DuckDuckGo to retrieve real-time documentation, library information, and technical solutions.

Web Crawler & Search Indexer

Crawls entire websites and builds searchable full-text indexes of content converted into Markdown format.

Web Scraper Quality Assurance

Enforces a rigorous, test-driven development workflow for building and maintaining web scrapers and data extraction agents.

Web Search

Retrieves real-time information, news, images, and videos from the web using DuckDuckGo to provide up-to-date data and resources.

Add Awesome Tool

Analyzes AI tool URLs to extract metadata and automatically categorizes and adds them to the awesome-ai-tools repository.

MCP Opportunity Pipeline

Automates the end-to-end lifecycle of discovering, validating, building, and publishing Model Context Protocol (MCP) servers and automation tools.

Web Scraper

Performs concurrent, recursive web scraping to extract clean text content while maintaining URL-based directory structures and respecting domain boundaries.

Research Reporter

Automates information gathering from web searches and authoritative sources to generate and save structured research reports.

Web Reference Fetcher

Fetches web documentation, extracts specific topics using AI subagents, and generates structured markdown summaries.

Industry Research

Conducts comprehensive market analysis and trend forecasting across the consumer, technology, healthcare, and finance sectors.

Competitive Research with Bright Data

Conducts enterprise-grade company research, competitive analysis, and market intelligence using professional web scraping and search tools.

Zhipu AI Search

Performs intelligent web searches via the Zhipu search engine with automated relative date resolution.

YouTube Info MCP Installer

Installs and configures the YouTube Info MCP server to enable automated extraction of video metadata and details within Claude Code.

Document Processor

Processes, analyzes, and transforms various file formats into structured data or new document types using a standardized CLI.

Pure.md MCP Installer

Automates the installation and configuration of the Pure.md MCP server for seamless web-to-markdown conversion within Claude Code.

Jina AI MCP Reader

Installs and configures the Jina AI MCP server to enable high-quality web scraping and URL content extraction within Claude Code.

VeyraX MCP Installer

Automates the installation and configuration of the VeyraX MCP server for enhanced web search and content extraction.

WebMind DOM Analyzer MCP

Integrates the WebMind MCP server into Claude Code to provide advanced DOM analysis and specialized Ragic assistant capabilities.

Gemini Web Search

Performs advanced web searches using Gemini to retrieve real-time, high-relevance information for complex technical queries.

Genome Assembly Dataset Curator

Automates the end-to-end processing and metadata curation of genome assembly datasets for VEuPathDB resources.

Wget URL Reader

Fetches and downloads content from any URL using the powerful wget command-line utility.