How does it handle duplicate records?

It includes a specialized deduplication tool that can filter records based on specific unique fields, such as URLs or product IDs.

What input file formats does this skill support?

The skill is designed to handle raw data in both JSON and CSV formats commonly used by web crawlers.

Can I customize the data cleaning logic?

Absolutely. The skill provides a Python-based framework using pandas, allowing you to easily adjust formatting rules or statistical metrics.

Does the Excel report include visual charts?

Yes, it uses openpyxl to automatically generate statistical charts and summaries within the exported Excel workbook.

Web Scraper Data Processor

Name: Web Scraper Data Processor
Author: zsutxz

byzsutxz

Web Scraping & Data Collection

Processes raw web scraping data into clean, formatted Excel reports with automated statistical analysis and deduplication.

About

This skill automates the post-collection workflow for web scraping projects by cleaning raw JSON or CSV data. It handles essential data engineering tasks such as deduplication, missing value imputation, and format standardization. By leveraging pandas and openpyxl, it transforms unstructured crawl results into professional Excel workbooks complete with statistical summaries and visual charts, making it an essential tool for developers and analysts who need to turn raw data into actionable insights.

Key Features

High-performance deduplication for JSON and CSV datasets
0 GitHub stars
Built-in data visualization using automated BarCharts
Generation of multi-sheet Excel reports with statistical summaries
Format standardization for prices and numerical fields
Automated data cleaning and missing value filling

Use Cases

Deduplicating and formatting lead lists extracted from web directories
Transforming raw e-commerce scrapes into clean price-comparison spreadsheets
Generating automated weekly market research reports from recurring crawls

About

Key Features

High-performance deduplication for JSON and CSV datasets
0 GitHub stars
Built-in data visualization using automated BarCharts
Generation of multi-sheet Excel reports with statistical summaries
Format standardization for prices and numerical fields
Automated data cleaning and missing value filling

Use Cases

Deduplicating and formatting lead lists extracted from web directories
Transforming raw e-commerce scrapes into clean price-comparison spreadsheets
Generating automated weekly market research reports from recurring crawls