Why am I getting 'Loading...' text instead of actual page content?

For JavaScript-heavy sites or SPAs, you need to configure the 'waitFor' parameter (e.g., 5000ms) to allow the browser to render the content before scraping.

Does Firecrawl handle robots.txt automatically?

Yes, Firecrawl honors robots.txt by default. This skill helps you ensure you aren't accidentally bypassing these safety features which could lead to IP bans.

How can I prevent Firecrawl from burning through my credits too fast?

Always set an explicit 'limit' parameter, use includePaths/excludePaths to target specific data, and set a maxDepth to avoid crawling unnecessary sections of a site.

Why is my Firecrawl output returning job IDs instead of data?

This is a common async model pitfall. You must use the job ID to poll the status via checkCrawlStatus until the job is completed before you can access the results.

Firecrawl Best Practices & Pitfalls

Name: Firecrawl Best Practices & Pitfalls
Author: micsapp

bymicsapp

0•

网络抓取与数据收集

Identifies and avoids common Firecrawl integration mistakes and anti-patterns to optimize web scraping efficiency.

This skill serves as an expert auditor for Firecrawl integrations, helping developers navigate the complexities of asynchronous crawl models, credit-based billing, and JavaScript rendering. It provides actionable guidance on handling job statuses, setting strict crawl limits to prevent unexpected credit exhaustion, and ensuring high-quality data extraction by correctly configuring output formats and rendering timeouts. Whether you are onboarding new developers or auditing production code, this skill ensures your web scraping workflows are robust, cost-effective, and compliant with site policies.

主要功能

010 GitHub stars

02Optimizes JavaScript rendering configurations for Single Page Applications (SPAs).

03Prevents credit depletion by enforcing crawl limits and URL inclusion/exclusion filters.

04Ensures correct output formatting for Markdown, HTML, or structured data.

05Audits code for robots.txt compliance and internal rate-limiting best practices.

06Detects inefficient async polling patterns that waste API credits.

使用场景

01Reviewing Firecrawl code to fix issues like empty results or job polling timeouts.

02Onboarding developers to web scraping projects to prevent accidental credit burn on large sites.

03Auditing existing automation scripts for performance and compliance with site-specific constraints.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add micsapp/micstec-skills firecrawl-known-pitfalls

For use in Claude.ai and ChatGPT

主要功能

010 GitHub stars

02Optimizes JavaScript rendering configurations for Single Page Applications (SPAs).

03Prevents credit depletion by enforcing crawl limits and URL inclusion/exclusion filters.

04Ensures correct output formatting for Markdown, HTML, or structured data.

05Audits code for robots.txt compliance and internal rate-limiting best practices.

06Detects inefficient async polling patterns that waste API credits.

使用场景

01Reviewing Firecrawl code to fix issues like empty results or job polling timeouts.

02Onboarding developers to web scraping projects to prevent accidental credit burn on large sites.

03Auditing existing automation scripts for performance and compliance with site-specific constraints.