Which AI crawlers are considered most critical?

Tier 1 crawlers like GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot are essential for appearing in active AI search products like ChatGPT Search and Perplexity.

Can I block AI training while still appearing in AI search results?

Yes, this skill helps you differentiate between search-only bots (like OAI-SearchBot) and training-specific bots (like CCBot) to achieve that balance.

Does allowing AI crawlers affect my standard Google Search ranking?

No, specific crawlers like Google-Extended control AI training and Gemini features without impacting your traditional Google Search indexing or ranking.

What is the primary purpose of the geo-crawlers skill?

It audits website technical settings to ensure AI search bots from OpenAI, Anthropic, and others can access and index your content for AI-powered responses.

AI Crawler Access Analysis (GEO)

Name: AI Crawler Access Analysis (GEO)
Author: lev-os

bylev-os

•

Marketing Automation

Analyzes website accessibility for AI crawlers to maximize visibility in generative search engines and AI-powered responses.

This skill performs a comprehensive audit of a website's robots.txt file, meta tags, and HTTP headers to determine which AI bots—such as GPTBot, ClaudeBot, and PerplexityBot—can index its content. By providing a detailed access map and strategic recommendations, it helps developers and marketers optimize their sites for Generative Engine Optimization (GEO), ensuring that high-quality content appears in AI-generated answers while maintaining granular control over training data usage.

Key Features

01Strategic recommendations for maximizing AI search visibility

02Comprehensive Tier 1, 2, and 3 crawler access mapping

03Detailed robots.txt parsing for AI-specific user agents

04Analysis of meta tags and HTTP headers for bot control

05Ready-to-use robots.txt templates for optimized AI reach

061 GitHub stars

Use Cases

01Auditing a site's visibility in ChatGPT, Claude, and Perplexity search results

02Configuring access for Apple Intelligence and Gemini without affecting standard search rankings

03Protecting proprietary content from training bots while allowing search-centric AI crawlers

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add lev-os/agents geo-crawlers

For use in Claude.ai and ChatGPT