Generative Engine Optimization (GEO) is the process of optimizing web content so it is easily discovered, indexed, and cited by AI search engines like ChatGPT Search and Perplexity.

Can I allow AI search while blocking AI training?

Yes, this skill helps you differentiate between crawlers. You can allow search-focused bots like OAI-SearchBot while blocking training-centric bots like CCBot or anthropic-ai.

What happens if I block GPTBot?

If GPTBot is blocked, your content will not appear in ChatGPT Search results and the model will be unable to browse or summarize your URLs when requested by users.

Which AI crawlers are considered most critical?

Tier 1 crawlers like GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot are critical because they power the live AI search products used by hundreds of millions of users.

Does blocking Google-Extended affect my standard Google ranking?

No. Google-Extended only controls whether your content is used to train Gemini models; it has no impact on your standard Google Search rankings or indexing.

AI Crawler Analysis & GEO

Name: AI Crawler Analysis & GEO
Author: lev-os

bylev-os

•

Marketing Automation

Analyzes and optimizes website accessibility for AI crawlers to maximize visibility in AI-generated search results.

This skill performs a technical audit of how AI crawlers perceive and access your website by examining robots.txt configurations, meta tags, and HTTP headers. It identifies accidental blocks that prevent major AI platforms like ChatGPT, Claude, and Perplexity from indexing your content, which is the foundational requirement for Generative Engine Optimization (GEO). By providing a detailed access map and tiered strategic recommendations, it helps developers ensure their content is discoverable and cited within the AI-native search ecosystem while maintaining granular control over model training data.

Key Features

01Meta tag and HTTP header accessibility diagnostic

021 GitHub stars

03Generation of optimized robots.txt for maximum AI reach

04Tiered crawler classification (Search Visibility vs. Model Training)

05Automated robots.txt parsing for 10+ major AI user-agents

06Strategic impact analysis for blocking specific AI bots

Use Cases

01Auditing a production site to ensure it appears in ChatGPT Search and Perplexity results

02Implementing GEO (Generative Engine Optimization) best practices for content discovery

03Configuring site permissions to allow AI search while blocking unauthorized model training

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add lev-os/agents geo-crawlers

For use in Claude.ai and ChatGPT