Can it extract data from private repositories?

Yes, provided the authenticated user or provided token has the necessary permissions to access those repositories.

How does it handle large README files?

It uses an intelligent chunking strategy that breaks documents down by Markdown headings (H1-H3) to preserve context for LLM ingestion.

Does this skill require a GitHub token?

Yes, while some public data can be accessed without one, an authenticated GITHUB_TOKEN or GitHub CLI login is recommended to avoid rate limits.

What languages does it support for code mining?

It includes built-in detection and extraction for Python, JavaScript, TypeScript, Go, Rust, Java, Ruby, and Bash.

Can I harvest issue comments as well?

Yes, the harvester recursively fetches issue bodies along with their associated comments and labels for complete discussion context.

GitHub Harvester

Name: GitHub Harvester
Author: mindmorass

bymindmorass

0•

Web Scraping & Data Collection

Extracts and processes comprehensive data from GitHub repositories for ingestion into RAG pipelines and LLM knowledge bases.

GitHub Harvester is a specialized utility designed to bridge the gap between GitHub's vast repositories and AI-powered development. It provides structured methods for mining documentation, code examples, issues, and discussions, transforming raw repo data into digestible formats for Retrieval-Augmented Generation (RAG). By automating the extraction of READMEs, changelogs, and language-specific code blocks, it enables developers to quickly build context-aware AI tools and documentation assistants directly from source control.

Key Features

010 GitHub stars

02Multi-method extraction using GitHub CLI and PyGithub API

03Automated documentation mining from READMEs and .github folders

04Metadata extraction including stars, topics, and release history

05Intelligent chunking strategies for READMEs, code, and issues

06Recursive code example harvesting with language detection

Use Cases

01Collecting training data for code-specific LLM context windows

02Automating documentation audits and changelog aggregations

03Building RAG-based AI assistants for specific open-source libraries

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add mindmorass/reflex github-harvester

For use in Claude.ai and ChatGPT

Download Skill