What is the benefit of LoRA over full fine-tuning?

LoRA (Low-Rank Adaptation) reduces compute and memory requirements by up to 70% by training only a small subset of parameters, allowing for fast iteration and the ability to serve multiple specialized adapters on a single base model.

When should I use fine-tuning instead of prompt engineering?

Fine-tuning is preferred when a model lacks specific domain knowledge (like medical or legal), needs strict style/format adherence, or when the task complexity exceeds what can be achieved through few-shot prompting.

How much data is required for effective fine-tuning?

For instruction tuning, 2,000 to 10,000 diverse, high-quality examples are typically required. For domain adaptation, you may need between 10,000 and 100,000 domain-specific documents.

How do I prevent a model from forgetting its general knowledge during training?

To prevent 'catastrophic forgetting,' use a smaller learning rate, train for fewer epochs, and mix in a small percentage (10-20%) of general instruction data with your domain-specific training set.

LLM Fine-Tuning Strategies

Name: LLM Fine-Tuning Strategies
Author: lev-os

bylev-os

0•

Ciencia de Datos y ML

Provides comprehensive frameworks and best practices for adapting foundation models to specialized domains using full fine-tuning and parameter-efficient methods.

This skill equips Claude with expert-level knowledge on adapting Large Language Models (LLMs) to specific tasks and industries. It covers the full lifecycle of model adaptation, from selecting the right strategy—such as LoRA, QLoRA, or bottleneck adapters—to data preparation, hyperparameter optimization, and evaluation. It is particularly useful for developers and data scientists when off-the-shelf models lack domain-specific expertise or when cost and latency constraints require specialized, smaller models over massive general-purpose APIs.

Características Principales

010 GitHub stars

02Comprehensive guidance on PEFT methods including LoRA, QLoRA, and DoRA

03Data quality checklists and volume guidelines for instruction tuning

04Optimal hyperparameter configurations for learning rates and batch sizes

05Best practices for multi-adapter serving and production deployment

06Strategies to mitigate catastrophic forgetting and overfitting

Casos de Uso

01Building specialized customer support chatbots with company-specific tone and knowledge

02Adapting models for high-stakes domains like medical radiology or legal analysis

03Fine-tuning code generation models to understand internal APIs and proprietary conventions

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add lev-os/agents fine-tuning-strategies

For use in Claude.ai and ChatGPT

Download Skill