What is 'token healing' in Guidance?

Token healing is a feature that fixes unnatural token boundaries between the prompt and generated text, preventing awkward spacing and improving model performance.

What is the main benefit of using Guidance?

It guarantees that LLM outputs follow specific structures, such as valid JSON or regex patterns, by filtering tokens during generation rather than just prompting.

Can I use Guidance for local models?

Absolutely; Guidance supports local backends like Transformers and llama.cpp in addition to API-based models from OpenAI and Anthropic.

Does Guidance work with Anthropic Claude?

Yes, it fully supports Claude models via context managers for system, user, and assistant roles, ensuring clean chat-formatted interactions.

How does Guidance reduce latency?

By constraining the model to specific formats, it often requires fewer tokens and eliminates the need for expensive retry logic caused by invalid or malformed outputs.

Guidance: Structured LLM Control

Name: Guidance: Structured LLM Control
Author: Orchestra-Research

byOrchestra-Research

•

3,983

•

Data Science & ML

Enforces structured outputs and grammatical constraints on Large Language Models to guarantee valid JSON, code, and regex-compliant text.

Guidance by Microsoft Research is a specialized framework designed to eliminate the unpredictability of LLM responses by applying strict regex and context-free grammar constraints during the generation process. It allows developers to build complex, multi-step workflows and ReAct agents that are guaranteed to follow specific formats, such as valid JSON, XML, or predefined selection lists. By utilizing 'token healing' to fix unnatural boundaries and providing Pythonic context managers for chat interactions, Guidance ensures higher reliability and lower latency for production-grade AI applications using models like Claude, GPT-4, and local LLMs.

Key Features

01Guarantee valid JSON, XML, and code generation through grammatical constraints

023,983 GitHub stars

03Build complex, stateful multi-step workflows and autonomous ReAct agents

04Optimize token boundaries with automatic token healing for natural text flow

05Constrain model outputs to specific selections or multiple-choice labels

06Enforce specific formats using regex for dates, emails, and custom identifiers

Use Cases

01Building reliable data extraction pipelines that must output 100% valid JSON

02Creating AI agents that interact with external tools using strictly defined action formats

03Improving classification accuracy by restricting responses to a predefined set of labels

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add orchestra-research/ai-research-skills guidance

For use in Claude.ai and ChatGPT

Key Features

01Guarantee valid JSON, XML, and code generation through grammatical constraints

023,983 GitHub stars

03Build complex, stateful multi-step workflows and autonomous ReAct agents

04Optimize token boundaries with automatic token healing for natural text flow

05Constrain model outputs to specific selections or multiple-choice labels

06Enforce specific formats using regex for dates, emails, and custom identifiers

Use Cases

01Building reliable data extraction pipelines that must output 100% valid JSON

02Creating AI agents that interact with external tools using strictly defined action formats

03Improving classification accuracy by restricting responses to a predefined set of labels