What is the Norvig tokenize-then-parse pattern?

It is a foundational programming pattern for interpreters where input text is first broken into a list of tokens (lexing) and then consumed to build a nested data structure (parsing).

Does this support multi-type token conversion?

Yes, the pattern includes an 'atom' function that automatically attempts to convert tokens into integers or floats before defaulting to symbols.

Can I use this pattern for JSON-like data?

Absolutely. Any data format with clear delimiters and nested structures can be processed efficiently using this recursive descent approach.

Is this skill suitable for complex programming languages?

It is perfect for DSLs and simple interpreters; however, for highly complex grammars, a dedicated parser generator like ANTLR or Yacc may be more appropriate.

When should I use this skill instead of a simple split?

Use this skill when your input has nested structures, like parentheses, or requires recursive evaluation that a simple string split cannot handle.

Norvig Tokenize & Parse Pattern

Name: Norvig Tokenize & Parse Pattern
Author: jimmc414

byjimmc414

•

开发者工具

Implements a multi-stage pipeline for building interpreters and compilers by separating lexical analysis from recursive descent parsing.

This skill equips Claude with the specialized design pattern popularized by Peter Norvig for building clean, efficient interpreters and structured text processors. It facilitates a robust two-stage approach—first converting raw strings into a clean token stream and then recursively parsing those tokens into a structured Abstract Syntax Tree (AST). This is particularly useful for developers building custom Domain Specific Languages (DSLs), Lisp-like parsers, or any application requiring the transformation of nested, parenthesized text into executable code structures.

主要功能

01Separates lexing (tokenization) from parsing for modular architecture

02Includes atom conversion logic for handling integers, floats, and symbols

03Automates the conversion of strings into Abstract Syntax Trees (ASTs)

04Standardizes error handling for malformed input and unexpected EOF

05Implements recursive descent parsing for handling nested structures

061 GitHub stars

使用场景

01Processing complex structured text with clear token boundaries

02Developing lightweight compilers or Lisp-style interpreters

03Building custom Domain Specific Languages (DSLs) or expression evaluators

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jimmc414/claude-code-plugin-marketplace tokenize-then-parse

For use in Claude.ai and ChatGPT

Download Skill