Does it include cost optimization strategies?

Yes, the skill includes a model selection matrix to help developers choose between high-accuracy models and cost-efficient alternatives like Gemini Flash.

What is the Multimodal LLM skill for Claude Code?

It is a specialized plugin providing implementation patterns for integrating image, audio, and video capabilities into AI applications using Claude Code.

Can this skill help with document OCR?

Yes, it provides specific rules for document understanding, PDF page processing, and extracting structured data from visual charts and diagrams.

How does it handle real-time audio?

It offers patterns for low-latency voice agents, speech-to-text (STT), and expressive text-to-speech (TTS) using models like Gemini Live and Whisper.

Which video generation APIs does this skill support?

The skill includes patterns and best practices for leading video providers including Kling 3.0, OpenAI Sora, Google Veo, and Runway Gen-4.5.

Multimodal LLM Integration

Name: Multimodal LLM Integration
Author: yonatangross

byyonatangross

•

116

•

Data Science & ML

Integrates advanced vision, audio, and video generation capabilities using production-ready multimodal LLM patterns.

The Multimodal LLM skill is a comprehensive toolkit for developers building applications that interact with the physical world through AI. It provides standardized patterns for image analysis, OCR for complex documents, real-time audio processing, and high-fidelity video generation. By offering implementation guides for industry-leading models like Kling, Sora, and Gemini, this skill helps developers navigate the complexities of base64 encoding, async API polling, and cost-optimized model selection for vision and voice workflows.

Key Features

01Audio integration patterns for STT, TTS, and real-time voice agents

02116 GitHub stars

03Advanced vision patterns for multi-image analysis and visual QA

04Model selection logic to optimize for cost, speed, and accuracy

05AI video generation workflows for Kling, Sora, Veo, and Runway

06Structured document understanding and OCR strategies for PDFs and charts

Use Cases

01Developing real-time voice assistants and expressive audio tools

02Creating AI-driven video content with character and scene consistency

03Building automated document processing pipelines with visual OCR

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add yonatangross/orchestkit multimodal-llm

For use in Claude.ai and ChatGPT

Download Skill