Can I use this for cross-modal search?

Yes, the skill includes specific code for embedding text queries into the same vector space as images, enabling 'search by description' functionality for image databases.

What is Multimodal RAG?

Multimodal RAG (Retrieval-Augmented Generation) is an AI architecture that allows models to retrieve and reason over both text and image data simultaneously, providing more accurate context for visual and textual queries.

Is it compatible with standard vector databases?

Absolutely. It includes setup patterns for Milvus and general guidance for any vector database that supports high-dimensional float vectors and cosine similarity.

Which embedding models are supported by this skill?

The skill provides implementation patterns for CLIP ViT-L/14, SigLIP 2, Voyage multimodal-3, and ColPali, covering a range of performance and context-window requirements.

How does it handle images within PDF documents?

It includes specialized chunking logic that uses libraries like PyMuPDF to extract text blocks and images separately, generating captions or embeddings for visual elements while maintaining their original context.

Multimodal RAG Patterns

Name: Multimodal RAG Patterns
Author: yonatangross

byyonatangross

•

データサイエンスとML

Implements advanced retrieval-augmented generation systems that integrate text and image data for hybrid search and visual question answering.

This skill provides production-ready patterns for building Multimodal RAG (Retrieval-Augmented Generation) systems. It streamlines the implementation of complex architectures, including joint embeddings with CLIP and SigLIP 2, long-context retrieval via Voyage multimodal-3, and hybrid pipelines that combine text captions with direct image indexing. Developers can use this skill to automate multimodal document processing, such as chunking PDFs containing charts and tables, performing cross-modal searches, and generating context-aware responses using Claude's vision capabilities combined with retrieved visual context.

主な機能

01Cross-modal search capabilities (text-to-image and image-to-text)

0269 GitHub stars

03Pre-configured patterns for Voyage multimodal-3, CLIP, and SigLIP 2

04Multimodal document chunking for PDFs with embedded charts and tables

05Optimized PDF batch processing for large document embedding

06Hybrid retrieval pipelines combining vector search and image captioning

ユースケース

01Developing RAG systems for technical documentation with diagrams

02Creating visual question-answering agents for complex financial or medical reports

03Building visual search engines for e-commerce and product catalogs

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add yonatangross/orchestkit multimodal-rag

For use in Claude.ai and ChatGPT

Download Skill