NLWeb Data Ingestion

•
32
•

Streamlines the ingestion of RSS feeds, JSON-LD, and CSV data into NLWeb vector stores for agentic commerce applications.

The nlweb-data-loading skill is a specialized utility for managing the content lifecycle within NLWeb's vector database. It automates the complex pipeline of fetching, normalizing, chunking, and embedding site content from diverse sources like RSS/Atom feeds, Schema.org JSON-LD, sitemaps, and CSV files. By enforcing consistent embedding providers and managing site-specific partitioning, this skill ensures that retrieval-augmented generation (RAG) systems stay accurate and performant. It is an essential tool for developers bootstrapping new commerce sites, migrating between retrieval backends, or establishing automated content refresh cycles.

Key Features

01Configurable batch sizing and parallel loading for high-volume data
02Site-based partitioning to enable scoped and multi-tenant retrieval
0332 GitHub stars
04Multi-source ingestion support including RSS, JSON-LD, Sitemap, and CSV
05Automated normalization of items to Schema.org JSON objects
06Support for incremental upserts and full 'delete-and-reload' refreshes

Use Cases

01Bootstrapping a RAG search index for a new e-commerce store using existing product feeds
02Migrating vector data between backends such as local Qdrant and Azure AI Search
03Automating daily content refreshes to keep an agent's knowledge base up to date