Evaluates long-term conversational memory performance using the ACL 2024 LoCoMo benchmark suite.
The LoCoMo Benchmark skill is a specialized evaluation tool designed to measure the effectiveness of long-term conversational memory within the cc-soul ecosystem. It automates the process of ingesting multi-session conversation data—extracting observations and speaker facts—and then subjects the system to rigorous QA testing. By measuring retrieval accuracy across multi-hop, temporal, and adversarial categories, it provides developers with a standardized F1 score to compare their AI's memory performance against human baselines and state-of-the-art models like GPT-4.
Key Features
01Detailed F1 score reporting for retrieval accuracy vs. ground truth
02Multi-category evaluation including multi-hop, temporal, and adversarial reasoning
03Integrated comparison against industry baselines and human performance ceilings
04Flexible execution modes for single conversation testing or full benchmark runs
05Automated ingestion of multi-session LoCoMo datasets into memory observations
060 GitHub stars
Use Cases
01Quantifying the accuracy of long-term memory retrieval in AI agents
02Benchmarking custom RAG implementations against established AI research standards
03Debugging temporal reasoning and fact-linkage capabilities in conversational models
