What happens to the papers that are filtered out?

While the 'core_set.csv' contains the top-ranked papers, a full deduped list is maintained in 'papers_dedup.jsonl' so no data is permanently lost during the ranking process.

Is the ranking process random or deterministic?

The process is deterministic. Rerunning the skill with the same input files will yield stable, repeatable results, which is essential for academic transparency.

Can I adjust the number of papers included in the final core set?

Yes, you can use the --core-size flag in the command line or define it within your queries.md file to set a specific target size for your output.

How does this skill identify duplicate research papers?

It uses a normalization process on the title and publication year to group similar entries, ensuring that only the version with the most complete metadata is retained.

Research Paper Deduplicator & Ranker

Name: Research Paper Deduplicator & Ranker
Author: WILLOSCAR

byWILLOSCAR

•

数据科学与机器学习

Streamlines research paper collections by deduplicating entries and ranking them to create high-quality core datasets for academic analysis.

The Dedupe-Rank skill is a deterministic curation tool designed to transform broad, noisy research retrieval results into a manageable 'core set' of papers. It utilizes sophisticated normalization of titles and years to eliminate duplicates while preserving the richest available metadata. By ranking entries based on relevance, recency, and specific architectural signals—such as pinning known classics or ensuring a quota of prior surveys—it provides a stable, repeatable foundation for building taxonomies, literature reviews, and research outlines.

主要功能

01Automated identification and pinning of classic papers and prior surveys

02Configurable core set sizing for flexible research depth and iteration speed

03Deterministic title and year-based deduplication to ensure data integrity

04Smart ranking based on relevance, recency, and specific research signals

0596 GitHub stars

06Generates structured CSV core sets with stable IDs and direct PDF links

使用场景

01Converting raw search results into a refined core set for systematic literature reviews

02Deduplicating research datasets compiled from multiple academic databases

03Selecting high-impact papers to serve as the foundation for AI-generated outlines

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add willoscar/research-units-pipeline-skills dedupe-rank

For use in Claude.ai and ChatGPT

Download Skill