Automates large-scale document processing and entity extraction using investigative journalism methodologies.
This skill implements a robust document processing pipeline designed for large-scale leak analysis, modeled after investigative techniques used in the Panama, Paradise, and Pandora Papers. It provides a comprehensive suite of tools including ICIJ Datashare for search, Apache Tika for universal extraction, Tesseract for OCR, and spaCy for high-accuracy Named Entity Recognition (NER). By coordinating between forensic validation and graph generation, it allows users to transform unstructured document troves into structured DuckDB databases, facilitating deep link analysis and entity co-occurrence mapping for complex investigations.
Key Features
01Automated batch processing for massive PDF and document corpora
02Advanced Named Entity Recognition (NER) for persons, organizations, and locations
038 GitHub stars
04Structured data modeling with DuckDB for relationship and co-occurrence analysis
05Self-hosted search and annotation via ICIJ Datashare integration
06Multi-format document extraction using Apache Tika and Tesseract OCR
Use Cases
01Extracting and mapping entity relationships for graph-based investigative reporting
02Analyzing massive leaked document sets to identify stakeholders and hidden networks
03Building searchable databases from diverse file types like PDFs, emails, and spreadsheets
