Trafilatura is a Python package and command-line tool for extracting the main text and metadata from web pages, removing noise and providing structured data in various formats.

What output formats does Trafilatura support?

Trafilatura supports a wide range of output formats, including TXT, Markdown, CSV, JSON, HTML, XML, and XML-TEI.

Is Trafilatura suitable for NLP tasks?

Absolutely! Trafilatura provides clean, structured text, making it ideal for text preprocessing, text mining, and other NLP applications. It is integrated in LLM pipelines by HuggingFace, IBM, and Microsoft Research, among others.

Can Trafilatura be used for web crawling?

Yes, Trafilatura includes web crawling capabilities with support for sitemaps and feeds, allowing for efficient discovery and download of web content.

Does Trafilatura support language detection?

Yes, Trafilatura has optional language detection capabilities to determine the language of the extracted text.

Trafilatura

Name: Trafilatura
Author: adbar

byadbar

•

5,463

•

Data Science & ML

Web Scraping & Data Collection

Developer Tools

Extracts text and metadata from web pages and online resources, offering various output formats.

Trafilatura is a Python package and command-line tool designed for efficient web crawling, scraping, and text extraction. It transforms raw HTML into structured data by gathering text, metadata, and comments from the web. With features like sitemap support, parallel processing, and customizable extraction options, Trafilatura aims to provide a balance between precision and recall, making it a robust and versatile solution for text-based web data collection and processing, also allowing output to commonly used formats. It is actively maintained and integrated into various projects by companies and institutions, offering comprehensive documentation and community support.

Key Features

01Parallel processing of online and offline HTML input

02Multiple output formats: TXT, Markdown, CSV, JSON, HTML, XML, and XML-TEI

034,118 GitHub stars

04Web crawling and text discovery with sitemap and feed support

05Configurable extraction of main text, metadata, and formatting

06Optional language detection and speed optimizations

Use Cases

01Creating news aggregators and content monitoring systems

02Building text corpora for research

03Extracting data for Natural Language Processing (NLP) and Machine Learning (ML) applications