Install trafilatura with Homebrew

Project links

• Homepage
• Repository
• Upstream docs
• Package manager page

Commands and aliases

• trafilatura

Project history

The official README says the work started as a PhD project at the crossroads of linguistics and NLP, initially launched to create text databases for research at the Berlin-Brandenburg Academy of Sciences, specifically the DWDS and ZDL units. The repository was created on April 8, 2019.

The project name comes from the Italian word trafilatura, referring to wire drawing and used by the author as a metaphor for refinement and conversion. Official citations connect the project to earlier research on metadata-enhanced web corpora in 2016, generic web content extraction in 2019, and the 2021 ACL/IJCNLP system demonstration paper.

Official GitHub release metadata shows v0.1.0 published on September 25, 2019 and v1.0.0 on November 30, 2021. The README notes that versions prior to v1.8.0 were GPLv3+ and that current versions are distributed under Apache 2.0.

Adoption history

The official README says Trafilatura is widely used and integrated into thousands of projects, naming companies and institutions including Hugging Face, IBM, Microsoft Research, the Allen Institute, Stanford, Tokyo Institute of Technology, and the University of Munich.

The official documentation has a uses and citations page, and the README emphasizes academic citation, benchmarks, and integration into the web data extraction ecosystem. Its adoption is therefore both software-package adoption and research-method adoption.

How it is used

Trafilatura can be used as a Python library or as the trafilatura command-line tool. Official docs describe crawling, downloads, scraping, main-text extraction, metadata extraction, comment extraction, language detection, and output as TXT, Markdown, CSV, JSON, HTML, XML, and XML-TEI.

For package-manager users, the CLI is useful for quick extraction from live URLs or stored HTML without writing code, while Python users can embed the same extraction functions in data pipelines.

Why package nerds care

Trafilatura matters to package nerds because it packages a research-grade text extraction stack behind one CLI and Python module. It is useful for building local corpora, testing scraping pipelines, and comparing extraction quality without assembling many separate tools.

Its packaging also reflects a common modern pattern: a Python data/NLP library that is simultaneously a CLI, a research artifact with citations, and an operational dependency for downstream software.

Timeline

• 2016: Author's related work on metadata-enhanced web corpora appears in the official citation list.
• 2019: Repository created and v0.1.0 released.
• 2021: Trafilatura ACL/IJCNLP system demonstration paper published; v1.0.0 released.
• 2022: v1.2.x releases appear in official GitHub release metadata.
• Current docs: Trafilatura 2.1.0 documentation is published on Read the Docs.

Related projects

• The official README lists jusText and readability as generic algorithms used in the extractor.
• The official README points to htmldate and related web data extraction packages in the author's software ecosystem.

Sources

• https://api.github.com/repos/adbar/trafilatura/releases - official GitHub release metadata.
• https://github.com/adbar/trafilatura - official README, repository metadata, history/context, adoption, license, and citations.
• https://trafilatura.readthedocs.io/en/latest/ - official documentation overview.
• https://trafilatura.readthedocs.io/en/latest/usage-cli.html - official command-line usage documentation.
• https://trafilatura.readthedocs.io/en/latest/used-by.html - official uses and citations page.