참고소스 수정본
This commit is contained in:
LASTA_DEV01\lasta
207
참고/trafilatura-master/README.md
Normal file
207
참고/trafilatura-master/README.md
Normal file
@@ -0,0 +1,207 @@
|
||||
# Trafilatura: Discover and Extract Text Data on the Web
|
||||
|
||||
<br/>
|
||||
|
||||
<img alt="Trafilatura Logo" src="https://raw.githubusercontent.com/adbar/trafilatura/master/docs/trafilatura-logo.png" align="center" width="60%"/>
|
||||
|
||||
<br/>
|
||||
|
||||
[](https://pypi.python.org/pypi/trafilatura)
|
||||
[](https://pypi.python.org/pypi/trafilatura)
|
||||
[](http://trafilatura.readthedocs.org/en/latest/?badge=latest)
|
||||
[](https://codecov.io/gh/adbar/trafilatura)
|
||||
[](https://pepy.tech/project/trafilatura)
|
||||
[](https://aclanthology.org/2021.acl-demo.15/)
|
||||
|
||||
<br/>
|
||||
|
||||
<img alt="Demo as GIF image" src="https://raw.githubusercontent.com/adbar/trafilatura/master/docs/trafilatura-demo.gif" align="center" width="80%"/>
|
||||
|
||||
<br/>
|
||||
|
||||
|
||||
## Introduction
|
||||
|
||||
Trafilatura is a cutting-edge **Python package and command-line tool**
|
||||
designed to **gather text on the Web and simplify the process of turning
|
||||
raw HTML into structured, meaningful data**. It includes all necessary
|
||||
discovery and text processing components to perform **web crawling,
|
||||
downloads, scraping, and extraction** of main texts, metadata and
|
||||
comments. It aims at staying **handy and modular**: no database is
|
||||
required, the output can be converted to commonly used formats.
|
||||
|
||||
Going from HTML bulk to essential parts can alleviate many problems
|
||||
related to text quality, by **focusing on the actual content**,
|
||||
**avoiding the noise** caused by recurring elements like headers and footers
|
||||
and by **making sense of the data and metadata** with selected information.
|
||||
The extractor strikes a balance between limiting noise (precision) and
|
||||
including all valid parts (recall). It is **robust and reasonably fast**.
|
||||
|
||||
Trafilatura is [widely used](https://trafilatura.readthedocs.io/en/latest/used-by.html)
|
||||
and integrated into [thousands of projects](https://github.com/adbar/trafilatura/network/dependents)
|
||||
by companies like HuggingFace, IBM, and Microsoft Research as well as institutions like
|
||||
the Allen Institute, Stanford, the Tokyo Institute of Technology, and
|
||||
the University of Munich.
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
- Advanced web crawling and text discovery:
|
||||
- Support for sitemaps (TXT, XML) and feeds (ATOM, JSON, RSS)
|
||||
- Smart crawling and URL management (filtering and deduplication)
|
||||
|
||||
- Parallel processing of online and offline input:
|
||||
- Live URLs, efficient and polite processing of download queues
|
||||
- Previously downloaded HTML files and parsed HTML trees
|
||||
|
||||
- Robust and configurable extraction of key elements:
|
||||
- Main text (common patterns and generic algorithms like jusText and readability)
|
||||
- Metadata (title, author, date, site name, categories and tags)
|
||||
- Formatting and structure: paragraphs, titles, lists, quotes, code, line breaks, in-line text formatting
|
||||
- Optional elements: comments, links, images, tables
|
||||
|
||||
- Multiple output formats:
|
||||
- TXT and Markdown
|
||||
- CSV
|
||||
- JSON
|
||||
- HTML, XML and [XML-TEI](https://tei-c.org/)
|
||||
|
||||
- Optional add-ons:
|
||||
- Language detection on extracted content
|
||||
- Speed optimizations
|
||||
|
||||
- Actively maintained with support from the open-source community:
|
||||
- Regular updates, feature additions, and optimizations
|
||||
- Comprehensive documentation
|
||||
|
||||
|
||||
### Evaluation and alternatives
|
||||
|
||||
Trafilatura consistently outperforms other open-source libraries in text
|
||||
extraction benchmarks, showcasing its efficiency and accuracy in
|
||||
extracting web content. The extractor tries to strike a balance between
|
||||
limiting noise and including all valid parts.
|
||||
|
||||
For more information see the [benchmark section](https://trafilatura.readthedocs.io/en/latest/evaluation.html)
|
||||
and the [evaluation readme](https://github.com/adbar/trafilatura/blob/master/tests/README.rst)
|
||||
to run the evaluation with the latest data and packages.
|
||||
|
||||
|
||||
#### Other evaluations:
|
||||
|
||||
- Most efficient open-source library in *ScrapingHub*'s [article extraction benchmark](https://github.com/scrapinghub/article-extraction-benchmark)
|
||||
- Best overall tool according to [Bien choisir son outil d'extraction de contenu à partir du Web](https://hal.archives-ouvertes.fr/hal-02768510v3/document)
|
||||
(Lejeune & Barbaresi 2020)
|
||||
- Best single tool by ROUGE-LSum Mean F1 Page Scores in [An Empirical Comparison of Web Content Extraction Algorithms](https://webis.de/downloads/publications/papers/bevendorff_2023b.pdf)
|
||||
(Bevendorff et al. 2023)
|
||||
|
||||
|
||||
## Usage and documentation
|
||||
|
||||
[Getting started with Trafilatura](https://trafilatura.readthedocs.io/en/latest/quickstart.html)
|
||||
is straightforward. For more information and detailed guides, visit
|
||||
[Trafilatura's documentation](https://trafilatura.readthedocs.io/):
|
||||
|
||||
- [Installation](https://trafilatura.readthedocs.io/en/latest/installation.html)
|
||||
- Usage:
|
||||
[On the command-line](https://trafilatura.readthedocs.io/en/latest/usage-cli.html),
|
||||
[With Python](https://trafilatura.readthedocs.io/en/latest/usage-python.html),
|
||||
[With R](https://trafilatura.readthedocs.io/en/latest/usage-r.html)
|
||||
- [Core Python functions](https://trafilatura.readthedocs.io/en/latest/corefunctions.html)
|
||||
- Interactive Python Notebook: [Trafilatura Overview](docs/Trafilatura_Overview.ipynb)
|
||||
- [Tutorials and use cases](https://trafilatura.readthedocs.io/en/latest/tutorials.html)
|
||||
|
||||
Youtube playlist with video tutorials in several languages:
|
||||
|
||||
- [Web scraping tutorials and how-tos](https://www.youtube.com/watch?v=8GkiOM17t0Q&list=PL-pKWbySIRGMgxXQOtGIz1-nbfYLvqrci)
|
||||
|
||||
|
||||
## License
|
||||
|
||||
This package is distributed under the [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0.html).
|
||||
|
||||
Versions prior to v1.8.0 are under GPLv3+ license.
|
||||
|
||||
|
||||
### Contributing
|
||||
|
||||
Contributions of all kinds are welcome. Visit the [Contributing
|
||||
page](https://github.com/adbar/trafilatura/blob/master/CONTRIBUTING.md)
|
||||
for more information. Bug reports can be filed on the [dedicated issue
|
||||
page](https://github.com/adbar/trafilatura/issues).
|
||||
|
||||
Many thanks to the
|
||||
[contributors](https://github.com/adbar/trafilatura/graphs/contributors)
|
||||
who extended the docs or submitted bug reports, features and bugfixes!
|
||||
|
||||
|
||||
## Context
|
||||
|
||||
This work started as a PhD project at the crossroads of linguistics and
|
||||
NLP, this expertise has been instrumental in shaping Trafilatura over
|
||||
the years. Initially launched to create text databases for research purposes
|
||||
at the Berlin-Brandenburg Academy of Sciences (DWDS and ZDL units),
|
||||
this package continues to be maintained but its future depends on community support.
|
||||
|
||||
**If you value this software or depend on it for your product, consider
|
||||
sponsoring it and contributing to its codebase**. Your support
|
||||
[on GitHub](https://github.com/sponsors/adbar) or [ko-fi.com](https://ko-fi.com/adbarbaresi)
|
||||
will help maintain and enhance this popular package.
|
||||
|
||||
*Trafilatura* is an Italian word for [wire
|
||||
drawing](https://en.wikipedia.org/wiki/Wire_drawing) symbolizing the
|
||||
refinement and conversion process. It is also the way shapes of pasta
|
||||
are formed.
|
||||
|
||||
### Author
|
||||
|
||||
Reach out via ia the software repository or the [contact
|
||||
page](https://adrien.barbaresi.eu/) for inquiries, collaborations, or
|
||||
feedback. See also social networks for the latest updates.
|
||||
|
||||
- Barbaresi, A. [Trafilatura: A Web Scraping Library and Command-Line
|
||||
Tool for Text Discovery and
|
||||
Extraction](https://aclanthology.org/2021.acl-demo.15/), Proceedings
|
||||
of ACL/IJCNLP 2021: System Demonstrations, 2021, p. 122-131.
|
||||
- Barbaresi, A. "[Generic Web Content Extraction with Open-Source
|
||||
Software](https://hal.archives-ouvertes.fr/hal-02447264/document)",
|
||||
Proceedings of KONVENS 2019, Kaleidoscope Abstracts, 2019.
|
||||
- Barbaresi, A. "[Efficient construction of metadata-enhanced web
|
||||
corpora](https://hal.archives-ouvertes.fr/hal-01371704v2/document)",
|
||||
Proceedings of the [10th Web as Corpus Workshop
|
||||
(WAC-X)](https://www.sigwac.org.uk/wiki/WAC-X), 2016.
|
||||
|
||||
|
||||
### Citing Trafilatura
|
||||
|
||||
Trafilatura is widely used in the academic domain, chiefly for data
|
||||
acquisition. Here is how to cite it:
|
||||
|
||||
[](https://aclanthology.org/2021.acl-demo.15/)
|
||||
[](https://doi.org/10.5281/zenodo.3460969)
|
||||
|
||||
``` shell
|
||||
@inproceedings{barbaresi-2021-trafilatura,
|
||||
title = {{Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction}},
|
||||
author = "Barbaresi, Adrien",
|
||||
booktitle = "Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations",
|
||||
pages = "122--131",
|
||||
publisher = "Association for Computational Linguistics",
|
||||
url = "https://aclanthology.org/2021.acl-demo.15",
|
||||
year = 2021,
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Software ecosystem
|
||||
|
||||
Jointly developed plugins and additional packages also contribute to the
|
||||
field of web data extraction and analysis:
|
||||
|
||||
<img alt="Software ecosystem" src="https://raw.githubusercontent.com/adbar/htmldate/master/docs/software-ecosystem.png" align="center" width="65%"/>
|
||||
|
||||
Corresponding posts can be found on [Bits of
|
||||
Language](https://adrien.barbaresi.eu/blog/tag/trafilatura.html).
|
||||
|
||||
Impressive, you have reached the end of the page: Thank you for your
|
||||
interest!
|
||||
Reference in New Issue
Block a user