lasta/AI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1be2c7d3a3c5e9b6c9aab8e2fc1f7c8778825afd
AI/참고/trafilatura-master/docs/usage-python.rst
LASTA_DEV01\lasta 2e9204243d 참고소스 수정본
2026-05-12 19:40:31 +09:00

493 lines
18 KiB
ReStructuredText
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
With Python
===========
.. meta::
:description lang=en:
This tutorial focuses on text extraction from web pages with Python code snippets.
Data mining with this library encompasses HTML parsing and language identification.
The Python programming language
-------------------------------
Python can be easy to pick up whether you're a first time programmer or you're experienced with other languages:
- Official `Python Tutorial <https://docs.python.org/3/tutorial/>`_
- `The Hitchhiker’s Guide to Python <https://docs.python-guide.org/>`_
- `The Best Python Tutorials (freeCodeCamp) <https://www.freecodecamp.org/news/best-python-tutorial/>`_
Step-by-step
------------
Quickstart
^^^^^^^^^^
For the basics see `quickstart documentation page <quickstart.html>`_.
.. note::
For a hands-on tutorial see also the Python Notebook `Trafilatura Overview <https://github.com/adbar/trafilatura/blob/master/docs/Trafilatura_Overview.ipynb>`_.
Extraction functions
^^^^^^^^^^^^^^^^^^^^
The functions can be imported using ``from trafilatura import ...`` and used on raw documents (strings) or parsed HTML (LXML elements).
Main text extraction, good balance between precision and recall:
- ``extract``: Wrapper function, easiest way to perform text extraction and conversion
- ``bare_extraction``: Internal function returning bare Python variables
Additional fallback functions:
- ``baseline``: Faster extraction function targeting text paragraphs and/or JSON metadata
- ``html2txt``: Extract all text in a document, maximizing recall
Output
^^^^^^
By default, the output is in plain text (TXT) format without metadata. The following additional formats are available:
- CSV
- HTML (from version 1.11 onwards)
- JSON
- Markdown (from version 1.9 onwards)
- XML and XML-TEI (following the guidelines of the Text Encoding Initiative)
To specify the output format, use one of the following strings: ``"csv", "json", "html", "markdown", "txt", "xml", "xmltei"``.
The ``bare_extraction`` function also accepts an additional ``python`` format to work with Python on the output.
To extract and include metadata in the output, use the ``with_metadata=True`` argument.
Examples
~~~~~~~~
.. code-block:: python
# some formatting preserved in basic XML structure
>>> extract(downloaded, output_format="xml")
# output in JSON format with metadata extracted
>>> extract(downloaded, output_format="json", with_metadata=True)
Note that combining TXT, CSV and JSON formats with certain structural elements (e.g. formatting or links) triggers output in Markdown format (plain text with additional elements).
Choice of HTML elements
^^^^^^^^^^^^^^^^^^^^^^^
Customize the extraction process by including or excluding specific HTML elements:
- Text elements:
``include_comments=True``
Include comment sections at the bottom of articles.
``include_tables=True``
Extract text from HTML ``<table>`` elements.
- Structural elements:
``include_formatting=True``
Keep structural elements related to formatting (``<b>``/``<strong>``, ``<i>``/``<emph>`` etc.)
``include_links=True``
Keep link targets (in ``href="..."``)
``include_images=True``
Keep track of images along with their targets (``<img>`` attributes: alt, src, title)
To operate on these elements, pass the corresponding parameters to the ``extract()`` function:
.. code-block:: python
# exclude comments from the output
>>> result = extract(downloaded, include_comments=False)
# skip tables and include links in the output
>>> result = extract(downloaded, include_tables=False, include_links=True)
# convert relative links to absolute links where possible
>>> extract(downloaded, output_format='xml', include_links=True, url=url)
Important notes
~~~~~~~~~~~~~~~
- ``include_comments`` and ``include_tables`` are activated by default.
- Including extra elements works best with conversion to XML formats or using ``bare_extraction()``. This allows for direct display and manipulation of the elements.
- Certain elements may not be visible in the output if the chosen format does not allow it.
- Selecting Markdown automatically includes text formatting.
.. hint::
The heuristics used by the main algorithm change according to the presence of certain elements in the HTML. If the output seems odd, try removing a constraint (e.g. formatting) to improve the result.
The precision and recall presets
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The main extraction functions offer two presets to adjust to focus of the extraction process: ``favor_precision`` and ``favor_recall``.
These parameters allow you to change the balance between accuracy and comprehensiveness of the output.
.. code-block:: python
>>> result = extract(downloaded, url, favor_precision=True)
Precision
~~~~~~~~~
- If your results contain too much noise, prioritize precision to focus on the most central and relevant elements.
- Additionally, you can use the ``prune_xpath`` parameter to target specific HTML elements using a list of XPath expressions.
Recall
~~~~~~
- If parts of your documents are missing, try this preset to take more elements into account.
- If content is still missing, refer to the `troubleshooting guide <troubleshooting.html>`_.
Additional functions for text extraction
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``html2txt`` and ``baseline`` functions offer simpler approaches to extracting text from HTML content, prioritizing performance over precision.
html2txt()
~~~~~~~~~~
The ``html2txt`` function serves as a last resort for extracting text from HTML content. It emulates the behavior of similar functions in other packages and can be used to output all possible text from a given HTML source, maximizing recall. However, it may not always produce accurate or meaningful results, as it does not consider the context of the extracted sections.
.. code-block:: python
>>> from trafilatura import html2txt
>>> html2txt(downloaded)
baseline()
~~~~~~~~~~
For a better balance between precision and recall, as well as improved performance, consider using the ``baseline`` function instead. This function returns a tuple containing an LXML element with the body, the extracted text as a string, and the length of the text. It uses a set of heuristics to extract text from the HTML content, which generally produces more accurate results than ``html2txt``.
.. code-block:: python
>>> from trafilatura import baseline
>>> postbody, text, len_text = baseline(downloaded)
For more advanced use cases, consider using other functions in the package that provide more control and customization over the text extraction process.
Guessing if text can be found
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The function ``is_probably_readerable()`` has been ported from Mozilla's Readability.js, it is available from version 1.10 onwards and provides a way to guess if a page probably has a main text to extract.
.. code-block:: python
>>> from trafilatura.readability_lxml import is_probably_readerable
>>> is_probably_readerable(html) # HTML string or already parsed tree
Language identification
^^^^^^^^^^^^^^^^^^^^^^^
The target language can also be set using 2-letter codes (ISO 639-1), there will be no output if the detected language of the result does not match and no such filtering if the identification component has not been installed (see above `installation instructions <installation.html>`_) or if the target language is not available.
.. code-block:: python
>>> result = extract(downloaded, url, target_language="de")
.. note::
Additional components are required: ``pip install trafilatura[all]``.
This feature currently uses the `py3langid package <https://github.com/adbar/py3langid>`_ and is dependent on language availability and performance of the original model.
Optimizing for speed
^^^^^^^^^^^^^^^^^^^^
Execution speed not only depends on the platform and on supplementary packages (``trafilatura[all]``, ``htmldate[speed]``), but also on the extraction strategy.
The available fallbacks make extraction more precise but also slower. The use of fallback algorithms can also be bypassed in *fast* mode, which should make extraction about twice as fast:
.. code-block:: python
# skip algorithms used as fallback
>>> result = extract(downloaded, no_fallback=True)
The following combination usually leads to shorter processing times:
.. code-block:: python
>>> result = extract(downloaded, include_comments=False, include_tables=False, no_fallback=True)
Extraction settings
-------------------
.. hint::
See also `settings page <settings.html>`_.
Function parameters
^^^^^^^^^^^^^^^^^^^
Starting from version 1.9, the ``Extractor`` class provides a convenient way to define and manage extraction parameters. It allows users to customize all options used by the extraction functions and offers a convenient shortcut compared to multiple function parameters.
Here is how to use the class:
.. code-block:: python
# import the Extractor class from the settings module
>>> from trafilatura.settings import Extractor
# set multiple options at once
>>> options = Extractor(output_format="json", with_metadata=True)
# add or adjust settings as needed
>>> options.formatting = True # same as include_formatting
>>> options.source = "My Source" # useful for debugging
# use the options in an extraction function
>>> extract(my_doc, options=options)
See the ``settings.py`` file for a full example.
Metadata extraction
^^^^^^^^^^^^^^^^^^^
- ``with_metadata=True``: extract metadata fields and include them in the output
- ``only_with_metadata=True``: only output documents featuring all essential metadata (date, title, url)
Date
~~~~
Among metadata extraction, dates are handled by an external module: `htmldate <https://github.com/adbar/htmldate>`_. By default, focus is on original dates and the extraction replicates the *fast/no_fallback* option.
`Custom parameters <https://htmldate.readthedocs.io/en/latest/corefunctions.html#handling-date-extraction>`_ can be passed through the extraction function or through the ``extract_metadata`` function in ``trafilatura.metadata``, most notably:
- ``extensive_search`` (boolean), to activate further heuristics (higher recall, lower precision)
- ``original_date`` (boolean) to look for the original publication date,
- ``outputformat`` (string), to provide a custom datetime format,
- ``max_date`` (string), to set the latest acceptable date manually (YYYY-MM-DD format).
.. code-block:: python
# import the extract() function, use a previously downloaded document
# pass the new parameters as dict
>>> extract(downloaded, output_format="xml", date_extraction_params={
"extensive_search": True, "max_date": "2018-07-01"
})
URL
~~~
Even if the page to process has already been downloaded it can still be useful to pass the URL as an argument. See this `previous bug <https://github.com/adbar/trafilatura/issues/75>`_ for an example:
.. code-block:: python
# define a URL and download the example
>>> url = "https://web.archive.org/web/20210613232513/https://www.thecanary.co/feature/2021/05/19/another-by-election-headache-is-incoming-for-keir-starmer/"
>>> downloaded = fetch_url(url)
# content discarded since necessary metadata couldn't be extracted
>>> bare_extraction(downloaded, only_with_metadata=True)
>>>
# date found in URL, extraction successful
>>> bare_extraction(downloaded, only_with_metadata=True, url=url)
Memory use
^^^^^^^^^^
Trafilatura uses caches to speed up extraction and cleaning processes. This may lead to memory leaks in some cases, particularly in large-scale applications. If that happens you can reset all cached information in order to release RAM:
.. code-block:: python
# import the function
>>> from trafilatura.meta import reset_caches
# use it at any given point
>>> reset_caches()
Input/Output types
------------------
Python objects as output
^^^^^^^^^^^^^^^^^^^^^^^^
The extraction can be customized using a series of parameters, for more see the `core functions <corefunctions.html>`_ page.
The function ``bare_extraction`` can be used to bypass output conversion, it returns Python variables for metadata (dictionary) as well as main text and comments (both LXML objects).
.. code-block:: python
>>> from trafilatura import bare_extraction
>>> bare_extraction(downloaded)
Raw HTTP response objects
^^^^^^^^^^^^^^^^^^^^^^^^^
The ``fetch_response()`` function can pass a response object straight to the extraction.
This can be useful to get the final redirection URL with ``response.url`` and then pass is directly as a URL argument to the extraction function:
.. code-block:: python
# necessary components
>>> from trafilatura import fetch_response, bare_extraction
# load an example
>>> response = fetch_response("https://www.example.org")
# perform extract() or bare_extraction() on Trafilatura's response object
>>> bare_extraction(response.data, url=response.url) # here is the redirection URL
LXML objects
^^^^^^^^^^^^
The input can consist of a previously parsed tree (i.e. a *lxml.html* object), which is then handled seamlessly:
.. code-block:: python
# define document and load it with LXML
>>> from lxml import html
>>> my_doc = """<html><body><article><p>
Here is the main text.
</p></article></body></html>"""
>>> mytree = html.fromstring(my_doc)
# extract from the already loaded LXML tree
>>> extract(mytree)
'Here is the main text.'
Interaction with BeautifulSoup
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Here is how to convert a BS4 object to LXML format in order to use it with Trafilatura:
.. code-block:: python
>>> from bs4 import BeautifulSoup
>>> from lxml.html.soupparser import convert_tree
>>> from trafilatura import extract
>>> soup = BeautifulSoup("<html><body><time>The date is Feb 2, 2024</time></body></html>", "lxml")
>>> lxml_tree = convert_tree(soup)[0]
>>> extract(lxml_tree)
Navigation
----------
Three potential navigation strategies are currently available: feeds (mostly for fresh content), sitemaps (for exhaustivity, all potential pages as listed by the owners) and discovery by web crawling (i.e. by following the internal links, more experimental).
Feeds
^^^^^
The function ``find_feed_urls`` is a all-in-one utility that attempts to discover the feeds from a webpage if required and/or downloads and parses feeds. It returns the extracted links as list, more precisely as a sorted list of unique links.
.. code-block:: python
# import the feeds module
>>> from trafilatura import feeds
# use the homepage to automatically retrieve feeds
>>> mylist = feeds.find_feed_urls('https://www.theguardian.com/')
>>> mylist
['https://www.theguardian.com/international/rss', '...'] # and so on
# use a predetermined feed URL directly
>>> mylist = feeds.find_feed_urls('https://rss.nytimes.com/services/xml/rss/nyt/HomePage.xml')
>>> mylist is not []
True # it's not empty
.. note::
The links are seamlessly filtered for patterns given by the user, e.g. using ``https://www.un.org/en/`` as argument implies taking all URLs corresponding to this category.
An optional argument ``target_lang`` makes it possible to filter links according to their expected target language. A series of heuristics are applied on the link path and parameters to try to discard unwanted URLs, thus saving processing time and download bandwidth.
.. code-block:: python
# the feeds module has to be imported
# search for feeds in English
>>> mylist = feeds.find_feed_urls('https://www.un.org/en/rss.xml', target_lang='en')
>>> mylist is not []
True # links found as expected
# target_lang set to Japanese, the English links are discarded
>>> mylist = feeds.find_feed_urls('https://www.un.org/en/rss.xml', target_lang='ja')
>>> mylist
[]
For more information about feeds and web crawling see:
- This blog post: `Using RSS and Atom feeds to collect web pages with Python <https://adrien.barbaresi.eu/blog/using-feeds-text-extraction-python.html>`_
- This Youtube tutorial: `Extracting links from ATOM and RSS feeds <https://www.youtube.com/watch?v=NW2ISdOx08M&list=PL-pKWbySIRGMgxXQOtGIz1-nbfYLvqrci&index=2&t=136s>`_
Sitemaps
^^^^^^^^
- Youtube tutorial: `Learn how to process XML sitemaps to extract all texts present on a website <https://www.youtube.com/watch?v=uWUyhxciTOs>`_
.. code-block:: python
# load sitemaps module
>>> from trafilatura import sitemaps
# automatically find sitemaps by providing the homepage
>>> mylinks = sitemaps.sitemap_search('https://www.theguardian.com/')
# the target_lang argument works as explained above
>>> mylinks = sitemaps.sitemap_search('https://www.un.org/', target_lang='en')
The links are also seamlessly filtered for patterns given by the user, e.g. using ``https://www.theguardian.com/society`` as argument implies taking all URLs corresponding to the society category.
Web crawling
^^^^^^^^^^^^
See the `documentation page on web crawling <crawls.html>`_ for more information.
.. hint::
For more information on how to refine and filter a URL collection, see the underlying `courlan <https://github.com/adbar/courlan>`_ library.
Deprecations
------------
The following functions and arguments are deprecated:
- extraction:
- ``process_record()`` function → use ``extract()`` instead
- ``csv_output``, ``json_output``, ``tei_output``, ``xml_output`` → use ``output_format`` parameter instead
- ``bare_extraction(as_dict=True)`` → the function returns a ``Document`` object, use ``.as_dict()`` method on it
- ``bare_extraction()`` and ``extract()``: ``no_fallback`` → use ``fast`` instead
- ``max_tree_size`` parameter moved to ``settings.cfg`` file
- downloads: ``decode`` argument in ``fetch_url()`` → use ``fetch_response`` instead
- utils: ``decode_response()`` function → use ``decode_file()`` instead
- metadata: ``with_metadata`` (include metadata) had once the effect of today's ``only_with_metadata`` (only documents with necessary metadata)
Reference in New Issue View Git Blame Copy Permalink