{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "59249398-24e8-4339-a6ad-1c0ef69fb25e",
   "metadata": {},
   "source": [
    "# Trafilatura: Overview and main functions\n",
    "\n",
    "## 1. Installation\n",
    "\n",
    "`pip install trafilatura`\n",
    "\n",
    "#### Updating\n",
    "\n",
    "`pip install -U trafilatura`\n",
    "\n",
    "For more info see [Installation](https://trafilatura.readthedocs.io/en/latest/installation.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8505f5ea-b7dd-455e-94a2-0950ee5ebdc5",
   "metadata": {},
   "source": [
    "## 2. Downloads"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8998cb12-efd0-4822-b214-77a94c79aead",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura import fetch_url\n",
    "\n",
    "document = fetch_url('https://www.example.org')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64f173b8-0778-4126-912e-f788b622ddb2",
   "metadata": {},
   "source": [
    "For parallel and mass downloads see [Downloads page](https://trafilatura.readthedocs.io/en/latest/downloads.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5186981-567b-4720-8a53-a7a456ace98b",
   "metadata": {},
   "source": [
    "## 3. Function `extract()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a127cca1-7bd6-41dc-851b-3920b6372e84",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.\n",
      "More information...\n"
     ]
    }
   ],
   "source": [
    "from trafilatura import extract\n",
    "\n",
    "text = extract(document)\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53309bee-cf37-4182-b0e9-1b5c40439a8b",
   "metadata": {},
   "source": [
    "### Several output formats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "69f9c7c8-1cb8-43d7-9ef9-b304d7f60f3f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<doc title=\"Example Domain\" categories=\"\" tags=\"\" fingerprint=\"FYMpE8PW9rnzogCzwvwGlIXzkHw=\">\n",
      "  <main>\n",
      "    <p>This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.</p>\n",
      "    <p>More information...</p>\n",
      "  </main>\n",
      "  <comments/>\n",
      "</doc>\n"
     ]
    }
   ],
   "source": [
    "xml_text = extract(document, output_format='xml')\n",
    "print(xml_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7eb251d-97aa-42f2-b18e-a7b820c0d56b",
   "metadata": {},
   "source": [
    "## Further extraction functions\n",
    "\n",
    "Documentation pages:\n",
    "\n",
    "- [extract](https://trafilatura.readthedocs.io/en/latest/corefunctions.html#trafilatura.extract)\n",
    "- [bare_extraction](https://trafilatura.readthedocs.io/en/latest/corefunctions.html#trafilatura.bare_extraction)\n",
    "- [baseline](https://trafilatura.readthedocs.io/en/latest/corefunctions.html#trafilatura.baseline)\n",
    "- [extract_metadata](https://trafilatura.readthedocs.io/en/latest/corefunctions.html#trafilatura.extract_metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09f30bc4-29c9-49bc-85ca-cec93ffc34ce",
   "metadata": {},
   "source": [
    "### Input formats\n",
    "\n",
    "#### Option 1: Unicode strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9f3ecad5-3fe5-409e-870c-99f0970aafb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# filename = 'myfile.html'\n",
    "# with open(filename, encoding='utf-8') as f:\n",
    "#    document = f.read()\n",
    "# extract(document)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed69754e-5e3d-4d16-a254-0d68fc2e469d",
   "metadata": {},
   "source": [
    "#### Option 2: Parsed trees (LXML objects)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8818602a-2a2f-4fdb-8e47-f0bd8d560bfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from lxml import html\n",
    "\n",
    "tree = html.fromstring(document)\n",
    "#extract(tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c558b83a-79be-481b-a5cb-ffada86444c7",
   "metadata": {},
   "source": [
    "### Output comparison: Baseline vs. Full extract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "432fa488-bcc9-42d6-9afe-59cb28427ad4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura import baseline\n",
    "\n",
    "lxml_object, text, length = baseline(document)\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3849cc24-1628-4a9a-8a20-1017a038a99b",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = extract(document)\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecc6a5b7-ca81-4c79-bcbe-678547d73d83",
   "metadata": {},
   "source": [
    "### Work with Python objects using `bare_extraction()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b5538b3-14f7-4a4e-bdbc-fee2a991d323",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura import bare_extraction\n",
    "\n",
    "doc_dict = bare_extraction(document)\n",
    "print(doc_dict.keys())\n",
    "print(doc_dict['sitename'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c0c2685-ae86-45f8-ad5c-9c10a4fcb92a",
   "metadata": {},
   "source": [
    "### Focus on metadata with `extract_metadata()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de489a51-2799-498d-aa2d-9c58f790ad24",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura import extract_metadata\n",
    "\n",
    "extract_metadata(document)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5bcb55b-60a6-4dae-ad86-bb16a39ddd88",
   "metadata": {},
   "source": [
    "## Text discovery\n",
    "\n",
    "For more info see documentation page on [Python functions](https://trafilatura.readthedocs.io/en/latest/usage-python.html).\n",
    "\n",
    "### Feeds\n",
    "\n",
    "Link discovery over [Web Feeds](https://en.wikipedia.org/wiki/Web_feed):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d0b2191-97a4-4347-a872-f835597a4583",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura.feeds import find_feed_urls\n",
    "\n",
    "links = find_feed_urls('https://www.nzz.ch')\n",
    "# 5 first links in the feed\n",
    "print('\\n'.join(links[:5]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "deb23164-1985-4de7-b6fe-bfb478193453",
   "metadata": {},
   "source": [
    "### Sitemaps\n",
    "\n",
    "Link discovery over [Sitemaps](https://en.wikipedia.org/wiki/Sitemaps):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2760f39-2c1c-4cff-a06a-740dbc220f8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura.sitemaps import sitemap_search\n",
    "\n",
    "links = sitemap_search('https://www.sitemaps.org', target_lang='de')\n",
    "# 5 first links found in the sitemap\n",
    "print('\\n'.join(links[:5]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7f8f173-a6c0-47bf-8a8f-3184d876e495",
   "metadata": {},
   "source": [
    "### Web crawling\n",
    "\n",
    "Link discovery over [Web Crawling](https://en.wikipedia.org/wiki/Web_crawler), i.e. exploratory process to find internal links and potential text-based web pages.\n",
    "\n",
    "Such tools are often called *crawler* or *spider*. The pages that are still to visit are often called *crawl frontier*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da261699-6bdb-4bc2-bd7a-145ba69dd55f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura.spider import focused_crawler\n",
    "\n",
    "to_visit, known_urls = focused_crawler('https://www.telebasel.ch/', max_seen_urls=3)\n",
    "print(len(to_visit), len(known_urls))\n",
    "# has to be converted to a list in order to select such slices\n",
    "print('\\n'.join(list(to_visit)[:5]))\n",
    "print('---')\n",
    "print('\\n'.join(list(known_urls)[:5]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "028230fa-bc47-4ace-8db7-8e294a74d1bd",
   "metadata": {},
   "source": [
    "Pages of the type \"author\", \"category\", \"page\", etc. are visited first."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b1b527f-b72a-4435-8995-7b694f4c3f78",
   "metadata": {},
   "source": [
    "Limits of web crawling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c463e6d-3963-427c-948b-31e4c88d5a5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "to_visit, known_urls = focused_crawler('https://www.zhdk.ch/', max_seen_urls=3)\n",
    "print(len(to_visit), len(known_urls))\n",
    "# nothing is found as the content is dynamic in nature\n",
    "print('\\n'.join(list(to_visit)[:5]))\n",
    "print('---')\n",
    "print('\\n'.join(list(known_urls)[:5]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "042d888a-9fa0-442f-acc0-0b0a343c1038",
   "metadata": {},
   "source": [
    "For more info see documentation page on [Web crawling](https://trafilatura.readthedocs.io/en/latest/crawls.html).\n",
    "\n",
    "### Mass downloads\n",
    "\n",
    "- For \"politeness rules\" see [Downloads](https://trafilatura.readthedocs.io/en/latest/downloads.html)\n",
    "- Using already existing web archives:\n",
    "    - Internet Archive, CommonCrawl and others [sources pour corpus web](https://trafilatura.readthedocs.io/en/latest/sources.html)\n",
    "    - Methods, e.g. prefix search with https://index.commoncrawl.org\n",
    "    - Tools, e.g. [cdx_toolkit](https://github.com/cocrawler/cdx_toolkit/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "208cdb04-7573-41f8-a4ec-dff2398d31b1",
   "metadata": {},
   "source": [
    "## Validation of XML-TEI documents\n",
    "\n",
    "### Validation of output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ddad7c3-e2a9-4206-968a-f97a69f4b87d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from trafilatura import bare_extraction\n",
    "from trafilatura.xml import validate_tei\n",
    "\n",
    "doc_dict = bare_extraction(document, output_format='xmltei')\n",
    "tei_tree = doc_dict['body']\n",
    "# not valid (happens frequently)\n",
    "validate_tei(tei_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4d3beeb-9521-437b-8630-48470377392a",
   "metadata": {},
   "source": [
    "### Opening and validating files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c18ad5a4-d163-407a-bd2e-50d9e03dce11",
   "metadata": {},
   "outputs": [],
   "source": [
    "from lxml import etree\n",
    "from trafilatura.xml import validate_tei\n",
    "\n",
    "# Example: file named \"document.xml\"\n",
    "# mytree = etree.parse('document.xml')\n",
    "# validate_tei(mytree)\n",
    "# Output: True or False & error message"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "toc-autonumbering": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}