참고소스 수정본

참고/trafilatura-master/docs/corpus-data.rst

@@ -0,0 +1,76 @@
+Working with corpus data
+========================
+
+
+.. meta::
+    :description lang=en:
+        After gathering texts from the Web, what to do next? This page lists options to work with output generated by Trafilatura.
+
+
+After gathering texts from the Web, what to do next? This page lists options to work with output generated by Trafilatura.
+
+
+Generic solutions in Python
+---------------------------
+
+
+Data science
+~~~~~~~~~~~~
+
+- Load the input into the data analysis library Pandas:
+   - `read_csv <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html>`_
+   - `read_json <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_json.html>`_
+
+
+Natural language processing
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For a first hand approach to NLP pipelines, see `Textblob <https://textblob.readthedocs.io/en/dev/>`_ or the `Natural Language Toolkit (NTLK) <https://www.nltk.org/>`_.
+
+
+Accessible tutorials:
+    - `Part-of-Speech Tagging <https://melaniewalsh.github.io/Intro-Cultural-Analytics/Text-Analysis/POS-Keywords.html>`_
+    - `TF-IDF with Scikit-Learn <https://melaniewalsh.github.io/Intro-Cultural-Analytics/Text-Analysis/TF-IDF-Scikit-Learn.html>`_
+
+
+Specific tools:
+    - Topic modeling, including word2vec models: `Gensim tutorials <https://radimrehurek.com/gensim/auto_examples/>`_
+    - `Scattertext <https://github.com/JasonKessler/scattertext>`_ is a tool for finding distinguishing terms in corpora, and presenting them in an interactive scatter plot.
+
+
+
+Formats and software used in corpus linguistics
+-----------------------------------------------
+
+
+Input/Output formats: TXT, XML and XML-TEI are quite frequent in corpus linguistics.
+
+- Han., N.-R. (2022). "Transforming Data", The Open Handbook of Linguistic Data.
+
+
+The XML and XML-TEI formats
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+See `A Gentle Introduction to XML <https://tei-c.org/release/doc/tei-p5-doc/en/html/SG.html>`_ or the Python package `xmltodict <https://github.com/martinblech/xmltodict>`_ which provide ways to directly read the files and work with the data as if it were in JSON format.
+
+
+Corpus analysis tools
+~~~~~~~~~~~~~~~~~~~~~
+
+
+- `Antconc <https://www.laurenceanthony.net/software/antconc/>`_ is expected to work with TXT files
+- `CorpusExplorer <https://notes.jan-oliver-ruediger.de/software/corpusexplorer-overview/>`_ supports CSV, TXT and various XML formats
+- `Corpus Workbench (CWB) <https://cwb.sourceforge.io/>`_ uses verticalized texts whose origin can be in TXT or XML format
+- `LancsBox <http://corpora.lancs.ac.uk/lancsbox/>`_ support various formats, notably TXT & XML
+- `TXM <https://txm.gitpages.huma-num.fr/textometrie/en/>`_ (textometry platform) can take TXT, XML & XML-TEI files as input
+- `Voyant <https://voyant-tools.org/>`_ support various formats, notably TXT, XML & XML-TEI
+- `Wmatrix <https://ucrel.lancs.ac.uk/wmatrix/>`_ can work with TXT and XML
+- `WordSmith <https://lexically.net/wordsmith/index.html>`_ supports TXT and XML
+
+Further corpus analysis software can be found on `corpus-analysis.com <https://corpus-analysis.com/>`_.
+
+
+Generic NLP solutions
+---------------------
+
+For natural language processing see this list of open-source/off-the-shelf `NLP tools for German <https://github.com/adbar/German-NLP>`_ and `further lists for other languages <https://github.com/adbar/German-NLP#Comparable-lists>`_.