77 lines
3.3 KiB
ReStructuredText
77 lines
3.3 KiB
ReStructuredText
|
Working with corpus data
|
||
|
|
========================
|
||
|
|
|
||
|
|
|
||
|
|
.. meta::
|
||
|
|
:description lang=en:
|
||
|
|
After gathering texts from the Web, what to do next? This page lists options to work with output generated by Trafilatura.
|
||
|
|
|
||
|
|
|
||
|
|
After gathering texts from the Web, what to do next? This page lists options to work with output generated by Trafilatura.
|
||
|
|
|
||
|
|
|
||
|
|
Generic solutions in Python
|
||
|
|
---------------------------
|
||
|
|
|
||
|
|
|
||
|
|
Data science
|
||
|
|
~~~~~~~~~~~~
|
||
|
|
|
||
|
|
- Load the input into the data analysis library Pandas:
|
||
|
|
- `read_csv <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html>`_
|
||
|
|
- `read_json <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_json.html>`_
|
||
|
|
|
||
|
|
|
||
|
|
Natural language processing
|
||
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
|
|
||
|
|
For a first hand approach to NLP pipelines, see `Textblob <https://textblob.readthedocs.io/en/dev/>`_ or the `Natural Language Toolkit (NTLK) <https://www.nltk.org/>`_.
|
||
|
|
|
||
|
|
|
||
|
|
Accessible tutorials:
|
||
|
|
- `Part-of-Speech Tagging <https://melaniewalsh.github.io/Intro-Cultural-Analytics/Text-Analysis/POS-Keywords.html>`_
|
||
|
|
- `TF-IDF with Scikit-Learn <https://melaniewalsh.github.io/Intro-Cultural-Analytics/Text-Analysis/TF-IDF-Scikit-Learn.html>`_
|
||
|
|
|
||
|
|
|
||
|
|
Specific tools:
|
||
|
|
- Topic modeling, including word2vec models: `Gensim tutorials <https://radimrehurek.com/gensim/auto_examples/>`_
|
||
|
|
- `Scattertext <https://github.com/JasonKessler/scattertext>`_ is a tool for finding distinguishing terms in corpora, and presenting them in an interactive scatter plot.
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
Formats and software used in corpus linguistics
|
||
|
|
-----------------------------------------------
|
||
|
|
|
||
|
|
|
||
|
|
Input/Output formats: TXT, XML and XML-TEI are quite frequent in corpus linguistics.
|
||
|
|
|
||
|
|
- Han., N.-R. (2022). "Transforming Data", The Open Handbook of Linguistic Data.
|
||
|
|
|
||
|
|
|
||
|
|
The XML and XML-TEI formats
|
||
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
|
|
||
|
|
See `A Gentle Introduction to XML <https://tei-c.org/release/doc/tei-p5-doc/en/html/SG.html>`_ or the Python package `xmltodict <https://github.com/martinblech/xmltodict>`_ which provide ways to directly read the files and work with the data as if it were in JSON format.
|
||
|
|
|
||
|
|
|
||
|
|
Corpus analysis tools
|
||
|
|
~~~~~~~~~~~~~~~~~~~~~
|
||
|
|
|
||
|
|
|
||
|
|
- `Antconc <https://www.laurenceanthony.net/software/antconc/>`_ is expected to work with TXT files
|
||
|
|
- `CorpusExplorer <https://notes.jan-oliver-ruediger.de/software/corpusexplorer-overview/>`_ supports CSV, TXT and various XML formats
|
||
|
|
- `Corpus Workbench (CWB) <https://cwb.sourceforge.io/>`_ uses verticalized texts whose origin can be in TXT or XML format
|
||
|
|
- `LancsBox <http://corpora.lancs.ac.uk/lancsbox/>`_ support various formats, notably TXT & XML
|
||
|
|
- `TXM <https://txm.gitpages.huma-num.fr/textometrie/en/>`_ (textometry platform) can take TXT, XML & XML-TEI files as input
|
||
|
|
- `Voyant <https://voyant-tools.org/>`_ support various formats, notably TXT, XML & XML-TEI
|
||
|
|
- `Wmatrix <https://ucrel.lancs.ac.uk/wmatrix/>`_ can work with TXT and XML
|
||
|
|
- `WordSmith <https://lexically.net/wordsmith/index.html>`_ supports TXT and XML
|
||
|
|
|
||
|
|
Further corpus analysis software can be found on `corpus-analysis.com <https://corpus-analysis.com/>`_.
|
||
|
|
|
||
|
|
|
||
|
|
Generic NLP solutions
|
||
|
|
---------------------
|
||
|
|
|
||
|
|
For natural language processing see this list of open-source/off-the-shelf `NLP tools for German <https://github.com/adbar/German-NLP>`_ and `further lists for other languages <https://github.com/adbar/German-NLP#Comparable-lists>`_.
|