Background data for: Advancing our understanding of dispersion measures in corpus researchdoi:10.18710/FVHTFMDataverseNO2024-11-261Sönning, Lukas, 2024, "Background data for: Advancing our understanding of dispersion measures in corpus research", https://doi.org/10.18710/FVHTFM, DataverseNO, V1Background data for: Advancing our understanding of dispersion measures in corpus researchdoi:10.18710/FVHTFMSönning, LukasUniversity of Bamberg2023-06-28Bamberg, GermanyMAXQDA PlusRDataverseNOThe Tromsø Repository of Language and Linguistics (TROLLing)Sönning, LukasSönning, Lukas2023-12-19Arts and Humanitiesdispersioncorpus linguisticsmethodologycorpus designBrown Corpusdispersion measureslexical dispersionword importancevocabulary listsword frequency liststext-level analysisfrequencyJuilland's DGries' DPDAEnglish<p><b>Dataset description</b></p> <p>This dataset contains background data and supplementary material for Sönning (forthcoming), a study that looks at the behavior of dispersion measures when applied to text-level frequency data. For the literature survey reported in that study, which examines how dispersion measures are used in corpus-based work, it includes tabular files listing the 730 research articles that were examined as well as annotations for those studies that measured dispersion in the corpus-linguistic (and lexicographic) sense. As for the corpus data that were used to train the statistical model parameters underlying the simulation study reported in that paper, the dataset contains a term-document matrix for the 49,604 unique word forms (after conversion to lower-case) that occur in the Brown Corpus. Further, R scripts are included that document in detail how the Brown Corpus XML files, which are available from the Natural Language Toolkit (Bird et al. 2009; https://www.nltk.org/), were processed to produce this data arrangement.</p><p><b>Abstract: Related publication</b></p> <p>This paper offers a survey of recent corpus-based work, which shows that dispersion is typically measured across the text files in a corpus. Systematic insights into the behavior of measures in such distributional settings are currently lacking, however. After a thorough discussion of six prominent indices, we investigate their behavior on relevant frequency distributions, which are designed to mimic actual corpus data. Our evaluation considers different distributional settings, i.e. various combinations of frequency and dispersion values. The primary focus is on the response of measures to relatively high and low sub-frequencies, i.e. texts in which the item or structure of interest is over- or underrepresented (if not absent). We develop a simple method for constructing sensitivity profiles, which allow us to draw instructive comparisons among measures. We observe that these profiles vary considerably across distributional settings. While D and DP appear to show the most balanced response contours, our findings suggest that much work remains to be done to understand the performance of measures on items with normalized frequencies below 100 per million words.</p>1961-01-011961-12-312023-06-142023-06-28United Statestextual linguistic datacorpus dataobservational data<p>A Standard Corpus of Present-Day Edited American English, for use with Digital Computers (the Brown Corpus). 1964, 1971, 1979. Compiled by W. N. Francis and H. Kučera. Brown University. Providence, Rhode Island.</p> <p>Brown Corpus XML files are available from the Natural Language Toolkit (<a href="https://www.nltk.org">https://www.nltk.org</a>).</p> <p>The extracted words included in the data files of this dataset represent insubstantial portions of the Brown Corpus; they do not represent coherent stretches of text. Reuse of such excerpts is permitted under exceptions in IPR and database protection regulations, such as the Norwegian Copyright Act (cf. <a href="https://lovdata.no/lov/2018-06-15-40/§24">§ 24 Eneretten til databaser</a>), the <a href="http://data.europa.eu/eli/dir/1996/9/oj">EU Database Directive</a> (cf. art 8 Rights and obligations of lawful users), and Fair use (cf. <a href="https://www.copyright.gov/fair-use/more-info.html">US Copyright Act</a>).</p><p>With the exception of the tab-delimited term-document matrix brown_tdm.tsv and the tab-delimited document-term matrix brown_dtm.tsv, the dataset "Background data for: Advancing our understanding of dispersion measures in corpus research" has been marked as dedicated to the public domain, as described here: <a href="https://creativecommons.org/publicdomain/zero/1.0/">https://creativecommons.org/publicdomain/zero/1.0/</a>.</p> <p>Our Community Norms as well as good scientific practices expect that proper credit is given via citation. Please use the data citation shown on the dataset page.</p> <p>The tab-delimited term-document matrix brown_tdm.tsv and the tab-delimited document-term matrix brown_dtm.tsv contain word forms that have been extracted from the Brown Corpus, available from the Natural Language Toolkit (<a href="https://www.nltk.org/">https://www.nltk.org/</a>), under limitations and exceptions to IPR and database protection regulations. The contribution of the author of the present dataset to these files, as detailed in the ReadMe file, is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license, as described here: <a href="https://creativecommons.org/licenses/by/4.0/">https://creativecommons.org/licenses/by/4.0/</a>. Reusers should note that this license does not apply to the word forms extracted from the Brown Corpus.</p> Sönning, Lukas. 2025. Advancing our understanding of dispersion measures in corpus research. Corpora 20(1). 3-35.10.3366/cor.2025.0326Sönning, Lukas. 2025. Advancing our understanding of dispersion measures in corpus research. Corpora 20(1). 3-35.00ReadMe_understanding_dispersion.txtFile describing the datasettext/plain2023-12-08_dispersion_survey_all_articles.tsvTab-delimited data table containing the 730 research articles that entered our literature surveytext/tsv2023-12-09_dispersion_survey.tsvTab-delimited data table containing annotations for the 38 studies in our survey that assessed dispersiontext/tsvbrown_dtm.tsvTab-delimited document-term matrix for the word forms in the Brown Corpustext/tsvbrown_tdm.tsvTab-delimited term-document matrix for the word forms in the Brown Corpustext/tsvscript_brown_data_retrieval.qmdR quarto script documenting retrieval of the data from the Brown XML filesapplication/octet-stream