Word Form Record Set

A Word Form Record Set is a lexical dataset that is a dataset of word form records.

• AKA: Word Inventory, Word Database, Word List, Word Form Inventory.
• Context:
  • It can (typically) represent a Word Form Set as a structured collection.
  • It can (typically) contain word form records, each describing a surface form of a word.
  • It can (typically) store orthographic forms of words.
  • It can (typically) associate each word form record with a lemma.
  • It can (typically) index word forms for fast lookup operations.
  • It can (typically) support word membership testing against a known vocabulary.
  • It can (typically) be organized by word frequency or alphabetical order.
  • It can (often) include part-of-speech tags for each word form.
  • It can (often) include morphological features such as grammatical number, grammatical gender, and grammatical tense.
  • It can (often) include pronunciation information via phonetic transcriptions.
  • It can (often) record word form frequency counts from a reference corpus.
  • It can (often) be derived from a text corpus through tokenization and word form extraction.
  • It can (often) be filtered by language to represent a specific natural language.
  • It can (often) exclude stopwords or explicitly enumerate them.
  • It can support a Word-level Analysis Task.
  • It can support a Spell-Checking Task as a reference vocabulary.
  • It can support a Text Normalization Task via word form matching.
  • It can support an Autocompletion Task via prefix lookup.
  • It can be serialized as a plain text file, a database table, or a trie data structure.
  • It can be curated by a lexicographer or generated by an automated pipeline.
  • It can be versioned to track vocabulary changes over time.
  • It can range from being a Small Word Form Record Set to being a Large Word Form Record Set, depending on its record count.
  • It can range from being a Monolingual Word Form Record Set to being a Multilingual Word Form Record Set, depending on its language coverage.
  • It can range from being a Flat Word Form Record Set to being a Structured Word Form Record Set, depending on its record schema.
  • It can range from being a Manually-Curated Word Form Record Set to being an Automatically-Generated Word Form Record Set, depending on its construction method.
  • …
• Example(s):
  • Stopword Lists, such as:
    • the NLTK Stopword List for stopword removal.
    • the Snowball Stopword List for text preprocessing.
  • Common Word Lists, of common words, such as:
    • the Google 10000 Most Common English Words list.
    • the Dolch Word List (1936) of sight words.
  • Frequency Word Lists, such as:
    • the British National Corpus Word Frequency List.
    • the COCA Word Frequency List from the Corpus of Contemporary American English.
  • Spell-Check Dictionarys, such as:
    • the Hunspell Dictionary used by LibreOffice.
    • the GNU Aspell Word List.
  • Wordlists for password cracking, such as the RockYou Wordlist (2009).
  • …
• Counter-Example(s):
  • a Word Sense Inventory, which enumerates word senses rather than surface forms.
  • a Lexical Database such as WordNet (1995), which encodes semantic relations beyond a flat word form listing.
  • a Character Set, which enumerates characters rather than word forms.
  • a N-Gram Model, which stores word sequence probabilities rather than individual word form records.
  • a Named Entity List, which enumerates named entitys rather than general word forms.
  • …
• See: Lexical Database, Word Sense Inventory, Word Form Record, Word Form Set, Vocabulary, Lexicon, Dictionary, Stopword List, Word Frequency, Tokenization.

---

References

• • (Manning et al., 2008) ⇒ Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. (2008). "Introduction to Information Retrieval." Cambridge University Press. [1]