Word Mention Segmentation Task

From GM-RKB
Jump to: navigation, search

A word mention segmentation task is a mention segmentation task that is restricted to the detection of the word mentions within a text.

References

2009

  • (Wikipedia, 2009) ⇒ http://en.wikipedia.org/wiki/Text_segmentation#Word_segmentation
    • Word segmentation is the problem of dividing a string of written language into its component words. In English and many other modern languages using some form of the Latin alphabet dividing text using the space character is a good approximation to word segmentation. (Some examples where the space character alone may not be sufficient include contractions like can't for can not.) However the equivalent to this character is not found in all written scripts and without it word segmentation is a difficult problem. Languages which do not have a trivial word segmentation process include Chinese, Japanese and Thai.
  • (Lin, 2009) ⇒ Dekang Lin. (2009). "Combining Language Modeling and Discriminative Classification for Word Segmentation." In: Proceedings of the CICLing Conference (CICLing 2009).

2007

  • (Schmid, 2007) ⇒ Helmut Schmid. (2007). "Tokenizing." In: Corpus Linguistics: An International Handbook. Walter de Gruyter, Berlin.

2005

2004

  • (Sproat, 2004) ⇒ Richard Sproat. (2004). "Issue of Chinese Word Segmentation." In: Journal of Chinese Language and Computing 14(3).

2003

2000

1999

  • (Ge et al, 1999) ⇒ Xianping Ge, Wanda Pratt, and Padhraic Smyth. (1999). "Discovering Chinese Words from Unsegmented Text." In: Proceedings of [[SIGIR]-1999.
  • (Manning and Schütze, 1999) ⇒ Christopher D. Manning and Hinrich Schütze. (1999). "Foundations of Statistical Natural Language Processing." The MIT Press.
    • Many languages do not put spaces in between words at all, and so the basic word division algorithm of breaking on whitespace is of no use at all. Such languages include major East-Asian languages/scripts such as Chinese, Japanese, and Thai. Ancient Greek was also written by Ancient Greeks without words spaces. Spaces were introduced (together with accents marks, etc.) by those who came afterwards. In such languages, word segmentation is a much more major and challenging task.
    • While maintaining most words spaces, in German compound nouns are written as single words, for example Lebensversicherungsgesellschaftsangestellter 'life insurance company employee.' In many ways this makes linguistic sense, as compounds are a single words, at least phonologically. But for process purposes one may wish to divide such a compound, or at least to be aware of the internal structure of the words, and this becomes a limited words segmentation task. While not the rule, joining of compunds sometimes also happens in English, especially when they are common and have a specialized meaning. We noted above that one finds both data base and database. As another examples, while hard disk is more common, one sometimes finds harddisk in the computer press.
    • Until now, the problems we have dealt with have mainly involved splitting apart sequence of characters where the word division are not shown by whitespace. But the opposite problem of wanting to lump things together also occurs. Where, things are separate by whitespace but we may with to regard them as a single word. One possible case is the reverse of the German compound problem. If one decides to treat database as one word, one may wish to treat it as one word even when it is written as database. More common cases are things such as phone numbers, where we may with to recard 9465 1873 as a single 'word,' or in the cases of multi-part names such as New York or San Francisco. An especially difficult case is when this problem interacts with hyphenation as in a phrase like this one: "the New York-New Haven railroad." Here the hyphen does not express grouping of just the immediate adjacent graphic words - treating York-New as a semantic unit would be a big mistake.
    • Other cases are of more linguistic interest. For many purposes, one would want to regard phrasal verbs (make up, work out) as single lexemes (section 3.1.4), but this case is especially tricky since in many cases the particle is separable from the verb (I couldn't work' the answer out), and so in general identification of possible phrasal verbs will have to be left to subsequent processing. One might also want to treat as a single lexeme creating other fixed phrases, such as in spite of/, in order to, and because of, but typically a tokenizer will regard them as separate words. A partial implementation of this approach occurs in the LOB corpus where certain pairs of words such as because of are tagged with a single part of speech, here preposition, by means of using so-called ditto tags.

1997

  • David D. Palmer. (1997). "A Trainable Rule-based Algorithm for Word Segmentation." In: Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
    • ABSTRACT: This paper presents a trainable rule-based algorithm for performing word segmentation. The algorithm provides a simple, language-independent alternative to large-scale lexical-based segmenters requiring large amounts of knowledge engineering. As a stand-alone segmenter, we show our algorithm to produce high performance Chinese segmentation. In addition, we show the transformation-based algorithm to be effective in improving the output of several existing word segmentation algorithms in three different languages.

1996

  • (Sproat et al, 1996) ⇒ Richard Sproat, William A. Gale, Chilin Shih, and Nancy Chang. (1996). "A Stochastic Finite-state Word-Segmentation Algorithm for Chinese." In: Computational Linguistics, 22(3).
    • ABSTRACT: The initial stage of text analysis for any NLP task usually involves the tokenization of the input into words. For languages like English one can assume, to a first approximation, that word boundaries are given by whitespace or punctuation. In various Asian languages, including Chinese, on the other hand, whitespace is never used to delimit words, so one must resort to lexical information to "reconstruct" the word-boundary information. In this paper we present a stochastic finite-state model wherein the basic workhorse is the weighted finite-state transducer. The model segments Chinese text into dictionary entries and words derived by various productive lexical processes, and---since the primary intended application of this model is to text-to-speech synthesis---provides pronunciations for these words. We evaluate the system's performance by comparing its segmentation "judgments" with the judgements of a pool of human segmenters, and the system is shown to perform quite well.
    • It includes an example of correct and incorrect Word Segmentation of "日文章魚怎麼說" (How do you say octopus in Japanese?)
      • [日文] [章魚] [怎麼] [說] ([Japanese] [octopus] [how] [say])
      • [日] [文章] [魚] [怎麼] [說] ([Japan] [essay] [fish] [how] [say]).
    • Distinguishes between: Orthographic Word, Syntactic Word, Dictionary Word, and Phonological Word.

1994

  • (Grefenstette & Tapanainen) ⇒ Gregory Grefenstette, and Pasi Tapanainen. (1994). "What is a Word, What is a Sentence? Problems of Tokenization." In: Proceedings of 3rd Conference on Computational Lexicography and Text Research (COMPLEX 1994).
  • (Sproat & al, 1994) ⇒ Richard Sproat, Chilin Shih, William A. Gale, Nancy Chang. (1994). "A Stochastic Finite-State Word-Segmentation Algorithm for Chinese." In: Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics: Proceedings of the Conference (ACL 1994).

1990

  • (Sproat & Shi, 1990) ⇒ Richard Sproat, and Chilin Shih. (1990). "A Statistical Method for Finding Word Boundaries in Chinese Text." In: Computer Processing of Chinese and Oriental Languages, 4.

1987

Retrieved from "http://www.gabormelli.com/RKB/index.php?title=Word_Mention_Segmentation_Task&oldid=7062"
Personal tools
Namespaces

Variants
Views
Actions
Navigation
Toolbox