2008 GenreOrientedSummarization

• (Goldstein Stewart, 2008) ⇒ Jade Goldstein Stewart. (2008). “Genre Oriented Summarization." Ph.D. Thesis, Cargegie Mellon University.

Subject Headings: Genre Oriented Summarization Task, Multi-Document Summarization Algorithm, Maximal Marginal Relevance Summarization Algorithm

Cited By

• ~2 http://scholar.google.com/scholar?q=%22Genre+Oriented+Summarization%22+2008

Quotes

Abstract

Summaries are used in daily life to condense information in a manner suitable for the intended recipient’s use and ideally suit the recipient’s information seeking goals. In the case of text, examples of such summaries include newswire articles, headlines, and the information snippets returned by Google. Previous research has focused on summarizing newswire articles or clusters of newswire documents, scientific articles, books, and extracting opinion (sentiment) sentences from reviews.

Our research addresses how to create short multi-sentence summaries meeting user’s goals within specific genres. The methodology is first to determine the genre of the document and then, based on the genre, present applicable summaries designed to address the user’s information seeking goals. For example, in the movie review genre, a goal-focused summary for a review could be an overview summary, a plot summary, or an opinion summary, each of which has a different focus and hence a different summary composition.

We describe our experiments with genre identification using different sets of features and varying numbers of training documents and show that genre tagging using classifiers (Support Vector Machines and Random Forests) is probably at a sufficient level of accuracy to inform a summarization system. We discuss the creation of goal focused single document summaries for seven genres (newswire articles, editorials, interviews, biographies, movie reviews, product reviews, and product press releases). Our results indicate that genre oriented goal-focused summarization algorithms perform better than our two baselines, lead sentence and the newswire summarization algorithm.

We also examine email summarization and, based on previous research in speech acts, present categories of the communicative intent of the sender. We discuss our experiments in identifying these email speech acts using a small annotated corpus of personal emails. In addition, for textual summaries of a sender’s email, we analyze a human annotated subset of the Enron corpus and based on a user study, suggest that the subject line and one sentence extracted from the email text body may be an effective summary length.

We briefly explore multi-document summarization for the newswire genre and present results indicating that, by using maximal marginal relevance (MMR) to eliminate redundancy, there is more coverage of the subtopics in a cluster than our baseline - which uses our single document newswire summarization algorithm on the concatenation of all articles and no MMR. MMR based summaries were also preferred in a ranking produced by one unbiased human evaluator.

Table of Contents

Chapter 1 Introduction............................................................................................... 1
1.1 Motivation........................................................................................................... 1
1.2 Motivation by Example..................................................................................... 13
1.2.1 Goal Focused Genre Oriented Summarization ......................................... 13
1.2.2 Motivation for Multi-Document Summarization Enhancements ............. 16
1.3 Thesis Statement ............................................................................................... 22
1.4 Thesis Contributions ......................................................................................... 22
1.5 Thesis Outline and Reader’s Guide .................................................................. 23

Chapter 2 Discussion of Summarization................................................................. 25
2.1 Dimensions of Summarization.......................................................................... 25
2.2 Essentials for Useful Summaries ...................................................................... 28
2.2.1 Creating a Summary Addressing the End User’s Goals ........................... 28
2.2.2 Creating a Summary with Good Internal Composition ............................ 30
2.2.3 Ideal Summary Length.............................................................................. 32
2.3 Genres ............................................................................................................... 33
2.4 Factors for Goal Focused Summarization......................................................... 35
2.5 Factors for Multi-Document Summarization.................................................... 37

Chapter 3 Discussion of Our Summarization Systems.......................................... 42
3.1 Single Document Summarization System Description..................................... 42
3.2 Maximal Marginal Relevance........................................................................... 43
3.3 MMR and Single-Document Summarization Evaluation................................. 45
3.3.1 Single Document Summarization Evaluation Dry Run ............................ 46
3.3.2 Single Document Summarization MMR Analysis ................................... 46
3.3.3 Single Document Summarization SUMMAC Evaluation ........................ 47
3.4 Summary Compression Rates ........................................................................... 49
3.5 Multi-Document System Design....................................................................... 51
3.6 Multi-Document Update Summaries ................................................................ 53

Chapter 4 Evaluation Metrics for Summarization Systems ................................. 59
4.1 Metrics Overview............................................................................................ 60
4.2 Metrics for Extractive Summaries ................................................................ 61
4.2.1 Precision-Recall ........................................................................................ 61
4.2.2 Relative Utility.......................................................................................... 62
4.3 Metrics for Generative or Abstractive Summaries...................................... 63
4.3.1 Cosine Similarity ...................................................................................... 63
4.3.2 ROUGE..................................................................................................... 64
4.4 Metrics that take into account semantic equivalence........................................ 66
4.4.1 Human Judgments..................................................................................... 67
4.4.2 Factoids..................................................................................................... 68
4.4.3 [[ROUGE]] with Basic Elements................................................................... 69
4.4.4 The PYRAMID Method ........................................................................... 70
4.5 Other Metrics .................................................................................................. 75
4.5.1 BLEU........................................................................................................ 75
4.5.2 WSummACCY......................................................................................... 76
4.5.3 METEOR.................................................................................................. 77
4.5.4 NUGGET PYRAMIDS and POURPRE................................................... 78
4.5.5 NUGGETEER........................................................................................... 79
4.6 Issues with Gold Standards Summaries............................................................ 80

Chapter 5 Formal Summarization Evaluations ..................................................... 82
5.1 Overview of SUMMAC, DUC, MSE, TAC, NTCIR, GALE .......................... 82
5.1.1 SUMMAC................................................................................................. 82
5.1.2 DUC, MSE and TAC ................................................................................ 84
5.1.3 NTCIR....................................................................................................... 85
5.1.4 GALE........................................................................................................ 85
5.2 Discussion of Summarization Evaluations ....................................................... 86
5.3 Multilingual Summarization Evaluation (MSE) Results .................................. 88
5.3.1 MSE Overview.......................................................................................... 89
5.3.2 MSE Content Evaluation: Summaries Can Be Misleading ..................... 90
5.4 Discussion – Good Summaries and Evaluations .............................................. 96

Chapter 6 Genre Identification................................................................................ 99
6.1 Introduction..................................................................................................... 100
6.2 Related Work .................................................................................................. 101
6.3 Genre Identification Data................................................................................ 102
6.4 Classifiers and Features .................................................................................. 104
6.5 Experimental Results ...................................................................................... 105

Chapter 7 Genre Oriented Goal-Focused Summaries......................................... 114
7.1 Genre Oriented Document Data Set Description............................................ 115
7.2 Summary System ............................................................................................ 118
7.3 Movie Reviews ............................................................................................... 127
7.4 Scientific Articles............................................................................................ 129
7.5 Discussion....................................................................................................... 130

Chapter 8 Email Summarization........................................................................... 131
8.1 The Email Genre ............................................................................................. 131
8.2 Genres (Speech Acts) of Email....................................................................... 133
8.3 Personal Email Corpus.................................................................................... 139
8.4 Features........................................................................................................... 139
8.4.1 Verbs....................................................................................................... 140
8.4.2 Email Specific Features .......................................................................... 140
8.4.3 Classification........................................................................................... 141
8.5 Enron Email Corpus........................................................................................ 144
8.6 Annotation of Email Corpus ........................................................................... 145
8.7 Email Summarization Results......................................................................... 146
8.7.1 Email Subject Line and Content ............................................................. 146
8.7.2 One Email Text Body Sentence as a Summary ...................................... 147
8.7.3 One Email Text Body Sentence as a Summary ...................................... 148
8.8 Discussion....................................................................................................... 152

Chapter 9 Multi-document Summarization ......................................................... 154
9.1 Multi Document Evaluation Corpus Description ........................................... 154
9.2 Data Sets Analysis .......................................................................................... 157
9.3 Evaluation of our MDS system....................................................................... 160
9.3.1 Cosine Similarity Evaluation .................................................................. 162
9.3.2 Subtopics Evaluation .............................................................................. 163
9.3.3 Human Judgment Summary Evaluation ................................................. 165

Chapter 10 Conclusions and Future Work............................................................. 167
10.1 Summary of Contributions.............................................................................. 167
10.2 Future Work.................................................................................................... 170
10.3 Conclusion ...................................................................................................... 171
Bibliography .............................................................................................................. 173

{#ifanon:|

Chapter 1 Introduction

• “We are drowning in information, while starving for wisdom. The world henceforth will be run by synthesizers, people able to put together the right information at the right time, think critically about it, and make important choices wisely.” --E.O. Wilson: Consilience: The Unity of Knowledge

Summaries are a method of conveying a condensed version of information to people. They can take various forms - text, graphics, video, speech or a combination of these forms. Effective summaries are tailored to the user’s information seeking goals; they convey the significant facts of the original text, condense descriptive details and are dependent on the genre.

In this thesis, we examine the creation of textual summaries that take into account the user’s information seeking goals as well as the genre – what we call “genre oriented goal focused summaries”.

…

Multi-document summarization -- capable of summarizing either complete documents sets, or clusters of documents in the context of previously summarized ones, referred to as update summaries -- is likely to be essential in such situations.

In the case of multi-document summarization (MDS), at one end of the summarization spectrum, this can be considered to be just simply single-document summarization, with all the documents concatenated together.

A group of articles may contain a temporal dimension, typical in a stream of news reports about an unfolding event. In this case later information may override earlier more tentative or incomplete accounts.

The compression ratio (i.e. the size of the summary with respect to the size of the document set) will typically be much smaller for collections of dozens or hundreds of topically related documents than for single document summaries.

• p.38 … information space exploration: Users need to be able to create different types of summaries to explore the information space (summary varieties). These summaries can be:
• Common Text Only Summary: Each summary passage must be linked with (similar to) at least one related passage in another document in the collection, providing a summary that contains many of the main repetitive points of the document collection.
• Unique Text Only Summary: Each summary passage contains a relevant point that only occurs in one document of the collection.
• Common and Unique Text Summary: The summary contains a combination of passages that are linked to related passages in other documents and some relevant text that only occurs in one document of the collection.
• Representative Single Document Summary: The summary is a single document summary from the centroid document in the collection.
• Novel Differing Points Summary: The summary contains only passages from outlier documents (from the centroid document) in the collection, thus differing from the central focus of the main documents.
• Document Set Overview Summary: The summary contains passages from the centroid document and the outlier documents.
• Latest Document Summary: The summary is a single document summary from the latest document.
• Common Text Time Weighted Summary: The summary contains only passages that have similar related passages in other documents and all summary passages are extracted from the most recent documents.
• Common Text Time Weighted plus Unique Text Summary: The summary contains a combination of (1) passages that have similar related passages in other documents in which all such passages are extracted from the most recent documents and (2) selected passages which are unique to one document.
• Comparative Summaries:
  • Differing Points Summary: The summary contains differing points of view or differing facts within the document collection. This includes differing opinions as well as source inconsistencies.
  • Similar Points Summary: The summary contains similar points of view or facts within the document collection.
  • Combination of Differing and Similar Points Summary
• Evolution Summary: The summary contains earlier information presented in the document collection and updates to that information.
• Update Summary: Given a previous summary, the summary contains updated information to the points contained in the summary.

}}

---

,

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2008 GenreOrientedSummarization	Jade Goldstein			Genre Oriented Summarization			http://www.lti.cs.cmu.edu/Research/Thesis/goldstein jade (2).pdf	10.1109/ICDM.2009.141