Clark-Ji-Smith Neural Narrative Text Generation Task

A Clark-Ji-Smith (CJS) Neural Narrative Text Generation Task is a Text Generation Benchmark Task that evaluates the quality of entity mentions automatically generated by a Neural Narrative Text Generation System.

• AKA: Neural Text Generation via Entity Mentions.
• Context:
  • Task Input(s): Story data (text document).
  • Task Output(s): Entity-labeled story data.
  • Task Requirement(s):
    • Benchmark Datasets:
      • Took Book Corpus (Zhu et al., 2015) - training Datasets include 312 adventure books which are divided in segments of up to 50 sentences (33,279 segments in total); test and development datasets include 39 additional adventure books, consisting of 4,037 and 4,577 segments, respectively.
      • Entities annotation datasets obtained using Stanford CoreNLP system (Clark and Manning, 2016a, b).
    • Benchmark Performance Metrics:
      • Mean Average Precision (MAP) Score of correct candidates.
      • CJS log-probability mean accuracy and standard deviation of candidate sentences pairs.
      • Amazon mechanical turkers evaluation ratings of generated sentences.
    • Baseline Models:
      • Entity-based Generation Model (ENGEN);
      • ENGEN-S2SA7;
      • ENGEN-ENTITYNLM;
      • Reverse Order ENGEN.
  • It can be solved by CJS Neural Narrative Text Generation Learning System that implements CJS Neural Narrative Text Generation Learning Algorithms.
• Example(s):
  • Clark et al. (2018)'s MAP Score on the mention generation task:

model	cluster and mention	cluster only	mention only
1. Reverse order	0.12	0.38	0.15
2. S2SA	—	—	0.44
3. ENTITYNLM	0.52	0.46	0.54
4. ENGEN	0.53	0.46	0.55

• CJS Log-Probability Mean Accuracy and Standard Deviation calculated across the five rounds of pairwise sentence selection(Clark et al., 2018):

model	mean accuracy	s.d.
1.S2SA	0.546	0.01
2. ENTITYNLM	0.534	0.006
3. ENGEN	$\star$0.566	0.008

•  Clark et al.(2018)'s example of ENGEN(second clumn) and ENGEN-S2SA (third column) generated sentence for three different contexts (first column) and respective Amazon Mechanical Turkers Voting Scores (last column) :

Context	ENGEN	S2SA	#
he says that it was supposed to look random , but he feels it was planned . i was the target . The ’s not sure , but he feels that you might have something to do with this , ” cassey said sadly . “ he ca n’t do that ! ” manny yelled . “ he ca n’t accuse me with no justification .	"it's not me".	he has nothing to do with my life	10
he was wearing brown slacks and a tan button-down shirt , with wool slippers . he looked about sixty ,a little paunchy , with balding brown hair and a bushy mustache . ice blue eyes observed alejo keenly , then drifted over to wara .“ welcome to my home. ” the man ’s voice was deep and calm .	“i’m proud of you” he said. ’	“what’s goin gon?'	4
bearl looked on the scene , and gasped. this was the white rock of legend , the rock that had lured him to this land . then he stopped . “ look , geron . the white rock we saw from the sea. ” the struggle was taking place on the white rock . the monster had his back to bearl .	“oh my god!”	he could not believe his eyes	1

• Counter-Example(s):
  • GSGAN Benchmark Task,
  • LeakGAN Benchmark Task.
  • MaliGAN Benchmark Task,
  • MaskGAN Benchmark Task,
  • RankGAN Benchmark Task,
  • SeqGAN Benchmark Task,
  • TextGAN Benchmark Task,
  • Texygen Benchmark Task.
• See: Mention Generation Task, Text Generation Task, Natural Language Processing Task, Natural Language Generation Task, Natural Language Understanding Task, Natural Language Inference Task, Computing Benchmark Task.

---

---

References

2018

• (Clark et al., 2018) ⇒ Elizabeth Clark, Yangfeng Ji, and Noah A. Smith. (2018). “Neural Text Generation in Stories Using Entity Representations As Context.” In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Volume 1 (Long Papers). DOI:10.18653/v1/N18-1204.
  • QUOTE: We propose an entity-based generation model (ENGEN) that combines three different sources of contextual information for text generation:

    1. The content that has already been generated within the current sentence;

    2. The content that was generated in the previous sentence;

    3. The current state of the entities mentioned in the document so far

    Each of these types of information is encoded in vector form, following extensive past work on recurrent neural network (RNN) language models. The first source of context is the familiar hidden state vector of the RNN; more precisely, our starting point is a sequence-to-sequence model (Sutskever et al., 2014). Representations of the second and third forms of context are discussed in 2.1 and 2.2, respectively. The combination of all three context representations is described in 2.3.

2017

• (Neubig et al., 2017) ⇒ Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, and Pengcheng Yin. (2017). “DyNet: The Dynamic Neural Network Toolkit.” In: ePrint: arXiv:1701.03980.

2016a

• (Clark & Manning, 2016) ⇒ Kevin Clark, and Christopher D. Manning. (2016). “Deep Reinforcement Learning for Mention-Ranking Coreference Models.” In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP 2016). DOI:10.18653/v1/D16-1245.

2016b

• (Clark & Manning, 2016) ⇒ Kevin Clark, and Christopher D. Manning. (2016). “Improving Coreference Resolution by Learning Entity-Level Distributed Representations.” In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016) Volume 1: Long Papers. DOI:10.18653/v1/P16-1061.

2015a

• (Bahdanau et al., 2015) ⇒ Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. (2015). “Neural Machine Translation by Jointly Learning to Align and Translate.” In: Proceedings of the Third International Conference on Learning Representations, (ICLR-2015).

2015b

• (Zhu et al., 2015) ⇒ Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. (2015). “Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books". In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV 2015). DOI:10.1109/ICCV.2015.11.

2014

• (Sutskever et al., 2014) ⇒ Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. (2014). “Sequence to Sequence Learning with Neural Networks.” In: Proceedings of the Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems (NIPS 2014).

---