Clark-Ji-Smith Neural Narrative Text Generation System

A Clark-Ji-Smith (CJS) Neural Narrative Text Generation System is a Text Generation System that can solve a CJS Neural Narrative Text Generation Task.

• Context:
  • Resource(s):
    • Software repository is available at https://github.com/eaclark07/engen
  • System's Architecture:
    • It is based on Sutskever-Vinyals-Le Sequence-to-Sequence Model (Sutskever et al., 2014) and Bahdanau-Cho-Bengio Attention Mechanism Model (Bahdanau et al., 2015).
  • Trainings and other ML Tools:
    • It is implemented using a DyNet with GPU support (Neubig et al. 2017) with input and hidden layers fixed to 512 size:
      • It maximizes the log-probability defined in Eq.6, Clark et al., 2018;
      • It uses a SGD optimization with a learning rate $\lambda=0.1$;
      • Word embedding inputs are randomly initialized with DyNet default method and updated during training.
    • It implements a Clark-Manning Neural Coreference Resolution System for obtaining entities annotations (Clark & Manning, 2016).
• Example(s):
  • …
• Counter-Example(s):
  • GSGAN System,
  • LeakGAN System,
  • Lee-Krahmer-Wubben Data-To-Text Generation System.
  • MaliGAN System,
  • MaskGAN System,
  • RankGAN System,
  • SeqGAN System,
  • TextGAN System,
  • Texygen System.
• See: Mention Generation System, Text Generation System, Natural Language Processing System, Natural Language Generation System, Natural Language Understanding System, Natural Language Inference System, Computing Benchmark System.

---

---

References

2018

• (Clark et al., 2018) ⇒ Elizabeth Clark, Yangfeng Ji, and Noah A. Smith. (2018). “Neural Text Generation in Stories Using Entity Representations As Context.” In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Volume 1 (Long Papers). DOI:10.18653/v1/N18-1204.
  • QUOTE: We propose an entity-based generation model (ENGEN) that combines three different sources of contextual information for text generation:

    1. The content that has already been generated within the current sentence;

    2. The content that was generated in the previous sentence;

    3. The current state of the entities mentioned in the document so far

    Each of these types of information is encoded in vector form, following extensive past work on recurrent neural network (RNN) language models. The first source of context is the familiar hidden state vector of the RNN; more precisely, our starting point is a sequence-to-sequence model (Sutskever et al., 2014). Representations of the second and third forms of context are discussed in 2.1 and 2.2, respectively. The combination of all three context representations is described in 2.3.

2017

• (Neubig et al., 2017) ⇒ Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, and Pengcheng Yin. (2017). “DyNet: The Dynamic Neural Network Toolkit.” In: ePrint: arXiv:1701.03980.

2016a

• (Clark & Manning, 2016) ⇒ Kevin Clark, and Christopher D. Manning. (2016). “Deep Reinforcement Learning for Mention-Ranking Coreference Models.” In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP 2016). DOI:10.18653/v1/D16-1245.

2016b

• (Clark & Manning, 2016) ⇒ Kevin Clark, and Christopher D. Manning. (2016). “Improving Coreference Resolution by Learning Entity-Level Distributed Representations.” In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016) Volume 1: Long Papers. DOI:10.18653/v1/P16-1061.

2015a

• (Bahdanau et al., 2015) ⇒ Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. (2015). “Neural Machine Translation by Jointly Learning to Align and Translate.” In: Proceedings of the Third International Conference on Learning Representations, (ICLR-2015).

2015b

• (Zhu et al., 2015) ⇒ Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. (2015). "Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books". In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV 2015). DOI:10.1109/ICCV.2015.11.

2014

• (Sutskever et al., 2014) ⇒ Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. (2014). “Sequence to Sequence Learning with Neural Networks.” In: Proceedings of the Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems (NIPS 2014).

---