Jordan Network

A Jordan Network is a Simple Recurrent Network in which activations occur at the output layer, not at a hidden layer.

• AKA: Jordan Recurrent Neural Network.
• Context:
  • It was first developed by Jordan (1997).
  • …
• Example(s):
  • the one described in Jordan (1997).
  • …
• Counter-Example(s)
  • a Bidirectional Associative Memory (BAM) Network,
  • a DAG Recurrent Neural Network,
  • an Echo State Network,
  • an Elman Network,
  • a Gated Recurrent Unit (GRU) Network,
  • a Hopfield Recurrent Neural Network,
  • a Long Short-Term Memory (LSTM) Network,
  • a Recurrent Multilayer Perceptron Network.
• See: Jeff Elman, Activation Function, Multilayer Perceptron, Michael I. Jordan.

---

---

References

2018

• (Wikipedia, 2018) ⇒ https://en.wikipedia.org/wiki/Recurrent_neural_network#Elman_networks_and_Jordan_networks Retrieved:2018-3-4.
  • Jordan networks are similar to Elman networks. The context units are fed from the output layer instead of the hidden layer. The context units in a Jordan network are also referred to as the state layer. They have a recurrent connection to themselves.

    Elman and Jordan networks are also known as "simple recurrent networks" (SRN).

    Elman network :

    [math]\displaystyle{ \begin{align} h_t &= \sigma_h(W_{h} x_t + U_{h} h_{t-1} + b_h) \\ y_t &= \sigma_y(W_{y} h_t + b_y) \end{align} }[/math]

    Jordan network :

    [math]\displaystyle{ \begin{align} h_t &= \sigma_h(W_{h} x_t + U_{h} y_{t-1} + b_h) \\ y_t &= \sigma_y(W_{y} h_t + b_y) \end{align} }[/math]

    Variables and functions

    • [math]\displaystyle{ x_t }[/math] : input vector
    • [math]\displaystyle{ h_t }[/math] : hidden layer vector
    • [math]\displaystyle{ y_t }[/math] : output vector
    • [math]\displaystyle{ W }[/math] , [math]\displaystyle{ U }[/math] and [math]\displaystyle{ b }[/math] : parameter matrices and vector
    • [math]\displaystyle{ \sigma_h }[/math] and [math]\displaystyle{ \sigma_y }[/math] : Activation functions

1997

• (Jordan, 1997) ⇒ Jordan, M. I. (1997). Serial order: A parallel distributed processing approach (PDF). In Advances in psychology (Vol. 121, pp. 471-495). North-Holland. doi: 10.1016/S0166-4115(97)80111-2
  • QUOTE: A theory of learned sequential behavior is presented, with a focus on coarticulatory phenomena in speech. The theory is implemented as a recurrent parallel distributed processing network that is trained via a generalized error-correcting algorithm. The basic idea underlying the theory is that both serial order and coarticulatory overlap can be represented in terms of relative levels of activation in a network if a clear distinction is made between the state of the network and the output of the network.

---