File:2016 ScalingMemoryAugmentedNeuralNet Fig5.png

Summary

Figure 5: A schematic of the memory efficient backpropagation through time. Each circle represents an instance of the SAM core at a given time step. The grey box marks the dense memory. Each core holds a reference to the single instance of the memory, and this is represented by the solid connecting line above each core. We see during the forward pass, the memory’s contents are modified sparsely, represented by the solid horizontal lines. Instead of caching the changing memory state, we store only the sparse modifications — represented by the dashed white boxes. During the backward pass, we “revert” the cached modifications to restore the memory to its prior state, which is crucial for correct gradient calculations. In: Rae et al. (2016)