2017 FractalNetUltraDeepNeuralNetwor

From GM-RKB
Jump to navigation Jump to search

Subject Headings: FractalNet; Self-Similarity Neural Network, Fractal.

Notes

Cited By

Quotes

Abstract

We introduce a design strategy for neural network macro-architecture based on self-similarity. Repeated application of a simple expansion rule generates deep networks whose structural layouts are precisely truncated fractals. These networks contain interacting subpaths of different lengths, but do not include any pass-through or residual connections; every internal signal is transformed by a filter and nonlinearity before being seen by subsequent layers. In experiments, fractal networks match the excellent performance of standard residual networks on both CIFAR and ImageNet classification tasks, thereby demonstrating that residual representations may not be fundamental to the success of extremely deep convolutional neural networks. Rather, the key may be the ability to transition, during training, from effectively shallow to deep. We note similarities with student-teacher behavior and develop drop-path, a natural extension of dropout, to regularize co-adaptation of subpaths in fractal architectures. Such regularization allows extraction of high-performance fixed-depth subnetworks. Additionally, fractal networks exhibit an anytime property: shallow subnetworks provide a quick answer, while deeper subnetworks, with higher latency, provide a more accurate answer.

References

BibTeX

@inproceedings{DBLP:conf/iclr/LarssonMS17,
  author    = {Gustav Larsson and
               Michael Maire and
               Gregory Shakhnarovich},
  title     = {FractalNet: Ultra-Deep Neural Networks without Residuals},
  booktitle = {Proceedings of the 5th International Conference on Learning Representations
               (ICLR 2017) Conference Track},
  publisher = {OpenReview.net},
  year      = {2017},
  url       = {https://openreview.net/forum?id=S1VaB4cex},
}


 AuthorvolumeDate ValuetitletypejournaltitleUrldoinoteyear
2017 FractalNetUltraDeepNeuralNetworMichael Maire
Gustav Larsson
Gregory Shakhnarovich
FractalNet: Ultra-Deep Neural Networks Without Residuals2017