Concatenated Rectified Linear Activation Function

A Concatenated Rectified Linear Activation Function is a Rectified-based Activation Function that is defined as [math]\displaystyle{ f(x)=(max(0,x),max(0,−x)) }[/math].

• AKA: CReLU.
• Context:
  • It can (typically) be used in the activation of Concatenated Rectified Linear Neurons.
• Example(s):
  • tf.nn.crelu [1]
  • chainer.functions.crelu [2]
• Counter-Example(s):
  • a Clipped Rectifier Unit Activation Function,
  • an Exponential Linear Activation Function,
  • a Leaky Rectified Linear Activation Function,
  • a Noisy Rectified Linear Activation Function,
  • a Parametric Rectified Linear Activation Function,
  • a Randomized Leaky Rectified Linear Activation Function,
  • a Scaled Exponential Linear Activation Function,
  • a Softplus Activation Function,
  • a S-shaped Rectified Linear Activation Function.
• See: Artificial Neural Network, Artificial Neuron, Neural Network Topology, Neural Network Layer, Neural Network Learning Rate.

---

---

References

2018a

• (TensorFlow.org, 2018) ⇒ https://www.tensorflow.org/versions/r0.12/api_docs/python/nn/activation_functions_#crelu Retrieved: 2018-2-18
  • QUOTE: tf.nn.crelu(features, name=None) Computes Concatenated ReLU.

    Concatenates a ReLU which selects only the positive part of the activation with a ReLU which selects only the negative part of the activation. Note that as a result this non-linearity doubles the depth of the activations.

    Source: https://arxiv.org/abs/1603.05201

    Args:

    • features: A Tensor. Must be one of the following types: float32, float64, int32, int64, uint8, int16, int8, uint16, half.
    • name : A name for the operation (optional).

Returns:

A Tensor. Has the same type as features.

2018b

• (Chainer, 2018) ⇒ http://docs.chainer.org/en/stable/reference/generated/chainer.functions.crelu.html Retrieved:2018-2-18
  • QUOTE: chainer.functions.crelu(x, axis=1)source

     Concatenated Rectified Linear Unit function.

    This function is expressed as follows

    [math]\displaystyle{ f(x)=(max(0,x),max(0,−x)) }[/math].

    Here, two output values are concatenated along an axis.

    See: https://arxiv.org/abs/1603.05201

    Parameters:

    • x (Variable or numpy.ndarray or cupy.ndarray) – Input variable. A [math]\displaystyle{ (s_1,s_2,\cdots,s_N) }[/math]-shaped float array.
    • alpha (float) – Parameter [math]\displaystyle{ \alpha }[/math]. Default is 1.0.

Returns: Output variable. A [math]\displaystyle{ (s_1,s_2,\cdots,s_N) }[/math]-shaped float array.

Return type: Variable

Example:

>>> x = np.array([ [-1, 0], [2, -3] ], 'f')
>>> x
array([ [-1.,  0.],
       [ 2., -3.] ], dtype=float32)
>>> y = F.crelu(x, axis=1)
>>> y.data
array([ [0., 0., 1., 0.],
       [2., 0., 0., 3.] ], dtype=float32)

2016

• (Shang et al., 2016) ⇒ Shang, W., Sohn, K., Almeida, D., & Lee, H. (2016, June). Understanding and improving convolutional neural networks via concatenated rectified linear units. In: Proceedings of The International Conference on Machine Learning (pp. 2217-2225).arXiv:1603.05201
  • ABSTRACT: Recently, convolutional neural networks (CNNs) have been used as a powerful tool to solve many problems of machine learning and computer vision. In this paper, we aim to provide insight on the property of convolutional neural networks, as well as a generic method to improve the performance of many CNN architectures. Specifically, we first examine existing CNN models and observe an intriguing property that the filters in the lower layers form pairs (i.e., filters with opposite phase). Inspired by our observation, we propose a novel, simple yet effective activation scheme called concatenated ReLU (CRelu) and theoretically analyze its reconstruction property in CNNs. We integrate CRelu into several state-of-the-art CNN architectures and demonstrate improvement in their recognition performance on CIFAR-10/100 and ImageNet datasets with fewer trainable parameters. Our results suggest that better understanding of the properties of CNNs can lead to significant performance improvement with a simple modification.

---