Rectified Linear Neuron

A Rectified Linear Neuron is an artificial neuron that uses a Rectified Linear Activation Function.

• AKA: Rectified Linear Activation Unit, Linear Threshold Neuron, Rectified Linear Unit, ReLU.
• Context:
  • It can be mathematically described as

    [math]\displaystyle{ y_j=\begin{cases} \alpha z_j, & \mbox{for } z_j \lt 0 \\z_j, & \mbox{for } z_j \geq 0 \end{cases} \text{with} \quad z_j=\sum_{i=0}^nw_{ji}x_i+b \quad \text{for}\quad j=0,\cdots, p }[/math]

    where [math]\displaystyle{ x_i }[/math] are the Neural Network Input vector, [math]\displaystyle{ y_j }[/math] are the Neural Network Output vector, [math]\displaystyle{ w_{ji} }[/math] is the Neural Network Weights and [math]\displaystyle{ b }[/math] is the Bias Neuron, [math]\displaystyle{ \alpha }[/math] is usually a small positive constant (i.e. [math]\displaystyle{ 0\leq\alpha\leq 1 }[/math]) known as leakage coeffcient.

  • It ranges from being a Simple Rectified Linear Neuron to being a Parametric Rectified Linear Neuron.
• Example(s):
  • a Leaky Rectified Linear Neuron,
  • a Randomized Leaky Rectified Linear Neuron,
• Counter-Example(s):
  • a S-shaped Rectified Linear Neuron,
  • an Adaptive Linear Neuron,
  • a Scaled Exponential Linear Neuron,
  • a Exponential Linear Neuron,
  • a Bent Identity Neuron.
  • a Hyperbolic Tangent Neuron,
  • a Sigmoid Neuron,
  • a Heaviside Step Neuron,
  • a Stochastic Binary Neuron,
  • a SoftPlus Neuron,
  • a Softmax Neuron.
• See: Artificial Neural Network, Perceptron, Linear Neuron.

---

---

References

2017

• (Mate Labs, 2017) ⇒ Mate Labs Aug 23, 2017. Secret Sauce behind the beauty of Deep Learning: Beginners guide to Activation Functions
  • QUOTE: It trains 6 times faster than tanh. Output value will be zero when input value is less than zero. If input is greater than or equal to zero, output is equal to the input. When the input value is positive, derivative is 1, hence there will be no squeezing effect which occurs in the case of backpropagating errors from the sigmoid function.

    [math]\displaystyle{ f(x) = \begin{cases} 0, & \mbox{for } x \lt 0 \\ x, & \mbox{for } x \geq 0 \end{cases} }[/math]

    Range:[math]\displaystyle{ [0, x] }[/math]

    Examples: [math]\displaystyle{ f(-5) = 0, f(0) = 0 \;\& \;f(5) = 5 }[/math]

    

---