Rectified Linear Neuron

Jump to: navigation, search

A Rectified Linear Neuron is an artificial neuron that uses a Rectified Linear Activation Function.



  • (Mate Labs, 2017) ⇒ Mate Labs Aug 23, 2017. Secret Sauce behind the beauty of Deep Learning: Beginners guide to Activation Functions
    • QUOTE: It trains 6 times faster than tanh. Output value will be zero when input value is less than zero. If input is greater than or equal to zero, output is equal to the input. When the input value is positive, derivative is 1, hence there will be no squeezing effect which occurs in the case of backpropagating errors from the sigmoid function.

      [math]f(x) = \begin{cases} 0, & \mbox{for } x \lt 0 \\ x, & \mbox{for } x \geq 0 \end{cases}[/math]

      Range:[math][0, x][/math]

      Examples: [math]f(-5) = 0, f(0) = 0 \;\& \;f(5) = 5[/math]