Rectified Linear Neuron

From GM-RKB
Jump to navigation Jump to search

A Rectified Linear Neuron is an artificial neuron that uses a Rectified Linear Activation Function.



References

2017

  • (Mate Labs, 2017) ⇒ Mate Labs Aug 23, 2017. Secret Sauce behind the beauty of Deep Learning: Beginners guide to Activation Functions
    • QUOTE: It trains 6 times faster than tanh. Output value will be zero when input value is less than zero. If input is greater than or equal to zero, output is equal to the input. When the input value is positive, derivative is 1, hence there will be no squeezing effect which occurs in the case of backpropagating errors from the sigmoid function.

      [math]\displaystyle{ f(x) = \begin{cases} 0, & \mbox{for } x \lt 0 \\ x, & \mbox{for } x \geq 0 \end{cases} }[/math]

      Range:[math]\displaystyle{ [0, x] }[/math]

      Examples: [math]\displaystyle{ f(-5) = 0, f(0) = 0 \;\& \;f(5) = 5 }[/math]