Simple Rectified Linear Neuron
A Simple Rectified Linear Neuron is a Rectified Linear Neuron in its simplest form, i.e. with [math]\displaystyle{ \alpha = 0 }[/math].
- Context:
- It can be mathematically described as
[math]\displaystyle{ y_j=\begin{cases} 0, & \mbox{for } z_j \lt 0 \\z_j, & \mbox{for } z_j \geq 0 \end{cases} \text{with} \quad z_j=\sum_{i=0}^nw_{ji}x_i+b \quad \text{for}\quad j=0,\cdots, p }[/math]
where [math]\displaystyle{ x_i }[/math] are the Neural Network Input vector, [math]\displaystyle{ y_j }[/math] are the Neural Network Output vector, [math]\displaystyle{ w_{ji} }[/math] is the Neural Network Weights.
- It can be mathematically described as
- Example(s)
- …
- Counter-Example(s)
- See: Artificial Neural Network, Perceptron, Linear Neuron.
References
2017
- (Mate Labs, 2017) ⇒ Mate Labs Aug 23, 2017. Secret Sauce behind the beauty of Deep Learning: Beginners guide to Activation Functions
- QUOTE: It trains 6 times faster than tanh. Output value will be zero when input value is less than zero. If input is greater than or equal to zero, output is equal to the input. When the input value is positive, derivative is 1, hence there will be no squeezing effect which occurs in the case of backpropagating errors from the sigmoid function.
[math]\displaystyle{ f(x) = \begin{cases} 0, & \mbox{for } x \lt 0 \\ x, & \mbox{for } x \geq 0 \end{cases} }[/math]
Range:[math]\displaystyle{ [0, x] }[/math]
Examples: [math]\displaystyle{ f(-5) = 0, f(0) = 0 \;\& \;f(5) = 5 }[/math]
- QUOTE: It trains 6 times faster than tanh. Output value will be zero when input value is less than zero. If input is greater than or equal to zero, output is equal to the input. When the input value is positive, derivative is 1, hence there will be no squeezing effect which occurs in the case of backpropagating errors from the sigmoid function.