An activation function based on the logarithm function.
This type of activation function can be useful to prevent saturation. A
hidden node of a neural network is said to be saturated on a given set of
inputs when its output is approximately 1 or -1 "most of the time". If this
phenomena occurs during training then the learning of the network can be
slowed significantly since the error surface is very at in this instance.
public final double derivativeFunction(double b,
Calculate the derivative. For performance reasons two numbers are provided.
First, the value "b" is simply the number that we would like to calculate
the derivative of.
Second, the value "a", which is the value returned by the activation function,
when presented with "b".
We use two values because some of the most common activation functions make
use of the result of the activation function. It is bad for performance to
calculate this value twice. Yet, not all derivatives are calculated this way.
By providing both the value before the activation function is applied ("b"),
and after the activation function is applied("a"), the class can be constructed
to use whichever value will be the most efficient.