Why do you want to regularize input in deep learning?

The author of this article is original, please indicate the source.

Today we explain why you want to regularize input (also called standardized input)?

The regularization input is actually the local normalization described in the paper. It was originally a data standardization method used by Krizhevsky and Hinton in the paper on ImageNet.

In practical applications, we may encounter different dimensional data or the distribution of features in space. Just like the picture below

This adds to the difficulty of training, we can look at what kind of gradient descent graph we will get if it is such data.

Obviously, this is a long and narrow three-dimensional figure. In the process of backpropagation, if the gradient begins to decrease at both ends, the whole process becomes very long. So in order to solve this situation, we used regularized input to solve. The following is the calculation formula for regularized input:

According to this formula, our gradient decline becomes like this.

In this case, no matter which point we start the gradient, the effect is the same.

Then let's take a look at how it is implemented in Tensorflow. The API in Tensorflow is tf.nn.lrn, and the alias is also called tf.nn.local_response_normalization. These two are one thing. Let’s see how the function is defined:

Local_response_normalization( input, depth_radius= 5 , bias= 1 , alpha= 1 , beta= 0.5 , name= None )

There are so many parameters inside, what do they mean? First, input is the tensor we want to input. depth_radius is n/2 in the above formula. In fact, why is this variable name called depth_radius? Is radius not a radius? What is the relationship with the radius? Let me explain why. Next, bias is the offset, alpha is the alpha in the formula, and beat is the β in the formula .

In fact, LRN can also be seen as "each pixel" divided by "the sum of the squares of other corresponding pixels within the radius" after zeroing, which is the value of the given variable depth_radius.

Then let's look at the effect with the code:

Import numpy as np import tensorflow as tfa = 2 * np.ones([ 2 , 2 , 2 , 3 ]) print (a)b = tf.nn.local_response_normalization(a , 1 , 0 , 1 , 1 ) with tf. Session() as sess: print (sess.run(b))

The output a is:

The output result b is: