In tensorflow, there is an initialization function: tf.contrib.layers.variance_scaling_initializer. Introduction Tensorflow official website is:
variance_scaling_initializer(
factor=2.0,
mode='FAN_IN',
uniform=False,
seed=None,
dtype=tf.float32
)
1
2
3
4
5
6
7
Returns an initializer that generates tensors without scaling variance.
When initializing a deep network, it is in principle advantageous to keep the scale of the input variance constant, so it does not explode or diminish by reaching the final layer. This initializer use the following formula:
if mode='FAN_IN': # Count only number of input connections.
n = fan_in
elif mode='FAN_OUT': # Count only number of output connections.
n = fan_out
elif mode='FAN_AVG': # Average number of inputs and output connections.
n = (fan_in + fan_out)/2.0
truncated_normal (Shape, 0.0, STDDEV = sqrt (factor / n-))
. 1
2
. 3
. 4
. 5
. 6
. 7
. 8
these words may be understood, by using this initialization method, we can guarantee scale change input variables constant, so as to avoid changes in scale or dispersed in an explosion last layer of the network.
This method is Xavier initialization method, you can go about this method from these two the following papers:
·X. Glorot and Y. Bengio. Understanding the difficulty of training deepfeedforward neural networks. In International Conference on Artificial Intelligence and Statistics, pages 249–256, 2010.
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S.Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast featureembedding. arXiv:1408.5093, 2014.
或者可以通过这些文章去了解：
CNN value
heavy weights initialization method three
deep learning --Xavier initialization method
---------------------
Author: Although the road is farther down the road
Source: CSDN
Original: https: //blog.csdn.net/u010185894/article/details/71104387
Disclaimer: This article is a blogger original article, reproduced, please attach Bowen link!
