Optimizer :
tf.train.GradientDescentOptimizer
tf.train.AdadeltaOptimizer
tf.train.AdagradOptimizer
tf.train.AdagradDAOptimizer
tf.train.MomentumOptimizer
tf.train.AdamOptimizer
tf.train.FtrlOptimizer
tf.train.ProximalGradientDescentOptimizer
tf.train.ProximalAdagradOptimizer
tf.train.RMSPropOptimizer
各种优化器对比:
标准梯度下降法:
标准梯度下降先计算所有样本汇总误差,然后根据总误差来更新权值随机梯度下降法:
随机梯度下降随机抽取一个样本来计算误差,然后更新权值批量梯度下降法:
批量梯度下降算是一种折中的方案,从总样本中选取一个批次(比如一共有10000个样本,随机选取100个样本作为一个batch),然后计算这个batch的总误差,根据总误差来更新权值。代码:
import tensorflow as tf from tensorflow.examples.tutorials.mnist import input_data #载入数据集 #当前路径 mnist = input_data.read_data_sets("MNISt_data", one_hot=True)
运行结果:
Extracting MNISt_data/train-images-idx3-ubyte.gz Extracting MNISt_data/train-labels-idx1-ubyte.gz Extracting MNISt_data/t10k-images-idx3-ubyte.gz Extracting MNISt_data/t10k-labels-idx1-ubyte.gz
代码:
#每个批次的大小 #以矩阵的形式放进去 batch_size = 100 #计算一共有多少个批次 n_batch = mnist.train.num_examples // batch_size #定义两个placeholder #28 x 28 = 784 x = tf.placeholder(tf.float32, [None, 784]) y = tf.placeholder(tf.float32, [None, 10]) #创建一个简单的神经网络 #输入层784,没有隐藏层,输出层10个神经元 W = tf.Variable(tf.zeros([784, 10])) b = tf.Variable(tf.zeros([1, 10])) prediction = tf.nn.softmax(tf.matmul(x, W) + b) #交叉熵 loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=prediction)) #使用梯度下降法 #train_step = tf.train.GradientDescentOptimizer(0.2).minimize(loss) train_step = tf.train.AdamOptimizer(1e-2).minimize(loss) #初始化变量 init = tf.global_variables_initializer() #结果存放在一个布尔型列表中 #tf.argmax(y, 1)与tf.argmax(prediction, 1)相同返回True,不同则返回False #argmax返回一维张量中最大的值所在的位置 correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(prediction, 1)) #求准确率 #tf.cast(correct_prediction, tf.float32) 将布尔型转换为浮点型 accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) with tf.Session() as sess: sess.run(init) #总共21个周期 for epoch in range(21): #总共n_batch个批次 for batch in range(n_batch): #获得一个批次 batch_xs, batch_ys = mnist.train.next_batch(batch_size) sess.run(train_step, feed_dict={x:batch_xs, y:batch_ys}) #训练完一个周期后准确率 acc = sess.run(accuracy, feed_dict={x:mnist.test.images, y:mnist.test.labels}) print("Iter" + str(epoch) + ", Testing Accuracy" + str(acc))
运行结果:
Iter0, Testing Accuracy0.9199 Iter1, Testing Accuracy0.9229 Iter2, Testing Accuracy0.9284 Iter3, Testing Accuracy0.9293 Iter4, Testing Accuracy0.9263 Iter5, Testing Accuracy0.9299 Iter6, Testing Accuracy0.9311 Iter7, Testing Accuracy0.9306 Iter8, Testing Accuracy0.9303 Iter9, Testing Accuracy0.9303 Iter10, Testing Accuracy0.9303 Iter11, Testing Accuracy0.9291 Iter12, Testing Accuracy0.9325 Iter13, Testing Accuracy0.9316 Iter14, Testing Accuracy0.9338 Iter15, Testing Accuracy0.9282 Iter16, Testing Accuracy0.9306 Iter17, Testing Accuracy0.9331 Iter18, Testing Accuracy0.9315 Iter19, Testing Accuracy0.9276 Iter20, Testing Accuracy0.9323