1. introduction
In this blog, I want firstly to learn some basic functions in tensor flow frame and then to design a convolution network to address one classical problem - mnist recognition. Meanwhile I also would to note some basic usage for my unfamiliar function. Lastly, I would like to learn Tensorboard to visualize the learning phase and explore how to analyze these information.
2. designing network
we firstly import two important modules involved in our project:
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
Then, we would better define initial functions for weight and bias, which are frequently called in a network:
def weight_variable(shape):
initial = tf.truncated_normal(shape, stddev=0.1)
return tf.Variable(initial)
def bias_variable(shape):
initial = tf.constant(0.1, shape=shape)
return tf.Variable(initial)
here, 'shape' is a tensor that includes four dimensions.
if we finish these works, we can to design our network, that is our graph. In tensor flow frame, network designing and training are divided into two different phases - graph and session.
myGraph=tf.Graph()
with myGraph.as_default():
with tf.name_scope('inputsAndLabels'):
x_raw = tf.placeholder(tf.float32, shape=[None, 784])
y = tf.placeholder(tf.float32, shape=[None, 10])
with tf.name_scope('hidden1'):
x = tf.reshape(x_raw, shape=[-1, 28, 28, 1]) # -1 donates recognition automatically
W_conv1 = weight_variable([5, 5, 1, 32])
b_conv1 = bias_variable([32])
l_conv1 = tf.nn.relu(tf.nn.conv2d(x, W_conv1, strides=[1, 1, 1, 1], padding='SAME') + b_conv1)
l_pool1 = tf.nn.max_pool(l_conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
tf.summary.image('x_input', x, max_outputs=10)
tf.summary.histogram('W_con1', W_conv1)
tf.summary.histogram('b_con1', b_conv1)
with tf.name_scope('hidden2'):
W_conv2 = weight_variable([5, 5, 32, 64])
b_conv2 = bias_variable([64])
l_conv2 = tf.nn.relu(tf.nn.conv2d(l_pool1, W_conv2, strides=[1, 1, 1, 1], padding='SAME')+b_conv2)
l_pool2 = tf.nn.max_pool(l_conv2, ksize=[1, 2, 2, 1],strides=[1, 2, 2, 1], padding='SAME')
tf.summary.histogram('W_con2', W_conv2)
tf.summary.histogram('b_con2', b_conv2)
with tf.name_scope('fc1'):
W_fc1 = weight_variable([64*7*7, 1024])
b_fc1 = bias_variable([1024])
l_pool2_flat = tf.reshape(l_pool2, [-1, 64*7*7])
l_fc1 = tf.nn.relu(tf.matmul(l_pool2_flat, W_fc1) + b_fc1)
keep_prob = tf.placeholder(tf.float32)
l_fc1_drop = tf.nn.dropout(l_fc1, keep_prob)
tf.summary.histogram('W_fc1', W_fc1)
tf.summary.histogram('b_fc1', b_fc1)
with tf.name_scope('fc2'):
W_fc2 = weight_variable([1024, 10])
b_fc2 = bias_variable([10])
y_conv = tf.matmul(l_fc1_drop, W_fc2) + b_fc2
tf.summary.histogram('W_fc1', W_fc1)
tf.summary.histogram('b_fc1', b_fc1)
with tf.name_scope('train'):
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_conv, labels=y))
train_step = tf.train.AdamOptimizer(learning_rate=1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
tf.summary.scalar('loss', cross_entropy)
tf.summary.scalar('accuracy',
- myGraph=tf.Graph(): design a new graph and return a object.
- myGraph.as_default(): overlay the default graph.
- with tf.name_scope('name'): used to administrate operations in graph, and return a context manager which named by scope_name. The graph will maintain the stack of name_space, so we can define different operations or children namespace in each name_space. So all in all, using it is to administrate the graph orderly, and in case of conflicts among operations.
- x_raw = tf.placeholder(tf.float32, shape=[None, 784]) &
y = tf.placeholder(tf.float32, shape=[None, 10] : when we design network graph, we do not know the batch of input images and labels. so we adopt place holder. The first parameter of shape - none- is the size of batch, which can be assigned automatically once the session runs. 784 = 28*28 is the size of input image. 10 is the image is classed into ten classes. - x = tf.reshape(x_raw, shape=[-1, 28, 28, 1]): in order to make sure dimension match, when input batch is feed to first layer we should reshape the shape of input tensor. Then common shape is [-1, in_height, in_width, in_channel]. we should notice "-1", which means the program can recognize the size of batch once the session runs.
- tf.nn.conv2d(input, filter, strides, padding, use_cudnn_on_gpu=None, data_format=None, name=None)
input = [batch, in_height, in_width, in_channel]
filter = [filter_height, filter_width, in_channel, out_channel]
strides = [batch_s, imheight_s, imwidth_s, imchannel_s] #stride
padding = 'VALID' or 'SAME' - tf.nn.relu(features, name = None)
- tf.nn.max_pool(input, ksize, strides, padding, name=None)
input = [batch, in_height, in_width, in_channel]
ksize =[batch_d, imheight_d, imwidth_d, imchannel_d ] #down sample
strides = [batch_s, imheight_s, imwidth_s, imchannel_s] #stride
padding = 'VALID' or 'SAME' - tf.nn.dropout(x, keep_prob, noise_shape=None, seed=None,name=None) : always occurs in full connection layer.
- cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits( ogits=y_conv, labels=y)): measure function
and then;
train_step = tf.train.AdamOptimizer(learning_rate=1e-4).minimize()
3. run session and train
Run a session:
with tf.Session(graph=myGraph) as sess:
sess.run(tf.global_variables_initializer())
saver = tf.train.Saver()
merged = tf.summary.merge_all()
summary_writer = tf.summary.FileWriter('./mnistEven/', graph=sess.graph)
for i in range(3001):
batch = mnist.train.next_batch(100)
sess.run(train_step, feed_dict={x_raw: batch[0], y: batch[1], keep_prob: 0.8})
if i % 100 == 0:
train_accuracy = accuracy.eval(feed_dict={x_raw: batch[0], y: batch[1], keep_prob: 0.8})
print('step %d training accuracy:%g' % (i, train_accuracy))
summary = sess.run(merged, feed_dict={x_raw: batch[0], y: batch[1], keep_prob: 0.8})
summary_writer.add_summary(summary, i)
4. test
Lastly, test the network
test_accuracy = accuracy.eval(feed_dict={x_raw: mnist.test.images, y: mnist.test.labels, keep_prob: 1.0})
print('test accuracy:%g' % test_accuracy)
saver.save(sess, save_path='mnist_model/mnistmodel', global_step=1)
5. visualizing with tensorflow
python /home/shenziheng/anaconda3/lib/python3.6/site-packages/tensorflow/tensorboard/tensorboard.py --logdir=/home/shenziheng/PycharmProjects/advance_mnist/mnistEven