First, the single programming framework
Stand-alone program is up and running refers to a process completed in a machine because there is no network overhead, very few parameters for less computation model.
Step to create a single data flow diagrams, create and run a single session.
saver = tf.train.Saver() Sex = tf.InteractiveSession () tf.global_variables_initializer().run() for i in range(1000): batch_xs,batch_ys = mnist.train.next_batch(100) sess.run(train_step,feed_dict={x:batch_xs,y_=batch_ys}) if i%100 = 0: saver.save(sess,'mnist.ckpt') correct_prediction = tf.equal(tf.argmax(y,1),tf.argmax(y_,1)) accuracy = tf.reduce_mean(tf.cast(correct_prediction,tf.float32)) sess.run(accuracy,feed_dict={x:mnist.test.images,y_:mnist.test.labels})
If you want to specify the device on the machine, such as cpu, gpu
can use
with tf.device('/cpu:0'):
……
Second, distributed programming framework
PS-worker is a classic distributed architecture, it has a wide range of applications, tensorFlow provides support for PS-worker in large-scale distributed machine learning and deep learning.
step
(1) .pull, each worker according to the topology of the data flow diagram, to pull in the model parameters from the PS
(2) .feed, each worker to fill different batches batch data according to certain rules
(3) .compute, each worker the same model parameters and batch data for calculating a gradient of a different, a different gradient value obtained
(4) .push, each worker gradient value calculated in the previous step push PS
(5) .update, PS aggregate data, update model parameters calculated average gradient
To create a distributed program running cluster, creating a distributed data flow diagrams, create a distributed session
Cluster creation, tf.train.Server (host, job_name, task_index)
The operation of the device is placed on the target
with tf.device('/job:PS/task:0'): weights_1 = tf.Variable() with tf.device('/job:PS/task:1'): weights_2 = tf.Variable() with tf.device('/job:worker/task:1'): tf.nn.relu()
3. training mechanism
Synchronous training mechanism
Each independent worker training, after all worker until the calculated gradient values are aggregated to calculate the model parameters, and updates the current parameters of the model training step of calculating a faster worker needs to be blocked to wait for the slower computing worker
y = tf.nn.softmax(tf.nn.xw_plus_b(hid,sm_w,sm_b)) cross_entropy = -tf.reduce_sum(FLAGS.learning_rate) if FLAGS.sync_replicas: opt = tf.train.SyncReplicasOptimizer(opt,replicas_to_aggregate=10,total_num_replicas=100,name='mnist_sync') opt.minimize(cross_entropy,global_step=1)
Asynchronous training mechanism
Each worker is independently after training, immediately calculate the model parameter calculated gradient values, each non-blocking wait for all other worker worker gradient calculation is complete.