Tensors
TensorFlow中数据的核心单元是tensor张量，一个tensor可以是任意维度的数组。tensor的rank指它的维度。
3 # a rank 0 tensor; this is a scalar with shape []
[1., 2., 3.] # a rank 1 tensor; this is a vector with shape [3]
[[1., 2., 3.], [4., 5., 6.]] # a rank 2 tensor; a matrix with shape [2, 3]
[[[1., 2., 3.]], [[7., 8., 9.]]] # a rank 3 tensor with shape [2, 1, 3] 好好理解一下[2, 1, 3]啊，一层一层来看。
TensorFlow Core tutorial
引入TensorFLow
import tensorflow as tfThe Computational Graph
计算图包括两个部分：
- 构造计算图
- 运行计算图
一个计算图是一系列排列成节点图的TensorFlow操作
Let’s build a simple computational graph. Each node takes zero or more tensors as inputs and produces a tensor as an output. One type of node is a constant（node的类型之一是常量）. Like all TensorFlow constants, it takes no inputs, and it outputs a value it stores internally. We can create two floating point Tensors node1 and node2 as follows:(来搞俩常量node玩玩)
node1 = tf.constant(3.0, dtype=tf.float32)
node2 = tf.constant(4.0) # also tf.float32 implicitly 默认float32
print(node1, node2)Tensor("Const:0", shape=(), dtype=float32) Tensor("Const_1:0", shape=(), dtype=float32)
傻了吧…输出不是3.0和4.0 [3.0,4.0]
Instead, they are nodes that, when evaluated, would produce 3.0 and 4.0, respectively. To actually evaluate the nodes, we must run the computational graph within a session. A session encapsulates(封装) the control and state of the TensorFlow runtime.
node只是用来指向TF构建的图中的节点，那两个节点能产生3.0和4.0，session用来控制和指示TF运行时的状态。
说白了，你建立的图只是一个外世界（Python的世界），外世界与里世界建立一一对应的关系，里世界（C++底层计算）为了计算速度快而存在，session是两个世界沟通的桥梁。
通过下面的代码创建session，并且运行计算图。
sess = tf.Session()
print(sess.run([node1, node2]))[3.0, 4.0]
如你所愿，输出了[3.0 ,4.0]
We can build more complicated computations by combining Tensor nodes with operations (Operations are also nodes). For example, we can add our two constant nodes and produce a new graph as follows:
节点也可以指向节点，比如：
node3 = tf.add(node1, node2)
print("node3:", node3)
print("sess.run(node3):", sess.run(node3))node3: Tensor("Add:0", shape=(), dtype=float32)
sess.run(node3): 7.0
TF提供了一个叫做TensorBoard的工具，可以用来可视化计算图，下面是可视化的一个截图：
As it stands, this graph is not especially interesting because it always produces a constant result. A graph can be parameterized to accept external inputs, known as placeholders. A placeholder is a promise to provide a value later.
计算图可以接受占位符参数，什么是占位符：就是先占个地方，可以之后在对这个地方赋值。
a = tf.placeholder(tf.float32)
b = tf.placeholder(tf.float32)
adder_node = a + b # + provides a shortcut for tf.add(a, b)The preceding three lines are a bit like a function or a lambda in which we define two input parameters (a and b) and then an operation on them. We can evaluate this graph with multiple inputs by using the feed_dict argument to the run method to feed concrete values to the placeholders:
可以通过feed_dict给placeholders传参，如下：
print(sess.run(adder_node, {a: 3, b: 4.5}))
print(sess.run(adder_node, {a: [1, 3], b: [2, 4]}))7.5
[ 3. 7.]
TB里面长这样：
通过增加其他操作符，可以使计算图变得更加灵活，比如：
add_and_triple = adder_node * 3.
print(sess.run(add_and_triple, {a: 3, b: 4.5}))22.5
可视化：
In machine learning we will typically want a model that can take arbitrary inputs, such as the one above. To make the model trainable, we need to be able to modify the graph to get new outputs with the same input. Variables allow us to add trainable parameters to a graph. They are constructed with a type and initial value:
ML的过程中需要提供arbitrary任意的输入，为了使得模型可以训练，还必须能够对输入进行改变，所以引出了Variables，允许向计算图里面添加可供训练的参数，Variables由数值类型和初始值两个参数构成。
举个例子：
W = tf.Variable([.3], dtype=tf.float32)
b = tf.Variable([-.3], dtype=tf.float32)
x = tf.placeholder(tf.float32)
linear_model = W * x + bConstants are initialized when you call tf.constant, and their value can never change. By contrast, variables are not initialized when you call tf.Variable. To initialize all the variables in a TensorFlow program, you must explicitly call a special operation as follows:
当你调用tf.constant常量就已经初始化了，与之相反，变量需要调用下面这条语句进行初始化：
init = tf.global_variables_initializer()
sess.run(init)It is important to realize init is a handle to the TensorFlow sub-graph that initializes all the global variables. Until we call sess.run, the variables are uninitialized.
init只是一个句柄，只有调用sess.run(init)时，变量才真正初始化。
Since x is a placeholder, we can evaluate linear_model for several values of x simultaneously as follows:
x是占位符，我们可以给x几个值来计算这个线性模型：
print(sess.run(linear_model, {x: [1, 2, 3, 4]}))[ 0. 0.30000001 0.60000002 0.90000004]
We’ve created a model, but we don’t know how good it is yet. To evaluate the model on training data, we need a y placeholder to provide the desired values, and we need to write a loss function.
我们已经创建了一个模型，但是如何评价他的好坏？我们需要一个占位符y表示预测值和一个loss function损失函数。
A loss function measures how far apart the current model is from the provided data. We’ll use a standard loss model for linear regression, which sums the squares of the deltas between the current model and the provided data. linear_model - y creates a vector where each element is the corresponding example’s error delta. We call tf.square to square that error. Then, we sum all the squared errors to create a single scalar that abstracts the error of all examples using tf.reduce_sum:
然后这段解释了loss function是干嘛的，显而易见，是指每个样本的loss，为了最优化模型，我们需要整个样本的loss，也就是cost function，通过tf.reduce_sum计算所有样本的loss之和。
y = tf.placeholder(tf.float32)
squared_deltas = tf.square(linear_model - y)
loss = tf.reduce_sum(squared_deltas)
print(sess.run(loss, {x: [1, 2, 3, 4], y: [0, -1, -2, -3]}))23.66
We could improve this manually by reassigning the values of W and b to the perfect values of -1 and 1. A variable is initialized to the value provided to tf.Variable but can be changed using operations like tf.assign. For example, W=-1 and b=1 are the optimal parameters for our model. We can change W and b accordingly:
可以自定义W and b:
fixW = tf.assign(W, [-1.])
fixb = tf.assign(b, [1.])
sess.run([fixW, fixb])
print(sess.run(loss, {x: [1, 2, 3, 4], y: [0, -1, -2, -3]}))0.0
tf.train API
A complete discussion of machine learning is out of the scope of this tutorial.
关于ML完整的探讨已经超过了这个教程的范围= =。意思是你不会ML关我P事，去学完理论再来用TF把。
However, TensorFlow provides optimizers that slowly change each variable in order to minimize the loss function.
TF提供optimizers优化器，最小化loss function的过程中可以缓慢的改变每个变量。
The simplest optimizer is gradient descent. It modifies each variable according to the magnitude of the derivative of loss with respect to that variable.
最简单的优化器是梯度下降优化器。
In general, computing symbolic derivatives manually is tedious and error-prone. Consequently, TensorFlow can automatically produce derivatives given only a description of the model using the function tf.gradients. For simplicity, optimizers typically do this for you. For example,
通常，手动计算导数是乏味和容易出错的，因此，TF能够通过自动的计算这些导数，只需要给模型一个描述，然后使用tf.gradients函数就可以了：
optimizer = tf.train.GradientDescentOptimizer(0.01)
train = optimizer.minimize(loss)
sess.run(init) # reset values to incorrect defaults.
for i in range(1000):
sess.run(train, {x: [1, 2, 3, 4], y: [0, -1, -2, -3]})
print(sess.run([W, b]))[array([-0.9999969], dtype=float32), array([ 0.99999082], dtype=float32)]
Now we have done actual machine learning! Although doing this simple linear regression doesn’t require much TensorFlow core code, more complicated models and methods to feed data into your model necessitate more code. Thus TensorFlow provides higher level abstractions for common patterns, structures, and functionality. We will learn how to use some of these abstractions in the next section.
至此一次ML实践结束，用TF能方便简洁的实现你的ML IDEA。当然，这个复杂的model也可以通过TB来可视化：
tf.estimator
tf.estimator是TF的高级库接口，它可以简化机器学习的结构，包括：
- 运行训练循环
- 运行评估循环
- 组织数据集
tf.estimator定义了很多公共模型
Basic usage
用它来做linear regression线性回归会更加简单
import tensorflow as tf
# NumPy is often used to load, manipulate and preprocess data.
import numpy as np定义特征列表：我们只使用一个数字特征。除了数字外还有很多其他复杂和有用的类型
feature_columns = [tf.feature_column.numeric_column("x", shape=[1])]estimator是调用训练和评估功能的前端。预先定义好的类型有线性回归，线性分类，和很多神经网络分类器和回归器，下面的代码实现了线性回归：
estimator = tf.estimator.LinearRegressor(feature_columns=feature_columns)INFO:tensorflow:Using default config.
WARNING:tensorflow:Using temporary folder as model directory: C:\Users\JLY\AppData\Local\Temp\tmpf71cj6rd
INFO:tensorflow:Using config: {'_keep_checkpoint_max': 5, '_tf_random_seed': 1, '_session_config': None, '_save_checkpoints_steps': None, '_log_step_count_steps': 100, '_save_summary_steps': 100, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'C:\\Users\\JLY\\AppData\\Local\\Temp\\tmpf71cj6rd', '_save_checkpoints_secs': 600}
TF提供了很多helper methods助手方法用来读取和建立数据集。
下面有两个数据集：训练集和评估集。
我们需要做的是设置batch_size和num_batch：
x_train = np.array([1., 2., 3., 4.])
y_train = np.array([0., -1., -2., -3.])
x_eval = np.array([2., 5., 8., 1.])
y_eval = np.array([-1.01, -4.1, -7, 0.])
input_fn = tf.estimator.inputs.numpy_input_fn(
{"x": x_train}, y_train, batch_size=4, num_epochs=None, shuffle=True)
train_input_fn = tf.estimator.inputs.numpy_input_fn(
{"x": x_train}, y_train, batch_size=4, num_epochs=1000, shuffle=False)
eval_input_fn = tf.estimator.inputs.numpy_input_fn(
{"x": x_eval}, y_eval, batch_size=4, num_epochs=1000, shuffle=False)我们可以设定1000次训练迭代，每次调用训练方法和传递训练集。
estimator.train(input_fn=input_fn, steps=1000)INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Saving checkpoints for 1 into C:\Users\JLY\AppData\Local\Temp\tmpf71cj6rd\model.ckpt.
INFO:tensorflow:step = 1, loss = 14.0
INFO:tensorflow:global_step/sec: 1768.63
INFO:tensorflow:step = 101, loss = 0.158421 (0.058 sec)
INFO:tensorflow:global_step/sec: 2149.03
INFO:tensorflow:step = 201, loss = 0.0259584 (0.046 sec)
INFO:tensorflow:global_step/sec: 2149.02
INFO:tensorflow:step = 301, loss = 0.0107812 (0.047 sec)
INFO:tensorflow:global_step/sec: 1998.61
INFO:tensorflow:step = 401, loss = 0.00192279 (0.051 sec)
INFO:tensorflow:global_step/sec: 2018.76
INFO:tensorflow:step = 501, loss = 0.000100472 (0.050 sec)
INFO:tensorflow:global_step/sec: 1768.67
INFO:tensorflow:step = 601, loss = 7.03774e-05 (0.057 sec)
INFO:tensorflow:global_step/sec: 1998.45
INFO:tensorflow:step = 701, loss = 1.27031e-05 (0.051 sec)
INFO:tensorflow:global_step/sec: 1885.44
INFO:tensorflow:step = 801, loss = 9.27439e-07 (0.053 sec)
INFO:tensorflow:global_step/sec: 1921.73
INFO:tensorflow:step = 901, loss = 3.1162e-07 (0.052 sec)
INFO:tensorflow:Saving checkpoints for 1000 into C:\Users\JLY\AppData\Local\Temp\tmpf71cj6rd\model.ckpt.
INFO:tensorflow:Loss for final step: 8.52581e-08.
<tensorflow.python.estimator.canned.linear.LinearRegressor at 0x2b248748be0>
Here we evaluate how well our model did.
train_metrics = estimator.evaluate(input_fn=train_input_fn)
eval_metrics = estimator.evaluate(input_fn=eval_input_fn)
print("train metrics: %r"% train_metrics)
print("eval metrics: %r"% eval_metrics)INFO:tensorflow:Starting evaluation at 2017-09-26-05:49:05
INFO:tensorflow:Restoring parameters from C:\Users\JLY\AppData\Local\Temp\tmpf71cj6rd\model.ckpt-1000
INFO:tensorflow:Finished evaluation at 2017-09-26-05:49:06
INFO:tensorflow:Saving dict for global step 1000: average_loss = 1.38662e-08, global_step = 1000, loss = 5.54647e-08
INFO:tensorflow:Starting evaluation at 2017-09-26-05:49:06
INFO:tensorflow:Restoring parameters from C:\Users\JLY\AppData\Local\Temp\tmpf71cj6rd\model.ckpt-1000
INFO:tensorflow:Finished evaluation at 2017-09-26-05:49:07
INFO:tensorflow:Saving dict for global step 1000: average_loss = 0.00253582, global_step = 1000, loss = 0.0101433
train metrics: {'global_step': 1000, 'loss': 5.5464675e-08, 'average_loss': 1.3866169e-08}
eval metrics: {'global_step': 1000, 'loss': 0.010143298, 'average_loss': 0.0025358244}
Notice how our eval data has a higher loss, but it is still close to zero. That means we are learning properly.
A custom model
自定义模型
tf.estimator不只局限于预设模型。
Suppose we wanted to create a custom model that is not built into TensorFlow. We can still retain the high level abstraction of data set, feeding, training, etc. of tf.estimator. For illustration, we will show how to implement our own equivalent model to LinearRegressor using our knowledge of the lower level TensorFlow API.
To define a custom model that works with tf.estimator, we need to use tf.estimator.Estimator. tf.estimator.LinearRegressor is actually a sub-class of tf.estimator.Estimator. Instead of sub-classing Estimator, we simply provide Estimator a function model_fn that tells tf.estimator how it can evaluate predictions, training steps, and loss. The code is as follows:
import numpy as np
import tensorflow as tf
# Declare list of features, we only have one real-valued feature
def model_fn(features, labels, mode):
# Build a linear model and predict values
W = tf.get_variable("W", [1], dtype=tf.float64)
b = tf.get_variable("b", [1], dtype=tf.float64)
y = W * features['x'] + b
# Loss sub-graph
loss = tf.reduce_sum(tf.square(y - labels))
# Training sub-graph
global_step = tf.train.get_global_step()
optimizer = tf.train.GradientDescentOptimizer(0.01)
train = tf.group(optimizer.minimize(loss),
tf.assign_add(global_step, 1))
# EstimatorSpec connects subgraphs we built to the
# appropriate functionality.
return tf.estimator.EstimatorSpec(
mode=mode,
predictions=y,
loss=loss,
train_op=train)
estimator = tf.estimator.Estimator(model_fn=model_fn)
# define our data sets
x_train = np.array([1., 2., 3., 4.])
y_train = np.array([0., -1., -2., -3.])
x_eval = np.array([2., 5., 8., 1.])
y_eval = np.array([-1.01, -4.1, -7, 0.])
input_fn = tf.estimator.inputs.numpy_input_fn(
{"x": x_train}, y_train, batch_size=4, num_epochs=None, shuffle=True)
train_input_fn = tf.estimator.inputs.numpy_input_fn(
{"x": x_train}, y_train, batch_size=4, num_epochs=1000, shuffle=False)
eval_input_fn = tf.estimator.inputs.numpy_input_fn(
{"x": x_eval}, y_eval, batch_size=4, num_epochs=1000, shuffle=False)
# train
estimator.train(input_fn=input_fn, steps=1000)
# Here we evaluate how well our model did.
train_metrics = estimator.evaluate(input_fn=train_input_fn)
eval_metrics = estimator.evaluate(input_fn=eval_input_fn)
print("train metrics: %r"% train_metrics)
print("eval metrics: %r"% eval_metrics)INFO:tensorflow:Using default config.
WARNING:tensorflow:Using temporary folder as model directory: C:\Users\JLY\AppData\Local\Temp\tmpo1jwtfp8
INFO:tensorflow:Using config: {'_keep_checkpoint_max': 5, '_tf_random_seed': 1, '_session_config': None, '_save_checkpoints_steps': None, '_log_step_count_steps': 100, '_save_summary_steps': 100, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'C:\\Users\\JLY\\AppData\\Local\\Temp\\tmpo1jwtfp8', '_save_checkpoints_secs': 600}
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Saving checkpoints for 1 into C:\Users\JLY\AppData\Local\Temp\tmpo1jwtfp8\model.ckpt.
INFO:tensorflow:step = 1, loss = 22.819291271
INFO:tensorflow:global_step/sec: 1940.36
INFO:tensorflow:step = 101, loss = 0.0428618415374 (0.052 sec)
INFO:tensorflow:global_step/sec: 2018.8
INFO:tensorflow:step = 201, loss = 0.00551747471313 (0.050 sec)
INFO:tensorflow:global_step/sec: 2060.36
INFO:tensorflow:step = 301, loss = 0.000166904336648 (0.049 sec)
INFO:tensorflow:global_step/sec: 2060.44
INFO:tensorflow:step = 401, loss = 5.22266242886e-06 (0.049 sec)
INFO:tensorflow:global_step/sec: 1978.78
INFO:tensorflow:step = 501, loss = 2.44514779169e-06 (0.051 sec)
INFO:tensorflow:global_step/sec: 2103.73
INFO:tensorflow:step = 601, loss = 7.78953573342e-08 (0.048 sec)
INFO:tensorflow:global_step/sec: 2103.82
INFO:tensorflow:step = 701, loss = 2.1247222487e-08 (0.048 sec)
INFO:tensorflow:global_step/sec: 2060.37
INFO:tensorflow:step = 801, loss = 3.11453845345e-09 (0.048 sec)
INFO:tensorflow:global_step/sec: 2039.35
INFO:tensorflow:step = 901, loss = 1.54485405925e-10 (0.050 sec)
INFO:tensorflow:Saving checkpoints for 1000 into C:\Users\JLY\AppData\Local\Temp\tmpo1jwtfp8\model.ckpt.
INFO:tensorflow:Loss for final step: 1.85340461243e-11.
INFO:tensorflow:Starting evaluation at 2017-09-26-05:49:13
INFO:tensorflow:Restoring parameters from C:\Users\JLY\AppData\Local\Temp\tmpo1jwtfp8\model.ckpt-1000
INFO:tensorflow:Finished evaluation at 2017-09-26-05:49:14
INFO:tensorflow:Saving dict for global step 1000: global_step = 1000, loss = 1.42244e-11
INFO:tensorflow:Starting evaluation at 2017-09-26-05:49:14
INFO:tensorflow:Restoring parameters from C:\Users\JLY\AppData\Local\Temp\tmpo1jwtfp8\model.ckpt-1000
INFO:tensorflow:Finished evaluation at 2017-09-26-05:49:15
INFO:tensorflow:Saving dict for global step 1000: global_step = 1000, loss = 0.0101004
train metrics: {'global_step': 1000, 'loss': 1.4224399e-11}
eval metrics: {'global_step': 1000, 'loss': 0.010100436}
