Stanford CS20学习笔记(2)

1 Constants, Sequences

1.1 常量

(1)tf.constant
(
value,
dtype=None,
shape=None,
name=’Const’,
verify_shape=False
)

import tensorflow as tf

a = tf.constant([2,2], name = 'a')
b = tf.constant([[0,1], [2,3]],name='b')

#Broadcasting like Numpy
x = tf.multiply(a, b, name='mul')

with tf.Session() as sess:
    print(sess.run(x))

# [[0 2]
# [4 6]]

1.2 创建指定值的tensor

(2)tf.zeros(shape, dtype=tf.float32, name=None)
Description: creates a tensor of shape and all elements will be zeros

e.g.
tf.zeros([2, 3], tf.int32) ==> [[0, 0, 0], [0, 0, 0]]

(3)tf.zeros_like(input_tensor, dtype=None, name=None, optimize=True)

Description: creates a tensor of shape and type (unless type is specified) as the input_tensor but all elements are zeros.

eg.

 input_tensor is [[0, 1], [2, 3], [4, 5]]

tf.zeros_like(input_tensor) ==> [[0, 0], [0, 0], [0, 0]]

(4)tf.ones(shape, dtype=tf.float32, name=None)

(5)tf.ones_like(input_tensor, dtype=None, name=None, optimize=True)

(6)tf.fill(dims, value, name=None)
Description: creates a tensor filled with a scalar value.

e.g.
tf.fill([2, 3], 8) ==> [[8, 8, 8], [8, 8, 8]]

1.3 创建序列constant tensor

(1)tf.lin_space(start, stop, num, name=None)
tf.lin_space(10.0, 13.0, 4) ==> [10. 11. 12. 13.]

(2)tf.range(start, limit=None, delta=1, dtype=None, name=’range’)

tf.range(3, 18, 3) ==> [3 6 9 12 15]
tf.range(5) ==> [0 1 2 3 4]

1.4 随机生成常量

tf.random_normal
tf.truncated_normal
tf.random_uniform
tf.random_shuffle
tf.random_crop
tf.multinomial
tf.random_gamma

1.5 TF中的数据类型

TensorFlow integrates seamlessly with NumPy

tf.int32 == np.int32   # ⇒ True

对于 tf.Session.run(fetches),如果fetches是Tensor，返回的则是numpy.ndarray类型

sess = tf.Session()
a = tf.zeros([2, 3], np.int32)
print(type(a)) # ⇒ <class 'tensorflow.python.framework.ops.Tensor'>
a = sess.run(a)
print(type(a)) # ⇒ <class 'numpy.ndarray'>

在实践中尽量使用TF的数据类型，因为：

• Python native types: TensorFlow has to infer Python type
• NumPy arrays: NumPy is not GPU compatible

$\color{Red}{\text{★}}$ 有一点不是很懂， 它说用不建议多用常量，因为常量会造成 loading graphs expensive 虽然不懂，但是可以先记着。

2 变量 Variables

2.1 两种创建variable的方式

tf.Variable与tf.get_variable。教程中说倾向于用后者来创建变量。后者创建变量时必须给与名称。这个函数比较复杂

# create variables with tf.Variable
s = tf.Variable(2, name="scalar")
m = tf.Variable([[0, 1], [2, 3]], name="matrix")
W = tf.Variable(tf.zeros([784,10]))

# create variables with tf.get_variable
s = tf.get_variable("scalar", initializer=tf.constant(2))
m = tf.get_variable("matrix", initializer=tf.constant([[0, 1], [2, 3]]))
W = tf.get_variable("big_matrix", shape=(784, 10), initializer=tf.zeros_initializer())

2.2 tf. c onstant 与 tf. V ariable的区别

1. tf. constant 是一个op
2. tf. Variable 是一个有着很多op的class

x = tf.Variable(...)

x.initializer # init op
x.value() # read op
x.assign(...) # write op
x.assign_add(...) # and more
#.........还有很多op

2.3 variable需要初始化才能使用

# create variables with tf.get_variable
s = tf.get_variable("scalar", initializer=tf.constant(2))
m = tf.get_variable("matrix", initializer=tf.constant([[0, 1], [2, 3]]))
W = tf.get_variable("big_matrix", shape=(784, 10), initializer=tf.zeros_initializer())
with tf.Session() as sess:
    print(sess.run(W)) 

输出
$\color{Red}{\text{>> FailedPreconditionError: Attempting to use uninitialized value Variable}}$

初始化
1. 最简单的方式是一次性初始化所有变量

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
#Initializer 是一个op，因此需要在session中执行也会有效

2.可以初始化指定的多个变量

# Initialize only a subset of variables:
with tf.Session() as sess:
    sess.run(tf.variables_initializer([a, b]))

3.初始化单个变量, tensor.initializer

# Initialize a single variable
W = tf.Variable(tf.zeros([784,10]))
with tf.Session() as sess:
    sess.run(W.initializer)

2.4 Variable的一些op

(1) .eval() ：执行op

# W is a random 700 x 100 variable object
W = tf.Variable(tf.truncated_normal([700, 10]))
with tf.Session() as sess:
    sess.run(W.initializer)
    print(W.eval()) # Similar to print(sess.run(W))

(2) .assign ： 赋值op

W = tf.Variable(10)
W.assign(100)
with tf.Session() as sess:
    sess.run(W.initializer)
    print(W.eval()) # >> 10
# W.assign(100) 实际生成了一个op,但是这个op并没有被执行,因此没有生效

# -------------------------------------
W = tf.Variable(10)
assign_op = W.assign(100)
with tf.Session() as sess:
sess.run(W.initializer)
    sess.run(assign_op) #执行赋值op
    print(W.eval()) # >> 100

# create a variable whose original value is 2
my_var = tf.Variable(2, name="my_var")

# assign a * 2 to a and call that op a_times_two
my_var_times_two = my_var.assign(2 * my_var)

with tf.Session() as sess:
    sess.run(my_var.initializer)
    sess.run(my_var_times_two) # >> the value of my_var now is 4
    sess.run(my_var_times_two) # >> the value of my_var now is 8
    sess.run(my_var_times_two) # >> the value of my_var now is 16

# 因为每一次调用my_var_times_two这个op时,都意味着将2*my_var赋值给my_var

(3) .assign_sub() and .assign_add()

my_var = tf.Variable(10)
With tf.Session() as sess:
    sess.run(my_var.initializer) 

    # increment by 10
    sess.run(my_var.assign_add(10)) # >> 20

    # decrement by 2
    sess.run(my_var.assign_sub(2)) # >> 18

2.5 多个session共存, 每个session有各自的variable的copy

W = tf.Variable(10)

sess1 = tf.Session()
sess2 = tf.Session()

sess1.run(W.initializer)
sess2.run(W.initializer)

print(sess1.run(W.assign_add(10))) # >> 20
print(sess2.run(W.assign_sub(2))) # >> 8

print(sess1.run(W.assign_add(100))) # >> 120
print(sess2.run(W.assign_sub(50))) # >> -42

sess1.close()
sess2.close()

2.6 placeholder

(1) 背景
一个TF程序由两个阶段组成
1. Assemble a graph
2. Use a session to execute operations in the graph.
$\qquad$存在这么一种情况: Assemble the graph first without knowing the values needed for computation.
$\qquad$而placeholders则让We, or our clients, can later supply their own data when they
need to execute the computation.

(2) placeholder的使用
**tf.placeholder(dtype, shape=None, name=None)**

# create a placeholder for a vector of 3 elements, type tf.float32
a = tf.placeholder(tf.float32, shape=[3])
b = tf.constant([5, 5, 5], tf.float32)
# use the placeholder as you would a constant or a variable
c = a + b # short for tf.add(a, b)

# 错误使用---------------
with tf.Session() as sess:
    print(sess.run(c)) # >> InvalidArgumentError: a doesn’t an actual value\

#正确使用 ----------
with tf.Session() as sess:
    print(sess.run(c, feed_dict={a: [1, 2, 3]})) # the tensor a is the key, not the string ‘a’
# >> [6, 7, 8]

# 另一种使用:feed 多个数据
with tf.Session() as sess:
    for a_value in list_of_values_for_a:
        print(sess.run(c, {a: a_value}))

• 注意 shape这个参数,用以指定输入的形状. 如果设置为None, 则意味着可以给与任何形状的输入. 但是这也意味着除了bug的时候难以调试, 因此尽量不要用None.

(3) feedable tensor
feed_dict这一行为不止可以作用于placeholder上,实际上它可以作用于任何feedable的tensor上. 也就是说feed_dict并不是为placeholder独创的. 用下面的语句可以看tensor是否可以feedable.
tf.Graph.is_feedable(tensor): True if and only if tensor is feedable.

# create operations, tensors, etc (using the default graph)
a = tf.add(2, 5)
b = tf.multiply(a, 3)
with tf.Session() as sess:
    # compute the value of b given a is 15
    sess.run(b, feed_dict={a: 15}) # >> 45

3 Normal loading && Lazy loading

3.1 Normal loading

定义: 在执行graph前,创建好节点

= tf.Variable(10, name='x')
y = tf.Variable(20, name='y')
z = tf.add(x, y)    # create the node before executing the graph

writer = tf.summary.FileWriter('./graphs/normal_loading', tf.get_default_graph())

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for _ in range(10):
        sess.run(z)

writer.close()

使用tf.get_default_graph().as_graph_def() 可以查看graph内部:

node {
    name: "Add"
    op: "Add"
    input: "x/read"
    input: "y/read"
    attr {
        key: "T"
        value {
            type: DT_INT32
        }
    }
}

可以发现仅向图的内部添加了一次”Add”节点

3.2 Lazy loading

定义 : Defer creating/initializing an object until it is needed

x = tf.Variable(10, name='x')
y = tf.Variable(20, name='y')
writer = tf.summary.FileWriter('./graphs/normal_loading', tf.get_default_graph())
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for _ in range(10):
        sess.run(tf.add(x, y)) # someone decides to be clever to save one line of code

writer.close()

使用tf.get_default_graph().as_graph_def() 可以查看graph内部:

node {
name: "Add_1"
    op: "Add"
    ...
    }

...

node {
    name: "Add_10"
    op: "Add"
...
    }

节点”Add”向图的定义添加了10次!

3.3 summary

一定要避免lazy loading,!!!!

1. Separate definition of ops from computing/running ops
2. Use Python property to ensure function is also loaded once the first time it is
   called*(这一点我不是很了解)