17- Create models using Keras in TensorFlow (TensorFlow series) (deep learning)

knowledge points

• Keras is a high-level neural network API written in Python
• The square root of the data: np.sqrt(784)       # 28
• Code runs tuned to CPU or GPU:

import tensorflow as tf
cpu=tf.config.list_physical_devices("CPU")
tf.config.set_visible_devices(cpu)

• Model display: model.summary ()

Create a model:

• Model creation: model = Sequential ()
• Add convolutional layer: model.add (Dense(32, activation='relu', input_dim=100))  # The first layer needs input_dim
• Add dropout: model.add (Dropout(0.2))
• Add the second network: model.add(Dense( 512, activation='relu') )   # Except first, other layers do not input shape
• Add output layer: model.add( Dense(num_classes, activation='softmax') )  # last usually use softmax
• In TensorFlow, use the model.compile method to select the optimizer and loss function :

  • optimizer: Optimizer : Mainly include: tf.train.AdamOptimizer , tf.train.RMSPropOptimizer , or tf.train.GradientDescentOptimizer .

  • loss: loss function : mainly include: mean square error (mse, regression ), categorical_crossentropy ( multi-classification ) , and binary_crossentropy ( two-classification ).

  • metrics : Algorithm evaluation criteria, general classification uses accuracy .

model.compile(loss='categorical_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

• model.fit (x_train, y_train, batch_size = 64, epochs = 20, validation_data = (x_test, y_test) )    # model training

Model Evaluation:

• score = model.evaluate (x_test, y_test, verbose=0) two return values: [ loss rate, accuracy rate]

Large data set processing:

• Turn large dataset data into dataset: dataset = tf.data.Dataset.from_tensor_slices ((data, labels))
  • Specify the size of each batch of data: dataset = dataset.batch(32) .repeat()
  • dataset data training : model.fit (dataset, epochs=10, steps_per_epoch=30)
• Save the model : model.save ('my_model.h5')
• Load the model : model = tf.keras.models.load_model ('my_model.h5')

---

1 Keras Basics

1.1 Introduction

Keras is a high-level neural network API written in Python that can run with TensorFlow, CNTK or Theano as a backend. "Deep Learning for humans" is written on Keras's official github, mainly because it can create neural networks simply and quickly without considering many intermediate processes like Tensorflow.

To put it bluntly, Keras is a shell that needs to be run in combination with back-end frameworks such as TensorFlow, CNTK or Theano. Based on these characteristics, getting started with Keras is very simple. TensorFlow has integrated keras.

Chinese Official Documentation: Homepage - Keras Chinese Documentation

1.2 Keras use

1.2.1 Create a model

• The easiest way to use it is to use Keras Sequential , which is called a sequential model in Chinese, which can be used to represent the linear stacking of multiple network layers. When using it, you can pass a list of network layer instances to Sequential , for example:

from keras.models import Sequential
from keras.layers import Dense, Activation

model = Sequential([
    Dense(32, input_shape=(784,)),
    Activation('relu'),
    Dense(10),
    Activation('softmax'),
])
model.summary()

 

• Layers can also be added to the model simply using the .add() method :

model = Sequential()
model.add(Dense(32, input_dim=784))
model.add(Activation('relu'))
model.summary()

 1.2.2 Model parameter setting

For the input layer, you need to specify the size of the input data, through the input_shape attribute in the Dense object. Note that there is no need to write the size of the batch. input_shape=(784,) is equivalent to the shape we defined in the neural network as (None, 784) Tensor .

After the model is successfully created, it needs to be compiled. Use the .compile() method to compile the created model. The compile() method mainly needs to specify the following parameters:

• Optimizer : It can be the string name of the optimizer defined by Keras, such as 'rmsprop' or an instance object of the Optimizer class. Common optimizers include: SGD, RMSprop, Adagrad, Adadelta , etc.

• Loss function loss : The objective function for model view minimization, which can be a string form of an existing loss function, such as: categorical_crossentropy, or an objective function.

• Evaluation metrics . Metrics to evaluate the performance of the algorithm. For classification problems, it is recommended to set metrics = ['accuracy'] . The evaluation standard can be an existing standard string identifier, or a custom evaluation standard function.

The following is a common way of writing compile:

# 多分类问题
model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# 二分类问题
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# 均方误差回归问题
model.compile(optimizer='rmsprop',
              loss='mse')

# 自定义评估标准函数
import keras.backend as K

def mean_pred(y_true, y_pred):
    return K.mean(y_pred)

model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy', mean_pred])
model.summary()

• Training model: Use the .fit() method to pass the training data, training times (epoch) , and batch size (batch_size) to the fit() method.

# 训练模型，以 32 个样本为一个 batch 进行迭代
model.fit(data, labels, epochs=10, batch_size=32)

1.3 Keras functional API

The Sequential sequential model encapsulates too many things and is not flexible enough. If you want to define a complex model, you can use Keras's functional API.

The following is an example of a fully connected network:    # Handwritten digit recognition

from keras.layers import Input, Dense
from keras.models import Model
from keras.datasets import mnist
import tensorflow as tf

# 导入手写数字数据集
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# 对数据进行初步处理
x_train = x_train.reshape(60000, 784)
x_train = x_train.astype('float32')
x_train /= 255
# 将标记结果转化为独热编码
y_train = tf.keras.utils.to_categorical(y_train, num_classes=10)

# 这部分返回一个张量
inputs = Input(shape=(784))
# 层的实例是可调用的，它以张量为参数，并且返回一个张量
output_1 = Dense(64, activation='relu')(inputs)
output_2 = Dense(64, activation='relu')(output_1)
predictions = Dense(10, activation='softmax')(output_2)

# 这部分创建了一个包含输入层和三个全连接层的模型
model = Model(inputs=inputs, outputs=predictions)
model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
model.fit(x_train, y_train)  # 开始训练

This example can help us make some simple understandings.

• Layer instances are callables that take tensors and return a tensor

• Both input and output are tensors , both of which can be used to define a model (Model)

• Such a model can be trained like Keras' Sequential model .

2. Using Keras in TensorFlow

keras is integrated in tf.keras.

2.1 Create a model

To create a simple model, use tf.keras.sequential .

import tensorflow as tf
import tensorflow.keras.layers as layers
model = tf.keras.Sequential()
# 创建一层有64个神经元的网络:
model.add(layers.Dense(64, activation='relu'))
# 添加另一层网络:
model.add(layers.Dense(64, activation='relu'))
# 输出层:
model.add(layers.Dense(10, activation='softmax'))

2.2 Configure layers

layers contains the following three sets of important parameters:

• activation : activation function, 'relu', 'sigmoid', 'tanh' .

• kernel_initializer and bias_initializer : Initializers for weights and biases . Glorot uniform is the default initializer. Generally do not need to be changed.

• kernel_regularizer and bias_regularizer : Regularization of weights and biases: L1, L2 .

The following are examples of configuration models:

import tensorflow.keras.layers as layers
# 激活函数为sigmoid:
layers.Dense(64, activation='sigmoid')
# Or:
layers.Dense(64, activation=tf.sigmoid)

# 权重加了L1正则:
layers.Dense(64, kernel_regularizer=tf.keras.regularizers.l1(0.01))

# 给偏差加了L2正则
layers.Dense(64, bias_regularizer=tf.keras.regularizers.l2(0.01))

# 随机正交矩阵初始化器:
layers.Dense(64, kernel_initializer='orthogonal')

# 偏差加了常数初始化器
layers.Dense(64, bias_initializer=tf.keras.initializers.constant(2.0))

2.3 Training and Evaluation

2.3.1 Configuration Model

Using compile to configure the model , there are mainly the following sets of important parameters.

• optimizer: Optimizer : Mainly include: tf.train.AdamOptimizer , tf.train.RMSPropOptimizer , or tf.train.GradientDescentOptimizer .

• loss: loss function : mainly include: mean square error (mse, regression ), categorical_crossentropy ( multi-classification ) , and binary_crossentropy ( two-classification ).

• metrics : Algorithm evaluation criteria, general classification uses accuracy .

The following is an example of compile:

# 配置均方误差的回归.
model.compile(optimizer = tf.train.AdamOptimizer(0.01),
              loss = 'mse',       # mean squared error
              metrics = ['mae'])  # mean absolute error

# 配置多分类的模型.
model.compile(optimizer=tf.train.RMSPropOptimizer(0.01),
              loss=tf.keras.losses.categorical_crossentropy,
              metrics=[tf.keras.metrics.categorical_accuracy])

2.3.2 Training

Use the fit method of the model for training, mainly with the following parameters:

• epochs: training times

• batch_size: How much data is in each batch

• validation_data: test data

For small-scale data , you can directly pass the training data into fit.

import numpy as np

data = np.random.random((1000, 32))
labels = random_one_hot_labels((1000, 10))

val_data = np.random.random((100, 32))
val_labels = random_one_hot_labels((100, 10))

model.fit(data, labels, epochs=10, batch_size=32,
          validation_data=(val_data, val_labels))

For large-scale training data , use dataset in tensorflow.

# 把数据变成dataset
dataset = tf.data.Dataset.from_tensor_slices((data, labels))
# 指定一批数据是32, 并且可以无限重复
dataset = dataset.batch(32).repeat()

val_dataset = tf.data.Dataset.from_tensor_slices((val_data, val_labels))
val_dataset = val_dataset.batch(32).repeat()

# 别忘了steps_per_epoch, 表示执行完全部数据的steps
model.fit(dataset, epochs=10, steps_per_epoch=30)

model.fit(dataset, epochs=10, steps_per_epoch=30,
          validation_data=val_dataset,
          validation_steps=3)

2.3.3 Assessment and Forecast

Use tf.keras.Model.evaluate and tf.keras.Model.predict to evaluate and predict. Evaluate prints the algorithm's loss and score.

data = np.random.random((1000, 32))
labels = random_one_hot_labels((1000, 10))
#  普通numpy数据
model.evaluate(data, labels, batch_size=32)
# tensorflow dataset数据
model.evaluate(dataset, steps=30)

predict:

result = model.predict(data, batch_size=32)
print(result.shape)

2.4 Using the functional API

Functional API mainly needs to define the objects of each component by itself and pass them manually.

inputs = tf.keras.Input(shape=(32,))  # 返回placeholder
# A layer instance is callable on a tensor, and returns a tensor.
x = layers.Dense(64, activation='relu')(inputs)
x = layers.Dense(64, activation='relu')(x)
predictions = layers.Dense(10, activation='softmax')(x)
model = tf.keras.Model(inputs=inputs, outputs=predictions)
# The compile step specifies the training configuration.
model.compile(optimizer=tf.train.RMSPropOptimizer(0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])
# Trains for 5 epochs
model.fit(data, labels, batch_size=32, epochs=5)

2.5 Save and restore

• Use model.save to save the entire model as an HDF5 file

model.save('my_model.h5')

• Restore to use tf.keras.models.load_model.

model = tf.keras.models.load_model('my_model.h5')

Note: If you use the optimizer of tensorflow, there is no model configuration information in the saved model , and it needs to be reconfigured after recovery. It is recommended to use the optimizer of keras.