3.1 Anatomy of a neural network
As introduced in the previous chapters, training neural networks mainly focuses on the following four aspects.
Layers, multiple layers are combined into a network (or model).
Enter the data and corresponding goals.
The loss function is the feedback signal used for learning.
The optimizer decides how to proceed with the learning process.
3.1.1 Layer: the basic component of deep learning
- Simple vector data is stored in a 2D tensor of shape (samples, features), usually densely connected layer (densely connected layer, also called fully connected layer) or dense layer, corresponding to Keras Dense Class] to deal with. The sequence data is stored in a 3D tensor of shape (samples, timesteps, features), which is usually processed by a recurrent layer (such as Keras's LSTM layer). The image data is stored in a 4D tensor, usually processed with a two-dimensional convolutional layer (Keras Conv2D).
from keras import layers layer = layers.Dense(32, input_shape=(784,))#这个input_shape说明输入的张量是（N*784）的
We created a layer that only accepts the first 2D tensor with a dimension of 784 (the 0th axis is the batch dimension, and its size is not specified, so it can take any value) as input. This layer will return a tensor, and the size of the first dimension becomes
32.
from keras import models from keras import layers model = models.Sequential()
model.add(layers.Dense(32, input_shape=(784,)))
model.add(layers.Dense(32))
The second layer has no input shape (input_shape) parameter, it can automatically deduce that the input shape is equal to the output shape of the previous layer.
3.1.3 Loss function and optimizer: the key to the configuration learning process
Once the network architecture is determined, you also need to select the following two parameters.
Loss function (objective function) -it needs to be minimized during the training process. It can measure whether the current task has been successfully completed.
Optimizer -decides how to update the network based on the loss function. It performs a variant of Stochastic Gradient Descent (SGD).
The choice of loss function:
For example, for binary classification problems, you can use the binary crossentropy loss function; for multi-classification problems, you can use the categorical crossentropy loss function; for regression problems, you can use the average Mean-squared error loss function; for sequence learning problems, connectionist temporal classification (CTC, connectionist temporal classification) loss function can be used
3.2 Introduction to Keras
You have already seen an example of the Keras model, the MNIST example. The typical Keras workflow is similar to that example.
(1) Define training data: input tensor and target tensor.
(2) Define the network (or model) composed of layers and map the input to the target.
(3) Configure the learning process: select the loss function, optimizer, and indicators that need to be monitored.
(4) Call the fit method of the model to iterate on the training data.
There are two ways to define the model: one is to use the Sequential class (only used for linear stacking of layers, which is currently the most common network architecture), and the other is to use functional API (functional API, which is used for layer composition directed Acyclic graph, allowing you to build any form of architecture)
The two-layer model defined by the Sequential class (note that we passed the expected shape of the input data to the first layer):
from keras import models from keras import layers
model = models.Sequential()
model.add(layers.Dense(32, activation='relu', input_shape=(784,)))
model.add(layers.Dense(10, activation='softmax'))
Here is the same model defined with the functional API:
input_tensor = layers.Input(shape=(784,))
x = layers.Dense(32, activation='relu')(input_tensor)
output_tensor = layers.Dense(10, activation='softmax')(x)
model = models.Model(inputs=input_tensor, outputs=output_tensor)
Once the model architecture is defined, it does not matter whether the Sequential model or the functional API is used. The next steps are the same.
The configuration learning process is in the compilation step. You need to specify the optimizer and loss function used by the model, as well as the indicators you want to monitor during the training process. The following is an example of a single loss function, which is currently the most common
from keras import optimizers model.compile(optimizer=optimizers.RMSprop(lr=0.001), loss='mse',
metrics=['accuracy'])
model.fit(input_tensor, target_tensor, batch_size=128, epochs=10)#这种方法 不好，这个是把全部的数据丢进网络中，太占内存了
3.4 Classification of Movie Criticisms: Two Classification Problems
from keras.datasets import imdb from keras.datasets import reuters import numpy as np from keras import models from keras import layers from keras import optimizers from keras import losses from keras import metrics import matplotlib.pyplot as plt(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)#在线Load data#
train_data and test_data These two variables are a list of comments, #each comment is a list
of word indexes (Represents a series of words).
#train_labels and #test_labels are both lists of 0 and 1, where 0
# stands for negative and 1 stands for positive.
Data preparation:
Perform one-hot encoding on the list and convert it into a vector of 0 and 1. For example, the sequence [3, 5] will be converted into a 10,000-dimensional vector, only the elements with indices 3 and 5 are 1, and the rest are 0. Then the first
layer of the network can use the Dense layer, which can handle floating-point vector data
#The function below I understand this way, train_data is a list containing the index of each word of the comment. What this function does is first create a matrix of (len(sequences), dimension), and then the row represents the size of the data , The column represents, the index is 10000.
Then set the corresponding output to 1.
def vectorize_sequences(sequences, dimension=10000):
results = np.zeros((len(sequences), dimension))
for i, sequence in enumerate(sequences) :
results[i, sequence] = 1.
return results
x_train = vectorize_sequences(train_data)
x_test = vectorize_sequences(test_data)
#Label vectorization
y_train = np.asarray(train_labels).astype('float32')
y_test = np.asarray(test_labels).astype('float32')
#构建网络
model = models.Sequential()
model.add(layers.Dense(16, activation=‘relu’, input_shape=(10000,)))
model.add(layers.Dense(16, activation=‘relu’))
model.add(layers.Dense(1, activation=‘sigmoid’))
model.compile(optimizer='rmsprop', loss='binary_crossentropy',
metrics=['accuracy'])
model.compile(optimizer='rmsprop',loss='binary_crossentropy',metrics=['accuracy'])书上也有例子这个我看，和下面正式使用的差别是：acc，和accuracy的差别，具体用哪个不清楚
The above code passes the optimizer, loss function and indicator as strings, because rmsprop, binary_ crossentropy and accuracy are all built-in parts of Keras. Sometimes you may want to configure the parameters of a custom optimizer, or pass in a custom loss function or indicator function. The former can be implemented by passing an optimizer class instance to the optimizer parameter, as shown in the code listing 3-5; the latter can be implemented by passing a function object to the loss and metrics parameters,
as shown in the code listing 3-6.
The following is a custom optimizer, which
actually adds parameters
from keras import optimizers model.compile(optimizer=optimizers.RMSprop(lr=0.001), loss='binary_crossentropy',
metrics=['accuracy'])
Reserve data for verification
x_val = x_train[:10000]
partial_x_train = x_train[10000:]
y_val = y_train[:10000]
partial_y_train = y_train[10000:]
model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])
history = model.fit(partial_x_train, partial_y_train, epochs=20,
batch_size=512,
validation_data=(x_val, y_val))
Note that calling model.fit() returns a History object. This object has a member history, which is a dictionary containing all the data during the training process. Let's take a look.
history_dict = history.history
history_dict.keys()
dict_keys(['val_acc','acc','val_loss','loss']) The
dictionary contains 4 entries, corresponding to the indicators monitored during training and verification. In the following two code listings, we will use Matplotlib to plot the training loss and the verification loss on the same graph
绘制函数如下：
history_dict = history.history
loss_values = history_dict[‘loss’]
val_loss_values = history_dict[‘val_loss’]
epochs = range(1, len(loss_values) + 1)
plt.plot(epochs, loss_values, ‘bo’, label=‘Training loss’)
plt.plot(epochs, val_loss_values, ‘b’, label=‘Validation loss’)
plt.title(‘Training and validation loss’)
plt.xlabel(‘Epochs’)
plt.ylabel(‘Loss’)
plt.legend()
plt.show()
plt.clf() 清空图像
acc = history_dict[‘acc’] val_acc = history_dict[‘val_acc’]
plt.plot(epochs, acc, ‘bo’, label=‘Training acc’) plt.plot(epochs, val_acc, ‘b’, label=‘Validation acc’) plt.title(‘Training and validation accuracy’) plt.xlabel(‘Epochs’) plt.ylabel(‘Accuracy’) plt.legend()
plt.show()
I’m too lazy to post a code that can run. Among them, the bug is corrected because the data set cannot be loaded.
from keras.datasets import imdb
from keras.datasets import reuters
import numpy as np
from keras import models
from keras import layers
from keras import optimizers
from keras import losses
from keras import metrics
import matplotlib.pyplot as plt
(train_data, train_labels), (test_data, test_labels) = imdb.load_data(
num_words=10000)
print(len(train_data[0]))
word_index = imdb.get_word_index()
reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])
decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in train_data[0]])
def vectorize_sequences(sequences, dimension=10000):
results = np.zeros((len(sequences), dimension))
for i, sequence in enumerate(sequences):
results[i, sequence] = 1.
return results
x_train = vectorize_sequences(train_data)
x_test = vectorize_sequences(test_data)
#标签向量化
y_train = np.asarray(train_labels).astype('float32')
y_test = np.asarray(test_labels).astype('float32')
#构建网络
model = models.Sequential()
model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))
model.add(layers.Dense(16, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))
x_val = x_train[:10000]
partial_x_train = x_train[10000:]
y_val = y_train[:10000]
partial_y_train = y_train[10000:]
model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])
history = model.fit(partial_x_train, partial_y_train, epochs=20,
batch_size=512,
validation_data=(x_val, y_val))
history_dict = history.history
loss_values = history_dict['loss']
val_loss_values = history_dict['val_loss']
epochs = range(1, len(loss_values) + 1)
plt.plot(epochs, loss_values, 'bo', label='Training loss')
plt.plot(epochs, val_loss_values, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.show()
plt.clf()
acc = history_dict['acc']
val_acc = history_dict['val_acc']
plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.xlabel('Epochs') plt.ylabel('Accuracy')
plt.legend()
plt.show()
There are two other examples that are the same as the above example.
import numpy as np
from keras.datasets import reuters
from keras import models
from keras import layers
(train_data, train_labels), (test_data, test_labels) = reuters.load_data(
num_words=10000)#加载数据集
def vectorize_sequences(sequences, dimension=10000):
results = np.zeros((len(sequences), dimension))
for i, sequence in enumerate(sequences):
results[i, sequence] = 1.
return results
x_train = vectorize_sequences(train_data)
x_test = vectorize_sequences(test_data)
def to_one_hot(labels, dimension=46):
results = np.zeros((len(labels), dimension))
for i, label in enumerate(labels):
results[i, label] = 1.
return results
one_hot_train_labels = to_one_hot(train_labels)
one_hot_test_labels = to_one_hot(test_labels)
model = models.Sequential()
model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(46, activation='softmax'))
model.compile(optimizer='rmsprop', loss='categorical_crossentropy',
metrics=['accuracy'])
x_val = x_train[:1000]
partial_x_train = x_train[1000:]
y_val = one_hot_train_labels[:1000]
partial_y_train = one_hot_train_labels[1000:]
history = model.fit(partial_x_train, partial_y_train, epochs=20,
batch_size=512,
validation_data=(x_val, y_val))
import matplotlib.pyplot as plt
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(1, len(loss) + 1)
plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.show()