LM perplexity by using tensorflow
1、Language model perplexity是衡量语言模型好坏的重要指标,其计算公式P(sentence)^-(1/N)
2、tensorflow的RNN模型如何使用
参考API文档:tf.contrib.legacy_seq2seq.sequence_loss_by_example,这个函数会返回一个大小为N的列表,N为句子数目,每个值代表该句话的log-perplexity。
tf.contrib.legacy_seq2seq.sequence_loss_by_example(
logits,
targets,
weights,
average_across_timesteps=True,
softmax_loss_function=None,
name=None
)
#计算这个batch的log-perplexity,返回的是shape=[batch]的值,代表每个句子的log-perplexity
loss = legacy_seq2seq.sequence_loss_by_example([self.logits],
[tf.reshape(self.targets, [-1])],
[tf.ones([args.batch_size * args.seq_length])],
args.vocab_size)
#计算这个batch中所有句子的平均log-perplexity
self.cost = tf.reduce_sum(loss) / args.batch_size
#计算这个batch中所有句子的平均perplexity
self.perplexity = tf.exp(self.cost)
可以看一下tensorflow对sequence_loss_by_example的源码
def sequence_loss_by_example(logits,
targets,
weights,
average_across_timesteps=True,
softmax_loss_function=None,
name=None):
"""Weighted cross-entropy loss for a sequence of logits (per example).
Args:
logits: List of 2D Tensors of shape [batch_size x num_decoder_symbols].
targets: List of 1D batch-sized int32 Tensors of the same length as logits.
weights: List of 1D batch-sized float-Tensors of the same length as logits.
average_across_timesteps: If set, divide the returned cost by the total
label weight.
softmax_loss_function: Function (labels-batch, inputs-batch) -> loss-batch
to be used instead of the standard softmax (the default if this is None).
name: Optional name for this operation, default: "sequence_loss_by_example".
Returns:
1D batch-sized float Tensor: The log-perplexity for each sequence.
Raises:
ValueError: If len(logits) is different from len(targets) or len(weights).
"""
if len(targets) != len(logits) or len(weights) != len(logits):
raise ValueError("Lengths of logits, weights, and targets must be the same "
"%d, %d, %d." % (len(logits), len(weights), len(targets)))
with ops.name_scope(name, "sequence_loss_by_example",
logits + targets + weights):
log_perp_list = []
for logit, target, weight in zip(logits, targets, weights):
if softmax_loss_function is None:
# TODO(irving,ebrevdo): This reshape is needed because
# sequence_loss_by_example is called with scalars sometimes, which
# violates our general scalar strictness policy.
target = array_ops.reshape(target, [-1])
crossent = nn_ops.sparse_softmax_cross_entropy_with_logits(
labels=target, logits=logit)
else:
crossent = softmax_loss_function(target, logit)
log_perp_list.append(crossent * weight)
log_perps = math_ops.add_n(log_perp_list)
if average_across_timesteps:
total_size = math_ops.add_n(weights)
total_size += 1e-12 # Just to avoid division by 0 for all-0 weights.
log_perps /= total_size
return log_perps
计算的过程是:
1、求句子的每个时间点(RNN timestep)处的loss,然后对每个时间点的loss求和。
2、求句子的长度(timestep),然后loss/timestep。(默认需要对average_across_timesteps )
参考链接:
tensorflow的legacy_seq2seq
How to calculate perplexity of RNN in tensorflow
tf.contrib.legacy_seq2seq.sequence_loss_by_example
rnn_ptb_perlexity