"Natural Language Understanding - from the rule to deep learning" reading notes

Original link: https://yq.aliyun.com/articles/158691
Author: Li Yongbin
Published: 2017-08-04 10:53:48

In this paper, combined with the specific natural language understanding NUI platform, detailed, description and classification intent attribute extraction system two core algorithms.

Difficulties natural language understanding

• Linguistic diversity
  • I want to hear the king called me to patrolled
  • Give me sow king called me to patrolled
  • I would like to listen to music king called me to patrolled
  • King told me to put the first park ranger
  • King told me to sing a park ranger
  • Play music king called me to patrolled
  • King asked me to put a song Mountaineer
  • Uncle to be the first king called me to patrolled
• Language Ambiguity
  • I'm going to Lhasa
    • Train tickets?
    • Airline tickets?
    • music?
    • Find or attractions?
• Robust language
  • Typo: King asked me to Johor Bahru
  • Many words: king called me to patrolled
  • Few words: king told me to park ranger
  • Nickname: Bear Bear two (finger of bears)
  • Incoherence: I want to see the grace to spend one thousand bone
  • Noise: King family only told me to go Mountaineer slightly
• Knowledge of the language-dependent
  • Large pear: in addition to that fruit, it can also indicate the name of the restaurant
  • Seven days: in addition to that time, but also can indicate the name of the hotel
  • General Staff: General Staff in addition to that, it can also indicate the name of the restaurant
  • Weather Forecast: or a song
  • Goodnight: This is a song
• Context of language
  • Conversation context
  • Device Context
  • Application Context
  • User portrait
  • ...

Implementation intentions classification
- based rule (rule-based)
- CFG
- JSGF
- ......
- traditional machine learning methods
- SVM
- ME
- ......
- depth learning
- CNN
- RNN / LSTM
- ......

Rule-based approach (based on CFG way of an example)

References: Ward & Issar, CMU Phoenix System, 1996

Air tickets from Beijing to Hangzhou

基于传统统计的方法（基于SVM的方法）

基于深度学习的方法，两种典型网络结构：

• CNN（卷积神经网络）
• RNN（循环神经网络）

几种衍生变型：

• CNN （Yoon Kim, Neural Networks for Sentence Classification, EMNLP, 2014）
• LSTM （Suman Ravuri and Andreas Stolcke, Recurrent Neural Network and LSTM Models for Lexical Utterance Classification, InterSpecch, 2015）
• RCNN （Siwei Lai, Liheng Xu, Kang Liu, Jun Zhao, Recurrent Convolutional Neural Networks for Text Classification, AAAI, 2015）
• C-LSTM （Chunting Zhou, Chonglin Sun, Zhiyuan Liu, Francis C.M. Lau, A C-LSTM Neural Network for Text Classification, arXiv, 2015）

单纯的CNN分类效果无法超越复杂特征工程的SVM分类器，尤其是在像音乐、视频等大量依赖世界知识的领域中。

深度学习在取得巨大成功后，慢慢开始显露出瓶颈，比如如何表示知识、存储知识，如何推理等。其中一个探索方向就是试图把联结主义和符号主义进行融合。纯粹的基于联结主义的神经网络的输入是distributed representation，把基于符号主义的symbolic representation融合到网络中，可以大大提高效果。

属性抽取的实现方法

• 基于规则（rule-based）
  • Lexicon-based
  • CFG
  • JSGF
  • ……
• 传统机器学习方法
  • HMM
  • CRF
  • ……
• 深度学习方法
  • RNN/LSTM
  • ……

基于规则的方法

• JSGF（JSpeech Grammar Format）

JSGF is a BNF-style, platform-independent, and vendor-independent textual representation of grammars for use in speech recognition.

示例

展开图

“帮我打开空调”匹配路径

基于传统统计的方法（基于CRF的方法）

基于深度学习的方法（Grégoire Mesnil, et. al, Using Recurrent Neural Networks for Slot Filling in Spoken Language Understanding, TASLP, 2015）

• RNN
• LSTM
• Bi-LSTM
• Bi-LSTM-Viterbi
• Bi-LSTM-CRF

还有一些联合模型：

• Xiaodong Zhang, HoufengWang, A Joint Model of Intent Determination and Slot Filling for Spoken Language Understanding, IJCAI, 2016
• Bing Liu, Ian Lane, Joint Online Spoken Language Understanding and Language Modeling with Recurrent Neural Networks, arxiv, 2016

在实际的系统中，基于规则的方法和基于深度学习的方法并存。基于规则的方法主要用来快速解决问题，比如一些需要快速干预的BUG；基于深度学习的方法是系统的核心。