Pradeep Dasigi translation of a long article
Knowledge-Aware Natural Language Understanding
Knowledge-based perception of natural language understanding
Summary
Natural Language Understanding (NLU) systems need to encode human gener- ated text (or speech) and reason over it at a deep semantic level. Any NLU system typically involves two main components: The first is an encoder, which composes words (or other basic linguistic units) within the input utterances compute encoded representations, which are then used as features in the second component, a predic- tor, to reason over the encoded inputs and produce the desired output. We argue that the utterances themselves do not contain all the information needed for understanding them and identify two kinds of additional knowledge needed to fill the gaps: background knowledge and contextual knowledge. The goal of this thesis is to build end-to-end NLU systems that encode inputs along with relevant background knowledge, and reason about them in the presence of contextual knowledge.
Natural language understanding (NLU) system needs to be created by the human voice or text to encode, then the semantic level of depth to its reasoning. Any typical natural language understanding system composed of two parts: The first part is an encoder , a word in the input sentence it (or other basic unit of language) composition calculation encoded representation, and a second portion ( forecast period ) of characteristics, and encoding the input reasoning and produce the desired output. We believe that these statements itself does not contain all the information they need to understand, nor determined to fill the two additional knowledge required Blank: background knowledge and context knowledge . The goal of this paper is to create an end-to-natural language understanding system, the input sentence and background knowledge encoded together, and reasoning in the case of the context of existing knowledge.
The first part of the thesis deals with encoding background knowledge. While distributional methods for encoding sentences have been used to represent meaning of words in context, there are other aspects of semantics that are out of their reach. These are related to commonsense or real world information which is part of shared human knowledge but is not explicitly present in the input. We address this limitation by having the encoders also encode background knowledge, and present two approaches for doing so. First, we leverage explicit symbolic knowledge from WordNet to learn ontology-grounded token-level representations of words. We show sentence encodings based on our token representations outperform those based on off-the-shelf word embeddings at predicting prepositional phrase attachment and textual entailment. Second, we look at cases where the required background knowledge cannot be stated symbolically. We model selectional restrictions verbs place on their semantic role fillers to deal with one such case. We use this model to encode events, and show that these representations are better at detecting anomalies in newswire texts than sentence representations produced by LSTMs.
The first part of this process "of background knowledge is encoded." Although the distribution method has been encoded sentence meaning in the context of the words used to represent, but other aspects of the semantics are beyond their scope. This information with information or knowledge about the real world, they are part of the shared knowledge of mankind, but not explicitly represented in the input. Let us be resolved by the encoder also "background knowledge encoded" This restriction, and proposed two methods: the first method, we use the explicit symbol of knowledge to learning words from WordNet ontology-based, token level representation. Based on the statement indicates that the coding our token representation, better than those embedded into the existing text with the prepositional phrase attachment contains a word prediction; the second method, we studied the case of a symbolic representation of the required background knowledge can not be. We model the verb semantically choose to limit the role of filler words to handle this situation. We use this model to encode the event, and show these representations (representations) performance in the detection of abnormalities newswire text, than LSTM class model generation better sentences.
The second part focuses on reasoning with contextual knowledge. We look at Question-Answering (QA) tasks where reasoning can be expressed as sequences of discrete operations, (i.e. semantic parsing problems), and the answer can be obtained by executing the sequence of operations (or logical form) grounded in some context. We do not assume the availability of logical forms, and build weakly supervised semantic parsers. This training setup comes with significant challenges since it involves searching over an exponentially large space of logical forms. To deal with these challenges, we propose 1) using a grammar to constrain the output space of the semantic parser; 2) leveraging a lexical coverage measure to ensure the relevance of produced logical forms to input utterances; and 3) a novel iterative training scheme that alternates between searching for logical forms, and maximizing the likelihood of the retrieved ones, thus effectively transferring the knowledge from simpler logical forms to more complex ones. We build neural encoder-decoder models for semantic parsing that use these techniques, and show state-of-the-art results on two complex QA tasks grounded in structured contexts.
The second part of this paper is to use contextual knowledge reasoning. We have studied the question and answer (QA) tasks, reasoning can be expressed as a discrete sequence of operations (ie, semantic parsing issues), and the answer can be based on the sequence of operations in some contexts (or logical form) obtained by execution. We do not assume the availability of logical form, but weak supervision and construction of semantic parser. This training is set to bring a huge challenge, because it is you want to search in the form of exponentially large logical space. To meet these challenges, we propose: 1) use syntax to constrain output space semantic analyzer; 2) the use of vocabulary coverage measurement, ensure the relevance of the logical form generated by the input utterance; 3) a new iterative training scheme, the possibility of looking for alternate between logical form and to maximize the retrieval, effectively transferring knowledge from simple to more complex logic in the form of logical form. We use this technology to build the neural coding for semantic analysis - decoder model (encoder-decoder models), and two complex QA tasks based on structured content shows the results of SOTA.
Overall, this thesis presents a general framework for NLU with encoding and reasoning as the two core components, and how additional knowledge can augment them. While the tasks presented in this thesis are hard language understanding challenges themselves, they also serve as examples to highlight the role of background and contextual knowledge in encoding and reasoning components. The models built for the tasks provide empirical evidence for the need for additional knowledge, and pointers for building effective knowledge-aware models for other NLU tasks.
In summary, this paper presents a general framework for encoding and NLU reasoning for the two core components, and how to enhance them with additional knowledge. Although the task of this paper is difficult to understand the challenges of the language itself, but can also be used as an example to emphasize the role of background knowledge and context knowledge and reasoning in the encoding process. To do this task of building a model of the need for additional knowledge to provide empirical evidence, and "Building effective knowledge perceptual model for other NLU mission" to provide guidance.
table of Contents
1 Introduction / Overview
1.1 Natural Language Understanding / Natural Language Understanding
1.1.1 Definition / definitions
1.1.2 Parts of an NLU system / natural language understanding component of the system
1.2 Knowledge / Knowledge
1.2.1 Background Knowledge for Encoding / for background encoding
1.2.2 Contextual Knowledge for Reasoning / context for knowledge reasoning
1.3 Knowledge-Aware NLU / knowledge perception of natural language understanding
1.3.1 Better encoding with background knowledge / background knowledge-based coding
1.3.2 Reasoning with contextual knowledge / context-based knowledge reasoning
1.3.3 Evaluating NLU Performance / Natural Language Understanding Performance Evaluation
1.4 Thesis Contributions and Outline / outline and contributed papers
I Encoding with Background Knowlege / background knowledge-based coding
2 Related Work: Learning to Encode / related work: Coding Learning
2.1 Representation Learning for Lexical Semantics / lexical semantic representation of learning
2.1.1 from distributional to distributed represantations / xxx
2.2 Incorporating Knowledge / Knowledge integration
2.2.1 Multi-prototype word vectors / multi-word prototype vector
2.2.2 Relying on symbolic knowledge / xxx
2.3 Selectional Preference / xxx
2.4 WordNet as a source for Selectional Preferences / xxx
3 Encoding Sentences with Background Knowledge from Ontologies / coding is based on statements from the body of background knowledge
3.1 Introduction / Introduction
3.2 Ontology-Aware Token Embeddings / token embedded sensing body
3.3 WordNet-Grounded Context-Sensitive Token Embeddings / xxx
3.4 PP Attachment
3.5 Textual Entailment
3.6 Conclusion / Conclusion
3.6.1 Summary / summary
3.6.2 Future Work / Future Work
4 Leveraging Selectional Preferences as Background Knowledge for Encoding Events / Reference use as background knowledge to coding events
4.1 Understanding Events / Event understanding
4.2 Model For Semantic Anomaly Detection / semantic anomaly detection model
4.2.1 Training / Training
4.3 Data / Data
4.4 Results / results
4.5 Conclusion / Conclusion
II Reasoning with Contextual Knowledge / context-based knowledge reasoning
5 Related Work: Learning to Reason / related work: Inference Learning
5.1 Learning to Reason with Latent Variables / reasoning learning based latent variables
5.2 Semantic Parsing / semantic parser
5.3 Weak Supervision / weak oversight
5.3.1 Maximum marginal likelihood training / maximum likelihood training border
5.3.2 Reinforcement learning methods / reinforcement learning method
5.3.3 Structured learning algorithms / structured learning algorithm
5.3.4 Bridging objectives / target engagement
Constraints on decoder 6 Constrained Decoding for Semantic Parsing / semantic analysis
6.1 Need for Grammar Constraints / grammar constraints demand
6.2 Need for Entity Linking / entity link demand
6.3 Grammar-Constrained Neural Semantic Parser / semantic parser grammar constraints nerve
6.3.1 Encoder / encoder
6.3.2 Decoder / decoder
6.4 Experiments with "WIKI TABLE QUESTIONS" based on "wiki table issues" experimental /
6.4.1 xxx
6.4.2 xxx
6.4.3 xxx
6.4.4 xxx
6.4.5 xxx
6.4.6 xxx
6.4.7 xxx
6.4.8 xxx
6.5 Conclusion / Conclusion
7 Training Semantic Parsers using Iterative Coverage-Guided Search / retrieval using an iterative training guide covers the semantic parser
7.1 Coverage-guided search / retrieval coverage Wizard
7.2 Iterative search / retrieval iteration
7.3 Task Details / task details
7.4 Experiments / Experimental
7.5 Related Work / related work
7.6 Conclusion / Conclusion
8 Conclution / conclusion
8.1 Summary / summary
8.2 Future work in Knowledge-Aware Encoding / perceptual coder knowledge of future work
8.3 Future work in Knowledge-Aware Reasoning / decoder perceived knowledge of future work