ACL2022 Document-Level Event Argument Extraction via Optimal Transport

Document-Level Event Argument Extraction via Optimal Transport

Paper: https://aclanthology.org/2022.findings-acl.130/

Code:-

Journal/Conference: ACL 2022

Summary

Event Argument Extraction (EAE) is one of the subtasks of event extraction, which aims to identify the role of each entity in a specific event trigger word. Although previous work has achieved success in sentence-level EAE, document-level has been less explored. In particular, although the syntactic structure of sentences has been shown to be effective for sentence-level EAE, previous document-level EAE models completely ignore the syntactic structure of documents . Therefore, in this work, we investigate the importance of syntactic structure in document-level EAE. Specifically, we propose Optimal Transport (OT) to induce document structure based on sentence-level syntactic structure, tailored to the EAE task. Furthermore, we propose a novel regularization technique to explicitly constrain the contribution of irrelevant context words in the final prediction of EAE. We achieve state-of-the-art performance through extensive experiments on the benchmark document-level EAE dataset RAMS. Furthermore, our experiments on the ACE 2005 dataset reveal the effectiveness of the proposed model in sentence-level EAE by establishing new state-of-the-art results.

1 Introduction

Better methods are needed to prune dependency-based structures in documents to better preserve important words and exclude noisy words. Unlike previous work, which employs simple grammar-based rules, i.e., distances to dependency paths (Zhang et al, 2018), we argue that the pruning operation should also be aware of the semantics of words. In other words, two criteria should be considered, namely syntactic and semantic relevance . Specifically, if a word has a small distance from an event trigger word/argument word in the dependency structure (i.e., based on grammatical importance), and it is semantically related to a word in the dependency path (i.e., According to the semantic importance), the word is retained in the document structure of the document-level EAE. Note that semantic similarity between words can be obtained from model-induced word representations. A key challenge to this idea is the different nature of syntactic and semantic distance, which complicates the combination of information that determines the importance of words to structure. Furthermore, the retention decision of a word should also incorporate the potential contributions of other words in the EAE document structure. Thus, inspired by dependency paths as anchors for document structure pruning, we propose to transform the joint consideration of syntactic and semantic distances of words into finding optimal alignments between non-DP and DP words. Optimal alignment will be addressed by an optimal transfer (OT) approach, in which both the syntactic and semantic distances of words from words along dependency paths are modeled in a joint optimization problem. OT is an established mechanism for efficiently Find the optimal transportation plan (ie, alignment) between the two sets of points. We propose in our document-level EAE problem to leverage the semantic similarity of words to obtain their transport costs, while exploiting the syntactic distance to event-triggering words/arguments to compute the word quality distribution for OT. Finally, to prune the document structure, a non-DP word is considered important to the document structure (and thus preserved) if it is aligned with an in-DP word by an OT solution. The pruned document structure will be used to learn representation vectors for input documents to perform argument role prediction using graph convolutional networks (GCNs) (Kipf and Welling, 2017).

Although OT-based pruning methods can help exclude irrelevant words of EAE in the document structure, their noise information may still be encoded in the representation of related words due to contextualization in the input encoder (e.g., BERT). Therefore, to improve representation learning, we propose to explicitly constrain the impact of irrelevant words on representation learning via a novel regularization technique based on the pruned document structure. In particular, we try to add irrelevant words back into the pruned structure (thus restoring the original tree) and ensure that changes in representation vectors due to such additions are minimized. Therefore, in addition to the pruned structure, we apply the GCN model on the original dependency structure to obtain another set of representation vectors for words. Finally, in the final loss function, we introduce the difference between the representation vectors obtained from the pruned structure and the original structure to implement the contribution constraint on irrelevant words. In our experiments, we evaluate our model on both sentence-level and document-level EAE benchmark datasets, demonstrating the effectiveness of the proposed model by establishing new state-of-the-art results in both cases.

2. Model

Problem Definition : The goal of EAE is to identify the effect of entity mentions on specific event trigger words. This task can be defined as a multi-classification problem. Formally speaking, given a document $D=[w_1,w_2,\dots ,w_n]$ , the trigger word$w_t$and candidate argument $w_a$, this goal is to predict labels $L=[l_1,l_2,\dots ,l_m]$ as a candidate argument$w_a$In the trigger word $w_t$induced effect. Label Set $L$ contains a special tagNoneto denote the argument$w_a$and the trigger word $w_t$There is no relationship between them.

Model Overview : The proposed model consists of four main components: 1) Input Encoder, which represents words in documents using high-dimensional vectors; 2) Dependency Pruning, which uses Optimal Transfer (OT) to prune the 3) regularization, which explicitly minimizes the contribution of irrelevant words to representation learning; and 4) prediction, which uses word-induced representations for documents to make final predictions.

2.1 Input Encoder

Use high-dimensional vector $x_i$To represent each word $w_i\in D.$ _ The vector xi is formed by the concatenation of the following vectors: A) Contextualized word embedding: we input text$[CLS]w_1{w}_2\dots w_n\left[SEP\right]$ into the BERTbase model (Devlin et al, 2019); we use$w_i$The hidden states in the last layer serve as contextualized word embeddings. Note that for words consisting of multiple tokens, we take the average of their token representations; and B) distance embedding: we use high-dimensional vectors obtained from the distance embedding table (randomly initialized) to represent the word wi $w_i$to trigger words and argument words (i.e., $|i-t|$ sum$|i-a|$ ​) relative distance. Update the distance embedding table during training. Also, in our experiments, we found that fixing BERT parameters is more helpful. Therefore, for the vector$x_i$Customized as an EAE task, we set the vector $X=[x_1,x_2,\dots ,x_n]$ are fed to a bidirectional long short-term memory network (BiLSTM). Hidden state obtained from BiLSTM,$H=[h_1,h_2,\dots ,h_n]$ , which will be used by subsequent components.

2.2 Dependency pruning

In order to use the input document $D\text{'}$ s syntactic structure, we exploit the dependency tree of the sentences in the document. Here we use an undirected version of the dependency tree generated by the Stanford CoreNLP parser. In order to join the dependency trees of sentences to formDDA single dependency graph for$D$$Each\; pair\; of\; consecutive\; sentences\; in\; D$ adds an edge between the roots of the dependency tree. Therefore, the generated$D\text{'}$ s syntax tree, called$T$ , will contain all words$w_i\in D$。$D\text{'}$ s complete tree$T$ may contain$w_a$Relative to the event trigger word $w_t$Related and irrelevant words predicted by the argument roles of . Therefore, it is necessary to prune the tree to keep only relevant words, thus preventing potential noise introduced by irrelevant words for representation learning. Inspired by the effectiveness of sentence-level EAE-dependent paths in previous work (Li et al, 2013), we use TTEvent trigger wordwt w_t$in\; T$$w_t$and argument candidate $w_a$The dependency path (DP) between them acts as an anchor to prune irrelevant words. In particular, we try to only retrain $Non-DP\; words\; in\; T$ that are syntactically and semantically close to words in DP (i.e., align DP words with non-DP words). We propose to use Optimal Transfer (OT) to jointly consider the syntax and semantics of this word alignment. In the following, we first formally describe OT. Then, we will provide details on how to leverage OT to realize our idea.

OT is an established method for finding the best plan to convert (i.e. transport) one allocation into another. Formally, given the domain $\mathcal{X},$Probability distribution p ( x ) p(x)on$Y$$p\left(x\right)$ and$q\left(y\right)$ , and cost/distance function$C(x,y):\mathcal{X}\times \mathcal{Y}\to {\mathbb{R}}_{+}$For $X$到$The\; mapping\; of\; Y$ , OT, is found with marginal$p\left(x\right)$ andq ( y ) q(y)The optimal joint permutation/distribution of$q$$($$y$$)$$Pi\ast (x,y)$（在$\mathcal{X},Y$ ), that is, from$p\left(x\right)$到$q\left(y\right)$ by default:
$$\begin{gathered} {Pi}^{\ast }(x,y)=\underset{\pi \in \prod (x,y)}{\min }{\int }_y{\int }_{\mathcal{X}}\pi (x,y)C(x,y)dxdy \\ s.t.x\sim p(x),y\sim q\left(y\right) \end{gathered}$$
where,$\prod (x,y)$ is with margin$p\left(x\right)$ andq ( y ) q(y)The set of all joint distributions of$q$$($$y$$)\; .$Note that if the distribution$p\left(x\right)$ and$q\left(y\right)$ is discrete, then the integral in the above equation is replaced by a sum, and the joint distribution${Pi}^{\ast }(x,y)$ is represented by a matrix whose entries$(x,y)(x\in \mathcal{X},y\in Y)$ indicates that the data pointxxconvert$x$$y$ will be distributed$p\left(x\right)$ transforms toq ( y ) q(y)Probability of$q$$($$y$$)\; .$Note that to obtain data points$X,$For hard alignment between$Y$${Pi}^{\ast }(x,y)$ is aligned to the column with the highest probability, i.e.,$y^{\ast }=\arg {\max }_{y\in Y}{Pi}^{\ast }(x,y)$，其中$y^{\ast }$是$Y$ with data point$x\in X-$ aligned data points.

The most important and useful feature of OT in our problem is that it can find the transfer (i.e., alignment) between two sets of data points at the lowest cost according to two criteria: 1) the distance between the data points, and 2) The difference between their probability masses. In particular, these two criteria can be used to capture the required semantic and syntactic similarity in our model to find the coherence between non-DP words and DP words. Specifically, we use words on DP as domain $data\; points\; in\; Y$ , use words outside DP as areaXXThe data points in$X\; will\; be.$To compute$x\in \mathcal{X},y\in$Distribution of$Y$$p\left(x\right)$ and$q\left(y\right)$ (i.e., the probability mass of the data point), we use a grammar-based importance score. Formally, for the word$w_i$, we calculate its distance to the trigger word and candidate argument in the dependency tree (the length of the dependency path), namely $d_i^t$sum $d_i^a$. Then, calculate word $x=w_i\in$The probability mass of$X$$p(x)=min(d_i^t,d_i^a)$ . Note that the distribution$p\left(y\right)$ is computed similarly;$p\left(x\right)$ and$q\left(y\right)$ are also normalized with softmax over their corresponding sets to obtain distributions. In order to get each pair of words$(x,y)\in \mathcal{X}\times$Distance between$Y$$C(x,y)$ , we propose based on their representation vector$h_x$and $h_y$in $H$：$C(x,y)=\parallel h_x-h_y\Vert$ to use their semantic information.

With this setup, solving the above equation returns the best alignment ${Pi}^{\ast }(x,y)$ , this alignment can be used to placeX \mathcal XEach data point in$X$ is related to Y \mathcal YA data point in$Y\; is\; aligned.$However, in our problem we look forX \mathcal XSubset of data points in$X$ and Y \mathcal YThe data points in$Y$$D\text{'}$ s dependency structure is preserved. Therefore, we add an extra data point "NULL" to$Y$ , which is represented by the pair$The\; representation\; of\; all\; data\; points\; in\; X$ is averaged and the probability mass isX \mathcal XAverage of the probability masses of the data points in$X.$withY \mathcal YThe alignment of this data point in$Y$$Aligned\; data\; points\; in\; X$ , i.e. deviated from DP words, should not be preserved in the pruned tree. $Other\; words\; in\; X$ with non-empty alignment, called$I(I\subset X)$ , will remain in$D\text{'}$ s pruned tree. fromTTRemoving the words assigned to DP in$T$NULL will generate a new graph that may contain the most important words in D's argument role prediction, we also keep the trigger words/argument words along with$Any\; word\; in\; the\; dependency\; path\; between\; words\; in\; I$ , resulting in a new graph$T^{\text{'}}$ to denote DDwith important context words$D$。

In the next step, we will $T^{\prime }$ input into a graph convolutional network (GCN) (Kipf and Welling, 2017; Nguyen and Grishman, 2018) to utilizethe HHThe BiLSTM induction vector in$H$$T^{\text{'}}$ More abstract representation vectors for words in . We denote the hidden vector produced in the last layer of the GCN model GCN as:$H^{\prime }=h_{i_1}^{\prime },\dots ,h_{i_m}^{\prime }＝GCN(H，T^{\prime })$, where$m$ is$T^{\prime }$ (m<n) the number of words in, and$h_{i_k}^{\prime }$is the word $w_{i_k}$(i.e. $T^{\text{'}}$ in$k$ words) vector.

2.3 Regularization

By pruning the tree T'T' using$T^{\text{'}}$ to calculate$H^{\prime }$ , we expect to explicitly guide these vectors to: (i) encode relevant/important context words, and (ii) exclude potential noise information from irrelevant words for wa$w_a$role predictions. However, due to the context in the input encoder with BERT, noise information of irrelevant words may still be included in the pruned tree $T^{}$The representation of the selected word in${}^{\text{'}}$$H$ , thus propagated by GCN to represent$H^{\text{'}}$ in. Therefore, to further limit the contribution of irrelevant words to representation learning, we introduce a new regularization technique that encourages learning fromDDThe representation obtained for each word in$D$$T^{\prime }$ (i.e. adding irrelevant words does not change the representation significantly). Since the character prediction will use the output vector of the GCN model, we implement this regularization technique based on the representation vector derived from the GCN. Formally, we first put$D\text{'}$ s$H$ and full dependency tree$T$ is fed into the same GCN model, i.e.$H"=GCN(H,T)$ . We then compute the set H'H'by performing max pooling$H^{\prime }$ (based on$T$ ) and$H"$ (based on$T$ ) vector represents vector$\overline{h^{\prime }}$and $\overline{h^{\prime \prime }}$，即，$\overline{h^{\prime }}=\text{MAX-POOL}(h_{i_1}^{\prime },\dots ,h_{i_m}^{\prime }),\overline{h^{\prime \prime }}=\text{MAX-POOL}(h_{i_1}^{\prime \prime },\dots ,h_{i_m}^{\prime \prime })$ . Finally, we augment h ′ ‾ \overline{h'}by adding their L2 distance to the overall loss function$\overline{h^{\prime }}$and $\overline{h^{\prime \prime }}$similarity: $L_{reg}=\parallel \overline{h^{\prime }}-\overline{h^{\prime \prime }}\parallel$。

2.4 Forecast

for $w_a$and $w_t$For argument role prediction, we form the overall vector $V=[h_t^{\prime },h_a^{\prime },\overline{h^{\prime }}]$ , among which$h_t^{\prime },h_a^{\prime }$is $H^{\text{'}}$中$w_a$and $w_t$The representation vector of . Therefore, $V$ will be used by the two-layer feed-forward network to obtain the distribution Pover possible argument roles$P(\cdot |D,w_t,w_a)$ . To train the model, we use a negative log-likelihood loss:$L_{before\_}=-\log P(l|D,w_t,w_a)$ , where$l$ is the correct label. Therefore, the overall loss function of our model is:$L=L_{before\_}+\beta L_{reg}$, where $\beta$ is a balance parameter.

3. Experiment

Experimental results :

Ablation experiment :

4. Summary

In this work, we propose a new document structure-aware model for document-level EAE. Our model adopts the dependency tree of sentences and proposes a new technique based on optimal transfer to prune the dependency tree of documents in the EAE task. Furthermore, we introduce a novel regularization method to explicitly constrain the contribution of irrelevant words to representation learning. Our extensive experiments demonstrate the effectiveness of the proposed model. In the future, we plan to apply our model to other IE tasks.