日期：2019年12月28日

Permutation Language Modeling

一. Introduction

• 自回归语言模型（Autoregressive LM）

Since an AR language model is only trained to encode a uni-directional context (either forward or backward), it is not effective at modeling deep bidirectional contexts

• 自编码语言模型（Autoencoder LM）

aims to reconstruct the original data from corrupted input. (like [MASK] in BERT)

• contributions

二. Proposed Method

1.Background

2. Objective: Permutation Language Modeling

3. Architecture: Two-Stream Self-Attention for Target-Aware Representations

• The content representation hθ(xz≤t ), or abbreviated as hzt , which serves a similar role to the standard hidden states in Transformer. This representation encodes both the context and xzt itself.

• The query representation gθ(xz<t , zt), or abbreviated as gzt , which only has access to the contextual information xz<t and the position zt, but not the content xzt , as discussed above.

4. Incorporating Ideas from Transformer-XL

• relative positional encodings
• recurrence mechanism

5. Modeling Multiple Segments

Relative Segment Encodings

Given a pair of positions i and j in the sequence, if i and j are from the same segment, we use a segment encoding sij = s+ or otherwise sij = s , where s+ and s are learnable model parameters for each attention head. When i attends to j, the segment encoding sij is used to compute an attention weight aij =transpose (qi + b)*sij , where qi is the query vector as in a standard attention operation and b is a learnable head-specific bias vector. Finally, the value aij is added to the normal attention weight.

paper XLNet:运行机制及和Bert的异同比较 slides

简单总结：该文认为大部分语言模型（SKIP-GRAM，BERT，XLNet等）都是求局部信息（masked word或者包含负样本的n-gram ）和全局信息（masked word’s sentence）的Mutual Information Maximization，并提出改进：局部信息为负样本。

一. Introduction

provide an alternative view and show that these methods also maximize a lower bound on the mutual information between different parts of a word sequence

二. Mutual information maximization

1. mutual information

2. InfoNCE

三. Mutual information maximization based on MODELS

1. SKIP-GRAM

2. BERT

3. XLNET

4. INFOWORD

4.1 the mutual information between global representation and local representation

4.2 objective function of InfoWord

四. Experiments

1.result

2. discussion

• Span-based models

JDIM is related to these span-based models such as SpanBERT and MASS

• Mutual information maximization

InfoNCE is widely accepted as a good representation learning objective

• Regularization

Our analysis and the connection we draw to representation learning methods used in other domains provide an insight into possible ways to incorporate prior knowledge into language representation learning models(objective func加入正则化去学习prior knowledge，正则与先验的关系)

paper