其他分享
首页 > 其他分享> > BERT论文阅读(二): CG-BERT:Conditional Text Generation with BERT for Generalized Few-shot Intent Detection

BERT论文阅读(二): CG-BERT:Conditional Text Generation with BERT for Generalized Few-shot Intent Detection

作者:互联网

目录

The proposed method

Input Representation

The Encoder 

 The Decoder

 fine-tuning


discriminate a joint label space consisting of both existing intent which have enough labeled data and novel intents which only have a few examples for each class.

==> Conditional Text Generation with BERT

The proposed method

CG-BERT: adopts the CVAE(Condiional  Variational AutoEncoder) framework and incorporates BERT into both the encoder and the decoder.

采用条件变分自编码器,并将BERT融入到encoder-decoder中

编码器:同时将话语x和意图y编码为一个潜变量z,并且模拟z的后验概率分布p(z|x,y),y是CVAE模型中的条件。 ==> encoder模拟few-shot intent的数据概率分布

解码器:同时解码变量z和意图y,以便重构输入话语x ==> 利用masked attention的特性限制attend,以保持文本生成这种特定任务left-to-right的特性,保留其autoregressive特性!

为新意图y生成新话语,我们通过从一个先验分布p(z|y)采样潜变量z,并且用解码器解码变量z和y to 新话语。

It's able to generate more utterances for the novel intent through sampling from the learned distribution.

通过从学到的概率分布中采样,为新意图生成更多的话语

Input Representation

input: intent + utterance text sentences (concatenated)

句子S1: CLS token + intent y + SEP token --> first intent sentence

句子S2: utterance x + SEP --> second utterance sentence

whole input: S1 + S2

CLS: as the representation for the whole input

variable z: encode the embeddings for [CLS] to the latent variable z

Text are tokenized into subword units by WordPiece

embedding: obtained for each token --> token embeddings, position embeddings, segment embeddings

a given token: constructed by summing these three embeddings and represented as H^{0} = [h_{1}^{0}, h_{2}^{0}, ....,h_{T}^{0}]  with a total length of T tokens.

The Encoder 

models the distribution of diverse utterances for a given intent.

对给定intent,即few-shot intent,的不同话语分布进行建模

to obtain deep bidirectional context information <-- models the attention between the intent tokens and the utterance tokens

为获得深度双向上下文信息 <-- 利用意图令牌和话语令牌之间的attention进行建模

the input representation: H^{0} = [h_{1}^{0}, h_{2}^{0}, ....,h_{T}^{0}] 

multiple self-attention heads: 

output of the previous layer H^{l-1} --> a triple of queries, keys and values

embeddings for the [CLS] token in the 6-th transformer block h_{1}^{6} --> sentence-level representation

sentence-level representation h_{1}^{6}  --> a latent variable z = a latent vector z, where prior distribution p(z|y) is a multivariate standard Gaussian distribution.

 u and \sigma in the Gaussian distribution q(z|x,y) = N(u, \sigma) --> to sample z

 The Decoder

 aims to reconstruct the input utterance x using the latent variable z and the intent y.

目的是用潜变量z和意图y重构输入话语x

residual connection from input representation H0 --> decoder H6'残差连接z和H0

==> input of the decoder  H_{6}^{'} = [z, h_{2}^{0}, ....,h_{T}^{0}]

left-to-right manner ==> 掩码masked attention

the attention mask --> helps the transformer blocks fit into the conditional text generation task. 

attention掩码 --> 帮助transformer块适应有条件文本生成任务

not whole bidirectional attention to the input ==> instead a mask matrix to determine whether a pair of tokens can be attended to each other.

并不是全部双向attention的输入 ==> 而是用一个掩码矩阵去决定一对令牌是否要相互关注

updated Attention:

 

 output of 12-th transformer block in decoder H^{12} = [h_{1}^{12}, h_{2}^{12}, ....,h_{T}^{12}]h_{1}^{12} is the embeddings for the latent variable z

To further increase the impact of z and alleviate the vanishing latent variable problem,

embeddings of z with all the tokens H^{12'} = [h_{1}^{12}||h_{1}^{12}, h_{2}^{12}|| ....||h_{1}^{12}, h_{2}^{12},...,h_{T}^{12}||h_{1}^{12}]

Two fully-connected layers with a layer normalization to get the final representation

H^{f} = g(f(f(H^{12'}W_{1} + b1)W_{2} + b_{2}))

to predict the next token at position t+1 <-- the embeddings in Hf at position t

p(X_{t+1}) = f(H_{t}^{f}W_{e}^{T} + be)

 fine-tuning

in order to improve the performance in the few-shot intent of model learned from existing intents with enough labeled data.

reference: Cross-Lingual Natural Language Generation via Pre-training

标签:BERT,shot,--,Conditional,intent,embeddings,variable,input,latent
来源: https://blog.csdn.net/qq_33419476/article/details/118752179