Paper Today:
'Incorporating Copying Mechanism in Sequence-to-Sequence Learning'
This paper develops a model called COPYNET which performs well in an important mechanism called 'copy mechanism'.
In human language communication, there are many situations that we will use 'copy mechanism', such as in a dialogue:
In order to make machine generate such dialogue, there are two things to do.
- First, to identify what should be copied.
- Second, to decide where the copy part should be addressed.
Currently there are some popular models like seq2seq, and adding Attention Mechanism to seq2seq.
COPYNET is also an encoder-decoder model, but a different strategy in neural network based models.
RNN and Attention Mechanism requires more 'understanding', but COPYNET requires high 'literal fidelity'.
There are mainly 3 improvements in the decoder part.
Prediction:
Based on the mix of two probabilistic modes, generate mode and copy mode, the model can pick the proper subsentence and generate some OOV words.
State Update:
There's a minor change that they designed a selective read for copy mode, which enables the model to notice the location information.
Reading M:
This model can get a hybrid of content based addressing and location based addressing.
In the experiment, this model did very well in tasks like text summarization.
