分类：NLP
【EMNLP-2016】Sequence-Level Knowledge Distillation
【ICLR-2016】Unifying distillation and privileged information
【CoRR-2019】Making Neural Machine Reading Comprehension Faster
【CoRR-2015】Distilling the Knowledge in a Neural Network
Advantage of soft target：
When the soft targets have high entropy, they provide much more information per training case than hard targets and much less variance in the gradient between training cases, so the small model can often be trained on much less data than the original cumbersome model and using a much higher learning rate.
