week06_da

Lecture slides

More materials

• Russian videos: lecture, seminar
• An overview of proxy-label approaches for semi-supervised learning - blog post
• A Little Review of Domain Adaptation in 2017 - blog post
• Awesome Transfer Learning - list of papers repo
• Awesome Domain Adaptation - list of papers repo

Practice

homework.ipynb

Videos

• The Future of Natural Language Processing by Thomas Wolf [youtube]
• Tutorial on Domain Adaptation by Hal Daumé III [youtube]

Articles

General theory

• Learning Bounds for Domain Adaptation Blitzer et al. 2007 [article]
• A theory of learning from different domains Shai Ben-David, John Blitzer 2010 [article]
• Analysis of Representations for Domain Adaptation Ben-Daivid et al. 2006 [article]

In-domain vs out-domain generalization

• Learning and Evaluating General Linguistic Intelligence Yogatama et al. 2019 [article]
• BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance R. Thomas McCoy, Junghyun Min, Tal Linzen 2019 [article]

Unsupervised pre-training

• Don't Stop Pretraining: Adapt Language Models to Domains and Tasks Gururangan et al. 2020 [article]
• Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling Xiaochuang Han, Jacob Eisenstein 2020 [article]

Instance weighting

• Instance Weighting for Domain Adaptation in NLP Jiang, Zhai 2007 [pdf]
• Instance Weighting for Neural Machine Translation Domain Adaptation Wang et al. 2017 [pdf]
• Cost Weighting for Neural Machine Translation Domain Adaptation Chen et al. 2017 [pdf]

Proxy-label methods

• Asymmetric Tri-training for Unsupervised Domain Adaptation Saito et al. 2017 [arxiv]
• Strong Baselines for Neural Semi-supervised Learning under Domain Shift Sebastian Ruder, Barbara Plank 2018 [arxiv]

Back-translation

• Improving Neural Machine Translation Models with Monolingual Data Sennrich et al. 2016 [arxiv]
• Iterative Back-Translation for Neural Machine Translation Hoang et al. 2018 [pdf]

Knowledge Distillation

• Distilling the Knowledge in a Neural Network Hinton et al. 2015 [arxiv]
• Fine-Tuning for Neural Machine Translation with Limited Degradation across In- and Out-of-Domain Data Dakwale, Monz 2017 [pdf]

Deep Feature Alignment

• Unsupervised Domain Adaptation by Backpropagation Yaroslav Ganin, Victor Lempitsky 2014 [arxiv]
• Learning Transferable Features with Deep Adaptation Networks Long et al. 2015 [arxiv]

Batch Normalization-like:

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey Ioffe, Christian Szegedy, 2015 [arxiv]
• Revisiting Batch Normalization For Practical Domain Adaptation Li et al. 2016 [arxiv]
• AutoDIAL: Automatic DomaIn Alignment Layers Carlucci et al. 2017 [arxiv]