The official implementation of [ACMMM2022] Pay Attention to Your Positive Pairs: Positive Pair Aware Contrastive Knowledge Distillation pdf
| Teacher Student |
ResNet56 ResNet20 |
ResNet110 ResNet32 |
ResNet32x4 ResNet8x4 |
WRN-40-2 WRN-16-2 |
WRN-40-2 WRN-40-1 |
VGG13 VGG8 |
|---|---|---|---|---|---|---|
| Teacher | 72.75 | 74.07 | 79.55 | 76.00 | 76.00 | 74.81 |
| Student | 68.93 | 71.45 | 72.22 | 73.03 | 71.32 | 70.68 |
| KD | 71.12 | 73.08 | 73.53 | 75.47 | 73.66 | 73.14 |
| PACKD | 72.59 | 75.31 | 77.09 | 77.32 | 77.15 | 75.94 |
| Teacher Student |
ResNet32x4 ShuffleNet-V1 |
WRN-40-2 ShuffleNet-V1 |
ResNet32x4 ShuffleNet-V2 |
ResNet50 MobileNet-V2 |
ResNet50 vgg8 |
|---|---|---|---|---|---|
| Teacher | 79.55 | 76.00 | 79.55 | 78.87 | 78.87 |
| student | 71.60 | 71.60 | 72.90 | 64.39 | 70.68 |
| KD | 74.20 | 75.69 | 74.94 | 67.83 | 73.33 |
| PACKD | 79.02 | 79.19 | 79.63 | 72.83 | 76.71 |
-
Download the
cifar_teachers.tarat https://github.com/smuelpeng/PACKD/releases/tag/checkpoints and untar it to./save_tviatar xf save_t.tar.# Training basleine. sh scripts/run_vanilla.sh # Distillation method. sh scripts/run_distill.sh
PACKD released under the MIT license. See LICENSE for details.
- Thanks for CRD and ReviewKD. We build this library based on the CRD's codebase