GPUs and GPU usage

[greeneggsandyaml]

Hello authors, thank you for your great repo and for ContraGAN!

I had a couple of quick questions:

• How many GPUs do you use to train the pretrained models provided in the README (especially the BigGAN-2048 on ImageNet)?
• I'm finding that GPU utilization is quite low when using multiple GPUs -- do you know why this might be?
• Also, I saw you recently added LARS, but it looks like none of the pretrained models were trained using it. Have you had success speeding up training with LARS?

Thank you so much for your help!

[mingukkang]

Hi.

I am sorry for the late reply.

1. How many GPUs do you use to train the pre-trained models provided in the README (especially the BigGAN-2048 on ImageNet)?
   =>
   Models trained on CIFAR10: 1 GPU (2080Ti, RTX-TITAN, V100, A100, etc.)
   Models trained on Tiny_ImageNet: RTX-TITAN x 1 (From DCGAN to SAGAN), RTX-TITAN x 4 (From BigGAN to ContraGAN +ADA)
   Models trained on ImageNet: V100 32GB x 4 with Sync_BN (From SNGAN to BigGAN 256 B.S.), V100 32GB x 8 with Sync_BN and DP (BigGAN 2048 B.S., training takes almost a month)

2. I'm finding that GPU utilization is quite low when using multiple GPUs
   => yes, it is because you might train the model using DataParallel (DP)
   If you train a model using DistributedDataParallel (DDP), you can make the most of GPUs for training.

3. Also, I saw you recently added LARS, but it looks like none of the pre-trained models were trained using it. Have you had success speeding up training with LARS?
   => No, I didn't try to train GAN models using LARS. I merely added LARS on the codebase since the optimizer is widely used to train big models using a large batch size.

Thank you:)

Best,

Minguk

[greeneggsandyaml]

Hi Minguk,

Thank you for the response! I really appreciate the detailed answers! Also your updates to the repo today were great.

To clarify, what were the exact commands used to train the BigGAN-256 and BigGAN-2048 models? If I'm going to train one of these for 3 or 4 weeks, I want to get the command right :)

For the models you have trained with 4/8 GPUs, did you use DP or DDP? From your previous response I gather that you used DP -- have you had success training models with DDP? Have you had any success with mixed precision training?

Also, when you use standing statistics for evaluation, -std_stat --standing_step STANDING_STEP, what do you use for STANDING_STEP? You don't have to do anything differently to use standing statistics, correct?

Best,
Luke

[mingukkang]

Hi!

Training generative models sometimes requires tremendous patience. Good Luck :)

1. what were the exact commands used to train the BigGAN-256 and BigGAN-2048 models?

Training BigGAN-256
CUDA_VISIBLE_DEVICES=0,1,2,3 python3 src/main.py -c "src/configs/ILSVRC2012/BigGAN256.json" -t -e -l -rm_API -sync_bn --eval_type "valid"

Training BigGAN-2048
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python3 src/main.py -c "src/configs/ILSVRC2012/BigGAN2048.json" -t -e -l -rm_API -sync_bn --eval_type "valid"

2. For the models you have trained with 4/8 GPUs, did you use DP or DDP?
   To see if DDP and Mixed_Precision training work appropriately, I've verified GANS' convergence on CIFAR10 only.

3. What do you use for STANDING_STEP? You don't have to do anything differently to use standing statistics, correct?
   => Actually, I did not use -std_stat and instead used the moving average of the previous statistics.
   (This is the trick I found. You don't have to find a STANDING_STEP).

You can train your model using by "not typing -stat_otf" on the command line.

Please refer to the new updated README for ImageNet experiments.

I highly recommend training your model by comparing it to the log I uploaded.

Thank you!

Sincerely,

MinGuk Kang

[greeneggsandyaml]

Amazing, thank you! I This was a super helpful response.

I am definitely going to use this repo for my future work/research, so I may have more questions as things come up. :)

Thanks again, and all the best with your work!