nccl_max_nchannels dynamically set in the program

[Rainlin007]

I want to dynamically set nccl_max_nchannels in the program. hope allreduce uses different SM numbers under different conditions, is there any way to achieve this?

[gcongiu]

You can use ncclCommInitRankConfig and set maxCTAs in ncclConfig_t. This will allow you to set the number of max channels (CTAs) per communicator.

[sjeaugey]

Note this property is per-communicator. If you want different allreduce operations to have different maxCTAs, then you'll need to create multiple communicators.

You can, however, use ncclCommSplit with config.splitShare=1 to create a new identical communicator with a different configuration but without using extra resources.