The expert-selecting Gating strategy of MOE.

[Leopold2333]

Hi. I'm reading the code snippets of Moirai-MOE and got some problems.
I see that each MOEFeedforward layer would use centroid to calculate the Euclidean distance to each "cluster".

uni2ts/src/uni2ts/module/ffn.py

Line 138 in 10c1b4c

cdist = torch.cdist(x_temp, centroid)

However, I'm confused how these clusters (or, centroid I guess) are generated. In the module TransformerEncoder, the centroid is created using Pytorch register_buffer function. So the centroid here would be a zero tensor all the time? If so, why does it work as the "clusters"? The snippets here seems different from that described in the paper.

uni2ts/src/uni2ts/module/transformer.py

Lines 182 to 184 in 10c1b4c

	self.register_buffer(
	"centroid", torch.empty(num_layers, 32, d_model, dtype=torch.float64)
	)