exclude_from_weight_decay for AdamW

[markub3327]

Hello,

please can you add an exclude_from_weight_decay parameter to AdamW? Some parameters during weight decay must be excluded.

For more details please look here.

Thanks a lot.
Have a nice day.

[chunduriv]

@markub3327,

Can you please elaborate about your feature and please specify the use cases for this feature. Thanks!

[markub3327]

I need to exclude some parameters from the weight decay process. Useful during training the Transformer model. List of regex patterns of variables excluded from weight decay. Variables whose names contain a substring matching the pattern will be excluded.

Here is the use case.

The code behind exclude_from_weight_decay from tensorflow/addons is here.

[jvishnuvardhan]

@markub3327 What is the difference between this issue when compared to #16195?

Can you please state the differences clearly or close this issue this is a duplicate of #16195. Thanks!

[bhack]

This is a proposal to add exclude_from_weight_decay in AdamW that was added in Keras 30 days ago (it was already in TFA).

#16195 is related to port RectifiedAdam (a new optimizer) with exclude_from_weight_decay in Keras from TFA

[markub3327]

@bhack

Thanks a lot ... Very good explanation!

[haifeng-jin]

@chenmoneygithub Any comments on this?

[chenmoneygithub]

@markub3327 Thanks for reporting the issue! Are you willing to open a PR to add exclude_from_weight_decay to AdamW? Thanks!

[markub3327]

Yes. Thanks.