When using transformer to load pre-trained T5 encoder parameters, it often appears:
Some weights of T5EncoderModel were not initialized from the model checkpoint
at t5-large and are newly initialized: ['encoder.embed_tokens.weight']
This ['encoder.embed_tokens.weight']parameter, as the name suggests, is the embedding weight of the encoder.
Many people worry about whether the random initialization of embedding will affect the performance of the model when they see this warning.
After many investigations by the author, this ['encoder.embed_tokens.weight']initialization has no impact on the performance of the model and can be ignored.
If you want to avoid this error, upgrade transformer to the latest version.
For details, see: Warning when loading T5 encoders