@Narsil\n\t ?\n","updatedAt":"2023-10-02T07:12:27.800Z","author":{"_id":"5e3aec01f55e2b62848a5217","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5e3aec01f55e2b62848a5217/PMKS0NNB4MJQlTSFzh918.jpeg","fullname":"Lysandre","name":"lysandre","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":582}},"numEdits":0,"identifiedLanguage":{"language":"fr","probability":0.24416381120681763},"editors":["lysandre"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/5e3aec01f55e2b62848a5217/PMKS0NNB4MJQlTSFzh918.jpeg"],"reactions":[],"isReport":false}},{"id":"651a72842d69a6fd7e39193a","author":{"_id":"65002a7c2ad36636be85d7d8","avatarUrl":"/avatars/8d8ab3bf5eb8e10fbbabff732a354dfb.svg","fullname":"Nolwenn O","name":"NolwennO","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isOwner":false,"isOrgMember":false},"createdAt":"2023-10-02T07:34:28.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"FYI, here is the related issue description https://github.com/huggingface/text-generation-inference/issues/541#issuecomment-1740913948\n","html":"
FYI, here is the related issue description https://github.com/huggingface/text-generation-inference/issues/541#issuecomment-1740913948
\n","updatedAt":"2023-10-02T07:34:28.365Z","author":{"_id":"65002a7c2ad36636be85d7d8","avatarUrl":"/avatars/8d8ab3bf5eb8e10fbbabff732a354dfb.svg","fullname":"Nolwenn O","name":"NolwennO","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7217327356338501},"editors":["NolwennO"],"editorAvatarUrls":["/avatars/8d8ab3bf5eb8e10fbbabff732a354dfb.svg"],"reactions":[{"reaction":"👍","users":["Cyrile"],"count":1}],"isReport":false}},{"id":"651bd7b78010d2458e1de040","author":{"_id":"5e2967b819407e3277369b95","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1608285816082-5e2967b819407e3277369b95.png","fullname":"Nicolas Patry","name":"Narsil","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":247,"isOwner":false,"isOrgMember":true},"createdAt":"2023-10-03T08:58:31.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Well those foundation model work.\n\nIf loading the model and saving it back in transformers changes it that's an issue IMO.\n\nWe can make something for TGI but this feels like legacy support, would you agree ?","html":"Well those foundation model work.
\nIf loading the model and saving it back in transformers changes it that's an issue IMO.
\nWe can make something for TGI but this feels like legacy support, would you agree ?
\n","updatedAt":"2023-10-03T08:58:31.205Z","author":{"_id":"5e2967b819407e3277369b95","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1608285816082-5e2967b819407e3277369b95.png","fullname":"Nicolas Patry","name":"Narsil","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":247}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9453772306442261},"editors":["Narsil"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1608285816082-5e2967b819407e3277369b95.png"],"reactions":[],"isReport":false}},{"id":"651bda5b650b79ca1360c61a","author":{"_id":"5e2967b819407e3277369b95","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1608285816082-5e2967b819407e3277369b95.png","fullname":"Nicolas Patry","name":"Narsil","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":247,"isOwner":false,"isOrgMember":true},"createdAt":"2023-10-03T09:09:47.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This should fix it: https://github.com/huggingface/text-generation-inference/pull/1090","html":"This should fix it: https://github.com/huggingface/text-generation-inference/pull/1090
\n","updatedAt":"2023-10-03T09:09:47.365Z","author":{"_id":"5e2967b819407e3277369b95","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1608285816082-5e2967b819407e3277369b95.png","fullname":"Nicolas Patry","name":"Narsil","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":247}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.6885347366333008},"editors":["Narsil"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1608285816082-5e2967b819407e3277369b95.png"],"reactions":[],"isReport":false}},{"id":"651d6c7ca9afacbd050d139c","author":{"_id":"6169f0f969935bfeaec043c9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6169f0f969935bfeaec043c9/irX3qrIK-573iqgSAIc1N.png","fullname":"Cyrile Delestre","name":"Cyrile","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":9,"isOwner":false,"isOrgMember":false},"createdAt":"2023-10-04T13:45:32.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"Hi Narsil,\nI agree that the issue seems to be more related to the naming of modules in the foundation models rather than a TGI problem. What I find strange is that the planned prefix in the code is \"transformer.[PyTorch module name],\" but in the foundation model, this prefix is absent.\nIf I refer to the BERT model, for example, there is the prefix \"bert.[etc]\" on the module names, as stipulated in the code: base_model_prefix = \"bert\".","html":"Hi Narsil,
I agree that the issue seems to be more related to the naming of modules in the foundation models rather than a TGI problem. What I find strange is that the planned prefix in the code is \"transformer.[PyTorch module name],\" but in the foundation model, this prefix is absent.
If I refer to the BERT model, for example, there is the prefix \"bert.[etc]\" on the module names, as stipulated in the code: base_model_prefix = \"bert\".
Indeed, allowing flexibility in TGI to let the user define the prefix would be a more robust solution.
\n","updatedAt":"2023-10-04T14:02:09.327Z","author":{"_id":"6169f0f969935bfeaec043c9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6169f0f969935bfeaec043c9/irX3qrIK-573iqgSAIc1N.png","fullname":"Cyrile Delestre","name":"Cyrile","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":9}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9086240530014038},"editors":["Cyrile"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6169f0f969935bfeaec043c9/irX3qrIK-573iqgSAIc1N.png"],"reactions":[],"isReport":false}}],"pinned":false,"locked":false,"collection":"discussions","isPullRequest":false,"isReport":false},"repo":{"name":"bigscience/bloom","type":"model"},"activeTab":"discussion","discussionRole":0,"watched":false,"muted":false,"repoDiscussionsLocked":false}">base_model_prefix = "transformer"
#265
by
Cyrile
- opened
@Narsil\n\t ?\n","updatedAt":"2023-10-02T07:12:27.800Z","author":{"_id":"5e3aec01f55e2b62848a5217","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5e3aec01f55e2b62848a5217/PMKS0NNB4MJQlTSFzh918.jpeg","fullname":"Lysandre","name":"lysandre","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":582}},"numEdits":0,"identifiedLanguage":{"language":"fr","probability":0.24416381120681763},"editors":["lysandre"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/5e3aec01f55e2b62848a5217/PMKS0NNB4MJQlTSFzh918.jpeg"],"reactions":[],"isReport":false}},{"id":"651a72842d69a6fd7e39193a","author":{"_id":"65002a7c2ad36636be85d7d8","avatarUrl":"/avatars/8d8ab3bf5eb8e10fbbabff732a354dfb.svg","fullname":"Nolwenn O","name":"NolwennO","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isOwner":false,"isOrgMember":false},"createdAt":"2023-10-02T07:34:28.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"FYI, here is the related issue description https://github.com/huggingface/text-generation-inference/issues/541#issuecomment-1740913948\n","html":"
FYI, here is the related issue description https://github.com/huggingface/text-generation-inference/issues/541#issuecomment-1740913948
\n","updatedAt":"2023-10-02T07:34:28.365Z","author":{"_id":"65002a7c2ad36636be85d7d8","avatarUrl":"/avatars/8d8ab3bf5eb8e10fbbabff732a354dfb.svg","fullname":"Nolwenn O","name":"NolwennO","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7217327356338501},"editors":["NolwennO"],"editorAvatarUrls":["/avatars/8d8ab3bf5eb8e10fbbabff732a354dfb.svg"],"reactions":[{"reaction":"👍","users":["Cyrile"],"count":1}],"isReport":false}},{"id":"651bd7b78010d2458e1de040","author":{"_id":"5e2967b819407e3277369b95","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1608285816082-5e2967b819407e3277369b95.png","fullname":"Nicolas Patry","name":"Narsil","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":247,"isOwner":false,"isOrgMember":true},"createdAt":"2023-10-03T08:58:31.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Well those foundation model work.\n\nIf loading the model and saving it back in transformers changes it that's an issue IMO.\n\nWe can make something for TGI but this feels like legacy support, would you agree ?","html":"Well those foundation model work.
\nIf loading the model and saving it back in transformers changes it that's an issue IMO.
\nWe can make something for TGI but this feels like legacy support, would you agree ?
\n","updatedAt":"2023-10-03T08:58:31.205Z","author":{"_id":"5e2967b819407e3277369b95","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1608285816082-5e2967b819407e3277369b95.png","fullname":"Nicolas Patry","name":"Narsil","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":247}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9453772306442261},"editors":["Narsil"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1608285816082-5e2967b819407e3277369b95.png"],"reactions":[],"isReport":false}},{"id":"651bda5b650b79ca1360c61a","author":{"_id":"5e2967b819407e3277369b95","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1608285816082-5e2967b819407e3277369b95.png","fullname":"Nicolas Patry","name":"Narsil","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":247,"isOwner":false,"isOrgMember":true},"createdAt":"2023-10-03T09:09:47.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This should fix it: https://github.com/huggingface/text-generation-inference/pull/1090","html":"This should fix it: https://github.com/huggingface/text-generation-inference/pull/1090
\n","updatedAt":"2023-10-03T09:09:47.365Z","author":{"_id":"5e2967b819407e3277369b95","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1608285816082-5e2967b819407e3277369b95.png","fullname":"Nicolas Patry","name":"Narsil","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":247}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.6885347366333008},"editors":["Narsil"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1608285816082-5e2967b819407e3277369b95.png"],"reactions":[],"isReport":false}},{"id":"651d6c7ca9afacbd050d139c","author":{"_id":"6169f0f969935bfeaec043c9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6169f0f969935bfeaec043c9/irX3qrIK-573iqgSAIc1N.png","fullname":"Cyrile Delestre","name":"Cyrile","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":9,"isOwner":false,"isOrgMember":false},"createdAt":"2023-10-04T13:45:32.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"Hi Narsil,\nI agree that the issue seems to be more related to the naming of modules in the foundation models rather than a TGI problem. What I find strange is that the planned prefix in the code is \"transformer.[PyTorch module name],\" but in the foundation model, this prefix is absent.\nIf I refer to the BERT model, for example, there is the prefix \"bert.[etc]\" on the module names, as stipulated in the code: base_model_prefix = \"bert\".","html":"Hi Narsil,
I agree that the issue seems to be more related to the naming of modules in the foundation models rather than a TGI problem. What I find strange is that the planned prefix in the code is \"transformer.[PyTorch module name],\" but in the foundation model, this prefix is absent.
If I refer to the BERT model, for example, there is the prefix \"bert.[etc]\" on the module names, as stipulated in the code: base_model_prefix = \"bert\".
Indeed, allowing flexibility in TGI to let the user define the prefix would be a more robust solution.
\n","updatedAt":"2023-10-04T14:02:09.327Z","author":{"_id":"6169f0f969935bfeaec043c9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6169f0f969935bfeaec043c9/irX3qrIK-573iqgSAIc1N.png","fullname":"Cyrile Delestre","name":"Cyrile","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":9}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9086240530014038},"editors":["Cyrile"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6169f0f969935bfeaec043c9/irX3qrIK-573iqgSAIc1N.png"],"reactions":[],"isReport":false}}],"pinned":false,"locked":false,"collection":"discussions","isPullRequest":false,"isReport":false},"primaryEmailConfirmed":false,"repo":{"name":"bigscience/bloom","type":"model"},"discussionRole":0,"acceptLanguages":["*"],"hideComments":true,"repoDiscussionsLocked":false,"isDiscussionAuthor":false}">Hello, why doesn't the nomenclature of the modules in the Bloom and Bloomz models adhere to those created by the BloomPreTrainedModel class: base_model_prefix = "transformer"?
The issue is that in TGI, which has adapted to Bloom modeling, models trained by Transformers do not work because the TGI library looks for model names without the "transformer" prefix.
FYI, here is the related issue description https://github.com/huggingface/text-generation-inference/issues/541#issuecomment-1740913948
Well those foundation model work.
If loading the model and saving it back in transformers changes it that's an issue IMO.
We can make something for TGI but this feels like legacy support, would you agree ?
Hi Narsil,
I agree that the issue seems to be more related to the naming of modules in the foundation models rather than a TGI problem. What I find strange is that the planned prefix in the code is "transformer.[PyTorch module name]," but in the foundation model, this prefix is absent.
If I refer to the BERT model, for example, there is the prefix "bert.[etc]" on the module names, as stipulated in the code: base_model_prefix = "bert".
Indeed, allowing flexibility in TGI to let the user define the prefix would be a more robust solution.