@ybelkada\n\t ,

I see there is an example for fsdp with TPU, can you please provide an example for Nvidia GPU.

Example: https://huggingface.co/google/gemma-7b/blob/main/examples/example_fsdp.py

Also, I tried removing few lines to run in Nvidia GPU,

import torch\n\nfrom datasets import load_dataset\nfrom peft import LoraConfig, get_peft_model\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments\nfrom trl import SFTTrainer\n\nmodel_id = \"google/gemma-7b\"\n\n# Load the pretrained model and tokenizer.\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map={\"\":0})\n\n# Set up PEFT LoRA for fine-tuning.\nlora_config = LoraConfig(\n    r=8,\n    target_modules=[\"k_proj\", \"v_proj\"],\n    task_type=\"CAUSAL_LM\",\n)\n\n# Load the dataset and format it for training.\ndata = load_dataset(\"Abirate/english_quotes\", split=\"train\")\nmax_seq_length = 1024\n\n# Set up the FSDP config. To enable FSDP via SPMD, set xla_fsdp_v2 to True.\nfsdp_config = {\"fsdp_transformer_layer_cls_to_wrap\": [\n        \"GemmaDecoderLayer\"\n    ],\n    \"xla\": True,\n    \"xla_fsdp_v2\": True,\n    \"xla_fsdp_grad_ckpt\": True}\n\n# Finally, set up the trainer and train the model.\ntrainer = SFTTrainer(\n    model=model,\n    train_dataset=data,\n    args=TrainingArguments(\n        per_device_train_batch_size=64,  # This is actually the global batch size for SPMD.\n        num_train_epochs=100,\n        max_steps=-1,\n        output_dir=\"./output\",\n        optim=\"adafactor\",\n        logging_steps=1,\n        dataloader_drop_last = True,  # Required for SPMD.\n        fsdp=\"full_shard\",\n        fsdp_config=fsdp_config,\n    ),\n    peft_config=lora_config,\n    dataset_text_field=\"quote\",\n    max_seq_length=max_seq_length,\n    packing=True,\n)\n\ntrainer.train()\n

but I get the following error ValueError: Using fsdp only works in distributed training.

can you please provide your input here?

\n","updatedAt":"2024-04-04T21:37:45.027Z","author":{"_id":"644f1d03030210812f469d17","avatarUrl":"/avatars/9c8ef28358d8ff81e6bc1dc4558de43d.svg","fullname":"IamexperimentingNow","name":"Iamexperimenting","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5062827467918396},"editors":["Iamexperimenting"],"editorAvatarUrls":["/avatars/9c8ef28358d8ff81e6bc1dc4558de43d.svg"],"reactions":[],"isReport":false}},{"id":"66165d1451c9b8e0e02b6226","author":{"_id":"62441d1d9fdefb55a0b7d12c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1648631057413-noauth.png","fullname":"Younes B","name":"ybelkada","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":502,"isOwner":false,"isOrgMember":false},"createdAt":"2024-04-10T09:34:12.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Sure @Iamexperimenting thanks!\nI advise you to read this article from the PEFT official documentation: https://huggingface.co/docs/peft/accelerate/fsdp and simply change the model id with gemma, you can also get started with the official scripts which are located here: https://github.com/huggingface/peft/tree/main/examples/sft","html":"

Sure \n\n@Iamexperimenting\n\t thanks!
I advise you to read this article from the PEFT official documentation: https://huggingface.co/docs/peft/accelerate/fsdp and simply change the model id with gemma, you can also get started with the official scripts which are located here: https://github.com/huggingface/peft/tree/main/examples/sft

\n","updatedAt":"2024-04-10T09:34:12.415Z","author":{"_id":"62441d1d9fdefb55a0b7d12c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1648631057413-noauth.png","fullname":"Younes B","name":"ybelkada","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":502}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9298200607299805},"editors":["ybelkada"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1648631057413-noauth.png"],"reactions":[],"isReport":false}}],"pinned":false,"locked":false,"collection":"discussions","isPullRequest":false,"isReport":false},"repo":{"name":"google/gemma-7b","type":"model"},"activeTab":"discussion","discussionRole":0,"watched":false,"muted":false,"repoDiscussionsLocked":false}">

FSDP with Nvidia GPU

#84

by Iamexperimenting - opened Apr 4, 2024

Discussion

@ybelkada\n\t ,

I see there is an example for fsdp with TPU, can you please provide an example for Nvidia GPU.

Example: https://huggingface.co/google/gemma-7b/blob/main/examples/example_fsdp.py

Also, I tried removing few lines to run in Nvidia GPU,

import torch\n\nfrom datasets import load_dataset\nfrom peft import LoraConfig, get_peft_model\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments\nfrom trl import SFTTrainer\n\nmodel_id = \"google/gemma-7b\"\n\n# Load the pretrained model and tokenizer.\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nmodel = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map={\"\":0})\n\n# Set up PEFT LoRA for fine-tuning.\nlora_config = LoraConfig(\n    r=8,\n    target_modules=[\"k_proj\", \"v_proj\"],\n    task_type=\"CAUSAL_LM\",\n)\n\n# Load the dataset and format it for training.\ndata = load_dataset(\"Abirate/english_quotes\", split=\"train\")\nmax_seq_length = 1024\n\n# Set up the FSDP config. To enable FSDP via SPMD, set xla_fsdp_v2 to True.\nfsdp_config = {\"fsdp_transformer_layer_cls_to_wrap\": [\n        \"GemmaDecoderLayer\"\n    ],\n    \"xla\": True,\n    \"xla_fsdp_v2\": True,\n    \"xla_fsdp_grad_ckpt\": True}\n\n# Finally, set up the trainer and train the model.\ntrainer = SFTTrainer(\n    model=model,\n    train_dataset=data,\n    args=TrainingArguments(\n        per_device_train_batch_size=64,  # This is actually the global batch size for SPMD.\n        num_train_epochs=100,\n        max_steps=-1,\n        output_dir=\"./output\",\n        optim=\"adafactor\",\n        logging_steps=1,\n        dataloader_drop_last = True,  # Required for SPMD.\n        fsdp=\"full_shard\",\n        fsdp_config=fsdp_config,\n    ),\n    peft_config=lora_config,\n    dataset_text_field=\"quote\",\n    max_seq_length=max_seq_length,\n    packing=True,\n)\n\ntrainer.train()\n

but I get the following error ValueError: Using fsdp only works in distributed training.

can you please provide your input here?

\n","updatedAt":"2024-04-10T09:34:12.415Z","author":{"_id":"62441d1d9fdefb55a0b7d12c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1648631057413-noauth.png","fullname":"Younes B","name":"ybelkada","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":502}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9298200607299805},"editors":["ybelkada"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1648631057413-noauth.png"],"reactions":[],"isReport":false}}],"pinned":false,"locked":false,"collection":"discussions","isPullRequest":false,"isReport":false},"primaryEmailConfirmed":false,"repo":{"name":"google/gemma-7b","type":"model"},"discussionRole":0,"acceptLanguages":["*"],"hideComments":true,"repoDiscussionsLocked":false,"isDiscussionAuthor":false}">

Iamexperimenting

Apr 4, 2024

Hi @ybelkada ,

I see there is an example for fsdp with TPU, can you please provide an example for Nvidia GPU.

Example: https://huggingface.co/google/gemma-7b/blob/main/examples/example_fsdp.py

Also, I tried removing few lines to run in Nvidia GPU,

import torch

from datasets import load_dataset
from peft import LoraConfig, get_peft_model
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments
from trl import SFTTrainer

model_id = "google/gemma-7b"

# Load the pretrained model and tokenizer.
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map={"":0})

# Set up PEFT LoRA for fine-tuning.
lora_config = LoraConfig(
    r=8,
    target_modules=["k_proj", "v_proj"],
    task_type="CAUSAL_LM",
)

# Load the dataset and format it for training.
data = load_dataset("Abirate/english_quotes", split="train")
max_seq_length = 1024

# Set up the FSDP config. To enable FSDP via SPMD, set xla_fsdp_v2 to True.
fsdp_config = {"fsdp_transformer_layer_cls_to_wrap": [
        "GemmaDecoderLayer"
    ],
    "xla": True,
    "xla_fsdp_v2": True,
    "xla_fsdp_grad_ckpt": True}

# Finally, set up the trainer and train the model.
trainer = SFTTrainer(
    model=model,
    train_dataset=data,
    args=TrainingArguments(
        per_device_train_batch_size=64,  # This is actually the global batch size for SPMD.
        num_train_epochs=100,
        max_steps=-1,
        output_dir="./output",
        optim="adafactor",
        logging_steps=1,
        dataloader_drop_last = True,  # Required for SPMD.
        fsdp="full_shard",
        fsdp_config=fsdp_config,
    ),
    peft_config=lora_config,
    dataset_text_field="quote",
    max_seq_length=max_seq_length,
    packing=True,
)

trainer.train()

but I get the following error ValueError: Using fsdp only works in distributed training.

can you please provide your input here?

ybelkada

Apr 10, 2024

Sure @Iamexperimenting thanks!
I advise you to read this article from the PEFT official documentation: https://huggingface.co/docs/peft/accelerate/fsdp and simply change the model id with gemma, you can also get started with the official scripts which are located here: https://github.com/huggingface/peft/tree/main/examples/sft

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment