\n

Now create a batch of examples using DataCollatorForLanguageModeling. It’s more efficient to dynamically pad the sentences to the longest length in a batch during collation, instead of padding the whole dataset to the maximum length. Use the end-of-sequence token as the padding token and set mlm=False. This will use the inputs as labels shifted to the right by one element:

\n

from transformers import DataCollatorForLanguageModeling\n\ntokenizer.pad_token = tokenizer.eos_token\ndata_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)\n

\n

However, if I use DataCollatorForLanguageModeling, I get the error:

ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`labels` in this case) have excessive nesting (inputs type `list` where type `int` is expected).\n

Environment

!cat /etc/os-release\nPRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"\n\n!transformers-cli env\n- `transformers` version: 4.28.0\n- Platform: Linux-4.14.309-231.529.amzn2.x86_64-x86_64-with-debian-10.6\n- Python version: 3.7.10\n- Huggingface_hub version: 0.13.4\n- Safetensors version: not installed\n- PyTorch version (GPU?): 1.13.1+cu117 (True)\n- Tensorflow version (GPU?): not installed (NA)\n- Flax version (CPU?/GPU?/TPU?): not installed (NA)\n- Jax version: not installed\n- JaxLib version: not installed\n- Using GPU in script?: YES\n- Using distributed or parallel set-up in script?: <fill in>\n

Code for Tokenization

DATASET_STREAMING: bool = False\ntrain = load_dataset(\"xsum\", split=\"train\", streaming=DATASET_STREAMING)\n\n# --------------------------------------------------------------------------------\n# Function to generate prompt from XSUM dataset\n# --------------------------------------------------------------------------------\ndef get_convert_to_prompt(template: Template) -> Callable:\n    def _convert_to_prompt(example: Dict[str, str]) -> Dict[str, str]:\n        \"\"\"Generate prompt as a dictionary:\n        {\n            \"prompt\": \"Summarize: <document>\\n<summary>\"\n        }\n\n        Args:\n            example: single {document, summary} pair to be able to apply template\n        Returns: a dictionary of prompt\n        \"\"\"\n        # assert isinstance(example, dict), f\"expected dict but {type(example)}.\\n{example}\"\n        assert isinstance(example['document'], str), f\"expected str but {type(example['document'])}.\"\n\n        prompt, response = template.apply(example=example, truncate=False)\n        return {\n            \"prompt\": \" \".join(\n                re.sub(r'[\\s\\'\\\"]+', ' ', prompt).split(' ')[:MAX_REQUEST_LENGTH-1]  # -1 for \\n\n            ) + \"\\n\" + \" \".join(\n                re.sub(r'[\\s\\'\\\"]+', ' ', response).split(' ')[:MAX_RESPONSE_LENGTH-1]\n            ) + \"\\n\"\n        }\n\n    return _convert_to_prompt\n\nconvert_to_prompt: Callable = get_convert_to_prompt(template=template)\n\n# --------------------------------------------------------------------------------\n# Function to tokenize prompt\n# --------------------------------------------------------------------------------\ndef tokenize_prompt(example):\n    \"\"\"Generate the model inputs in the dictionary with format:\n    {\n        \"input_ids\": List[int], \n        \"attention_mask\": List[int]\",\n        \"labels\": List[int]\n    }\n    \n    Args:\n        example:   a dictionary of format {\n            \"prompt\": \"Summarize:<document>\\n<summary>\\n\",\n        }\n    \"\"\"    \n    assert isinstance(example['prompt'], str), f\"expected str, got {type(example['prompt'])}\"\n    inputs: Dict[str, List[int]] = tokenizer(\n        example['prompt'], \n        max_length=MAX_TOKEN_LENGTH,   \n        truncation=True,\n        # padding='max_length',\n    )\n    inputs[\"labels\"] = inputs[\"input_ids\"].copy()   # Casual LM get the same tokens as inputs and label\n    \n    return inputs\n\nremove_column_names: List[str] = list(train.features.keys())\n\n# --------------------------------------------------------------------------------\n# Tokenization by applying function\n# --------------------------------------------------------------------------------\ntokenized_train = train.map(\n    function=convert_to_prompt, \n    batched=False,\n    remove_columns=remove_column_names,\n    num_proc=NUM_CPUS\n).map(\n    function=tokenize_prompt, \n    batched=False,\n    remove_columns=['prompt'],\n    num_proc=NUM_CPUS\n).shuffle(\n    seed=42\n).with_format(\n    \"torch\"\n)\n

Training:

data_collator = DataCollatorForLanguageModeling(\n   tokenizer=tokenizer, \n   mlm=False,\n   return_tensors='pt'\n)\n\ntraining_args = TrainingArguments(\n    output_dir=\"bloom_finetuned\",\n    max_steps=MAX_STEPS,\n    num_train_epochs=3,\n    per_device_train_batch_size=1,\n#    per_device_eval_batch_size=1,\n    learning_rate=2e-5,\n    weight_decay=0.01, \n    fp16=USE_FLOAT16,\n    no_cuda=False,\n#    evaluation_strategy=\"epoch\",\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=tokenized_train,\n    tokenizer=tokenizer,\n    data_collator=data_collator,\n    compute_metrics=compute_metrics,\n)\n