Habana

/

bert-base-uncased

like 0

I get

Traceback (most recent call last):                                                                                                                                                            
  File "/root/workspace/optimum-habana/examples/question-answering/run_qa.py", line 732, in <module>                                                                                          
    main()                                                                                                                                                                                    
  File "/root/workspace/optimum-habana/examples/question-answering/run_qa.py", line 678, in main                                                                                              
    train_result = trainer.train(resume_from_checkpoint=checkpoint)                                                                                                                           
  File "/usr/local/lib/python3.10/dist-packages/optimum/habana/transformers/trainer.py", line 516, in train                                                                                   
    self._load_from_checkpoint(resume_from_checkpoint)                                                                                                                                        
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2493, in _load_from_checkpoint                                                                                 
    load_result = model.load_state_dict(state_dict, False)                                                                                                                                    
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 2156, in load_state_dict                                                                                    
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(                                                                                                                 
RuntimeError: Error(s) in loading state_dict for BertForQuestionAnswering:                                                                                                                    
        size mismatch for bert.embeddings.word_embeddings.weight: copying a param with shape torch.Size([30522, 1024]) from checkpoint, the shape in current model is torch.Size([30522, 768])
.                                                                                                                                                                                             
        size mismatch for bert.embeddings.position_embeddings.weight: copying a param with shape torch.Size([512, 1024]) from checkpoint, the shape in current model is torch.Size([512, 768])
.                                                                                                                                                                                             
        size mismatch for bert.embeddings.token_type_embeddings.weight: copying a param with shape torch.Size([2, 1024]) from checkpoint, the shape in current model is torch.Size([2, 768]). 
        size mismatch for bert.embeddings.LayerNorm.weight: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).                   
        size mismatch for bert.embeddings.LayerNorm.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).                     
        size mismatch for bert.encoder.layer.0.attention.self.query.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([768
, 768]).                                                                                                                                                                                      
        size mismatch for bert.encoder.layer.0.attention.self.query.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).     
        size mismatch for bert.encoder.layer.0.attention.self.key.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([768, 
768]).                                                                                                                                                                                        
        size mismatch for bert.encoder.layer.0.attention.self.key.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).       
        size mismatch for bert.encoder.layer.0.attention.self.value.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([768
, 768]).                                                                                                                                                                                      
        size mismatch for bert.encoder.layer.0.attention.self.value.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).     
        size mismatch for bert.encoder.layer.0.attention.output.dense.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([7
68, 768]).                                                                                                                                                                                    
        size mismatch for bert.encoder.layer.0.attention.output.dense.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).   
        size mismatch for bert.encoder.layer.0.attention.output.LayerNorm.weight: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768
]).                                                                                                                                                                                           
        size mismatch for bert.encoder.layer.0.attention.output.LayerNorm.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768])
.                                                                                                                                                                                             
        size mismatch for bert.encoder.layer.0.intermediate.dense.weight: copying a param with shape torch.Size([4096, 1024]) from checkpoint, the shape in current model is torch.Size([3072,
 768]).                                                                                                                                                                                       
        size mismatch for bert.encoder.layer.0.intermediate.dense.bias: copying a param with shape torch.Size([4096]) from checkpoint, the shape in current model is torch.Size([3072]).      
        size mismatch for bert.encoder.layer.0.output.dense.weight: copying a param with shape torch.Size([1024, 4096]) from checkpoint, the shape in current model is torch.Size([768, 3072])
.                                                                                                                                                                                             
        size mismatch for bert.encoder.layer.0.output.dense.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).             
        size mismatch for bert.encoder.layer.0.output.LayerNorm.weight: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).       
        size mismatch for bert.encoder.layer.0.output.LayerNorm.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).         
        size mismatch for bert.encoder.layer.1.attention.self.query.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([768
, 768]).                                                                                                                                                                                      
        size mismatch for bert.encoder.layer.1.attention.self.query.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).     
        size mismatch for bert.encoder.layer.1.attention.self.key.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([768, 
768]).
...

when running

PT_HPU_LAZY_MODE=0 python run_qa.py \
  --model_name_or_path bert-base-uncased \
  --gaudi_config_name Habana/bert-base-uncased \
  --dataset_name squad \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 24 \
  --per_device_eval_batch_size 8 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --output_dir /tmp/squad/ \
  --use_habana \
  --torch_compile_backend hpu_backend \
  --torch_compile \
  --use_lazy_mode false \
  --throughput_warmup_steps 3 \
  --bf16

on Gaudi2 with SynapseAI v1.16.0 and installing Optimum Habana from main.

That's strange I was able to make it running with exactly same cmd, so there must be some difference.

My CMD history:
pip install optimum[habana]
git clone https://github.com/huggingface/optimum-habana.git
cd optimum-habana/examples/question-answering/
pip install -r requirements.txt
PT_HPU_LAZY_MODE=0 python run_qa.py --model_name_or_path bert-base-uncased --gaudi_config_name Habana/bert-base-uncased --dataset_name squad --do_train --do_eval --per_device_train_batch_size 24 --per_device_eval_batch_size 8 --learning_rate 3e-5 --num_train_epochs 2 --max_seq_length 384 --output_dir /tmp/squad/ --use_habana --torch_compile_backend hpu_backend --torch_compile --use_lazy_mode false --throughput_warmup_steps 3 --bf16

Output:
95%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 7000/7378 [04:09<00:11, 34.08it/s][INFO|trainer.py:1610] 2024-07-15 15:16:03,185 >> Saving model checkpoint to /tmp/squad/checkpoint-7000
[INFO|configuration_utils.py:471] 2024-07-15 15:16:03,239 >> Configuration saved in /tmp/squad/checkpoint-7000/config.json
[INFO|modeling_utils.py:2590] 2024-07-15 15:16:03,825 >> Model weights saved in /tmp/squad/checkpoint-7000/model.safetensors
[INFO|tokenization_utils_base.py:2488] 2024-07-15 15:16:03,825 >> tokenizer config file saved in /tmp/squad/checkpoint-7000/tokenizer_config.json
[INFO|tokenization_utils_base.py:2497] 2024-07-15 15:16:03,825 >> Special tokens file saved in /tmp/squad/checkpoint-7000/special_tokens_map.json
[INFO|configuration_utils.py:125] 2024-07-15 15:16:03,835 >> Configuration saved in /tmp/squad/checkpoint-7000/gaudi_config.json
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 7376/7378 [04:22<00:00, 34.03it/s][INFO|trainer.py:1067] 2024-07-15 15:16:15,965 >>

Training completed. Do not forget to share your model on huggingface.co/models =)

{'train_runtime': 263.236, 'train_samples_per_second': 717.752, 'train_steps_per_second': 29.91, 'train_loss': 1.025522436285187, 'epoch': 2.0, 'memory_allocated (GB)': 1.9, 'max_memory_allocated (GB)': 94.61, 'total_memory_available (GB)': 94.62}
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7378/7378 [04:23<00:00, 28.03it/s]
[INFO|trainer.py:1610] 2024-07-15 15:16:16,427 >> Saving model checkpoint to /tmp/squad/
[INFO|configuration_utils.py:471] 2024-07-15 15:16:16,478 >> Configuration saved in /tmp/squad/config.json
[INFO|modeling_utils.py:2590] 2024-07-15 15:16:17,054 >> Model weights saved in /tmp/squad/model.safetensors
[INFO|tokenization_utils_base.py:2488] 2024-07-15 15:16:17,055 >> tokenizer config file saved in /tmp/squad/tokenizer_config.json
[INFO|tokenization_utils_base.py:2497] 2024-07-15 15:16:17,055 >> Special tokens file saved in /tmp/squad/special_tokens_map.json
[INFO|configuration_utils.py:125] 2024-07-15 15:16:17,064 >> Configuration saved in /tmp/squad/gaudi_config.json
***** train metrics *****
epoch = 2.0
max_memory_allocated (GB) = 94.61
memory_allocated (GB) = 1.9
total_flos = 32318061GF
total_memory_available (GB) = 94.62
train_loss = 1.0255
train_runtime = 0:04:23.23
train_samples = 88524
train_samples_per_second = 717.752
train_steps_per_second = 29.91
07/15/2024 15:16:17 - INFO - main - *** Evaluate ***
[INFO|trainer.py:786] 2024-07-15 15:16:17,067 >> The following columns in the evaluation set don't have a corresponding argument in BertForQuestionAnswering.forward and have been ignored: example_id, offset_mapping. If example_id, offset_mapping are not expected by BertForQuestionAnswering.forward, you can safely ignore this message.
[INFO|trainer.py:1779] 2024-07-15 15:16:17,069 >> ***** Running Evaluation *****
[INFO|trainer.py:1781] 2024-07-15 15:16:17,069 >> Num examples = 10784
[INFO|trainer.py:1784] 2024-07-15 15:16:17,069 >> Batch size = 8
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍| 1345/1348 [00:41<00:00, 37.32it/s]07/15/2024 15:17:07 - INFO - utils_qa - Post-processing 10570 example predictions split into 10784 features.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10570/10570 [00:25<00:00, 421.39it/s]
07/15/2024 15:17:32 - INFO - utils_qa - Saving predictions to /tmp/squad/eval_predictions.json.████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 10568/10570 [00:25<00:00, 419.57it/s]
07/15/2024 15:17:32 - INFO - utils_qa - Saving nbest_preds to /tmp/squad/eval_nbest_predictions.json.
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1348/1348 [01:19<00:00, 16.98it/s]
***** eval metrics *****
epoch = 2.0
eval_exact_match = 80.3784
eval_f1 = 87.9444
eval_runtime = 0:00:41.28
eval_samples = 10784
eval_samples_per_second = 261.2
eval_steps_per_second = 32.65
max_memory_allocated (GB) = 94.61
memory_allocated (GB) = 1.4
total_memory_available (GB) = 94.62
[INFO|modelcard.py:450] 2024-07-15 15:17:36,853 >> Dropping the following result as it does not have all the necessary fields:
{'task': {'name': 'Question Answering', 'type': 'question-answering'}, 'dataset': {'name': 'squad', 'type': 'squad'}}

My PIP list:

Package Version

absl-py 2.1.0
accelerate 0.27.2
aiohttp 3.9.5
aiosignal 1.3.1
async-timeout 4.0.3
attrs 23.2.0
av 9.2.0
cachetools 5.3.3
certifi 2024.2.2
cffi 1.15.1
cfgv 3.4.0
charset-normalizer 3.3.2
cmake 3.29.3
coloredlogs 15.0.1
datasets 2.19.2
diffusers 0.26.3
dill 0.3.8
distlib 0.3.8
evaluate 0.4.2
expecttest 0.2.1
filelock 3.14.0
frozenlist 1.4.1
fsspec 2024.3.1
google-auth 2.29.0
google-auth-oauthlib 0.4.6
grpcio 1.64.0
habana_gpu_migration 1.16.0.527
habana-media-loader 1.16.0.527
habana-pyhlml 1.16.0.527
habana_quantization_toolkit 1.16.0.527
habana-torch-dataloader 1.16.0.527
habana-torch-plugin 1.16.0.527
huggingface-hub 0.22.2
humanfriendly 10.0
identify 2.5.36
idna 3.7
importlib_metadata 8.0.0
Jinja2 3.1.4
lightning 2.2.4
lightning-habana 1.5.0
lightning-utilities 0.11.2
Markdown 3.6
MarkupSafe 2.1.5
mpi4py 3.1.4
mpmath 1.3.0
multidict 6.0.5
multiprocess 0.70.16
networkx 3.3
ninja 1.11.1.1
nodeenv 1.9.0
numpy 1.23.5
oauthlib 3.2.2
optimum 1.21.2
optimum-habana 1.12.1
packaging 24.0
pandas 2.0.1
pathspec 0.12.1
perfetto 0.7.0
pillow 10.4.0
Pillow-SIMD 7.0.0.post3
pip 23.3.1
platformdirs 4.2.2
pre-commit 3.3.3
protobuf 3.20.3
psutil 6.0.0
pyarrow 16.1.0
pyarrow-hotfix 0.6
pyasn1 0.6.0
pyasn1_modules 0.4.0
pybind11 2.10.4
pycparser 2.22
pydantic 1.10.13
pynvml 8.0.4
python-dateutil 2.9.0.post0
pytorch-lightning 2.2.5
pytz 2024.1
PyYAML 6.0
regex 2023.5.5
requests 2.32.3
requests-oauthlib 2.0.0
rsa 4.9
safetensors 0.4.3
sentencepiece 0.2.0
setuptools 70.0.0
six 1.16.0
symengine 0.11.0
sympy 1.12.1
tdqm 0.0.1
tensorboard 2.11.2
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
tokenizers 0.19.1
torch 2.2.2a0+gitb5d0b9b
torch_tb_profiler 0.4.0
torchaudio 2.2.0+08901ad
torchdata 0.7.1+5e6f7b7
torchmetrics 1.4.0.post0
torchtext 0.17.0+400da5c
torchvision 0.17.0+b2383d4
tqdm 4.66.4
transformers 4.40.2
typing_extensions 4.12.0
tzdata 2024.1
urllib3 1.26.18
virtualenv 20.26.2
Werkzeug 3.0.3
wheel 0.43.0
xxhash 3.4.1
yamllint 1.35.1
yarl 1.9.4
zipp 3.19.2

Just stepped at this problem when running bert-large. The root cause is very simple: we store checkpoints inside /tmp/squad and then try to reuse them on different model :) So the solution is we need to delete the /tmp/suad before running the training.

LGTM!