smolm-autoreg-bpe-counterfactual_babylm_aann_low_variability_all-1e-3

This model was trained from scratch on the kanishka/counterfactual_babylm_aann_low_variability_all dataset. It achieves the following results on the evaluation set:

• Loss: 3.4168
• Accuracy: 0.4101

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 64
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
3.6021	1.0	18594	3.8030	0.3580
3.3886	2.0	37188	3.6019	0.3808
3.2595	3.0	55782	3.4546	0.3925
3.1812	4.0	74376	3.4454	0.3980
3.1282	5.0	92970	3.4115	0.4015
3.0804	6.0	111564	3.3765	0.4046
3.0457	7.0	130158	3.3680	0.4062
3.0127	8.0	148752	3.3718	0.4069
2.9897	9.0	167346	3.3723	0.4080
2.9602	10.0	185940	3.3669	0.4083
2.9377	11.0	204534	3.3622	0.4096
2.9136	12.0	223128	3.3741	0.4098
2.8939	13.0	241722	3.3820	0.4091
2.8736	14.0	260316	3.3767	0.4099
2.8493	15.0	278910	3.3848	0.4101
2.831	16.0	297504	3.3952	0.4100
2.8158	17.0	316098	3.4012	0.4106
2.8018	18.0	334692	3.4028	0.4104
2.7777	19.0	353286	3.4115	0.4103
2.7582	20.0	371880	3.4168	0.4101

Framework versions

• Transformers 4.38.0
• Pytorch 2.3.1+cu121
• Datasets 2.16.1
• Tokenizers 0.15.2