Update README.md

README.md

@@ -21,7 +21,7 @@ from transformers import AutoModelForCausalLM, AutoTokenizer
 tokenizer = AutoTokenizer.from_pretrained("stockmark/stockmark-100b")
 model = AutoModelForCausalLM.from_pretrained("stockmark/stockmark-100b", device_map="auto", torch_dtype=torch.bfloat16)
 
-inputs = tokenizer("人工知能とは、", return_tensors="pt").input_ids.to(model.device)
+input_ids = tokenizer("人工知能とは、", return_tensors="pt").input_ids.to(model.device)
 with torch.inference_mode():
     tokens = model.generate(
         input_ids,
@@ -33,4 +33,29 @@ with torch.inference_mode():
     
 output = tokenizer.decode(tokens[0], skip_special_tokens=True)
 print(output)
-```
+```
+
+## Dataset (pretraining)
+
+Stockmark-100b was trained using a total of about 910B tokens of Japanese and English text corpus. The detail of Japanese data is summarized in the below table.
+
+| corpus | tokens after preprocessing |
+|:---:|:---:|
+| Stockmark Web Corpus (This dataset will not be released) | 8.8 billion |
+| Patent | 37.5 billion |
+| Wikipedia |1.5 billion |
+| mC4 | 52.6 billion |
+| CommonCrawl (snapshot: 2020-50 ~ 2024-10) | 203.7 billion|
+
+English data is sampled from [RedPajama-Data](https://github.com/togethercomputer/RedPajama-Data/tree/rp_v1).
+
+## Environment
+- GPU: 48 nodes of 8*H100 instances
+- Library: [Megatron-LM](https://github.com/NVIDIA/Megatron-LM)
+
+## License
+[MIT](https://opensource.org/licenses/MIT)
+
+## Developed by
+[Stockmark Inc.](https://stockmark.co.jp/)
+