OKLMdocumentation v1.0.3
On this page

Open-Source LLM Training Datasets

Status: reference.

Reference list of free, openly licensed datasets for pretraining, fine-tuning, and aligning a local LLM. All datasets load via the Hugging Face datasets library unless noted. Always re-check the license on the dataset card before training, since terms occasionally change.

pip install datasets
from datasets import load_dataset
ds = load_dataset("HuggingFaceFW/fineweb-edu", name="sample-10BT", split="train", streaming=True)

1. Web-Scale Pretraining Corpora

Use these for training a base model from scratch or for continued pretraining of an existing base model. Most are far too large for consumer hardware in full; use the sample subsets noted below.

Dataset HF Path Size License Notes
FineWeb HuggingFaceFW/fineweb ~15T tokens (English) ODC-By Current de facto baseline. Has sample-10BT, sample-100BT, sample-350BT configs.
FineWeb-Edu HuggingFaceFW/fineweb-edu ~1.3T tokens ODC-By Educational-quality filter of FineWeb. Best choice for small models; punches above its token count. sample-10BT config available.
FineWeb-2 HuggingFaceFW/fineweb-2 Multilingual, 1000+ languages ODC-By Use for non-English or multilingual models.
Dolma allenai/dolma ~3T tokens ODC-By Allen AI's OLMo training data. Mix of Common Crawl, Wikipedia, The Stack, Reddit, arXiv, academic text. Fully documented provenance.
RedPajama-V2 togethercomputer/RedPajama-Data-V2 100T+ tokens raw Apache 2.0 Raw web data with 46 per-document quality signals for custom filtering. Multilingual (en, de, fr, es, it).
RedPajama-V1 togethercomputer/RedPajama-Data-1T ~1.2T tokens Apache 2.0 Reproduction of LLaMA-1 training mix.
SlimPajama cerebras/SlimPajama-627B 627B tokens Apache 2.0 Deduplicated, cleaned cut of RedPajama-V1. Good mid-size option. 6B-token sample available (DKYoon/SlimPajama-6B).
Common Corpus PleIAs/common_corpus ~2T tokens Varies (all open/permissive) Public-domain and openly licensed text only. Best choice if license cleanliness is a priority. Multilingual.
Common Pile common-pile/comma_v0.1_training_dataset ~8T tokens Openly licensed sources EleutherAI's successor to The Pile with clean provenance.
C4 allenai/c4 ~750 GB (English) ODC-By Cleaned Common Crawl used for T5. en, en.noclean, multilingual configs.
The Pile EleutherAI/pile ~825 GB Mixed Older; some subsets have questionable licensing (Books3 removed). Fine for research.
Common Crawl commoncrawl.org (not on HF as a single dataset) Petabyte-scale, monthly CC0 / terms of use Raw crawl. Requires your own filtering pipeline (see tooling section).

Recommendation for local training: Start with HuggingFaceFW/fineweb-edu sample-10BT. If you need more tokens, scale to sample-100BT or mix in cerebras/SlimPajama-627B.


2. Code

Dataset HF Path Size License Notes
The Stack v2 bigcode/the-stack-v2 ~67 TB, 600+ languages Permissive source licenses only Standard code pretraining corpus (StarCoder2). Gated; accept terms on HF. the-stack-v2-dedup and the-stack-v2-train-smol-ids are the practical configs.
StarCoderData bigcode/starcoderdata ~783 GB Permissive source licenses Pre-filtered training mix used for StarCoder v1.
The Stack v1 (dedup) bigcode/the-stack-dedup ~3 TB Permissive source licenses Older but simpler to work with.
OpenCodeInstruct nvidia/OpenCodeInstruct 5M samples CC BY 4.0 SFT-stage. Each sample: question, solution, test cases, execution feedback, quality score.
Magicoder OSS-Instruct ise-uiuc/Magicoder-OSS-Instruct-75K 75K MIT SFT-stage. Instructions grounded in real open-source code snippets.
CodeFeedback-Filtered-Instruction m-a-p/CodeFeedback-Filtered-Instruction 157K Apache 2.0 SFT-stage. Multi-source filtered code instructions.

3. Instruction / Chat Fine-Tuning (SFT)

Use these on top of a pretrained base model to produce an instruction-following chat model.

Dataset HF Path Size License Notes
Tulu 3 SFT Mixture allenai/tulu-3-sft-mixture ~940K ODC-By Best single general-purpose SFT set. Used for OLMo 2 / Tulu 3. Covers chat, math, code, safety, multilingual.
OpenHermes-2.5 teknium/OpenHermes-2.5 ~1M Varies by source Very popular for local fine-tunes. Aggregated from many sources; check sub-licenses if going commercial.
SmolTalk HuggingFaceTB/smoltalk ~1.1M Apache 2.0 Mix used for SmolLM2-Instruct. Tuned for small models.
Magpie (various) Magpie-Align/Magpie-Pro-300K-Filtered, Magpie-Align/Magpie-Llama-3.1-Pro-MT-300K-Filtered 300K each Llama license (derived) High-quality synthetic instructions generated from open models. Check the generating model's license.
OpenAssistant oasst2 OpenAssistant/oasst2 ~135K messages Apache 2.0 Human-written multi-turn conversations. Needs tree-to-conversation flattening.
OpenAssistant Guanaco timdettmers/openassistant-guanaco ~10K Apache 2.0 Pre-flattened oasst1 subset; good for quick tests.
databricks-dolly-15k databricks/databricks-dolly-15k 15K CC BY-SA 3.0 Human-written, commercially usable. Small.
Alpaca tatsu-lab/alpaca 52K CC BY-NC 4.0 Classic but non-commercial and GPT-3.5-generated. Research only.
Alpaca-GPT4 vicgalle/alpaca-gpt4 52K CC BY-NC 4.0 Same prompts, GPT-4 responses. Research only.
FLAN v2 Open-Orca/FLAN or SirNeural/flan_v2 Very large Apache 2.0 Thousands of academic NLP tasks in instruction format. Boosts zero-shot task performance.
OpenOrca Open-Orca/OpenOrca ~4.2M MIT FLAN prompts with GPT-4/3.5 reasoning responses.
UltraChat 200k HuggingFaceH4/ultrachat_200k 200K MIT Multi-turn synthetic chats; used for Zephyr.
LIMA GAIR/lima 1K CC BY-NC-SA Tiny, high-quality; demonstrates "less is more" for SFT.
No Robots HuggingFaceH4/no_robots 10K CC BY-NC 4.0 Human-written by professional annotators.
WizardLM Evol-Instruct WizardLMTeam/WizardLM_evol_instruct_V2_196k 196K Varies Complexity-evolved instructions.
Open-Platypus garage-bAInd/Open-Platypus ~25K Varies STEM/logic focused.

Recommendation: Start with allenai/tulu-3-sft-mixture for a general model, or HuggingFaceTB/smoltalk if the model is under ~2B parameters.


4. Preference / Alignment (DPO, ORPO, RLHF)

Use after SFT to improve helpfulness, harmlessness, and response quality.

Dataset HF Path Size License Notes
UltraFeedback Binarized HuggingFaceH4/ultrafeedback_binarized ~64K pairs MIT The standard DPO dataset. Used for Zephyr and many Mistral fine-tunes. Drop-in for TRL DPOTrainer.
UltraFeedback (raw) openbmb/UltraFeedback ~64K prompts, 4 responses each MIT Full multi-response version with fine-grained scores.
Tulu 3 Preference Mixture allenai/llama-3.1-tulu-3-8b-preference-mixture ~270K pairs ODC-By Matched to the Tulu 3 SFT set. On-policy data included.
Anthropic HH-RLHF Anthropic/hh-rlhf ~170K pairs MIT Human helpfulness/harmlessness preferences.
Nectar berkeley-nest/Nectar ~183K prompts, 7 ranked responses Apache 2.0 Ranked multi-response; used for Starling.
Stanford SHP stanfordnlp/SHP ~385K Varies Reddit-derived human preferences across 18 domains.
Argilla DPO Mix 7K argilla/dpo-mix-7k 7K MIT Small, curated, high-quality mix. Good for quick DPO runs.
OpenAssistant oasst1 (ranked) OpenAssistant/oasst1 — Apache 2.0 Contains human rankings usable for preference pairs.
HelpSteer2 nvidia/HelpSteer2 ~21K CC BY 4.0 Multi-attribute human ratings (helpfulness, correctness, coherence, etc.).

Recommendation: HuggingFaceH4/ultrafeedback_binarized for DPO. Use allenai/llama-3.1-tulu-3-8b-preference-mixture if you used the Tulu SFT set.


5. Math and Reasoning

Dataset HF Path Size License Notes
OpenMathInstruct-2 nvidia/OpenMathInstruct-2 ~14M CC BY 4.0 Problem/solution pairs from GSM8K and MATH seeds, Llama-3.1-405B generated.
NuminaMath-CoT AI-MO/NuminaMath-CoT ~860K Apache 2.0 Competition math with chain-of-thought.
NuminaMath-1.5 AI-MO/NuminaMath-1.5 ~900K Apache 2.0 Updated, cleaner version.
OpenThoughts open-thoughts/OpenThoughts-114k / open-thoughts/OpenThoughts3-1.2M 114K / 1.2M Apache 2.0 Long reasoning traces (math, code, science) for training "thinking" models.
OpenR1-Math open-r1/OpenR1-Math-220k 220K Apache 2.0 DeepSeek-R1 distilled math reasoning traces.
MetaMathQA meta-math/MetaMathQA ~395K MIT Augmented GSM8K/MATH.
GSM8K openai/gsm8k 8.5K MIT Grade-school math; mostly used for eval but fine for SFT.
MATH hendrycks/competition_math 12.5K MIT Competition problems with solutions.
Orca-Math microsoft/orca-math-word-problems-200k 200K MIT Synthetic word problems.

6. Domain-Specific and Supplementary

Dataset HF Path Size License Notes
Wikipedia wikimedia/wikipedia ~20 GB (en) CC BY-SA 4.0 Clean encyclopedic text. Common pretraining ingredient.
arXiv (RedPajama subset) togethercomputer/RedPajama-Data-1T config arxiv ~28B tokens Varies Scientific papers.
peS2o allenai/peS2o ~40M papers ODC-By Cleaned academic papers from Semantic Scholar.
Project Gutenberg manu/project_gutenberg ~70K books Public domain Pre-1929 books; long-form narrative text.
PubMed (The Pile subset) via EleutherAI/pile — Mixed Biomedical abstracts and full text.
StackExchange HuggingFaceH4/stack-exchange-preferences ~10M CC BY-SA Q&A with vote-based preferences.
Cosmopedia HuggingFaceTB/cosmopedia ~25B tokens Apache 2.0 Synthetic textbooks, blogs, stories. Strong for small models.
OpenWebMath open-web-math/open-web-math ~14.7B tokens ODC-By Math-heavy web pages.
Aya Collection CohereForAI/aya_collection ~513M instances Apache 2.0 Multilingual instruction data, 100+ languages.

7. Suggested Training Pipeline

Stage 1 — Pretraining (optional; skip if starting from an open base model)
  FineWeb-Edu sample-10BT  (+ Cosmopedia, + The Stack v2 subset for code)

Stage 2 — Supervised Fine-Tuning
  allenai/tulu-3-sft-mixture   OR   HuggingFaceTB/smoltalk (small models)
  + domain data as needed (OpenMathInstruct-2, OpenCodeInstruct)

Stage 3 — Preference Optimization (DPO / ORPO)
  HuggingFaceH4/ultrafeedback_binarized
  OR allenai/llama-3.1-tulu-3-8b-preference-mixture

Stage 4 — Evaluation
  lm-evaluation-harness (EleutherAI), MT-Bench, IFEval, GSM8K, HumanEval

8. Tooling

  • Data processing / filtering: datatrove (Hugging Face), NeMo Curator (NVIDIA), dolma toolkit (Allen AI)
  • Fine-tuning: TRL (SFTTrainer, DPOTrainer), Axolotl, LLaMA-Factory, torchtune, Unsloth
  • Evaluation: lm-evaluation-harness, lighteval
  • Streaming large datasets: use streaming=True in load_dataset to avoid downloading full corpora

9. License Quick Reference

License Commercial use Notes
Apache 2.0, MIT Yes Fully permissive.
ODC-By Yes Attribution required.
CC BY 4.0 Yes Attribution required.
CC BY-SA Yes Share-alike applies to derived datasets.
CC BY-NC No Research only (Alpaca, LIMA, No Robots).
Llama-derived data Conditional Subject to Llama community license terms.
GPT-generated data Caution OpenAI terms restrict using outputs to train competing models.

10. Directories for Further Browsing