Open-Source LLM Training Datasets
Status: reference.
Reference list of free, openly licensed datasets for pretraining,
fine-tuning, and aligning a local LLM. All datasets load via the Hugging
Face datasets library unless noted. Always re-check the
license on the dataset card before training, since terms occasionally
change.
pip install datasetsfrom datasets import load_dataset
ds = load_dataset("HuggingFaceFW/fineweb-edu", name="sample-10BT", split="train", streaming=True)1. Web-Scale Pretraining Corpora
Use these for training a base model from scratch or for continued pretraining of an existing base model. Most are far too large for consumer hardware in full; use the sample subsets noted below.
| Dataset | HF Path | Size | License | Notes |
|---|---|---|---|---|
| FineWeb | HuggingFaceFW/fineweb |
~15T tokens (English) | ODC-By | Current de facto baseline. Has sample-10BT,
sample-100BT, sample-350BT configs. |
| FineWeb-Edu | HuggingFaceFW/fineweb-edu |
~1.3T tokens | ODC-By | Educational-quality filter of FineWeb. Best choice for small models;
punches above its token count. sample-10BT config
available. |
| FineWeb-2 | HuggingFaceFW/fineweb-2 |
Multilingual, 1000+ languages | ODC-By | Use for non-English or multilingual models. |
| Dolma | allenai/dolma |
~3T tokens | ODC-By | Allen AI's OLMo training data. Mix of Common Crawl, Wikipedia, The Stack, Reddit, arXiv, academic text. Fully documented provenance. |
| RedPajama-V2 | togethercomputer/RedPajama-Data-V2 |
100T+ tokens raw | Apache 2.0 | Raw web data with 46 per-document quality signals for custom filtering. Multilingual (en, de, fr, es, it). |
| RedPajama-V1 | togethercomputer/RedPajama-Data-1T |
~1.2T tokens | Apache 2.0 | Reproduction of LLaMA-1 training mix. |
| SlimPajama | cerebras/SlimPajama-627B |
627B tokens | Apache 2.0 | Deduplicated, cleaned cut of RedPajama-V1. Good mid-size option.
6B-token sample available (DKYoon/SlimPajama-6B). |
| Common Corpus | PleIAs/common_corpus |
~2T tokens | Varies (all open/permissive) | Public-domain and openly licensed text only. Best choice if license cleanliness is a priority. Multilingual. |
| Common Pile | common-pile/comma_v0.1_training_dataset |
~8T tokens | Openly licensed sources | EleutherAI's successor to The Pile with clean provenance. |
| C4 | allenai/c4 |
~750 GB (English) | ODC-By | Cleaned Common Crawl used for T5. en,
en.noclean, multilingual configs. |
| The Pile | EleutherAI/pile |
~825 GB | Mixed | Older; some subsets have questionable licensing (Books3 removed). Fine for research. |
| Common Crawl | commoncrawl.org (not on HF as a single dataset) |
Petabyte-scale, monthly | CC0 / terms of use | Raw crawl. Requires your own filtering pipeline (see tooling section). |
Recommendation for local training: Start with
HuggingFaceFW/fineweb-edu sample-10BT. If you
need more tokens, scale to sample-100BT or mix in
cerebras/SlimPajama-627B.
2. Code
| Dataset | HF Path | Size | License | Notes |
|---|---|---|---|---|
| The Stack v2 | bigcode/the-stack-v2 |
~67 TB, 600+ languages | Permissive source licenses only | Standard code pretraining corpus (StarCoder2). Gated; accept terms
on HF. the-stack-v2-dedup and
the-stack-v2-train-smol-ids are the practical configs. |
| StarCoderData | bigcode/starcoderdata |
~783 GB | Permissive source licenses | Pre-filtered training mix used for StarCoder v1. |
| The Stack v1 (dedup) | bigcode/the-stack-dedup |
~3 TB | Permissive source licenses | Older but simpler to work with. |
| OpenCodeInstruct | nvidia/OpenCodeInstruct |
5M samples | CC BY 4.0 | SFT-stage. Each sample: question, solution, test cases, execution feedback, quality score. |
| Magicoder OSS-Instruct | ise-uiuc/Magicoder-OSS-Instruct-75K |
75K | MIT | SFT-stage. Instructions grounded in real open-source code snippets. |
| CodeFeedback-Filtered-Instruction | m-a-p/CodeFeedback-Filtered-Instruction |
157K | Apache 2.0 | SFT-stage. Multi-source filtered code instructions. |
3. Instruction / Chat Fine-Tuning (SFT)
Use these on top of a pretrained base model to produce an instruction-following chat model.
| Dataset | HF Path | Size | License | Notes |
|---|---|---|---|---|
| Tulu 3 SFT Mixture | allenai/tulu-3-sft-mixture |
~940K | ODC-By | Best single general-purpose SFT set. Used for OLMo 2 / Tulu 3. Covers chat, math, code, safety, multilingual. |
| OpenHermes-2.5 | teknium/OpenHermes-2.5 |
~1M | Varies by source | Very popular for local fine-tunes. Aggregated from many sources; check sub-licenses if going commercial. |
| SmolTalk | HuggingFaceTB/smoltalk |
~1.1M | Apache 2.0 | Mix used for SmolLM2-Instruct. Tuned for small models. |
| Magpie (various) | Magpie-Align/Magpie-Pro-300K-Filtered,
Magpie-Align/Magpie-Llama-3.1-Pro-MT-300K-Filtered |
300K each | Llama license (derived) | High-quality synthetic instructions generated from open models. Check the generating model's license. |
| OpenAssistant oasst2 | OpenAssistant/oasst2 |
~135K messages | Apache 2.0 | Human-written multi-turn conversations. Needs tree-to-conversation flattening. |
| OpenAssistant Guanaco | timdettmers/openassistant-guanaco |
~10K | Apache 2.0 | Pre-flattened oasst1 subset; good for quick tests. |
| databricks-dolly-15k | databricks/databricks-dolly-15k |
15K | CC BY-SA 3.0 | Human-written, commercially usable. Small. |
| Alpaca | tatsu-lab/alpaca |
52K | CC BY-NC 4.0 | Classic but non-commercial and GPT-3.5-generated. Research only. |
| Alpaca-GPT4 | vicgalle/alpaca-gpt4 |
52K | CC BY-NC 4.0 | Same prompts, GPT-4 responses. Research only. |
| FLAN v2 | Open-Orca/FLAN or SirNeural/flan_v2 |
Very large | Apache 2.0 | Thousands of academic NLP tasks in instruction format. Boosts zero-shot task performance. |
| OpenOrca | Open-Orca/OpenOrca |
~4.2M | MIT | FLAN prompts with GPT-4/3.5 reasoning responses. |
| UltraChat 200k | HuggingFaceH4/ultrachat_200k |
200K | MIT | Multi-turn synthetic chats; used for Zephyr. |
| LIMA | GAIR/lima |
1K | CC BY-NC-SA | Tiny, high-quality; demonstrates "less is more" for SFT. |
| No Robots | HuggingFaceH4/no_robots |
10K | CC BY-NC 4.0 | Human-written by professional annotators. |
| WizardLM Evol-Instruct | WizardLMTeam/WizardLM_evol_instruct_V2_196k |
196K | Varies | Complexity-evolved instructions. |
| Open-Platypus | garage-bAInd/Open-Platypus |
~25K | Varies | STEM/logic focused. |
Recommendation: Start with
allenai/tulu-3-sft-mixture for a general model, or
HuggingFaceTB/smoltalk if the model is under ~2B
parameters.
4. Preference / Alignment (DPO, ORPO, RLHF)
Use after SFT to improve helpfulness, harmlessness, and response quality.
| Dataset | HF Path | Size | License | Notes |
|---|---|---|---|---|
| UltraFeedback Binarized | HuggingFaceH4/ultrafeedback_binarized |
~64K pairs | MIT | The standard DPO dataset. Used for Zephyr and many Mistral
fine-tunes. Drop-in for TRL DPOTrainer. |
| UltraFeedback (raw) | openbmb/UltraFeedback |
~64K prompts, 4 responses each | MIT | Full multi-response version with fine-grained scores. |
| Tulu 3 Preference Mixture | allenai/llama-3.1-tulu-3-8b-preference-mixture |
~270K pairs | ODC-By | Matched to the Tulu 3 SFT set. On-policy data included. |
| Anthropic HH-RLHF | Anthropic/hh-rlhf |
~170K pairs | MIT | Human helpfulness/harmlessness preferences. |
| Nectar | berkeley-nest/Nectar |
~183K prompts, 7 ranked responses | Apache 2.0 | Ranked multi-response; used for Starling. |
| Stanford SHP | stanfordnlp/SHP |
~385K | Varies | Reddit-derived human preferences across 18 domains. |
| Argilla DPO Mix 7K | argilla/dpo-mix-7k |
7K | MIT | Small, curated, high-quality mix. Good for quick DPO runs. |
| OpenAssistant oasst1 (ranked) | OpenAssistant/oasst1 |
— | Apache 2.0 | Contains human rankings usable for preference pairs. |
| HelpSteer2 | nvidia/HelpSteer2 |
~21K | CC BY 4.0 | Multi-attribute human ratings (helpfulness, correctness, coherence, etc.). |
Recommendation:
HuggingFaceH4/ultrafeedback_binarized for DPO. Use
allenai/llama-3.1-tulu-3-8b-preference-mixture if you used
the Tulu SFT set.
5. Math and Reasoning
| Dataset | HF Path | Size | License | Notes |
|---|---|---|---|---|
| OpenMathInstruct-2 | nvidia/OpenMathInstruct-2 |
~14M | CC BY 4.0 | Problem/solution pairs from GSM8K and MATH seeds, Llama-3.1-405B generated. |
| NuminaMath-CoT | AI-MO/NuminaMath-CoT |
~860K | Apache 2.0 | Competition math with chain-of-thought. |
| NuminaMath-1.5 | AI-MO/NuminaMath-1.5 |
~900K | Apache 2.0 | Updated, cleaner version. |
| OpenThoughts | open-thoughts/OpenThoughts-114k /
open-thoughts/OpenThoughts3-1.2M |
114K / 1.2M | Apache 2.0 | Long reasoning traces (math, code, science) for training "thinking" models. |
| OpenR1-Math | open-r1/OpenR1-Math-220k |
220K | Apache 2.0 | DeepSeek-R1 distilled math reasoning traces. |
| MetaMathQA | meta-math/MetaMathQA |
~395K | MIT | Augmented GSM8K/MATH. |
| GSM8K | openai/gsm8k |
8.5K | MIT | Grade-school math; mostly used for eval but fine for SFT. |
| MATH | hendrycks/competition_math |
12.5K | MIT | Competition problems with solutions. |
| Orca-Math | microsoft/orca-math-word-problems-200k |
200K | MIT | Synthetic word problems. |
6. Domain-Specific and Supplementary
| Dataset | HF Path | Size | License | Notes |
|---|---|---|---|---|
| Wikipedia | wikimedia/wikipedia |
~20 GB (en) | CC BY-SA 4.0 | Clean encyclopedic text. Common pretraining ingredient. |
| arXiv (RedPajama subset) | togethercomputer/RedPajama-Data-1T config
arxiv |
~28B tokens | Varies | Scientific papers. |
| peS2o | allenai/peS2o |
~40M papers | ODC-By | Cleaned academic papers from Semantic Scholar. |
| Project Gutenberg | manu/project_gutenberg |
~70K books | Public domain | Pre-1929 books; long-form narrative text. |
| PubMed (The Pile subset) | via EleutherAI/pile |
— | Mixed | Biomedical abstracts and full text. |
| StackExchange | HuggingFaceH4/stack-exchange-preferences |
~10M | CC BY-SA | Q&A with vote-based preferences. |
| Cosmopedia | HuggingFaceTB/cosmopedia |
~25B tokens | Apache 2.0 | Synthetic textbooks, blogs, stories. Strong for small models. |
| OpenWebMath | open-web-math/open-web-math |
~14.7B tokens | ODC-By | Math-heavy web pages. |
| Aya Collection | CohereForAI/aya_collection |
~513M instances | Apache 2.0 | Multilingual instruction data, 100+ languages. |
7. Suggested Training Pipeline
Stage 1 — Pretraining (optional; skip if starting from an open base model)
FineWeb-Edu sample-10BT (+ Cosmopedia, + The Stack v2 subset for code)
Stage 2 — Supervised Fine-Tuning
allenai/tulu-3-sft-mixture OR HuggingFaceTB/smoltalk (small models)
+ domain data as needed (OpenMathInstruct-2, OpenCodeInstruct)
Stage 3 — Preference Optimization (DPO / ORPO)
HuggingFaceH4/ultrafeedback_binarized
OR allenai/llama-3.1-tulu-3-8b-preference-mixture
Stage 4 — Evaluation
lm-evaluation-harness (EleutherAI), MT-Bench, IFEval, GSM8K, HumanEval
8. Tooling
- Data processing / filtering:
datatrove(Hugging Face), NeMo Curator (NVIDIA),dolmatoolkit (Allen AI) - Fine-tuning: TRL (
SFTTrainer,DPOTrainer), Axolotl, LLaMA-Factory, torchtune, Unsloth - Evaluation:
lm-evaluation-harness,lighteval - Streaming large datasets: use
streaming=Trueinload_datasetto avoid downloading full corpora
9. License Quick Reference
| License | Commercial use | Notes |
|---|---|---|
| Apache 2.0, MIT | Yes | Fully permissive. |
| ODC-By | Yes | Attribution required. |
| CC BY 4.0 | Yes | Attribution required. |
| CC BY-SA | Yes | Share-alike applies to derived datasets. |
| CC BY-NC | No | Research only (Alpaca, LIMA, No Robots). |
| Llama-derived data | Conditional | Subject to Llama community license terms. |
| GPT-generated data | Caution | OpenAI terms restrict using outputs to train competing models. |
10. Directories for Further Browsing
- Hugging Face Hub datasets: https://huggingface.co/datasets (filter by task: text-generation, sort by downloads)
mlabonne/llm-datasets(GitHub): curated, categorized list with quality notesEvanJaye/InstructionZoo(GitHub): instruction-tuning dataset collectionraunak-agarwal/instruction-datasets(GitHub): instruction dataset index with paper linkseugeneyan/open-llms(GitHub): open LLMs and the datasets they used- RedPajama repo: https://github.com/togethercomputer/RedPajama-Data
- Common Corpus paper: https://arxiv.org/abs/2506.01732
- OpenCodeInstruct paper: https://arxiv.org/abs/2504.04030