The Complete LLM Training Engineer Guide

From Tokenizer to Post-Training, from single GPU to 10K-GPU clusters, from text to multimodal.

For engineers seeking full-stack mastery: foundational theory, systems engineering, frontier architectures, and production practice.

中文版 / Chinese Version | Read online: yingwang.github.io/llm-tutorial

Overview: From Data to Deployment

flowchart LR
    subgraph DATA["<b>① Data</b>"]
        D1["Web Crawl\nWikipedia\nCode/Math"] --> D2["Cleaning\nDedup\nFiltering"]
    end

    subgraph TOK["<b>② Tokenizer</b>"]
        D2 --> T1["BPE / Unigram\nVocab Training"]
        T1 --> T2["Token IDs"]
    end

    subgraph PRE["<b>③ Pretraining</b>"]
        T2 --> P1["Transformer\nGQA · MoE · MLA\nRoPE · SwiGLU"]
        P1 --> |"Next Token\nPrediction"| P2["Base Model"]
    end

    subgraph POST["<b>④ Post-Training</b>"]
        P2 --> S1["SFT\nInstruction Tuning"]
        S1 --> S2["RLHF / DPO\nPreference Opt."]
        S2 --> S3["Safety\nTraining"]
    end

    subgraph DEPLOY["<b>⑤ Deployment</b>"]
        S3 --> Q1["Quantization\nINT4/FP8"]
        Q1 --> Q2["Serving\nvLLM · SGLang"]
    end

    classDef step fill:#ffffff,stroke:#555,color:#222
    class D1,D2,T1,T2,P1,P2,S1,S2,S3,Q1,Q2 step

    style DATA fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    style TOK fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    style PRE fill:#e8eaf6,stroke:#283593,color:#1a237e
    style POST fill:#fff3e0,stroke:#e65100,color:#bf360c
    style DEPLOY fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20

Knowledge Map: Chapter Dependencies

flowchart TB
    subgraph core["Core Training Pipeline"]
        direction LR
        ch1["<a href='01-tokenizer.md'>① Tokenizer</a>"]
        ch2["<a href='02-architecture.md'>② Architecture</a>"]
        ch3["<a href='03-pretraining.md'>③ Pretraining</a>"]
        ch4["<a href='04-post-training.md'>④ Post-Training</a>"]
        ch1 --> ch2 --> ch3 --> ch4
    end

    subgraph infra["Engineering Infrastructure"]
        direction LR
        ch5["<a href='05-peft.md'>⑤ PEFT\nLoRA/QLoRA</a>"]
        ch6["<a href='06-infra.md'>⑥ Distributed\nTP/PP/DP</a>"]
        ch7["<a href='07-inference.md'>⑦ Inference\nQuant/vLLM</a>"]
    end

    subgraph extend["Capability Extensions"]
        direction LR
        ch8["<a href='08-embedding-rag.md'>⑧ Embedding\n& RAG</a>"]
        ch9["<a href='09-multimodal.md'>⑨ Multimodal\nVLM/Video</a>"]
        ch10["<a href='10-safety-alignment.md'>⑩ Safety\n& Alignment</a>"]
    end

    subgraph ref["Reference"]
        direction LR
        ch11["<a href='11-sota-models.md'>⑪ SOTA\nModels</a>"]
        ch12["<a href='12-distillation-merging.md'>⑫ Distillation\n& Merging</a>"]
        ch13["<a href='13-roadmap.md'>⑬ Roadmap\n& Resources</a>"]
    end

    ch4 -.-> ch5
    ch3 -.-> ch6
    ch4 -.-> ch7
    ch7 -.-> ch8
    ch3 -.-> ch9
    ch4 -.-> ch10

    classDef chapter fill:#ffffff,stroke:#555,color:#222
    class ch1,ch2,ch3,ch4,ch5,ch6,ch7,ch8,ch9,ch10,ch11,ch12,ch13 chapter

    style core fill:#e8eaf6,stroke:#3949ab
    style infra fill:#e0f2f1,stroke:#00695c
    style extend fill:#fce4ec,stroke:#c62828
    style ref fill:#f3e5f5,stroke:#6a1b9a

Table of Contents

#	Chapter	Topics
1	Tokenizer	BPE, WordPiece, Unigram, multilingual, byte-level
2	Architecture	Transformer, RoPE, GQA, MoE, MLA, Scaling Laws
3	Pretraining	Data pipeline, objectives, optimizer, stability, long context
4	Post-Training	SFT, RLHF, DPO, GRPO, Reasoning Models
5	PEFT	LoRA, QLoRA, Adapters, practical tips
6	Training Infrastructure	GPU, TP/PP/DP/EP, frameworks, fault tolerance, MFU
7	Inference & Deployment	KV Cache, quantization, vLLM, Speculative Decoding, cost
8	Embedding & RAG	Embedding models, vector search, RAG pipeline
9	Multimodal	VLM, image generation, audio, video, Omni Models
10	Safety & Alignment	Benchmarks, Red Teaming, Guardrails, structured output
11	SOTA Models	LLaMA 3, DeepSeek, Claude, Gemini, GPT-4, Qwen
12	Distillation & Merging	KD, TIES, DARE, mergekit
13	Practical Roadmap	Learning path, paper list, resources, hardware, cost

Training Cost Reference

Model Size	Hardware	Tokens	Est. Cost	Time
1B	1x A100 80GB	20B	~$500	~2 days
7B	8x A100 80GB	1T	~$50K	~2 weeks
13B	32x A100 80GB	2T	~$200K	~3 weeks
70B	256x H100	2T	~$2M	~1 month
405B	16384x H100	15T	~$50M+	~2 months
671B MoE	2048x H800	14.8T	~$5.5M	~2 months

The final row reflects DeepSeek-V3: MoE architecture slashes pretraining compute costs by nearly an order of magnitude compared to dense models of comparable capability.

Companion Code

Runnable implementations reside in llm-tutorial-code, organized by chapter: from first-principles BPE tokenizers and attention kernels to full-scale pretraining, SFT, DPO, LoRA, and high-throughput inference serving.

Glossary

Need clarity on terminology? Consult the Glossary: 80+ structured entries spanning model architectures, training dynamics, PEFT, distributed infrastructure, serving engines, RAG systems, and multimodal paradigms.

Getting Started

If you are new to LLM systems engineering, we recommend the following sequence:

1. Chapter 1: Tokenizer: understand how raw text becomes discrete token sequences
2. Chapter 2: Architecture: master the fundamental building blocks of modern LLMs
3. Chapter 13: Roadmap: chart your learning path and toolchain selection
4. Get hands-on: build intuition with nanoGPT
5. Return to explore the specialized systems chapters

Already familiar with the foundations? Jump directly to the specific topic you need.

Key Links

Resource	Link
nanoGPT	github.com/karpathy/nanoGPT
HuggingFace TRL	github.com/huggingface/trl
vLLM	github.com/vllm-project/vllm
Megatron-LM	github.com/NVIDIA/Megatron-LM
Flash Attention	github.com/Dao-AILab/flash-attention
LLM Evaluation	github.com/EleutherAI/lm-evaluation-harness
Chatbot Arena Leaderboard	huggingface.co/spaces/lmsys/chatbot-arena-leaderboard

Author

Ying Wang

Citation

If this guide is useful in your work, please cite:

@misc{wang2026llmtutorial,
  author = {Ying Wang},
  title  = {LLM 训练工程师完全指南 / The Complete LLM Training Engineer Guide},
  year   = {2026},
  url    = {https://github.com/yingwang/llm-tutorial}
}

License

This repository is dual-licensed:

• Prose and diagrams (Mermaid flowcharts, explanatory text, chapter content): CC BY-NC-SA 4.0 (Attribution · NonCommercial · ShareAlike)
• Code snippets (Python/Bash examples within chapters): MIT (free to use, including commercial, with copyright notice retained)

In practice: code snippets are free to copy and adapt for any use, including commercial software. The non-commercial restriction applies only to the prose, book text, and original diagrams.

---

Last updated: 2026-04-25