Tools, notes and ideas I collect while learning to build a large language model.

Foundations

• PyTorch: the core framework.
• tiktoken: the GPT-2 tokenizer used in Raschka’s book. Check the splits in Tiktokenizer, an online tokenizer playground by dqbd.
• Jupyter or VS Code: for code and experiments.

Reference implementations

• LLMs-from-scratch (Sebastian Raschka): the official code for the book, with extra bonus chapters.
• nanoGPT and nanochat (Andrej Karpathy): leaner training pipelines, good for seeing a full pretraining run end to end.
• LitGPT (Lightning AI): from-scratch implementations of common open models, with finetuning scripts.

Data

• Hugging Face datasets and tokenizers: load corpora and train a custom tokenizer.
• SentencePiece (Google): an unsupervised text tokenizer, another way to train your own vocabulary.
• FineWeb: a large web-text corpus for pretraining, and a paper on how it was filtered and deduplicated.
• datatrove (Hugging Face): a library for filtering, deduplicating and tokenizing text at scale.

Training method

• A Recipe for Training Neural Networks (Andrej Karpathy): a step-by-step approach to training. One of its checks is to overfit a single batch of a few examples first: if the loss cannot reach its minimum there, there is a bug.
• Scaling Laws for Neural Language Models (Kaplan et al., 2020) and Training Compute-Optimal Large Language Models (Hoffmann et al., 2022): how loss depends on model size, data and compute, and how to split a fixed compute budget between model size and training tokens.

Training efficiently

• Mixed precision. bf16 needs a GPU with compute capability 8.0 or higher (Ampere and later). Turing cards such as the T1000 (7.5) use fp16 with gradient scaling, see the PyTorch AMP docs.
• torch.compile. It can speed up training. It relies on Triton, which has no official Windows build, so on Windows it needs community-built wheels.
• Gradient accumulation. Simulates a large batch when GPU memory is too small to hold one. The linked post walks through finetuning an LLM on a single GPU with it.

Distributed training

• FSDP (PyTorch): shards parameters, gradients and optimizer states across GPUs.
• DeepSpeed: a library for distributed training and inference. It matters once the model no longer fits on one GPU.

Watching and debugging

• nvidia-smi: memory use, for tracking down out-of-memory errors.
• TensorBoard or Weights & Biases: loss curves.
• PyTorch Profiler: finds where training time goes.

Post-training

• LoRA and QLoRA: low-rank adapters, and the same idea on a quantized base model, for finetuning with little memory.
• PEFT: the Hugging Face library for parameter-efficient finetuning.
• TRL: Hugging Face library to train language models with reinforcement learning, including supervised finetuning and preference optimization.

Evaluation

• lm-evaluation-harness: the standard benchmark runner.
• Ollama: in chapter 7 of the book it runs the judge model that scores instruction-following answers. On a small GPU, a smaller judge model is the practical choice.

Inference and deployment

• llama.cpp: LLM inference in C/C++, runs quantized models on modest hardware.
• vLLM: a high-throughput, memory-efficient inference and serving engine.
• GPTQ: post-training quantization for generative pretrained transformers.

Learning resources

• Neural Networks: Zero to Hero (Andrej Karpathy): a video course that builds up to a GPT.
• Stanford CS336: Language Modeling from Scratch: course site with schedule, assignments and materials.
• LLaMA: Open and Efficient Foundation Language Models (Touvron et al., 2023): the paper behind a widely copied open architecture.