Building every single component of an LLM from scratch using PyTorch.
The goal is to understand how LLMs actually work from first principles.
Covered so far:
1. Data & tokenization pipeline
- Text cleaning and normalization
- Tokenization pipeline for autoregressive training
- Dataset integration using Hugging Face corpora
2. Positional encoding strategies
- Learned (linear) positional embeddings
- Rotary positional embeddings (rope)
3. Attention mechanisms
- Scaled dot-product attention
- Multi-head attention
- Advanced attention variants:
- Grouped query attention (gqa)
- Multi-head latent attention (mla)
- Sliding window attention (swa)
4. Transformer architecture design
- Complete transformer block implementation
- Layer normalization
- Mlp / feed-forward networks
- Residual connections
5. Scaling & efficiency techniques
- Mixture-of-experts (moe) routing
- Kv caching for optimized inference
6. Training & adaptation
- Autoregressive pretraining on large-scale corpora
- Classification fine-tuning
- Supervised instruction fine-tuning