Big ideas.
Right here.

How model size, training data and compute relate to language-model performance. A foundation for understanding scaling as an empirical relationship, with limits.

The Chinchilla study revisits how to divide a training budget between model size and data. It argues that many contemporary large models were undertrained.

The transformer replaces recurrence with attention. It became a foundation for language models and many other systems that process sequences.

BERT learns representations from text in both directions and then adapts them to language tasks. It helped establish pretraining followed by fine-tuning.

GPT-3 investigates how a large language model can perform tasks from instructions and examples in its prompt, without task-specific weight updates.

The GPT-4 technical report documents capabilities, evaluations and limitations of a multimodal model, while withholding many architecture and training details.

Chain-of-thought prompting explores whether intermediate reasoning examples improve language-model performance on multi-step tasks.

InstructGPT studies fine-tuning language models using demonstrations and human preference judgments to better follow instructions.

Constitutional AI explores training a more helpful and harmless assistant with written principles and AI-generated feedback.

Direct Preference Optimization derives a preference-learning objective that avoids training a separate reward model in the studied setup.

Denoising diffusion probabilistic models learn to generate data by reversing a noise-adding process. This became a foundation of modern image generation.

Latent diffusion moves the denoising process into a compressed representation to reduce the cost of generating high-resolution images.

CLIP learns visual concepts from paired images and text, connecting language descriptions to image representations.

The Vision Transformer applies a transformer to sequences of image patches and studies large-scale image recognition.

Residual networks introduce skip connections that make very deep networks easier to optimize for visual recognition.

Generative adversarial networks train a generator and a discriminator together. Their competing objectives can learn to synthesize data.

Variational autoencoders provide a tractable approach to learning probabilistic latent representations and generating data.

Adam combines adaptive learning rates with moving averages of gradients, becoming a widely used optimizer for neural networks.

Batch normalization studies normalizing layer inputs during training to improve optimization and training speed.

Dropout regularizes neural networks by randomly omitting units during training, reducing dependence on particular combinations of features.

Double Q-learning uses two value estimates to reduce the tendency of standard Q-learning to overestimate action values, and combines this with deep networks for learning from high-dimensional observations.

Proximal Policy Optimization proposes a policy-gradient objective that limits overly large updates while simplifying implementation.

Retrieval-Augmented Generation combines a language model with retrieved documents, bringing external evidence into generation.

LoRA adapts large models with low-rank updates, reducing the number of trainable parameters during fine-tuning.

QLoRA investigates fine-tuning quantized models with low-rank adapters to reduce memory requirements.

FlashAttention reorganizes attention computation to reduce costly reads and writes between GPU memory levels.

FlashAttention-2 further improves GPU attention efficiency through work partitioning and parallelism.

PagedAttention manages attention key-value memory in blocks to improve the efficiency of language-model serving.

Mamba investigates selective state-space models as an alternative approach to sequence modeling with efficient scaling.

Switch Transformers studies sparse mixture-of-experts models, activating selected expert networks rather than every parameter.

Sparsely gated mixture-of-experts layers explore conditional computation using a learned routing network.

LLaMA studies efficient foundation language models and the relationship between model size, data and inference budgets.