2015 · Representation learning
Distilling the Knowledge in a Neural Network
Knowledge distillation trains a smaller model to reproduce the informative predictions of a larger teacher or ensemble.
Paper & contextGeoffrey Hinton
Learning useful internal representations from data.
Hinton studies how neural networks discover internal structure in data. His work connects distributed representations, generative learning, distillation, and alternatives to backpropagation. This selection follows that thread from dropout and capsules to the Forward-Forward algorithm.
Selected work
10 papers
2012 · Optimization
Improving neural networks by preventing co-adaptation of feature detectors
Dropout regularizes neural networks by randomly omitting units during training, reducing dependence on particular combinations of features.
Paper & context2017 · Representation learning
Dynamic Routing Between Capsules
Capsule networks represent object properties with vectors and route information according to agreement between parts and wholes.
Paper & context2021 · Representation learning
How to represent part-whole hierarchies in a neural network
GLOM is a conceptual proposal for representing image-dependent part-whole hierarchies inside a network with a fixed architecture.
Paper & context2017 · Architecture
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Sparsely gated mixture-of-experts layers explore conditional computation using a learned routing network.
Paper & context2022 · Representation learning
The Forward-Forward Algorithm: Some Preliminary Investigations
Selected research in representation learning. Read the full paper, including the methods, experiments, and reported results.
Paper & context2022 · Representation learning
Meta-Learning Fast Weight Language Models
Selected research in representation learning. Read the full paper, including the methods, experiments, and reported results.
Paper & context2022 · Representation learning
Testing GLOM's ability to infer wholes from ambiguous parts
Selected research in representation learning. Read the full paper, including the methods, experiments, and reported results.
Paper & context2022 · Representation learning
Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning
Selected research in representation learning. Read the full paper, including the methods, experiments, and reported results.
Paper & context2026 · Representation learning
International AI Safety Report 2026
Selected research in representation learning, deep learning. Read the full paper, including the methods, experiments, and reported results.
Paper & contextAn independent editorial profile. Inclusion does not imply Council membership or endorsement. Research is collaborative; coauthorship does not imply sole credit.
Selection & sources