The idea in plain language.
Represent observations with a learned vocabulary of discrete codes.
How it works
An encoder maps an observation to a representation, which is replaced by a nearby entry in a learned codebook. A decoder reconstructs the observation from those discrete codes. A separate prior learns patterns over the code sequence for generation. The design addresses a problem where powerful decoders can otherwise ignore their latent representation.
What to keep in mind
The codebook and decoder constrain what can be represented. Strong reconstruction or generation results do not imply that the discrete codes match human concepts.
Source: Neural Discrete Representation Learning. The original manuscript contains the methods, experiments, figures, and references. An arXiv posting date may follow an earlier conference publication. Read the linked record for version history.