Zunkiree Labs
AI Models

Transformer

Definition

The neural network architecture behind modern language models, using attention to weigh how much each word matters to the others.

The transformer is the neural network architecture that modern language models are built on, introduced in 2017. Its central idea is self-attention: for each token, the model computes how relevant every other token in the input is, so meaning that depends on distant context is captured directly rather than passed along step by step. Because those comparisons are independent of one another, they run in parallel — which is what made training on internet-scale text practical and is the reason the architecture displaced the recurrent networks that came before it. GPT, Claude, and Llama are all transformers.