Skip to content
forward pass

LLaMA leaks and open weights take off

Meta releases GPT-3-class models small enough for a single GPU to researchers; the weights leak within a week, and the open-model ecosystem builds itself on them.

category
model
significance
4 of 5
people
Hugo Touvron, Guillaume Lample, Yann LeCun
organisations
Meta AI

what had to happen · 37 events back to 1943

Every event this one built on, transitively, in order. The organism has the same path lit. Direct influences are marked.

00 · One neuron · 1

  1. 1943A logical calculus of nervous activity

I · Foundations · 6

  1. 1949Cells that fire together wire together
  2. 1958The perceptron learns
  3. 1960ADALINE and the least-mean-squares rule
  4. 1965Moore's law
  5. 1969Perceptrons
  6. 1970Reverse-mode automatic differentiation

W1 · The first winter · 2

  1. 1974Werbos applies backpropagation to neural networks
  2. 1980The Neocognitron

II · Connection · 3

  1. 1982The Hopfield network
  2. 1985The Boltzmann machine
  3. 1986Backpropagation

W2 · The second winter · 3

  1. 1989LeNet reads handwritten postcodes
  2. 1990Finding structure in time
  3. 1991The vanishing gradient problem

III · Statistics and data · 9

  1. 1997Long short-term memory
  2. 1998MNIST and LeNet-5
  3. 1999The first GPU
  4. 2003A neural probabilistic language model
  5. 2006Deep belief networks and the word 'deep'
  6. 2007CUDA
  7. 2009Deep learning moves to GPUs
  8. 2009ImageNet
  9. 2010Rectified linear units

IV · Deep learning · 7

  1. 2012Google Brain's network discovers cats
  2. 2012Dropout
  3. 2012AlexNet wins ImageNet
  4. 2014Attention
  5. 2014Sequence to sequence learning
  6. 2015Batch normalisation
  7. 2015Residual networks

V · Transformers · 5

  1. 2017Attention is all you need
  2. 2018GPT: generative pre-training
  3. 2019GPT-2 and the model too dangerous to release
  4. 2020Scaling laws for neural language models
  5. 2020GPT-3direct

VI · Everyone · 1

  1. 2022Chinchilla: the models were undertraineddirect

Meta's LLaMA models, released to approved researchers on 24 February 2023, ranged from 7 to 65 billion parameters and had been trained on more than a trillion tokens of public data, far past the Chinchilla-optimal point, so that a small model would be as good as possible at inference time. The 13-billion version matched GPT-3 on most benchmarks and ran on one graphics card. Within a week the weights were on BitTorrent.

What followed was the fastest bloom of derivative work in the field's history. Stanford's Alpaca fine-tuned the 7-billion model on instructions for $600; Vicuna, Koala and hundreds of others followed; Georgi Gerganov's llama.cpp ran the models on a MacBook and then a phone; the quantisation, LoRA fine-tuning and serving tools of the open ecosystem were built in months around a model that was technically not licensed for any of it. In July 2023 Meta released Llama 2 with a licence permitting commercial use, and Llama 3 in 2024.

LLaMA settled the shape of the industry into closed frontier models from a few laboratories and open weights, mostly from Meta, Mistral and the Chinese labs, a step behind. The DeepSeek releases of 2024 and 2025 that shook the markets came from that second tradition.

what it led to · 5 events downstream, through 2025

Built on it directly:

  1. 2024DeepSeek-V3 trained for $5.6 millionVI

And, through them, by era:

VII · Agents · 4
  1. 2025DeepSeek-R1
  2. 2025Nvidia is worth four trillion dollars
  3. 2025Gold at the Mathematical Olympiad
  4. 2025Gemini 3

sources · 2

Trace the lineage of this event on the timeline →Back to the ledger