What word would you put in this blank?
She spread the toast with ___.
Maybe you guessed butter. The writer used jam. Both make sense. To score your prediction, we look at how much chance you gave the word that actually appeared.
A language model is a computer program that learns patterns from examples of text. Perplexity measures how well it predicts the next piece of text. The more chance it gives the pieces that actually appear, the lower its score.
Say you gave jam a 50% chance. When jam appears, that's the same chance you'd have given it if you were choosing between 2 equally likely answers. At 25%, it's like choosing between 4. At 10%, it's like choosing between 10.
Across a whole test, perplexity gives you one number in those same units. A score of 2.7 means the model did about as well as a guesser facing 2.7 equally likely choices at each step. The decimal comes from combining many predictions. The model itself can consider thousands of possible next pieces.
You and the steady guesser are filling in the same toast sentence. The steady guesser gives jam a 50% chance. Use the slider to change the chance you give jam, then compare your perplexity scores.
<!doctype html>
<html lang="en">
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>The toast quiz</title>
<style>
.prompt { margin: 0 0 .8rem; }
.adjust { display: block; }
.adjust input { display: block; width: 100%; max-width: 24rem; padding: 0; margin: .35rem 0; accent-color: var(--sdoc-app-accent-color); }
.results { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 1rem; margin: .9rem 0 .3rem; }
.results div { padding: .6rem .8rem; background: var(--sdoc-app-surface); border-radius: var(--sdoc-app-radius); }
.results dt { font-weight: 600; }
.results dd { margin: 0; font-size: 1.55em; font-variant-numeric: tabular-nums; }
.odds { color: var(--sdoc-app-muted-color); font-size: .9em; }
.outcome { margin: .4rem 0 0; font-size: .9em; }
</style>
</head>
<body>
<p class="prompt">She spread the toast with <strong>jam</strong>.</p>
<label class="adjust">
Chance you give jam <strong class="chance" aria-hidden="true">90%</strong>
<input type="range" min="1" max="100" value="90" aria-valuetext="90 percent">
</label>
<dl class="results" aria-label="Perplexity scores">
<div><dt>Steady <span class="odds">(50%)</span></dt><dd>2.00</dd></div>
<div><dt>Your guess</dt><dd><output class="score" aria-live="off">1.11</output></dd></div>
</dl>
<p class="outcome" role="status">Your guess scores better. Lower is better.</p>
<script>
const slider = document.querySelector('.adjust input');
const chance = document.querySelector('.chance');
const score = document.querySelector('.score');
const outcome = document.querySelector('.outcome');
function update() {
const percent = Number(slider.value);
const perplexity = 100 / percent;
let message = 'Both guesses score the same.';
if (percent > 50) message = 'Your guess scores better.';
if (percent < 50) message = 'The steady guesser scores better.';
chance.textContent = percent + '%';
slider.setAttribute('aria-valuetext', percent + ' percent');
score.value = perplexity.toFixed(2);
outcome.textContent = message + ' Lower is better.';
}
slider.addEventListener('input', update);
update();
</script>
</body>
</html>
At 50%, both guessers score 2.00. Giving jam a 90% chance lowers your score to 1.11 because jam was the word the writer used.
For one blank, divide 1 by the chance you gave the word that appeared.
| Chance for jam | Perplexity |
|---|---|
| 50% | 1 ÷ 0.50 = 2.00 |
| 90% | 1 ÷ 0.90 ≈ 1.11 |
On a longer test, the score combines the chances for every piece of text that appears.
During pretraining, the first major stage of learning, the model practises predicting the next piece over and over across a huge amount of material. It adjusts based on how much chance it gave the piece that came next.
Those pieces are called tokens. A token might be a whole word, part of a word, or punctuation. To measure perplexity, testers give the model text set aside for the test. At each step the model sees the text so far and assigns chances to possible next tokens. The tester records the chance it gave the token the writer actually used, reveals that token, and goes on. Each new guess starts with the actual text up to that point.
A compressor packs a message into less space so the receiver can unpack it and get the exact original back.
Imagine you and a friend are sending breakfast orders. You agree on three signals.
| Topping | Signal |
|---|---|
| Jam | 0 |
| Butter | 10 |
| Honey | 11 |
You expect jam to come up most often, so you give it the shortest signal. If the orders are jam, jam, butter, jam, you send 0 0 10 0. That's 5 digits. Giving every topping a two-digit signal would take 8 digits for the same orders. Your friend can still recover all four orders exactly. A 0 on its own means jam, and a 1 tells them to read one more digit.
A computer calls each of those 0s and 1s a bit. A language model supplies the chances for what comes next, and a separate program uses those chances to pack the exact text into bits. Better predictions can mean fewer bits. That's why OpenAI's article calls a predictor a compressor. Patterns it learned from one set of examples can help it pack new text, too.
Jakub Pachocki, OpenAI's chief scientist, reports that GPT-3.5 scored about 0.99 and GPT-4 about 0.91 on a fixed collection of OpenAI's own software code set aside for testing. The article uses log-perplexity (also called cross-entropy loss here). That's a different scale for the same score. On the quiz's scale, those numbers are about 2.69 and 2.48. Lower is better on either scale. If the model gave every piece that appeared a 100% chance, it would score 0 on the article's scale and 1 on the quiz's scale. Perplexity and Intelligence, OpenAI
Picking up patterns in how software code works can help a model predict how a line of code will finish. Those same patterns may help it write or fix code. Pachocki argues that lower scores can accompany new abilities, and proposes using perplexity alongside other tests to help decide when to slow down development for safety. To compare perplexity scores fairly, use the same test material and the same way of splitting it into tokens. To learn whether a model gives true answers, solves a particular problem, or behaves safely, test those things directly. Perplexity and Intelligence, OpenAI
Default for code blocks, blockquotes, and charts. Override individually below.
npm link from project root~/.sdocs/styles.yamlDownload as PDF document
One slide per landscape page with selectable text
Editable shapes and text in PowerPoint / Keynote / Slides
Download as Word document
Plain markdown, no styles
Markdown with YAML front matter styles
Copy install promptcurl -fsSL https://smalldocs.org/install | sh
Discord
GitHub
For business
You agree to our Terms