Prompting · Few-shot inference
In-Context Learning
A frozen autoregressive model can use task demonstrations—examples in the prompt—to figure out what to do without changing any of its parameters.
Evidence & limits
- What the further reading establishes
- Larger autoregressive language models do much better on tasks given through zero-, one-, and few-shot prompts.
- What this lab runs
- You'll run a real quantized 135M model locally on the same fixed classification cases with three different prompt setups.
- What it does not prove
- The local model isn't GPT-3, and a two-case browser test can't recreate the paper's benchmark claims.
Dataset
Opaque Review Labels
- Source
- Course-authored synthetic evaluation set
- License
- Not separately licensed
- Size
- 4 demonstrations · 2 held-out cases
Further reading
- Language Models are Few-Shot Learners
Primary · Tom Brown et al. · 2020
Popularized the modern zero-, one-, and few-shot setup for large autoregressive models.
- Rethinking the Role of Demonstrations
Paper · Sewon Min et al. · 2022
Tests which parts of prompt examples actually help classification performance.
- What Can Transformers Learn In-Context?
Paper · Shivam Garg et al. · 2022
Studies in-context learning as a learning algorithm using controlled function tasks.
Lesson progressRestoring progress…
- CodeRestoring
- ExperimentRestoring
- CheckRestoring
Summary
The weights stay frozen. In-context learning doesn't run gradient descent on the examples in the prompt. The model reads the instruction, examples, and query as one causal token sequence. Those earlier tokens change the hidden activations and KV cache used to predict the next token, but every learned weight stays fixed.
Examples show the model what you mean. Demonstrations can spell out an output format, label mapping, style, or latent task that the instruction leaves unclear. Their order and formatting change the prompt prefix, so they can also change the model's output probabilities.
Keep the comparison fair. For a fair comparison, change only the number of examples. Keep the instruction, held-out queries, decoding settings, label extractor, and exact-match metric the same. In this browser experiment, a provided local evaluator runs that whole comparison. It doesn't import or run your prompt and scoring functions; the IDE checks those separately. If you pick prompts after seeing the test results, you're quietly tuning on the test set.
infer mapping · return K or MSame test questionsmoving story · tedious storyinstruction → query0 demonstrationsOne-shotinstruction → 1 example → query1 demonstrationFew-shotinstruction → 4 examples → query4 demonstrationsprompt tokens change activations + KV cacheweights updated: 0| Condition | Moving / K | Tedious / M | Exact match |
|---|---|---|---|
| 0 examples | prediction / K | prediction / M | ? / 2 |
| 1 example | prediction / K | prediction / M | ? / 2 |
| 4 examples | prediction / K | prediction / M | ? / 2 |
Model size matters. The paper's best results come from a model far larger than anything this browser can run. Two held-out items can show whether examples changed this local model's answers. They can't prove that few-shot prompting usually improves accuracy or recreate GPT-3's benchmark results.
Knowledge check
Implementation
Format labeled examples without changing their order.
- Signature
def format_demonstrations(examples):- Inputs
- examples list[{input: str, label: str}]
- Returns
- str containing the records in input order
- Rule
trim each field; format `Input: …\nLabel: …`; join records with one blank line- Example
[{input: " aa ", label: "K"}] → "Input: aa\nLabel: K"
Reference solution
Approach Format labeled examples without changing their order.
def format_demonstrations(examples): records = [ f"Input: {example['input'].strip()}\nLabel: {example['label'].strip()}" for example in examples ] return "\n\n".join(records)Build predictable prompt formatting and scoring functions in Python. The IDE checks your functions on their own. The experiment below uses a separate fixed evaluator, so changing these practice cells can't change the comparison it reports.
Saved results
Results created after your saved code passes its checks
Saved results
Results created after your saved code passes its checks
The replay is course data. The validation result is tied to the code you saved and checked.
Loading…