Answers & rationales

Project A (house email style) is an SFT problem: collect demonstration pairs — prompts with gold house-style emails — and "grade" implicitly by imitation (the loss matches the demonstration). There is no objective right answer to score, so you teach by example. Project B (competition math with known answers) is an RL problem with a verifier: collect prompts with ground-truth answers, sample rollouts, and grade each by a deterministic check (does the extracted answer match?). They differ because math has checkable ground truth that can reward exploring new solution paths, whereas style has no ground truth — only examples to imitate. Common wrong answer: "collect human preference pairs for both" — style is demonstration-SFT (no preferences needed) and math already has a verifier, so preference labeling wastes effort and introduces a gameable proxy where a clean check exists.

Reasoning comes in two moves: chain-of-thought SFT teaches the model to show step-by-step work (imitating reasoning traces), then RL with verifiable rewards lets it discover novel strategies beyond the demonstrations — DeepSeek R1 is the canonical example, where GRPO on math/code verifiers produced reasoning the SFT data never showed. Safety comes from Constitutional AI / RLAIF: a written constitution drives the model to self-critique and revise its outputs and to generate its own preference labels, so harmlessness scales without armies of human annotators. The throughline: pre-training gives raw capability; post-training shapes it into reasoning format and safe behavior. Common wrong answer: "reasoning and safety come from a bigger pre-training corpus" — scale helps raw capability, but the step-by-step reasoning habit and the safety alignment are post-training behaviors, instilled by SFT+RL and Constitutional AI respectively.

Pre-training optimizes broad next-token prediction over internet-scale text (knowledge + language). Mid-training continues on curated data to sharpen fluency, add modalities (code/math), and extend context. Post-training (SFT + RL) optimizes targeted behavior: instruction-following, safety, reasoning, tool use. The common mistake is to conflate mid- and post-training — mid-training is still self-supervised on curated corpora, whereas post-training optimizes behavior against demonstrations or a reward signal.

Start from the base model (DeepSeek-V3, broad capability). (1) Cold-start CoT SFT — a small, high-quality set of reasoning traces bootstraps the "think step by step" format. (2) RL with verifiers (GRPO) on math/code — discovers reasoning strategies beyond the demonstrations, graded by deterministic correctness, no learned reward model. (3) Synthetic reasoning data + filtering — the improved model generates ~600K reasoning examples, aggressively filtered for correctness. (4) Final SFT mix (~800K, ~75/25 reasoning/non-reasoning) — rebalances so reasoning gains do not regress general capability. (5) Deploy, plus distillation into smaller checkpoints. Each stage adds one thing: format → capability → data scale → balance → reach. Common wrong answer: "base → SFT → RLHF with a reward model" — R1's signature is verifier-based RL (not a learned reward model for the reasoning) plus a heavy synthetic-data generate-and-filter loop, not vanilla RLHF.

SFT first, then RL. Phase 1 (SFT on the 800 examples) is stable and quickly teaches valid SQL syntax/structure — expect reliable formatting in 1–2 epochs. Phase 2 (RL with the execution checker as a deterministic verifier) pushes past the demonstration ceiling, discovering correct handling of joins, NULLs, and edge cases the 800 examples never covered. Justification via the tradeoff: SFT alone is bounded by demonstration coverage; RL alone is unstable for format-heavy tasks early on. The common wrong answer is "RL only, since we have a verifier" — it wastes the cheap stability SFT provides and risks slow, unstable convergence on syntax.

1. RL only
2. SFT then RL
3. SFT only ✓
4. Neither — collect more data first

Correct: SFT only

Data: multi-turn chat pairs plus chain-of-thought traces showing the diagnostic reasoning (symptom → ranked causes → next step), and refusal/guardrail examples for out-of-scope medical questions. Grading: this task is only partly verifiable, so SFT on curated CoT is the backbone; for any RL, prefer an LLM-as-judge on a safety+helpfulness rubric over a single scalar, and monitor for reward hacking (e.g. over-cautious boilerplate). The common mistake is reaching for a deterministic verifier — triage quality is subjective, so there is no pass/fail oracle, and a naive reward invites gaming.

Diagnosis: reward hacking (Goodhart's law). The judge rewards a proxy — confident, citation-shaped, fluent prose — and the policy maximized that proxy while factual accuracy fell. The saturating reward with falling held-out accuracy is the tell. Fixes: (1) add a verifiable component to the reward (retrieval-grounded fact checks) so correctness dominates the judge score; (2) add a KL penalty to the SFT baseline to limit drift into the judge's blind spot; (3) ensemble or periodically re-calibrate the judge, and monitor output diversity / hedging rate. The common wrong answer is "train longer / raise the reward weight" — that accelerates the hack.

1. Increase the reward weight so the signal is stronger
2. Train for more steps to escape the collapse
3. Raise the sampling temperature to force more diverse rollouts
4. Strengthen the KL-divergence penalty against the reference policy ✓

Correct: Strengthen the KL-divergence penalty against the reference policy

No — a flat refusal over-weights harmlessness and violates helpfulness (and arguably honesty, since chemical safety is legitimate knowledge). Under Constitutional AI the goal is the response that maximizes all principles simultaneously: explain general lab-safety principles (ventilation, PPE, incompatible-reagent storage), point to authoritative sources (OSHA, the SDS), and decline only the genuinely dangerous specifics. Key idea: over-refusal is a constitutional violation just as over-compliance is — the skill is the balance point, not blanket caution.

Both start from a pre-trained base and use SFT, but their RL differs. DeepSeek R1 leans on RL with **deterministic verifiers** (GRPO on math/code correctness) and even demonstrated RL-only reasoning emergence (R1 Zero), then used rejection-sampling SFT to re-stabilize. LLaMA 3 uses **RLHF with PPO** against a human-preference reward model for general alignment, with iterative human eval and data refresh. One key difference: R1's reward is largely machine-checkable correctness (low reward-hacking surface); LLaMA's is a learned preference model (more subjective, more hacking surface). The common mistake is calling both "RLHF" — R1's reasoning RL is reward-from-verifier, not from human preferences.

1. It lacks world knowledge until it is fine-tuned
2. It predicts fluent tokens but isn't instruction-following or aligned ✓
3. It is too large to serve without quantization
4. Its weights are randomly initialized until post-training

Correct: It predicts fluent tokens but isn't instruction-following or aligned

Loss at this position is −log(0.2) ≈ 1.61 nats. The logit gradient is p_i − 1[i=y]; for the correct class that is 0.2 − 1 = −0.8 (magnitude 0.8), a strong corrective push. Contrast a confident-correct token at p=0.95: loss −log(0.95) ≈ 0.05 and gradient 0.95 − 1 = −0.05 — a gentle nudge. The point: the update self-scales with how wrong the model is, so most learning is spent on uncertain tokens. Common wrong answer: forgetting the indicator term and reporting the gradient as just p (0.2).

Random splitting lets near-duplicate examples land in both train and test, so the test score partly measures memorization and is inflated. Use a time-based split (e.g. train on 2024, test on 2025) plus a held-out out-of-distribution set to measure real generalization. The RL-specific rule: the RL test set must be graded by a *different* reward model than training used — otherwise you measure how well the policy games a familiar scorer, not whether it improved. Common wrong answer: "shuffle and split 80/20," which is exactly the leakage trap.

Deduplication removes not just identical examples but *similar* ones, which otherwise leak across splits and let the model memorize rather than generalize. The standard pipeline is exact-hash → MinHash LSH (approximate near-duplicate detection) → manual review; frontier labs discard ~99% of candidate data here. Common wrong answer: "dedup only removes exact duplicates" — the value is in catching near-duplicates at scale.

The LoRA update is scaled by α/r, so it is the *ratio* that sets adaptation magnitude. Going r=8→16 while holding α=16 changes the ratio from 16/8=2 to 16/16=1 — you've silently *halved* the effective update strength, so the more expressive higher-rank adapter may actually adapt *less*, confounding your experiment. Fix: scale α with r (set α=32 to keep α/r=2), or use a rank-stabilized variant. Common wrong answer: "higher rank always means stronger adaptation" — true for capacity, false for magnitude once α/r drops.

Full update of that matrix: d² = 4096² ≈ 16.78M parameters. LoRA trains A (d×r) and B (r×d), so 2·d·r = 2·4096·16 = 131,072 parameters ≈ 0.78% of the full matrix (131,072 / 16,777,216). So LoRA at r=16 trains under 1% of one projection's parameters. Two checks people miss: A and B together are 2dr (not dr — both matrices are trained), and the saving scales with how small r is relative to d. Common wrong answer: computing d·r (one matrix) and reporting ~0.39%.

1. Train for more epochs
2. Reduce the learning rate by ~10× ✓
3. Increase the learning rate to escape the plateau
4. Increase the batch size to stabilize the gradient estimate

Correct: Reduce the learning rate by ~10×

Loss is computed only on completion tokens; the prompt is masked. Without masking, the model spends gradient updates learning to reproduce the prompt — which it already receives as input — diluting the signal for the actual task. For instruction data the prompt is often ~80% of the sequence, so unmasked training misallocates ~80% of updates to prompt reproduction. In TRL: completion_only_loss=True. Reasonable exception: few-shot in-context tasks where you *want* the model to learn prompt patterns — there a partial mask (mask the system prompt, keep the examples) is the middle ground. Common wrong answer: "masking saves compute" — the forward pass still covers all tokens; masking changes *where the gradient flows*, not the FLOPs.

Halve the per-device batch size to 16 and set gradient_accumulation_steps=2. The effective batch size stays 32 (16 × 2), so the optimizer sees the same averaged gradient and training dynamics are unchanged — only peak activation memory drops, because you process 16 examples at a time and accumulate. Keep batch sizes powers of two for memory alignment. Common wrong answers: "just lower the batch to 16" (changes the effective batch and thus the learning dynamics / effective LR), or "lower the learning rate" (doesn't address the memory limit at all).

1. Full fine-tuning with gradient checkpointing to save memory
2. LoRA (r=8) with the base loaded in fp16
3. QLoRA — 4-bit NF4 base with 16-bit adapters ✓
4. 8-bit LoRA — base loaded in int8, adapters in 16-bit

Correct: QLoRA — 4-bit NF4 base with 16-bit adapters

Loss = −log σ(r(y_w) − r(y_l)), where y_w is the preferred (winning) output and y_l the dispreferred. It trains the reward model to score the preferred output higher by a margin (the sigmoid of the score *difference*). Crucially it learns **relative** preference, not an absolute quality score — which matches what humans can reliably label (A-vs-B is far more consistent than "rate this 0–10"). Common wrong answer: claiming the reward model predicts an absolute goodness; it only ever sees and learns differences.

1. DPO requires a separate reward model; PPO and GRPO do not
2. DPO simply fine-tunes on the preferred outputs, while PPO/GRPO add a reward signal
3. DPO uses a critic and GRPO a group baseline, but they are otherwise identical
4. DPO trains offline on a fixed preference dataset; PPO/GRPO generate fresh rollouts online ✓

Correct: DPO trains offline on a fixed preference dataset; PPO/GRPO generate fresh rollouts online

Both are a loss of generality, but from different pressures. Catastrophic forgetting is an **SFT/fine-tuning** failure: training on a narrow dataset overwrites broadly-useful pre-trained weights, so the model loses capabilities it used to have. Mitigations: LoRA (caps weight change), mix in general data, keep epochs low. Mode collapse is an **RL** failure: under reward pressure the policy converges onto a single high-reward output pattern, losing diversity (all responses look alike). Mitigations: KL penalty to the reference, entropy bonus, group-relative methods (GRPO). Common wrong answer: treating them as the same thing — they share the symptom (lost generality) but differ in cause (data overwriting vs reward concentration) and fix.

The four: (1) policy — the LLM being trained; (2) reference — a frozen copy, for the KL penalty; (3) reward model — scores generated outputs; (4) value model (critic) — estimates expected reward for advantage/GAE. GRPO removes the **value/critic** model, replacing it with a group-mean baseline, so it runs with three. The most-forgotten one is the **reference** model — omitting it in the answer signals you've never wired RLHF. Note: with verifiers instead of a learned reward model, GRPO can drop further (no reward model either).

Group baseline = mean = (1+1+0+0)/4 = 0.5. Advantages = r − baseline = [+0.5, +0.5, −0.5, −0.5]: the two correct rollouts get a positive push (more likely), the two wrong ones negative. The zero-gradient case: if all G rollouts score the *same* (all 1s or all 0s), the baseline equals every reward, so every advantage is 0 and the prompt contributes no gradient — which is why GRPO wants prompts of intermediate difficulty (some pass, some fail). Common wrong answer: using a fixed 0.5 baseline regardless of the group, or forgetting that uniform groups vanish.

The KL term −β·KL(π_θ ‖ π_ref) keeps the trained policy close to the frozen reference, so RL maximizes reward *without* drifting into degenerate regions — it's the main guard against reward hacking and mode collapse. If β is too small, the policy is under-constrained: it drifts far from the reference and games the reward (e.g. collapsing onto one high-scoring pattern). If β is too large, the policy can barely move and never improves. So β trades exploration against stability. Common wrong answer: "the KL term improves the reward" — it does the opposite locally (it's a penalty); its job is to keep gains *honest*.

This is mode collapse: under reward pressure the policy has converged onto a single high-reward pattern and lost diversity, while the *measured* reward keeps climbing because the reward model/verifier rewards that pattern. Diagnostic signals: collapsing output diversity, a rising KL from the reference, reward up but human/held-out quality flat or down. Mitigations: increase the KL penalty (β), add an entropy bonus, or use a group-relative method (GRPO) whose within-group comparison preserves spread. Common wrong answer: "train longer" or "raise the learning rate" — both accelerate the collapse.

1. It caps how far an action's probability moves from the old policy ✓
2. It clips the reward to a fixed range to prevent reward hacking
3. It limits the gradient norm to prevent exploding gradients
4. It truncates long sequences to fit the context window

Correct: It caps how far an action's probability moves from the old policy

1. GRPO with the verifier as reward — no critic and no learned reward model to host ✓
2. PPO with the verifier as reward, keeping the learned critic for token-level credit
3. GRPO but drop the KL term, since a verifiable reward is exploit-proof
4. DPO on verifier-labeled pass/fail pairs, refreshed each round

Correct: GRPO with the verifier as reward — no critic and no learned reward model to host

The alignment tax is the gap between the automated reward-model score and actual human preference. A *growing* tax means the model's reward keeps rising while real quality doesn't — it's optimizing the proxy without improving the true objective, i.e. Goodhart's Law / reward hacking in progress. It's a primary RL health signal precisely because reward-going-up looks like success; the tax reveals it's hollow. Corroborate with KL divergence (a spike past ~0.5 nats) and collapsing rollout diversity, and act: raise the KL penalty, fix/strengthen the reward (or switch to a verifier), or early-stop on a held-out human/quality metric rather than reward. Common wrong answer: "a growing tax means the reward model needs retraining" — retraining may help, but the immediate read is that you're over-optimizing the proxy.

Embedding failures (with a sentence transformer) and clustering them (k-means) surfaces structural patterns that manual inspection misses at scale — it groups the dozens of "looks different but fails the same way" cases so you can see the real failure modes and their relative size. The same embedding space then drives *data selection*: take each cluster's centroid, find the nearest existing training examples by cosine similarity, and seed targeted counterexamples from those neighbors. That closes the loop from diagnosis to curation — why ten similarity-matched examples often beat a hundred random ones. Common wrong answer: "clustering just visualizes errors" — its real payoff is choosing *which* training data to add.

Frequency alone says fix reasoning (30 > 12), but prioritize by impact ÷ effort. Format errors (right answer, wrong format) are cheap and certain to fix — ~20–50 targeted SFT examples, an hour of work, and all 12 disappear. Reasoning errors (wrong approach entirely) are the hardest class: they need new reasoning chains in the training data, days-to-a-week of effort, and only a partial fix. So you bank the cheap, certain win (format) first, then invest in the high-effort reasoning fix. The principle: rank by frequency × severity ÷ effort, not raw frequency. Common wrong answer: "always fix the biggest bucket first" — that ignores that a small, trivially-fixable bucket can be the higher ROI.

Because if the capability isn't in your eval suite, you can't *measure* whether training improved it — you'd be optimizing blind, unable to tell progress from regression or to catch the fix introducing new failures. Adding it first gives a baseline and a target the whole eval-train loop can steer by. The canonical example: when GPT-4o was found to be sycophantic, the fix began by adding sycophancy to the eval suite, *then* training against it — "evals before fixes, always." Common wrong answer: "train first, then build an eval to confirm" — that risks training in the dark and only discovering the regression after it ships.

1. Exact-match against a single reference answer
2. The held-out training-loss curve
3. Pass@k with a deterministic verifier
4. LLM-as-judge with a calibrated, human-checked rubric ✓

Correct: LLM-as-judge with a calibrated, human-checked rubric

1. Human preference labels are inherently noisy, so the reward is unreliable
2. The reward is a proxy; pushed too hard, the policy exploits the proxy-vs-true-objective gap ✓
3. Reward models are too small to represent quality, so they underfit
4. The KL penalty makes the reward signal non-stationary during training

Correct: The reward is a proxy; pushed too hard, the policy exploits the proxy-vs-true-objective gap

1. A jailbreak, since the model's safety guardrails were bypassed
2. A hallucination, since the model invented the instruction
3. A prompt injection, since the instruction arrived through the data ✓
4. An over-refusal, since the model should have declined the task

Correct: A prompt injection, since the instruction arrived through the data

1. Reward hacking or mode collapse — the policy is drifting to game the reward ✓
2. Healthy exploration — a high KL means the model is learning quickly
3. The KL penalty β is too large and should be reduced
4. Underfitting — the model needs many more training steps

Correct: Reward hacking or mode collapse — the policy is drifting to game the reward

Pass@1 = c/n = 30/100 = 0.30. Pass@10 = 1 − C(70,10)/C(100,10) = 1 − ∏_{i=0}^{9}(70−i)/(100−i) ≈ 1 − 0.022 ≈ 0.97. So the model produces a correct answer ~30% of the time single-shot but ~97% within ten tries. A large Pass@1↔Pass@10 gap means the capability is *there* but sampling/decoding is unreliable — the fix is best-of-N or majority voting (or temperature tuning), NOT more training. If Pass@1 ≈ Pass@10 instead, the model genuinely doesn't know the answer and needs training. Common wrong answer: treating a low Pass@1 as "needs more data" without checking Pass@k first.

Probe multiple attack surfaces, not just capability: (1) jailbreaks — adversarial user prompts that bypass guardrails (roleplay bypass); (2) prompt injection — malicious instructions hidden in the data/tools the agent reads (the higher-risk vector for a tool-using agent); (3) tool misuse — getting the agent to call tools with harmful parameters or chain them into damage; (4) automated adversarial generation — use one LLM to mass-produce attacks (garak/promptfoo) beyond manual effort. Watch for agentic misalignment (pursuing goals against user intent) and sleeper-agent behavior (passes evals, triggers on a condition). Defense-in-depth: pair training-time alignment with a runtime monitor — a separate model screening each output/action before it executes. Common wrong answer: only listing jailbreaks — for an *agent* the data/tool vectors matter more.

Three signals that the reward is being gamed rather than the task improved: (1) reward score rises while a held-out human/quality metric stays flat (the alignment tax grows); (2) rollout diversity collapses — outputs converge on one template; (3) output length (or another spurious feature the reward correlates with) grows monotonically. The single confirming metric is **KL divergence** from the reference policy: a spike past ~0.5 nats shows the policy has diverged far to chase the proxy. The unifying habit: always plot accuracy/quality *alongside* reward — when they diverge, it's Goodhart. Common wrong answer: "rising reward proves improvement" — rising reward is exactly what reward hacking also produces.

Task A (SQL) → verifier: execute the generated query and compare result rows to ground truth. High reliability (deterministic), cheap to run, low Goodhart risk because it checks the real outcome, not a proxy. Task B (email helpfulness) → learned reward model or LLM-as-judge: "helpful" has no ground truth to check, so you need a learned proxy — but it drifts, costs more to train and host, and carries higher Goodhart risk, so cross-check it against ~100 human ratings and watch the alignment tax. Principle: verifiers where ground truth exists; a calibrated judge where it does not. Common wrong answer: "use a learned reward model for both — it is more flexible" — that throws away SQL's checkable ground truth and invites reward hacking where a clean, cheap verifier already exists.

A verifier is a deterministic ground-truth check (test-suite pass, exact match): highly reliable, reproducible, cheap to run, and essentially ungameable (low Goodhart risk) — but it only exists where there's a checkable answer (math, code, format). A learned reward model scores subjective, open-ended quality (helpfulness, tone) that no verifier can reach — but it drifts, is expensive to train and host, and is a gameable proxy (high Goodhart risk). So: use verifiers wherever ground truth exists (the DeepSeek R1 stance); fall back to a learned reward model or a calibrated LLM-as-judge for subjective quality, cross-checked against humans, and always watch the alignment tax. The honest framing isn't "verifiers win" — it's "verifiers where you can, judges where you must, monitor either." Common wrong answer: claiming verifiers are strictly better — they can't grade most of what users actually care about.

1. The supervised loss recorded during the earlier SFT stage
2. The tokenizer's vocabulary coverage on the evaluation set
3. The KL divergence from the reference policy ✓
4. The warmup length of the learning-rate schedule

Correct: The KL divergence from the reference policy

This is a strong case for Constitutional AI and RLAIF. The constraint is exactly the one CAI was built for: human preference labeling is expensive and does not scale to thousands of scenarios, while a written constitution lets the model critique and revise its own outputs and generate its own preference labels at far lower cost and with more consistency. The safeguard to keep: calibrate the AI feedback against a human-labeled validation set. AI feedback can drift systematically from human values, and a model judging itself can entrench its own blind spots, so spend the limited human budget not on labeling everything but on auditing whether the AI's judgments track human preference. Common wrong answer: "avoid RLAIF and just label what you can by hand" — a few hundred hours cannot cover thousands of scenarios, so you would get thin, inconsistent coverage where CAI plus a human-validated check gives broad coverage with a drift guard.

Collapse detection flags and removes near-identical synthetic outputs — measured by embedding similarity or deduplicated response prefixes — that appear when the generator falls into a repetitive mode. Without it, a dataset can look large because it has many rows, but those rows are near-duplicates the model simply memorizes, so effective diversity (and the learning it would provide) is far lower than the row count suggests. The failure is silent: headline volume rises while real coverage stalls. It is distinct from catastrophic forgetting, which is about losing prior capability rather than generating duplicates. Common wrong answer: "collapse detection removes low-accuracy examples" — that is rejection sampling on a quality axis; collapse detection targets redundancy, not correctness.

Allocate examples proportionally to the failure distribution, weight by severity, and source each from ground truth. Hallucinated APIs (45%) is the largest and most damaging bucket — build roughly 120 examples pairing prompts with verified, runnable code that uses real library calls, drawn from official docs so the correct signal is authoritative. Off-by-one bounds (35%) — about 90 examples of correct loop-boundary handling, ideally paired with a unit-test verifier so RL can reinforce them. Missing error handling (20%) — about 60 examples demonstrating proper try/except and edge-case checks. Then re-run the full eval suite, not just these three clusters, to confirm the additions did not regress anything else. Common wrong answer: "add a few thousand general high-quality coding examples" — volume without targeting mostly re-teaches what the model already does well and leaves the specific failures unfixed.

Rejection sampling fails when the threshold is set on a single axis: filter only on length or fluency and you keep confident, well-written, wrong answers while biasing the distribution. LLM-as-judge fails on the judge's own biases — it tends to reward verbose, confident, or self-similar outputs, so a fluent wrong answer scores well. Majority vote fails when a shared misconception makes the wrong answer the consensus, so the samples agree and reinforce the error. The implication: no single filter is sufficient — combine them on orthogonal axes (a correctness check the judge cannot fake, a diversity or collapse check, a consensus check) so a candidate must clear several independent hurdles. Common wrong answer: "use the strongest LLM judge and trust it" — a single judge concentrates its biases instead of cancelling them.

1. Reward hacking, where the model found an exploit in an automated grader
2. Over-generalization of the narrow counterexample set — catastrophic forgetting of uncovered behavior ✓
3. The 150 examples were too few to change behavior, so the new refusal is unrelated noise
4. Test-set leakage into the counterexample data, which inflated the apparent fix

Correct: Over-generalization of the narrow counterexample set — catastrophic forgetting of uncovered behavior

1. It removes the reward model and optimizes the policy directly from raw text
2. It replaces reinforcement learning with supervised fine-tuning on revised outputs
3. It restricts training to verifiable tasks where a deterministic checker exists
4. The reward model's preference labels come from an AI judge scoring against a constitution, not humans ✓

Correct: The reward model's preference labels come from an AI judge scoring against a constitution, not humans

Start around 10K to 20K prompts — the default starting band, enough signal without frontier-scale cost — and sample about 8 outputs per prompt, the standard default that balances gradient signal against compute (more outputs sharpen the estimate but cost linearly). For the reward, use a verifier rather than a learned reward model: competition math has a checkable ground-truth answer, so a deterministic numeric check is cheaper, reproducible, and ungameable, with no reward-model drift to track. Scale prompts toward 100K only after the pipeline shows gains. Common wrong answer: "train a reward model on human ratings of the solutions" — that adds cost and a gameable proxy for a task that already has objective ground truth a verifier can check directly.

1. A rollout gives only a sparse scalar reward, so each example teaches less than an SFT pair — you need many more ✓
2. Rollouts are longer sequences, so they consume more tokens per example and train more slowly
3. RL optimizers require smaller learning rates than SFT, which slows convergence
4. Reward models are noisiest early in training, which delays downstream RL progress

Correct: A rollout gives only a sparse scalar reward, so each example teaches less than an SFT pair — you need many more

Start small — about 20 to 50 examples. The model already knows the content; you are only changing the output shape, which sits at the bottom of the SFT scale ladder where formatting and structure are learned. Spending thousands of examples here wastes annotation budget and risks overfitting to your exact phrasings, which can degrade the model's general ability. Fine-tune on the small set, evaluate, and add more only if the format is not reliably followed. Common wrong answer: "a few thousand, to be safe" — that confuses changing behavior with changing format, and over-trains a problem that 20 to 50 demonstrations usually solve.

Generate, filter, transform, score — with filtering as the quality-critical step. Generate: build templates with variable slots (question type, user persona, difficulty) seeded from the 200 real tickets and the API docs, sampling at varied temperature to produce thousands of candidate question-answer pairs. Filter: this is where quality is created — apply multi-axis rejection sampling (correctness against the docs, relevance, fluency) plus collapse detection to drop near-duplicates, expecting to keep only 10 to 20%. Transform: restyle survivors across tones and difficulty so the set is not monochrome. Score: assign multi-axis quality scores to weight or rank examples for the final mix. Keep the 200 real tickets in the mix as a grounding anchor. Common wrong answer: "generate 50K examples and train on all of them" — unfiltered synthetic data is dangerously easy to produce badly, and skipping the filter is the classic way to train on thousands of confident, wrong, near-duplicate examples.

The persona-varying engineer's data. Temperature only adds token-level randomness to the same response style, so the outputs cluster around one voice and risk collapsing into near-duplicates. Varying the persona slot changes the structure and register of the answer — a patient teacher explains step by step, a terse senior engineer compresses — so the model sees genuinely different ways to answer the same question and learns stylistic range. Persona variation is one of the highest-leverage diversity levers in template engineering precisely because it changes structure, not just surface tokens. Common wrong answer: "temperature is enough for diversity" — high temperature mostly produces noisier versions of one style, not structurally different examples, and pushed too high it degrades coherence.

The reward is sparse and delayed: a single signal arrives at the end of a long action sequence, so RL faces a credit-assignment problem — it cannot tell which of the ten steps caused the failure (a bad early tool call? a sound plan with one wrong final step?). Choices that make the signal more learnable: reward the final answer but add shaped, verifiable intermediate signals where you can (did each tool call succeed, was the retrieved fact used); use process- or step-level rewards on checkable sub-goals; and keep episodes short enough that credit is not smeared across dozens of steps. Common wrong answer: "give a big reward for the final answer and let RL sort it out" — with only a terminal reward over a long horizon, the gradient is too diffuse to attribute blame, so learning is slow and noisy.

1. Distillation is faster to run, and the caveat is that it needs more GPUs than RL
2. The student copies the teacher's weights directly, and the caveat is licensing
3. Distillation works for math but not code, and that is its main limitation
4. SFT on teacher traces beats the student's own RL — bounded by student capacity ✓

Correct: SFT on teacher traces beats the student's own RL — bounded by student capacity

No model change plus an accuracy drop means the world changed, not the model — diagnose the environment before retraining. Hypothesis 1 (most likely): data drift — user queries shifted (new topics, phrasing, seasonality). Test: compare embedding distributions of month-1 vs month-3 production queries; a large centroid distance confirms drift. Hypothesis 2: evaluation drift — the held-out set has gone stale relative to live traffic. Test: sample ~100 recent production queries, grade them by hand, and compare to the test-set accuracy; a gap means the test set needs refreshing. (A third candidate: an upstream infra/tokenizer change — diff configs and replay month-1 inputs.) Common wrong answer: "retrain immediately on fresh data" — that may help, but without diagnosing why it dropped you might be papering over a stale test set or an infra bug, wasting a training cycle.

Annual difference = (0.08 - 0.03) x 1,000,000 x 365 = about $18.25M/year — that is the price of choosing Model B. The single fact that justifies it: Model B's quality edge must directly drive enough revenue (or avoid enough cost/risk) to clear ~$18M — for example a premium math-tutoring product where higher accuracy raises subscriptions, or a safety-critical setting where B's lower error rate avoids large liabilities. Absent that revenue/risk link, the cheaper model wins. Common wrong answer: "B is better, so deploy B" — a quality edge that does not pay for its serving cost is a net loss at scale; serving cost is a first-class promotion gate, not an afterthought.

1. Run a fresh round of RL with a format reward
2. Fine-tune (SFT) on a few hundred correctly-formatted examples
3. Adjust the prompt or output template — a ~1-day format fix ✓
4. Retrain from the cold-start SFT stage to rebuild formatting

Correct: Adjust the prompt or output template — a ~1-day format fix

Cold-start SFT fixes formatting and language instability (pure-RL R1 Zero mixed languages and formats) — it gives RL a clean starting point. Rejection sampling plus filtering fixes quality variance — it keeps only verifier-passing traces so the next SFT learns from correct reasoning. Combining with ~200K non-reasoning data fixes capability narrowness — pure RL stayed stuck in verifiable math/code, so general helpfulness and safety data broaden it. The pipeline needs both RL and SFT because RL explores (discovers new strategies but is unstable and verifier-limited) while SFT consolidates (stabilizes and broadens but cannot discover); alternating captures the strengths of each. Common wrong answer: "just run RL longer" — more RL does not fix language-mixing or extend capability beyond verifiable domains; those need SFT consolidation and non-reasoning data.

Gates: (1) aggregate quality improves by a meaningful, statistically significant margin (confidence intervals, not noise); (2) no behavioral slice regresses past its threshold — including a "policy-accuracy" slice; (3) a hard ceiling on the policy-accuracy slice, where any regression is a blocker, since wrong policy answers are the top risk; (4) cost per query stays in budget. The hard blocker is the policy-accuracy slice ceiling: aggregate gains cannot buy back a regression on the safety-critical behavior. All gates run in a frozen test environment (same set, verifiers, hardware) so candidates are comparable. Common wrong answer: "promote if overall accuracy went up" — an aggregate metric can rise while the critical policy slice silently regresses, which is exactly the failure the slice ceiling guards against.

1. RL explores and expands the capability frontier; SFT consolidates and stabilizes what works ✓
2. SFT is cheaper than RL, so alternating spreads compute cost evenly across the run
3. RL can only run on verifiable tasks, so SFT stages exist to cover everything else
4. The stages must alternate because GRPO requires a fresh SFT checkpoint before each update

Correct: RL explores and expands the capability frontier; SFT consolidates and stabilizes what works

Suspect reward collapse (production reward hacking): the model has found a degenerate strategy that scores high on the automated reward while producing worse outputs — which is why the dashboard looks great while users complain. Confirm by checking the tells: is output diversity dropping (unique-n-gram ratio), is length growing without added information, and does a human-judgment sample diverge from the reward? Prevent it by always monitoring a human-judgment metric alongside reward, tracking diversity, and setting SLOs on both reward and user satisfaction so a divergence triggers rollback automatically. Common wrong answer: "the reward is high, so the model is fine — investigate the tickets separately" — the high reward is the symptom, and trusting the automated metric is exactly how reward collapse hides.

1. A shadow deploy uses synthetic test traffic while a canary uses real traffic
2. A shadow runs on live traffic but doesn't serve its outputs; a canary serves a small slice of users ✓
3. A canary runs offline against a frozen test set while a shadow runs online
4. A shadow deploy is only for safety tests and a canary is only for latency tests

Correct: A shadow runs on live traffic but doesn't serve its outputs; a canary serves a small slice of users

Performance (task accuracy, response-quality score, safety-violation rate), Cost (tokens per request, GPU utilization, cost per query), and Reliability (latency p50/p95/p99, error rate, timeout rate). Accuracy alone is insufficient because a model can be accurate yet too slow, too expensive, or too unreliable to ship — one that scores well but runs 3x slower or doubles serving cost can be unusable, and accuracy says nothing about latency tails or error rates that users actually feel. Monitoring all three means a regression in any dimension can trigger an alert or rollback. Common wrong answer: "just track accuracy and safety" — that misses cost and reliability, the dimensions that most often block a technically-good model from deploying.

Parametric memory is stale and unauditable: a price, date, or order status the model "remembers" from training is likely wrong and cannot be traced, whereas a tool result is current and checkable. The principle is to prefer tools over internal knowledge for factual lookups. Train the preference with SFT on tool-use transcripts that demonstrate when to call the tool, then RL that rewards correct tool invocation plus final-answer accuracy — so calling the tool and using its result is the higher-reward path. Common wrong answer: "add more facts to the training data so its memory is accurate" — facts go stale immediately and you cannot retrain for every price change; the fix is to route factual lookups to tools, not to memorize more.