Sice Huang 黄思策

01. SmartStart: An interview coach that fits on one GPU

Built for Hack Apertus 2026 (Track 2A, the FHGR challenge "AI-Powered Job Interview Coach") by team ccnewbee: Chu Jia and me. It placed third in the preliminary round.

The problem

Teenagers in Switzerland apply for apprenticeships at 14 to 16, and for most of them the interview is the first one of their lives. The challenge asked for a coach that runs a realistic practice interview in German, French or Italian, understands Swiss German input, and gives feedback that follows a fixed professional rubric: 11 assessment criteria and 27 feedback guidelines from FHGR.

Three constraints shaped everything else:

• The model was fixed. Submissions had to use the Apertus family. We used Apertus 1.5 at 8B parameters, a small model by current standards.
• The budget was fixed. Under 32 GB of GPU memory and fewer than 5 LLM calls per candidate answer.
• The users are minors. Feedback has to be constructive, must not invent things the candidate never said, and has to notice signs of distress.

A small model asked to grade an answer against 11 criteria in one shot is inconsistent. That inconsistency was the real problem to solve.

The approach

The design principle was to let the LLM do only what needs language, and make everything else deterministic.

A state machine runs the interview. The sequence from introduction through motivation, strengths, situational questions and closing is planned in code from the job posting and the candidate profile. No LLM call decides what happens next.

Retrieval calibrates the grader. Before the model assesses an answer, we retrieve expert-annotated answers to the same question from the challenge's 706 reference annotations and show them as anchors. The model no longer has to guess what "level 2" looks like; it compares. Similarity is plain character trigrams, with no embedding model, which keeps the memory budget for the LLM and handles dialect spelling reasonably well.

One call assesses, one call speaks. The assessment call returns rubric levels, an evidence quote, missing elements and a coach action (continue, probe, redirect, or support and refer). A deterministic policy turns that into the next move, with at most one follow-up per question and no repeated gaps. A second call phrases the turn in the interviewer's persona.

Rules check the final feedback. After the closing feedback is generated, a rule checker tests it against the guidelines that can be verified mechanically. If it finds violations, the model gets exactly one rewrite. Quotes are verified against what the candidate actually wrote.

How we evaluated

We wanted numbers that could tell us whether a change helped, so we built three layers:

• Assessment accuracy against expert labels. 100 answers with reference annotations, 25 each in German, French, Italian and Swiss German. We measured whether the predicted quality level and coach action matched.
• An ablation. The same 100 answers with retrieval switched off, to isolate what the anchors contribute.
• End-to-end simulated interviews. All 28 challenge scenarios, scored by an LLM judge. Before trusting the judge we validated it on 29 pairs of good and poor feedback from the challenge data, where it picked the better one 89.7% of the time.

Results

Retrieval is what makes the small model usable as a grader: the choice of coaching action nearly doubled in accuracy. Asking the same question in German, French and Italian led to the same follow-up decision 87 to 93% of the time, and repeat runs were fully consistent.

What the numbers do not show

• German was the weakest language (action accuracy 0.76, exact level 0.52), although it is the one most candidates will use.
• The LLM-judge scores are near the ceiling (about 4.96 out of 5 on every dimension). A judge that gives almost everything full marks cannot rank systems, so I treat those scores as a sanity check and not as evidence.
• Distress detection was measured on two examples. Recall went from 0 to 1.0, but with n = 2 that is an anecdote.
• The reference annotations are synthetic, provided by the challenge. Agreement with them is not the same as agreement with a human career counsellor.
• We gave up accuracy on purpose late in the build. The final version adds order-verified quotes and gap-targeted follow-ups, and action accuracy moved from 0.83 to 0.80 as a result. We judged that a coach which never misquotes a teenager was worth three points.

What I would do next

Test with real students and a career counsellor, since every number above is against synthetic or model-generated judgments. Replace the trigram retriever with a small multilingual embedding model and measure whether German improves. And build a harder judge, or drop it in favour of pairwise comparison, so that end-to-end quality can be distinguished.

References

1. Source code, data schema and evaluation scripts
2. Full evaluation tables
3. Architecture notes

02. Agentic RAG Compliance Engine: Retrieval that can refuse

A solo project: a question-answering service over compliance policies and financial reports, built to be measured and not just demoed.

The problem

The standard recipe of retrieve, paste into the prompt and generate has three weaknesses that matter a great deal in compliance, legal and financial settings:

• Recall. People ask in everyday language ("what is the most I can spend on a client gift?") while the policy says "the value of a single item must not exceed RMB 500". Pure vector search and pure keyword search each miss part of this.
• It never says "I don't know". When the answer is not in the knowledge base, the model produces a plausible threshold or deadline anyway. In compliance, an invented number is worse than no answer.
• It cannot be checked. No per-claim sources, and no metrics to show whether a change improved anything.

The approach

Hybrid retrieval with reranking. Dense retrieval (FAISS with a BGE embedding model) covers meaning; BM25 covers exact figures and terms such as amounts and grade codes. The two rankings are merged with Reciprocal Rank Fusion, which uses only ranks and so needs no score calibration between systems, and a cross-encoder then reranks the merged list. Everything runs on ONNX Runtime on CPU, with no PyTorch or GPU in the image.

Structure-aware chunking. Documents are split by Markdown heading, then packed by sentence, and every chunk is prefixed with its document and section title. Short clauses that would be meaningless in isolation stay findable.

A Self-RAG loop as a state graph. In LangGraph, the agent retrieves, then grades whether the passages can answer the question. If they cannot, it rewrites the query once in the style of the source documents and retrieves again. If that fails too, it refuses. After generating, a second check verifies each claim against the sources and looks for internal contradictions; a failed check triggers one regeneration with the reasons fed back, and if that fails as well the system falls back to quoting the source directly. An answer that has not passed the check is never returned.

An offline mode with no LLM. Every node has a deterministic fallback: the reranker's score serves as the relevance grader, and answers are extracted from the sources. The project runs, tests and evaluates without an API key, and degrades to this mode if the LLM endpoint fails instead of returning an error.

Things a user can inspect. A web UI highlights the path the agent took through the graph, shows timing and reasoning per node, and lists every candidate passage with its vector rank, BM25 rank, fusion score and rerank score. The same agent is exposed over REST, server-sent events and as an MCP server.

How I evaluated

I wrote 26 questions over five policy documents: 20 answerable, mostly paraphrased into colloquial language, and 6 whose answer is deliberately absent, to test refusal. Each answerable question is annotated with the key facts a correct answer must contain.

Four configurations run on the same set, each adding one component: vector-only, hybrid, hybrid with reranking, and the full agent. Two kinds of metric are reported:

• Deterministic metrics from the annotated key facts (Hit@4, MRR, whether the answer contains every key fact, refusal accuracy). These are reproducible without any API.
• RAGAS metrics judged by an LLM (context precision, faithfulness, context recall).

Results

With GLM-4-Flash as the generator:

In the offline mode, where there is no LLM to decline on its own, refusal accuracy on the six unanswerable questions went from 0 for every baseline to 0.67 for the agent.

What the numbers do not show

• Faithfulness went down, and I report it. A per-question breakdown points to two causes. Refusals reach the judge with an empty context, and it scored four identical refusal texts as 1.00, 1.00, 0.25 and 0.00, which is noise. And the grader often keeps a single passage, so explanatory sentences in the answer lose their supporting evidence.
• With an LLM, the plain baseline also refused correctly on all six unanswerable questions. The agent's refusal advantage is clear only in offline mode. Its advantage with an LLM is in what it retrieves, not in whether it refuses.
• The evidence is thin. 26 questions, one run per configuration, a corpus of 29 chunks, and the judge is the same model as the generator. Differences of a few points are within noise, and the relevance threshold was tuned on this same set.
• The cost is latency. Three to four LLM calls per question make the agent two to three times slower than the baselines.

What I would do next

Split the questions into a calibration set and a held-out test set. Move to a larger public corpus, such as real annual reports, where reranking has room to matter. Pass the candidate passages to the judge on refusals, use a stronger judge that differs from the generator, and average over several runs. For latency, use a small model for grading and a larger one only for generation.

References

1. Source code, tests and Docker setup
2. Technical report with the per-question analysis
3. Benchmark output

01. Backpropagation by shapes: Deriving gradients without memorizing them

Most backprop mistakes I made early on were not calculus mistakes. They were bookkeeping mistakes: a transpose in the wrong place, a sum over the wrong axis. These notes are the version of backpropagation I wish I had started with, where every step is checked by asking one question: what shape must this be?

Start with the shape contract

Take one dense layer in the batch-first convention. A batch of $B$ inputs with $d_{in}$ features is a matrix $X \in \mathbb{R}^{B \times d_{in}}$. The layer owns $W \in \mathbb{R}^{d_{out} \times d_{in}}$ and $b \in \mathbb{R}^{d_{out}}$, and computes

so $Z, A \in \mathbb{R}^{B \times d_{out}}$. The batch axis passes through untouched; only the feature axis changes. Writing this contract down before anything else is the cheapest debugging tool in deep learning, because almost every later formula can be recovered from it.

Why the nonlinearity is not optional

If $\phi$ is the identity, stacking layers buys nothing:

A composition of affine maps is one affine map. Depth only adds expressive power once something nonlinear sits between the layers. This is also why the choice of activation matters for training and not just for expressiveness: its derivative $\phi'(Z)$ multiplies every gradient that flows back through the layer.

The three gradients of a dense layer

Let $L$ be a scalar loss and write $G_A = \partial L / \partial A$ for the gradient arriving from above. It has the same shape as $A$. One rule covers everything that follows: a gradient has the shape of the thing it is a gradient of.

Through the activation, which acts elementwise:

Through the affine map there are three things to differentiate, and for each one only a single arrangement of $G_Z$, $X$ and $W$ produces the right shape:

Check them against the contract. $G_Z^\top X$ is $(d_{out} \times B)(B \times d_{in})$, which is the shape of $W$. The bias was broadcast across the batch on the way forward, so its gradient is summed across the batch on the way back. And $G_Z W$ is $(B \times d_{out})(d_{out} \times d_{in})$, the shape of $X$, which becomes the $G_A$ of the layer below. If you store $W$ as $d_{in} \times d_{out}$ instead, every transpose moves. That is the reason to derive from shapes instead of memorizing.

A single neuron, with numbers

Let $z = wx + b$, $a = \sigma(z)$ and $L = \tfrac12 (a - y)^2$, with $x = 1.5$, $w = 0.8$, $b = -0.2$, $y = 1$.

Forward: $z = 1.0$, $a = \sigma(1.0) \approx 0.7311$, $L \approx 0.0362$.

Backward, one local derivative per edge of the graph:

• $\partial L/\partial a = a - y \approx -0.2689$
• $\partial a/\partial z = a(1-a) \approx 0.1966$
• $\partial z/\partial w = x = 1.5$ and $\partial z/\partial b = 1$

Multiplying along the path gives $\partial L/\partial w \approx -0.0793$ and $\partial L/\partial b \approx -0.0529$. Both are negative, so gradient descent increases $w$ and $b$, which pushes $a$ toward the target of 1. Being able to predict the sign before computing it is a good test of whether the derivation is understood or only executed.

Softmax with cross-entropy

Softmax is not elementwise: every output depends on every logit, so it has a full Jacobian. Combined with cross-entropy, though, the result collapses to something simple. With probabilities $p = \operatorname{softmax}(z)$ and a one-hot target $y$:

Frameworks fuse the two operations for this reason. The fused form is cheaper and avoids taking the log of a probability that has underflowed to zero.

What it costs

A dense layer does $O(B \cdot d_{in} \cdot d_{out})$ multiply-adds going forward and holds $O(d_{in} \cdot d_{out})$ parameters. The backward pass is the same order as the forward pass, since it is again a few matrix products of the same sizes. The real price of backprop is memory: every intermediate activation has to be kept until the gradient has passed back through it. That is why the memory used in training grows with depth and batch size, while inference can discard activations as it goes.

How I test myself

• State the shape of every tensor in a two-layer MLP before writing any formula.
• Derive $\partial L/\partial W$ with $W$ stored as $d_{in} \times d_{out}$ and explain which transposes moved.
• Explain why the bias gradient is a sum over the batch and not a mean. (The mean, if there is one, already lives inside $L$.)
• Predict the sign of a gradient in a small example before computing it.

References

1. Rumelhart, Hinton & Williams (1986), Learning representations by back-propagating errors
2. Goodfellow, Bengio & Courville (2016), Deep Learning, chapter 6
3. PyTorch documentation, Autograd mechanics

02. Keeping signals alive: Initialization, AdamW and learning-rate schedules

Knowing how to compute a gradient does not mean a network will train. Three decisions determine whether the signal survives the trip through a deep model: how the weights start, how each step is scaled, and how the step size changes over time. These notes collect the reasoning behind the defaults I now reach for.

Initialization is a variance budget

Consider one pre-activation $z = \sum_{i=1}^{n_{in}} w_i x_i$ with independent, zero-mean weights and inputs. Then

If that factor $n_{in}\operatorname{Var}(w)$ is larger than 1, activations grow layer by layer; if it is smaller, they shrink. After fifty layers either one is fatal. Keeping the forward variance constant asks for $\operatorname{Var}(w) = 1/n_{in}$. Asking the same of the backward pass gives $1/n_{out}$. Xavier (Glorot) initialization compromises between the two:

ReLU sets roughly half of its inputs to zero, which halves the variance that gets through. He initialization compensates by doubling it:

The practical rule is short: match the initializer to the activation. Xavier suits tanh-like units, He suits ReLU-like units. Residual connections and normalization layers make modern networks far more forgiving, but they do not make initialization irrelevant.

From SGD to Adam

Plain SGD moves against the gradient $g_t$ with one global step size. Momentum averages recent gradients so that consistent directions accumulate and oscillating ones cancel. Adam goes one step further and keeps two running averages, of the gradient and of its square:

Both start at zero, so early values are biased toward zero. Dividing by $1-\beta^t$ corrects that:

The effect is a separate step size per parameter. Parameters that rarely receive gradient take larger steps; parameters with large, noisy gradients take smaller ones.

Why AdamW and not Adam with L2

With plain SGD, adding an L2 penalty to the loss and shrinking the weights directly (weight decay) are the same thing. With Adam they are not. An L2 term enters through the gradient, so it is divided by $\sqrt{\hat v_t}$ like everything else. Weights with a large gradient history end up being regularized less, which is the opposite of what one wants. AdamW removes the decay from the gradient and applies it straight to the weights:

This is why AdamW became the default for Transformers. Biases and normalization gains are usually excluded from the decay, since shrinking them has no regularizing benefit.

Warmup, then cosine

At step zero Adam's variance estimate is built from very few gradients and the weights are random, so a full-size step can do lasting damage. Linear warmup raises the learning rate from near zero to its peak over the first few hundred or thousand steps. After that, cosine decay lowers it smoothly:

Large steps early cover ground; small steps late let the model settle. The schedule has few knobs, which is a large part of its appeal.

Regularization in one paragraph each

Dropout zeroes each activation with probability $p$ during training and scales the survivors by $1/(1-p)$, so that the expected activation is unchanged and nothing needs rescaling at inference.

Weight decay pulls weights toward zero and limits how much the model can rely on any single feature.

Normalization layers are not regularizers by design, but by keeping activation statistics in a stable range they widen the set of learning rates that work.

Early stopping is the cheapest of all: stop when validation loss stops improving.

When the loss will not go down

The order I check things in, cheapest first:

• Can the model overfit a single batch? If not, the bug is in the model, the loss or the data pipeline, not in the hyperparameters.
• Is the loss at initialization what it should be? For $C$ balanced classes it should start near $\ln C$.
• Is the learning rate off by an order of magnitude? Try a factor of ten in both directions before touching anything else.
• Are gradients vanishing or exploding? Log the gradient norm per layer.
• Does validation loss diverge from training loss? That is a regularization or data question, not an optimization one.

References

1. Glorot & Bengio (2010), Understanding the difficulty of training deep feedforward neural networks
2. He et al. (2015), Delving Deep into Rectifiers
3. Kingma & Ba (2014), Adam: A Method for Stochastic Optimization
4. Loshchilov & Hutter (2017), Decoupled Weight Decay Regularization
5. Loshchilov & Hutter (2016), SGDR: Stochastic Gradient Descent with Warm Restarts
6. Srivastava et al. (2014), Dropout: A Simple Way to Prevent Neural Networks from Overfitting

03. Attention from scratch: Shapes, masks and what it costs

Attention is easier to understand as an answer to a specific problem than as a formula. These notes follow that route: the bottleneck it removed, the four lines of the mechanism, how it is assembled into a Transformer block, and what the bill looks like.

The bottleneck attention removed

A recurrent encoder-decoder reads the source sentence one token at a time and squeezes everything into a final hidden state. The decoder then has to generate the whole output from that single vector. Long inputs suffer most, since early tokens have been overwritten many times by the end. Bahdanau and colleagues let the decoder look back at all encoder states at every step and learn how much weight to give each one. The Transformer then asked whether the recurrence was needed at all, and kept only the looking-back.

Scaled dot-product attention in four lines

Given queries $Q \in \mathbb{R}^{T_q \times d_k}$, keys $K \in \mathbb{R}^{T_k \times d_k}$ and values $V \in \mathbb{R}^{T_k \times d_v}$:

Read it as four steps.

1. $QK^\top$ scores every query against every key. Shape: $T_q \times T_k$.
2. Dividing by $\sqrt{d_k}$ keeps those scores in a sensible range.
3. Adding the mask $M$ and taking a softmax along the key axis turns each row into weights that sum to one.
4. Multiplying by $V$ takes a weighted average of the values. Shape: $T_q \times d_v$.

Two consequences fall out of the shapes. The output has one row per query, so its length is set by the queries, not by what is attended to. And each output row is a mixture of values, which means attention by itself can only average; the nonlinear work happens in the feed-forward layer that follows.

Why divide by the square root of the key dimension

If the components of a query and a key are independent with zero mean and unit variance, their dot product is a sum of $d_k$ such terms and has variance $d_k$. As the dimension grows, the scores spread out, the softmax saturates toward a one-hot vector, and its gradient becomes vanishingly small. Dividing by $\sqrt{d_k}$ brings the variance back to one regardless of dimension.

Masks are added, not multiplied

A mask is a matrix of zeros and $-\infty$ added to the scores before the softmax, so that masked positions receive exactly zero weight. Two kinds matter. A padding mask hides positions that exist only to make sequences in a batch the same length. A causal mask hides the future: position $t$ may attend only to positions up to $t$. This is what lets a decoder be trained on every position of a sequence in a single parallel pass while still behaving autoregressively.

Many heads, one block

Multi-head attention splits the model dimension into $h$ heads of size $d_k = d_{model}/h$, runs attention independently in each, concatenates the results and applies an output projection. The cost is about the same as one full-width head, but different heads can track different relations.

A Transformer block wraps two sublayers, attention and a position-wise feed-forward network, in residual connections with normalization. In the pre-norm arrangement used by most recent models:

The residual path gives gradients a direct route to early layers. Attention has no built-in sense of order, so position has to be injected separately; the original paper added sinusoidal encodings to the input embeddings.

The bill

For batch $B$, sequence length $T$ and model dimension $d$:

Compare a recurrent layer at $O(B \cdot T \cdot d^2)$: linear in $T$, but each step waits for the previous one. Self-attention pays a quadratic term in exchange for two things, a direct path between any pair of tokens and full parallelism across positions during training. Most later efficiency work, from optimized attention kernels to sparse and chunked variants, is an attempt to soften that $T^2$.

One block, three families

The decoder-only family won the scaling race partly because its objective needs no labels and no task-specific head: every position of every document is a training example.

References

1. Bahdanau, Cho & Bengio (2014), Neural Machine Translation by Jointly Learning to Align and Translate
2. Vaswani et al. (2017), Attention Is All You Need
3. Devlin et al. (2018), BERT: Pre-training of Deep Bidirectional Transformers
4. Raffel et al. (2019), Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
5. Brown et al. (2020), Language Models are Few-Shot Learners

04. Inside a modern LLM block: RMSNorm, SwiGLU, RoPE, GQA, the KV cache and MoE

Open the config of a recent open-weight language model and it reads like a list of acronyms. The backbone is still the decoder-only Transformer from 2017; what changed is a set of components, each swapped in to relieve one specific bottleneck. I find them much easier to remember by the problem they solve than as a list.

The skeleton stayed the same

A current decoder block is still attention plus a feed-forward network, each wrapped in a residual connection:

Normalization moved in front of each sublayer (pre-norm), which leaves the residual path as a clean identity and makes deep stacks easier to optimize. Everything below is a replacement for one of the boxes in this picture.

RMSNorm and SwiGLU: cheaper normalization, gated feed-forward

LayerNorm subtracts the mean and divides by the standard deviation. RMSNorm drops the centering and keeps only the rescaling:

It is a little cheaper and works about as well, which is all a component needs to be adopted at scale.

The feed-forward network changed from one activation between two projections to a gated product:

There are now three weight matrices instead of two, so the hidden width is usually cut to about two thirds of the classic $4d$ to keep the parameter count level.

RoPE: position as rotation

The original Transformer added a position vector to each token embedding. Rotary position embedding instead rotates the query and key vectors inside attention. Each pair of coordinates is treated as a 2D point and turned by an angle proportional to the token's position $m$:

Rotations compose, so for a query at position $m$ and a key at position $n$:

The attention score depends only on the offset $n - m$. The model gets relative position for free, with no extra parameters, and different coordinate pairs use different frequencies so that both nearby and distant offsets are distinguishable.

The KV cache and why GQA exists

When generating token by token, the keys and values of earlier tokens never change, so they are computed once and stored. That removes a great deal of repeated work, but the store grows with everything:

for $L$ layers, batch $B$, cached length $T$, $H_{kv}$ key/value heads of dimension $d_h$, and $b$ bytes per number.

A worked example makes the scale concrete. Take $L = 32$, $B = 1$, $T = 8192$, $d_h = 128$ and 16-bit floats ($b = 2$). With standard multi-head attention and 32 heads, the cache is about 4 GiB for a single sequence. Grouped-query attention lets several query heads share one key/value head. With 8 KV heads serving the same 32 query heads, the cache drops to about 1 GiB. Queries stay diverse; only the stored side is shared.

The lesson I take from this is that for long contexts the bottleneck is no longer arithmetic but memory capacity and bandwidth, and that is where architectural effort has gone.

Mixture of experts: more parameters, same compute per token

A mixture-of-experts layer replaces the single feed-forward network with many, plus a small router. For each token the router scores the experts, keeps the top $k$, and mixes their outputs:

Total capacity grows with the number of experts while the work per token grows only with $k$. The costs appear elsewhere: experts can receive very uneven load, tokens have to be dispatched to wherever their experts live, and across devices that means all-to-all communication.

One distinction is worth keeping sharp, since model announcements often blur it: total parameters, active parameters per token, training compute and inference latency are four different numbers. A sparse model can be large on the first and modest on the second.

Each component, and the bottleneck it targets

Exact configurations differ between model families and between versions of the same family, so the table describes roles, not a single canonical recipe.

References

1. Zhang & Sennrich (2019), Root Mean Square Layer Normalization
2. Shazeer (2020), GLU Variants Improve Transformer
3. Su et al. (2021), RoFormer: Enhanced Transformer with Rotary Position Embedding
4. Ainslie et al. (2023), GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
5. Touvron et al. (2023), Llama 2: Open Foundation and Fine-Tuned Chat Models
6. Kaplan et al. (2020), Scaling Laws for Neural Language Models

05. After pretraining: SFT, preference optimization and RL with verifiable rewards

A pretrained language model has learned to continue text. Turning it into something that follows instructions, prefers good answers to bad ones, and reasons through a problem takes further stages of training, each with a different kind of signal. These notes lay the stages side by side, since what distinguishes them is mostly where the supervision comes from.

One loss, used twice

Pretraining factorizes the probability of a sequence with the chain rule and minimizes the negative log-likelihood of each token given the ones before it:

Supervised fine-tuning (SFT) uses the very same loss. What changes is the data, now curated prompt and response pairs, and the mask: the loss is normally computed only on the response tokens, so the model learns to answer prompts instead of learning to write them. Padding positions are masked out for the same reason. SFT teaches format and behaviour by imitation, which also sets its ceiling. The model can become as good as its demonstrations, not better.

Preferences without a reward model

Often it is easier to say which of two answers is better than to write the ideal one. Direct Preference Optimization trains on such pairs directly. For a prompt $x$ with a preferred answer $y_w$ and a rejected answer $y_l$:

The model is rewarded for raising the probability of the preferred answer relative to a frozen reference model more than it raises the rejected one. The reference terms act as an anchor that keeps the policy from drifting far from where it started. No separate reward model and no sampling loop are needed, which is why the method is attractive in practice.

PPO: improve, but not too far in one step

When there is a reward signal instead of fixed pairs, the model can be treated as a policy and trained with reinforcement learning. Proximal Policy Optimization compares the new policy with the one that generated the data through a probability ratio, and clips it:

If an action was better than expected ($A_t > 0$), the objective raises its probability, but the gain stops counting once the ratio leaves $[1-\epsilon, 1+\epsilon]$. With a negative advantage the logic runs the other way. The clip is a cheap way to say: trust this batch of data only for a small step. A full implementation also carries a value network to estimate advantages, usually an entropy bonus, and often a KL term.

GRPO: let the group be the baseline

Training a value network for a large language model is expensive and hard to get right. Group Relative Policy Optimization avoids it. For each prompt it samples a group of answers, scores each one, and uses the group's own statistics as the baseline:

An answer is good if it beats its siblings on the same prompt. That advantage then drives a PPO-style clipped update, with a KL penalty toward a reference policy. A caution I keep attached to this formula: group size, reward scaling, clipping and the exact token-level objective differ between papers and codebases, so it describes the idea, not every variant that carries the name.

Why verifiable rewards matter

Reinforcement learning is only as good as its reward. A learned reward model can be flattered: the policy finds outputs that score well without being good. Tasks with a checkable outcome avoid that problem. A maths answer either matches or does not; code either passes its tests or does not.

DeepSeek-R1 is the best-known demonstration. The paper reports that large-scale RL with rule-based rewards on such tasks was enough for long chains of reasoning to emerge in the model's outputs, and that the released model combined this with supervised stages to improve readability and general usefulness. The broader point is that the bottleneck moves from collecting demonstrations to building environments where correctness can be checked.

Reasoning: prompted versus trained

Chain-of-thought prompting asks a model to write intermediate steps before the answer, and often improves accuracy on multi-step problems with no change to the weights. A reasoning-trained model goes further: producing the working is part of what it was optimized to do. In both cases the cost is the same kind, more tokens per answer, so more latency and compute. That is test-time scaling: spending inference compute, instead of parameters, to buy accuracy.

The stages at a glance

References

1. Schulman et al. (2017), Proximal Policy Optimization Algorithms
2. Rafailov et al. (2023), Direct Preference Optimization: Your Language Model is Secretly a Reward Model
3. Shao et al. (2024), DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
4. DeepSeek-AI (2025), DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
5. Wei et al. (2022), Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

06. Diffusion in latent space: From adding noise to Stable Diffusion

Diffusion models generate images by learning to undo noise. The idea is simple enough to state in a sentence, yet the path from that sentence to a system that turns a text prompt into a high-resolution picture goes through several distinct design decisions. These notes take them in order.

Destroying an image on purpose

The forward process adds a little Gaussian noise at each of many steps until nothing of the image is left. Conveniently, the result after $t$ steps has a closed form, so any noise level can be sampled directly from the clean image $x_0$:

Here $\bar\alpha_t$ is a schedule that falls from near 1 (almost clean) to near 0 (almost pure noise). Nothing is learned in this direction. Its only job is to manufacture training pairs: a noisy image, and the exact noise that was added to make it.

Learning to undo it

The network $\epsilon_\theta$ sees a noisy image and the step index, and predicts the noise:

That is an ordinary regression loss, which is a large part of why diffusion models train so stably compared with adversarial ones: there is no second network to balance against. One model is trained across all noise levels at once, with the step index telling it how much noise to expect.

Sampling is where the cost is

Generation starts from pure noise and applies the network repeatedly, removing a little noise each time. If one evaluation of the denoiser costs $C$ and sampling takes $S$ steps, an image costs about $S \cdot C$. Training looks at one random step per example, but sampling has to walk the whole chain. This is the central weakness of the approach, and much of the follow-up work, faster samplers and distillation into few-step models, is aimed at shrinking $S$.

Moving to latent space

Running the denoiser on raw pixels is expensive. A $512 \times 512$ RGB image has $512 \cdot 512 \cdot 3 = 786{,}432$ values. Latent diffusion first trains an autoencoder that compresses the image by a factor of 8 along each side into a 4-channel latent of size $64 \times 64 \times 4 = 16{,}384$ values. That is 48 times fewer numbers for every one of the $S$ denoising steps.

The pipeline then has three parts with clearly separated jobs:

• The autoencoder handles perceptual detail: textures and sharp edges are reconstructed by its decoder.
• The denoiser works only in the compact latent space and handles composition and semantics.
• The decoder is applied once, at the very end, to turn the final latent back into pixels.

The price is that image quality is now bounded by the autoencoder. Fine structures it cannot reconstruct, small text being the classic example, cannot be recovered by the denoiser however well it is trained.

Telling it what to draw

Text conditioning enters through cross-attention. A text encoder turns the prompt into a sequence of embeddings, and at several resolutions inside the denoiser the image features attend to them, with image features as queries and text embeddings as keys and values. It is the same mechanism as in a translation model, pointed at a different pair of modalities.

Following the prompt closely takes one more step. Classifier-free guidance trains the same network both with and without the prompt, by dropping the text some of the time, and at sampling extrapolates from the unconditional prediction toward the conditional one:

With $s = 1$ this is plain conditional sampling. Larger values follow the prompt more strictly at the cost of diversity, and eventually of image quality. It also doubles the work per step, since each step needs two predictions.

What SDXL changed

Stable Diffusion XL keeps the latent-diffusion recipe and scales its parts: a considerably larger denoiser, two text encoders used together for a richer reading of the prompt, conditioning on image size and crop so that training on mixed resolutions does not leak into the outputs, and an optional refiner model for the last denoising steps. None of this alters the principle. It moves the balance between quality and cost.

Where the time goes

References

1. Ho, Jain & Abbeel (2020), Denoising Diffusion Probabilistic Models
2. Kingma & Welling (2013), Auto-Encoding Variational Bayes
3. Rombach et al. (2022), High-Resolution Image Synthesis with Latent Diffusion Models
4. Ho & Salimans (2022), Classifier-Free Diffusion Guidance
5. Podell et al. (2023), SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis