PYNQ-Z1 · Stage 5a → 5b · 2026-09-18

Reaching the Wall

At the end of the previous entry I learned that speeding up only the matrix multiply is pointless. So I turned attention into a circuit. It became 15.6× faster, and softmax didn't cost a single byte of bandwidth. All four circuits are pinned to the 2.0 GB/s wall.

And at the very end, I found out that the way I had been measuring was wrong. The direction of the conclusions didn't change, but I redid every number (section 6).

Update, 2026-09-21 — I made the same kind of mistake again

After publishing, I found that the CPU implementation I compared against had been compiled with -O2 (NEON disabled). Adding -mfpu=neon makes the same code 1.78× faster. So every "N× the CPU" in the text was overstated by about 1.8×. The numbers are corrected throughout this page (attention: 28× → 15.6×).

Right after writing "suspect the measurement formula" in section 6, I made the same bias again, this time in how much effort went into the baseline. I measured what I built carefully, and what I compared it against carelessly. That happened twice in a row.

01The homework was "the remaining 78%"

the 78% that was still on the CPU

A measurement I added at the end of The Wall Doesn't Move broke the premise of the plan. Even with the matrix multiply turned into a circuit and cut to 14.2 ms, attention left on the CPU takes 46 ms at context 512. The CPU takes 78% of the whole system, and the circuit I worked so hard on barely helps.

So Stage 5 had one job. Put attention in a circuit too.

Time per token (context 512, same time scale) Matmul (PL) attention (CPU) attention (PL) Small ops (CPU) attention on CPU through Stage 4 46.2 ms 15.7 tok/s attention on PL too Stage 5b 20.6 ms 14.20 + 2.97 + 3.47 48.5 tok/s 0 20 40 60 ms The green matmul bar is unchanged. The red part shrank to 1/15.6. That is this entry.
Figure 1 / What shrank is the red part. The green (matmul) is still the Stage 4 circuit, and its length hasn't changed at all. The only change is that attention went from 46.2 ms to 2.97 ms. That alone cuts the whole system to one third. Start with the longest bar. Speed work always goes like this.

02Attention becomes a two-phase circuit

two phases, one memory in between

The matmul circuit was simple. Each incoming weight gets multiplied and added, one after another, and it never has to look back. Attention doesn't work that way.

First you stream all of K to build the scores. Then you use those scores as weights and stream V. V can't start until K has been read to the end, so you need a buffer for the scores in between. That is the biggest structural difference.

1 group = one KV head of one layer T q 192 B K ── T × 64 B V ── T × 64 B Out: 195 words 192 + 3 denom. Phase 1 ── stream K, build scores Receive K 8 B / beat 24 multipliers → 3-level adder tree Accumulate → clamp 8-bit score Score buffer s8 BRAM 2048 words × 3 heads + max mx (3 of them) Phase switch. Wait 12 cycles for the max to settle Phase 2 ── weight V by score Read s8 from buffer mx − s8 = distance d ★ exp table 256 words this is softmax 24 multipliers × V 192 accumulators + 3 denominators Multipliers: 24 in phase 1 + 24 in phase 2 = 48. Just over 20% of the 220 DSPs The ★ lookup is the softmax. It never touches DDR ── see Figure 4 below No division. Output 192 numerators and 3 denominators, divide downstream (only 3 per group)
Figure 2 / Two phases with a buffer between them. Data flows left to right twice. The first pass (K) builds scores and stores them in the buffer. The second pass (V) reads them back and uses them as weights. The matmul circuit never had to "store along the way." This is the first time it was needed. There are 48 multipliers in total, just over 20% of the chip's 220 DSPs.
I changed my approach ── always the simulator before the board

In Stage 5a I spent a whole day on the board chasing a single one-cycle slip in this circuit. Three times I confidently named a different cause, and twice I was wrong. The real cause: "the data was 2 stages deep, but I took the valid flag from 3 stages." A counting mistake.

In Stage 5b I changed the order. Pass the Vivado simulator first. I swept context length over 1, 2, 3, 5, 8, 12, 16 and 64, dropped upstream data with a 10% probability, and stalled downstream one time in four, then checked 1,560 words against numpy. It passed on the first try, and matched on the board on the first try too. Logic errors in the simulator; only timing and bandwidth on the board. That is the split.

03Using each byte 3 times

why attention is denser than matmul

This was the surprise. Attention is a "denser" computation than matmul.

In a matmul, each weight byte you read is used only once. Then it is thrown away. So one multiply per byte. The height of the wall is directly the ceiling on compute.

Attention is different. This model (SmolLM2-135M) uses a structure called GQA, where one K/V head is shared by 3 q heads. So each byte of K you read can be handed to 3 separate computations at once.

Matmul (Stages 3 & 4) 1 weight byte read from DDR 1 multiply Used once, thrown away attention (Stage 5) 1 byte of K/V read from DDR (only once) q head 0 multiply q head 1 multiply q head 2 multiply Read once, used 3 times Measured ── multiplies per byte Matmul 1.00 attention 2.99
Figure 3 / For the circuit, GQA was a bonus. GQA was originally a trick to save KV-cache memory, not a speed feature. But in a world where "the wall is bandwidth," being able to read once and use three times is itself 3× the performance. The measured 2.99 is almost exactly the theoretical 3.00. A design choice on the model side just happened to suit the circuit.

04Softmax comes for free

the exponent never leaves the chip

On the CPU, softmax is expensive. Measured: 6.9 ms of the 46 ms of attention. It computes an exponential for every position in the context.

In the circuit, it becomes one table lookup. The score is rounded to 8 bits, so it can take only 256 values. Just bake in the 256 answers ahead of time. The exp operation itself disappears from the circuit.

And the key point: this table lives inside the chip. Not a single extra byte moves between the chip and DDR.

Off-chip / on-chip DDR KV cache read K read V 195-word result Only these 3 arrows cross the boundary Everything here is on-chip (BRAM and LUTs) Score buffer + max mx mx − s8 ★ exp table 256 words folded into LUTs Softmax is just these 3 Not one more arrow crosses the boundary = DDR reads unchanged = no extra time Measured ── same conditions, same method, only softmax on/off (1 HP port) no softmax 1.01 with softmax 1.00 GB/s
Figure 4 / If no more arrows cross the boundary, the time doesn't go up. You could dismiss this as "obvious," but I didn't stop at obvious. I changed just one variable and measured. The same bitstream on one HP port gave 1.01 and 1.00, a difference within measurement noise. Work that took 6.9 ms on the CPU takes 0 ms in the circuit.
A small trick ── where to put the denominator

Softmax needs a division, but the circuit doesn't do it. It outputs 192 numerators and 3 denominators and leaves the division to downstream. Building hardware for a division that happens only 3 times per group isn't worth it.

But growing the output from 192 to 195 words was surprisingly tricky. The output stage has a huge selector that picks one of 192 values, and it has only 0.436 ns of slack per clock. Adding even one more 2-way choice could break timing. So I put the denominators at free addresses that the selector's address formula already produces (192, 200, 208). That added 3 words without changing the formula, and the circuit is not one level deeper.

05With 2 ports I only got 1.46

when the number refuses to make sense

Up to here, things went smoothly. Use 2 HP ports to double the bandwidth. It should have been the same thing I did in Stages 3 and 4.

But I only got 1.46 GB/s. One port gives 1.00, so that's 1.46×. Matmul had scaled 1.72×. Every result checked against numpy matched, so the circuit was working correctly. Only the speed didn't add up.

Blame the circuit here and you get stuck in a swamp

A reason came to mind right away. The attention circuit stalls upstream while it emits the 195 output words. The matmul circuit never stalls. That had to be it, I thought.

But the numbers didn't fit. It stalls for 216 of the 8,217 beats in a group, only 2.6%. That can't explain 26%.

So before touching the circuit, I measured at context lengths 128, 512 and 2048 and solved simultaneous equations with two unknowns: "the stall time per group, S" and "the fixed overhead per measurement, P." Here is the answer.

S = 216 cycles, P = 0.82 ms. All three conditions fit exactly. S is exactly the stall time I designed. The problem wasn't the circuit. I had forgotten that P existed.

06I was measuring it wrong

measure the slope, not the ratio

P is the measurement program itself. Starting a transfer and waiting for it to finish took 0.7–0.8 ms per channel pair. In an 8 ms measurement, that alone eats 10%.

And it had been in every measurement since Stage 1.

0 4 MB 8 MB 0 3 6 ms Transfer size Old ── total ÷ total time = slope from the origin 1.47 GB/s (too low) New ── slope between 2 points 2.00 GB/s (net) 3.6 ms 5.7 ms Intercept 1.5 ms overhead The line misses the origin. So one point joined to the origin always reads low
Figure 5 / I had assumed the line passes through the origin. A measurement follows the line y = size ÷ bandwidth + fixed overhead. Measuring one point and dividing "total size ÷ total time" means reading the slope of the line from the origin to that one point, and it always comes out low by the intercept. Measure two points and take the difference, and the intercept cancels out. It is just the linear function from middle school, and an embarrassing kind of mistake.

I remeasured everything. Not one conclusion changed direction, but the numbers now look like this.

MeasurementOld (1 point)New (slope)Meaning
Stage 2 · no compute · 1 HP port1.131.143exactly 100.0% of datapath width
Stage 2 · no compute · 2 HP ports1.972.00995.7% of the DDR theoretical 2.1
Stage 3 · 16 INT8 MACs1.911.98298.7% of the wall
Stage 4 · ternary1.831.8949.47 G weights/s
Stage 5b · attention + softmax1.461.9999.3% of the wall

The first row surprised me most. The 1-port circuit had been running at exactly 100.0% of datapath width. For two weeks I had thought "98.7%, so 1.3% is being lost somewhere," but the only loss was inside the measurement program.

Lesson

When the numbers don't fit, suspect the measurement formula first. I got lucky this time only because I swept context length and solved the simultaneous equations before rebuilding the circuit. If I had gone ahead with "adding shadow accumulators should remove the stall," I would have spent two weeks building something that improved things by only 2.6%, and been at a loss when the numbers still didn't fit.

07All four are pinned to the wall

four different circuits, one number

With everything remeasured, here are all the circuits I have built so far, side by side.

Datapath width 2.286 The DDR wall 2.009 Stage 2 · no compute 2.009 Stage 3 · INT8 1.982 Stage 4 · ternary 1.894 Stage 5b · attention 1.990 0 1.0 2.0 GB/s Totally different insides: discard-only / 16 MACs / 80 ternary / 48 MACs + exp table ── same number
Figure 6 / Whatever I build, I get the same number. The four bars are a circuit that only discards, 16 multipliers, 80 ternary units, and 48 multipliers + an exp table. Inside, they are completely different things. Yet the numbers are nearly the same. This is the most direct view of what "the bottleneck is bandwidth, not the number of arithmetic units" means. The blue line (datapath width) still has room, but everything stops at the orange line (DDR).
Stage 5b circuitValue
Read (2 HP ports)1.99 GB/s
Against the DDR wall99.3%
Multiplies5.95 G MAC/s
Per byte2.99 MAC
LUT12,820 / 53,200 (24%)
DSP108 / 220 (49%)
Timing slack+0.201 ns
Attention at context 512 (1 token)2.97 ms ── 15.6× the CPU

08What I decided not to build

the optimisation that is not worth it

Stage 5b had a third planned item: removing the stall (2.6%) while the 195 output words are emitted. With a second set of accumulators, the circuit could take in the next group while emitting, and the stall would go away.

I dropped it. The measurements told me why. The 1-port circuit runs at 97.4%, exactly the theoretical value computed from the stall. But with 2 ports it is already at 99.3% of the wall. Even with the stall gone, DDR won't give any more.

The plan's premise became the decision

"The bottleneck is bandwidth, not area." The reading this plan started from became, here, the reason not to build. Adding another set of 192 32-bit accumulators (6,144 registers × 2 sets) is pointless when the wall won't allow it.

The chip still has 76% of its LUTs and 51% of its DSPs free. Plenty left over, and nothing to use it on. That's where this board stands.

09What I can and can't claim so far

what is measured and what is not

What I can claim. All of it is measured on the board.

What I can't claim. I'll be honest about these.

Next is streaming the real weights. So far the question was "does the box run correctly and fast?" Next, the question is whether actual sentences come out of that box.