PYNQ-Z1 · Stage 2 → 4 · 2026-09-08

The Wall Doesn't Move

The memory wall sits at 1.97 GB/s, and no design moves it. So I stopped widening the road and made the cargo smaller instead. With ternary weights, the same wall delivers 4.8× as many weights. Not a single multiplier is used.

Update (2026-09-13): I later measured "everything except the matrix multiply" and found that attention takes 83 ms on the CPU. I added this to section 7.

01What the three stages did

what the three stages measured

In Stage 2 I measured where the wall is. In Stage 3 I built a circuit that can compute right up to the wall. In Stage 4 I pulled 5× out of the same wall. Everything was measured on this one board.

Bytes moved from DDR GB/s · all about the same = the wall wall 1.97 Stage 2 · no compute 1.97 Stage 3 · INT8 1.91 Stage 4 · ternary 1.83 0 1.0 2.0 Weights extracted G weights/s · this is what grew 4.8× Stage 3 · INT8 1.91 Stage 4 · ternary 9.16 4.8× 0 4 8
Figure 1 / I didn't change how much was moved. I changed how much came out of it. The top rows are all about the same. That is the wall, and it doesn't move. What changed is the bottom—how many weights come out of the same single byte. INT8 gives 1. Ternary gives 5. The 4.8× came not from faster multiplies or more compute units, but from how the data is packed.

02Stage 2 · Measuring where the wall is

where the wall actually is

The first job was to pin down "how fast can I read at all." I wrote this up in detail in a separate entry (Two Ceilings), so here are just the key points.

There are two ceilings. The speed of the DDR itself (2.1 GB/s, fixed when you buy the board), and the datapath width on the PL side (8 bytes × clock, which design can change). With one HP port, the datapath width runs out first and only half the DDR gets used. Only with two ports did the DDR side saturate.

The real limit of this board is 1.97 GB/s. That is 93.6% of the theoretical 2.1. The rest is unavoidable loss such as refresh. From here on, I design around this number.

03Stage 3 · Computing up to the wall

does computing cost bandwidth?

In Stage 2 I used a circuit that just accepts data and throws it away. I swapped that for real multipliers. There was only one question: can it hold 1.97 GB/s while actually computing?

It could easily have failed. If the arithmetic can't keep up, the circuit stalls the upstream, and bandwidth drops the moment that happens. So I set one absolute design rule: Keep the accept signal at 1, always. Never stall the upstream.

Shape of the input stream x (512 B) captured first row 1 (512 B) row 2 (512 B) row 3 (512 B) … 65,535 rows x needs no separate path, so no AXI slave and no BRAM controller are required DDR weights AXI DMA carrier MAC array 8 multiplies/clock 5-stage pipeline FIFO DDR results 8 B/clock 4 B / 512 B Results are only 0.8% of the reads. Writing them back costs almost no bandwidth Measured 1.91 GB/s = 96.7% of no-compute. All 131,070 rows match numpy
Figure 2 / The throw-away circuit, swapped for one that multiplies. The key to this design is capturing the first 512 bytes of the input stream as the input vector. Thanks to that, the whole separate path for distributing x (an AXI slave, a BRAM controller) is unnecessary. Even with computation, 96.7% of the bandwidth remained. Most of the gap is explained by writing back the results.

04Short by 0.5 ns

the first build did not meet timing

The first build failed timing. It was short by almost exactly 0.5 ns. On an FPGA, "does the signal arrive within one clock" is an absolute requirement. If it doesn't, there is no error. Wrong values just flow through, silently.

0 2 4 6 ns 1 clock = 7.00 ns Before 4 steps in 1 clock counter routing RAM Multiply built from LUTs (9 logic levels) 7.39 over After split over 2 clocks ← clock 1: capture in a register here DSP multiply ← clock 2: on a DSP, not in LUTs +0.062
Figure 3 / Two causes, stacked. ① Reading x and multiplying were crammed into the same cycle. ② The 8×8 multiply was judged "too small" for a DSP, so it was built from LUTs, 9 levels deep. I added one pipeline stage to separate them and forced the multiply onto a DSP with use_dsp. That took it from −0.502 ns → +0.062 ns. The real oversight: the chip has 220 DSPs, and 0 of them were being used.
This kind of failure happens silently

I could have run it anyway with timing unmet. It does run. But when data gets corrupted, it just returns wrong answers without saying anything. Then I could no longer tell whether a problem was in the circuit's logic or in timing. So I fixed it first, before moving on to measurement.

05Stage 4 · Shrinking the cargo

three values per weight, five weights per byte

The wall doesn't move. So the only option is to get more weights out of the same single byte. That is ternary quantization (BitNet b1.58). Each weight is rounded to one of three values: −1 / 0 / +1.

With three values, each weight carries log₂3 = 1.585 bits. In practice I use 3⁵ = 243 ≤ 256 and pack exactly 5 per byte. That is 5× the density of INT8.

178 1 byte lookup Base-3 digits (178 = 1 + 2·3 + 1·9 + 0·27 + 2·81) 1 2 1 0 2 −1 digit − 1 gives ternary Weights (ternary) 0 +1 0 −1 +1 Input vector x x₀ x₁ x₂ x₃ x₄ Resulting terms — a 3-way choice, not a multiply 0 +x₁ 0 −x₃ +x₄ Adder tree 40 terms → 1 "Pass through, zero out, flip the sign" — not a single multiplier appears here
Figure 4 / A multiply turns into a 3-way choice. Since weights are only −1 / 0 / +1, no multiplication happens at all. All that's needed is selecting, flipping the sign, and adding. That is what "build it from LUTs, not DSPs" actually means. One 8-byte beat yields 40 terms, so a 40-term adder tree is needed (writing that by hand always goes wrong, so I emit it from a generator script).

06The multipliers disappeared

what it cost in silicon

Here is how Stage 3 and Stage 4 used the chip's resources, side by side. This comparison is the thing I most wanted to show in this whole project.

LUT 53,200 total 8,175 (15%) 9,557 (18%) only +1,382 DSP 220 total 20 (9%) 0 none used Throughput G weights/s 1.91 9.16 ■ Stage 3 INT8 / ■ Stage 4 ternary — 17% more LUTs, every multiplier gone, 4.8×
Figure 5 / A few LUTs in exchange for giving up every multiplier. On a chip with 220 DSPs, the main circuit uses not a single one. The guess I made at the very start of the project—"the bottleneck is bandwidth, not the number of compute units"— held all the way to the end.
StageCompute unitsReadWeight throughputDSPLUTChecked vs numpy
2 · no compute—1.97————
3 · INT8161.911.91208,175131,070 rows
4 · ternary801.839.1609,557524,278 rows

The last column is the result of checking every single row the PL produced against numpy. Fast but broken is worthless, so I did this at every stage. In fact, a bug where the generator script's pipeline depth was off by one would not have been found without this check—it shows up as "runs fine, but every answer is wrong."

07What isn't done yet

what this is not, yet

At this speed, a ternary-quantized 135M-parameter model (27 MB of weights) works out to 14.7 ms per token = about 68 tok/s. But that is an upper bound for the matrix multiply alone. Attention, layernorm, softmax and the KV cache haven't been built at all yet.

So I measured how heavy the rest actually is (update, 2026-09-13). I wrote everything except the matrix multiply in C at the real dimensions of SmolLM2-135M and ran it on the board's CPU. It was far heavier than expected, and it changed the premise of the plan.

Time per token, by part (normalized to 100%) matmul (PL) attention (CPU) small ops (CPU) context 128 26.2 tok/s context 512 9.8 tok/s context 2048 2.8 tok/s The matmul doesn't depend on context length (always 14.68 ms). Only attention grows with it
Figure 6 / How much of the whole the fast part actually covers. Green is the share handled by the circuits built in this entry. The longer the context, the thinner the green gets—the matrix multiply doesn't depend on context length, but attention grows in proportion to it. At context 2048, the PL's share is 4%. At that point the circuit barely helps at all.
Update, 2026-09-13 — "Matrix multiply on the PL, the rest on the CPU" doesn't work

At context 512, attention alone takes 83 ms. That is 5.7× the matrix multiply's 14.68 ms. The whole system only reaches 9.8 tok/s, one seventh of 68 tok/s.

It isn't a bad implementation. A naive version that converts to float took 158 ms. Computing directly in INT8 still takes 83 ms, an effective 0.21 G MAC/s. That is close to the measured upper limit of the CPU's INT8 dot product, 0.28 G MAC/s. It is the raw performance of the Cortex-A9. Rewriting it won't make it shorter.

The next step is clear. Attention becomes a circuit too. Luckily, what needs building is almost the same as the existing engine—the KV cache is just another INT8 stream, and QKT and AV are nothing but dot products.

Only the sizes differ. With GQA, 3 K/V heads are shared by 9 heads, so each byte is reused 3.15 times (the matrix multiply used it once). That means I want 48 MACs—still only a little over 20% of the 220 DSPs. Softmax's exp can also go into the circuit as a BRAM lookup table.

What happens if I go that far (estimate)

Attention becomes bandwidth-bound: 2.94 ms at context 512. The budget is 14.68 + 2.94 + 4.2 ≈ 21.8 ms, or about 46 tok/s. On top of that, it holds up much better as the context grows— the PL's bandwidth advantage applies directly to reading the KV cache as well.

After that, this is what remains.