"The PL drew 74% of DDR bandwidth." That is what I wrote last time. It was wrong. What I had measured was not DDR, but a lower ceiling in front of it. Today I measured the two separately and found the real limit: 1.97 GB/s. This becomes the design value for my own accelerator.
why bandwidth is the number that matters
When a small language model emits one token, the CPU or FPGA reads all of the model's weights once more. At 135M parameters in INT8, that is 135 MB. It reads this every time. So "how many tokens per second" is decided almost entirely by "how many bytes per second can be read."
On top of that, in INT8 each byte of weight read produces exactly one multiply. Read speed and compute speed are tied one to one. So adding more compute units does nothing if the data can't be read. Bandwidth is the only number worth measuring.
the DDR wall and the datapath wall
On the way from DDR to the FPGA fabric, data passes through two narrow points. They are two separate ceilings, set for entirely different reasons. Last time, I measured only one of them and assumed I was talking about the other.
the loopback hid the answer
The circuit I built last time was a loopback. Read from DDR, pass the data once around the PL, and write it straight back to DDR. In that setup, reads and writes are always the same amount.
It gave "read 0.77 + write 0.77 = 1.54 GB/s total," and I read that as "74% of DDR's 2.1." But the 0.77 for reads and the 0.77 for writes are each 96% of their own 0.80 datapath. So the one pinned at its limit was blue, and amber still had plenty of headroom. A loopback cannot separate the two ceilings.
assign tready = 1'b1;.
Real inference also only reads weights and writes nothing back, so this is closer to the real thing, too.
if it tracks the clock, it is the datapath
To say "blue is the culprit," I need evidence. I used a simple trick: change only the datapath width and see whether the measurement follows. The datapath width is 8 bytes × clock, so raising the clock widens only the datapath, and DDR stays exactly the same.
I built two versions, at 100 MHz and 142.86 MHz, and measured both. The results came out like clockwork.
two HP ports, and the bottleneck moves
Zynq has four "HP ports" that go from the PL to DDR. With one port, even at 142.86 MHz, I could use only about half of DDR (53.7%). So I set up two DMAs and split them across HP0 and HP2. The datapath becomes 1.14 × 2 = 2.29 GB/s, which for the first time exceeds DDR's 2.1.
| Config | Clock | Datapath | Measured | % of datapath | % of DDR |
|---|---|---|---|---|---|
| 1 HP port | 100.00 MHz | 0.80 | 0.79 | 99.1% | 37.8% |
| 1 HP port | 142.86 MHz | 1.14 | 1.13 | 98.7% | 53.7% |
| 2 HP ports | 142.86 MHz | 2.29 | 1.97 | 86.0% | 93.6% |
what 1.97 GB/s buys you
Once the bandwidth is known, the number of compute units you need falls out of a single division. In INT8, one multiply per byte read. 1.97 GB/s at 142.86 MHz means 1.97 ÷ 0.14286 ≈ 13.8, so about 14 bytes arrive per clock. Fourteen multipliers are enough.
If the weights are made ternary, −1 / 0 / +1, about 5 fit in one byte (about 0.2 B/weight). The same 1.97 GB/s path can then carry 9.8 G weights per second, effectively 5 times as many. On top of that, multiplication turns into "add, subtract, or do nothing," so it can be built from LUTs without even using DSPs. Since bandwidth is the wall, this is what to push on. This measurement confirmed that once again.
what I got wrong last time
I wrote: "The PL drew 74% of DDR, more than double the CPU's 33%." The measured value itself, 1.54 GB/s total, was correct. But it was not about DDR at all; it was just pinned at 96% of the 0.80 GB/s datapath. At that point I wasn't using even half of DDR.
The direction of the conclusion, that the PL can draw more from DDR than the CPU, still holds. In fact the correct number is 93.6%, 2.8 times the CPU's 33% (0.70 GB/s). That is a stronger result than last time. But it requires using two HP ports; with one, it reaches only 53.7%. This is an important design condition that was missing last time.
stage 3
The null sink was a circuit that "just discards." Next I replace it with a real row of multipliers (a MAC array). There is only one question: can it hold 1.97 GB/s while actually computing? If it can, the next step is to move the weight representation to ternary.
Stage 3 reached 1.91 GB/s (96.7% of the no-compute case), and the ternary Stage 4 reached 9.16 G weights/s (4.8 times INT8) with 0 DSPs. The 1.97 GB/s design value settled here serves as the yardstick for both. Continued in "The Wall Doesn't Move".