FPGA inference accelerator / Progress report

From CPU to Fabric

Where the plan to run a small language model fast on an FPGA stands now. Why it is slow on a CPU, what changes when it moves to an FPGA, and how I measured that for the first time yesterday.

2026-09-05 Measured on: PYNQ-Z1 (XC7Z020) Before that: ADRFT1 (MPFS095T)

There is a follow-up — Update, 2026-09-08

This document reflects my understanding as of 2026-09-05. When I re-measured three days later, I found that my reading here, "the PL gets 74% of DDR", was wrong. The 74% was not stuck at the DDR. It was stuck at a lower ceiling in front of it. The direction of the conclusion is unchanged. The correct value was 93.6% (1.97 GB/s).

I have left this document as it was, to keep the record. For what changed and how, see "Two Ceilings". The affected passages in the text are marked as well.

What this project is aiming for

What I want to do is run a small quantized inference model on compute circuits I build myself in an FPGA. I know a GPU would be faster. The point is to build it myself.

But the research and measurements so far have overturned my initial picture twice. This document uses figures to share what I know now, and what I will do next.

Figure 1 — What is this chip, anyway?

The PS and the PL live on the same chip and share the same DDR memory. The PS is a finished CPU. The PL is "an empty lot for circuits", and what gets built there is up to me. The road from the PL to DDR is the HP port (High Performance). Yesterday I opened one of those roads and measured how wide it is.

The important point is that the PL starts out empty. It has 220 DSPs on it, but they are not wired to anything, so they do nothing. You design what the circuit should do, build a bitstream (a file with the circuit design baked in), and load it.

And the PL cannot reach DDR on its own. There is a gate called the HP port, and only after you build a circuit that goes through it can the PL touch memory. That gate being blocked is exactly where I got stuck on the ADRFT1.

Figure 2 — The fabric's raw material: the LUT

A LUT is a "rewritable lookup table". Depending on its contents, it can become any logic circuit. It holds 64 answers, one for each 6-bit input. Just by rewriting those 64 bits, it becomes an AND, an XOR, or an adder. A bitstream is a file with the table contents of every LUT and every wiring choice baked in. The moment it is loaded, the empty fabric becomes the intended circuit.

As a real example, here is how much the circuit I built yesterday actually used.

Resource	Used	Total	Share	Role
LUT	4,360	53,200	8.2%	Anything: control, decisions, address calculation
Flip-flop	6,199	106,400	5.8%	Holds a value for one clock
Block RAM	11.5	140	8.2%	Temporary buffer for chunks of data
DSP	0	220	0.0%	Multiply only. Unused, since this circuit computes nothing

The DSP count of exactly 0 makes the point nicely. That circuit only moves data and does no computation, so it uses not a single multiplier. The division of labor is: LUTs do anything, DSPs only multiply.

You can build a multiplier out of LUTs too, but a multiplier of any reasonable width eats hundreds of them and runs slow. That is why dedicated circuits come baked into the chip from the start. The DSP on this chip is a 25-bit × 18-bit signed multiplier with a 48-bit accumulator, which is far too big for INT8.

Going ternary makes DSPs unnecessary

When weights are ternary (+1 / 0 / −1), a multiply becomes a three-way choice: add, do nothing, or subtract. That is not a multiplication, so it is built from LUTs, not DSPs. So on the ternary path, the budget that matters is not the 220 DSPs but the 53,200 LUTs. A rough estimate says about 1,000–1,500 are needed, so LUTs will be left over too. In the end, the conclusion stays the same: the only thing in short supply is bandwidth.

Figure 3 — Why bandwidth is the wall

What sets inference speed is not the number of compute units but how fast you can read from memory. Making one token means reading every weight once, and with INT8 each byte read triggers just one multiply. So "how many bytes you can read per second" is directly the upper limit on "how many computations you can do per second". Of the 220 DSPs, this bandwidth can feed only about 8. (Update, 2026-09-08: the bandwidth is now settled at 1.97 GB/s, so this number becomes about 14. That is 6% of the total. The point, that compute units are over-provisioned by an order of magnitude, still stands.)

This was the first of my assumptions in this project to be overturned. At first I thought, "Hundreds of DSPs are sitting idle, so this should get fast." In reality, there were more than enough DSPs. What was short was the bandwidth from memory.

The conclusion that follows

To go faster, the most effective thing is to cut the number of bytes per weight. Going from INT8 (1 byte/weight) to ternary (about 0.2 bytes/weight) lets the same bandwidth carry 5 times as many weights. And with ternary weights, multiplies become additions and subtractions, so you don't even need DSPs. That is why models like BitNet b1.58 suit this board.

Figure 4 — What I measured yesterday: how much of the road can be used

How much the CPU and the PL could each pull from the same DDR on the same board. The CPU, with both cores working flat out, got 33%. The PL, with a single DMA, got 74%. Given the losses from refresh and bank switching, 74% is close to the practical upper limit, so the PL side can be said to use the DDR fully.

That is what I meant by "the PL does the same job at 74%". From the same memory, the CPU can pull only 30%, while the PL pulls a little over 70%. A circuit that drives memory directly in hardware uses memory far more efficiently than a general-purpose CPU.

Update, 2026-09-08 — the 74% in this section was wrong

The measured value of 1.54 GB/s itself is correct. The interpretation was wrong. It is not 74% of the DDR's 2.1 GB/s. It is 96% of the PL-side datapath width of 0.80 GB/s. The AXI path is 64 bits wide, so it carries only 8 bytes per clock. At 100 MHz the limit is 8 × 100,000,000 = 0.80 GB/s. Both the 0.77 read and the 0.77 write were hitting this ceiling, not the DDR.

I rebuilt the circuit as read-only and measured again. The true limit for the PL reading from DDR was 1.97 GB/s (93.6% of DDR). Against the CPU's 33%, that is 2.8×, so the conclusion that "the PL can pull more out of DDR" actually got stronger. The condition, though, is using two HP ports. With one, it only reaches 53.7%. Details in "Two Ceilings".

And on the ADRFT1, the CPU was pulling only 13%. What felt "slow" on that board was not the DDR's performance. It was the CPU failing to draw that performance out. When I say the hypothesis that "moving to the fabric is worth it" was backed by measurement, this gap is what I mean.

Figure 5 — The circuit I built yesterday does nothing

A circuit that just reads from DDR, loops the data once through the PL, and writes it back to DDR. It computes nothing. The goal was not to build the factory. It was to measure the width of the road that brings materials to it. Reads and writes go through the same gate, so from the DDR's point of view the work is twice 0.77, or 1.54 GB/s.

There was a reason to go out of my way to build a circuit that does nothing. If I build the compute units first and the result is slow, I can't tell whether the circuit is bad or the road is narrow. If I pin down the width of the road first, the next compute circuit only has to be sized to consume that road fully. It will be neither too big nor too small.

After each transfer I compared the source data and the result byte by byte and confirmed an exact match. Fast but broken would be pointless.

Figure 6 — It all runs from one Mac

On an Apple Silicon Mac alone, the loop from FPGA design to measurement closes. Vivado runs only on x86 Linux, but Docker and Rosetta 2 made it work. No programming hardware such as JTAG is needed. You just send the generated file over the network and load it from Python.

It looks mundane, but it may be the biggest result of this project so far. On the ADRFT1, I never managed to build this loop. Reprogramming its FPGA needed dedicated hardware (over ¥30,000), and a failed attempt could not be recovered. On the PYNQ-Z1 that came out of a drawer, the loop ran on the first day.

How I got here

8/29–30

Found the cause of the ADRFT1's instability A mistake in the device tree had the CPU claiming a memory region it cannot use as ordinary RAM. A workaround that adds two lines to the boot script fixed it completely. It is applied to both boards, and they still run stably.

9/3

The vendor said it won't fix it They replied that they know about the bug but have no plans to fix it. The workaround became a permanent configuration, not a stopgap.

9/4

Measured the bandwidth, and the picture changed The ADRFT1's CPU was pulling only 13% of the DDR. The problem was not "DSPs are left over" but "the road to memory isn't being used". It was also confirmed that the way to reprogram the FPGA was blocked, so I decided to cancel the FPGA plan on the ADRFT1.

9/5

Closed the loop on the PYNQ-Z1 Set up a 2017 board that came out of a drawer. Built a Vivado environment on the Mac, generated my own bitstream, loaded it onto the board, and measured PL→DDR bandwidth at 1.54 GB/s (74% of the limit).

9/8

That 74% turned out not to be about DDR I built a "read-only" circuit with no write-back and re-measured at different clocks. Every time, the result stuck at 99% of the datapath width. The culprit was not the DDR but the width of the AXI path. Only with two HP ports did the DDR side saturate, and the true limit was settled at 1.97 GB/s (93.6% of DDR). → "Two Ceilings"

By the way, I had been considering a move to the KV260 ($199, roughly 10× the PYNQ-Z1's performance), but the PYNQ-Z1 turned out to be more capable than I expected, so that decision can probably wait. A ternary 135M-class model fits in 512MB of memory, so there is a chance I can run it end to end on this board at least once.

What's next

Stage	Task	What it tells us
Done	Measure read-only bandwidth	Done on 2026-09-08. 1.97 GB/s (93.6% of DDR). As expected, read-only got more, but for a different reason than I thought — the headroom was not in the write-back but in the datapath width. → "Two Ceilings"
Next	Line up INT8 multipliers	Size it to the width of the road. 220 are not needed. The settled design value is 14 at 142.86 MHz (6% of the total)
Main goal	Go ternary (BitNet b1.58)	1.0 → 0.2 bytes per weight. 5× as much through the same road. At 1.97 GB/s, that is the equivalent of 9.8 G weights per second

My expectation is that once ternary is done, a 135M-class model should run 10–20× faster than on the CPU, and around 30 tokens per second should be within reach. But this is an estimate from the bandwidth I measured this time. I won't know until it actually runs.

Glossary

PS: Processing System. The finished CPU side of the chip. It runs Linux.
PL: Programmable Logic. An empty lot for circuits, where I build my own. Nearly a synonym for "fabric".
Fabric: The contents of the PL: a sea of rewritable circuitry. The name compares it to woven cloth.
Bitstream: A circuit design turned into a file that can be loaded into the chip (.bit). Loading it turns the PL into that circuit.
DSP: Small dedicated multiply circuits embedded in the fabric. This chip has 220.
HP port: High Performance port. The gate from the PL to DDR memory. Unless you build a circuit that goes through it, the PL cannot touch memory.
AXI: The standard for connecting components inside the chip. Something like a road standard.
DMA: Direct Memory Access. A carrier circuit that moves data to and from memory without going through the CPU.
Bandwidth: The number of bytes that can be moved per second. Throughout this project, this is what sets the speed.
Quantization: Making a model smaller by representing its weights with coarser numbers. INT8 is 1 byte; ternary is about 0.2 bytes.