Some Real DDR4 and PCIe Bandwidth Numbers for Early FPGA System Design
In my previous post, I talked about how I usually translate requirements into an initial FPGA/ASIC system architecture. One thing that post was missing was some actual performance data.
So this is a small follow-up focused on DDR bandwidth and PCIe DMA bandwidth.
None of the numbers below are my own benchmarks. They come from published papers, vendor performance reports, and open-source benchmarking projects. I am basically just collecting the data in one place because I think these numbers are useful when doing early-stage bandwidth budgeting.
They should be treated as reference points, not guaranteed performance numbers.
DDR4 bandwidth: what do real access patterns look like?
A useful recent reference is the 2025 paper:
“A Benchmarking Platform for DDR4 Memory Performance in Data-Center-Class FPGAs”
The authors built an FPGA-based DDR4 benchmarking platform with a configurable traffic generator. It can generate:
- read-only / write-only / mixed read-write traffic
- sequential and random addresses
- single transactions and burst transactions
- different burst lengths
- different traffic-generation policies
Hardware counters are then used to measure throughput, latency, refresh effects, and other memory behavior.
The test setup is shown below.
[Figure 1 — DDR4 benchmark platform]
The platform uses an AMD Kintex UltraScale XCKU115, with DDR4 tested at:
- 1600 MT/s
- 1866 MT/s
- 2133 MT/s
- 2400 MT/s
For DDR4-1600, for example, the PHY runs at 800 MHz and the AXI interface at 200 MHz.
Some of the measured results are shown below.
[Figure 2 — DDR4 throughput vs. access pattern / burst length]
There are two practical observations here that are worth keeping in mind during architecture design.
First, sequential access is your friend. If the algorithm allows you to turn scattered accesses into long sequential transfers, do it.
Second, if random access cannot be avoided, transaction size matters a lot. Larger bursts generally make much better use of the DDR interface than many small, fragmented transactions. Random access does not suddenly become efficient just because the burst is larger, but the overhead can be amortized much better.
The paper also evaluates mixed read/write traffic:
[Figure 3 — Mixed read/write DDR4 performance]
This is another case that is easy to underestimate during spreadsheet-level bandwidth calculations. A design that says:
may look perfectly fine against the theoretical DDR bandwidth.
The controller, however, does not necessarily see an ideal 20 GB/s stream. Read/write turnaround, bank conflicts, row behavior, arbitration, refresh, burst size, and address mapping can all reduce the usable bandwidth.
That is why I normally leave a fairly large margin when sizing external memory bandwidth.
PCIe bandwidth: Xilinx/AMD QDMA Gen3 x16 data
Another useful public dataset is the Xilinx QDMA Performance Report, Answer Record 71453.
The reference platform is:
- VU9P / VCU1525
- PCIe Gen3 x16
- MPS = 256 bytes
- MRRS = 512 bytes
- Extended Tags enabled
- Relaxed Ordering enabled
The report covers both the DPDK driver and Linux kernel driver, and includes:
- ST mode
- MM mode
- BRAM-backed designs
- DDR-backed designs
- C2H
- H2C
- bidirectional traffic
- PF and VF
- different queue counts
- different packet sizes
For reference, PCIe Gen3 x16 provides:
8.0 GT/s × 128/130 × 16 = 126.03 Gbit/s
of link bandwidth in one direction after 128b/130b encoding.
This is not application payload bandwidth. TLP headers, DLLP traffic, Completion overhead, flow control, DMA behavior, host memory, and software overhead all reduce what the application actually sees.
The QDMA report provides detailed results for the DPDK TestPMD forwarding path, effectively exercising a path similar to:
C2H → host memory/software → H2C
The important packet-size points are summarized here:
| packet size | 8 queues | efficiency | 4 queues | efficiency | 2 queues | efficiency | 1 queue | efficiency |
|---|---|---|---|---|---|---|---|---|
| 64 B | 42.32 Gbps | 33.6% | 49.98 Gbps | 39.7% | 36.55 Gbps | 29.0% | 18.28 Gbps | 14.5% |
| 128 B | 66.51 Gbps | 52.8% | 67.31 Gbps | 53.4% | 63.98 Gbps | 50.8% | 31.99 Gbps | 25.4% |
| 256 B | 80.39 Gbps | 63.8% | 81.12 Gbps | 64.4% | 81.59 Gbps | 64.7% | 51.20 Gbps | 40.6% |
| 512 B | 90.03 Gbps | 71.4% | 90.80 Gbps | 72.0% | 90.87 Gbps | 72.1% | 73.14 Gbps | 58.0% |
| 1024 B | 94.26 Gbps | 74.8% | 96.55 Gbps | 76.6% | 96.81 Gbps | 76.8% | 93.09 Gbps | 73.9% |
| 1536 B | 96.86 Gbps | 76.9% | 99.11 Gbps | 78.6% | 99.34 Gbps | 78.8% | 99.75 Gbps | 79.1% |
| 2048 B | 100.57 Gbps | 79.8% | 100.90 Gbps | 80.1% | 101.01 Gbps | 80.1% | 101.04 Gbps | 80.2% |
| 4096 B | 101.27 Gbps | 80.4% | 103.03 Gbps | 81.7% | 103.48 Gbps | 82.1% | 103.44 Gbps | 82.1% |
[Figure 4 — QDMA PF forwarding throughput vs. packet size / queue count]
The report also provides VF results:
| packet size | 8 queues | efficiency | 4 queues | efficiency | 2 queues | efficiency | 1 queue | efficiency |
|---|---|---|---|---|---|---|---|---|
| 64 B | 42.50 Gbps | 33.7% | 50.06 Gbps | 39.7% | 36.55 Gbps | 29.0% | 18.28 Gbps | 14.5% |
| 128 B | 67.29 Gbps | 53.4% | 67.62 Gbps | 53.7% | 63.99 Gbps | 50.8% | 31.99 Gbps | 25.4% |
| 256 B | 81.26 Gbps | 64.5% | 82.08 Gbps | 65.1% | 81.81 Gbps | 64.9% | 51.20 Gbps | 40.6% |
| 512 B | 90.45 Gbps | 71.8% | 91.80 Gbps | 72.8% | 91.73 Gbps | 72.8% | 73.14 Gbps | 58.0% |
| 1024 B | 93.40 Gbps | 74.1% | 97.51 Gbps | 77.4% | 97.61 Gbps | 77.5% | 93.09 Gbps | 73.9% |
| 1536 B | 96.81 Gbps | 76.8% | 100.08 Gbps | 79.4% | 100.17 Gbps | 79.5% | 100.17 Gbps | 79.5% |
| 2048 B | 100.67 Gbps | 79.9% | 101.41 Gbps | 80.5% | 101.42 Gbps | 80.5% | 101.42 Gbps | 80.5% |
| 4096 B | 101.32 Gbps | 80.4% | 103.90 Gbps | 82.4% | 103.90 Gbps | 82.4% | 103.92 Gbps | 82.5% |
[Figure 5 — QDMA VF forwarding performance]
At large transfer sizes, PF and VF performance is quite close. At 4 KB, the measured throughput is already around the 100 Gbit/s class on this Gen3 x16 platform.
What I find more useful than the absolute number is the shape of the curve.
For QDMA, small packets are very sensitive to queue count and the number of outstanding/in-flight requests.
Once packet size reaches roughly 2 KB or 4 KB, the difference between 1, 2, 4, and 8 queues becomes much smaller.
This matches the recommendation in the Xilinx report:
- In ST mode, four or more queues with descriptor prefetch enabled can significantly improve performance, especially for smaller packets.
- In MM mode with BRAM, a 4 KB workload can reach line rate even with a single queue.
- When the MM backend is DDR, more queues may be required, and DDR access patterns start to matter as well.
That last point is important.
Sometimes what looks like a “PCIe bandwidth problem” is actually a DDR bandwidth problem behind the PCIe DMA engine.
A few engineering lessons I would take from these benchmarks
1. DDR efficiency is not just about cache-line alignment
Alignment matters, but DDR performance also depends heavily on how addresses map into:
row / bank / bank-group / column
The open-source Shuhai memory benchmark is a good demonstration of this.
Its experiments show that changing address mapping, stride, and burst size can significantly change achievable bandwidth, even though the nominal DDR interface bandwidth has not changed at all.
So if an FPGA design is heavily DDR-bound, I would not stop at:
That is only part of the story.
2. Test the AXI/DDR path around 4 KB boundaries
This is something worth testing separately.
An AXI burst cannot cross a 4 KB address boundary.
So a long transfer arriving close to the end of a 4 KB region has to be split into multiple bursts.
In a well-buffered large-transfer system this may not matter much, but for small packets, fragmented DMA traffic, or mixed read/write workloads, those extra transactions can become visible.
For a real design I normally include dedicated test cases for:
- naturally aligned bursts
- deliberately misaligned bursts
- transfers ending exactly at 4 KB
- transfers crossing a 4 KB boundary
- random packet sizes around the boundary
It is much better to discover this during the bandwidth benchmark than after the whole datapath has been integrated.
3. PCIe H2C is sensitive to host-address alignment
H2C DMA is fundamentally a PCIe Memory Read operation from the endpoint's point of view.
The FPGA issues Memory Read Requests, and the Root Complex returns data through Completion TLPs.
Completion packetization is affected by the Read Completion Boundary (RCB) and request/address alignment. A poorly aligned DMA buffer can therefore require more Completion TLPs than an equivalent aligned transfer.
For large DMA blocks this overhead may be negligible.
For many small DMA reads, it is not.
This is why I would include host-buffer alignment in any serious PCIe DMA benchmark rather than benchmarking only one perfectly aligned buffer.
4. Small PCIe DMA transactions can be extremely sensitive to the IOMMU
The public pcie-bench results are particularly interesting here.
In one of their tested Intel systems, with 4 KB IOMMU pages and sufficiently large DMA windows:
- 64-byte DMA Read throughput dropped by almost 70%
- 256-byte DMA Read still dropped by roughly 30%
- 64-byte DMA Write dropped by roughly 55%
The penalty largely disappears as the transfer size grows.
The reason is straightforward: when each DMA operation contains very little useful data, the cost of IOTLB misses and page-table walks becomes a significant part of the total transaction time.
This is a good example of why:
does not tell you very much about the performance of a real small-transfer DMA workload.
5. Public benchmark numbers are baselines, not acceptance criteria
This is probably the most important point.
For both DDR and PCIe DMA, measured bandwidth depends on the complete system:
FPGA side
- DDR controller configuration
- AXI width and clock
- outstanding transaction depth
- burst length
- address mapping
- bank conflicts
- DMA architecture
- queue depth
- descriptor prefetch
- MPS / MRRS
Host side
- Root Complex implementation
- CPU architecture
- NUMA topology
- memory-channel configuration
- IOMMU
- page size
- RCB
- driver
- DMA buffer placement
- CPU affinity
- competing memory traffic
NUMA deserves special attention.
On a multi-socket or multi-NUMA-node server, putting a PCIe device on one NUMA node while allocating its DMA buffers on another can hurt DMA latency and bandwidth badly. Xilinx explicitly warns about this in the QDMA performance report.
There is another effect that is often overlooked: PCIe DMA ultimately competes for host DRAM bandwidth.
A CPU may have a very large number of cores, but that does not mean memory bandwidth scales proportionally with core count. If many cores are already generating heavy DRAM traffic, the PCIe DMA engine is competing for the same memory-controller resources.
In that situation, you may have:
- a PCIe link running at Gen3 x16 or Gen4 x16
- a DMA engine capable of line rate
- no obvious FPGA bottleneck
and still fail to reach the expected DMA throughput because the bottleneck has moved into the host memory subsystem.
This is one reason I always prefer to benchmark:
FPGA ↔ PCIe ↔ Root Complex ↔ NUMA node ↔ DRAM
as one complete path.
Final takeaway
When doing early architecture work, theoretical bandwidth is still useful. You need it for the first sizing pass.
But I would separate three numbers:
Theoretical bandwidth → controller/link bandwidth → application-sustainable bandwidth
They are not the same thing.
For DDR, I would pay particular attention to burst size, address locality, read/write mixing, bank mapping, and outstanding transactions.
For PCIe, I would look at packet size, outstanding requests, queue depth, MPS/MRRS, alignment, IOMMU, NUMA placement, and host memory bandwidth.
Public benchmark data is very useful for establishing a baseline.
But before freezing the architecture, I would still build a small traffic generator and measure the actual board and host platform.
It is usually much cheaper to discover a 30% bandwidth gap with a traffic generator than after the application RTL is finished.
References
- Andrea Galimberti et al., A Benchmarking Platform for DDR4 Memory Performance in Data-Center-Class FPGAs, ISCAS 2025
- AMD/Xilinx, QDMA Performance Report, Answer Record 71453
- Wang et al., Shuhai: Benchmarking High Bandwidth Memory on FPGAs, FCCM 2020
- Huang et al., Shuhai: A Tool for Benchmarking High Bandwidth Memory on FPGAs, IEEE Transactions on Computers
- Neugebauer et al., Understanding PCIe Performance for End Host Networking, pcie-bench
Posted Aug 23, 2026, 2:48 AM UTC