Perf problem with pfSense on ESXi

Supermicro X10SL7-F NIC Intel® i210AT (onboard) Xeon 1220v3 3.1 GHz

ESXi 6.5 Running 2.3.4-RELEASE-p1 with suricata.

I get a perf peak in the first ~5s then the throuput converge to ~300 Mbps.

Sending: [ 5] 0.00-1.00 sec 94.0 MBytes 785 Mbits/sec
[ 5] 1.00-2.01 sec 86.9 MBytes 726 Mbits/sec
[ 5] 2.01-3.01 sec 98.5 MBytes 826 Mbits/sec
[ 5] 3.01-4.01 sec 95.6 MBytes 804 Mbits/sec
[ 5] 4.01-5.01 sec 101 MBytes 843 Mbits/sec
[ 5] 5.01-6.01 sec 102 MBytes 858 Mbits/sec
[ 5] 6.01-7.00 sec 48.1 MBytes 407 Mbits/sec
[ 5] 7.00-8.01 sec 45.5 MBytes 378 Mbits/sec
[ 5] 8.01-9.00 sec 30.6 MBytes 259 Mbits/sec
[ 5] 9.00-10.00 sec 48.8 MBytes 409 Mbits/sec

Receiving [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 65.0 MBytes 544 Mbits/sec
[ 5] 1.00-2.00 sec 74.7 MBytes 628 Mbits/sec
[ 5] 2.00-3.00 sec 72.3 MBytes 606 Mbits/sec
[ 5] 3.00-4.00 sec 83.7 MBytes 702 Mbits/sec
[ 5] 4.00-5.00 sec 27.8 MBytes 233 Mbits/sec
[ 5] 5.00-6.00 sec 32.0 MBytes 268 Mbits/sec
[ 5] 6.00-7.00 sec 33.0 MBytes 277 Mbits/sec
[ 5] 7.00-8.00 sec 30.4 MBytes 255 Mbits/sec
[ 5] 8.00-9.01 sec 26.7 MBytes 222 Mbits/sec
[ 5] 9.01-10.00 sec 29.3 MBytes 248 Mbits/sec

W/o suricata the throughput converge at ~500Mbps but there is still the peak at ~800 Mbps in the first ~5s.

[ ID] Interval Transfer Bandwidth [ 5] 0.00-1.01 sec 99.2 MBytes 827 Mbits/sec
[ 5] 1.01-2.00 sec 102 MBytes 859 Mbits/sec
[ 5] 2.00-3.00 sec 105 MBytes 878 Mbits/sec
[ 5] 3.00-4.01 sec 109 MBytes 911 Mbits/sec
[ 5] 4.01-5.00 sec 76.0 MBytes 641 Mbits/sec
[ 5] 5.00-6.00 sec 60.8 MBytes 510 Mbits/sec
[ 5] 6.00-7.00 sec 66.6 MBytes 558 Mbits/sec
[ 5] 7.00-8.01 sec 70.0 MBytes 584 Mbits/sec
[ 5] 8.01-9.01 sec 66.6 MBytes 559 Mbits/sec
[ 5] 9.01-10.00 sec 62.8 MBytes 528 Mbits/sec

last pid: 13781; load averages: 0.79, 0.63, 0.45 up 26+07:02:36 20:02:01 135 processes: 7 running, 106 sleeping, 22 waiting

Mem: 249M Active, 1213M Inact, 584M Wired, 603M Buf, 1885M Free Swap: 8192M Total, 8192M Free

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 155 ki31 0K 64K RUN 2 620.8H 79.98% [idle{idle: cpu2}] 11 root 155 ki31 0K 64K RUN 0 623.0H 78.96% [idle{idle: cpu0}] 11 root 155 ki31 0K 64K RUN 3 621.1H 77.98% [idle{idle: cpu3}] 11 root 155 ki31 0K 64K CPU1 1 623.8H 76.95% [idle{idle: cpu1}]

87685 root 52 0 349M 235M uwait 1 0:06 30.96% /usr/local/bin/suricata -i vmx0 -D -c /u 12 root -92 - 0K 384K CPU3 3 315:24 23.00% [intr{irq257: vmx0}] 12 root -92 - 0K 384K CPU0 0 296:09 15.97% [intr{irq258: vmx1}] 87685 root 25 0 349M 235M bpf 2 0:05 5.96% /usr/local/bin/suricata -i vmx0 -D -c /u 0 root -92 - 0K 208K - 0 28:46 1.95% [kernel{vmx1 taskq}] 30631 root 21 0 28924K 7092K kqread 3 1:00 0.98% nginx: worker process (nginx) 0 root -92 - 0K 208K - 2 12:43 0.00% [kernel{vmx0 taskq}] 91517 root 20 0 104M 37052K select 3 12:37 0.00% /usr/local/bin/vmtoolsd -c /usr/local/sh 44034 root 52 20 17000K 2384K wait 2 10:01 0.00% /bin/sh /var/db/rrd/updaterrd.sh 12 root -60 - 0K 384K WAIT 2 8:24 0.00% [intr{swi4: clock}] 62524 root 20 0 14504K 2304K select 2 5:30 0.00% /usr/sbin/syslogd -s -c -c -l /var/dhcpd 4 root -16 - 0K 32K - 3 4:20 0.00% [cam{doneq0}] 44072 unbound 26 0 89408K 69676K kqread 2 4:19 0.00% /usr/local/sbin/unbound -c /var/unbound/ 6 root -16 - 0K 16K pftm 1 4:07 0.00% [pf purge]

cpu is mostly idle during iperf running with suricata on. Running iperf with -P 4 gives the same result.

Any ideas on how to tweak or solve the problem? How can I get concurrent throughput with suricata >600Mbit ? Im thinking of getting a new box with perhaps G4650 to run pfSense on baremetal if I can't solve this.

Would 2 cores at 3.5GHz be enough for >600 Mbit with suricata ? ( ofc depends on the rule set etc...)