10Gbit ISP performance with Mikrotik x86?
I have a situation where we are doing about 4.5gbit of WISP traffic (~2000 subscribers, mostly residential) and our CCR1072 is starting to get overwhelmed, mostly with queues. Once it gets beyond 60% CPU we start to see glitchiness like 2-3 sec drops of all traffic, or intermittent maxing of all cores to 100% (profiler just showing "Networking" as the culprit).
Ideally we'd like to install a dedicated traffic shaping appliance (with that sweet sweet FQ_CoDel), so we can get better performance and reduce the load on the 1072 - but we only have 2U available in the rack we are subleasing so we can't fit the two CCRs + 1U server it'd take to do that. The next move up is a full rack + crossconnect, which is going to cost an arm and a leg extra in monthly fees.
One thought I had was to put in a Xeon 4 x blade server, as you can get some seriously powerful ones that will fit into 2U, and then run the x86 Mikrotik on two blades and the traffic shaping software on another. I think I can get interface cards with enough 10Gbit ports to make that work. I'm not able to find any recent forum posts out there with folks doing anything similar though.
I know we'd be dropping down from 72 cores to closer to 24 cores, but they'd be far more powerful cores than Tilera, and we'd have a boatload more RAM, CPU cache etc.
Or alternatively is there any way to have traffic from upstream get sent out to a traffic shaping box and have it return to the same router before being sent back out again to the network? I've tried messing around with VRFs and policy routes but couldn't make it work on the bench.
Posted Aug 21, 2020, 1:07 AM UTC · edited
I run X86 routeros with ~10000 queues for 2500 customers (4 queues per customer for different priorities). I'm using a fairly old HP dual xeon 2x6 core server (ebay special), and a intel 10gb nic. Customers push just under 3Gb of traffic during peak times and the CPU load never goes much above 25%. I used to have a CCR but I was having load issues, especially when gathering data from the queues using the API. X86 absolutely smokes the tile CPU. Be aware that X86 routeros is 32bit so any ram above 2GB isn't used. Not a problem when just doing stuff like queues and basic routing though.
Oh perfect this is exactly what I needed to know, thanks!
Hello! Can you explain why do you use 4 queues pee client? Thanks!
I break down the traffic into 3 categories: TV voip general (everything else) +a master parent queue to group everything together for each customer. The TV and voip queues guarantee a minimum bandwidth for each service (i.e. 256k/256k for voip, 4M/256k for TV).