10Gbit ISP performance with Mikrotik x86?

I have a situation where we are doing about 4.5gbit of WISP traffic (~2000 subscribers, mostly residential) and our CCR1072 is starting to get overwhelmed, mostly with queues. Once it gets beyond 60% CPU we start to see glitchiness like 2-3 sec drops of all traffic, or intermittent maxing of all cores to 100% (profiler just showing "Networking" as the culprit).

Ideally we'd like to install a dedicated traffic shaping appliance (with that sweet sweet FQ_CoDel), so we can get better performance and reduce the load on the 1072 - but we only have 2U available in the rack we are subleasing so we can't fit the two CCRs + 1U server it'd take to do that. The next move up is a full rack + crossconnect, which is going to cost an arm and a leg extra in monthly fees.

One thought I had was to put in a Xeon 4 x blade server, as you can get some seriously powerful ones that will fit into 2U, and then run the x86 Mikrotik on two blades and the traffic shaping software on another. I think I can get interface cards with enough 10Gbit ports to make that work. I'm not able to find any recent forum posts out there with folks doing anything similar though.

I know we'd be dropping down from 72 cores to closer to 24 cores, but they'd be far more powerful cores than Tilera, and we'd have a boatload more RAM, CPU cache etc.

Or alternatively is there any way to have traffic from upstream get sent out to a traffic shaping box and have it return to the same router before being sent back out again to the network? I've tried messing around with VRFs and policy routes but couldn't make it work on the bench.

Posted Aug 21, 2020, 1:07 AM UTC · edited

923
endz420•

I run X86 routeros with ~10000 queues for 2500 customers (4 queues per customer for different priorities). I'm using a fairly old HP dual xeon 2x6 core server (ebay special), and a intel 10gb nic. Customers push just under 3Gb of traffic during peak times and the CPU load never goes much above 25%. I used to have a CCR but I was having load issues, especially when gathering data from the queues using the API. X86 absolutely smokes the tile CPU. Be aware that X86 routeros is 32bit so any ram above 2GB isn't used. Not a problem when just doing stuff like queues and basic routing though.

9
lizardlikeOP•

Oh perfect this is exactly what I needed to know, thanks!

3
velhomonge•

Hello! Can you explain why do you use 4 queues pee client? Thanks!

2
endz420•

I break down the traffic into 3 categories: TV voip general (everything else) +a master parent queue to group everything together for each customer. The TV and voip queues guarantee a minimum bandwidth for each service (i.e. 256k/256k for voip, 4M/256k for TV).

2
caes95•

You're NATing right?, because that will cut your CCR performance, what I do in this type of setup is division of roles, maybe a CCR2004 with fasttrack (if you can) for receive the upstream links and do NAT and the CCR1072 for handle the subscribers queues.

I have an CCR1072 and a CCR1036-2S+ with this setup and the 1072 passing 2Gbps with near of 1800 subs, peak cpu usage is 16%. Conntrack is off in the 1072. The 1036 has 3 upstreams with PCC, max cpu 42%

That's my experience, hope you manage to solve it.

2
lizardlikeOP•

Yup, CGNAT with several thousand rules (although jump rules per /24 so it’s fairly efficient). Profiler shows firewall is only about 18% but queuing is 40%+ which is why we’d love to offload that part.

If we can’t figure a way to shoehorn a server and second router in, then yeah just splitting things across two routers will buy us some time for sure. We have a second 1072 that’s just a cold spare right now.

Queuing works with conntrack off? I had no idea, I thought it was required for that. Could make a big difference then if we split it up.

2
caes95•

yeaah, queues works with conntrack off, check my setup: https://imgur.com/a/jvx8y09

4
lizardlikeOP•

Sweet good to know thanks!

1
willricci•

Why is that device queueing at all?

You should be queueing on the cpe's.

Theoretically, a computer running ros may do better with a beefy enough cpu and dpdk support but things are fundamentally wrong if your doing all that on a single device anyway.

A random xeon will not be a solution, but a specially built machine might do it.

2
lizardlikeOP•

Mostly to be able to dynamically adjust speeds in response to changing backhaul capacities. A few other ISPs that I know that are the same size as this one are doing this with Preseem and Swift Fox and it works great. Backhaul gets overloaded in extreme weather or something and it automatically cuts package speeds for the clients using it so pings stay perfect.

Also it simplifies administration immensely. Can change a customer package in billing software and it pushes automatically. Can change the speeds offered for a package without having to log into hundreds of radios.

1
systemdad•

Also it simplifies administration immensely. Can change a customer package in billing software and it pushes automatically.

Frankly, the CPEs should already be 100% configuration managed one way or another, so it shouldn't really change where the change is made, it should be centralized either way.

2
willricci•

Yikes.

Man that's a name I never thought I'd hear again.

There's plenty of proper ways of standardizing speeds/radios. The problem with queues is shaping someone on the core means they can be hammering a wireless link that's just retransmitting like a mofo while the core is shaping things. It's an awful thing to do for congestion.

TE does the same thing, ping is an awful metric.

2
Irkutsk2745•

Turn off NAT and turn off Conntrack. NAT needs conntrack to work.

2
dbolly•

Running a web host using only Mikrotik CHRs as edge routers. 40 edges around the world, some pushing <100Mbit and others pushing >5Gbit. All IPs announced Anycast over all edges, then routed internally using hundreds of GRE tunnels managed with OSPF.

Using modern 4/6/8 core desktop Intel CPUs with the highest possible clocks. Mikrotik isn't a highly threaded beast and still relies on single threaded performance, so you want the fastest core speeds you can get (in favour of many cores) and fastest RAM. Avoid AMD Ryzen as the inter-core and memory latency is too high. Intel X520 and newer NICs are working best for us.

2
lizardlikeOP•

Oh that’s really interesting to know. Great info there, I would’ve expected cores to take precedence given how the CCR1072 sacrificed clockspeed to get more of them compared to the 1036.

I would expect Xeon to do even better than the desktop ones though right? Better on die cache and so forth.

1
dbolly•

You're forwarding packets, not caching them. Cache size is irrelevant. Besides, cache and cores are much, much slower on Xeon CPUs to fit tight datacentre power/thermal margins. Desktop CPUs are 100% the way to go. Even quad core i7/i5 CPUs hit 5+Gbit of traffic without issue.

2
dbolly•

Also, FWIW, our slowest CHR is running on an old Pentium G3258 CPU (dual core 3.2GHz) with 8GB DDR3 1600MHz. It's one of our cores at a compute site, and frequently hits 2Gbit+ traffic during evening peak, with 27% average CPU utilisation.

It runs 9 GREs, 41 OSPF, 301 queues (for 301 unique VMs behind it), about 50 NATted ports, and it does DNS caching.

2
Santyx32•

There's no fq_codel on ROS

1
lizardlikeOP•

Yup that’s why we’d have a non-ROS x86 server for it in addition to the Mikrotik servers to do our traffic shaping. It’s built into the mainline Linux kernel though so can’t be too hard to set up.

It’s really too bad it’s not built in though. Would save a lot of trouble.

2
mitchellrrrr•

I signed up with reddit just to respond to your question.

​

Preseem / Saisei

https://www.preseem.com/

https://www.saisei.com/

​

​

Preseem would be exactly what you want as it will handle all the shaping/queuing for your clients using that beauituful FQ_CoDel stuffz.

Saisei allows you to use more of your bandwidth itself and monitors your overall network

1
sharmeena•

I've Mikrotik CCR-1036-8G-2S+ Router board, on which I'm running PCQ with 2.1 Gbps data with CPU usage 40% to 60% and 146 firewall mangle rules

How do I increase data throughput from 2.1 Gbps to 6.2 Gbps with low CPU usage?

Is it possible on Mikrotik?

If not, can someone please suggest any other cheap solution.

1