Trying to build a system around 5Gbe sucks

So in a nutshell, I have AT&T Fiber and have the option to upgrade to 5gig service for next to nothing (I get great pricing on AT&T services through work and my SO gets a stipend to cover our home internet).

We have the new BGW 320 Router/ONT with fiber coming in from the street and a 5GBe port on the back that currently brings in full 1 gig symmetrical service.

Here’s where it gets annoying.

If I want to completely rebuild my network around 5 gig…….but it’s a nightmare. I can find routers that support 2.5Gbe, but very very few that support 5Gbe without going through SFP+ and maybe it works maybe it doesn’t.

Routing aside, it’s getting difficult to find a switch that does multi-gigabit as well as PoE. You can find unmanaged multi-gig switches easy enough, but I’ve really only found the Netgear switches that support PoE and multi-gig - Not only are the prohibitively expensive, but I have a personal vendetta against Netgear for some warranty issues they gave us through my work.

I get that 1/2.5/5/10 multi-gig functionality is relatively new, and that enterprise really just jumped right over it and is doing much higher bandwidth negotiation, but geez I didn’t think it would be this much of a nightmare.

Posted Jul 31, 2022, 4:44 PM UTC

29
MrBigOBX•

Also getting 5Gb on your network and then onto each machine is a whole diff conversation.

Routers are a little tough as youll prob need either a homegrown PFsense type rig or something with a 10gb WAN interface and even then those dont alway negotiate well.

from there youll prob want to get one main access switch with a 10gb uplink and 1gb ports for probably most of your gear and while no one device can get more than 1gb up to 5 can get 1gb at once which is pretty damn good.

then youll prob need to get one multi gig access switch with a 10gb uplink and "multi" other ports for those select few items that can even take advantage of these higher speeds.

My perspective is with these higher than 1gb WAN links is not abut getting that speeds to EVERY machine, its more about adding WAAAAY more lanes to the highway so multi users / devices can really jam on the pipe at once.

im hoping to get optimum 2gb fiber service and do the same thing but with a 2gb wan pipe.

Also Kudos on uber fast internet, i hope you pull it off and show us some nice speed tests and some big Downloads hahahah

5
MilkshakeGOP•

Yeah the ultimate goal is to massively open up bandwidth. When my partner and I are both working from home, we can saturate the full 1 gig symmetrical now. I’m thinking about just going to 2.5 service because it’s going to be a lot more manageable….

I’ll keep you posted, still in the planning phases currently 🙂

1
diamondintherimond•

May I ask what kind of data you’re accessing in order to saturate a gig connection? Just curious because my use case is pretty much Teams calls and Microsoft documents which has relatively low bandwidth requirements.

5
1911Cheeseburger•

What are the multi gig POE for? 6E WAPs?

What’s wrong with SFP+? I’d much rather have that, then I can DAC as much as I can.

4
MilkshakeGOP•

Nothing wrong with SFP+, just a matter of finding a module that will negotiate 10gig on one side and 5gig on the other. I’ve read into a few models, but so few people are going to 5gig that it’s going to likely be trial and error.

1
1911Cheeseburger•

When doing copper SFP+’s get the 30m version. Those will do 2.5/5/10. The 100m versions will only do 10G. The negotiations happen on the SFP transceiver itself in copper, not on the SFP port. For fiber it happens on the port.

In my home network I do everything 1 or 10G. If I had 5G WAN; that’s the only bit that would be 5G would be that WAN connection.

1
jasonwc•

Your best bet to get 5Gb speeds on wired devices is either 10GBase-T if you want to use your existing Cat6 or 6A cabling, or SFP+ if you have or are willing to run fiber for longer runs (DACs are great for short runs). For my network, I use a Brocade ICX6610-48p as the core switch as it provides 48 1000Base-T ports with PoE+ (30W/port, 1500W total), 16 10G ports (8 SFP+, 2 x QSFP+ breakout), and 2 40Gb QSFP+ ports with a well-documented CLI and all the features I need (redundant fans and power supplies, Layer 3 routing, VLANs, and even MACSEC on the SFP+ ports if I want to run fiber to a neighbor). The copper ports provide data and power to my APs and security cameras, and 1Gb is fast enough for all the non-PCs/servers. PCs and servers use 10G-LR or 40G-LR4 Lite transceivers for 10GbE or 40GbE over cheap duplex SMF.

As you note, you can't really find many 5 Gbe switches because it doesn't really solve a problem. 2.5GBase-T is significantly cheaper to manufacture than 10GBase-T and it will work up to 100M on ubiquitous Cat 5e. 5GBase-T doesn't appear to be significantly cheaper than 10GBase-T, and more importantly, it requires Cat 6 per the specs. 10GBase-T works up to 55M (180ft) over Cat 6, and that's plenty. If for some reason the port can't negotiate at 10GBase-T, a modern 10Gb switch can drop down to 5Gb, which seems to be where the standard is useful.

The main issue for me is that there are no affordable 2.5GBase-T managed switches with PoE+. You can work around the lack of PoE+ by using a power injector. However, I assign different VLANs to each SSID including having RADIUS assign an appropriate VLAN based on the client TLS certificate for EAP-TLS authentication. This requires a managed switch. The only switches I've seen with multi-gigabit (generally a mix of 10/5/2.5 and 2.5Gb copper ports, SFP28, and 100Gb QSFP28 ports) are very expensive enterprise switches. There are cheap 5 and 8-port 2.5Gb switches, but they are unmanaged and do not have PoE+.

In fact, you're probably more likely to find a used enterprise 10GBase-T switch with PoE than a 2.5Gb managed switch with PoE. The problem with these older 10GbE switches is that they were manufactured before N-BaseT was a thing, so they can't negotiate at 2.5G or 5G speeds.

Since I only use WiFi for mobile devices and IoT crap that can't be wired, I don't see a compelling reason to upgrade to 2.5Gb. I chose to run SMF for future-proofing (easy upgrade from 10Gb/40Gb to 25Gb/100Gb). There's really no future for twisted-pair copper after 10GbE. While there are 25GbE and 40GBe standards, they're mostly vaporware. I found a single 25GbE enterprise switch which seems to have been produced in very small numbers. The standard requires Cat 8 cables and is limited to 30M. 30M simply is inadequate even for a home, and Cat 8 and will be exceptionally difficult to terminate at-home to the standards required. If you're running new wiring anyway, and need factory-terminated cables, SMF is cheaper and takes up far less room in conduit.

In addition, there are lots of affordable used SFP+ switches with enterprise features, as well as some very cheap passively cooled Mikrotik switches (4 or 8 SFP+ managed switches for $150 and $269). The CRS309-1G-8S+IN looks particularly nice (8 SFP+ ports on a managed switch, passive cooling, max power consumption 23W, and can be powered from the single copper port via PoE+). 10GBase-LR used transceivers only cost me $5/each on Ebay and the 40GBase-LR4 Lite transceivers cost $12 (4 wavelengths of 10Gb over SMF). There's also great documentation for the Brocade switches and helpful instructions online to get everything working and licensed for free.

Most of the SFP+ 10G copper modules also negotiate at 1G, 2.5G, and 5G. I know the Mikrotik one does, and folks have confirmed it works with the Brocade switches. Brocade switches will basically work with any transceiver within power limits. See https://mikrotik.com/product/s_rj10

Users also report that the fs.com 10GBase-T SFP+ module will negotiate at 2.5G (and presumably 5G). https://www.fs.com/products/66613.html

Mikrotik also appears to work with any transceiver. Mikrotik says "Note: MikroTik devices and SFP/SFP+/SFP28/QSFP+/QSFP28 modules do not have any restrictions for other vendor equipment. As long as the other vendor modules and devices comply with transceiver multi-source agreement (MSA) they should be compatible with MikroTik."

3
craigmontHunter•

I have 1.5gbe service, my solution for routing is I use a seperate device for DHCP, then pass the ISP router as the default gateway. I losses some insight into network traffic, but it saved me the headache of building/buying a router capable of over 1gbps (there is a 10g port on the router).

My current DHCP solution is windows server, tied into my active directory and DNS, prior to the last hardware failure it was openwrt on a Linksys 1200ac router.

Beyond that I am really hoping for multigig switches to drop back to reasonable prices, Netgear has some with nice options for home use, but if you have a vendetta I have seen some trendy options that seem promising, I will probably use one with DAC when I have budget again.

1
MilkshakeGOP•

See, and I dont want to get too far down the path of homelabbing and tinkering because that’s my day job.

It was one thing when it was just me, but having a partner who works from home means 100% uptime no exceptions 🙃

1
Crimtide•

What'd you end up deciding? We are considering upping to 5Gbps from ATT in a couple weeks in our new home, mostly to feed 1Gbps to more devices at the same time, not necessarily to get 5Gbps to my workstation. Looking myself for something that has minimum 9 ports. I have 8 drops I need to wire, preferably one of those being multi-gig to hog bandwidth when needed. But I am having a tough time finding a switch that will do something like 8x1Gbps ports and 2x5/10Gbps ports.

1
MilkshakeGOP•

Still mulling it over unfortunately.

I think I’ve settled on doing 10g core switching, UDM pro with SFP+ (multi gig SFP adapter) for WAN, SFP+ out into an aggregation switch, and then a mix of 10g/1gbe switching. Wi-Fi/APs are going to have to be 1gbe until some more multi gig switching comes out.

1
Crimtide•

I ended up just buying a cheap Netgear that I found searching, model GS110EMX. 10 ports total, with 2 being up to 10 gig. So it gets 8 x 1 Gig ports and 2 x 10 gig ports. I thought about the UDM Pro myself, but it's got a bunch of stuff I will never use, like the built in security camera system / recording, etc.. so not worth the $400.

The netgear will do for now, until like you said, more multi-gig switches come out.

1
MilkshakeGOP•

Netgear gave me all sorts of warranty issues when a product was defective at work so I have a vendetta against them.

My other issue, multi-gig + Poe gets prohibitively expensive, especially if you go above 8 ports or so. (A 16 port from netgear is $2400. It gives you 1/2.5/5/10 and Poe across every port though so that’s nice)

The UDM pro is also about the only router that can (technically) do multi-gig. There’s a newer Mikrotik that could also has 2.5g WAN which is doable, but 5-gig/multi gig is still MIA from most company’s sku lists

1
Crimtide•

Yea, I don't plan to use their support for anything. The device will have to be bricked for me to not figure it out. This is only temporary, to get up and running in a new home build and with 2 x multi-gig ports it works for me because I only need to feed multi-gig to one room, all the others can have 1 gig. I also don't need POE as I already have a separate NVR supplying that. Which is why I don't need the added features of UDM Pro.

It was on sale for $190. Can't beat it for a quick setup. I guess I'll see how it goes in 2 weeks when we move in.

1