Garagerack v5 - 10Gbe, Wifi 6E and a few things learned
Posted Sep 9, 2023, 6:55 AM UTC
12
Definitely overkill, but I finally have solid 10Gbe in the house and Wifi6E for the few devices that support it at least for now. Much better coverage for all my home use-cases like 4K video editing on the Flashstor NAS and little to no interference on the 6Ghz Wifi6E is really nice to have.
Few things learned...
The Enterprise 24 PoE has been solid for APs, cameras, and various devices around the house. Aggregating 2 SFP+ ports for 20Gb to the USW-Aggregation helps raise the ceiling with all the cameras, random traffic and 2 hungry 10Gbe NAS devices.
One of the UNVRs runs 9 cameras which the UDM-Pro could handle fine, but it was really nice to offload protect onto the UNVR since it consumed 20-30% CPU on the UDM-Pro consistently. It's more peace of mind than anything, but the UDM-Pro has been running <=10% CPU with only the Network app (stopped Protect and uninstalled Talk). I put 3x Samsung 870 SSDs in the UNVR with RAID5 which was a massive difference in scrubbing videos and loading streams/events. While the UDMP did fine, having faster scrubs is a luxury as well as more record time and redundancy.
My spare UNVR I'm actually using solely as a backup NAS for my primary Flashstor 12 NAS. Protect app is stopped and a simple SMB hack was configured to give a little more data redundancy. Yes it survives reboots and upgrades (so far that I've tried) and it's easy to setup. I'm running 4x Exos X20 20TB HDDs in RAID10 and I'm very impressed with the SMB3 throughput (no encryption, no signing) where I'm seeing 800MBs read and 400MBs+ write consistently. I only need this as an onsite backup for peace of mind and fully aware it's not the intended purpose. Still very impressed with what you get for $300 even without any NAS software sitting on top. NFS works fine, but the clients I have accessing it perform better with SMB for the time being.
I've had OK luck with the UACC-CM-RJ45-MG SFP+ RJ45 adapters, but had one recently fail and RMA'ing now. It took me far too long to realize the SFP module was the issue and that was after running a new cat6a replacement cable for a few hours thinking that was it (sadly, no cable tester). I have a few backup IOPlex modules that have been working great and it runs about the same temp as the UI modules. Consider spare modules and a cable tester.
I've been using 10Gtek DACs which have been solid and cheap. They have many short lengths available like 0.25m, 0.3m which has helped keep cabling clean.
I have this all mounted in the garage with a higher ambient temp and I've been closely monitoring temps for all devices, drives, etc. under load and very happy how well the cooling works. I like the AC Infinity 1U that pulls air into the case from 2nd to bottom U and there are two AC Infinity fans in the top of the case linked to the same 1U to push out the top. Dust hasn't been an issue.. yet.
Almost finished the NUT config on a hidden RPi connected to the UPS for graceful UNVR and Flashstor shutdowns. This UPS is a bit underpowered, but just enough to get in a graceful shutdown before the battery runs out. I've never had an issue with sudden power loss corrupting the single WD Red Pro I used to have in the UDM-Pro for the cameras and haven't considered graceful shutdown for it.
I like the Patchbox /dev/mount cage nuts a little more than rackstuds, but both are better than standard cage nuts IMO. Definitely makes racking easier.
Had a laser cut acrylic plate made to put on the front of the 2x sonos amps and flashstor above it at the top of the rack sitting on a shelf. Still TBD if it helps airflow, but it was mostly for looks. Glued some magnets on the acrylic to make it easy to remove.
The middle patch panel I just spray painted the keystone blanks with the fan-favorite Rust-Oleum Metallic spray paint which is a pretty good color match. Still awkward that the UNVR SFP ports are in the back, so I hacked some of the blanks to wrap around the DAC, but still not as clean as I would like.
Happy with the upgrade path so far and it's been pretty painless, fun and not obscenely expensive. Hopefully holds me over until UI brings more 10Gbe devices to market or my need for more 10Gbe devices grows.
1
Beautiful install, thank you for sharing.
1
Thanks!
1
What SFP cables are you using to connect your switches?
1
The Enterprise 24 PoE has been solid for APs, cameras, and various devices around the house. Aggregating 2 SFP+ ports for 20Gb to the USW-Aggregation helps raise the ceiling with all the cameras, random traffic and 2 hungry 10Gbe NAS devices.
How did you set this up? When I tried to aggregate the two SFP+ ports on the 24 POE, i ran into some STP issues.
1
It's pretty vanilla, but using RTSP global switch settings and that's worked perfect for all my Sonos gear (none are wired). This has worked really well through several switches and several versions of UnifiOS/firmware. Luckily I haven't had STP issues since I stopped using Sonos devices over ethernet.
Otherwise, the aggregation was super simple to setup in the latest UnifiOS port mapper. Just have to do the aggregation on the downstream switch before the Agg otherwise.
1
So you are not using STP but RTSP for sonos? All sonos is wireless? Is sonos stable that way?
1
Yes RSTP and all Sonos is wireless. Yes it's been solid through multiple switch upgrades and APs over the years.
1
Where did you get the rack from?
1
[removed]
1
Nice! What are the blanks with holes you used to pass through your direct attach cables?
1
[removed]
1
Nice run down, and feed back on your experience! Thank you.
One question / request, when you complete the NUT config can you share that experience and script as well please?
1
Thanks! I will and I've definitely benefitted from other's sharing their experience with NUT
1
What's the need for the second and third unit having all the plugs going into each other? What is that device?
1
Not sure which you're asking about but for the silver unifi gear top to bottom its: 2.5Gb PoE switch, cat6 patch panel, 10G SFP aggregation switch, cat6 patch panel, UDM-Pro router and 2 of the UNVR on the bottom.
1
Sorry yeah The cat 6 patch panel. I have to look up more about that
1
Yeah it's the 24 port patch panel from UI with Cable Matters shielded RJ45 couplers.
1
I would assume Front to rear cat6, so all the patching (mess) is hidden at the back of the panel..
1
Yes cat6 all the way through. Definitely a hot mess at the back of the cabinet especially since it's only 15" of usable depth.
1
sorry what are your internet speeds
1
2g fiber (non complete)
1000/300 cable (primary for now)
1
Is one of those lessons that that Cyberpower UPS is woefully inadequate for what you’ve got going on there?
1
Yes, it's only 700VA/400W and I have about 10min of battery runtime under load with the current configuration. Definitely want more battery capacity, but that's tough in a 1U without spending more $/watt than a 2U
1
See I want to build something like this, but I hate cable management and I have no where near the money to buy all the unifi equipment or even a place to put it 😅.
1
Still a huge fan of buying used network gear and cable management takes time+patience which might be the toughest thing to acquire
1
Clean AF!
1
Thanks!!
1
Clean
1
Thanks!!
1
Nice setup - what’s the rack that this is mounted to. Looks exactly like what I’m looking for
1
[removed]
1
Nice setup. What model flashstor nas are you using and what's it for/what are you running on it?
1
Thanks! Flashstor 12 with 6x TeamGroup MP34 4TB that have been a great value so far. Mostly used for backups, photos + editing, videos + editing, streaming non-transcoded 4kUHD (plex/jellyfin), bunch of lightweight docker containers with a few personal projects with decent disk IO and massive data sets. Nothing close to a full server, but super useful for projects needing high disk IO and minimal CPU/RAM.
1
What the infinity ac role there at the bottom ?
1
Does the ac infinity at bottom helps a lot ?
1
It pulls colder air into the cabinet and there are two fans in the top of the case to pull hot air out. The top fans are controlled by the bottom AC Infinity unit and I’ve been experimenting with different probe location to make sure the fans are moving enough air for the hottest device.
1
How did you manage to setup 10Gbe? Which switch are you using? I guess the access points are connected to the switch as well ?
1
The USW-Aggregation is really where the 10Gbe support is and a separate USW-Enterprise-24-PoE only has half 1Gbe ports and other half 2.5Gbe ports which powers wifi6e APs with ease and the rest of the devices that need 1Gbe or slower speeds.
So few devices I need 10Gbe on all run through the aggregation switch and it's been working well. That will most likely be the next upgrade point if I add more 10Gbe devices .
1
2x UNVR‘s. You are a madman 😅
1
Thanks ;) Still impressed at what you get for $300 on the UNVR so justifying 2 of them was not hard
1
[removed]
1
Looks amazing. Why did you put it in the garage tho? I'm guessing noise and heat.
1
Thanks! Garage was only space beyond a small closet where airflow would be very difficult to manage. That and noise for sure even though the whole rack is quiet with the AC Infinity stock fans being the loudest part. Rack has been there for year with 0 issues.
1
Hello! Thanks for posting on r/Ubiquiti!
This subreddit is here to provide unofficial technical support to people who use or want to dive into the world of Ubiquiti products. If you haven’t already been descriptive in your post, please take the time to edit it and add as many useful details as you can.
Please read and understand the rules in the sidebar, as posts and comments that violate them will be removed. Please put all off topic posts in the weekly off topic thread that is stickied to the top of the subreddit.
If you see people spreading misinformation, trying to mislead others, or other inappropriate behavior, please report it!
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.