Homelab Rack and Network Card Upgrade
imgur.com/a/tFUSe
imgur.comPosted Mar 9, 2018, 6:24 PM UTC
8922
imgur.comI recently got a new half rack locally for $50 and my datacenter switch was showing signs of failing, so I decided to do a network upgrade. This included:
-
Upgrading my Cisco C240 M3S with a Cisco UCS VIC 1285 (2 x 40 Gb QSFP) at home
-
Replacing a malfunctioning Qlogic dual 10 Gb NIC and unused quad port Intel NIC with two Mellenox ConnectX-2 NICs
-
Installing a Cisco UCS 6120xp in my datacenter lab
-
Reusing my four Mellenox ConnectX-2 cards in my C6100 in the datacenter
-
Upgrading my 6100 with eight Intel Xeon X5670 (Used to be eight L5520's)
-
Organizing my cables in both location
3
No 10Gig for the pfsense router?
2
When I upgrade from that R310, I'll get 10 or 40 Gb NIC(s) for my routers.
3
I mean my R210ii can push with dual 10Gig links about 17Gb/s of traffic between VLANs.
2
What are you using that UCS 6120 for all by itself?
1
In the datacenter, it is the 10 Gb switch for the Dell PowerEdge C6100 that replaced the failing 1 Gb switch.
1
"Home" lab. hah! Very nice
3
Awesome dude. Just awesome. #homelabpro
1
wish I could find a local half height rack locally.
1
Very nice. Any idea how many watts or amps that setup is drawing?
1
Around 1,300 watts under load.
1
I guess that isn't horrible for the amount of stuff you are doing on it. Very nice work.
1
Why are you running all that on 120v?
1
For now, it is ran off two 120v lines. This summer, I'm looking to install two lines of two phase 240V.
1
Please don't zip tie your cables :(
1
Any intent on getting rid of those 2 Intel nics?
1
How does the FI handle port licensing with the QSFP>SFP breakout cable? Does it see it as a single in use port, or as 4 separate ports? (Or some other fun cisco gotcha...)
1
No licensing required. However, the 6120xp / Nexus 5010 doesn't combine 4 x 10 Gb to 40 Gb, but does provide some redundancy on ports.
2
It looks like by default, only 8 of the ports for the FI are licensed, unless you flashed to the Nexus firmware. So I guess it would see all 8 of the pre-licensed ports in use if you're using the QSFP cable, which is a bit of a bummer. Any specific reason you went with the 6120 instead of the 6248/etc? (Aside from it being like 3 times the cost) Was planing to grab it as it should run newer fw + 12 port licenses by default, and potentially lower power draw
1
I flashed mine to the Nexus 5010 firmware since I only use the vNIC feature set from my VIC NICs, along with having more servers and desktops that needed SFP+ ports than UCS hardware. I'm currently keeping an eye on the N3K-C3064PQ-10GX as a replacement for the 5010 / 6120xp as they're cheaper than a UCS 6332 or 6332-16UP at the moment.
Any specific reason you went with the 6120 instead of the 6248/etc? (Aside from it being like 3 times the cost)
I got two 6120xp for $15, plus shipping, off eBay. So price was the deciding factor.
2
Holy shit, $15 is insanely cheap, good find!
1
I'd love to know what services you are hosting.
Infrastructure (Veeam (5 proxies, 1 managment, 2 deduplication targets), VMware (PSC, vCenter, VMware Horizon, VSAN witness appliances), Ansible, 2 Active Directory, 2 AD DNS servers, TFTP, FTP, NFS servers)
Game servers (ARK, Starbound, Factorio, Terraria, Minecraft (FTB), )
Wife's servers (see game servers, Adobe Encoder)
Web servers, databases (MySQL, MSSQL, InfluxDB), iperf, memcached, guacamole, Plex, torrent clients
A ESXi lab (7 nested ESXi VM's, 9 VyOS BGP routers, 3 web servers, 5 DNS servers, 7 clients (Linux, Windows))
Desktop / SSH clients (dynamic client deployments via VMware Horizon)
Monitoring (LibreNMS, SexiGraf, nfsen, oxidized, smokeping)
I should re-deploy VMware VIC to play with Docker on top of ESXi.
Oh, be careful with that memcached server. Patch and ensure you aren't vulnerable to amplification attacks.
It's patched and the web server that uses it is only accessible on the internal network, not exposed in any way to the outside world.
As would I
See above