March 2018, WIYH?

PinnedMegapost

#Acceptable top-level responses to this post:

  • What are you currently running? (software and/or hardware.)
  • What are you planning to deploy in the near future? (software and/or hardware.)
  • Why are you running said hardware/software?
  • Any new hardware you want to show.

#Previous WIYH:


#View all previous megaposts here!


Tempest~

Posted Mar 16, 2018, 7:26 PM UTC

2265
NOTNlCER720XD | 192GB RAM | 11TB + 480GB•

Big progress here in the last week. Retired my whitebox storage server and my PowerVault MD3200i. Storage server was virtualized and VMs moved onto my new R720xd. Currently, I'm running:

  • R720xd (2 x E5-2660, 128GB RAM, 2x480GB SSD, 6x3TB HDD)
  • R710 (2 x X5650, 192GB RAM, 2x1TB HDD, 4x2TB HDD)
  • R210 II (E3-1220v2, 16GB RAM, 2x500GB HDD)

R720xd is the newest Hyper-V host, running a plethora of VMs. This weekend I'll configure replication to the R710. R210 II is backup DC/DHCP Failiver/RRAS.

11
didyoureset•

Are you using a domain in your primary network or just in a lab or both? I am running mine in my primary network but would like to set up a testing lab beside it as well. So tinker then deploy with out pissing off the Mrs

2
NOTNlCER720XD | 192GB RAM | 11TB + 480GB•

My primary and my lab are on the same network, so it's all on one domain, but "Production" services are mirrored and everything is in its own VM. If I bork something critical like DHCP or DNS, the R210 II kicks in for failover. Nifty feature in Server 2012+.

1
dazedman00•

This is exactly what I wanted to get configured for myself too. I just wish I could get my hands on a R720xd for the right price so I could get that going already. :)

1
Zxvy•

Do you have photos of your lab?

2
NOTNlCER720XD | 192GB RAM | 11TB + 480GB•

I'll post a thread this week. Finally started the cleanup down there yesterday evening. Got a few new items racked and cables aren't as much of a disaster as they were a few weeks ago.

1
endre84•

RB1200

RB3011UiAS-RM

RS814 (2x1TB R0 | 2x3TB R1)

4xR210II E3-1240 V2 32GB (2x1/2/3TB R1, nginx)

1xR210II G2120 8GB (2x1TB R1, MX)

2xSmart-UPS 1500 RM

2xDumb Gb switches

UAP-AC-PRO

Orange Pi R1 for proxying some requests from ISP3 to ISP1

WS: Dell T5810 / E5-1650v3 / 16GB ECC / Plextor PX-512M8PeY 512GB / 2x3TB R1 / Quadro P2000 / w10

Questions about the HW are welcome (if you are looking for impressions or a quick review)

Future plans: hopefully I can get some work done and not have to buy / setup / reorganize / upgrade anything for a while.

7
CSTutorRHCALAB (7 R610, 3 R510)•

I'm running a 42U rack on a 20A circuit.

  • Physical pfSense server (R610)
  • 4 node Proxmox compute cluster (R610s)
  • 5 node Ceph storage cluster (2 R610s, 3 R510s)
  • 8 TB raw SSD storage
  • 80 TG raw HDD storage

Some software I'm running:

  • Plex
  • LAMP stack
  • Mail server (Postfix + Dovecot) providing SMTP and IMAP
  • Grafana
  • Various python applications including a Reddit bot
  • Windows 10 VM

I'd like to upgrade to:

  • Change the RAID cards in my other two R510s to H200's with cables to allow for more storage on those servers but budget is tight right now (I'd totally take donations! lol)
  • STILL need UPS systems (looking for two 2U 1500VA 900W systems AND a consumer level UPS of basically any kind to give power to a single external low power device)
  • I would like 10 gigabit networking across the whole rack but I would settle for just 10 gigabit between the Ceph cluster nodes (cluster network). Honestly, the Ceph cluster network is the only place I'd really need it. The rest would be expensive bragging rights. It's 5 servers so I would need 5 single port SFP+ PCI cards, 6 DACs, and a 6 port SFP+ switch at minimum for this upgrade.
  • I would like to get enough RAM to bring compute3/compute4 on par with compute1/compute2 in capacity.
  • I would like to get two X5670 processors to bring compute3/compute4 on par with compute1/compute2
7
Team503ESX, 56TB, 20 Cores, 384gb RAM - Plex!•

Not much change for me - car mods, work, and life in general hasn't left me the inclination or time to mess with things. I keep my dockers up to date for the most part, and otherwise have left things alone.

#TexPlex Media Network

  • 20 Cores, 384gb of RAM, 2TB usable SSD and 56TB usable Platter Storage
  • Serving more than 50 people in the TexPlex community

##Notes

  • Unless otherwise stated, all *nix applications are running in Docker-CE containers
  • DFWpSEED01 could probably get by with 4gb, but Ombi is a whore, so I overkilled. Plan to reduce to 8GB when I get around to it.
  • The jump box is obsolete and will be retired soon, but I refuse to do it remotely in case my RDS farm get squirrle-y.

##DFWpESX01 - Dell T710

  • ESX 6.5, VMUG License
  • Dual Xeon hexacore x5670s @2.93 GHz with 288GB ECC RAM
  • 4x1GB onboard NIC
  • 2x1GB PCI NIC

Storage

  • 1x32gb USB key on internal port, running ESX 6.5
  • 4x960GB SSDs in RAID 10 on H700i for Guest hosting
  • 8x4TB in RAID5 on Dell H700 for Media array (28TB usable, 2TB free currently)
  • nothing on h800 - Expansion for next array
  • 1x3TB 7200rpm on T710 onboard SATA controller; scratch disk for NZBget
  • nVidia Quadro NVS1000 with quad mini-DisplayPort out, unused

Production VMs

  • DFWpPLEX01 - Ubuntu LTS 16.04, 8CPU, 8GB, Primary Plex server, all content except adult, plus PlexPy
  • DFWpPLEX02 - Ubuntu LTS 16.04, 2CPU, 2GB, Secondary Plex server, adult content only, plus PlexPy
  • DFWpPROXY01 - Ubuntu LTS 16.04, 1CPU, 1GB, NGINX, Reverse proxy
  • DFWpDC01 - Windows Server 2012R2, 1CPU, 4GB, Primary forest root domain controller, DNS
  • DFWpDC01a - Windows Server 2016, 1CPU, 4GB, Primary tree domain controller, DNS, DHCP
  • DFWpDC05 - Windows Server 2016, 1CPU, 4GB, Primary tree domain controller, Volume Activation Server
  • DFWpGUAC01 - Ubuntu LTS 16.04, 1CPU, 4GB, Guacamole for remote access (NOT docker)
  • DFWpFS01 - Windows Server 2012R2, 2CPU, 4GB, File server that shares 28TB array, NTFS
  • DFWpJUMP01 - Windows 10 Pro N, 2CPU, 32GB, Jump box for Guacamole
  • DFWpSEED01 - Ubuntu LTS 16.04, 2CPU, 8GB, Seed box for primary Plex environment, OpenVPN not containerized, dockers of Radarr, Sonarr, Ombi, Headphones, NZBHydra, and Jackett
  • DFWpNZB01 - Ubuntu LTS 16.04, 1CPU, 1GB, Docker of NZBGet
  • DFWpRDS01 - Windows Server 2012R2, 4CPU, 32GB, Primary Windows RDS host server
  • DFWpRDSbroker01 - Windows Server 2012R2, 2CPU, 8GB, Windows RDS connection broker
  • DFWpRDSgw01 - Windows Server 2012R2, 1CPU, 4GB, Windows RDS gateway server
  • DFWpRDSlicense01 - Windows Server 2012R2, 1CPU, 4GB, Windows RDS license server
  • DFWpRDSweb01 - Windows Server 2012R2, 2CPU, 8GB, Windows RDS web server
  • DFWpMB01 - Ubuntu LTS 16.04, 1CPU, 2GB, MusicBrainz (IMDB for music, local mirror for lookups)
  • VMware vCenter Server Appliance - 4CPU, 16GB
  • DFWpBACKUP01 - Windows Server 2012R2, 2CPU, 4GB, Windows Veeam Host
  • DFWpSQL01 - Windows Server 2016, 4CPU, 4GB, Backend MS SQL server for internal utilities like Veeam

Powered Off

  • DFWpCA01 - Windows Server 2012R2, 2CPU, 4GB, Subordinate Certificate Authority for tree domain
  • DFWpRCA01 - Windows Server 2012R2, 2CPU, 4GB, Root Certificate Authority for forest root domain

Build in process

  • None

##DFWpESX02 - Dell T610

  • ESX 6.5 VMUG License
  • Dual Xeon quadcore E5220 @2.27GHz with 96GB RAM
  • 2x1GB onboard NIC, 4x1GB to come eventually, or whatever I scrounge

Storage

  • 1x2TB 7200rpm on T610 onboard SATA controller; scratch disk for Deluge
  • 1x DVD-ROM
  • PERC6i with nothing on it
  • 8x4TB in RAID5 on H700

Production VMs

  • DFWpDC02A - Windows Server 2016, 1CPU, 4GB, Secondary tree domain controller, DNS, DHCP
  • DFWpDC04 - Windows Server 2012R2, 1CPU, 4GB, Secondary tree domain controller, DNS
  • DFWpFS02 - Windows Server 2012R2, 2CPU, 4GB, File server that shares 28TB array, NTFS
  • DFWpRDS01 - Windows Server 2012R2, 4PU, 32GB, Secondary RDS host server
  • DFWpTOR01 - Ubuntu LTS 16.04, 1CPU, 1GB, Docker of Deluge
  • DFWpWSUS01 - Windows Server 2016, 1CPU, 4GB, WSUS Server
  • Dell OpenManage Enterprise - 2CPU, 8GB, *nix Appliance

Powered Off

  • None

Build in process

  • None

#####Task List

  • Configure EdgeRouterX 192.168.20.x
  • Re-IP network
  • Add AD Activation for SQL, Win10N, Win10
  • Install H700/i in T610, upgrade firmware, move data array, remove H700
  • Configure WSUS policies and apply by OU
  • Patch both hosts with OME
  • Watch NZB box for CPU/RAM usage

######Recently Completed

  • Upgrade OMBI to v3
  • Design new IP schema and assign addresses
  • Disable Wifi on router
  • Server 2016 migration and domain functional level upgrade
  • Stand up replacement 2016 DCs
  • Demote and decomm 2012 DCs
  • Configure WSUS on WSUS01
  • Finish standing up WSUS01, joining to domain
  • Finish installing SQL for Veeam including instance, db, permissions, and AD Activation key
  • Deployed Dell OpenManage Enterprise
  • Create static entries in DNS for all Nix boxes
  • Configure new NZBGet install with new 3TB disk
  • Reconfigure DFWpSEED01: Remove Deluge and Sonarr dockers and their data, remove old 2TB scratch disk
  • Stand up a 2016 DC and install Active Directory Activation for Office and Server 2016
  • Stand up PiHole VM, configure Windows DNS servers to point to it
  • Move all TV to FS01 and all movies to FS02, update paths in Sonarr and Radarr to match
  • Configure Dell OMSA on both boxes
  • Build DFWpTOR01 on DFWpESX01
  • Build DFWpNZB01 on DFWpESX02
  • Install new hotswap bays and 3TB scratch disk in each server to onboard SATA controller
  • Replace RAID batteries for three of three H700

######Pending External Change

  • Move DHCP to Windows servers - Configured, not activated
  • Upgrade firmware on H700

######In Process

  • Migrate to EdgeRouterX and WAP and offload GigaPower 802.1x traffic to AT&T residential gateway
  • Re-IP and VLAN network
  • Deploy WSUS
  • Configure Veeam backup solution

######Up Next

  • Build OpenVPN appliance and routing/subnetting as needed
  • Build deployable Ubuntu and Windows templates in VMware
  • Stand up MuxiMux and stand down Organizr (??)
  • Configure SSO for VMware and the domain
  • Publish OMSA client as RemoteApp in RDS
  • Configure Lets Encrypt certificate with RDS and auto-renew
  • Reduce RAM to 1GB on DFWpGUAC01
  • Build an IPAM server (using MS IPAM)
  • Fix internal CAs
  • Deploy WDS server with MDT2013 and configure base Win10 image for deployment
  • Slipstream in Dell and HP drivers for in-house hardware in Win10 image
  • Configure pfSense with Squid, Squidguard
  • Deploy OwnCloud
  • Deploy Mattermost
  • Deploy SCOM/SCCM
  • Configure alerting to SMS
  • Deploy Ubooquity - Web-based eBook and Comic reader
  • Deploy SubSonic (or alternative)
  • Deploy Cheverto
  • Deploy Minecraft server
  • Deploy Space Engineers server
  • Deploy GoldenEye server
  • Configure automated backups of vSphere - Veeam?
  • Deploy Wiki - MediaWiki?
  • Set up monitoring of UPS and electricity usage collection
  • Deploy VMware Update Manager
  • Deploy vRealize Ops and tune vCPU and RAM allocation
  • Deploy vRealize Log Insights
  • Configure Storage Policies in vSphere
  • Convert all domain service accounts to Managed Service Accounts
  • Deploy Chef/Puppet/Ansible/Foreman
  • Upgrade ESX to u1
  • Write PowerShell for Server deployment
  • NUT server on Pi - Turns USB monitored UPSes into network monitored UPSes so WUG/SCOM can alert on power
  • Upgrade forest root to 2016 DCs and Functional Level

#####Stuff I've Already Finished

  • Migrate Plex from Windows-based to *nix deployment
  • Move datastore hosting media from Plex Windows server to dedicated file server VM
  • Build RDS farm
  • Build new forest root and tree domains
  • Build MuxiMux servers - Dockered onto Seedboxes
  • Build new MusicBrainz server with Docker
  • Set up new proxy server with Let's Encrypt certs with auto-renewal
  • Stand up Organizr docker
  • Stand down Muximux
  • Troubleshoot why Radarr isn't adding all my movies

#####Things I toss around as a maybe

  • Deploy book server - eBooks and Comics, hosted readers?
  • Host files for download via NGINX/IIS/Apache?
  • PXE options for Linux servers?
  • Grafana/InfluxDB/Telegraf - Graphing and Metrics applications for my VMs and hosts
  • Ubiquity wifi with mesh APs to reach roof
  • FTP server - Allow downloads and uploads in shared space (probably not)
  • Snort server - IPS setup for *nix
  • McAfee ePO server with SIEM - ePolicy Orchestrator allows you to manage McAfee enterprise deployments. SIEM is a security information and event manager
  • Wordpress server - for blogging I guess
  • Investigate Infinit and the possiblity of linking the community's storage through a shared virtual backbone

#####Tech Projects - Not Server Side

  • SteamOS box because duh and running RetroARCH for retro console emulation through a pretty display
  • Set up Munki box when we get some replacement Apple gear in the house
7
hawaiizachwww.creedthoughts.gov.www/creedthoughts•

This is amazing. Can we get some lab pics?

3
Team503ESX, 56TB, 20 Cores, 384gb RAM - Plex!•

Honestly, it's just a T710 and a T610 next to each other in my closet. The only neat stuff is virtual. :)

But sure, I'll grab some tonight; I've been meaning to.

3
Irravian•

I always look forward to your WIYH posts.

3
Team503ESX, 56TB, 20 Cores, 384gb RAM - Plex!•

Thanks!

2
mylittlelan•

How do I follow you to get your WIYH updates!? Very nicely organized. My lab is about to explode with new things. I look forward to being able to post mine. With a name of Team503, are you by chance in Oregon?

1
Team503ESX, 56TB, 20 Cores, 384gb RAM - Plex!•

Honestly, I have no idea; I'm a casual Redditor at best, sorry! And thanks - I try to keep it easy to read.

And no, I'm a Texan. :)

1
chrisbloemker•

are you using swarm for your docker environments? or just managing them all with something like portainer? What's you're preferred way of manging all things docker related? :)

1
Team503ESX, 56TB, 20 Cores, 384gb RAM - Plex!•

Honestly, I manage them manually at the command line. I almost never need to mess with them except to upgrade them every now and then, so I don't much bother.

1
chrisbloemker•

Ahh gotcha. Do you have them configured to automatically update or anything special?

1
Team503ESX, 56TB, 20 Cores, 384gb RAM - Plex!•

Nope!

1
WgnZillaHomeLab Blog ⇨ TheNerdGarage.xyz/HomeLab•

This will be worth the possibility of down votes, but... More about the car mods pls :p

1
Team503ESX, 56TB, 20 Cores, 384gb RAM - Plex!•

E92 335. Full bolt ons and Cobb exhaust with custom tune and M3 suspension all the way around. Forged wheels ordered (ETA 6 weeks). Next power steps are the new Flex Fuel box for MHD that lets me flip between E85 and 93 without reflashing the ECU, and then tuning for it. Putting around 450whp down now, hoping E85 to push me to 500whp. If not, maybe methanol injection, depending on control options. Avoiding upgrading turbos for now, saving up for an FD3S toy.

GTR style hood, Vorsteiner carbon fiber trunklid, and OEM M3 side skirts waiting for paint. Need to buy front and rear bumper masks. Would like to do a LSD and M3 steering rack at some point, but they're pricey.

3
WgnZillaHomeLab Blog ⇨ TheNerdGarage.xyz/HomeLab•

Sounds like you've built a fun daily that's for sure! Now leave it alone and save for the FD, I need an FD :(

1
Team503ESX, 56TB, 20 Cores, 384gb RAM - Plex!•

Yep, that was the goal. Fast enough to stomp any casual offenders, not so fast that it becomes unreliable.

It'll be my fourth FD, though my first in more than fifteen years. Long-term goal is a 20B, though I'd love a 26B I just can't justify 50k for a motor.

2
gac64k56VMware VSAN in the Lab•

After I performed my last upgrade, I've been playing with Docker and will probably deploy VMware VIC soon. If this works, I may just start deploying Minecraft and ARK: Survival images instead of VMware templates and installing everything with Ansible.

I'm also hoping soon to start upgrading my VMware VSAN disks from 320 GB SATA drives to 1 TB SAS, along with upgrading my SSDs from Samsung EVO 850 to Intel PCI-e NVMe drives.

6
harbinger_117•

+1 for Docker, use it extensively at my job, and with the larger adaption of kubernetes across all vendors it will be a huge plus.

3
MasterScrat•

What do you run it on? Have you tried VMware VIC?

1
harbinger_117•

I run almost everything other than my router (Guacamole, Plex, sonarr, radarr, VPN, smoke ping, webservices, etc). Since docker really is in much more demand and I use it heavily at work with Kubernetes, it's very easy to see the benefits of contanerized workloads.

2
x7C3•

So uh, after I picked up a GSA r710 for 500 NZD, I found another r710 for 300 NZD and I had to buy it. Will be using the second one as a NAS, will sell my current NAS as it isn't exactly fit for purpose (non ECC RAM), so it's not too bad.

4
sniperczar•

Progress moves forward on the hyperconverged lab. Since last month I have finally installed everything on rails (thanks in part to the 20% off eBay sale) and put the door back on the rack, which has dropped the noise level an impressive amount. With my wife's assistance, we finally got the 8U APC into the rack, which combined with the limited space in the closet was about half an evening's work by itself.

Upcoming expenses are a 240V feed to the closet and ventilation, the lack of which means the closet door remains off its hinges for the time being.

Future improvement projects are just finishing touches for the rack - an AC Infinity controller, 48 port patch panel, cable comb, and an AC WAP to replace my current AP.

I am finally starting to install some VMs on my Proxmox/Ceph cluster again after selling my SSDs and trading up for better write speed/capacity on the journal disks.

4
Bobbler23•

Currently building up having been bitten fully by the bug. I have worked in IT for 23 years, always had a lot of kit but this is another level even for me...used to be lots of desktops everywhere but consolidating to server based/Linux based setup now that I have a lot more knowledge on it through.

Previous setup: Custom built server running UnRAID, Dockers for media management - low power i5 T series, 16GB ram, 2 x 120 SSD cache, 1 x 6TB parity, 2 x 5TB storage. Moved over from Windows 7 based install to UnRAID which is when the server bug caught ;) Mixed network of CAT6 wired, Powerline ethernet and BT whole home wifi Router was Tomato shibby based Netgear R8000. No-name cheapy gigabit switches daisy chained together to provide enough points in my home office.

The few weeks added: APC UPS (fairly small one, I hadn't planned on the below at the time) Custom built PFSense box - £100 - fanless heatsink case, running a laptop low power i5, 4GB RAM and 60GB SSD. Wanted whole home VPN without the 30 meg limit of the R8000 as it is CPU constrained. Cisco 8 port Managed switch - now resigned to being a fall back as its didn't have enough ports and was non-rackmount. Netgear GSM 24 Port Gigabit managed switch - £25 - plus modded the fans for Noctua's as it was mental loud as has no temp control. Built a Lack Rack (Mobile Enterprise deluxe edition) - about £20 altogether with everything 2 x PDUs - one IEC, one UK standard Dusted off my Raspberry Pi 3 and is setup running Observium for SNMP events currently.

Pretty much silent and whole rack is consuming just about 100W at load.

On the way this week: HP DL380 G6 - 96GB RAM, 2x E5540 (ordered a pair of X5650's to upgrade it to hex cores - £40) - only £180 which I thought was good really for that much memory...no disks though so have also got 2 x 10K 146GB - £15 2 x 15k 146GB - £20 And that still didn't seem like enough storage so also bought a 2nd DL380 G6 which was sporting 8x 300GB 10K SAS drives but carrying a lower amount of RAM and single CPU. May move in the other CPUs or just keep for spares, it was significantly cheaper than buying the disks on their own even 2nd hand - cost me £109.

Plans: Playing really, I am going through Azure admin training in the next few weeks (first in my company) so am going to try getting the Azure stack going on it to get some familiarity with it all. Work won't let me play too much as I am only going to be looking after my departments cloud machines. Will then probably level it and go for a bare metal hyper visor install on the big machine, possibly setup a Linux cluster to mimic my work servers so that I can do some break fix type stuff - I look after an analytics cluster at work (SAS based on Redhat) and it's not often I get to try new stuff without worry as we have no full copy in the test environment, just the software as a standalone rather than the full cluster (due to licensing costs) Ansible, Azur and getting to grips with containers (Docker etc) along the way no doubt. Going to leave the original server and pfSense alone as I don't really want two rack mount servers constantly on the spin 24/7.

4
harbinger_117•

Jeez, I need to find where you get these deals haha. I always hit the tail end or miss the good deals completely.

2
Bobbler23•

EBay was all it took. They had another 7 un its of the 96GB based machines up until yesterday when the auction expired. Will post a link when they come back on probably once we get into next working day for them tomorrow. The second hard disk laden one turned up but has been bent on the track mount ears. They have me a £20 refund and I bent it back myself so not bad result

1
Bobbler23•

https://www.ebay.co.uk/itm/HP-Proliant-DL380-G6-2-x-Quad-Core-96gB-2U-Server-2596/192491113985?hash=item2cd15d4e01:g:Rw4AAOSwuxFYy9WZ

No disks, but 96GB of RAM. I bought mine from these guys the other week - decently packaged too.

1
Bobbler23•

And it continues to grow before I have even done anything meaningful. Got the Azure stack preview installed, took hours but it's up ready for play when I get back from holiday. Got a second 750 APC UPS which has network capability via an add-on card. I saw a peak load of over a 1000W during the Azure install which my current UPS didn't agree with so going to split the load on the PSUs to one clean and one unprotected, keeping the two DL380s on their own and the home server/Pfsense box left with plenty of run time on the original one. RAM upgrade for the second DL to 32GB with enough spare slots to go to 64 should I need to. Have to rejig my office around to accommodate a second rack for the servers I think is in order next. Also spoke to a local FTTP provider so going to be working on getting 250/250 fibre in with option of 500/500 if I can get some sign ups in the neighbourhood too.

1
EnigmaticNimrod•

Figured I might as well jump into this - long time lurker, first time actual-poster, etc etc.

Stuff I Have

I am currently running 4 whiteboxes in my homelab, which serves as the entirety of my home networking infrastructure (testing in production, if you will ;) ) All of my hardware is consumer-grade - I get enough practice with server-grade stuff at work, and while IPMI is nice it's not a necessary-to-have for me at this point. Power and noise are also a factor - I live in a 2 bedroom apartment with my partner, my lab is in the living room, and we split the electric bill.

Hosts in question (all hypervisors are Centos 7 running KVM/libvirt for virtualization):

  • hyp01 (i5-4670, 32GB DDR3 RAM, 1x128GB SSD, 2x2TB HDD)
  • hyp02 (i5-4670, 32GB DDR3 RAM, 1x256GB SSD, 2x1TB HDD)
  • hyp04 (FX-8320E, 32GB DDR3 RAM, 1x240GB SSD, 2x1TB HDD)
  • hyp05 (FX-8320E, 32GB DDR3 RAM, 1x240GB SSD, 2x1TB HDD)
  • 5x Raspberry Pi Model B (sitting around doing nothing currently)

(hyp03 was originally in here, but was converted into a gaming desktop for my partner)

Services being run:

hyp01

  • git (ubuntu 16) - gitlab, a few configuration files for other services live here
  • dns1 (ubuntu 16)- primary DNS host, config commited to git repo
  • docker02 (ubuntu 16) - secondary host for use with Rancher - currently unconfigured
  • fipa (centos 7) - FreeIPA - centralized login for entire homelab
  • ppt (ubuntu 16) - Puppet/Foreman server for deployment and configuration management - this is what I use to configure every other machine in my homelab

hyp02

  • fw01 - pfSense firewall VM
  • unifi (ubuntu 16) - unifi controller VM for my UAP-AC-PRO which serves WiFi to my apartment
  • docker01 (ubuntu16) primary host for use with Rancher - currently barebones-configured

hyp04

  • dns2 (ubuntu 16) - secondary dns node - config is commited to git repo
  • minecraft (ubuntu 16) - minecraft server for myself and a couple of friends.

hyp05

  • server1 (centos 7) - testing for RHCSA certs
  • tester1 (centos 7) - testing for RHCSA certs
  • outsider1 (centos 7) - testing for RHCSA certs

Other hardware:

  • Ubiquiti EdgeSwitch Lite 48 port
  • Ubiquiti UniFi UAP-AC-PRO

What I'm doing with it

I've got a pretty barebones homelab set up currently. Everything is exactly what it says on the tin - currently dedicating an entire hypervisor just for RHCSA/RHCE studying, but soon that'll get added back into the available "pool" of compute resources.

Stuff I want

  • Rackmount cases for all of these boxes - they're all still in their tower-style cases, and since I have a 13U rack I'd like to stuff the boxes into the rack so I don't have a stack-o-towers just sitting around.
  • NAS - I want to custom-build a NAS running FreeBSD. Why FreeBSD and not full-blown FreeNAS? Basically I just want the box to serve the storage, and that's it - and this means I can choose lower-power and quieter hardware in order to do it. ZFS runs great on regular FreeBSD, and if I add in NFS as well as packages to expose some block devices over iSCSI then I'll be happy. I have no problem running dedicated VMs/containers just for media download/consumption, since I've already got the available capacity to do so.

Stuff I want to do

  • Sensu monitoring - SensuV2 was recently released into beta status, so I'm excited to test that out. I was a big enough fan of SensuV1, but I never really took advantage of everything it had to offer.
  • Logging/Visualization - I want to get some sort of ELK stack set up so I can visualize and track everything that's going on in my lab. This kinda goes hand-in-hand with sensu above.
  • Highly available firewalls - I'd really like to get a HA pfSense solution set up so I can take one of my hypervisors completely down for updates/upgrades without bringing down the entire network. The issue is that my ISP won't give me a static IP address unless I buck up for a super-expensive "business plan" which my partner and I don't really want to pay for. I could get something like an EdgeRouter in front of the firewalls that I can use to forward all ports to the highly available CARP IP, but that might be overkill. I have a consumer-grade WiFi router that I could use, but I've tested the LAN speeds and they're garbage - I don't want to put my entire network behind that. Still some thinking to do on this one.
  • Shared VM storage - I'd like to use the NAS that I want to build as storage for my various VMs, so that I can live-migrate them around my various hypervisors. Possibly even a second NAS just for this, using the HDDs that are already in the hypervisors? Maybe I combine the two NASes via 10GBE to create an actual SAN? Who knows.
  • Figure out a use for the 5x Raspberry Pis - what kinds of interesting clustered stuff could I do with these?
  • More configuration management - I've got a Puppetserver sitting around that isn't doing too much. I'd like to configure some more services and get the manifests committed into my git repo to create a scenario in which I can restore a backup of my puppet server and gitlab host and be back online with just a few clicks. Not even close to there yet.
  • Playing with containers - I've only just recently discovered the awesomeness that is containterization (both with Docker and LXC/LXD), and there are a number of services that I'm running that don't require the full OS stack (dns, unifi, and gitlab are the three that primarily come to mind, but also potentially others). I want to play around with this more.

Always a work in progress. :)

4
Brandon4466Core 2 Duo :D•

What do you use for the Minecraft server? Just running it right from the command line or are you using something like Mine as to manage it/them?

1
EnigmaticNimrod•

I believe I'm just running Craftbukkit right on the command line from within a tmux session. Nothing fancy.

I've not heard of Mine, I'll have to look into it.

3
Brandon4466Core 2 Duo :D•

Typo on my part, its MineOS. Runs on almost any Linux distro, also available as a prebuilt turnkey image. Really nice Minecraft server management software. Controllable via a web interface, support Minecraft, Craftbukkit, Spigot, plus any custom anything, mods, plugins, you name it. Really a nice program, I'd check it out.

2
vesikk•

Current:

  • pfSense (physical machine)

  • Ubiquiti Unifi AP (Gen 1)

  • Ubiquiti US-24 (non POE)

  • Proxmox 5.1 whitebox running a Xeon E3-1220 v3, 8GB RAM, 120GB Kingston SSD

  • Synology DS216j (NFS for proxmox, External HDD for Plex, etc.)

  • Windows Server 2012R2 (AD,DNS,DHCP)

  • Windows Server 2012R2 (Plex & sometimes game servers)

  • Ubuntu 16.04 Server (Unifi Controller)

  • Ubuntu 16.04 Server (Grafana)

Planned

  • Web Server (Nginx)

  • Ubiquiti UAP-AC-PRO

  • Pi-Hole

  • Move pfSense to a virtual machine

  • change the current pfSense box into another Proxmox node

  • Setup Open vSwitch on Proxmox

I run all of this so that I can expand my knowledge and play around with cool things or ideas that I see others post here.

3
bytwokaapi2x(12C r710 w/144gb) | 60TB | Freenas | ESXI•

My 2c, leave pfSense in a dedicated box. I tried moving it into a VM a couple of days ago and I locked my self out soo many times, VLAN configuration was a pain...eventually gave up.

2
Hovertac•

Give the "LAN" nic VLAN ID 4095... I use virtual pfsense with dual WAN.

I use 4-port LACP to my ESXi box, WAN1 is VLAN 500, WAN2 is VLAN 501, and LAN is VLAN 4095. This allows me to have sub interfaces for VLAN1, VLAN10, VLAN20, VLAN30, VLAN40, and VLAN100.

The LACP trunk on my ProCurve is tagged on VLAN 1, 10, 20, 30, 40, 100, 500, and 501.

1
acre_18u of end table power•
  • Mikrotik RB2011 series for the Internetting. One line in from the modem, one out to my dumb 16 port gigabit switch. I've on occasion used the other ports for routing experiments, but otherwise everything just goes to the switch to make things neat.

  • 16 port TP-LINK "smart" switch. Basically means a half assed web GUI, VLAN support I think, I don't use it. Just one /24 for everything because I'm lazy.

  • ZOTAC ZBOX ID92 for a desktop hypervisor. Used to be a dedicated hypervisor, but I've since consolidated lab VMs to my other server. Core i5-4570T with 16GB RAM. Shitty old laptop hard disk running Ubuntu, but will be playing with Antergos.

  • ATOM C2750 for a ZFS storage server. Was running FreeNas for the last 2-3 years but runs PVE as of a week ago. Containers run fine on this processor, Plex can transcode if nothing else is running hard. I don't have to have both computers on to run containers now so I like this better.

  • Mikrotik hAP lite for home wifi.

3
inkarnata•

Running ESXi 6.5 u1 on an R710

Dual Xeon E5649, 120GB DDR3 A Mish mosh of 2x 1TB and 6x 2TB drives

  • 1x Server 2016 VM for general use
  • 1x Server 2012 R2 w/ Azure AD Connect for 365 testing stuff for work
  • 1x Mint Linux VM running Sonarr, Radarr and Deluge (currently offline while I figure out a USG->PIA VPN route for permanent VPN traffic)
  • 1x Zabbix on Ubuntu Server (not yet configured)
  • 1x Grafana on Ubuntu Server (not yet configured)

R610 Running Proxmox, Dual Xeon E5649, 96GB DDR3 4x 146gig Drives and 2 500 gig drives...I think.

  • 1x Mint Linux box for testing

(Powered off) R210ii 1x E3-1220v2 8gb RAM

  • Sophos Firewall (Decomissioned)

Running FreeNAS on a 4U Supermicro box: Intel Xeon E3-1280 V2 32 GB RAM 17.8 TB total storage (1x 2tb and 7x 3tb drives...I think)

  • Plex Plugin
  • Nextcloud
  • iSCSI target

USG Pro 4

HP 2920-48G-POE Switch w/ some bad PoE Ports

Avocent KVM console w/ possibly a bad power supply All housed in a 42u Dell Rack

Dell PowerConnect 2816 and a Ubiquiti AC Pro live inside

3
inkarnata•

March brought an unexpected surprise, another decomissioned R710, added to the lab, or at least the floor in front of the lab until i get some rails. Have not powered on yet, but currently has a measly 8 gigs of RAM 4x2 gig sticks, I got.some spares, 6x 2tb Drives, added idrac Enterprise to it....

1
inkarnata•

Upgraded the box, R710 2x E5520, 96 gigs of RAM, H700, 8x2tb Drives. Just need to get a battery and cables for the H700 on payday.

This may take over the FreeNAS duties from the Supermicro box for an all Dell (server at least) lab

1
b1nar3•

I’m really into infosec and have been for the past ten years. I always ran virtualbox for my virtual needs but I finally decided to get into homelabbing and within the last two weeks I bought a large 6’ rack (overkill), r710, 2950 gen3, Cisco 3750 layer 3.

I just decided to wear a white hat for a change and actually learn how to install, configure, and maintain servers and switches. It’s kind of crazy how I know how to break in but I have no idea how they are configured, setup, or maintained. Just a few weeks ago I found out what idrac6i is, configuring RAID, server ram config, and so much more. Now I’m completely addicted.

PowerEdge r710 2 x Processors Intel 2.53 Ghz E5649 6 cores
8MB 5.86 GT/s 64 bit processors 2 x 300 GB SAS 2.5” SFF Drives 1 x 570 watt power supply 4 x Onboard Quad Gigabit 1000 Pro Ethernet Embedded Matrox g200 8MB 2U Rack-mount (working on buying rails) PERC 6i RAID Controller w/ battery backup iDRAC6 Express Edition Backplane for 8x2.5” SAS Or SATA 2xRicer Boards 2xPCI-e g2 x8 slots per board Running VMware ESXi 6.5 R710 came with 16 gb of ram 8x2GB but I bought 48 GB 12x4GB sticks only reason I didn’t buy 8GB or 16GB sticks Is because I ended up winning the 48GB r710 ram for $58 including shipping Still a little confused on ram configuration but If I don’t use any 2GB sticks my system will have 48 GB of memory. 8x4GB 2Rx4 PC3-10600R 8x4GB 2Rx4 PC3-8500R 48 GB total running at 1066 mhz (pc3-8500) Server Cost:$160 Ram Cost: $60

Switch Specifications ———————————————- Cisco WS-c3750-24PS-E IP Services License IOS Upgraded to: c3750-ipservicesk9-mz.122-55.SE10-tar Running IOS 12.2(55) Looking to upgrade to IOS 15 (if possible?) Just received the switch today the reason I went with Cisco is because it has the highest market share and complex cli which I never used. Being into infosec knowing Cisco cli will be very beneficial in the long run.

Cost: $60

PowerEdge 2950 gen3 32 GB Ram 2x 72GB 15k IDrac enterprise Cost $50

6’ Rack with fan Not sure how many U but I’m 5’10 and the rack is taller than me. Cost: $100 + 20 mile drive.

Future plans? By the time I’m done I think my room will look like google mini data center. My next buy is definitely going to be a dedicated firewall. I’m leaning toward Cisco 55xx model perhaps 5510. I don’t want to virtualize my firewall so if anyone has a suggestion about a Cisco firewall with vpn that’s within $200 range.

All obtained equipment Was obtained within last 2 weeks. Been researching past 1 month. So my knowledge as far as enterprise Servers and equipment is limited. I’m completely addicted to home labbing with withdrawals and everything and I can’t Stop buying equipment. I need serious rehab help.

Thanks for reading.

  • B1nar3
3
njgreenwood•

You think that 6 foot rack is overkill now... Spend a year on this sub and you'll be looking for another one.

3
Rikairchy2•

I'm currently working with:

2x R410 (2xE5620s, 32GB of RAM, 4x2TB HDD)

1x R610 (2xE5620s, 72GB of RAM, 2x1TB HDD)

5x Raspberry Pi 3B

Unifi 24 port switch

Unifi USG

UAP-AC-Lite

The hosts run ESXi 6.5, with vCenter on one of the hosts.

I'm about to remove all my VMs and start over, since I like to rebuild my AD network every now and then.

Three of the Raspberry Pis are not used at the moment, but one runs nginx with as a reverse proxy with guacamole, and the other just runs the Unifi management software.

I'm picking up a rack (finally!) and an R510 tomorrow. I need to grab some more RAM for the R510, as well as racks for all the servers.

2
starkruzrProxmox on whitebox Shuttle XPC w/ USB3 RAID & HP Z240•

CURRENT

Network: Netgear N300 old-ass router in front of a FiOS gigabit connection and a couple gigabit switches, plus 1300Mbps powerline networking to the extra AP in the back of the house (am already regretting this decision as that connection seems to suck mightily)

Virtualization and storage: Whitebox Shuttle XPC i7 w/ 32GB RAM and a 4TB hardware RAID array hanging off of USB3 for storage (turns out this actually does run around 5Gbps as promised, pretty great IOPS considering), plus an HP Z420 workstation, Xeon something with another 32GB RAM and a 500GB SSD

Just Storage: old Dell i3 desktop with 12GB RAM and a bunch of laptop hard drives shoved into it running FreeNAS, exporting NFS to the two Proxmox nodes (I'm still amazed I got this working, it took what felt like an act of God)

FUTURE

Network: A Qotom box I'm trying to turn into a pfSense router for that gigabit connection

Other: a Pi I'd like to use as the front-end for a Grafana dashboard

2
auto-xkcd37•

old ass-router


^(Bleep-bloop, I'm a bot. This comment was inspired by )^xkcd#37

5
bytwokaapi2x(12C r710 w/144gb) | 60TB | Freenas | ESXI•

Qotom q355g4 nics are labled properly. I believe it is igb0 igb2 igb3 igb1...caused some frustration during setup. Otherwise, its a great pfsense box.

2
mrouija213•

Updates, yay! New server arrived and updated BIOS to support v2 CPUs which I have yet to order. Also got new system drive for the desktop, some sweet sweet NVMe goodness in a sale. Now I need to get some drives and a new switch to get moving on the new server. Looking at cheap, silent, gigabit switches and currently that looks like a used Dell Powerconnect 2816.

New Server

Supermicro CSE-825TQ Black 2U Rackmount Server Chassis
Supermicro X9DRi-F Motherboard
Xeon E5-2620 Processor, (single CPU right now)
16 GB RAM (4x 4GB) w/ another 16GB (4x 4GB) for when I get the second CPU
120GB SSD for System
No storage drives yet.
Proxmox

Running:
Nothing yet! :(

Desktop

i7 3770K CPU
Asrock Z77 Extreme 6
12GB RAM (2x 4GB, 2x 2GB I had laying around)
512GB MyDigitalSSD SBX m.2 NVMe card in Mailiya PCI-E adapter 

Running:
Android Studio
Unity
Games (Factorio, Elite: Dangerous, Path of Exile, Warframe, Rimworld, Dwarf Fortress mostly)
Plus media/etc, basically whatever tickles my fancy

Old Server

Unchanged.

HTPC

Unchanged.
2
TheBloodEagleXResident Noob•

Supermicro X9DRi-F

I love dualies! I looked at this board soooo many times while cruising the SM site.

I'm still rocking a 3770K too, hah, also on an Asrock board but a worse one, the ATX Pro4. I'm at 32GB of RAM though (with activity LEDs, lol). No NVMe drive yet, but several SATA SSDs.

1
Stylomax•

Systems:

Running ESXi 6.5 on a C1100 Dual Xeon X5570s, 16GB DDR3 (yeah I need more, also need the cash to buy it) with a mix of 1 & 2 TB drives

  • Windows 10 Pro - testing for any potential patching issues
  • Ubuntu 17.04 - Docker for Bitlocker vault, subsonic, and other docker images
  • Ubuntu 17.04 - Dedicated steam caching VM (decided to go this route as opposed to the docker image)
  • Ubuntu 17.04 - Dedicated backup server setup for sftp and clients are using duplicati for backup

Whitebox system

Running Windows Server 2008 R2 i5 2500K 16GB RAM 120 GB SSD (OS), mix of 1 & 2 TB HDDs

  • Minecraft FTB servers
  • Teamspeak 3 server
  • Terraria Servers
  • Plex server
  • Hyper V VM running Ubuntu 17.04 running a nginx reverse proxy with RDP enabled (used for producing twitch streams with multiple game camera views)

Lenovo M90 i5 650 4GB RAM 30 GB SSD Absolute overkill for a pfsense box, probably should virtualize it but I want more RAM in the C1100 first, just haven't had the cash.

HP Procurve 3500yl-48G - Network core switch (and who can say no to a switch like this for the price of free?)

D-Link DIR-615 - Flashed with DD-WRT and just running as an access point

Planned software & Hardware deployments:

Hardware - Get a decent Unifi AP but that actually comes after saving up for some DDR3-ECC RAM if I can figure out a way to afford it

Software - None planned but if I see something interesting & useful I may put it on a VM

2
ReasonablePriority•

Currently:
1x HP Microserver Gen8 (g1610t, 16GB, 2x 2TB HDD, 2x 1TB HDD) - ESXi 6.0x

  • Plex (Ubuntu 16.04)
  • Plex Test (Ubuntu 16.04) (1)
  • Internal Docker Registry (Ubuntu 16.04)
  • General Linux VM (Oracle Linux 7.4) (2)

1x HP Microserver Gen8 (g1610t, 16GB, 128GB SSD, 3x 2TB HDD) - Ubuntu 16.04 + Docker CE.

  • 2x Transmission
  • Portainer
  • Handbrake
  • MakeMKV
  • OpenVPN
    System also runs internal DNS (BIND)

1x HP Microserver Gen8 (g1610t, 16GB, 1x 128GB SSD, 4x 2TB HDD, 1x 4TB USB) - Ubuntu 16.04 - Backup Server 1
1x HP Microserver Gen8 (g1610t, 8GB, 1x 128GB SSD, 2x 2TB HDD, 2x 1TB HDD, 1x 4TB USB) - Fedora 26 - Backup Server 2

1x 1810-24g 24-Port gigabit switch

2x QNAP TS431x (4x 4TB HDD)
1x QNAP TS431 (4x 2TB HDD)
1x QNAP TS259 (2x 4TB HDD)

(1) Two Plex servers as I subscribe to the beta Plex-Pass channel so want to test releases before applying to my main Plex server.
(2) System can also VPN into my lab at work to give access to more kit there and has Ansible playbooks to spin up and down AWS EC2 instances if I want them.

In the future i would like to upgrade the CPU in the ESX system and install Docker on the backup servers so I can have a play for Swarm or Kubernetes. Moving to a house which is big enough to take some proper rack servers would be good too!

2
t34p075•

Running a Raspberry Pi with OpenHAB for home automation and another as a cloud server with Nextcloud. Another is a media server that houses my Kodi library via MySQL. The Kodi library one also houses a collection of games that can be played via other RPis hooked up to TVs throughout the house via RetroPie. I’ve also got one hooked up as a print server for my 3D printer.

...I’m kinda addicted to Raspberry Pi’s...

2
roothorick•

Just starting to assemble the pieces to transitioning my home's backbone to a proper mini-lab. The gear:

  • Recently acquired open 4-post rack, 42U
  • Heart and brain of the beast, an R815: 2x Opteron 6212 + 2 unpopulated sockets, 64GB RAM, will be outfitted with 6x1TB in RAID5
  • Pursuing a Cisco 3548XL on Craigslist
  • 1500VA APC rackmount UPS, I forget the model. It has network connectivity
  • I got a KVM console for free, primarily because it's in rough shape (but still, probably, functional)
  • Will be purchasing a USB enclosure of some description for backups. Have a 4TB drive to drop in it, but with the array in the server being 5TB I don't think that'll suffice. TBD
  • Will be stealing an Aeotec SS6 from my living room (the range on these things are just crap, I don't know why)
  • One preexisting UAP-AC-Lite, will be adding two more
  • Preexisting Aeotec Z-Stick Gen5 and Nortek HUSBZB-1 on an ancient (I don't think it's even USB2) unpowered hub. I'm kind of in awe that this works at all, now that I think about it.
  • Existing ISP-provided SB6121
  • Will be getting shelves to put in the rack

The plan:

  • Ubuntu Server 18.04 will be going on the server "bare metal", and it will be running pretty much every "backend" service I care about; highlights are iptables + tc firewall with classful traffic shaping, Samba, Home Assistant (with some supporting services like mosquitto and prosody), and rtorrent. Keeping open the option of adding some kind of Windows VM later should I have a desire to run something that doesn't support Linux (Civ 5/6 pitboss anyone?)
  • Will try to bond three of the NICs on the server together, connected to the Cisco switch. The existing UAP will be connected to the switch as well as two cables going out to other rooms. Lotsa empty ports, I know :)
  • The additional UAPs will be set up in a mesh configuration in an attempt extend range to and around my detached garage. If I pull this off, it will be a huge boon for an upcoming project (basically, making ESP8266 based keyfobs).
  • The new USB enclosure will be connected to the server and power plugged in via the SS6. It will be on a schedule to turn on a few minutes before a backup needs to be run and shut off again after. (The "off" won't be timer-based: I'll have the backup solution post to Hass "hey backup's done" somehow. Probably an MQTT blast.) Since it'll be practically on top of the Z-Stick here, the limited range shouldn't be a problem.
  • Everything's plugged directly into the UPS; not enough gear (yet) for a PDU
  • Modem will be dropped on a shelf in the rack and connected to the server's remaining NIC
  • Console connected to the server, obviously
  • The radio sticks will be plugged directly into the server on separate USB extension cables; no more hub. I'll try to tape them to the walls somewhere near the ceiling.

The new hardware will be piecemeal assembled and racked as it comes in. Server OS and software will be installed tail end of April (18.04 release). Configuration, data, and existing hardware will be transitioned in one fell swoop over the course of a single day in early May. Exact dates TBD pending work schedule.

This all leaves an aging whitebox server (Phenom II X4 965 BE, 4GB unregistered ECC, has some 3.5" 1TB drives in it) and a Smart-UPS C1500 without a purpose. I'll probably put my gaming and work gear on the UPS, just to utilize it. Thinking about parting out the server (it has a ridiculously nice PSU I'm thinking about swapping into my primary rig), but even a lot of the parts I don't have any compelling use for...

Currently, KVM console is mounted, the UPS is half-assedly mounted (I did not have the right hardware), the R815 is awaiting rails and just sitting loose in the bottom of the rack for now. Only thing that's hooked up is the UPS is plugged in. Wanted to give it a smoketest + charge and just left it because why not.

2