#Acceptable top level responses to this post:
#Previous WIYH:
#View all previous megaposts here!
MM~~
Posted Nov 16, 2017, 8:05 AM UTC
Dell T310 - NFS for my esxi
Dell R710 - esxi host
Puppet Enterprise x2 (HA)
AD/DNS/DHCP
Plex
ELK Stack
Nessus - patch and vulnerability scans run weekly
Gitlab
Squid Proxy
PiHole
OpenVPN Acess Server
Coming down the pipeline:
Fully puppetizing my environment - it currently just secures my servers, adds them to the domain, and installs filebeat
fine tuning ELK - I just set this up and still have a lot to learn
It is advised against using shared storage for elasticsearch according to elastic co. If you are only using gigabit switch it does not take much for you to have a performance problems.
Yeah, unfortunately my storage is very limited right now, so ELK is running off of the only local drive the R710 has; all of the other VMs live on the NFS share.
It's not ideal, but it works until I can devote more funds to redoing my whole storage setup.
I've just upgraded from my old Raspberry Pi + 4tb WD NAS drive.
Found myself a very reasonably priced Dell R710, dual X5650s, 24gb RAM. Picked up two cheap drives to get me started which I've setup in RAID 0 for now - my plan is to add more storage in a better RAID setup and move VMs across eventually as funds allow.
I've setup ESXi and have the following VMs:
Tonberry: pfSense - Bought myself a replacement modem and setup pfSense in a VM to replace my old, bad, PlusNet all-in-one thing. I had a pretty decent ASUS wireless router which I put in AP mode and have setup. Overall my network stability has increased ridiculously. Zero wifi dropouts, zero complaints from the wife, zero downtime reports from external checks. Very happy with this work.
Chocobo: Reverse nginx proxy
Cid: mariaDB server, which provides databases for...
Balthier: Web server running nginx + latest release of PHP via php-fpm. Currently serves a blank page. Aww yeah.
Moogle: FFXI server
Bahamut: Vanilla WoW server
Wedge: Minecraft server
(These 3 are rarely on at the same time, currently it's running the WoW server, but my friends will get bored in a week or two and we'll spin up a new Minecraft modpack or something to try out)
Fran: Plex - this has my 4tb NAS mounted which has all my media on it.
Cactuar: Home Assistant, currently offline as I was only really using it for the virtual bridge for Alexa and Tradfri lights which is now supported natively. Maybe at some point I will revisit and do something smart with this, but then again my projects list is getting really long.
Latest mission completed:
I installed Ubuntu on my desktop. I mainly use my MBP for everything and rarely turn the thing on, but fancied giving it a shot so now that's done.
Future expansion plans:
I named my virtual servers after final fantasy things too!!!
Probably not a relevant reply. I am getting into the process of wanting a homelab to learn Linux st a more intimate level. What would you recommend for a beginner? I was looking at building my own but was exceeding 1k. Also looking at the Lenovo ThinkServer TS140, HP ProLiant gen 8, the Dell PowerEdge T30. Also is eBay a relevant now? If so which direction should I be looking?
I'll be using this for file-sharing, plex, music, and housing VMs from my main system while running CENTOS as the main os.
Well homelab, its been 3 months. Killing it at that job I was starting back in July. Still a surprising amount of NDA-coverable stuff over online beauty orders.
WHAT HAS CHANGED
Hardware
Still running a single IBM server x3650 M1 (2u 2s, x5450s?) on lga 771, planning on switching config 100% in march, when taxes come in
Personal workstation/gaming rig got a new motherboard, went with the z270x-gaming 5 from Aorus/Gigabyte as the h170x1/3.1 i have has bent pins apparently from removing/replacing the cpu after waterblock application.
Still wanting to move to a Router-on-a-Stick Configuration using the cisco small business grade ISR (cisco 891W, which i want to put to use so badly, but is likely going to be setup in the living room behind the entertainment center to act as a seperate OSPF/EIGRP zone for the 29xx/19xx router with a lp 8 port switch to provide ethernet to my xbone (black friday deal?), raspi for retropi, and potential HTPC?) I have and the catalyst 2950. Just bought a 3750 non g/x PoE for powering the IP Phones and APs I am planning on deploying when i can figure out how to use the vController for Aironet LWAP/APs.
have an HP ProCurve Gigabit switch waiting to be deployed to replace the aging 2950 as core switch.
replaced my aging Macbook7,1 with the ever great thinkpad t430, planning on replacing the cpu, getting 16gb of ram and a couple ssds to make this thing screamin fast
Software
FUTURE UPGRADES/CHANGES
Still planning on upgrading to a couple 1u 2s 1366/2011 nodes and eventually putting water blocks on it, maybe (dependant on how feasible it is for me to do it, maybe r510s or r520s, maybe 1u whiteboxes?)
r3 1200/r5 1400 based HTPC for steam/youtube/etc?
Bridge Mode soon^tm
Def picking up a datacenter grade ISR G2 (2921/1941/2911) and figuring out a fan mod/watercooling to keep it cool and quiet
Netshelter or similar to replace my ghetto fab LackRack^tm
All in all thats kind of what i have planned. If anyone has questions/suggestions/feedback feel free to hit me up in PMs here or tag me on Discord (Can Flag + my name here ;) )
ESXI-01: HP ProLiant DL360G7 (8x 73GB SAS, 2x E5645 6-Core, 64GB RAM)
Mac OSX Client - Self explanatory
Windows 10 Client - Self explanatory
Ubuntu Client - Self explanatory
Torrents - Sonarr/Radarr/Deluge
Domain Controller - Self explanatory
DVR - Test VM for security cameras, not sure if I'm going to use VM or a phyiscal DVR
vCenter Server - Self explanatory
Exchange - Test purposes
pfSense - Self explanatory
Pi-Hole - Self explanatory
Plex - Self explanatory
HTTP/SFTP - Remote SSH access with mounted NFS share on FreeNAS, which is also a directory on my webserver.
Home Automation - Just read about this on this post, in the process of setting up now.
Minecraft - Self explanatory
SecurityOnion - Using SPAN to monitor all network traffic
Docker - Not set up yet, experimenting with it because I keep hearing about it
Guacamole - Additional remote access if necessary
FreeNAS WhiteBox:
Logisys 4U Chassis
AMD A6-7400K
F2A88X-UP4
8GB DDR3
5X 2TB RAIDZ
LAGG NIC's
Cisco Aironet 1142, 1x indoors 1x garage Currently using 2/4 HP ProCurve 2824 and 1 2600-8 PoE for cameras and AP's
Future upgrades include:
Verizon FIOS Gigabit install, 940mbps down, 880 up
Replacing (8) 73GB SAS in ProLiant with (8) 500GB and switching to RAID10 as opposed to RAID5.
UPS
Still deciding if I want to keep pfSense a VM or switch to a physical box with AES-NI
Mac OSX Client
Curious what guide you used to get this setup? I'd like to dabble in some iOS dev PoC before buying dedicated hardware.
I really don't remember off the top of my head, it's currently not in use but you need the ESXi unlocker. I've heard the unlocker is unstable on 6.5 so I installed ESXi on a VM and then ran the unlocker and installed the OSX VM on that 2nd ESXi server.
Cool, thanks for the response. I'm trying to avoid all the guides out there that require downloading a prebuilt image from some random on the internet that could have vulns/etc baked right into the image :(
Ah yeah, I would too unless you can compile your own image. But the one I tried was a script that had to be executed.
What do you use to run minecraft? McMyAdmin or something like that?
Ubuntu and an sh script that runs the Minecraft process inside of a screen process so I can logout/login and manage from SSH without interrupting
How does the server keep up runnign those servers? I'm trying to get a feel for what I'll be able to run when I get my lab started.
I also am weighting the same decision over running pfSense on physical hardware or virtualized. Is the server keeping up with the FiOS gigabit just fine?
Currently I have my Optimum 400 down and 40 up, and it's been great however I like more so I won't see the FIOS until Dec 2nd which was the earliest they could come to install.
https://i.imgur.com/IvPmt6x.png
Other than the drives being completely full, it's been running great. Memory is nearing capacity so maybe I'll order some more memory before I decide to run more VM's or maybe order a 2nd server all together.
I've been running pfSense as a VM for about 2-3 years now and it's been working perfectly fine. My server has 4 ethernet ports on it so 1 is dedicated to WAN, 1 is a mirror for SecOnion and the other 2 are LACP for the LAN. The only reason I personally considered going back to a physical box was for a WAN failover with a USB 4g stick but I decided to keep as a VM.
Once my FIOS is installed I'll be using FIOS as the primary ISP, Optimum as the secondary ISP. With the mirror port added, I won't have LACP anymore so I'll be adding in an additional 4 NICs which I will then LACP those 4 and have one open port leftover, which I'm sure I'll find a use for.
I'm not worried about it keeping up at all, I have all gigabit switches and NICs and CAT6 punched down and ran to drops all throughout my house.
Oh okay thanks for the explanation! That seems pretty robust. I'm still waiting for more than 75mb in my area.
I had 100/40 before, I had no clue this gigabit plan was available. I upgraded to the 400 last month all excited because faster speeds were (finally) available and then I discovered that Verizon had blown them out of the water haha
For the NVR I use a physical one which replicates data to my NAS so if they steal the NVR it acts as a dummy and I still have the recordings. Just an idea although I guess they won't steal your server.
Pfsense and pihole? Just use pfblocker
I've used it before in the past but I'm happier with this setup
You sure? It would be easier to hand off to the pfsense box for everything instead of having to deal with DNS on the Pi and you'd get a spare pi. I can point you to lists. Of course you do you
Well I use it as a VM, so It's not that it's taking up a Pi. I'll surely take a look at any resources you have, though
Currently I have, but not all running
1 HP DL370 G6 SFF
2 HP DL380 G7 SFF
2 HP DL385 G7 SFF
2 HP DL360 G7 SFF
1 HP DL380 G7 LFF
1 HP DL380 G6 LFF
1 HP Z820
1 Dell Optiplex i7 (what this Dell in the mixture for?)
1 HP Procurve 2510 x24
1 HP Procurve 2810 x48
all in a 36U rack solutions rack
Running VMWare, Plex, Server 2008/2012 and 2016.
Basically a sandbox for learning and a means of understanding client environments
Currently running:
Recent Changes:
Software:
Upcoming changes:
Longtime Lurker first time poster Sun Lab: Cisco 2950 T5120 serving as rsync and zfs storage appliance with a Netapp FC shelf Also veritas Netbackup 7.5 T5220 running various ldoms sharing the Netapp shelf T2000 running 7 solaris ldoms with HP sas array backend
VMware lab: Brocade Silkworm 4100 FC switch Supermicro Storage appliance running ESOS with FC target 2x HP proliant G6 w 48gb ram each running ESXi 6.0 1x IBM 3650 w 36gb ram running ESXi 5.5 FC backend share storage with 12x 450gb 6gb SAS
IBM Lab: IBM P520 running AIX 7.1 IBM 9110 running VIOS w 2 AIX lpars IBM P6 8203-E4A running VIOS w 4 AIX LPARS IBM P5 9110 running AIX 7.1 IBM 7310-CR4 running IBM HMC HP DL360 G6 running SUSE EL as nfs backed for IBM power systems
Up and Coming: I'm looking at Cisco Nexus 5010 or 5020 to learn FCOE and converged 10G networking If I do wind up getting the Nexus, I see a Cisco UCS blade system in my future
In my vmware lab, I design and build Exchange 2016 DAG's and Citrix XenAPP 7.5 instances. Currently running 3 full exchange labs and 2 Citrix labs with netscaler load balanced storefront Spamfilter and mail router running hardened debian running ASSP mail proxy
Since the last WIYH I now have a HP 10622 G2 22U rack, and am able to have much fewer exposed cables to hide from my cats. In the rear I have a tower fan blowing hot air away and into the rest of the room.
I was also able to fit HVAC filters inside of the door of the rack, averting future issues with dust for a while. (20x25 + 20x16 filters)
Thanks to FCp2p, everything that is not the storage server or an appliance is diskless aside from some USB drives I inserted for future-proofing.
Debian 10 will feature a stable release for SPARC64. I'm so excited!
| RU | Device | Specs | Purpose/notes |
|---|---|---|---|
| 22 | IBM BNT G8000R network switch [rear] | TOR and core | |
| 22 | APC PDU [front] | 9xNema 5-15 | for powering things on the shelf |
| 21 | cable management plate | ||
| 20 | blank [rear] | ||
| 20 | Shelf [front] | ||
| 19 | Dell PowerConnect 2724 | shite | Out-of-Band management |
| 18 | DL380G6 | Barebones atm | Possibly storage server |
| 16 | DL360G6 | Windows 2016 DC , x5672, 40GB RAM | Fibre Channel initiator |
| 14 | DL360G6 | ESXi 6.0, 2xL5630, 18GB RAM | Fibre Channel initiator |
| 11 | Sun T5120 | OpenBSD 6.1, Solaris 10, SPARC T2 [email protected], 32GB | Fibre Channel initiator |
| 8 | Supermicro 1026t-6rf+ | FreeBSD 11, e5520, 18GB, 1.6TB platter, 240GB flash | Fibre Channel target |
| 6 | Whitebox | FreeNAS 11, Athlon ii 170u, 8GB , 2TB raid10 | (NFS, SMB server for various shares) |
| 3 | HP UPS R1500 G2 | Needs batteries charged. | |
| 2 | APC SmartUPS 1500VA | ||
| Shelf | |||
| Shelf | Netgear 1GbE DOCSIS 3.0 Modem CM800 | ||
| Shelf | PCEngines APU1d4 | OpenBSD 6.2 | gateway, pf, dhcpd, dns, cronjobs for DDNS records. |
| Shelf | Philips Hue Bridge | ||
| Hypervisors | |||
| ESXi | unifi controller | Debian | |
| ESXi | game server | Debian | May move to Arch Linux because AUR |
| ESXi | testing environment | Arch Linux | |
| ESXi | plex | 'bout time | |
| ESXi | AI playground | ||
| SPARC | Runs a hypervisor natively in silicon. Primary domain accesses configuration, Guest domains are like VMs. | In this context, domains are refered to as logical domains or "ldoms" | |
| ldom | primary | OpenBSD 6.1 | Needs to be updated |
| ldom | testing | OpenBSD 6.1 | Needs to be updated |
| ldom | solaris | Solaris 10 | |
| ldom | deprecated network domains | OpenBSD 6.1 | never used |
Still using the whitebox NAS as it's the only thing that can carry LFF drives. May replace this with a DAS.
whats your runtime on the UPS like? Trying to find one that works for me
One time I shut down everything except for the router and the switch and it lasted north of an hour when I checked the LED panel and saw 2/5 LED panels lit. The router (apu1d4) consumes 6 watts and the switch I think is 50. The datasheets should tell you how long the UPSs will last under various loads.
I've been playing just a bit, so I don't have too much, and need to spend more time tinkering
In the very near future I'm working on getting a R210 running as a PF sense box/email server. But it needs a new DVD drive first so I can install the OS
After that I have a PowerValt MD3200 with both of the power supplies and controllers. No drives though. So later on I will get it filled and used it as attached storage. It's was only $50 with the bezel so I figured why the hell not? Once I get that going I would like to host some kind of cloud storage in raid 6 for a few friends along with photos and medial from side projects.
I'm on the hunt for a half rack at a good price locally. Once I get that I will get it all off the desk that it's on now.
Laptop running:
Elastic search
In the region of 20 different services for learning XML scripting within the healthcare sector. It's mostly HL7 and DICOM protocol we use, but some XDS also. Mixture of proprietary and in house developed.
What application are writing against for Healthcare? Cerner, Meditech, Epic, McKesson,etc
It's database systems for hospitals that are connected to journal systems or laboratory management systems.
SuperMicro X9DRL-iF
Dell PowerEdge R710 II
Networking
Misc
Software
About to actually make a few VLANs here. Currently all in VLAN1
Since I use nothing but Linux in my homelab I'd really want to get started with puppet or ansible, just to get some automation going on. I'll be upgrading my main pc in the coming month so I want to recuperate the hardware for building a new firewall.
Recently bought a H200 for my R710. Im planning to move my ESXi install to a thumb drive and my VM's to a SSD so I will have 12TB to push into a ZFS array. I have seen some stuff where people have difficulties getting their H200 to install correctly, any tips before I start my work?
I'm working on two projects this month:
Transplanting old NAS hardware - my previous-generation NAS hardware is two 4U rack servers with hotswap bays that I don't use anymore. So I'm taking the guts out and putting them both in these much smaller cases. It's an awesome case - 5.5" but room for a full ATX board and two PCI cards (with risers). Best of all, it's dirt cheap.
Creating a new VM deployment setup - probably something crazy with netbooting combined with saltstack.
could you share the case you are putting them in?
Can you tell me more about the Pi's as wifi collection nodes. What is that exactly ?
hardware:
[rack]
[networking]
unifi 48 switch (core)
unifi 8 switch (poe powered to light up the rest of my * apartments drops + poe passthrough to the AP) unifi AP AC pro
[server]
only one currently, but it does its job well with <20% load
1x xeon e5-2620 v4
64gb ddr4 ecc (1/2 full for cpu#1)
1000w atx psu non redundant
supermicro mobo, is dual slot. fully loaded keeping the same hardware i would have 32 threads @ 2.1ghz with 256gb ram
10gbe fiber to the core switch
notcura fans / cpu cooler. very quiet. hard to tell its on sometimes.
5tb data drive (temporary until i build a nas) / 120gb boot ssd
[htpc]
i5 7950k
16gb ddr4
rx 550
1tb ssd for local flac's / movies i want higher quality
[gpu compute node x2]
for hash cracking / mining / messing around with computer vision and ML stuff.
celeron 3930
8gb ddr4
13 consumer gpu's each
PSU fuckery to power them all...
case is some hacked ikea furniture.
software:
pfsense
plex
unifi control
ad / dns
1 vm for reverse proxying web requests
3x vm's each running its own website
znc
openvpn x2 (one for lan, one for a lab network that doesnt have internet outbound)
sooooooon (hopefully):
16 / 24 bay nas, 2u 24 bay would be nice, but more than likely going to be a 16 bay 3u.
10g sfp+ so i can have 10g between the server, switch, and storage.
looking at 8gb sas drives, but storage gets pricy fast.
software side, going with snapraid and mergerfs. will probably populate 1/4 the drives and add on every month or two until i fill it up completely.
Copy pasta plus a bit extra from my other post:
Raspberry Pi 1 Model B 512MB - 4G Backup Internet link
Raspberry Pi 1 Model B 256MB - Temperature and UPS monitoring
48 Port Linksys SRW2048 switch
8 Port unmanaged PoE Switch
2RU Server - ProxMox host but currently turned off - i5 655k with 16GB RAM, 2x 115GB SSDs, 9x 500GB 2.5" HDDs
QNAP 8 Bay NAS - 4x RAID1 Arrays for Home drives / Media / Home Drive Backups / VMs/other Storage
Cisco 887VA
Dell R210 - Proxmox host. Xeon X3430 with 16GB RAM
Has 2x 115GB SSD RAID1 array and 2x 500GB HDDS RAID1 array. Also has a 115GB SSD OS Disk.
Hosts the following services:
pfSense
Mediawiki
FreePBX
IRC box
OpenVPN endpoint to my VPS
WDS Server
another pfSense box for playing with
apt-cacher / PXE booting server
Virtual NAS (I don't remember why I'm running this one at the moment...)
NVR/Motion Server
Zabbix
Gitea
And a couple of web servers for Devving/Testing a webapp that I'm building.
A SiliconDust HD Home Run Networked TV Tuner, this allows me to watch Free To Air TV on any device in the house, because I couldn't be bothered running antenna cabling to the living or theatre room.
Cisco 7945G IP Phone
HP 560 AP
D-Link DAP1650
##Router: EspressoBin ARM board (3 gigabit ports)
##Docker Server: (old quad-core desktop with 4GB ram)
OS: Antergos Linux
Services:
##Kodi: Raspberry Pi 2
##NAS: (another old quad-core desktop with 4GB ram, 2TB drive)
OS: Ubuntu Mate
Services:
##Backup server: (old 32-bit desktop with 2TB drive)
OS: OpenMediaVault
Services:
So I'm currently running:
I'll do a long write up at some point, but I'm running vsphere across all the compute with NSX managing the network (ospf between all the routing devices). Primary use is security research, but i've got two hosts that i use to run my 'production home' services - plex, dns etc.
New hardware wise i'm waiting for a large order from FS to come so i can actually use the 40gb switch, plus i've got 5x large lengths of bend flexible fibre, so i can run fibre to each floor's distribution switch.
homelab overkill = achieved.
W-wwhat? How much Watts does it pull?
Haven’t got a clue, probably don’t want to know!
Actual live footage of your power meter
Actual live footage of his circuit breaker...
Hmm wouldn’t of thought it’s that bad. It would of been a whole load worse if I’d gone with my M1000e plan
What software are you using?
More detail above, but it’s based on a heavily modified version of a project called nzyme. Basically 802.11 frame collection
Can you give a brief rundown of what you're running on those?
What is the purpose of using those pis as wifi collection nodes? I'm not familiar them and a brief google search didn't pull up anything I thought related to that.
At some point I’ll do a proper write up, servers are just for the usual skills learning stuff. The network is controlled by NSX and a run full packet collection & inspection using bro and a few other tools.
WiFi is just a little prototype in working on, it’s based on a project called Nzyme (it’s on GitHub and there’s a post over on r/netsec about it) but I’ve heavily modified it. I’m basically collecting all 802.11 frames and then analysing them. Let’s me do anything from spotting fake WiFi access points, to detecting deauth attacks and in the most extreme case tracking people & inffering patterns of life. I do a little bit of cellular scanning too, but I’m limited by my hardware atm.
I’m in a very remote area so have little background noise, which makes all this possible.
pics??
In the middle of re-racking and improving how it's setup, once that's done i'll post some pics / a write up
Looking forward to it!
.
Getting there.. I’m away with work for another two weeks, so once I’m back I’ll get it sorted and do a write up!