Hope the summer has been treating y'all right and you didn't blow all the vacation money on the lab.
Posted Aug 15, 2019, 5:38 AM UTC
Started the journey into homelabbing this month. My parents never backup their PC so I built a Freenas whitebox at my parents' house. As it's mainly focused on storage I decided to go along with Freenas, even if i have some VM running.
Freenas Whitebox :
Running :
I'm planning to build an ESXi host around a NUC 8i5BEH 32GB DDR4 in order to let the Freenas box as a storage and media oriented server. All the backup, webhost, reverse proxy, thing will be handled by the ESXi box. I also plan to use the ESXi box to learn more about k8s. I will also certainly buy a managed switch and a router that support VLAN, looking toward Unifi gear, but also Mikrotik.
It has been a really good month because I made my first steps into homelabbing and it's endless possibilities, can't wait to learn more and more.
1 Dell r420 dual proc 32gb ram
3 dell r610's dual proc 32gb ram
1 dell r210I single proc 4gb ram
1 dell r710 dual proc 16gb ram
cisco 3570G (Home)
Juniper ex2200 (LAB)
Id like to swap out the 710 for a 720xd or at lest get a super micro with more 3.5 storage bays..
I’m in the process of reorganizing the rack and servers/vm since I just added two new servers (r620 & r720xd)
R210ii running Proxmox with Pfsense and soon Pihole
R710 it’s my old main server that had all my VMs now it will be my main storage with 20tb of raw storage on Unraid
R620 64gb ram running Esxi which will run all my VMs
R720xd will be my backup storage server running Unraid with 31tb of raw space
The two storage servers will be connected with a 10gb link which I just received and I’m ready to setup. I’m waiting to setup the 10gb before I do the initial sync that will take a lot longer on a 1gb connection
Wireless: I have two ubiquity AP in the house. I need to add one outside since my connection drops as soon as walk out the door
how are you planning your 10gb link?
One Mellanox card on each server. I have an Aruba S2500 that has 4 10gb ports.
Currently running an R710 with a bunch of virtual machines. My plans over the next few months are to build a NAS device. At this point I'm planning on using btrfs on Ubuntu Server for the storage drives so I can hot add more drives as I can afford them without rebuilding the RAID array. My plan is to have ~80-100 TB of usable space once its complete.
Active Servers:
Other Servers laying around:
Raspberry Pis:
Switches:
Wireless:
Opnsense:
the 2 HP Server that got replaced were too loud, had too much vibration and they were on my loft and i still could hear them when it is warm outside. The Dell Servers are damn quiet so I went the all Dell way.
ESXi (main server)
Supermicro X9DR3-F
Dual Xeon E5-2630v2
256GB DDR3-1600MHz
1TB Samsung SSD
1TB WD Blue SSD
2TB Seagate HDD (5400rpm)
5TB Misc HDD in JBOD
Docker Swarm, GlusterFS, Graylog, OpenVPN, Tor Relay, Electrum Server, Nginx, Pi-Hole, Cloudflare Argo Tunnel, DNS/DHCP
Intel NUC7i3BNH
i3-7100U (2c/4t)
32GB SODIMM
Samsung 960 EVO 500GB NVMe
Hyper-V 2016
Docker Swarm, GlusterFS, Pi-Hole, Cloudflare Argo, DNS/DHCP
Currently playing with Splunk and oh boy, what a shitshow. How this tool ever got to be so popular is beyond me.
With regards to Splunk - It's front end, log source support and comprehensive query language... But agreed it still sticks!
Been hanging around here for a few years and never bothered posting in one of these...
So here we go:
###R720xd (2) E5-2680, 128GB ram, quadro p2000, 232TB
Baremetal:
plex
random scripts
tensorflow training
Containers:
the usual sonarr,radarr,ombi, unifi, heimdall
VMs:
running homeassistant, tig stack, and guacamole
jump box for work keeps my work vpn off my desktop
testing, random stuff. I usually snapshot this before messing around and revert it when I am done
ml/ai playground. right now chatting on discord using a bunch of random data-sets I got off reddit and trained for a year
Storage:
VM storage array 8 x 1TB ssd raid0
Scratch array 4 x 4tb reds raid0
Mass storage 24 x 8tb media pool is 15 x 8tb with 3 x 8tb for snapraid storage pool is 6 x 8tb, using duplication scripts for backups and more important data
###R220 (maybe r210 ii) E3-1220 v3, 16GB
Currently running untangle
###HP desktop i5-2500, 16GB, 4TB
Blue iris
###Networking
###Off site
####Future Plans:
Physical:
Virtual (VMs and LXC containers):
Running: R720 R620*2
Planning on deploying windows server 2016 based vdi solution...for no reason in particular
3x i3 Dell Desktops. Working on my RHCSA and then RHCE.
Looking to buy three rack mount low power server setup to learn Docker, K8, and Openshift.
No new hardware to show. Hoping to maybe trade for a 3 node Atom system.
Running a 2U supermicro box with dual e5-2620s and 116GB of RAM, used for testing at the moment but will be running 24/7 once I move all VMs from my old i7-4770 white box with 16gb of RAM. Just need to find a place to put it where I can't hear it because it is LOUD. looking to build another white box and trying to find some cheap 1u chassis's to put everything in. Also planning on going 10Gb soon so looking for a cheap switch. All my servers are running CentOS 7 with KVM, managed through virt-manger and virsh although I'm looking to build a web interface with the libvirt node.js API
Nothing yet but I just oprdered my first Rack server. A Dell PE r720 with two 10 core Xeon E5-2680 v2 and 128 GB of memory.
I don't have a single drive for it so that's something I'll have to figure out. I'm also not sure if I should install windows server 2016 desktop and the run hyper-v and DC, or if I should install proxmox and then run the Win server as DC in a VM.
2* R610 (2* Xeon x5570, 32GB) - Virtual Machines for various things
Supermicro 828TQ (4* Opteron 6276, 64GB) - Arriving from eBay in 3 days, planning to use for render server, but need to do some research.
Generic 2u box (i7-2600, 8GB) - NAS (6TB + 1TB cache) I know it’s not much storage but I really don’t need more.
PowerConnect 2848
Planning on getting a Quanta LB6M for 10Gb but I need to make sure everything will work together first, and maybe get faster cache drives for the NAS.
Running: Odroid XU4 with Ubuntu Server 18.04 LTS onboard, 500 GB of storage (Seagate Barracuda)
Odroid is running Owncloud instance and also serves as dev server.
Planning: designing a modular "rack" for future expansions, additional SBCs.
Still generally the same as I have over the last year, and wishing I had money to upgrade it...
3x HP DL360 G6's with dual 6-core E5649's and 96GB RAM running my PVC software (https://github.com/parallelvirtualcluster/pvc - shamless plug since I first mentioned it in a thread the other day, if running a KVM+Ceph 3+ node cluster is something you're interested in!)
3x Whitebox nodes with single E3-1245 V2's and 32GB RAM for Ceph storage. Each node has a 10TB, 2x 8TB and 4x 3TB HDDs for bulk data, as well as an 800GB SSD for VM disks
2x DL360 G6's with a single quad-core CPU for routers
10GbE SFP+ DAC everywhere
My next major plan is to replace my "base", which is a server with identical specs to the Ceph nodes, with a LattePanda Alpha to help save some power. Alas I got it at the start of the month and mine was DoA, so I'm just waiting on the replacement now.
Next major step is actually not server-related - I need a service and electrical panel upgrade in order to move towards my new cooling plan - a chiller. But for now that's a next-summer project, giving all winter to deal with the electrical stuff.
Cubelab: Dell: T5500 8CPU PCI passtheough: Ubuntu to manage proxmox /workstation Proxmox Awx (Upstream to Ansible) bookstack WinAD/SQL Server Future: endpoint protection and endpoint encryption servers Future: certificate authority Future:VPs endpoint
Homelab: Ubiquity poe24 /firewall4 thing
Incomplete proxmox cluster: 2x HP dl360. 8CPU 72gb of ram Telegraf/grafana/docuwiki
1xHp dl360 8CPU 72gb of ram on order to rebuild and setup redundant/failover
—— Some HP surestore shelf with 300gb drives to be hooked up
3x d2700 - 25 bays each - I have ordered rack ears for the two that didn’t come with it. 1 connected but not really used.
Probably going to start loading up on drives around Christmas or after Christmas. “75” hungry bays to feed.
Connect HP lto4. G2 autoloader 8 tape
Future: Cisco ucs simulator Some GNS virtual routers BigIP appliance Cisco/devnet scripting Asterisk:pbx Remote Desktop/xen desktop stuff Akamai DDOS or cloudflare connection Dropbox replacement - forget the name Local copies of work applications for testing/development - smartcards/pgp Organizer Blockchain development? Honeypot
Not too much as changed. Most recent hardware is updating my media storage server from 16GB to 32GB RAM so ZFS will be happy with I add more 8TB hard drives
Top rack:
Bottom Rack:
I think you might have added the wrong photo, you're describing a rack layout but the pic is not of a rack.
Thanks, I edited it to make it more clear. I haven't taken any pics of the racks lately.
^(and also because I moved and just did enough wiring to make it all work, and haven't made it neat)
👍, No worries. Thanks for the contribution.
Hardware:
Software (In proxmox vms/containers):
My current setup, version 3 if you will:
Netowrking is handled by a cheap Asus Wireless Router and DNS/DHCP coming from my "server".
HP Z800 host with 2x Xeon X5670, 24 GB ECC RAM, single 250 GB Samsung SATA SSD, single 2 TB HDD of some variety.
Hypervisor is currently ESXi 6.
Three VMs running 24/7
DC01 as my domain controller, Windows Server 2012 R2 Evaluation
DHCP as my DHCP and WDS server, Windows Server 2012 R2 Evaluation
FileServer as my (you guessed it) file server, also running Windows Server 2012 R2 Evaluation
All three VMs are on running off the datastore on the SSD. FileServer has the 2 TB HDD mounted directly as a vmdk to it for direct access. I want to be able to easily pull it out and put it in another machine if I need access to all the files in the event of a drive crash ony my main desktop PC.
DC01 and DHCP are getting 2 vCPUs and 2 GB RAM. FileServer gets 4 vCPUs and 4 GB of RAM. I only have 1 Gbe networking going, so I could probably drop it down to 1 or 2 vCPUs and 2GB RAM quite easily as it's not accessed all that often.
Previous versions:
It came from eBay with 2x Xeon E5645 CPUs, 8 GB RAM, and an old WD Black that was making funny noises. Windows 10 Pro was installed and activated, so after doing a fresh install of Windows 10 I just used it as a hypervisor and had 2 servers deployed in Hyper-V as domain controller and DHCP, which was Version 1.
Version 2 involved upgrading to the current CPU/Memory/Storage that I have. I went with Server 2012 R2 Datacenter evaluation as my hypervisor and domain controller, with DHCP and FileServer their own VMs.
Future plans:
Install Plex on FileServer for easy movie streaming to the TV/phone.
3 more 2 TB HDDs for FileServer to run them in RAID10. Maybe 4 drives of a more robust model than the Seagate Barracuda Compute unit. Or maybe just bigger. Or 1 more Barracuda of the same size, run it in RAID 0 and have incremental backups to a 4 TB drive also passed through directly to FileServer. I'm still mulling this over.
Add a VM to run BOINC/Einstein at home. This may involve upgrading the cooling solution, which may be more money than I'm willing to spend right now, especially on such and old box.
Maybe a Ubiquiti network setup. USG, 8 port 60-watt switch, AP-Lite, run the controller on a Ubuntu Server VM.
Someday I'd like to have space for an actual rack for networking stuff and have something "small" like a Dell R210 II or a DIY solution in a closet just for file server duties, and use the Z800 solely as an educational tool, but for now it will live tucked under my desk. It was set up like it is as part of learning the ropes for my current job. I started as the tech support guy for a school, then got told one day I'd be taking over some sysadmin duties. We user ESXi/VMWARE essentials with all Server 2012 R2 and Server 2008 R2 in production at work, so it was a good learning experience. I've since gotten to deploy a semi-production ESXi server to host a minecraft server for the boarding students to play on (and me when I have free time).
It's been a while since I've made a thread here.
Currently running two hosts,
In no real order, here are the hosts and VM's.
1. X3650 M3, 288 GB RAM, 2x E5645
2. R720, 384 GB RAM, 2x E5-2697 v2
h-DBCLU01 - MySQL NDB Node1, Debian 10
h-DBCLU02 - MySQL NDB Node2, Debian 10
3. R820, x GB RAM, x
4. SuperMicro, 128 GB RAM, 2x E5-2640
Future Plans:
Future plans are getting done with all my little projects. I need to learn how to harden my services and hosted systems.
Expand networking, 10G - Not sure what hardware yet, but considering Ubiquiti.
An actual rack. Can't fit a 48U in my basement. But as I am looking to buy a house, this might change soon.
Getting a better handle on Docker. Might be a trust issue. It's me, not you.
May I ask what you have running on those databases? I've been thinking about putting a few on my R710, but dont know of any applications i would really have for one.
Also, Mikrotik makes some decent 10gig capable switches for a nice price. (Just my 2 cents)
Hello, and sorry for the very late reply!
So for MySQL - I use it for web apps and other hosting data, some people have wordpress, others have asset management sites and so on..
MongoDB is basically just a driver for things like Graylog, Foreman and so on. Just runs the actual systems.
Timeseries data is stored in Elasticsearch - This is interfaced with graylog, and on to Kibana. I also use Graphite to import RRD Data for storage systems.
My MSSQL server is only storing a very few systems like Veeam.
Hope this answers everything
Thank you for the tip, I'll check out Mikrotik - Not a lot in the used market in my area, but Ebay has lots!
Thanks for the run down!
If you're willing to share it, what's your electric bill and your electric rate? I'm in an area with 21c/kwh power and I'm cheap as shit, wondering what I can expect from running my 710/Storage/TS140 24x7. I realize my power bill is probably going to double.
Hello! Sorry for the late reply, swamped (tm) at work,
I haven't really measured my power consumption, but I'm going to install an UPS soon, so I can definitely gain some numbers of the entire lab for you.
For Power to Muscle ratio, the M3 is definitely the expensive one.
I want to put my unraid in a bit smaller box of some kind, with one CPU, since it really doesn't do much, and won't need the power.
21c/kwh? ouch...mine starts at 6.5c and tops out at 8.5c on the top tier of usage. It's nice to have some of the cheapest power in the country!
Yeah, well, Alaska is a bit harder than most places to work reliable power :)
How does your jump box work? I’ve been thinking of making something similar but not very sure of what’s going on or how it works.
Hey,
So my jump box, it's pretty simple, something you usually find in enterprise settings using Remote Desktop
Expose 443 towards a remote desktop gateway server, or set it up through a reverse proxy like WAP, HAProxy or Netscaler.
Connection hits a RDGW server, which forwards the connection to a remote desktop server running what ever you'd want, here I use Windows Server 2019, you can pretty much go as far back as remote desktop allows.
You can also expose a windows server directly through rdp protocol, but I highly advise you never ever to do that.
Guacamole also allows for setting up a access gateway you can expose to the Internet. It also supports 2-faktor authentication, you can even set up stuff like Google auth and so on.
My Setup looks a bit like this ADFS Running on my f-AD01 server WAP runs on my f-WAP01 which forwards my rdgw address with a Let'sEncrypt certificate to f-RDGW
I'm only using one management server, and not the "RD Farm" part of Remote Desktop Gateway, so under my RDP connection settings, I input my gateway address under advanced settings -> gateway, and my management server as the computer I connect to which is f-MGMT01
Hope this makes sense
Hey, can I ask which vlmcsd Docker container you are using? I can see two from Googling, one is a cached page and no longer accessible and the other is in Chinese. Thanks!
Hey,
I just looked and you are right. The one that 404's is the one I used, and is now gone.
I'll look into finding an alternative.