Hope you all have a great Easter weekend and get some good labbing in!
Posted Apr 15, 2019, 5:40 AM UTC
Your docker scripts look cool. Could you please tell more on how this whole thing works?
I would love to. There are actually quite a few moving parts, so let's look at an example. calling make nextcloud changes into nextcloud.d and runs mds.sh run. The first thing nextcloud.d/mds.sh does is source the top level mds.sh script. This holds the default definitions for functions, variables, etc. So nextcloud.d/mds.sh run really calls the top level mds.sh run. But before it actually runs anything, it sets some variables. In this case, its sets the name of the container, the db, and the container network, as well as container arguments like where to mount volumes, etc. This example also prompts the user for the root user and password for nextcloud. When we actually get to the run part of the script, the first thing that happens is we check if the container is already running. If it is, we quit. If we detect a Dockerfile in the nextcloud.d directory, then we build it with the same tag as conImg. Then we call the preConfig method. By defualt, this method does nothing, but we can override it in the container specific directories (like nextcloud.d) to do things like copy over files required before we run the contianer. If the conNet and conDB variables are set, we run the network and db container respectively. Then we run the actual container with a regular docker run command. It might look kind of confusing because we are evaluating the args array in that call, but that should equate to typing things like -d -p 80:80, etc. We then call postConfig which is similar to preConfig but obviously runs after the container is built. You can then do things like make -j CMD=restart all to restart all the containers in parallel.
Running make init from the top level allows you to search for containers, and upon selecting one, opens that specific container.d/mds.sh file for editing. It's prepopulated with some example stuff but you can really do whatever you want here. Overriding the default methods is really where this framework shines.
The final part of make init generates a reverse proxy config for all enabled containers. The term 'enabled' means two things. First, the directory ends in '.d'. Changning nextcloud.d to nextcloud disables everything in MDS around that container. You can't start, stop, build, or anything. Secondly, and more specific to the proxy, 'enabled' means there is an 'exposedPort' variable set in nextcloud.d/mds.sh. The proxy.d/autoconfig.sh loops through all of the enabled directories and specifies an upstream server block in the nginx config. This would add
upstream nextcloud_server {
<ipOfHost>:<exposedPort>
}
to the config. The ipOfHost by default comes from the IP of the interface connected to the default gateway but again, you can override that if you need to proxy things running on different machines. For example, my emby.d/mds.sh overrides all the base functions to do nothing, but provides an exposedPort and conIP so that the proxy will route to a different host running the emby vm.
The proxy also does some boilerplate stuff for proxy configs, then it runs the linuserver/letsencrypt image and specifies every enabled directory as a subdomain and generates all the https certs.
That was quite a long-winded answer but I think I covered everything. If you have any other questions, don't hesitate to ask! I also appreciate any pull requests or feature requests
Wow, thanks for a reply. This is a lot more comprehensive than i had expected. Thanks a lot
I somehow haven't seen keycloak before, I'm gonna have to check it out.
those docker scripts look interesting, but out of interest, why not kubernetes? Or at least treaffik.
I Was in the same 'kubernetes is hard' space as you, but actually surprised myself how much its not. Rancher let me get a working (3 node) cluster up an afternoon, with external load balancing, proper external dns resolution (servicename.home.example.com).
MetalLB, nfs-client-provisioner are the main two 'magic' pieces of kit that make it useable at home.
I tried getting Kubernetes to work but I couldn't get it to install haha I've tried kubeadm and some other methods. I also had issues getting openshift to work. They both just seemed to complicated for my needs. Plus, living the #GentooLife I enjoy doing things my own way haha it started as a fun side project and actually ended up being perfectly viable for me
Next steps (gear already purchased):
Replace Dell R710 (dual X5550) with R820 (quad E5-4620). Demote R620 to dev server and make R820 the new production server. Sell R710.
Add DVD drive and 96 GB RAM to the R820 (for a total of 128 GB) after taking 80 GB from the R710 and 16 GB from the R620.
Make a 200 GB SSD the new boot drive for the R820 and the two 146 GB 15k drives from the R710 the new boot drives for the R620. Sell the rest of the R620 drives (it came with fast expensive ones I don’t really need) and get a few smaller/cheaper ones for both servers.
Replace two UPS’ (APC SMX750I & SMT1000RMI2U) with three larger UPS’ (SMX1500RMI2U & two SMT1500RMI2U). Sell the two smaller ones.
Add the H810 controller to one of the Rx20's. Set up the Dell MD1220 and see if the noise level is acceptable. If yes, proceed with the plan to migrate from Synology NAS’ to Dell gear. If not, sell all drives and keep the MD as eye candy.
Most of that will hopefully happen over the next three weeks (I will only have 8 working days in the next 21 days).
As for software, next plans are setting up Grafana and Ansible.
What's the sound difference between the R710 and R820?
The R820 is about as quiet as my R620. The R710 was louder until I tuned down the fan speed with IPMI. Right now I’d say the R710 is slightly less noisy.
Idle power draw is about 200W (compared to 140W for the R620), then again it has rather low-end CPUs (I assume the E5-4650’s draw much more).
Edit: I‘m dumb. 200W power draw was with just one PSU plugged into my rack and the other to another circuit, so likely closer to 350W.
How about heat from it? I want to replace my r910 with something a little newer and the 820 with 4 socket looks great. But wonder if it would be hotter and louder
Louder, no. The R910 is one of the loudest ones there is, and the R820 is really quiet in comparison.
Same for heat (of course that also depends on the CPUs used, but the E5 Xeons are much more effective than their predecessors).
Haven't fired up the R820 yet. Will know more in a few days.
Running now:
ESXi 6.5 on a Dell R710:
Hardware:
VMs:
Old random Supermicro 1u running pfSense for my whole house:
1 x Mikrotik Cloud Smart Switch CSS326-24G-2S+RM, 24 x Gbit LAN, 2x 10Gbit SFP+
1 x UBIQUITI UniFi AP AC LR (no cloud key, set it up with the Android app)
Software:
Planning on getting sometime around summer:
###Current:
###vmWare Cluster Hardware: ######3 x Dell OptiPlex 9020M
###NAS Hardware: #####FreeNas-01 ######1 x Dell OptiPlex 9020M
#####FreeNas-02 ######1 x Acer Veriton x4620g
###VMs:
###To-Do:
Hardware:
VMs:
Docker Containers:
Changes soon™:
Notes:
Please pardon any formatting errors, I am mobile.
It's been an exciting 2 weeks.
Switched from my single node Packstack install to a 3 node Kolla-Ansible behemoth. Still tuning and working away at rebuilding VMs. This time, everything is managed by Ansible, so next time I make a major change, it should be much easier to rebuild the VMs.
I've also segmented my network into VLANs and properly configured a Firewall and added Suricata into the mix.
Hardware:
VMs (more to come):
Todo:
Currently I run a two node proxmox cluster. Dell 1700sff and whitebox dual e5-2670 256gb ram. I sold the third machine, an avoton whitebox nas.
In the mail are two xeon d 4bay machines and parts to build a whitebox dual e5 v3 server.
Once all is done I plan to run a 4 proxmox / ceph cluster with separate 10g networks for vm and storage traffic.
I unfortunately have to downsize the lab,(2100w total power draw is too darn high). Will hopefully just be running an r410, and r620 when I am done with the downsize. Currently have 2 x3550m3 with 2 x5650's each and 32GB RAM, 2 x3650m2 (l5640's, 32GB and 250gb), 2x3650m1 with 48GB RAM each 250gb boot drive and running ESXi. For storage I have two exn4000 IBM storage arrays with 4.2TB if usable storage each. All will be sold except for the r410 and new to me r620.
What I will hopefully be left with is 1 r410(x5650's, 32GB RAM, 500gb boot, and ESXi hypervisor), and 1 r620 (e5 2620's, 192GB RAM, and 300gb cheetah drive, ESXi hypervisor).
Software and such that I want to learn will be primarily a lab environment for messing with scripting, without the risk of Messing up production systems
My wallet disintegrated hearing your power draw.
Just brought my old Lenovo Ideacentre up as a second "server" to use for Veeam and logging, on top of ESXi. It has a Celeron CPU, but it can easily handle 2 VMs without running the CPU at 100%, funnily enough. Just had to upgrade it to 16 GB RAM by cannibalizing some broken laptops.
My R710 is trucking along, I would list what software I have on it, except I kinda lost track. It uses libvirt with QEMU for virtualization, and docker for containers.
My i7-920 machine is off for the most part, because it has really poor fan control and is thus louder than my R710, and it sucks power like no one's business. Should look for some more modern hardware which sucks less power second-hand.
No new hardware planned in the next 30 days, but I might replace the custom libvirt setup on the R710 with ESXi, simply because networking is a mess. Entirely depends on how crippled ESXi would be because of the older version and the various errata workarounds needed for things like PCIe pass-through, which is why I rolled libvirt and QEMU instead.
i7-920
How many watts was your box pulling? I never really thought the old i7s were that big on power.
Enough to warrant not keeping it on. My R710 uses less power, but is way more capable since it has more RAM.
What I’m currently running NAS:
Networking:
Home automation equipment:
Miscellaneous equipment in the rack:
Software running on NAS:
Future plans
I'm in the middle of a major upgrade, and moving my gear from the basement to the garage due to heat.
Current:
3x HP DL380 g7, each 2x X5650, 48GB ram, 4x1GbE - racked with cable arms in an HP 42U rack.
Running VMware 6.5
And a NetApp FAS2240-2 with a DS2246 shelf (24x600GB 10K SAS) and a DS4246 shelf (24x3TB SAS) fully licensed. The filer head is dual controller and has the 8G FC module installed.
The SAN connects to the vSphere cluster via a single HP (Brocade) 8G FC switch with MPIO (dual path connectivity for all links)
The network is an Untangle server (virtual machine) connected to a pair of Cisco 2960S-48FPS switches, stacked. All this backed by a pair of HP PDUs connecting to HP R5000 5kVA UPSes. And a KVM console and switch as well.
New: The DL380 Gen7 servers are being replaced with DL360 Gen8 servers with 96GB ram and 2x E5-2665 CPUs. Haven't decided on 2x146GB disks for ESXi or diskless boot from SAN.
The network is being replaced by a dedicated Untangle server (DL320e Gen8 v2 with 4x10GbE), 2x Cisco 5548UP 10G switches, 1x 2960S-FPD (PoE+, 10G uplinks), and 1x 2960S-48TS for management.
The SAN is being replaced by a dual controller, dual chassis NetApp FAS3240 with dual 10G (iSCSI) per controller, 512Gb flash cache, a DS2246 with 24x400GB SAS SSD, and 2x DS4246 with total 48x3TB SAS.
An additional server will serve as a backup for the SAN - DL380e Gen8, 14x3.5", with 2x400GB SSD for OS and 12x 8TB SAS (72TB usable) for the backup - using Veeam.
Use: Security cameras - Avigilon - we are using 5.0MP cameras to protect the house.
AD; Usenet download stack; Plex; Hass.io auto ation (planned). Some MySQL and MSSQL servers for dev. That sort of thing.
Wall of bulletpoints incoming!
Networking:
Servers / Storage:
Misc
Current setup:
In the near future:
Dell's mezanine board for 3&4 CPU already in the way to me, as well as couple of E5-4610s. Also in plans to expand x8 backplane in R820 to x16 - cage and backplane itself already purchased. Not to mention changing 6*3TB drives in FreeNAS to 4TBs, and filling those horrible empty hard drive slots in R820.
Physical:
Virtual (VMs and LXC containers):
I ordered an E3-1270 v2 to replace the i7-3770k in my desktop so I can do GPU passthrough and not have to primarily run Windows, so I'm going to put the 3770k into my server so I'm not so CPU limited. I'm also planning to move bookstack, gitea, and maybe nextcloud to docker, and finally delete that server 2016 VM. Also want to add a macOS VM for guacamole, though idk if that'll be too laggy.
Edit: E3-1270 v2 was DOA so I decided to just upgrade my desktop to Ryzen. Put the 3770k in my server and it's soooo much faster than that shitty Pentium.
ESXi 6.5 on a MacPro4,1 2009:
Hardware:
VMs:
High Sierra & Windows 7 on a MacPro5,1 2010:
Hardware:
Debian 9 on a MacMini3,1 2009 Server:
Hardware:
Software
This month has seen me add 128GB of ram and a couple of 1TB disks to my Dl380 G7 running vSphere 6.7.
This means I have enough resources to run Red Hat Satellite on it for the next couple of months (NFR license, but I'm losing access to that then as I'm moving jobs and I don't think the new company is a Red Hat partner).
Although I have signed up for a Red Hat Developer account so I'll still be able to access their software to maintain my skills in it.
What am I planning to deploy?
Im currently redoing everything I had sense I moved and got a new job a few months ago and deciced sense I am no longer doing sysadmin/networking work and more AWS to move most of my stuff to AWS.
Current setup:
Upgrades:
3 Skull Canyon NUCs Running ESXI 6.7u1
What is this? My Google-fu must be weak as I'm mostly getting results for personal box.com subscriptions.
Patent Pending.
I kid, I forgot the space between personal and box! lmao -- It's just a Win 10 VM for my wife to use for embroidery software (she has a Mac, and likes one that runs on Windows only).
Ha! Alright I gotcha. Thanks!
Are running blueiris in a vm? How is that working out for you? Do you use intel quick Sync?
I have 4 cameras assigned to it right now, and I've limited it to 3 cores of CPU:
https://i.imgur.com/QuAbnD5.png
I've ran it this way for 4 years. Never had any issues. :) I'm not taking advantage of any quicksync or hardware acceleration. My 4 cameras are set to 720p. I'll experiment with hardware passthrough for the Intel Iris graphics and see what comes of it; This is the only VM that would need it, so not being able to pass that through for others wouldn't be an issue.
Thanks for asking!