My god, it's a compute cluster!
Hi folks.
TL-DR: Mister Fan and the 5 Boys. And coffe. And networking. And storage. Stuff.
I had a lot of fun with a bunch of PI4s over the last few days and I feel like sharing it. :) This should be tagged as show-and-tell I suppose, but I don't see how I'd do that myself.
We just had an internal company coding convention. Practically, we'd shipped ~80 developers and 2 admins from all across europe into an office and we'd have 1.5 days to do something in a hackaton. Anything. It should propose a benefit for the company, but that's a big thing.
So, we've been struggling with our software deployments. At my current employer, current deployment and management strategies are failing already, because bringing a new service into production takes ~2-4 weeks from my own team. This works if we're dealing with 2-4 services, but recent changes in the company group dumps ~20 different new services on us right now, tendency growing. FAST.
So I've been figuring.. how should I push our current container experiments in a flashy way? Containerized deployments would cut the 2-4 weeks into 1 day or two.
Well. If you want to get peoples attention, use PIs. What'd be more fun than bringing a bunch of PIs, deploying the entire container orchestration on that, and implementing a software on that. I got budget approval for 2, 3 other admins also chipped in personally. As such, behold, a mighty containerized compute cluster.
Those are 5 PIs running hashicorps nomad + consul, running gitlab, continuously deploying ~7 applications in our core languages. The notebook in the background, a DELL XPS 15 serves as the nomad+consul server, as well as a storage node, since the PIs are running small and fragile SD cards.
The fan in the foreground had been installed on the fly, btw, because I had underestimated the heat issues of the PI4. After about 3 hours once I had 8 people throwing CI/CD jobs at the gitlab instance on the pi with a bit of tape on it, that poor thing started overheating and throttling seriously and fiercely. No one could build and deploy any software anymore! Such terror!
As such, we deployed the fan next to the PIs and put it right between the modes of "hurricane" and "death by tornado". This promptly vacated every single loose piece of paper and some other equipment from my desk onto the desk of one of the developers, who was rather confused by this sudden influx of paperwork and wind. However, that was a lower priority at the time! Our gitlab instance started running faster once we had active cooling for our PIs. Aha! :)
All in all, it was a horrible idea to do this on passively cooled PI4s. Those things heat up really, really fast once the processor is running at 90+% nonstop. On the other hand, it got a running gag started of Mr. Fan being the most valuable team member. Also, building ARM32 docker images if everything you have are AMD64 processor is a pain. Also, finding a 32 bit ARM JRE is surprisingly hard. There's so much weird things going on with this.
But it was a lot of fun to get this entire modern software development stack working on a jumble of PIs. We probably had the future of the current software development at the company running on 5 PIs for a day or two. That's a statement for these little boxes of compute power. I'll totally look at something similarly weird next time.
If you have more questions about this utterly insane project, let me know.
Posted Sep 27, 2019, 11:22 PM UTC
What are you thoughts on Nomad? I'm a fan of other Hashicorp products but never heard of this one.
It's a young project, but it has it's place in my opinion.
Practically, in a position like we're in,we're not asking: Nomad or no Nomad? We're rather asking: Nomad, Kubernetes, Swarm or DC/OS?
In my book, if you have the budget and you're allowed to, you should probably buy in Kubernetes via EKS / GKE or any of the kubernetes cloud providers. Currently, there is a lot of hype to integrate and support kubernetes as a deployment platform. Gitlab has native support for it, a bunch of applications start delivering helm charts besides modules for classical configuration management systems. In cases like these, nomad currently is a second class citizen and you generally either have to contribute these things yourself, or build such integrations yourself.
However, we have customers who object against using amazon or google. They don't want their placed data to be under control of a US company even with an EU subsidary.
And in this case, nomad scales down much much better than k8 in my opinion, while being more flexible than DC/OS. Operating K8 can easily turn into an extremely complex situation. And in this case, nomad is in a very nice spot. It's fairly simple to setup securely and robust, and then you add the components and complexity you need on demand, not all at once.
From there, we've spent ~2 weeks setting up a 3 server / 5 node nomad cluster at work. Worst part of that was a bit of TLS manglement to get the authentication working. Once we had that, other things caught on fire and we couldn't really manage the cluster, but apparently, the developers who're on there had uptimes beyond 2 nines without a hitch, in spite of one of the datacenters this is in being down for a couple of hours. I'm quite happy with that.