Best practices or production checklist for using multiple accounts to implement a DEV, TESTING/STAGING, PRODUCTION flow?
While developing our product, we have only used one account. Before we go live with the product, we think it's probably wise to create a new account for our production stack.
Much of our infrastructure is in cloudformation and we're working to get 90-95% of it in in cloudformation before creating the new account. Some things like Domains and Route53 and what not seem difficult to put into cloudformation.
We have two main github repos with codepipelines. I guess the developer account would watch for pushes to "development" or a feature branch while the production account should watch for pushes to master? We're also aware of consolidated billing and can get that setup, too.
Is there a best practices guide for setting this up that you like? Or a checklist that you follow? I've found a few guides online but none are too specific -- perhaps that's because this isn't as I'm worried that it will be.
Posted Nov 2, 2019, 7:31 PM UTC
This is a thing that often gets neglected until it's so ingrained that changing it is awful. I strongly recommend working out a branching and workflow model and then choosing CI/CD tooling to fit that workflow, they aren't all created equal. For instance, we're in the process of ripping out Codepipeline in favour of Gitlab and everyone is overjoyed that we're removing it, as it has been very limiting and caused us a lot of pain. Our devs settled on GitFlow, as we currently use a 2 week sprint cycle with corresponding releases. Gitflow wouldn't really work well with Codepipeline's restrictive relationship to branches anyway, but we've had a lot of problems in particular with the CodeDeploy part of our pipelines. It's one of those things where it looks great on paper, but in practice our implementation of it has been nothing but problems.
Nobody can tell you what the right choices are for your team, but I suggest your stakeholders give it some thought and come to an agreement sooner rather than later.
Alright, you might have just saved us a lot of pain, thanks! Codepipeline and deploy have always felt very limiting and I always assumed that that was just how life had to be or that I didn't know what I was doing.
(for the record, i'm using the royal we, so forming consensus should be quite easy :) )
This has been my experience with almost every aws product that sits a layer or two above the infrastructure. They do the bare minimum to get people onboard and then the product experience is miserable -- the part that I don't understand is that it slows down development which slows down my ability to pay aws money by deploying an application at scale
It seems as though you've pretty much got a handle on the nature of non-core AWS services. It might be an unpopular opinion in this sub, but my advice is to avoid them where possible.
I evaluated the following tools to replace Codepipeline:
I ultimately settled on Gitlab, primarily due to having a very feature rich pipeline syntax and the security benefit of running your own "runners" ie compute resources where pipelines are executed. For me, it combined the best parts of tools like Concourse and Drone (container-first, declarative, no plugin system to fuck things up) with the best of Jenkins (highly flexible, do pretty much anything, not locked to specific branches) and a little-known tool called ShipIt (https://engineering.shopify.com/blogs/engineering/introducing-shipit), which basically provides a stream of commits with deploy buttons next to them.
Here's an example: Imgur
Using gitflow, this is what happens on a commit to a
sprint-xxxxxxxxbranch. We :This is just what works for us, I don't even personally care for GitFlow. Point is, Gitlab allowed us to:
Lots of the above tools offer some combination of these, or even all - but we really found that Gitlab was the whole package. This is even before you talk about hosting your code there, using their project management features, etc. It's imperfect but I think it's the best tool out there, at least in the pre-kubernetes landscape (though it is built to work well with Kubernetes too, I just can't speak to how well as I haven't used those parts yet).
Thanks for the detailed write up and insight. I'm using flyway, too, and have had that integration in the back of my mind.
I'm excited that a little forethought and devops design upfront is going to greatly improve productivity moving forwards. Signing up for gitlab now and porting github repos over. Thanks again!
Glad it was helpful. FWIW you don't necessarily have to move your code to Gitlab, you can just use it as CI for an external repo if you prefer. This is currently a free feature but Gitlab has said they plan to charge for it in the future. We are GitHub users, for instance.