r/aws•
8solutions

Best practices or production checklist for using multiple accounts to implement a DEV, TESTING/STAGING, PRODUCTION flow?

technical question

While developing our product, we have only used one account. Before we go live with the product, we think it's probably wise to create a new account for our production stack.

Much of our infrastructure is in cloudformation and we're working to get 90-95% of it in in cloudformation before creating the new account. Some things like Domains and Route53 and what not seem difficult to put into cloudformation.

We have two main github repos with codepipelines. I guess the developer account would watch for pushes to "development" or a feature branch while the production account should watch for pushes to master? We're also aware of consolidated billing and can get that setup, too.

Is there a best practices guide for setting this up that you like? Or a checklist that you follow? I've found a few guides online but none are too specific -- perhaps that's because this isn't as I'm worried that it will be.

Posted Nov 2, 2019, 7:31 PM UTC

157
tastycidr•

We have two main github repos with codepipelines. I guess the developer account would watch for pushes to "development" or a feature branch while the production account should watch for pushes to master? We're also aware of consolidated billing and can get that setup, too.

This is a thing that often gets neglected until it's so ingrained that changing it is awful. I strongly recommend working out a branching and workflow model and then choosing CI/CD tooling to fit that workflow, they aren't all created equal. For instance, we're in the process of ripping out Codepipeline in favour of Gitlab and everyone is overjoyed that we're removing it, as it has been very limiting and caused us a lot of pain. Our devs settled on GitFlow, as we currently use a 2 week sprint cycle with corresponding releases. Gitflow wouldn't really work well with Codepipeline's restrictive relationship to branches anyway, but we've had a lot of problems in particular with the CodeDeploy part of our pipelines. It's one of those things where it looks great on paper, but in practice our implementation of it has been nothing but problems.

Nobody can tell you what the right choices are for your team, but I suggest your stakeholders give it some thought and come to an agreement sooner rather than later.

4
8solutionsOP•

Alright, you might have just saved us a lot of pain, thanks! Codepipeline and deploy have always felt very limiting and I always assumed that that was just how life had to be or that I didn't know what I was doing.

(for the record, i'm using the royal we, so forming consensus should be quite easy :) )

but we've had a lot of problems in particular with the CodeDeploy part of our pipelines. It's one of those things where it looks great on paper, but in practice our implementation of it has been nothing but problems.

This has been my experience with almost every aws product that sits a layer or two above the infrastructure. They do the bare minimum to get people onboard and then the product experience is miserable -- the part that I don't understand is that it slows down development which slows down my ability to pay aws money by deploying an application at scale

2
tastycidr•

It seems as though you've pretty much got a handle on the nature of non-core AWS services. It might be an unpopular opinion in this sub, but my advice is to avoid them where possible.

I evaluated the following tools to replace Codepipeline:

  • Jenkins (via years of experience with it)
  • Concourse CI
  • Github Actions
  • CircleCI (also years of experience)
  • Writing my own!
  • Various others which were more cursory glances through documentation (codefresh, DroneCI, some others I can't even remember
  • GitLab

I ultimately settled on Gitlab, primarily due to having a very feature rich pipeline syntax and the security benefit of running your own "runners" ie compute resources where pipelines are executed. For me, it combined the best parts of tools like Concourse and Drone (container-first, declarative, no plugin system to fuck things up) with the best of Jenkins (highly flexible, do pretty much anything, not locked to specific branches) and a little-known tool called ShipIt (https://engineering.shopify.com/blogs/engineering/introducing-shipit), which basically provides a stream of commits with deploy buttons next to them.

Here's an example: Imgur

Using gitflow, this is what happens on a commit to a sprint-xxxxxxxx branch. We :

  • run code quality tests, unit tests etc and build a .jar artifact (puts to S3).
  • build AMIs with packer for multiple environments
  • run flyway (database schema updates) and then deploy to our QA environment (this is a terraform based blue-green)
  • run a terraform plan for production
  • <Manual Approval> Run Flyway for Production
  • <Manual Approval> Deploy to production (again, terraform based blue-green)

This is just what works for us, I don't even personally care for GitFlow. Point is, Gitlab allowed us to:

  • have different build stages trigger for different branch names (for example, feature branches just run tests/sonar/build artifact)
  • Implement manual approval gates (if Github Actions had these, it probably would have won - looks like a great product!)
  • avoid using IAM users for S3 Uploads, terraform etc (because we control the runners, they're ec2 autoscaled workers with IAM Roles attached)
  • use custom build containers with whatever tooling we like in them (these are also built via gitlab =D)
  • keep the pipeline config declarative and next to our source code
  • Easily see what is deployed to each env and for each service
  • restrict who can run manual approval gates (this is a paid feature of GitLab)

Lots of the above tools offer some combination of these, or even all - but we really found that Gitlab was the whole package. This is even before you talk about hosting your code there, using their project management features, etc. It's imperfect but I think it's the best tool out there, at least in the pre-kubernetes landscape (though it is built to work well with Kubernetes too, I just can't speak to how well as I haven't used those parts yet).

3
8solutionsOP•

Thanks for the detailed write up and insight. I'm using flyway, too, and have had that integration in the back of my mind.

I'm excited that a little forethought and devops design upfront is going to greatly improve productivity moving forwards. Signing up for gitlab now and porting github repos over. Thanks again!

2
tastycidr•

Glad it was helpful. FWIW you don't necessarily have to move your code to Gitlab, you can just use it as CI for an external repo if you prefer. This is currently a free feature but Gitlab has said they plan to charge for it in the future. We are GitHub users, for instance.

1
Kubiedo•

You should take a look at the well architected service. And also the partner network has released a set of common best practices for your production services: https://aws.amazon.com/partners/technical-baseline-review/

3
powerbronx•

There's a simple way to do it through an AWS Service

See this post. Especially my comments: https://www.reddit.com/r/aws/comments/dq4ejs/am_i_understanding_aws_accounts_correctly/?utm_medium=android_app&utm_source=share

1