r/devops•
8solutions

Example of a gitlab CI pipeline that deploys to AWS SAM?

Hi, I've started working on my first gitlab CI pipeline (coming from AWS Codepipeline) and am struggling a bit.

My pipeline needs to: Build a docker image and push it to a repository (ECR), compile typescript code and deploy it with AWS SAM, and build

To get started, I attempted to modify this code to work with my own repository; however, I realized after a little while that it does not actually work. The Gitlab user is named "gitlab-examples" so I mistakenly took it to be an official repo, but I don't think that it is.

Does anybody know of a public repo with an example of a working Gitlab CI + AWS Cloudformation/SAM deployment?

Posted Nov 22, 2019, 3:41 AM UTC

212
davehodg•

I’ve done push to ECR then refresh ECS. It’s a case of getting the awscli right.

Build image Log in to AWS Tag image Push to ECR Refresh ECS

Push to ECR had its own IAM ID.

1
8solutionsOP•

Just replying to reformat so I can read it, Reddit markdown...

  1. Build image

  2. Log in to AWS

  3. Tag image

  4. Push to ECR

  5. Refresh ECS

1
davehodg•

About right.

1
83bytes•

We have a tool at work that uses ansible and gitlab runners to build an image, push it to ECR and then ECS.

we use ansible for CloudFormation templates. Each ECS job is a new Cloudformation stack and we can update/delete all resources related to a stack at the same time.

​

Unfortunately its an internal tool kinda thing and im not sure I can share the repo.

But I will be more than happy to answer questions and help out cloudformation

1
8solutionsOP•

Thank you.

I was making the mistake of leaping a chasm in two jumps by attempting to add gitlab CI and deploy from docker images at the same time. Instead, I'm going to write the docker files first and make sure I can use them for deployments before adding those stages to gitlab ci.

1
8solutionsOP•

The reference gitlab-ci.yml file I was using was a little too complex for me and I defined "image" in variables rather than at the top level of the yaml file, so I was in the wrong docker container!

1
83bytes•

if your pipeline needs to work for multiple projects, then I think it would be better to have the developers write the Dockerfile for each project.

That way your runner can just pick the dockerfile from the repo and build it.

Is that what you are doing ?

1
8solutionsOP•

I might not have been clear in that post. I was saying that my gitlab-ci.yml file had the following form:

variables:
  image: lambci:lambda/nodejs10.x
  ...other variables

rather than

image: lambci:lambda/nodejs10.x
variables:
    ...

Originally, it was using the standard docker image to run my stages because "variables.image" is not how you define the global docker image to use.

I'm not sure what you mean by "project". I intend to use different Dockerfiles for each stage. My gitlab-ci.yaml sits at the top of my monorepo and has different variables for a hotfix/wildcard, dev, test, and production envronment.

I am not currently writing the Dockerfiles but I do plan on doing so if I need them (or pushing the lambci image to my own gitlab docker registry). I'll post the gitlab-ci file in the next post... it's too large for this one.

1
8solutionsOP•

Needed to delete some blocks for space, but should have the gist

image: lambci/lambda:build-nodejs10.x #Docker image to use for build jobs - should have Python 3 installed

variables:
  #12-digit account ID
  PROD_ACCOUNT: "012345678901" #12-digit account ID for your production AWS environment
  STAGING_ACCOUNT: "012345678902" #12-digit account ID for your staging AWS environment
  DEV_ACCOUNT: "012345678903" #12-digit account ID for your dev AWS environment
       
  # region id
  DEV_REGION: us-east-1 #Deployment AWS region for your development environment
  STAGING_REGION: us-east-1 #Deployment AWS region for your staging environment
  PROD_REGION: us-east-1 #Deployment AWS region for your production environment
  
  # role name
  CI_ROLE_NAME: gitlab-ci-deployment #'SharedServiceRoleName' output from deployed setup-templates/roles.yml
  DEPLOYMENT_ROLE_NAME: gitlab-ci-passed-role #'SharedDeploymentRoleName' output from deployed x-account-roles/roles.yml
  CI_BUCKET: my-ci-bucket #'ArtifactBucket' output from deployed setup-templates/ci-bucket.yml
  
  # hosted zone name
  DEV_HOSTED_ZONE_NAME: dev.redacted.io #OPTIONAL - only needed if you want a DNS name for your review apps. Don't include trailing dot
  STAGING_HOSTED_ZONE_NAME: staging.redacted.io #OPTIONAL - only needed if you want a DNS name for your review apps. Don't include trailing dot
  PROD_HOSTED_ZONE_NAME: prod.redacted.io #OPTIONAL - only needed if you want a DNS name for your review apps. Don't include trailing dot

  # Certificate ids
  DEV_ACM_CERT_ID: asdfg12-sdfa-asdf-1234-0a2b3c4f5gasdf #OPTIONAL - only needed if you want a DNS name for your review apps.
  STAGING_ACM_CERT_ID: asdfg12-sdfa-asdf-1234-0a2b3c4f5gasdf #OPTIONAL - only needed if you want a DNS name for your review apps.
  PROD_ACM_CERT_ID: asdfg12-sdfa-asdf-1234-0a2b3c4f5gasdf #OPTIONAL - only needed if you want a DNS name for your review apps.
  
  # Uncomment to send weekly emails from this address
  #SES_SENDER_ADDRESS: [email protected] #Email address to send weekly emails from

  # Directory structure
  CFN_INFRASTRUCTURE_DIRECTORY: ./services/infrastructure
  LAMBDA_CODE_DIRECTORY: ./services/lambdas/code

.dev_variables: &dev_variables
  STAGE: ${CI_COMMIT_REF_NAME}
  REGION: ${DEV_REGION}
  SERVICE_ROLE: "arn:aws:iam::${DEV_ACCOUNT}:role/${CI_ROLE_NAME}"
  DEPLOYMENT_ROLE: "arn:aws:iam::${DEV_ACCOUNT}:role/${DEPLOYMENT_ROLE_NAME}"
  # HOSTED_ZONE_NAME: ${DEV_HOSTED_ZONE_NAME} ==>
  ACM_CERT_ARN: arn:aws:acm:${DEV_REGION}:${DEV_ACCOUNT}:certificate/${DEV_ACM_CERT_ID}
  VERIFIED_SES_SENDER: arn:aws:ses:${DEV_REGION}:${DEV_ACCOUNT}:identity/${SES_SENDER_ADDRESS}

.staging_variables: &staging_variables
  STAGE: staging
  REGION: ${STAGING_REGION}
  SERVICE_ROLE: "arn:aws:iam::${STAGING_ACCOUNT}:role/${CI_ROLE_NAME}"
  DEPLOYMENT_ROLE: "arn:aws:iam::${STAGING_ACCOUNT}:role/${DEPLOYMENT_ROLE_NAME}"
  HOSTED_ZONE_NAME: ${STAGING_HOSTED_ZONE_NAME}
  ACM_CERT_ARN: arn:aws:acm:${STAGING_REGION}:${STAGING_ACCOUNT}:certificate/${STAGING_ACM_CERT_ID}
  VERIFIED_SES_SENDER: arn:aws:ses:${STAGING_REGION}:${STAGING_ACCOUNT}:identity/${SES_SENDER_ADDRESS}

cache:
  key: ${CI_COMMIT_REF_NAME}
  untracked: true
  paths:
    - node_modules/
    - .yarn

stages:
  - test
  - build-dev
  - deploy-dev
  - build-staging
  - deploy-staging
  - create-change-prod
  - execute-change-prod

.build_script: &build_script
  script:
    - npm install -g yarn
    - echo "===== building Lambda packages ====="
    - cd $LAMBDA_CODE_DIRECTORY
    - yarn install
    - yarn run build
    - yarn run bundle-functions
    - cd $TOP_DIRECTORY &&

    # cfn.1 Package cloudformation templates
    - cd $CFN_INFRASTRUCTURE_DIRECTORY &&
    - sam package --template-file ./main.yaml \
      --region us-east-1 \
      --output-template-file ./main-packaged.yaml \
      --s3-bucket redacted &&
    - cd $TOP_DIRECTORY
    # source ./assume-role.sh "${REGION}" "${CI_COMMIT_TAG}" "${SERVICE_ROLE}"
    # sam package --template-file sam.yml --s3-bucket ${CI_BUCKET} --output-template-file deployment.yml

.shutdown_script: &shutdown_script
  script: |
    echo "===== Deleting stack ${CI_PROJECT_NAME}-${STAGE} ====="
    ls
    source ./assume-role.sh "${REGION}" "${CI_COMMIT_TAG}" "${SERVICE_ROLE}"
    aws cloudformation delete-stack --stack-name ${CI_PROJECT_NAME}-${STAGE} --region ${REGION}
    ./cfn-wait.sh ${CI_PROJECT_NAME}-${STAGE} ${REGION}

build:dev:
  image: lambci/lambda:build-nodejs10.x
  stage: build-dev
  <<: *build_script
  variables:
    <<: *dev_variables
  artifacts:
    paths:
      - deployment.yml
      - assume-role.sh
      - cfn-wait.sh
    expire_in: 1 week
  only:
    - branches
  except:
    - master

build:staging:
  stage: build-staging
  <<: *build_script
  variables:
    <<: *staging_variables
  artifacts:
    paths:
      - deployment.yml
    expire_in: 1 week
  only:
    - master

test:unit:
  stage: test
  only:
    - branches
  script: |
    if test -f package.json; then
      yarn install --loglevel error;
      yarn run test;
    else
      echo "no test file found to run tests from";
    fi
1
83bytes•

This is actually really nice.

Can you share the reference gitlab-ci.yml

​

and by "project" i mean every project in the org. There are different teams that work in different stack (node, golang, java etc) which use the same pipeline.

1