Unit 3 - Continuous Deployment with GitLab & AWS
About 8 min
Unit 3 - Continuous Deployment with GitLab & AWS κ΄λ ¨
3.1 Unit overview
- we will learn about deployments and take our website project and deploy it to the AWS cloud.
- learn about other DevOps practices such as CI/CD
3.2 A quick introduction to AWS
- AWS (Amazon Web Services) is a cloud platform provider offering over 200 products & services available in data centers all over the world
- you need an AWS account to continue with the rest of the course
π Resources
3.3 AWS S3
- the first AWS service that we will use is AWS S3 which stands for simple storage service
- the website is static and requires no computing power or a database
- we will use AWS S3 to store the public files and serve them over HTTP
- AWS S3 files (which AWS calls objects) are stored in buckets
- the name of the bucket needs to be unique
- the AWS console allows us to manually upload files through the web interface
3.4 AWS CLI
- to interact with the AWS cloud services, we need to use a CLI tool called AWS CLI
- we will use AWS CLI v2 throughout the course
- when using Docker images in pipelines, I highly recommend specifying a tag or at least a major version (if available)
- if you don't specify a tag, at least log the version of every tool you use, as this can help with debugging later on
- example:
aws --version
Pipeline after this lecture
.gitlab-ci.yml
stages:
- build
- test
- deploy
build website:
image: node:16-alpine
stage: build
script:
- yarn install
- yarn lint
- yarn test
- yarn build
artifacts:
paths:
- build
test website:
image: node:16-alpine
stage: test
script:
- yarn global add serve
- apk add curl
- serve -s build &
- sleep 10
- curl http://localhost:3000 | grep "React App"
deploy to s3:
stage: deploy
image:
name: amazon/aws-cli:2.4.11
entrypoint: [""]
script:
- aws --version
π Resources
3.5 Uploading a file to S3
- to upload a file to S3, we will use the copy command
cp
aws s3 cp
allows us to copy a file to and from AWS S3
Pipeline after this lecture
.gitlab-ci.yml
stages:
- build
- test
- deploy
.build website:
image: node:16-alpine
stage: build
script:
- yarn install
- yarn lint
- yarn test
- yarn build
artifacts:
paths:
- build
.test website:
image: node:16-alpine
stage: test
script:
- yarn global add serve
- apk add curl
- serve -s build &
- sleep 10
- curl http://localhost:3000 | grep "React App"
deploy to s3:
stage: deploy
image:
name: amazon/aws-cli:2.4.11
entrypoint: [""]
script:
- aws --version
- echo "Hello S3" > test.txt
- aws s3 cp test.txt s3://YOUR_BUCKET_NAME/test.txt
π Resources
3.6 Masking & protecting variables
- to define a variable, go to
[Settings]
>[CI/CD]
>[Variables]
>[Add variable]
- we typically store passwords or other secrets
- a variable has a key and a value
- it is a good practice to write the key in uppercase and to separate any words with underscores
- flags:
- Protect variable: if enabled, the variable is not available in branches, apart from the default branch (main), which is a protected branch
- Mask variable: if enabled, the variable value is never displayed in clear text in job logs
Pipeline after this lecture
.gitlab-ci.yml
stages:
- build
- test
- deploy
.build website:
image: node:16-alpine
stage: build
script:
- yarn install
- yarn lint
- yarn test
- yarn build
artifacts:
paths:
- build
.test website:
image: node:16-alpine
stage: test
script:
- yarn global add serve
- apk add curl
- serve -s build &
- sleep 10
- curl http://localhost:3000 | grep "React App"
deploy to s3:
stage: deploy
image:
name: amazon/aws-cli:2.4.11
entrypoint: [""]
script:
- aws --version
- echo "Hello S3" > test.txt
- aws s3 cp test.txt s3://$AWS_S3_BUCKET/test.txt
3.7 Identity management with AWS IAM
- we don't want to use our username and password to use AWS services from the CLI (I am not even sure if this is even possible!)
- as we only need access to S3, it makes sense to work with an account with limited permissions
- IAM allows us to manage access to the AWS services
- we will create a new user with the following settings:
- account type: programmatic access
- permissions: attach existing policy: AmazonS3FullAccess
- the account details will be displayed only once
- go to Settings > CI/CD > Variables > Add variable and define the following unprotected variables:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION
- AWS CLI will look for these variables and use them to authenticate
3.8 Uploading multiple files to S3
- using cp to copy individual files can take a lot of space in the pipeline config
- some file names are generated during the build process, and we can't know them in advance
- we will use sync to align the content between the build folder in GitLab and the S3 bucket
Pipeline after this lecture
.gitlab-ci.yml
stages:
- build
- test
- deploy
build website:
image: node:16-alpine
stage: build
script:
- yarn install
- yarn lint
- yarn test
- yarn build
artifacts:
paths:
- build
test website:
image: node:16-alpine
stage: test
script:
- yarn global add serve
- apk add curl
- serve -s build &
- sleep 10
- curl http://localhost:3000 | grep "React App"
deploy to s3:
stage: deploy
image:
name: amazon/aws-cli:2.4.11
entrypoint: [""]
script:
- aws --version
- aws s3 sync build s3://$AWS_S3_BUCKET --delete
π Resources
3.9 Hosting a website on S3
- files in the S3 bucket are not publicly available
- to get the website to work, we need to configure the bucket
- from the bucket, click on Properties and enable Static website hosting
- from the bucket, click on the Permissions tab and disable Block public access
- from the bucket, click on the Permissions tab and set a bucket policy
S3 bucket policy example
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicRead",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*"
}
]
}
3.10 Controlling when jobs run
- we donβt want to deploy to production from every branch
- to decide which jobs to run, we can use
rules:
to set a condition CI_COMMIT_REF_NAME
gives us the current branch that is running the pipelineCI_DEFAULT_BRANCH
gives us the name of the default branch (typically main or master)
Pipeline after this lectur
.gitlab-ci.yml
stages:
- build
- test
- deploy
build website:
image: node:16-alpine
stage: build
script:
- yarn install
- yarn lint
- yarn test
- yarn build
artifacts:
paths:
- build
test website:
image: node:16-alpine
stage: test
script:
- yarn global add serve
- apk add curl
- serve -s build &
- sleep 10
- curl http://localhost:3000 | grep "React App"
deploy to s3:
stage: deploy
image:
name: amazon/aws-cli:2.4.11
entrypoint: [""]
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
script:
- aws --version
- aws s3 sync build s3://$AWS_S3_BUCKET --delete
π Resources
3.11 Post-deployment testing
- we will use
cURL
to download the index.html file from the website - with
grep
, we will check to see if the index.html file contains a specific string
Pipeline after this lecture
.gitlab-ci.yml
stages:
- build
- test
- deploy
- post deploy
variables:
APP_BASE_URL: http://YOUR-BUCKET-NAME.s3-website-YOUR-REGION.amazonaws.com
build website:
image: node:16-alpine
stage: build
script:
- yarn install
- yarn lint
- yarn test
- yarn build
artifacts:
paths:
- build
test website:
image: node:16-alpine
stage: test
script:
- yarn global add serve
- apk add curl
- serve -s build &
- sleep 10
- curl http://localhost:3000 | grep "React App"
deploy to s3:
stage: deploy
image:
name: amazon/aws-cli:2.4.11
entrypoint: [""]
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
script:
- aws --version
- aws s3 sync build s3://$AWS_S3_BUCKET --delete
production tests:
stage: post deploy
image: curlimages/curl
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
script:
- curl $APP_BASE_URL | grep "React App"
3.12 What is CI/CD?
- the pipeline goes through multiple stages: build, test & deploy
- right now, we consider the website hosted at AWS S3 our production environment
- quite often, pipelines will also have a staging environment
- a staging environment is a non-production, usually non-public environment that is very close to the actual production environment
- we often use automation to create these environments and to ensure that they are indeed identical
- we use a staging environment as a pre-production environment
- essentially, we try out our deployment in the pre-production environment
- CD can refer to two concepts:
- Continuous Deployment - where every change is automatically deployed to production
- Continuous Delivery - where every change is automatically deployed to staging but where we need some manual intervention to deploy to production
3.13 Assignment
- create a staging environment and add it to the CI/CD pipeline
3.14 Assignment solution
Pipeline after this lecture
.gitlab-ci.yml
stages:
- build
- test
- deploy staging
- test staging
- deploy production
- test production
variables:
APP_BASE_URL: http://YOUR-BUCKET-NAME.s3-website-YOUR-REGION.amazonaws.com
APP_BASE_URL_STAGING: http://YOUR-BUCKET-NAME-STAGING.s3-website-YOUR-REGION.amazonaws.com
build website:
image: node:16-alpine
stage: build
script:
- yarn install
- yarn lint
- yarn test
- yarn build
artifacts:
paths:
- build
test website:
image: node:16-alpine
stage: test
script:
- yarn global add serve
- apk add curl
- serve -s build &
- sleep 10
- curl http://localhost:3000 | grep "React App"
deploy to staging:
stage: deploy staging
image:
name: amazon/aws-cli:2.4.11
entrypoint: [""]
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
script:
- aws --version
- aws s3 sync build s3://$AWS_S3_BUCKET_STAGING --delete
staging tests:
stage: test staging
image: curlimages/curl
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
script:
- curl $APP_BASE_URL_STAGING | grep "React App"
deploy to production:
stage: deploy production
image:
name: amazon/aws-cli:2.4.11
entrypoint: [""]
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
script:
- aws --version
- aws s3 sync build s3://$AWS_S3_BUCKET --delete
production tests:
stage: test production
image: curlimages/curl
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
script:
- curl $APP_BASE_URL | grep "React App"
3.15 Environments
- every system where we deploy an application is an environment
- typical environments include test, staging & production
- GitLab offers support for environments
- we can define environments in
[Deployments]
>[Environments]
Pipeline after this lecture
.gitlab-ci.yml
stages:
- build
- test
- deploy staging
- deploy production
build website:
image: node:16-alpine
stage: build
script:
- yarn install
- yarn lint
- yarn test
- yarn build
artifacts:
paths:
- build
test website:
image: node:16-alpine
stage: test
script:
- yarn global add serve
- apk add curl
- serve -s build &
- sleep 10
- curl http://localhost:3000 | grep "React App"
deploy to staging:
stage: deploy staging
environment: staging
image:
name: amazon/aws-cli:2.4.11
entrypoint: [""]
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
script:
- aws --version
- aws s3 sync build s3://$AWS_S3_BUCKET --delete
- curl $CI_ENVIRONMENT_URL | grep "React App"
deploy to production:
stage: deploy production
environment: production
image:
name: amazon/aws-cli:2.4.11
entrypoint: [""]
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
script:
- aws --version
- aws s3 sync build s3://$AWS_S3_BUCKET --delete
- curl $CI_ENVIRONMENT_URL | grep "React App"
π Resources
3.16 Reusing configuration
- sometimes, multiple jobs may look almost the same, and we should try to avoid repeating ourselves
- GitLab allows us to inherit from an exiting job with the
extends:
keyword
Pipeline after this lecture
.gitlab-ci.yml
stages:
- build
- test
- deploy staging
- deploy production
build website:
image: node:16-alpine
stage: build
script:
- yarn install
- yarn lint
- yarn test
- yarn build
artifacts:
paths:
- build
test website:
image: node:16-alpine
stage: test
script:
- yarn global add serve
- apk add curl
- serve -s build &
- sleep 10
- curl http://localhost:3000 | grep "React App"
.deploy:
image:
name: amazon/aws-cli:2.4.11
entrypoint: [""]
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
script:
- aws --version
- aws s3 sync build s3://$AWS_S3_BUCKET --delete
- curl $CI_ENVIRONMENT_URL | grep "React App"
deploy to staging:
stage: deploy staging
environment: staging
extends: .deploy
deploy to production:
stage: deploy production
environment: production
extends: .deploy
3.17 Assignment
- the goal of this assignment is to expand the post-deployment tests to ensure that the correct version has been deployed
- add a file named
version.html
which contains the current build number - the current build number is given by a predefined GitLab CI variable named
CI_PIPELINE_IID
π Resources
3.18 Assignment solution
Pipeline after this lecture
.gitlab-ci.yml
stages:
- build
- test
- deploy staging
- deploy production
variables:
APP_VERSION: $CI_PIPELINE_IID
build website:
image: node:16-alpine
stage: build
script:
- yarn install
- yarn lint
- yarn test
- yarn build
- echo $APP_VERSION > build/version.html
artifacts:
paths:
- build
test website:
image: node:16-alpine
stage: test
script:
- yarn global add serve
- apk add curl
- serve -s build &
- sleep 10
- curl http://localhost:3000 | grep "React App"
.deploy:
image:
name: amazon/aws-cli:2.4.11
entrypoint: [""]
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
script:
- aws --version
- aws s3 sync build s3://$AWS_S3_BUCKET --delete
- curl $CI_ENVIRONMENT_URL | grep "React App"
- curl $CI_ENVIRONMENT_URL/version.html | grep $APP_VERSION
deploy to staging:
stage: deploy staging
environment: staging
extends: .deploy
deploy to production:
stage: deploy production
environment: production
extends: .deploy
3.19 Continuous Delivery pipeline
- adding
when: manual
allows us to manually trigger the production deployment
Pipeline after this lecture
.gitlab-ci.yml
stages:
- build
- test
- deploy staging
- deploy production
variables:
APP_VERSION: $CI_PIPELINE_IID
build website:
image: node:16-alpine
stage: build
script:
- yarn install
- yarn lint
- yarn test
- yarn build
- echo $APP_VERSION > build/version.html
artifacts:
paths:
- build
test website:
image: node:16-alpine
stage: test
script:
- yarn global add serve
- apk add curl
- serve -s build &
- sleep 10
- curl http://localhost:3000 | grep "React App"
.deploy:
image:
name: amazon/aws-cli:2.4.11
entrypoint: [""]
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
script:
- aws --version
- aws s3 sync build s3://$AWS_S3_BUCKET --delete
- curl $CI_ENVIRONMENT_URL | grep "React App"
- curl $CI_ENVIRONMENT_URL/version.html | grep $APP_VERSION
deploy to staging:
stage: deploy staging
environment: staging
extends: .deploy
deploy to production:
stage: deploy production
when: manual
environment: production
extends: .deploy