Automating Static Website Deployment, Part 1

by AlphaGeek

Sun, Jun 11, 2017

Now that I have 8 static websites deployed into AWS using S3 and CloudFront I need to automate the deployment process so that I can make changes quickly and get them to production with less effort. To that end I have developed some scripts and configurations that automate the deployment of this site to a QA environment and production when changes are pushed to either branch. I will be describing this work in probably 3 parts. This is the first.

Note:

I describe creating a Hugo Website but you can probably drop in any static site generator.

Building a Hugo Website

A Hugo web site is extremely easy to start. First create a new hugo site:

$ hugo new site test.computersfearme.com

One thing you will almost certainly need is a theme. I use a git submodule link to import the theme. First you’ll need to initalize the git repo:

$ cd test.computersfearme.com
$ git init .

Next, you’ll need to add the submodule you want. I will use the hugo-bootstrap-premium theme:

$ mkdir -p themes
$ cd themes
$ git submodule add "https://github.com/appernetic/hugo-bootstrap-premium"
$ cd ..

Now you need to edit your config.toml:

baseURL = "http://test.computersfearme.com/"
languageCode = "en-us"
title = "CFM Test Site"
theme = "hugo-bootstrap-premium"

After that you need to start creating content. For instance:

$ hugo new post/hello-world.md

This will create a new file in the content directory named “post/hello-world.md” with some boilerplate front matter.

Please see Hugo’s Site for details on building and maintaining a website using hugo. At this point we have a small but functional website using a theme. Next we will look at how to build and deploy this website to AWS automatically after each push.

Continuous Delivery

We are going to develop a set of scripts and configuration files that will allow the site to be continously deployed to an appropriate environment based on the branch changes were made to. In this example we will use GitFlow to manage our code.

$ git flow init

NOTE: Be sure to have installed git-flow first.

If you took the defaults when prompted, you will be on a branch named “develop”. This will be our “QA” branch. When we push changes to this branch the system will automatically deliver our generated site to our QA environment.

Creating the Envronments

We have to have someplace to push this site. We are going to use S3 and Cloudfront to deploy our static website. Please see Hosting a Static Website for Pennies a Month for instructions on how to set this up. For now I am going to assume that we have already done this. That would mean we have a buckets named origin.qa.test.computersfearme.com for QA and origin.test.computersfearme.com for Production created and CloudFront distributions created that serves up content for qa.test.computersfearme.com and test.computersfearme.com.

Creating the Code Repository

Now we need to create the code repo in AWS CodeCommit.

$ aws codecommit create-repository \
    --repository-name test.computersfearme.com \
    --repository-description "A test website"

The output will look like this:

{
    "repositoryMetadata": {
        "repositoryName": "test.computersfearme.com",
        "cloneUrlSsh": "ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/test.computersfearme.com",
        "lastModifiedDate": 1497199943.645,
        "repositoryDescription": "A test website",
        "cloneUrlHttp": "https://git-codecommit.us-east-1.amazonaws.com/v1/repos/test.computersfearme.com",
        "creationDate": 1497199943.645,
        "repositoryId": "9663fef8-f28f-4806-aa09-e8218a774f7c",
        "Arn": "arn:aws:codecommit:us-east-1:############:test.computersfearme.com",
        "accountId": "############"
    }
}

Now set the remote and push the current code to it:

$ git remote add origin "https://git-codecommit.us-east-1.amazonaws.com/v1/repos/test.computersfearme.com"
$ git add .
$ git commit -m "initial checking of test.computersfearme.com"
$ git push --set-upstream origin develop

Now we can set the default branch so that new clones will use develop automatically:

$ aws codecommit update-default-branch --repository-name test.computersfearme.com \
    --default-branch-name develop

Building the Website

To build the website, we need to run hugo and package the public site into a tarball.

Here is a script that will do that. We will put it in a subdirectory named ‘website-tools’ and call it ‘build.sh’.

#!/bin/bash

usage() { 
  echo "usage: $0 -b <basedir> [-v <version>] [-d <domain-name>] [-D]" 1>&2
  echo "  -b Specifies the base directory to build in." 1>&2
  echo "  -v The version tag to put on the tarball. Defaults to 'local'" 1>&2
  echo "  -d The base domain for the site. Default uses baseURL from config." 1>&2
  echo "  -D Turn on drafts." 1>&2
  exit -1
}

HUGO_OPT=''
VERSION=local
DIR=$(pwd)
while getopts ":b:v:d:D" o; do
  case "${o}" in
    b)
      DIR=${OPTARG}
      ;;
    v)
      VERSION=${OPTARG}
      ;;
    d)
      HUGO_OPT="--baseURL http://${OPTARG}/ ${HUGO_OPT}"
      ;;
    D)
      HUGO_OPT="--buildDrafts ${HUGO_OPT}"
      ;;
    *)
      usage
      ;;
  esac
done
shift $((OPTIND-1))

cd "${DIR}" || exit -2

hugoOutput=$(mktemp /tmp/hugoOutput-XXXXXX)
hugo $HUGO_OPT 2>&1 | tee -a "${hugoOutput}"
ERRCOUNT=$(grep -c "^ERROR:" "${hugoOutput}")
if (( ERRCOUNT == 0 )); then
  cd "${DIR}/public" || exit -3
  tar czf "${DIR}/public-${VERSION}.tar.gz" -- *
  exit $?
else 
  echo "$ERRCOUNT errors in hugo output" 1>&2
  exit "${ERRCOUNT}"
fi

This script does 3 things:

  1. It builds the website using hugo
  2. It looks for lines beginning with “ERROR:” in the output and fails if one is encountered.
  3. It tars the “public/” folder into a tarball with the name public-.tar.gz where the is either ’local’ or a value passed into the script.

If we want drafts we can pass ‘-D’ on the command line for this script. The ‘-d’ option lets us override the domain name of the site so that links go to the correct domain name. Hugo will generate full URLs based on the “baseURL” setting in the config file. So overriding this for QA will prevent QA from having links into Production.

Presumably we want drafts in QA and not in Production so the command lines for our build in QA would be:

$ ./website-tools/build.sh . -D -d qa.test.computersfearme.com -v develop-1234567

And Prduction will be:

$ ./website-tools/build.sh . -v master-1234567

Deploying the Website

To deploy the website we need to copy the contents of the “public/” directory generated in the build step to the origin bucket in S3. After that is complete, we need to issue an invalidation request for the cloudfront distribution.

Here is a script that does that. We will put it in ./website-tools as well and call it deploy.sh.

#!/bin/bash

usage() {
  echo "usage: deploy.sh -h <host> [-b <basedir>] [-d <distributionId>]"
  exit -1
}

DIR=$(pwd)/public
while getopts ":b:h:d:" o; do
  case "${o}" in 
    h)
      HOST=${OPTARG}
      ;;
    b)
      DIR=${OPTARG}
      ;;
    d)
      DISTRIBUTION_ID=${OPTARG}
      ;;
    *)
      usage
      ;;
  esac
done

if [ -z "${HOST}" ]; then
  usage
fi

if ! aws s3 sync "${DIR}" "s3://${HOST}/" --acl 'public-read'; then
  echo "ERROR attempting to push content to origin"
  exit -2
fi

# if the distribution id is provided, then it is fronted by a CloudFront distribution.
# Issue the invalidation.
if [[ "${DISTRIBUTION_ID}" != "" ]]; then
  read -r -d '' JSON <<JSON
{
  "Paths": {
    "Quantity": 3,
    "Items": ["/*","/index.html","/"]
  },
  "CallerReference": "invalidate-all-$(TZ=GMT date "+%Y%m%dT%H%M%S")"  
}
JSON
  INVALIDATION_ID=$(aws cloudfront create-invalidation \
      --distribution-id "${DISTRIBUTION_ID}" \
      --invalidation-batch "${JSON}" \
      | jq -r .Invalidation.Id)
  LOOPS=0
  while : ; do
    sleep 10
    LOOPS=$((LOOPS + 1))
    INVALIDATION_STATUS=$(aws cloudfront get-invalidation \
        --distribution-id "${DISTRIBUTION_ID}" \
        --id "${INVALIDATION_ID}" \
        | jq -r .Invalidation.Status)
    echo -n "."
    if [ "$INVALIDATION_STATUS" == "Completed" ] || ((LOOPS == 30)); then
      break;
    fi
  done
  echo 
  if [ "$INVALIDATION_STATUS" != "Completed" ]; then
    echo "ERROR: Invalidation status check timed out. (300 seconds)"
    exit -3
  else 
    echo "Invalidation Complete."
  fi
fi

This script does 3 things:

  1. Uses aws s3 sync to transfer the content of public/ to the origin bucket.
  2. Issues an invalidation to the passed distribution id.
  3. Waits for that invalidation to move to a “Completed” state or upto 5 minutes which ever is shorter.

So now, to deploy our current code to QA we issue the following command:

$ ./website-tools/deploy.sh -h qa.test.computersfearme.com -d E3ODFZSADK1LN8

NOTE: Your distribution id will, of course, be different.

and Production will look like this:

$ ./website-tools/deploy.sh -h test.computersfearme.com -d E3EHZFMLC8IG8T

Coming Up

In Part 2, I will discuss the process of creating the AWS CodeBuild buildspec.yml including how to detect what branch you are building so that you can deploy to the correct environment.