TL;DR: Automatic updates of the certificate are definately possible but some setup is required and you have to be careful how you manage the access rights to the runner

Introduction

While this humble blog does not take any user data or other tracking I was still eager to secure it with https encryption. Browsers and search engines give more and more warnings and penalties to websites that are not offering https encryption. Since this page is a hobby and I want to spend as little money as possible I deceided to go with a free ssl certificate provider. I bet there are more out there but I went with Certbot from Let’s Encrypt. Certbot generates free ssl certificates for as many domains as you want. But there is one catch! The certificates expire after 3 months. Let’s Encrypt is doing this to ensure a more secure web. Long living certificates compromise the safety. If the private key of the certificate leaks there could be a lot of web pages that are affected. Since 3 months is to short for most people to manually update their certificates people tend to invest time in automation. Automation is key when certificate private keys leak. In a perfect world everybody could just press his “Update certificate” button and we are back to a secure https web. In reality a lot of certificates are updated manually and this makes them static and vulnerable. Read more about the short certificate lifetime HERE. I run this page on Gitlab Pages so I don’t have access to the servers hosting this web page. In most cases you can install certbot onto your server and it can replace the certificates. Thankfully Gitlab offers the possiblity to add SSL certificates for domains through the Gitlab settings and the Gitlab API. In this blog post I want to show how I automated the SSL certificate renewal for my Gitlab Pages.

Prequisits

This post will be more about the code to automate the procedure and not so much about how everything is working. You should have:

• A running GitLab pages project
• Verified custom domains in the Gitlab Pages settings ( If you are running with the default gitlab.io domain you don’t need an own ssl certificate )
• I would recommend running through the certificate generation process once just to get to know what the automation does. Read this Guide
• So in theory your page should already be running on https but you want to automate it

Source

You can find the source code HERE. Please read the LICENSE before using.

Steps

I will break the process up into multiple steps. I’ll give the overview here and then go into detail below.

1. Run Gitlab pipeline job automatically every 60 days to renew the certificate
2. Request new certificate for domains from Let's encrypt through certbot
3. Modify web page to succeed at the http challenge
4. Upload new certificate to Gitlab Pages Settings

All of these steps are execute by python scripts automatically given the correct parameters.

Scripts

This is the certbot job from the gitlab-ci.yml file from this website repository. The certbot job is marked to run only with schedules. The schedule is configured to run every 60 days (cron syntax: 0 0 1 */2 *). I also recommend to run this operation on a private runner. Remember that this generates the public and private key for your certificate. Shared runners on Gitlab are good but not waterproof when it comes to data security. Anybody getting access to the shared runner could steal your private key. Your private runner does not need much performance. It could be a Raspberry Pi or similar.

certbot:
  only:
    - schedules
  script:
    # Move repo content to own directory
    - shopt -s extglob dotglob
    - mkdir rootPackage
    - echo !(rootpackage)
    - mv !(rootPackage) rootPackage
    - BASE_URL=`echo $CI_REPOSITORY_URL | sed "s;\/*$CI_PROJECT_PATH.*;;"`
    - REPO_URL="${BASE_URL}/robinryf/gitlab-ci.git"
    - git clone ${REPO_URL} gitlab-ci
    # Starting certificate renew process
    - python3 gitlab-ci/certbot/gitlabRenew.py certbot rootPackage/sslChallenges gitlab-ci/certbot/template.html site@robinryf.com 23xxxx5 --domain robinryf.com --domain www.robinryf.com
  tags:
    - <privateRunnerName>

When the above script section is executed I want to checkout my gitlab-ci repository to use it. Since I don’t want to clone this repository into the website project I first move all of the website files into a directory called rootPackage. I just add one directory layer more.

The most important line is executing the gitlabRenew.py script. Usage: python3 gitlabRenew.py <workingDirectory> <targetSiteDirectory> <templateFile> <siteEmail> <gitlabProjectID> --domain <domain> --domain <optionalSecondDomain> Check the script gitlabRenew.py for more details on the parameters.

Request new certificate for domain(s)

DISCLAIMER: When using the script the options --agree-tos and --manual-public-ip-logging-ok are passed to certbot. By executing with these options you agree to the terms of service from Let's encrypt and agree that the public IP adress of the runner is logged by Let's encrypt. Read more HERE

Certbot will be executed in the certonly mode. Normally certbot runs on the webserver itself and has presets to update the certificate (e.g. Apache Server). We are not running on the machine which also hosts the certificate, we are running on some runner. The certonly mode gives us the certificates as files for us to further process.

To verify that you are the owner of the webpage certboot offers a few methods to prove it. I use the http-challenge method. Basically certbot asks you to upload a random string to a certain url. You do this and tell certbot you did it. Certbot now requests that specific url and if the response matches you have proven that you control the webpage and are the owner. In our case we have to modify the webpage while the Gitlab job of certbot is running.

If you are trying around different things please keep in mind that Let's encrypt sets a limit on how many certificates you can request peer week. There is the possiblity to use a Staging Environment but I did not include it in my initial script version.

Authentication

An example challenge would be that certbot expects the response wN8OYFL-rAHIdx33U4qZPvfqBJq9Js_nKxYDumY4-do when requesting via the url http://robinryf.com/.well-known/acme-challenge/JiQ9b4dBeE_DDuhYvL_Q. The code comes with a template file that should work for Jekyll but you can also use a custom template file if required. The template file looks like this:

---
layout: null
permalink: "/.well-known/acme-challenge/%TOKEN%/index.html"
---

%VALIDATION%

In my Jekyll setup this will add a page at the mentioned url above. %TOKEN% and %VALIDATION are replaced with the correct tokens given by certbot.

The script will create pages for the challenges in the <targetSiteDirectory> path given to the script at execution. Then the script pushes the changed files to Gitlab which causes the automatic pipeline job to start. This pipeline job updates the web page with the required challenge information and certbot is happy. If you have multiple domains there will also be multiple pushes to the repository.

Gitlab Pages settings update

If everything worked out we should now have a private key and a public key stored on the runner which represent our ssl certificate. This information has to be sent to Gitlab now. Using the Gitlab API the SSL information is uploaded. I use this API ENDPOINT.

Runner configuration

Since the runner must be able to push to the repository and change repository settings we have to configure it to have the correct rights. This is a problematic topic. Especially when running on shared runners you have to be careful since everybody optaining the information the runner has can mess around with your repositories. I use Personal Access Tokens which gets injected to the runner via CI-Variables. To keep the security risk as low as possible I took some measures:

• I only run this job on private runners which I fully control.
• I didn’t want to create a private token for my main Gitlab Account. Somebody aquiring a private token could mess with all my repositories. I created a seperate Gitlab account for my private runner. This Gitlab user only has access to the repositories which he needs. This way it is easier to control which repositories are at risk.

If you took the security measures which seem applicable to you then create a environment variable called PRIVATE_TOKEN in the GitLab Runner Environment settings. This variable should be filled with the private token.

Conclusion

Automatically renewing the SSL certificate on GitLab pages takes some work and time. But in the end it works very well and I don’t have to do anything manually anymore. Of course, you could also pay a hosting provider which does this renewing for you or buy a long living certificate.

I hope you enjoyed this summary and please use the code at your own risk.