You what can be intimidating? Doing things you’ve never done before. I know that growth happens when you extend yourself and go beyond your comfort zone, but that doesn’t make it any easier to stare into the void of the unknown.

In my case, I have been hesitant to touch an EC2 instance I have that’s running a web app I worked on a while back. Why? Well, it was my first deployment and I only vaguely have an idea of why things worked (even though much of the setup was done manually).

I launched the app in early March, and enabled SSL a few days later. If I’m going to be honest, I thought I made a document detailing how I enabled SSL, but when looking over it yesterday, I found myself scratching my head.

My SSL certificate issued from LetsEncrypt was set to expire this upcoming weekend, so like a good procrastinator, I thought I’d open up the black screen of my ec2 instance a few days before. In my defense, I’d done some research months ago to document renewal and automation, so I wasn’t starting from complete 0.

For some reference on my setup, to install certs my first time, I created a standalone docker container that downloaded and ran a certbot image. I then made some changes to my production default.conf files to incorporate the new certs and enable a https connection. I then restarted my nginx web server and my site went from http to https! (the setup I followed will be linked at the end)

I thought that to update my certificate, I would generally have to:

• re-run the certbot image to update the certs
• rebuild the nginx docker container in the background to pick up the changes
• use a tool to test/verify the new expiration date

Running the certbot docker yaml file did not yield the results I was expecting.

certbot-service  | Challenge failed for domain somerandomquotes.com
certbot-service  | http-01 challenge for somerandomquotes.com
certbot-service  | Notifying user: 
certbot-service  | Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
certbot-service  |   Domain: somerandomquotes.com
certbot-service  |   Type:   unauthorized
certbot-service  |   Detail: 54.85.4.220: Invalid response from http://somerandomquotes.com/.well-known/acme-challenge/hQgnMf3rLv_TbiUJ160puK_FiAOfMBApzpiuk2m3N9I: 404
certbot-service  | 
certbot-service  | Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

There are some good hints in there, I’m sure, but my first instinct was to re-run the script with a -v flag for more verbose output messages. It did help and it didn’t help. More information is good, if you know what to do with it. In this case, I had more to potentially Google and figure out.

I spent a couple of hours searching around, seeing if problems that other people faced seemed similar to my own. I tried out a few solutions like updating my nginx config (link), but nothing seemed to really work.

After eventually restarting my ec2 instance and re-running a dry-run certbot renew command, I was able to pass validation, but I have no idea what was the thing (or set of things) I did to make it happen.

I’m not sure what exactly did it, but here are things I did:

• added additional listeners for ipv6 addresses in my prod default.conf file (credit), probably what did it
• including a listener for /.well-known/acme-challenge/ in the ssl section of the default.conf file (which I don’t think actually helped)
• rebuilt my container images (by re-running docker-compose prod file)
• stop and restart the containers (with the newer images)
• restart the ec2 instance
• follow the certbot documentation for renewal (link)

With this, I was able to update my ssl cert and after checking, I won’t have to “worry” about this until September!

new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/somerandomquotes.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Requested authenticator webroot and installer None
Notifying user: 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Notifying user: Congratulations, all renewals succeeded: 
Congratulations, all renewals succeeded: 
Notifying user:   /etc/letsencrypt/live/somerandomquotes.com/fullchain.pem (success)
  /etc/letsencrypt/live/somerandomquotes.com/fullchain.pem (success)
Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
no renewal failures

### command to get expiration date: 
sudo openssl x509 -enddate -noout -in fullchain.pem
notAfter=Sep 10 21:08:50 2023 GMT

Ultimately, I still want to automate this process by configuring a cron job to obtain new certs and modify my nginx docker service to pick them up, but I’ll take this first step of making sure that my ssl didn’t expire!

Some takeaways

• new things are scary, but do them anyways! (beginner’s mind is 👍)
• docker is powerful, but like all tech, has a somewhat steep learning curve
• for my next project, I probably want a bigger ec2 instance (the one I have is quite cheap, but small and most importantly, slow)

Resources

• enabling https with letsencrypt on docker and amazon ec2: link
• certbot documentation: link