HAProxy and Certbot

Combine HAProxy and Certbot on the same container host allowing Certbot to keep certificates updated and HAProxy to proxy all traffic. HAProxy will redirect insecure requests to secure ports except for Certbot’s proof-of-ownership requests. Those are proxied to Certbot.

This example uses Docker.

Deployment

Test Resources

Let’s create a private docker network for our containers to use. That will let them connect dynamically to each other by container name so we can avoid hard-coding IPs.

docker network create -d bridge proxy-nw

It’s also useful to have a local web server to test against. Throw up a darkhttpd web server (a very simple web server) on the same container host, redirecting a high-level port on the container host to it’s port 80.

mkdir /tmp/www
echo "Look ma, http" > /tmp/www/index.html

docker run --name darkhttpd --net=proxy-nw --detach --volume /tmp/www:/www --publish 8008:80 shelmangroup/darkhttpd

curl localhost:8008

If everything went as planned you should get the expected output and docker logs darkhttpd will show the web request.

HAProxy

The creators of HAProxy suggest avoiding local dependencies by adding config and certificate files to the official image to create a custom image. For our tests however, we’ll start with a local config as we need it to work with a local Certbot instance.

The first step is to create volumes to hold the local configs; one for HAProxy and one for Certbot. The docker image provided by HAProxy defaults to /usr/local/etc/haproxy/haproxy.cfg. Certbot defaults to /etc/letsencrypt. The cert itself is in a subdirectory under ’live’ named for the site, such as live/server.your.org, but you need all of the subdirectories to persist or renewals won’t work right. We’ll present to the containers in the run command below.

docker volume create haproxy
docker volume create certbot

# Check the location just in case
docker volume inspect haproxy

Next, create a config file for HAProxy to specifies the darkhttpd server as a back-end server. Notice that darkhttpd itself is listening on port 80. Port 8008 above is the translated-port we published on the container host for testing and not what you want HAProxy to connect to.

sudo vi /var/lib/docker/volumes/haproxy/_data/haproxy.cfg

global
    maxconn 4000
    tune.ssl.default-dh-param 2048
 
frontend http
    bind *:80
    mode http
 
    default_backend http-server
    timeout client 1h
 
backend http-server
    mode http
    server darkhttpd darkhttpd:80
    timeout connect 1h
    timeout server 1h

Now lets run the HAProxy image.

docker run \
--detach \
--mount type=volume,source=haproxy,target=/usr/local/etc/haproxy,readonly \
--mount type=volume,source=certbot,target=/etc/letsencrypt/,readonly \
--name haproxy \
--net=proxy-nw \
--publish 80:80 \
--publish 443:443 \
--restart always \
haproxy

curl localhost

If all went well, you can now access the host on port 80 and hit the darkhttpd server. Importantly, you must be able to hit this from the Internet, as Let’s Encrypt will be hitting it to prove you own the host in question. Issue a docker ps and docker logs haproxy if something failed. Once you’re done testing, stop the containers and remove the unneeded darkhttpd server.

docker stop haproxy
docker stop darkhttpd; docker rm darkhttpd; docker rmi shelmangroup/darkhttpd

Certbot

The official image is designed to run-and-exit, requiring the container OS to have a cron job to manage renewals. Our goal is for Certbot manage the certificate without external dependencies, and to be protected from general internet traffic by HAProxy. So let’s create our own image based on theirs.

Docker Image Set-Up

# Make a directory
mkdir ~/docker-certbot
cd ~/docker-certbot

# Create the Docker file and a script that will renew the cert
touch Dockerfile
touch cert_request
chmod +x cert_request

Dockerfile

Create the following Docker file. It uses Certbot’s image as a base, places a cert request script and launches the cron daemon in the foreground (crond doesn’t run by default). With crond running this way you can examine the output easily with a docker log command.

vi Dockerfile

FROM certbot/certbot
COPY cert_request /etc/periodic/daily
ENTRYPOINT ["crond", "-f"]

Renewal Script

Below is the content of the script. The COPY above puts it in a location where crond will run it once a day. Let’s Encrypt actually recommends twice a day but this is close enough. We also concatenate the public and private key into a single file as required by HAProxy.

The reason we’re going to all this trouble is that issued certs are valid for 90 days. You have to check often.

TODO We could probably do something slick here and not recreate that file if we spent some time thinking about it. We could also use –post-hook to tell haproxy something has changed or at least see if that’s needed.

#!/bin/sh

HOST=server.your.org 
EMAIL=[email protected]

certbot certonly $1 --standalone -d $HOST --non-interactive --preferred-challenges http --agree-tos --email $EMAIL

cat /etc/letsencrypt/live/$HOST/fullchain.pem /etc/letsencrypt/live/$HOST/privkey.pem > /etc/letsencrypt/live/$HOST/$HOST.pem

Building and Testing the Image

Let’s build the image and publish port 80. We won’t want that published later on but it’s useful for a first test so we know Certbot itself doesn’t have any issues.

docker build -t allen/certbot .

docker run --name certbot --detach --publish 80:80 --mount type=volume,source=certbot,target=/etc/letsencrypt allen/certbot

Let’s run the first request manually against Let’s Encrypt’s test system to make sure it works as expected.

# Attach to the running container
docker exec -it certbot sh

# Execute the request script aginst let's encrypt's test system
/etc/periodic/daily/cert_request --staging

You should see output similar to:

Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/server.your.org/fullchain.pem

If not, start digging into if your web server is reachable or the host name is wrong. Importantly, add –staging to the request while you are debugging. The production Let’s Encrypt service is rate-limited.

Integrating HAProxy and Certbot

Now that you know Certbot works, recreate it without an external port (this is easier than a restart). We’ll use HAProxy to route requests to it. Note: now that it has a certificate, certbot will execute a renew operation which no longer requires the verification of accepting port 80 challenges. The private key it already has is used. But you still want to catch requests as any new certs you add later will require the verification.

docker stop certbot
docker rm certbot
docker run --name certbot --detach --net=proxy-nw --mount type=volume,source=certbot,target=/etc/letsencrypt --restart always allen/certbot

The HAProxy config file needs an update, both to handle reqeusts to Certbot and to take advantage of the certificate we got earlier.

sudo vi /var/lib/docker/volumes/haproxy/_data/haproxy.cfg

global
    tune.ssl.default-dh-param 2048

defaults
    timeout client 1h
    timeout connect 1h
    timeout server 1h

frontend http
    bind *:80
    mode http

    acl certbot-acl path_beg /.well-known/acme-challenge/
    http-request redirect scheme https if ! certbot-acl
    use_backend certbot-backend if certbot-acl

frontend https
    mode http
    bind *:443 ssl crt /etc/letsencrypt/live/wiki.your.org/wiki.your.org.pem

backend certbot-backend
    mode http
    server certbot certbot:80




docker stop haproxy

docker start haproxy

Let’s do some basic tests

# Test that non-secure requests are redirected with a 302

curl -D -  http://server.your.org/xyz

  HTTP/1.1 302 Found
  Cache-Control: no-cache
  Content-length: 0
  Location: https://wiki.your.org/xyz

# Test that non-secure requests to acme-challenge are not

curl http://server.your.org/.well-known/acme-challenge/

  <html><body><h1>503 Service Unavailable</h1>
  No server is available to handle this request.
  </body></html>

# Take a look at the certificate we have

openssl s_client -showcerts -connect localhost:443

  ...
  ...
  Verify return code: 20 (unable to get local issuer certificate)

These are as expected. Certbot isn’t isn’t in the middle of a request so nothing is listening. The cert displayed is from the testing server and so untrusted.

Now, let’s try requesting a cert from the production server and assuming it shows you success, restart HAProxy to use the new cert.

docker exec -it certbot sh

/etc/periodic/daily/cert_request --force-renew

exit

docker stop haproxy

docker start haproxy

openssl s_client -showcerts -connect localhost:443

  ...
  ...
  Verify return code: 0 (ok)

Now that you have a working proxy, you’ll probably want to proxy some things other than certbot. Let’s add an example xwiki running on it’s own network.

docker network connect xwiki-nw haproxy

# To the haproxy conf file, add 

...
...
frontend https
    mode http
    bind *:443 ssl crt /etc/letsencrypt/live/wiki.your.org/wiki.your.org.pem

    acl is_wiki hdr(host) -i wiki.your.org
    use_backend xwiki if is_wiki

backend xwiki
    mode http
    server xwiki xwiki:8080
...
...

docker stop haproxy
docker start haproxy

And enjoy!

Troubleshooting

Container exits immediately

Check the logs with docker logs haproxy to see what’s up.

‘server certbot’ : could not resolve address ‘certbot’…Failed to initialize server(s) addr.

HAProxy wont start if it can’t connect to the back-end server. Adding the init-addr option accomodates that.

cat: can’t open ‘/etc/letsencrypt/live/server.your.org/fullchain.pem**

Certbot can start saving certs under -0001 and such when you start requesting certs that it already knows about but can’t find. You can delete the /etc/letsencrypt/* dir

Moving a Custom Container

docker save allen/certbot >test.tar scp some.server:test.tar . docker load -i test.tar

To Remove old containers and images docker ps -a docker rm … docker images docker rmi … docker system prune –all docker image prune –all

Other Resources

https://www.lab-time.it/2018/09/20/running-haproxy-and-lets-encrypt-on-docker/ https://docs.docker.com/storage/volumes/ https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/ https://certbot.eff.org/docs/install.html https://wiki.alpinelinux.org/wiki/Alpine_Linux:FAQ#My_cron_jobs_don.27t_run.3F https://gist.github.com/andyshinn/3ae01fa13cb64c9d36e7

Feedback

Was this page helpful?

Glad to hear it! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.

Last modified April 14, 2026: Old site imports (677647f)