This is the multi-page printable view of this section. Click here to print.
Servers
- 1: Caddy
- 1.1: Install
- 1.2: Configure
- 1.3: Wildcard Sites
- 1.4: Log
- 1.5: Secure
- 1.6: WebDAV
- 1.7: MFA
- 1.8: Cloudflare
- 2: HAProxy
- 2.1: Basic Auth
- 2.2: HAProxy and Certbot
- 2.3: VHost
- 3: lighttpd
- 4: Log File Perms
- 5: nginx
- 5.1: Let's Encrypt Proxy
- 5.2: WebDAV on Alpine
- 6: WebDAV
1 - Caddy
Caddy is a web server that runs SSL by default by automatically grabing a cert from Let’s Encrypt. It comes as a stand-alone binary, written in Go, and makes a decent reverse proxy.
1.1 - Install
Installation
Caddy recommends “using our official package for your distro” and for debian flavors they include the basic instructions you’d expect. I recommend the testing build for the short-term there’s an bug with loging wild-cards sites (like *.you.org) that are handy to have available.
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/testing/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-testing-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/testing/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-testing.list
chmod o+r /usr/share/keyrings/caddy-testing-archive-keyring.gpg
chmod o+r /etc/apt/sources.list.d/caddy-testing.list
sudo apt update
sudo apt install caddy
System Alternatives
If you ever add a module to caddy, chances are it will be lost when you next apt upgrade. This is because caddy phones home, requests a build for any non-standard module, and replaces itself. The devs don’t think this will be an issue. In any event, you should prepare for it, because you will be adding modules.
Because we’ve diverted, apt upgrade will not upgrade caddy. You must use caddy upgrade instead. Since this stops unattended-updates from any automation of vulnerability patching, maybe add a cron job if you like.
sudo dpkg-divert --divert /usr/bin/caddy.default --rename /usr/bin/caddy
sudo mv ./caddy /usr/bin/caddy.custom
sudo update-alternatives --install /usr/bin/caddy caddy /usr/bin/caddy.default 10
sudo update-alternatives --install /usr/bin/caddy caddy /usr/bin/caddy.custom 50
sudo systemctl restart caddy
1.2 - Configure
Configure
The easiest way to configure Caddy is by editing the Caddyfile
sudo vi /etc/caddy/Caddyfile
Sites
You define websites with a block that includes a root and the file_server directive. Once you reload, and assuming you already have the DNS in place, Caddy will reach out to Let’s Encrypt, acquire a certificate, and automatically redirect from port 80 to 443
site.your.org {
root * /var/www/site.your.org
file_server
}
Authentication
You can add basic auth to a site by creating a hash and adding a directive to the site.
caddy hash-password
site.your.org {
root * /var/www/site.your.org
file_server
basic_auth {
allen SomeBigLongStringFromTheCaddyHashPasswordCommand
}
}
Reverse Proxy
Caddy also makes a decent reverse proxy.
site.your.org {
reverse_proxy * http://some.server.lan:8080
}
You can also take advantage of path-based reverse proxy. Note the rewrite to accommodate the trailing-slash potentially missing.
site.your.org {
rewrite /audiobooks /audiobooks/
handle_path /audiobooks/* {
uri strip_prefix /audiobooks/
reverse_proxy * http://some.server.lan:8080
}
}
Import
You can define common elements at the top (snippets) or in files and import them multiple times to save duplication. This helps when you have many sites.
# At the top in the global section of your Caddyfile
(logging) {
log {
output file /var/log/caddy/access.log
}
}
site.your.org {
import logging
reverse_proxy * http://some.server.lan:8080
}
Modules
Caddy is a single binary so when adding a new module (aka feature) you are actually downloading a new binary that has it compiled in. You can find the list of packages at their download page.
Do this at the command line with caddy itself.
sudo caddy add-package github.com/mholt/caddy-webdav
systemctl restart caddy
Now you can add the webdav directive to a site. You can have the same both serve up a directory listing and handle webdav with a litte extra config. The new module uses different ‘verbs’ when working, allowing it to cooexist as long as file_server gets the first shot.
# Global Option Block
{
order webdav last
}
site.your.org {
root * /var/www/site.your.org
file_server browse
webdav *
}
Troubleshooting
You can test your config file and look at the logs like so
caddy validate --config /etc/caddy/Caddyfile
journalctl --no-pager -u caddy
1.3 - Wildcard Sites
Caddy has an individual cert for every virtual host you create. This is fine, but Let’s Encrypt publishes these as part of certificate transparency and the bad guys are watching. If you create a new site in caddy, you’ll see bots probing for weaknesses within 30 min - without you even having published the URL. There’s no security in anonymity, but need-to-know suggests we shouldn’t be informing the whole world about sites of limited scope.
One solution is a wildcard cert. It’s published as just ‘*.some.org’ so there’s no information disclosed. These are usally done with DNS integration. Caddy supports this and it requires a little extra work, but it does make it a lot easier to sort out the riff-raff later.
Installation
We’ll connect to CloudFlare’s DNS service. Check https://github.com/caddy-dns to see if your DNS provider is available. If you haven’t already configured a system-alternative for caddy, now is the time to do so.
sudo caddy add-package github.com/caddy-dns/cloudflare
sudo systemctl restart caddy.service
DNS Provider Configuration
For Cloudflare, a decent example is below. Just use the ‘Getting the Cloudflare API Token’ part
https://roelofjanelsinga.com/articles/using-caddy-ssl-with-cloudflare/
Caddy Configuration
Use the acme-dns global option and then create a single site (used to determine the cert) and match the actual vhosts with subsites.
{
acme_dns cloudflare alotcharactersandnumbershere
}
*.some.org, some.org {
@site1 host site1.some.org
handle @site1 {
reverse_proxy * http://localhost:3200
}
@site2 host site2.some.org
handle @site2 {
root * /srv/www/site2
}
}
1.4 - Log
Access Logs
Simple Logs
In general, you should create a snippet and import into each block.
#
# Global Options Block
#
{
...
...
}
#
# Importable Snippets
#
(logging) {
log {
output file /var/log/caddy/access.log
}
}
#
# Host Blocks
#
site.your.org {
import logging
reverse_proxy * http://some.server.lan:8080
}
other.your.org {
import logging
reverse_proxy * http://other.server.lan:8080
}
Per VHost Logs
If you want separate logs for each vhosts you can still use a snippit by adding a parameter for the site name.
#
# Global Options Block
#
{
...
...
}
#
# Importable Snippets
#
(logging) {
log {
output file /var/log/caddy/{args[0]}.log
}
#
# Host Blocks
#
site.your.org {
# Add the argument "site.your.org" at the end sets the log file name to "site.your.org.log"
import logging site.your.org
reverse_proxy * http://some.server.lan:8080
}
other.your.org {
import logging other.your.org
reverse_proxy * http://other.server.lan:8080
}
Wildcard Sites
Wildcard sites only have one block but you must use the hostname matcher to separate logs. Note the slight difference in the handling of log names.
#
# Global Options Block
#
{
...
...
}
#
# Importable Snippets
#
(logging) {
log {
output file /var/log/caddy/{args[0]}
}
#
# Main Block
#
*.site.org, site.org {
# Everything goes to the file named here (access.log) unless it's filtered out by a more specific log block
import logging access.log
# This site will write to the access.log
@site host other.site.org
handle @site {
reverse_proxy * http://other.site
}
# This site will write to just this log file. Make sure to import logging BEFORE the '@www host' matcher or requests don't get logged here.
import logging www.site.org.log
@www host www.site.org
handle @www {
root * /var/www/www.site.org
file_server
}
Logging Credentials
If you want to track users that have authenticated, add a directive to the global headers so their usernames are included in the request log.
#
# Global Options Block
#
{
servers {
log_credentials
}
}
File Permissions
By default, only caddy can read the log files. This is a problem when you have a log analysis package. In recent versions of caddy however, you can set the mode.
log {
output file /var/log/caddy/access.log {
mode 644
}
}
If the log file doesn’t change modes, check the version of caddy. It must be newer than v2.8.4 for the change and you must restart.
Troubleshooting
You can have a case where the domain specific file never gets created. This usually happens when there us nothing to write to it. Check the hostname is correct.
1.5 - Secure
Web servers are constantly under attack. So much so some people way back before Y2K termed this “internet background radiation”.
There are two broad categories of attacks; ones that are just probing IPs, and ones that know a little about your web presence.
If an adversary just connects to your IP without knowing the domain, you can:
- Return an error
- Abort the request
The default configuration, and one that is consistent with the protocol, is to return an error. But the general consensus is to drop invalid requests. This prevents adversaries from profiling the system.
If they know your domains it more difficult. But they are usually asking for things that don’t exist, or trying to access a login page as fast as possible to guess passwords. In either case, you should apply a limit to how fast they can make requests.
So as a fist step, let’s drop or rate-limit every request.
Drop Unknown Domains
A default config will take connections to port 80, announce that it’s a Caddy web server and redirect you to https. Instead, let’s look at the host headers and SNI (Server Name Indicator) first. Then we can immediately abort any requests for things we don’t even have.
#
# Global Options Block
#
{
...
...
}
#
# Importable Sections
#
...
...
#
# Site Blocks
#
# Accept secure requests specific to our domain
https://*.some.org, https://some.org {
# Serve any known host
@some host some.site.org
handle @some {
encode
reverse_proxy * http://some:8080
}
@other host other.site.org
handle @other {
import auth
encode
reverse_proxy * http://other:8080
}
# Abort any unknown host
handle {
abort
}
}
# Accept any insecure request to our domain
http://*.some.org, http://some.org {
# Redirect only known host headers to https
@secure {
host some.site.org
host other.site.org
}
handle @secure {
redir https://{host}{uri} permanent
}
# Abort anything else.
handle {
abort
}
}
# Handle requests without the right domain or SNI
# so we can abort them early
:80, :443 {
# Log it so we have a record of the probe
import logging
abort
}
Monitor Valid Domains
Other attackers will connect to a valid domain and making requests for potential exploits. You’ll send them back 401 Unauthorized or 404 Not Found, but these are bots and they’ll just keep chugging along, ignoring the errors.
Rate Limiting
A basic protection is to rate-limit your responses. You’re already aborting requests for invalid sites, but if you’re hosting a web form or basic-auth protected site it can be useful. While a web app using a form may have it’s own rate limiting mechanism, there’s no inherent limit to the BA prompt. The only way to handle this is apply a rate limit to the specific site.
sudo caddy add-package github.com/mholt/caddy-ratelimit
...
...
https://*.some.org, https://some.org {
# A fairly generous over-all limit
rate_limit {
zone descriptive-name-1 {
key {http.request.remote_ip}
events 200
window 5s
}
}
@some host some.site.org
handle @some {
# A conservative one for this site
rate_limit {
zone descriptive-name-1 {
key {http.request.remote_ip}
events 20
window 5s
}
}
encode
reverse_proxy * http://other:8080
}
}
You’ll need to tune this as different sites may have radically different setups, some needing just one large request and another 50 small ones to fill a page load. It’s also mostly targeted at mitigating denial of service and credential stuffing. A distributed or patient bot makes it through just fine. Importantly, if you’re using a proxy service like Cloudflare, you’ll need to pull out the actual client IP.
Deploy a Protection System
For actual threat detection & response, consider adding Fail2Ban or CrowdSec
1.6 - WebDAV
Caddy can also serve WebDAV requests with the appropriate module. This is important because for many clients, such as Kodi, WebDAV is significantly faster.
sudo caddy add-package github.com/mholt/caddy-webdav
sudo systemctl restart caddy
{ # Custom modules require order of precedence be defined
order webdav last
}
site.your.org {
root * /var/www/site.your.org
webdav *
}
You can combine WebDAV and Directly Listing - highly recommended - so you can browse the directory contents with a normal web browser as well. Since WebDAV doesn’t use the GET method, you can use the @get filter to route those to the file_server module so it can serve up indexes via the browse argument.
site.your.org {
@get method GET
root * /var/www/site.your.org
webdav *
file_server @get browse
}
Sources
https://github.com/mholt/caddy-webdav https://marko.euptera.com/posts/caddy-webdav.html
1.7 - MFA
The package caddy-security offers a suite of auth functions. Among these is MFA and a portal for end-user management of tokens.
Installation
# Install a version of caddy with the security module
sudo caddy add-package github.com/greenpau/caddy-security
sudo systemctl restart caddy
Configuration
/var/lib/caddy/.local/caddy/users.json
caddy hash-password
Troubleshooting
journalctl –no-pager -u caddy
1.8 - Cloudflare
Cloudflare offers and excellent free service that proxies your web server, removing you from the front lines of the internet and blocking many known bad-actors. That deserves it’s own section, but we should make changes to caddy’s logging to accommodate the proxy. Most specifically, to log the actual IP address of the end user.
Add a Trusted Proxy
Cloudflare will inject the header CF-Connecting-IP into the request and we can use that to identify the end-user. This is preferred over other headers, which can be easily spoofed. But in order to make sure it’s Cloudfare doing the injecting, and not some internet rando, we need to explicitly know all the Cloudflare exit node IPs to trust them. That would be hard to manage, but happily, there’s a handy [caddy-cloudflare-ip] module for that. Many thanks to WeidiDeng!
sudo caddy add-package github.com/WeidiDeng/caddy-cloudflare-ip
sudo vi /etc/caddy/Caddyfile
#
# Global Options Block
#
{
servers {
trusted_proxies cloudflare
client_ip_headers CF-Connecting-IP
}
}
After restarting Caddy, we can see the header change
sudo head /var/log/caddy/access.log | jq '.request'
sudo tail /var/log/caddy/access.log | jq '.request'
Before
"remote_ip": "172.68.15.223",
"client_ip": "172.68.15.223",
After
"remote_ip": "172.71.98.114",
"client_ip": "109.206.128.45",
You can now update any rate_limit from the remote_ip to the client_ip
rate_limit {
zone descriptive-name-1 {
key {http.request.client_ip} # Uses the real client IP
events 20
window 5s
}
}
2 - HAProxy
HAProxy is the pre-eminent open source software load balancer, as evidenced by its nearly ubiquitous use by instagram, dropbox, twitter, etc. It incorporates many layer 4 (TCP) and layer 7 (HTTP) features that provide advanced load-balancing, protection and logging.
2.1 - Basic Auth
HAProxy can prompt for basic auth. This saves you from modifying back-end services.
Change a backend like this:
backend logs
mode http
server logs 10.40.0.8:5601 check
To:
backend logs
mode http
server logs 10.40.0.8:5601 check
acl authorized http_auth(users)
http-request auth unless authorized
http-request del-header Authorization
This requires a user list as well, so I included this at the bottom
userlist users
user admin password aBigLongHashHere
It seems one can add groups and such for more granular control. A remote password database would be a better solution.
Resources
https://blog.sleeplessbeastie.eu/2018/03/08/how-to-define-basic-authentication-on-haproxy/
2.2 - HAProxy and Certbot
Combine HAProxy and Certbot on the same container host allowing Certbot to keep certificates updated and HAProxy to proxy all traffic. HAProxy will redirect insecure requests to secure ports except for Certbot’s proof-of-ownership requests. Those are proxied to Certbot.
This example uses Docker.
Deployment
Test Resources
Let’s create a private docker network for our containers to use. That will let them connect dynamically to each other by container name so we can avoid hard-coding IPs.
docker network create -d bridge proxy-nw
It’s also useful to have a local web server to test against. Throw up a darkhttpd web server (a very simple web server) on the same container host, redirecting a high-level port on the container host to it’s port 80.
mkdir /tmp/www
echo "Look ma, http" > /tmp/www/index.html
docker run --name darkhttpd --net=proxy-nw --detach --volume /tmp/www:/www --publish 8008:80 shelmangroup/darkhttpd
curl localhost:8008
If everything went as planned you should get the expected output and docker logs darkhttpd will show the web request.
HAProxy
The creators of HAProxy suggest avoiding local dependencies by adding config and certificate files to the official image to create a custom image. For our tests however, we’ll start with a local config as we need it to work with a local Certbot instance.
The first step is to create volumes to hold the local configs; one for HAProxy and one for Certbot. The docker image provided by HAProxy defaults to /usr/local/etc/haproxy/haproxy.cfg. Certbot defaults to /etc/letsencrypt. The cert itself is in a subdirectory under ’live’ named for the site, such as live/server.your.org, but you need all of the subdirectories to persist or renewals won’t work right. We’ll present to the containers in the run command below.
docker volume create haproxy
docker volume create certbot
# Check the location just in case
docker volume inspect haproxy
Next, create a config file for HAProxy to specifies the darkhttpd server as a back-end server. Notice that darkhttpd itself is listening on port 80. Port 8008 above is the translated-port we published on the container host for testing and not what you want HAProxy to connect to.
sudo vi /var/lib/docker/volumes/haproxy/_data/haproxy.cfg
global
maxconn 4000
tune.ssl.default-dh-param 2048
frontend http
bind *:80
mode http
default_backend http-server
timeout client 1h
backend http-server
mode http
server darkhttpd darkhttpd:80
timeout connect 1h
timeout server 1h
Now lets run the HAProxy image.
docker run \
--detach \
--mount type=volume,source=haproxy,target=/usr/local/etc/haproxy,readonly \
--mount type=volume,source=certbot,target=/etc/letsencrypt/,readonly \
--name haproxy \
--net=proxy-nw \
--publish 80:80 \
--publish 443:443 \
--restart always \
haproxy
curl localhost
If all went well, you can now access the host on port 80 and hit the darkhttpd server. Importantly, you must be able to hit this from the Internet, as Let’s Encrypt will be hitting it to prove you own the host in question. Issue a docker ps and docker logs haproxy if something failed. Once you’re done testing, stop the containers and remove the unneeded darkhttpd server.
docker stop haproxy
docker stop darkhttpd; docker rm darkhttpd; docker rmi shelmangroup/darkhttpd
Certbot
The official image is designed to run-and-exit, requiring the container OS to have a cron job to manage renewals. Our goal is for Certbot manage the certificate without external dependencies, and to be protected from general internet traffic by HAProxy. So let’s create our own image based on theirs.
Docker Image Set-Up
# Make a directory
mkdir ~/docker-certbot
cd ~/docker-certbot
# Create the Docker file and a script that will renew the cert
touch Dockerfile
touch cert_request
chmod +x cert_request
Dockerfile
Create the following Docker file. It uses Certbot’s image as a base, places a cert request script and launches the cron daemon in the foreground (crond doesn’t run by default). With crond running this way you can examine the output easily with a docker log command.
vi Dockerfile
FROM certbot/certbot
COPY cert_request /etc/periodic/daily
ENTRYPOINT ["crond", "-f"]
Renewal Script
Below is the content of the script. The COPY above puts it in a location where crond will run it once a day. Let’s Encrypt actually recommends twice a day but this is close enough. We also concatenate the public and private key into a single file as required by HAProxy.
The reason we’re going to all this trouble is that issued certs are valid for 90 days. You have to check often.
TODO We could probably do something slick here and not recreate that file if we spent some time thinking about it. We could also use –post-hook to tell haproxy something has changed or at least see if that’s needed.
#!/bin/sh
HOST=server.your.org
EMAIL=[email protected]
certbot certonly $1 --standalone -d $HOST --non-interactive --preferred-challenges http --agree-tos --email $EMAIL
cat /etc/letsencrypt/live/$HOST/fullchain.pem /etc/letsencrypt/live/$HOST/privkey.pem > /etc/letsencrypt/live/$HOST/$HOST.pem
Building and Testing the Image
Let’s build the image and publish port 80. We won’t want that published later on but it’s useful for a first test so we know Certbot itself doesn’t have any issues.
docker build -t allen/certbot .
docker run --name certbot --detach --publish 80:80 --mount type=volume,source=certbot,target=/etc/letsencrypt allen/certbot
Let’s run the first request manually against Let’s Encrypt’s test system to make sure it works as expected.
# Attach to the running container
docker exec -it certbot sh
# Execute the request script aginst let's encrypt's test system
/etc/periodic/daily/cert_request --staging
You should see output similar to:
- Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/server.your.org/fullchain.pem
If not, start digging into if your web server is reachable or the host name is wrong. Importantly, add –staging to the request while you are debugging. The production Let’s Encrypt service is rate-limited.
Integrating HAProxy and Certbot
Now that you know Certbot works, recreate it without an external port (this is easier than a restart). We’ll use HAProxy to route requests to it. Note: now that it has a certificate, certbot will execute a renew operation which no longer requires the verification of accepting port 80 challenges. The private key it already has is used. But you still want to catch requests as any new certs you add later will require the verification.
docker stop certbot
docker rm certbot
docker run --name certbot --detach --net=proxy-nw --mount type=volume,source=certbot,target=/etc/letsencrypt --restart always allen/certbot
The HAProxy config file needs an update, both to handle reqeusts to Certbot and to take advantage of the certificate we got earlier.
sudo vi /var/lib/docker/volumes/haproxy/_data/haproxy.cfg
global
tune.ssl.default-dh-param 2048
defaults
timeout client 1h
timeout connect 1h
timeout server 1h
frontend http
bind *:80
mode http
acl certbot-acl path_beg /.well-known/acme-challenge/
http-request redirect scheme https if ! certbot-acl
use_backend certbot-backend if certbot-acl
frontend https
mode http
bind *:443 ssl crt /etc/letsencrypt/live/wiki.your.org/wiki.your.org.pem
backend certbot-backend
mode http
server certbot certbot:80
docker stop haproxy
docker start haproxy
Let’s do some basic tests
# Test that non-secure requests are redirected with a 302
curl -D - http://server.your.org/xyz
HTTP/1.1 302 Found
Cache-Control: no-cache
Content-length: 0
Location: https://wiki.your.org/xyz
# Test that non-secure requests to acme-challenge are not
curl http://server.your.org/.well-known/acme-challenge/
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
# Take a look at the certificate we have
openssl s_client -showcerts -connect localhost:443
...
...
Verify return code: 20 (unable to get local issuer certificate)
These are as expected. Certbot isn’t isn’t in the middle of a request so nothing is listening. The cert displayed is from the testing server and so untrusted.
Now, let’s try requesting a cert from the production server and assuming it shows you success, restart HAProxy to use the new cert.
docker exec -it certbot sh
/etc/periodic/daily/cert_request --force-renew
exit
docker stop haproxy
docker start haproxy
openssl s_client -showcerts -connect localhost:443
...
...
Verify return code: 0 (ok)
Now that you have a working proxy, you’ll probably want to proxy some things other than certbot. Let’s add an example xwiki running on it’s own network.
docker network connect xwiki-nw haproxy
# To the haproxy conf file, add
...
...
frontend https
mode http
bind *:443 ssl crt /etc/letsencrypt/live/wiki.your.org/wiki.your.org.pem
acl is_wiki hdr(host) -i wiki.your.org
use_backend xwiki if is_wiki
backend xwiki
mode http
server xwiki xwiki:8080
...
...
docker stop haproxy
docker start haproxy
And enjoy!
Troubleshooting
Container exits immediately
Check the logs with docker logs haproxy to see what’s up.
‘server certbot’ : could not resolve address ‘certbot’…Failed to initialize server(s) addr.
HAProxy wont start if it can’t connect to the back-end server. Adding the init-addr option accomodates that.
cat: can’t open ‘/etc/letsencrypt/live/server.your.org/fullchain.pem**
Certbot can start saving certs under -0001 and such when you start requesting certs that it already knows about but can’t find. You can delete the /etc/letsencrypt/* dir
Moving a Custom Container
docker save allen/certbot >test.tar scp some.server:test.tar . docker load -i test.tar
To Remove old containers and images docker ps -a docker rm … docker images docker rmi … docker system prune –all docker image prune –all
Other Resources
https://www.lab-time.it/2018/09/20/running-haproxy-and-lets-encrypt-on-docker/ https://docs.docker.com/storage/volumes/ https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/ https://certbot.eff.org/docs/install.html https://wiki.alpinelinux.org/wiki/Alpine_Linux:FAQ#My_cron_jobs_don.27t_run.3F https://gist.github.com/andyshinn/3ae01fa13cb64c9d36e7
2.3 - VHost
Overview
- Create the DNS Name
- Configure HAProxy
- Attach to certbot, edit the renewal script in daily, and run the script
- Edit the haproxy config to use the new keyfile
Create the DNS Name
Login to the DNS server and create a CNAME to the proxy02 server
vi some file
some entry
Configure HAProxy
sudo vi /var/lib/docker/volumes/haproxy/_data/haproxy.cfg
frontend https
...
...
acl SOMESERVER-acl hdr(host) SOMESERVER.marietta.edu
use_backend SOMESERVER-backend if SOMESERVER-acl
...
...
backend SOMESERVER-backend
mode http
server SOMESERVER 10.40.1.216:8080 check
Test and reload the config
(see below)
Configure certbot and request a cert
Login to your proxy server and attach to the running certbot
docker exec -it certbot sh
vi /etc/periodic/daily/cert_request
# add to the variable at top
/etc/periodic/daily/cert_request
Configure haproxy to use the new cert
vi /var/lib/docker/volumes/haproxy/_data/haproxy.cfg
# add 'cert path/certname.pem' along with the others
docker kill -s HUP haproxy
Troubleshooting
You can check the config by attaching and running the test parameter
docker exec -it haproxy bash
haproxy -c -V -f /usr/local/etc/haproxy/haproxy.cfg
The below does a graceful restart. This is preferred over the hard stop and start
docker kill -s HUP haproxy
3 - lighttpd
Pronounced “lighty”, it describes itself as being two to three times faster than apache. More importantly, it consumes very few resources; important on a small system.
Install
sudo apt-get install lighttpd
Configure
Configuration files are in /etc/lighttpd. Interestingly, one edits (or replaces) the files in the conf-available folder, which correspond to the available modules, and then turns them on with the lighty-enable-mod command.
SSL
Generate a self-signed certificate. (remember to use the public facing DNS name when it asks for Common Name)
sudo openssl req -new -x509 -keyout /etc/lighttpd/server.pem -out /etc/lighttpd/server.pem -days 3650 -nodes
sudo chmod 400 /etc/lighttpd/server.pem
Enable SSL and restart (see SSL notes below)
sudo lighty-enable-mod ssl
sudo service lighttpd restart
Proxy
Let’s say we want to SSL wrap service, like transmission, that may not provide SSL on it’s own. In this example it’s running on the same host on port 9091
Edit the config file
cd /etc/lighttpd/conf-available
sudo cp -a 10-proxy.conf 10-proxy.conf.bak
sudo vim 10-proxy.conf
And make it look like so
server.modules += ( "mod_proxy" )
$HTTP["url"] =~ "^/transmission/" {
proxy.server = ( "" =>
(
( "host" => "127.0.0.1",
"port" => "9091"
)
)
)
}
And enable and restart
sudo lighty-enable-mod proxy
sudo service lighttpd force-reload
Notes
SSL -
At one time you edited the main conf file /etc/lighttpd/lighttpd.conf so it contained:
$SERVER["socket"] == ":443" {
ssl.engine = "enable"
ssl.pemfile = "/etc/lighttpd/server.pem"
}
But as of Ubuntu 13+, the above is included in a sub conf file referenced in the main file
include_shell "/usr/share/lighttpd/include-conf-enabled.pl"
So if you try to add it in the main, you get a red-herring error message of
(network.c.379) can’t bind to port: 443 Address already in use
The better way is a virtual host file that combines all the settings for one service, in one place.
One would do that by adding include “domain1.com.conf” at the bottom of lighttpd.conf file and start it with a $HTTP[“host”] == “www2.example.org” and then some directives. Haven’t tried it though.
Links
http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs:ModProxy#Example
4 - Log File Perms
Make the existing log files readable by apache and change the log rotation config so the new ones are too.
sudo chown root:www-data /var/log/apache2/*
sudo vi /etc/logrotate.d/apache2
...
...
create 640 root www-data
...
...
Make the apache logs viewable in the config
sudo vi /etc/apache2/sites-enabled/you.your.org.conf
<VirtualHost *:80>
ServerAdmin webmaster@localhost
DocumentRoot /var/www/test.marietta.edu
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
...
...
...
Alias "/logs" "/var/log/apache2"
<Directory "/var/log/apache2">
Options Indexes
Require all granted
</Directory>
Note:
You may have to change the execute bit on the directory so the group has permission to list the contents.
https://serverfault.com/questions/961749/changing-default-apache-log-permissions
5 - nginx
This replaced Apache’s Web Server as the go-to after 2010 and is still in the top spot.
5.1 - Let's Encrypt Proxy
When you’re proxying a back-end server that’s making let’s encrypt challenges for it’s own purposes, you need a webroot to handle those requests.
vi /etc/nginx/conf.d/some.server.org.conf
#
# Redirect requests for the site that are unencrypted, except for let's encrypt challenges
#
server {
listen 80;
server_name some.your.org;
location / {
return 301 https://$host$request_uri;
}
# Allow access to the ACME Challenge for Let's Encrypt
location ^~ /.well-known/acme-challenge {
allow all;
root /var/www/some.your.org/htdocs;
}
}
server {
listen 443 ssl;
...
...
5.2 - WebDAV on Alpine
With nginx, WebDAV is only available in edge. (or at least it was in 2018 or so)
sed -i -e 's/v[[:digit:]]\..*\//edge\//g' /etc/apk/repositories
apk update;apk upgrade
Then you can install nginx and the htpasswd too
apk add nginx-mod-http-dav-ext
apk add apache2-utils
htpasswd -c /etc/nginx/conf.d/passwords user1
sudo mkdir /var/lib/nginx/tmp/
server {
listen 80 default_server;
listen [::]:80 default_server;
# Try some media
location /video/ {
alias /srv/media/video/;
autoindex on;
dav_ext_methods PROPFIND OPTIONS;
auth_basic "Authentication Required";
auth_basic_user_file /etc/nginx/conf.d/passwords;
}
}
6 - WebDAV
Background
The best approach to maintaining content on a small-scale webserver is to use WebDAV. This lets you remain with to HTTP protocols and conventions on for the whole stack. No messing about with firewalls, ftp or windows sharing.
On the content creator’s side, it’s built into the Operating System so they can use the File Manager they use everyday to manage their web content just like a local file. This simplifies support greatly.
Server Testing
Apache is the only major web server that fully supports the native file managers on major OSs (as of 2022). Nginx almost get’s there in our testing, but write-level access doesn’t quite work with Mac OS Finder. IIS is similarly lacking.
Deployment
Installation
We’re installing from repo and enabling the DAV modules using the debian tools rather than creating the symlinks ourselves.
sudo apt-get update
sudo apt-get install apache2
sudo a2enmod dav dav_fs dav_lock
Configuration
The best method is to use a site-based approach where we put our content and config in their own locations. We’re using the debian conventions for file locations here.
# Create the content folder
sudo mkdir /var/www/www.your.org
sudo chown www-data:www-data /var/www/www.your.org
# Check ownership on the WebDAV file lock database folder
# and chown -R www-data:root as needed.
ls -la /var/lock/apache2
# Create the config file
sudo vi /etc/apache2/sites-enabled/www.your.org
Here’s our example site.
DavLockDB /var/lock/apache2/DavLock
<VirtualHost *:80>
ServerAdmin [email protected]
DocumentRoot /var/www/www.your.org
ServerName www.example.org
ErrorLog ${APACHE_LOG_DIR}/wwwerror.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
<Directory /var/www/test.your.local/>
DAV On
AuthType Digest
AuthName "webdav"
AuthUserFile /etc/apache2/users.password
<LimitExcept GET POST OPTIONS HEAD>
Require user test
</LimitExcept>
</Directory>
</VirtualHost>
Useful notes:
If you leave out the DocumentRoot, apache will use a default value and present the default web site even when you have no sites enabled, driving you nuts until you finally find a hint about the default value being compiled differently for different distributions.
For authentication you should at least use HTTP Digest Authentication. Though it’s generally recognized that SSL should be deployed everywhere.
Limiting Access to WebDAV
The Require Directive
These two methods appear functionally equivalent. The first is the example from the Require Directive
<RequireAny>
Require method GET POST OPTIONS
Require valid-user
</RequireAny>
The second is the example from the WEBDAV module. NOTE - if you leave the ‘all granted’ you seem to mask
Require all granted
<LimitExcept GET POST OPTIONS>
Require valid-user
</LimitExcept>
Apache’s LimitExcept directive is part of the core module, and the webdav example uses it, so we’re going with it here.
This article suggest the three methods below. HEAD is a common one, so it makes sense.
<Directory />
<LimitExcept GET POST HEAD>
deny from all
</LimitExcept>
</Directory>
Or possibly
<LimitExcept GET POST OPTIONS PROPFIND>
The AuthBasicProvider is, in this case, optional, since file is the default value for this directive.
User Database LDAP/AD
Now that you’re requiring auth, you have to compare it against a user database. LDAP is a possibility.
First, enable the LDAP module:
a2enmod authnz_ldap
Then config with an ldap source.
DavLockDB /usr/local/apache2/var/DavLock
<VirtualHost *:80>
ServerAdmin webmaster@localhost
DocumentRoot /var/www/test.your.local
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
<Directory /var/www/test.your.local/>
DAV On
AuthName "YOUR"
AuthType Basic
AuthBasicProvider ldap
AuthLDAPBindDN [email protected]
# Creds are "svcapache/XXXXXX"
AuthLDAPBindPassword XXXXX
AuthLDAPURL "ldap://ldap.your.local:389/OU=People,DC=your,DC=local?uid?sub?(objectClass=*)"
Require all granted
<LimitExcept GET POST OPTIONS>
Require valid-user
</LimitExcept>
</Directory>
</VirtualHost>