So, Let's Encrypt is now in public beta. Yet it is not so easy to use for everybody. Of course for Mainstream Linux (Let's say Debian), it is fine. For OpenBSD, euh…

OpenBSD also has its own minimalistic webserver. So we need to adapt. Impossible to use the provided turnkey solution.

I write this article to help others and myself when it's about to have new or renew certificates.

I expect this article to be usefull to other hackers who wish or have to work their own solution.

Foreplay

Using SSL/TLS isn't easy, for several reasons:

  • It is a nightmare to use and setup properly (read my previous article).
  • It is supposed to improve security, but it actually creates other security problems !
  • It is not just one thing to setup. It is actually a set of tools to set, making them to work together in a smooth way and adapt to each situation. Therefore, you cannot copy and paste my work. You have to mind, adapt your own installation and my scripts to it.

My scripts are written for OpenBSD, with Korn Shell. Please test them and check that they are all good on your server.

There is no special dependency, except python. And tlsa_rdata if you want to use DANE.

Generate, maintain and rotate certificates

I wish to use DANE (TLS cert's with fingerprints in DNSSEC secured zones). This compels me to create cert's and publish their fingerprints in the DNS before to put them into service (2 TTL according to these recommandations and those).

During this short time, there are two deck of valid certificates, current and latest - those which have just been created and shall be serving soon.

All of this compels me to hugely slice the whole process.

Generate cert's one by one

To create the cert's, we are going to use the CA with an unofficial acme client. It's cool, it's free software, Let's Encrypt really wishes to have that.

Just to mention also that I could never setup the official client, even less make it to work.

I have created a system user le with a working dir' in /var/le to contain the main scripts, acme_tiny and the repositories to the certificates - take care for the permissions in this one.

First script, whose job is to create and obtain signing of the certificates and combine them in usable files. It takes the FQDN to the certificate as a parameter - so the whole name, from the start to the tld. It returns the timestamp, which indicate where is the generated certificate in the file tree.

/var/le/generate - script - GPG signature

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
#!/bin/sh

# needs https://github.com/diafygi/acme-tiny/

# first time
# mkdir -p /var/le/domains/{domains you need}

# create account master key for Letsencrypt to run
# openssl genrsa 4096 > master.key

# Think about creating web vhost and DNS records

timestamp=$(date +%s)
domain=$1

if [ ! -d /var/www/$1 ] ; then
echo "webserver not ready"
exit
fi

mkdir -p /var/le/domains/$domain/$timestamp/
cd /var/le/domains/$domain/$timestamp/

# create key
openssl genrsa 4096 > serverkey.pem

# create CSR
openssl req -new -sha256 -key serverkey.pem -subj "/CN=$domain" > csr

# create repository in web vhost for challenge
# user le should be able to write here
# user www should be able to read and walk the path (r and x rights)
mkdir -p /var/www/$1/.well-known/acme-challenge/

# asking for cert
python2.7 /var/le/acme-tiny/acme_tiny.py --account-key /var/le/master.key \
 --csr /var/le/domains/$domain/$timestamp/csr \
     --acme-dir /var/www/$1/.well-known/acme-challenge/ \
    > /var/le/domains/$domain/$timestamp/cert.pem

# create the files you need (full and partial chain…)
# intermediate.pem contains the chain from letsencrypt AC certificate
cat /var/le/domains/$domain/$timestamp/cert.pem \
    /var/le/intermediate.pem > /var/le/domains/$domain/$timestamp/chain.pem

# This full chain file with the key is needed for haproxy
# (to deliver SNI in front of Httpd)
cat /var/le/domains/$domain/$timestamp/cert.pem \
    /var/le/intermediate.pem \
    /var/le/domains/$domain/$timestamp/serverkey.pem \
    > /var/le/domains/$domain/$timestamp/full.pem

rm -rf /var/www/$1/.well-known/*

# if everything went ok, then modify the rights of the certificates files.
# And return the timestamp.
if [ -r cert.pem ] ; then
    chmod 400 *
    return=$timestamp
else
    return=10
fi

echo $return
return $return

This script should run as le. It has these rights:

-r-sr-xr-x  1 le        tls    1412 Dec 30 21:05 generate

I set S so it will be ran by le, whatever it takes. Even if root launches the script, it will be le the actual runner, and thus le running the acme client. Root doesn't speak with the outside nasty world.

I get writing rights off. So if the script is compromised, neither le nor anyother user can modify it.

It reduces potential risks.

Loop to create the certificates

The previous script create each certificate in a repository and combine the files to be used by various servers (web, imap, smtp…).

The following script is to regenerate the whole certificates deck (so, for all the hosted services), to set the proper permissions (The owner should be root or smtpd won't start ! Described here.), and generated all TLSA fingerprints for DANE with a third script I am going to introduce later.

All certificates come with latest statute.

/var/le/regenerate-all - script - GPG signature

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#!/bin/sh

for domain in $(ls /var/le/domains/)
do
    echo $domain

    # create the certs and same time get the timestamp for which it has been generated.
    time=$(su - le /var/le/generate $domain)

    # give the certs to root
    chown -R root:tls /var/le/domains/$domain/$time/

    # If the previous command did fine
    # then link the certs to be the new certs (that will be loaded in a few days).
    if [[ $? -eq 0 ]]; then
        echo "renew $domain certs"
        ln -s /var/le/domains/$domain/$time/ /var/le/domains/$domain/latest
    fi

done

# create the tlsa RR for the newly generated certs
/var/nsd/tlsa-gen latest

Generate TLSA

/var/nsd/tlsa-gen - script - GPG signature

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#!/bin/sh

zone=22decembre.eu

output=/var/nsd/$zone/$1.db

rm -f $output

while read host usage selector match preambule
do
    domain=$host.$zone
    cert=/var/le/domains/$domain/$1/full.pem
    echo $preambule.$host "IN TLSA" $(/usr/local/bin/tlsa_rdata $cert $usage $selector $match) \
        >> $output
    echo $domain " tlsa done"
done < /var/nsd/tlsa-list

zone=$zone.

echo "Resigning zone:"
/usr/local/bin/zkt-signer -v -f $zone || exit

echo "Update NSD:"
/usr/sbin/nsd-control reload $zone

This script takes the state of the certificates to read and write the fingerprints with tlsa_rdata, in a current or latest.db file.

This file is included in the DNS zone (there are two $INCLUDE), the whole zone is signed and loaded again.

The file tlsa-list contains the fingerprints list to create:

www             3 1 1 _443._tcp
photos          3 1 1 _443._tcp
blackblock      3 1 1 _443._tcp
blackblock      3 1 1 _143._tcp
blackblock      3 1 1 _587._tcp
blackblock      3 1 1 _25._tcp

The first is the simple machine name, then come the three special DANE settings (you can read the RFC, I did it on the part related to those, it is understandable ! YYYESSSSSS !).

This allows the use of TLSA with all services and protocoles you might want. Here I have imap, smtp and https.

It is really important that you set only the machine name (blackblock, photos or www here). The signer will append the zone name. To set here the FQDN would result in something like that:

_143._tcp.blackblock.22decembre.eu.22decembre.eu.

You ought to be cautious with the DNS, DNSSEC and DANE. If your DNS server has a single problem (connectivity, trouble while signing or other), DNSSEC and DANE validations could make your server unreachable for one, several or all hosted services.

Putting the certificates into service

Here, we are simply to remove the link to the certificates created one or two months ago (that are still acting), pass the created certificates from latest (to be used soon) to current (acting now), and reload or restart services.

These services have to be setted up to use the current cert's. For example:

/var/le/domains/www.example.com/current/cert.pem

Which certificate file matters as well (just the cert', the whole chain…). Read your server's doc to know about it.

The script also copy the new certificates to Radicale's (a light self-hosted agenda service) repository to use them (Radicale is too much simple to load them as root before dropping priviledges).

/var/le/transition-cert - script - GPG signature

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#!/bin/sh

for domain in $(ls /var/le/domains/)
do
    # end of current cert by removing the link
    rm /var/le/domains/$domain/current

    # certificates that have been created few days ago linked
    ln -s $(readlink -f /var/le/domains/$domain/latest) /var/le/domains/$domain/current

    # these certificates are not anymore new or latest, they are current !
    rm /var/le/domains/$domain/latest
done

cp /var/le/domains/blackblock.22decembre.eu/current/serverkey.pem /var/radicale/
cp /var/le/domains/blackblock.22decembre.eu/current/chain.pem /var/radicale/
chown _radicale /var/radicale/*.pem

rm /var/nsd/22decembre.eu/current.db
rm /var/nsd/22decembre.eu/latest.db

# The signer bugs if latest.db doesn't exist.
touch /var/nsd/22decembre.eu/latest.db

/var/nsd/tlsa-gen current

rcctl reload haproxy
rcctl restart smtpd
rcctl restart spamd
rcctl reload dovecot
rcctl restart radicale

Crontab

The scripts are launched by crontab as root, every second month at a five days interval in order to let the fingerprints propagate, as said before :

#
SHELL=/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin
HOME=/var/log
#
#minute hour    mday    month       wday    command
#
# renew letsencrypt cert'
13      1       10      1,3,5,7,9,11    *       /var/le/regenerate-all

# Loading
15      1       15      1,3,5,7,9,11    *       /var/le/transition-cert

Httpd and Haproxy

As Let's Encrypt validate certificates with HTTP, the webserver configuration has a big impact here.

The new OpenBSD http daemon does not allow to have several certificates on a single server. If you have more than one virtual server, you're doomed. So I use Haproxy, like Vigdis.

I won't describe other services. Your duty to look at it.

Httpd

Roughly said, you put in httpd.conf the web conf' only (php, a few redirections, basic things) and each virtual server has its own port.

prefork 4
types { include "/usr/share/misc/mime.types" }

server "www.22decembre.eu" {
    listen on ::1   port 2000

    root "/www.22decembre.eu"
    location "/.well-known/*"    { directory auto index }

    log error www-error.log
    log access www-access.log

    location "/bibliotheque/" { block return 301 "https://biblib.22decembre.eu" }
    location "/biblib/"     { block return 301 "https://www.22decembre.eu/2013/12/17/softwares-projects-en/" }

    location "/drafts/*"    { directory auto index }
    location "/downloads/*" { directory auto index }

    location "*/feed/" { directory index atom.xml }

    location match "/2012/02/27/*"  { block return 301 "https://www.22decembre.eu/2015/03/31/5-sign-mail-fr/" }
}

server "photos.22decembre.eu" {
    listen on ::1   port 2001

    root "/photos.22decembre.eu"
    location "/.well-known/*"    { directory auto index }

    log error photos-error.log
    log access photos-access.log

    directory index index.php

    location "*.php" { fastcgi socket "/tmp/php.sock" }
}

server "blackblock.22decembre.eu" {
    listen on ::1 port 2002

    root "/blackblock.22decembre.eu"
    directory index index.html

    log error blackblock-error.log
    log access blackblock-access.log

    location "/.well-known/*"    { directory auto index }

    location "/links/"      { directory index index.php }

    location "*.php" { fastcgi socket "/tmp/php.sock" }
}

Yes, a bit messy. Life is Life

Haproxy

Then all tls settings and https redirections.

global
    log 127.0.0.1   daemon debug
    maxconn 1024
    chroot /var/haproxy
    daemon
    pidfile /var/run/haproxy.pid
    tune.ssl.default-dh-param 2048
    ssl-default-bind-options no-sslv3 no-tls-tickets
    ssl-default-bind-ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!aNULL:!MD5:!DSS
    ssl-default-server-options no-sslv3 no-tls-tickets
    ssl-default-server-ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!aNULL:!MD5:!DSS

defaults
    log     global
    mode http
    option forwardfor
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    option  httplog
    option  dontlognull
    option  redispatch
    retries 3
    maxconn 2000

########################

frontend https
    bind :::443 ssl crt /var/le/domains/www.22decembre.eu/current/full.pem crt /var/le/domains/photos.22decembre.eu/current/full.pem
    bind *:443 ssl crt /var/le/domains/www.22decembre.eu/current/full.pem crt /var/le/domains/photos.22decembre.eu/current/full.pem

    acl is_domain0 hdr_end(host) -i www.22decembre.eu
    acl is_domain1 hdr_end(host) -i photos.22decembre.eu
    acl is_domain2 hdr_end(host) -i blackblock.22decembre.eu

    acl is_domain9 hdr_end(host) -i lectures.22decembre.eu

    use_backend www if is_domain0
    use_backend photos if is_domain1
    use_backend blackblock if is_domain2

    use_backend lectures if is_domain9

frontend http
    bind *:80
    bind :::80

    acl is_domain0 hdr_end(host) -i www.22decembre.eu
    acl is_domain1 hdr_end(host) -i photos.22decembre.eu
    acl is_domain2 hdr_end(host) -i blackblock.22decembre.eu

    acl is_domain9 hdr_end(host) -i lectures.22decembre.eu

    use_backend www if is_domain0
    use_backend photos if is_domain1
    use_backend blackblock if is_domain2

    use_backend lectures if is_domain9

backend www
    server www ::1:2000
    redirect scheme https code 301 if !{ ssl_fc }
    http-response set-header Strict-Transport-Security "max-age=16000000; preload;"

backend photos
    server www ::1:2001
    redirect scheme https code 301 if !{ ssl_fc }
    http-response set-header Strict-Transport-Security "max-age=16000000; preload;"

backend blackblock
    server www ::1:2002
    redirect scheme https code 301 if !{ ssl_fc }
    http-response set-header Strict-Transport-Security "max-age=16000000; preload;"

backend lectures
    redirect location https://www.22decembre.eu/category/lectures.html code 301

Explanations:

  • In ssl-default-bind-options and ssl-default-bind-ciphers, you define tls configuration. Mine is inspired by https://cipherli.st/.
  • With acl is_domain1 hdr_end(host) -i www.22decembre.eu, I assign web requests to the corresponding backends and virtual webservers.
  • The crt attributes in liges bind … ssl contain names of files to tls certificates whose ought to contain the whole chain, including the private key.
  • redirect makes it mandatory to connect with https if connecting using http.
  • http-response set-header Strict-Transport-Security sets the website enforce HSTS.

These two last options should be in the backend. If you set them as a generic option, then you will never be able to generate new certificates : the acme client will try to connect with https on a website that does not have yet a correct certificate, whereas it is there to provide it ! Chicken and egg problem.

Create a new certificate

Here, I am going to describe the procedure to create and sign a new certificate. I will reuse some parts above, but there is a lot to do yourself.

You first need to add the name to your DNS and to the domain repository in /var/le/domains/, and setup the web server.

The virtual host on the server itself is standard too : /var/www/FQDN. This directory should contain as well a subdirectory .well-known where le can write and www can read and go (so: reading and execution rights).

DNS

You need to add your new name in your DNS zone:

$ORIGIN 22decembre.eu.
$INCLUDE soa
@               IN      NS      blackblock.22decembre.eu.
@               IN      NS      ns6.gandi.net.
@               IN      MX 10   blackblock.22decembre.eu.

@               IN      TXT     "v=spf1 mx -all"

blackblock      IN      A       90.185.111.213
        IN      AAAA    2001:16d8:dd00:8207::0
        IN      AAAA    2001:16d8:dd00:207::2

www             IN      CNAME blackblock
photos          IN      CNAME blackblock
ntp             IN      CNAME blackblock
lectures        IN      CNAME blackblock
biblib          IN      CNAME blackblock
piwik           IN      CNAME blackblock

_keybase        IN TXT "keybase-site-verification=brQmXLHBZ1ZHELHYwuWWIshIQ-PTj1Q7Kj_W4vlmzWs"

selector1._domainkey IN TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDOYPdaCG3yT4KKFmW6/CipMqAzE9t8Osusj7MxgER6/t5Q4ZLkAv/53H5GZHDDydYhABFBZF4S2kD/wZJel9aZ5XrvJnuwHzAAUpAYYrbte1q4LLPelsC+WyRaB9tb/nrbDK+k3/sEfydo9lDHVmq8JtsTmwVJWHnzD6NExNZNTwIDAQAB"

$INCLUDE latest.db
$INCLUDE current.db

I suppose it is slightly transparent. When you want a new certificate, just add the CNAME and voila. Don't forget to resign and load the zone again.

Checkout the two includes in the bottom. yes it's for dane.

I also share the SOA between my zone, so include it here. One has to be lazy enough. The last weird thing is probably the domainkey RR. I advice you to read here to have more information about it. There are also articles about DKIM on the web, search for them.

Serveur web

Create the webserver repository with good permissions

$ cd /var/www
$ mkdir -p www.example.com/.well-known/
$ doas chown le:www www.example.com/.well-known/
$ doas chmod 750 www.example.com/.well-known/

Chmod is probably not necessary.

Conf' Httpd

server "www.example.com" {
    listen on ::1   port 2000

    root "/www.example.com"
    location "/.well-known/*"    { directory auto index }

    log error error.log
    log access access.log
}

Take care of

location "/.well-known/*"    { directory auto index }

It seems to me to be quite important (in case of debugging, or if you use scripts then you have to set it's bare files there).

Haproxy

Here, you add a backend and an acl, as mentionned before. When your certificates are on, you can setup an https redirection and hsts. Not before.

Generate the certificate

Think about creating the directory in le domains repository:

$ cd /var/le/domains
$ doas -u le mkdir www.example.com

Aim and fire:

$ doas -u le /var/le/generate www.example.com
$ cd /var/le/domains/www.example.com/

Ownership to root and link to current, directly:

$ doas chown -R root 123456789
$ doas ln -s 123456789 current

You can now setup your services to use the current certificate, then relaunch them. You can also generate TLSA:

$ doas /var/nsd/tlsa-gen current

What does it lack, what could be improved

Root

There are too much things done by root. And it's not cool. Same time, using one machine as web and DNS server makes it mandatory to check and lock files and permissions.

Thus, root has to generate the TLSA, because it's the owner and only reader of the tls cert's files. A solution would be to generate the TLSA just after the certificates, but you got the problem of including them in the DNS zone.

See, each solution creates new trouble.

This is also what makes mandatory to manage Radicale on the side.

Automation

Automation don't like special situations. Everything uncommon is bad and must be avoided at all cost, because it ruins the process, and we cannot adapt (same as the Radicale comment)

The TLSA scripts are written with a single domain in mind. If you have two or more, you are screwed and must adapt my scripts by yourself.

Optimization

I suppose one can greatly optimize all this mess, particularly asking less certificates (by putting more names in one cert'). But I have no idea how to do that.

Rate limits

The whole process regenerate all the certificates at once. This implies that you cannot have more than five certificates per domain (cf this LE thread), or you have to find some way to solve the problem mentionned above.

Or, again, you have to find a way to spread the renewing week by week. And congrats if you succeed on this one !

Conf'

I don't have a single one conf' file for all that mess. So if I forget one thing, there is a risk of bug. I did not either think of not setting DANE, whereas it could greatly ease the thing (you just have to renew the cert's and reload services, no need to wait several days).

I suppose the TLSA listing could have been a good place for a central configuration (with the list of certificates, all the DNS zones to include and sign…) but I think it raise the level too high for me.

I believe the previous problems (automation, optimization and rate limites) should be understood as a whole, so the solution should be systemic.

Monitoring

You should monitor the whole system. For now, I don't know what happens in case of bug. Think of that ! I tried to imagine or avoid many problems, but there are still some. Obviously ! So, what happens ? The system itself does not know about.

The cron tasks should report you by mail, but anyway, monitoring has to be done carefully.

DNS, Httpd and haproxy

It's your job to set up and check the good running state of those.

I could have written a whole script to init repositories and create certificates at start, but anyway, you still have to add a DNS record, and a virtual server in Httpd and Haproxy.

Housekeeping

Nothing cleans behind you there. It's your duty to erase used certificates (I think one year after expirate is good thing).

Remarks

This article and all the included scripts are under CC By licence. You can use and improve it, but you must provide some link back.

This was actually a lot of work. Approximately one month and a half. The french version is about 4590 words, while the english one is 3790 words long.

If you have remarks, comments, find bugs or think about an improvement in the scripts, please write me.