it would be great if, as part of the SSL integration, ikiwiki-hosting would call up on the [letsencrypt.org gods][] to install proper certs.

[[!toc]]

First implementation
--------------------

so far I have done this by hand, using the main client (previously called letsencrypt, now called certbot, those instructions may need to be updated):

1. define some handy variables:

        domain=example.com
        email=nobody@$domain
        username=w-example
        webroot=/home/$username/public_html/

2. install the letsencrypt client software:

   * if you are on Debian sid, stretch or jessie (with backports):

            apt install certbot

   * otherwise, run the magic commands:

            git clone https://github.com/certbot/certbot
            cd certbot
            ./cerbot-auto --help

3. request a certificate:

        sudo -s
        [ -d .git ] && source venv/bin/activate # only use virtualenv if checking out from git
        certbot certonly --webroot \
          --agree-tos \
          --webroot-path="$webroot" \
          --domains $domain --email $email

    ... notice how you need to change the two last lines for your
    environment.

4. this will create a certificate in
      `/etc/letsencrypt/live/example.com`. install it where ikiwiki
      expects it:

        ln -s /etc/letsencrypt/live/$domain/cert.pem /etc/ikiwiki-hosting/config/$username/ssl.crt
        ln -s /etc/letsencrypt/live/$domain/fullchain.pem /etc/ikiwiki-hosting/config/$username/ssl.chain
        ln -s /etc/letsencrypt/live/$domain/privkey.pem /etc/ikiwiki-hosting/config/$username/ssl.key

5. regenerate the apache configs and restart apache:

        ikisite enable $domain
        service apache2 reload # or restart?

Ikiwiki hosting would need to perform steps 3 to 5, and probably have
a `Recommends` on certbot. -- [[anarcat]]

Potential clients
-----------------

Finally, note that there is now a fairly [large set of Let's Encrypt clients][], here's a short list (those have not been reviewed or tested for compatiblity with ikiwiki-hosting):

 * [dehydrated][], bash-only client that should work all Debian
   releases back to Wheezy (formerly known as letsencrypt.sh, in [Debian jessie, stretch and sid](https://tracker.debian.org/pkg/letsencrypt.sh))
 * [simp_le][], much simpler implementation in Python ([not in Debian](https://github.com/kuba/simp_le/issues/69))
 * [acme][], root-less, more minimal implementation than the official
   client, in Go ([in Debian stretch and sid](https://tracker.debian.org/pkg/acmetool))
 * [lego][], root-less, fairly featureful, also [in Debian](https://tracker.debian.org/pkg/lego)
 * [acme-tiny][] is a very small client (~200 lines of python code)
   that aims to be easily audited and to do only one thing well. for
   example, you need to generate your RSA key yourself, which may
   actually be a good thing considering how ikiwiki-hosting works
   ([in Debian stretch and sid][])

 [letsencrypt.org gods]: https://letsencrypt.org
 [plugin]: https://github.com/letsencrypt/letsencrypt/wiki/Plugins
 [large set of Let's Encrypt clients]: https://github.com/letsencrypt/letsencrypt/wiki/Links
 [simp_le]: https://github.com/kuba/simp_le
 [dehydrated]: https://dehydrated.de/
 [acme]: https://github.com/hlandau/acme
 [letsacme]: https://github.com/neurobin/letsacme
 [acme-tiny]: https://github.com/diafygi/acme-tiny/
 [ITP for acmetool]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=817091
 [in Debian stretch and sid]: https://tracker.debian.org/pkg/acme-tiny
 [lego]: https://github.com/xenolf/lego

Note that I have also considered simply using the `--apache` plugin to just have LE manage all those certs itself. This could conflict with ikiwiki-hosting, but since it runs fairly frequently, it could just win in the end. It's not a great solution for sites managed by the control panel, but for my use case, it could just work. Unforunately, it [parses configs in sites-available instead of just sites-enabled](https://github.com/certbot/certbot/issues/3545) which breaks it for me. --[[anarcat]]

Renewals
--------

Also, renewals need to be processed in some way. It seems that it's possible to just run:

    letsencrypt -c /etc/letsenctypt/renewal/$domain.conf --renew-by-default

> `--renew-by-default` renews every time it's run; 
> `--keep-until-expiring` instead waits until 30 days before expiry and 
> renews then. I think that's the simpler approach, and the same
> letsencrypt command line can then both obtain the original cert and renew
> it. --[[Joey]]

>> True, makes a lot more sense! --[[anarcat]]

>>> Actually, the `-c /etc/letsenctypt/renewal/$domain.conf` doesn't work at all. The 0.4 client features a new `renew` command that does the right thing (including renewing only if necessary). I tested it here and it seems to work fairly well. My complete update run looks more something like this now: --[[anarcat]]

>>>     letsencrypt renew --non-interactive
>>>     git -C /etc add -A letsencrypt
>>>     git commit -m"renew all certs"

>>>> So the software name changed: it's certbot now. And yes, the renew command seems to work better. There is a `--post-hook` that can restart apache and do various things with certs, but unfortunately prior to 0.7, it runs all the time. Normally, the `--post-hook` is one of those variables that would be saved on renewal, but I haven't tested this because I don't want to have that hook run twice a day. So I'm waiting for this to hit backports before automating this completely, especially since I do not want to diverge from the package's `/etc/cron.d/certbot` cronjob. My test renewal line now looks like:
>>>>
>>>>     certbot --dry-run renew --renew-hook 'service apache2 reload; etckeeper commit "certbot renewed domains: $RENEWED_DOMAINS"'
>>>> 
>>>> I'll test this again in a few weeks, hopefully when the [new package hits backports](http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=826147). --[[anarcat]]

>>>> The software has caught up. To avoid customizing the config file, I had to put the renew-hook in the `/etc/letsencrypt/cli.ini` config file, as such:
>>>> 
>>>>     renew-hook = service apache2 reload & etckeeper commit "certbot renewed domains: $RENEWED_DOMAINS"
>>>>
>>>> Notice the `&` instead of `;` because the latter breaks the LE config file parser. With that, the configured domains are renewed automatically! To add new domains, I just need to repeat steps 3 to 5 above. Note that extra webroots can be specified with:
>>>>
>>>>     certbot certonly -w /home/foo -d foo.example.com -w /home/bar -d bar.example.com
>>>>
>>>> I.e. the last webroot path is attached to the next domain specified. Next step is to code this in Perl, I guess? --[[anarcat]]

Rate limits
-----------

Regarding [rate limits](https://community.letsencrypt.org/t/rate-limits-for-lets-encrypt/6769), they don't seem to be a problem unless you run a really huge site (500 registrations per 3 hours per IP!). The limits are:

 * 100 Names/Certificate (how many domain names you can include in a single certificate)
 * 20 Certificates per Domain per week
 * 500 Registrations/IP address per 3 hours
 * 300 Pending Authorizations/Account per week

> That seems to be 5 certificates per domain, not subdomain. 
> Eg, I asked for tmp.kitenet.net 5 times and now it's not allowing
> getting a cert for downloads.kitenet.net.
> 
> So, this seems a pretty big problem; ikiwiki-hosting sites often use
> subdomains, and if so, quite likely use more than 5. It could keep a site
> site http-only until it got a cert. But with 100 sites, that would be nearly
> half a year of delay! (Update: With the new 20 certs/domain/week, that
> drops to 5 weeks delay.)

> And, what about renewal, wouldn't that be rate limited too?
> (Update: No, it seems that renewals are not rate limited in the 20
> certs/domain/week bucket.)
> 
> Seems like getting a cert for 100 subdomains under branchable.com (eg),
> and then moving onto the next cert etc would be the best we could do. But
> that's a lot more complicated. In particular, renewal needs to use the
> same set of 100 domains, and if one of the domains is deleted in the
> meantime, renewal of those 100 probably won't work, and a new certificate
> would need to be generated for the 99 remaining.
> --[[Joey]]

>> Ouch, yes, that is a significant problem for branchable!!! Maybe this should be brought up upstream... I wonder if this couldn't be a way to finance letsencrypt, to have larger providers pay a monthly fee for the service, and bypass those limits. --[[anarcat]]

>>> [WP.com switched to LE](https://en.blog.wordpress.com/2016/04/08/https-everywhere-encryption-for-all-wordpress-com-sites/), so there's probably a way around those. --[[anarcat]]

>>> I [asked around](https://news.ycombinator.com/item?id=11460480) and the limits were bumped to 20 certs/domain/week recently. Also, they can tweak the limits upon request. --[[anarcat]

Discussion
----------

Oh, and this could be a [plugin][], not sure if that would make sense, but it's the way it works on that side of the world. --[[anarcat]]

> I would much rather use something small enough to be reviewable, particularly if we're
> going to have to run it unattended as root; the official Let's Encrypt client is *huge*.
> We don't have to worry about portability to non-Debian environments or non-Apache
> web servers, which takes away a lot of the reason to use something so big. --smcv

>> Kind of agreed, but I'm more interested in ameanability to automation
>> than overall code size. <https://github.com/hlandau/acme> is one client
>> that seems promising. At the moment only letsencrypt is seems packaged
>> in debian yet and that's the first bar. --[[Joey]]

>>> Agreed as well. Just scratching an itch here... The `acme`
>>> client looks interesting, but it's go, which brings a whole set of
>>> dependencies in. But I like its philosophy and it could integrate
>>> well. Also note that the above instructions fail to take into account
>>> the canonical and alias names of the site, which breaks openid
>>> logins... Something to fix in my future renewals... --[[anarcat]]

Here's an interesting [puppet implementation][] that mentions another (!)
golang client ([lego][]), which has the interesting property of
supporting DNS challenges. The puppet code itself uses the shell script
previously mentioned. --[[anarcat]]

 [puppet implementation]: https://bzed.de/post/2016/02/bzed-letsencrypt_puppet_module/
 [lego]: https://github.com/xenolf/lego


Note that Let's Encrypt started [supporting wildcard certificates](https://arstechnica.com/information-technology/2017/07/lets-encrypt-to-start-offering-free-wildcard-certificates-for-https/) recently, but only with DNS challenges.. --[[anarcat]]

---- 

## how to only enable ssl once a certificate is available

Tip from [[Joey]]:

Letsencrypt has a bit of a chicken and egg problem, in that the http server
has to be up to get the certificate, but the apache site config file
has to contain both the http and the https vhost. And the https vhost needs
to point to cert files not available until letsencrypt has run.

smcv was thinking about dealing with that by generating a temporary
self-signed cert first. But, I don't like the possibility that a http site
would come up with such a cert during normal operation.

I found a way around this when adding letsencrypt support to propellor:
Make the apache site config file contain only the http vhost, and use
IncludeOptional to include another file which contains the https vhost.
Generate that file only once letsencrypt has generated the cert.

Note that IncludeOptional has to be used with a wildcard to avoid it being
an error if the included file doesn't exist. So, something like:

	IncludeOptional /etc/apache2/sites-available/$domain/*.conf

>> The way I did this here was to generate the cert with the HTTP-only vhost, and only after the cert was generated create the HTTPS vhost... Can't you  have a modified HTTPS vhost generated only *after* the cert is generated? --[[anarcat]]

>>> Yes, I think your approach at the top works; `ikisite enable $domain`
>>> notices the certs and regenerates an apache config to use them.
>>> --[[Joey]]

## Worked design

Add a `use_letsencrypt` config setting in the ikiwiki-hosting plugin.
(done)

Make /etc/ikiwiki-hosting/config/$username/domain.{crt,key,chain} be
used for a domain when available, instead of ssl.{crt,key,chain}.
This allows urlaliases to each get their own letsencrypt cert. (done)

Add `ikisite letsencrypt site`, which first sets `use_letsencrypt`
and then runs certbot (or whatever, it can be made configurable if
desired) to try to get a certificate for all of the site's domain(s).
If certbot succeeds, it symlinks the cert into place,
and calls `ikisite enable site` to update the web server config
to use the cert. (done)

Add `ikisite letsnotencrypt site` which undoes that. (done)

Then add a daily cron job that looks for sites that have `use_letsencrypt`
set, but don't have letsencrypt certs. (Check that the cert files are
symlinks to tell). For each such site, it runs `ikisite letsencrypt`
to try again to get the cert. (done in `ikisite maintaincerts`)

Another cron job can run `certbot renew` to handle all renewals.
Seems this doesn't need to rejigger the certificate symlinks or the
apache config for renewed sites, so it should be nice and simple.
(done in `ikisite maintaincerts`)

`ikisite domains` will need to check `use_letsencrypt`. If it's set, and a
new domain is being added to a site, or a domain removed, it needs to
delete the flag file and run `ikisite letsencrypt` to (try to) get a new
certificate. What if that fails? It might be possible to keep using the old
letsencrypt certificate in some cases, eg when adding
an alias. Certianly not in the case where the site's primary domain has
changed. The safe option is certainly to remove the old letsencrypt
certificate if it fails, and `ikisite enable` will then disable ssl support
for that site, until the cron job later succeeds in getting a certificate.
(done)

Once this is tested and working well, it can be made the default,
by setting `use_letsencrypt` in the autosetup/ files. For that to work,
every ikisite command that creates a site (including branching,
restoring from backup, etc) should run `ikisite letsencrypt` at the end
when the site has `use_letsencrypt` configured. It can take a minute or so
for letsencrypt to run, so best to do that in the background so as to not
slow down site creation etc.

> Note: there is already a cron job that runs certbot renew in the Debian package,
> watch out for dupes... Otherwise: design sounds okay, although I'm not sure 
> why the flag file is necessary. --[[anarcat]]

## issues

* If a site's urlalias includes some domain that is not in the DNS,
  or no longer points to the host, this is normally not a problem.
  But, it will prevent certbot getting a cert for that site, because
  it cannot verify it.

  This affects both `ikisite letsencrypt` and `ikisite domains`.

  `ikisite domains` does not currently ensure that all urlaliases are set
  in the dns. This check could be added, it might cause some problem
  though.

  Note that this is only a problem because the urlaliases are passed
  to certbot as --host. That is needed because
  `templates/apache-sitealias.tmpl` includes a https configuration.
  That lets redirection from the https urlalias to the main url work.

  But also, if a cert is issued for domains A and B, and B later falls out
  of the DNS, renewal of the cert will fail. So this can lead to renewal
  failures later down the line.

  Partial fix would be re-running certbot without the urlaliases if it
  fails with them. Better to have the main url encrypted even if
  redirection to it cannot be. But, that would leave the
  apache-sitealias.tmpl generating a config that uses the wrong cert for
  https, which is bad if some of the urlaliases are accessible in the dns.
  So, this fix needs a way to prevent that template from using https in
  this case.

  Alternatively, could improve the UI so the site admin can see when
  letsencrypt failed, and why. (Worth doing in any case, but I'm not sure
  where to put it in the UI.)

  Or, could make apache-sitealias.tmpl use per-domain cert files. This
  avoids all this complexity. Only the Registration/IP rate limit might be
  a problem with doing this, but it's sufficiently high. (done)

* Sites with a apache.conf.tmpl password protecting the site 
  will not be able to use letencrypt, unless it's relaxed to let
  .well-known be accessed w/o password. This can be fixed by
  adding this to the end of their apache.conf.tmpl file:

	<Directory <TMPL_VAR HOME>/public_html/.well-known>
		AuthType None
		Require all granted
	</Directory>
