Apache2 with mod_md - Automatic Restart after Certificate Renewal

Abstract

For Apache HTTPD, mod_md is a good choice in automating certificate renewal with Let’s Encrypt. However, it is not fully automatic due to the need to schedule a server restart for the certificate change to become effective.

This post collects some ideas about how to implement an adequate restart mechanism.

Introduction

The standard way to achieve TLS connectivity on the Internet today is using certificates from Let’s Encrypt. Due to their short validity times, it is generally required to fully automate the renewal process.

Due to the high popularity of Let’s Encrypt, modern webservers integrate the renewal functionality and all common webservers can be combined with suitable tools to automate the renewal process.

Apache HTTPD (just called “Apache” in the following) is no longer as common a webserver as it used to be. However, it is still of interest for certain deployments because it offers a unique and rich feature set and is generally well-supported.

An experimental module mod_md for Apache automates most of the process flows for automating Let’s Encrypt certificate renewals. However, one key thing is missing: In order to actually apply a certificate change, Apache needs to be restarted.

While mod_md offers hooks for triggering actions in event of certificate changes, these run with the limited privileges of the www-data user who is generally not allowed to perform administrative changes like restarting the web server.

Basic Configuration for mod_md

Before digging into the main topic of this post, this section shows an example configuration for mod_md. On the top level of the config, an adequate global configuration for mod_md might look as follows. Substitute the masysma.net parts with suitable ones as applicable to your own domain:

ServerName             masysma.net
ServerAdmin            info@masysma.net
MDContactEmail         info@masysma.net
MDCertificateAgreement accepted
MDomain                masysma.net www.masysma.net
MDRenewMode            always

Check Status

After this setup, the /server-status page may be used to check the status of mod_md requesting certificates from Let’s Encrypt:

Screenshot of the server-status endpoint relevant section

Automatic Restart Ideas

This section contains some ideas about the restart which were not implemented. The next section then continues by showing what I ended up running with and which seems to work OK so far.

Restart the Server on a Fixed Schedule

Many servers have some sort of maintenance window or perform planned restarts due to software updates etc. In such cases, the issue of restarting Apache is handled implicitly and automatically.

It may also be valid and viable to setup such a schedule just for restarting the webserver. This could work the same way as described in Systemd Timer and Conditional Restart by modifying the ExecStart= to the following:

ExecStart=/usr/sbin/apachectl graceful

I don’t particularly like this approach because in general I think periodic restarting should be avoided as much as possible as it could hide a lot of errors that would better be resolved at the root of the issue.

However, his approach seems to be the most commonly deployed solution to the issue at hand.

Check the Apache Logs for Renewal Information

When performing a renewal, logs are generated in error.log. The following shows an example:

# grep -n "has been setup and changes" /var/log/apache2/error.log
8:[Sun Oct 20 14:56:40.129747 2024] [md:notice] [pid 32166:tid 32167] AH10059: The Managed Domain masysma.net has been setup and changes will be activated on next (graceful) server restart.
883:[Fri Dec 20 02:03:54.257738 2024] [md:notice] [pid 39086:tid 39087] AH10059: The Managed Domain masysma.net has been setup and changes will be activated on next (graceful) server restart.
1495:[Tue Feb 18 23:45:29.875672 2025] [md:notice] [pid 2508336:tid 2508337] AH10059: The Managed Domain masysma.net has been setup and changes will be activated on next (graceful) server restart.

One could then setup a timer to periodically check the logs and in event of detecting the relevant lines, trigger a restart. The difficult part is to keep track of which of the log entries have already been processed. This could e. g. be addressed by keeping a copy of the most recently processed log file and checking only the differences. Alternatively, it might be possible that there is some “restart-indicating” log line that could be detected with grep, too and the restart then be triggered only if the “has been setup and changes” lines appear after any restart-related lines.

An alternative would be to use the time of the last restart (if available) to compare with the log file and only if the log entry is newer than the last restart tigger a restart of Apache.

In general, I try to avoid building my system logic on log outputs because logs should typically be used for analysis purposes only and thus not be assumed to have a stable enough fromat as to build custom logic upon.

Check Server Status Information

While the renewal is pending, the following may be observed on the server-status page:

Managed Certificates
Domain  Names   Status  Valid   CA  Stapling    CheckAt Activity
masysma.net 
masysma.net www.masysma.net
    good    until 2025-03-20    LetsEncrypt     crt.sh[rsa]
    finished, 1 new certificate staged. Ongoing...

Note the “1 new certificate staged. Ongoing…” line.

Its presence might be checked from a script and this may be slightly more reliable compared to parsing the log file. Still, this server-status is probably also intended to be more used for informational purposes.

It is possible to get the information in an easier-to parse format by enabling the /md-status endpoint via SetHandler md-status. Refer to the Monitoring section of the Apache mod_md documentation: https://httpd.apache.org/docs/2.4/mod/mod_md.html. It may be easier to build a script on top of that as opposed to parsing the entire server-status.

Create a Marker File from within a Hook

While www-data may not restart the server all by itself, it could very well write to a dedicated location and the presence of a new file there could be detected, taken as trigger for the restart and then deleted.

One advantage of this implementation is that no “free-form” text data needs to be processed in order to trigger the restart. The disadvantage is that the external state keeping complicates the setup a little bit.

Allow www-data to restart the server with sudo

In general, a mere restart of the web server might not be considered critical and hence the particular command of choice could be allwed to user www-data with sudo allowing for a restart to be invoked from inside a hook.

Despite the hard-to-assess security implications there is also a race condition in that performing a restart inside the hook is aikin to killing a process from within as the restart might cause the hook execution to terminate.

In general I advise against this option especially considering that the other alternatives seem to be much better.

Estimate Restart Time based on Certificate Validity

By periodically querying the server it is easily possible to obtain the current certificate. It might be enough to check if the expiry date is nearing and then trigger a server restart unconditionally.

There is also a /.httpd/certificate-status endpoint which indicates the validity times in a JSON format making them easier to process compared to the real certificate files.

While a restart based on validity time is thus probably easy to implement it has multiple disadvantages:

It is very prone to flickering (i.e. continuous restarting) in event that something is amiss with the renewal process. Specifically, the renewal process must trigger before the restart process starts.
The effective certificate validity times may vary and decrease over time making it possible that the restart triggers too early even if it used to be aligned to the renewal process.
The process is more sensitive to the system clock and fluctuations in the system time may trigger server restarts which may not be desired.

I wanted to implement this solution but it turns out that the primary certificate file is not even updated by the automatic renewal. Instead, the new certificate is stored in a different location. This is a little bit easier to detect, cf. next section.

Systemd Timer and Conditional Restart

My current and preferred solution is as follows: Periodically check the existence of a “staged” certificate for renewal and if it exists trigger the server restart.

The following systemd unit files support this process. It may be necessary to adjust the domain name for your usage purposes. Also, as you can see, this is currently limited to one domain although it might be possible to extend the scheme to multiple domains. In such a case it might be helpful to factor out the “one-liner” into its own dedicated shell script.

# masysma-apache-cert-check.service
[Unit]
Description=Checks that the certificate of the server matches the one on-disk and restarts in event of mismatch.
Requires=network.target

[Service]
Type=oneshot
User=root
ExecStart=/bin/sh -ec 'if [ -e /etc/apache2/md/staging/masysma.net/pubcert.pem ]; then echo staging file found; apachectl graceful; fi'

# masysma-apache-cert-check.timer
[Unit]
Description=Trigger masysma-apache-cert-check.service daily

[Timer]
OnCalendar=daily

[Install]
WantedBy=multi-user.target

This task is run once per day in order to cause only an acceptable number of restarts in event that the logic would fail and behave as an “always restart” trigger. Also, this minimizes race conditions where the certificate renewal might just be going on but not be completed yet but already the filesystem state differs from the server state.

Advantages over the respective other solution ideas proposed above:

Alternative Idea	Advantage of the Systemd Timer and Conditional Restart Approach
Restart the Server on a Fixed Schedule	Does not hide other errors which might be automatically worked around by an unconditional restart.
Check the Apache Logs for Renewal Information	Independent of Log File format. Spamming and rotating the log won’t bypass the check.
Check Server Status Information	Independent of the Status Page format.
Create a Marker File from within a Hook	No need for external state keeping. This is probably the 2nd best option, though…
Allow www-data to restart the server with sudo	Smaller attack surface. Lower potential for race conditions.
Estimate Restart Time based on Certificate Validity	Lower Probability of flickering. Behaviour independent of system clock and renewal schedule.