--- section: 37 x-masysma-name: apache2_mod_md_automatic_restart title: Apache2 with mod_md - Automatic Restart after Certificate Renewal date: 2025/03/04 19:58:12 lang: en-US author: ["Linux-Fan, Ma_Sys.ma (info@masysma.net)"] keywords: ["kb", "apache", "cert", "acme", "letsencrypt", "mod_md"] x-masysma-version: 1.0.0 x-masysma-website: https://masysma.net/42/apache2_mod_md_automatic_restart.xhtml x-masysma-owned: 1 x-masysma-copyright: (c) 2025 Ma_Sys.ma . --- Abstract ======== For Apache HTTPD, `mod_md` is a good choice in automating certificate renewal with _Let's Encrypt_. However, it is not fully automatic due to the need to schedule a server restart for the certificate change to become effective. This post collects some ideas about how to implement an adequate restart mechanism. Introduction ============ The standard way to achieve TLS connectivity on the Internet today is using certificates from _Let's Encrypt_. Due to their short validity times, it is generally required to fully automate the renewal process. Due to the high popularity of Let's Encrypt, modern webservers integrate the renewal functionality and all common webservers can be combined with suitable tools to automate the renewal process. Apache HTTPD (just called “Apache” in the following) is no longer as common a webserver as it used to be. However, it is still of interest for certain deployments because it offers a unique and rich feature set and is generally well-supported. An experimental module `mod_md` for Apache automates most of the process flows for automating Let's Encrypt certificate renewals. However, one key thing is missing: In order to actually apply a certificate change, Apache needs to be restarted. While `mod_md` offers hooks for triggering actions in event of certificate changes, these run with the limited privileges of the `www-data` user who is generally not allowed to perform administrative changes like restarting the web server. Basic Configuration for mod_md ============================== Before digging into the main topic of this post, this section shows an example configuration for `mod_md`. On the top level of the config, an adequate global configuration for `mod_md` might look as follows. Substitute the `masysma.net` parts with suitable ones as applicable to your own domain: ServerName masysma.net ServerAdmin info@masysma.net MDContactEmail info@masysma.net MDCertificateAgreement accepted MDomain masysma.net www.masysma.net MDRenewMode always ## Check Status After this setup, the `/server-status` page may be used to check the status of `mod_md` requesting certificates from Let's Encrypt: ![Screenshot of the server-status endpoint relevant section](apache2_mod_md_automatic_restart_att/scr_server_status.png) ## See Also Examples of how to configure `mod_md` with more details compared to above configuration excerpt: * * * * The automatic restart part seems to not typically be solved by any of these guides. Automatic Restart Ideas ======================= This section contains some ideas about the restart which were not implemented. The next section then continues by showing what I ended up running with and which seems to work OK so far. ## Restart the Server on a Fixed Schedule Many servers have some sort of maintenance window or perform planned restarts due to software updates etc. In such cases, the issue of restarting Apache is handled implicitly and automatically. It may also be valid and viable to setup such a schedule _just_ for restarting the webserver. This could work the same way as described in _Systemd Timer and Conditional Restart_ by modifying the `ExecStart=` to the following: ExecStart=/usr/sbin/apachectl graceful I don't particularly like this approach because in general I think periodic restarting should be avoided as much as possible as it could hide a lot of errors that would better be resolved at the root of the issue. However, his approach seems to be the most commonly deployed solution to the issue at hand. ## Check the Apache Logs for Renewal Information When performing a renewal, logs are generated in `error.log`. The following shows an example: ~~~ # grep -n "has been setup and changes" /var/log/apache2/error.log 8:[Sun Oct 20 14:56:40.129747 2024] [md:notice] [pid 32166:tid 32167] AH10059: The Managed Domain masysma.net has been setup and changes will be activated on next (graceful) server restart. 883:[Fri Dec 20 02:03:54.257738 2024] [md:notice] [pid 39086:tid 39087] AH10059: The Managed Domain masysma.net has been setup and changes will be activated on next (graceful) server restart. 1495:[Tue Feb 18 23:45:29.875672 2025] [md:notice] [pid 2508336:tid 2508337] AH10059: The Managed Domain masysma.net has been setup and changes will be activated on next (graceful) server restart. ~~~ One could then setup a timer to periodically check the logs and in event of detecting the relevant lines, trigger a restart. The difficult part is to keep track of which of the log entries have already been processed. This could e. g. be addressed by keeping a copy of the most recently processed log file and checking only the differences. Alternatively, it might be possible that there is some “restart-indicating” log line that could be detected with `grep`, too and the restart then be triggered only if the “has been setup and changes” lines appear _after_ any restart-related lines. An alternative would be to use the time of the last restart (if available) to compare with the log file and only if the log entry is newer than the last restart tigger a restart of Apache. In general, I try to avoid building my system logic on log outputs because logs should typically be used for analysis purposes only and thus not be assumed to have a stable enough fromat as to build custom logic upon. ## Check Server Status Information While the renewal is pending, the following may be observed on the `server-status` page: Managed Certificates Domain Names Status Valid CA Stapling CheckAt Activity masysma.net masysma.net www.masysma.net good until 2025-03-20 LetsEncrypt crt.sh[rsa] finished, 1 new certificate staged. Ongoing... Note the “1 new certificate staged. Ongoing...” line. Its presence might be checked from a script and this may be slightly more reliable compared to parsing the log file. Still, this `server-status` is probably also intended to be more used for informational purposes. It is possible to get the information in an easier-to parse format by enabling the `/md-status` endpoint via `SetHandler md-status`. Refer to the _Monitoring_ section of the Apache mod_md documentation: . It may be easier to build a script on top of that as opposed to parsing the entire `server-status`. ## Create a Marker File from within a Hook While `www-data` may not restart the server all by itself, it could very well write to a dedicated location and the presence of a new file there could be detected, taken as trigger for the restart and then deleted. One advantage of this implementation is that no “free-form” text data needs to be processed in order to trigger the restart. The disadvantage is that the external state keeping complicates the setup a little bit. ### See also * A variant of this is implied here: Instead of a marker file, user `mhitza` proposes to send a message via dbus to a dedicated “restart” daemon. This would work as well but is probably more complex to implement compared to the marker file approach. ## Allow www-data to restart the server with sudo In general, a mere restart of the web server might not be considered critical and hence the particular command of choice could be allwed to user `www-data` with `sudo` allowing for a restart to be invoked from _inside_ a hook. Despite the hard-to-assess security implications there is also a race condition in that performing a restart _inside_ the hook is aikin to killing a process from within as the restart might cause the hook execution to terminate. In general I advise against this option especially considering that the other alternatives seem to be much better. ### See also * An example that is not using sudo but a dedicated setuid binary + script: ## Estimate Restart Time based on Certificate Validity By periodically querying the server it is easily possible to obtain the current certificate. It might be enough to check if the expiry date is nearing and then trigger a server restart unconditionally. There is also a `/.httpd/certificate-status` endpoint which indicates the validity times in a JSON format making them easier to process compared to the real certificate files. While a restart based on validity time is thus probably easy to implement it has multiple disadvantages: * It is very prone to flickering (i.e. continuous restarting) in event that something is amiss with the renewal process. Specifically, the renewal process must trigger _before_ the restart process starts. * The effective certificate validity times may vary and decrease over time making it possible that the restart triggers too early even if it used to be aligned to the renewal process. * The process is more sensitive to the system clock and fluctuations in the system time may trigger server restarts which may not be desired. I wanted to implement this solution but it turns out that the primary certificate file is not even updated by the automatic renewal. Instead, the new certificate is stored in a _different_ location. This is a little bit easier to detect, cf. next section. Systemd Timer and Conditional Restart ===================================== My current and preferred solution is as follows: Periodically check the existence of a “staged” certificate for renewal and if it exists trigger the server restart. The following systemd unit files support this process. It may be necessary to adjust the domain name for your usage purposes. Also, as you can see, this is currently limited to _one_ domain although it might be possible to extend the scheme to multiple domains. In such a case it might be helpful to factor out the “one-liner” into its own dedicated shell script. # masysma-apache-cert-check.service [Unit] Description=Checks that the certificate of the server matches the one on-disk and restarts in event of mismatch. Requires=network.target [Service] Type=oneshot User=root ExecStart=/bin/sh -ec 'if [ -e /etc/apache2/md/staging/masysma.net/pubcert.pem ]; then echo staging file found; apachectl graceful; fi' # masysma-apache-cert-check.timer [Unit] Description=Trigger masysma-apache-cert-check.service daily [Timer] OnCalendar=daily [Install] WantedBy=multi-user.target This task is run once per day in order to cause only an acceptable number of restarts in event that the logic would fail and behave as an “always restart” trigger. Also, this minimizes race conditions where the certificate renewal might just be going on but not be completed yet but already the filesystem state differs from the server state. Advantages over the respective other solution ideas proposed above: Alternative Idea Advantage of the Systemd Timer and Conditional Restart Approach --------------------------------------------------- -------------------------------------------------------------------------------------------------- Restart the Server on a Fixed Schedule Does not hide other errors which might be automatically worked around by an unconditional restart. Check the Apache Logs for Renewal Information Independent of Log File format. Spamming and rotating the log won't bypass the check. Check Server Status Information Independent of the Status Page format. Create a Marker File from within a Hook No need for external state keeping. This is probably the 2nd best option, though... Allow www-data to restart the server with sudo Smaller attack surface. Lower potential for race conditions. Estimate Restart Time based on Certificate Validity Lower Probability of flickering. Behaviour independent of system clock and renewal schedule. See Also ======== * Discussion about this topic in a bugreport for mod_md: