monotux.tech

Retrying With NGINX & systemd

• systemd, NGINX

After upgrading to Fedora 33 on my home server1, I noticed that my web server (NGINX) won’t come up after boot. This surprised me as I hadn’t made any changes to my configuration in a while, and when looking into the error message it was complaining that it couldn’t resolve one of the upstream sources.

It could look like this:

# journalctl -u nginx.service
-- Reboot --
mar 20 07:02:20 hilda systemd[1]: Starting The nginx HTTP and reverse proxy server...
mar 20 07:02:20 hilda nginx[1000]: nginx: [emerg] host not found in upstream "grafana.services.home.arpa:3000" in /etc/nginx/sites.d/grafana:2
mar 20 07:02:20 hilda nginx[1000]: nginx: configuration file /etc/nginx/nginx.conf test failed
mar 20 07:02:20 hilda systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
mar 20 07:02:20 hilda systemd[1]: nginx.service: Failed with result 'exit-code'.
mar 20 07:02:20 hilda systemd[1]: Failed to start The nginx HTTP and reverse proxy server.

…but my DNS server is on another machine, still up and running? Interesting!

But instead of really trying to understand why I ran into this problem, I decided just to fix it’s symptoms. A quick session on Google later and I came up with this:

# systemctl edit --full nginx.service
[Unit]
Description=The nginx HTTP and reverse proxy server
After=network.target remote-fs.target nss-lookup.target

# Here
StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
Type=forking
PIDFile=/run/nginx.pid
# Nginx will fail to start if /run/nginx.pid already exists but has the wrong
# SELinux context. This might happen when running `nginx -t` from the cmdline.
# https://bugzilla.redhat.com/show_bug.cgi?id=1268621
ExecStartPre=/usr/bin/rm -f /run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
KillSignal=SIGQUIT
TimeoutStopSec=5
KillMode=mixed
PrivateTmp=true

# ...and here!
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

I stole this from the first result on google, for which I’m forever grateful.

# journalctl -u nginx.service
-- Reboot --
mar 20 08:01:43 hilda systemd[1]: Starting The nginx HTTP and reverse proxy server...
mar 20 08:01:43 hilda nginx[892]: nginx: [emerg] host not found in upstream "grafana.services.home.arpa:3000" in /etc/nginx/sites.d/grafana:2
mar 20 08:01:43 hilda nginx[892]: nginx: configuration file /etc/nginx/nginx.conf test failed
mar 20 08:01:43 hilda systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
mar 20 08:01:43 hilda systemd[1]: nginx.service: Failed with result 'exit-code'.
mar 20 08:01:43 hilda systemd[1]: Failed to start The nginx HTTP and reverse proxy server.
mar 20 08:01:48 hilda systemd[1]: nginx.service: Scheduled restart job, restart counter is at 1.
mar 20 08:01:48 hilda systemd[1]: Stopped The nginx HTTP and reverse proxy server.
mar 20 08:01:48 hilda systemd[1]: Starting The nginx HTTP and reverse proxy server...
mar 20 08:01:48 hilda nginx[1900]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
mar 20 08:01:48 hilda nginx[1900]: nginx: configuration file /etc/nginx/nginx.conf test is successful
mar 20 08:01:48 hilda systemd[1]: Started The nginx HTTP and reverse proxy server.

  1. Yes, I unironically run Fedora Server on my home server. It’s auto patching and reboots every other day or so, and all my containers are auto updating as well. It’s wonderful! ↩︎