I wrote a little daemon that collects data from our backyard weather station and stores it in a local database. The station has it's own cloud service that does this, but I feel like weather data is precious enough to keep a local copy.
Sometimes the daemon stops running. I don't know exactly why, and I could probably add some logging to find out, but right now I don't have time so I needed to do something quickly to avoid loosing data silently.
I'm using Systemd to run the daemon which gives you a lot of options when it comes to how to start, stop and restart something so I spent the five spare minutes I had this morning looking to see if there was something I could fiddle with to resolve this for now.
I know that when the daemon is stopped, restarting it has always worked, so it surpried me that the original unit file wasn't sufficient:
[Unit]
Description=Logs weather data from AmbientWeather stations to a database
After=multi-user.target
[Service]
User=jason
ExecStart=/usr/local/bin/awnet-receiver
Type=simple
Restart=always
[Install]
WantedBy=multi-user.target
With this file in place I'd still find the service stopped, so why didn't Systemd restart it?
Looking more closely at the logs I noticed that Systemd was restarting it, but after restarting and failing a certain number of times, Systemd gave up. These restarts were happening very quickly (many times a second), so if the problem was connecting to the database it's easy to imagine that the problem wasn't resolved within less than a second.
I reviewed the Systemd docs and found a few more parameters that might help, here's the updated [Service] section of the unit file:
[Service]
User=jason
ExecStart=/usr/local/bin/awnet-receiver
Type=simple
Restart=always
RestartSec=15
StartLimitBurst=20
StartLimitIntervalSec=300
I added the following parameters to see if it helps:
RestartSec - The number of seconds to wait before restarting
StartLimitBurst - The maximum number of times to try restarting
StartLimitIntervalSec - The maximum amount of time to allow for restarts
I don't know what the default values for these are, but based on what I saw in the logs this should spread-out the restart attempts a lot more. Given that I suspect database connectivity to be the problem, this should give that connectivity a lot more time to resolve before giving-up on starting the service.
I put-off doing anything about this for awhile because I wanted to take a more surgical approach to understanding exactly what is wrong and learning more about Systemd to see if there's more sophisticated ways of addressing it, but in the process I lost almost a month of data. Sometimes a tactical approach is better, especially if time is a factor, and you can always improve things once you've bought yourself more time.
Jason J. Gullickson, 2026