Repository navigation
Recover from a failed wireguard server start - #434
Merged
Merged
Conversation
george-angel
force-pushed
the
wiresteward-startup-recovery
branch
3 times, most recently
from
October 21, 2026 04:24
a92d18f to
87b2e84
Compare
ffilippopoulos
approved these changes
Oct 21, 2026
During boot the server can lose the race with the network: oauth discovery fails and the process exits before it removes the wireguard device it created. The device lives in the host network namespace rather than in the container, so the leftover device and nat rule made every later start fail, and systemd's default start rate limit turned that into a unit that stayed down until someone logged in to the node. Startup failures now return through Run so the deferred cleanup runs, and the servers that run in their own goroutines report a listener they cannot bind instead of exiting, which would leave the device behind the same way. Signals are trapped before the device is created, so a stop request during startup still cleans up. A start that finds its device already in place replaces it rather than refusing, which covers a crash or a kill that skipped cleanup. The unit keeps retrying without a start limit, so a late network costs time instead of the tunnel.
george-angel
force-pushed
the
wiresteward-startup-recovery
branch
from
October 21, 2026 06:54
87b2e84 to
2211f78
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
wiresteward now recovers from a failed start instead of staying down until
someone logs in to the node.
A node that boots before its network is ready loses the race: oauth discovery
times out, the process exits, and its wireguard device stays behind.
wiresteward runs with
--net=host, sowg0belongs to the host rather than thecontainer, and the cleanup that would have removed it never ran. Every later
start hit
file exists, and systemd gave up after 5 starts in 10s. The only wayback was deleting
wg0by hand.What changed
os.Exit, so the deviceand its nat rule always get cleaned up.
exiting, which leaked the device the same way.
cleans up.
wg0already there replaces it rather than refusing, whichcovers a crash or a
SIGKILL.StartLimitIntervalSec=0,RestartSec=15s), so aslow network costs time instead of the tunnel.
The alert in utilitywarehouse/system-alerts#276 asks for
state="active" == 0instead ofstate="failed" == 1for the same reason: afterthis change the unit never reaches
failed, so the old rule would go quiet on atunnel that is down.
govulncheckis red here for GO-2026-6809 in Go 1.26'scrypto/x509; masterfails the same job. The Go version change is being done separately, so this PR
stays on the current toolchain.