hydra: use BindsTo= and auto-restart for hydra-init to fix services stuck on dependency failure

[Senthil455]

Summary

Fixes #367 - Hydra services never restart after hydra-init.service fails.

Problem

When hydra-init.service fails transiently (e.g. database not yet ready), hydra-server, hydra-evaluator, and hydra-queue-runner all fail with Dependency failed because they have Requires=hydra-init.service. The services never recover because:

1. hydra-init has no auto-restart: It is Type=oneshot with RemainAfterExit=true but no Restart= setting, so it defaults to Restart=no. Once it fails, it stays permanently in failed state.
2. No restart propagation: Even if hydra-init were retried and eventually succeeded, the dependent services' start jobs already failed. With Requires=, systemd does not automatically restart dependents when the dependency recovers.

Solution

1. Auto-restart hydra-init on failure

serviceConfig = {
  Restart = "on-failure";
  RestartSec = "5s";
};

This ensures transient failures (database not ready, network hiccup) are retried automatically.

2. Use BindsTo= instead of Requires=

bindsTo = [ "hydra-init.service" ];

BindsTo= creates a stronger lifecycle binding: when hydra-init transitions from failed to active (after a successful retry), systemd automatically restarts the bound services. The upstream module's Requires= stays in place (harmless, BindsTo is a superset).

Files Changed

• build/hydra.nix (production): Added serviceConfig.Restart and RestartSec to hydra-init; added bindsTo to hydra-server, hydra-evaluator, and hydra-queue-runner
• non-critical-infra/hosts/staging-hydra/hydra.nix (staging): Same changes for hydra-init, hydra-server, and hydra-evaluator (staging uses the separate hydra-queue-runner-dev service from the Rust queue-runner module, which does not have Requires=hydra-init.service)