Crypto: add a CLI flag to disable auto generation of a KEK

[wim07101993]

Currently a KEK is generated on startup when there is none configured. This behavior is useful when developing/testing the system. However, once in production, this can create unwanted behavior when something is misconfigured.

To solve this we should add a CLI flag which allows to disable this behavior.

[grvijayan]

To solve this we should add a CLI flag which allows to disable this behavior.

Should disabled be the default behavior?

[wim07101993]

I would keep the default behavior so that it generates the kek. It creates less friction for someone who wants to get started. Once someone configures production, I would assume they would add the flag. It's not that the config is less secure. It's just less configurable.

However, I do not have a hard opinion on it.

[muhlemmer]

Field evidence for this, from wiring the GCP deployment (#734).

On Cloud Run the auto-generated KEK is not just "unwanted behavior when something is misconfigured" — it is the default outcome of a specific, easy misconfiguration, and it fails silently in the direction that loses data.

The misconfiguration is easy to hit. server.master_keys is a map[string]*MasterKeyConfig keyed by key ID, and Viper cannot populate map keys from the environment. Setting NEXTGEN_SERVER_MASTER_KEYS_PRIMARY_PRIVATE_KEY and ..._USE_FOR_ENCRYPTION does nothing at all. I probed loadConfig directly with both set:

master_keys count = 1
  key "master-key.pem": use_for_encryption=true private_key="" file="…/master-keys/master-key.pem"
  generated file on disk: master-key.pem

The env values were dropped and a fresh key was generated instead. An operator who reasonably expects env configuration to work gets auto-generation and no signal that their configuration was ignored.

The consequence is worse on Cloud Run than on a VM. $NEXTGEN_SERVER_DATA_DIR/master-keys is ephemeral container storage, so every instance and every revision mints a different KEK. Project KEKs wrapped by one instance's key are undecryptable by the next. There is no error at the point of damage — the failure surfaces later, as data that cannot be unwrapped.

The only signal today is one line on stderr at startup:

created server master key file at /var/lib/zitadel/nextgen-data/master-keys/master-key.pem (generated for local/dev only; configure server.master_keys for production)

That is easy to miss in aggregated logs, and it is emitted after the decision has already been made.

So the flag proposed here would have turned a silent data-loss path into a startup failure. Worth considering alongside it:

• Make the flag's disabled state the default for anything non-interactive, or at least loud — ERROR rather than an unlevelled stderr write.
• Warn when server.master_keys env vars are present but ignored, since that is the specific trap that leads here.

#1132 and #1133 work around this by always providing a key — mounted as a file into the master-key directory, which is discovered, since keys there are picked up by filename. Verified against ghcr.io/zitadel/nextgen:1.0.0-alpha.20: with a key present the server reaches server listening for requests and logs migrate keys to latest master key, and the dev-only warning is absent. So this is not blocking the deployment — but the trap is still armed for the next person.