Skip to content

feat(chart): add startupProbe support to the pms container - #183

Open
alphayax wants to merge 2 commits into
plexinc:masterfrom
alphayax:feat/chart-startup-probe
Open

feat(chart): add startupProbe support to the pms container#183
alphayax wants to merge 2 commits into
plexinc:masterfrom
alphayax:feat/chart-startup-probe

Conversation

@alphayax

@alphayax alphayax commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Why

The chart supports livenessProbe and readinessProbe (added in #117) but not startupProbe. That gap forces a bad trade-off, and the reason is already written down in #117's own review discussion:

if it's a particularly long db migration, the restart will just result in it being run again from the beginning and will indefinitely until the migration is allowed enough time to complete.

That is exactly right, and it is the problem. PMS answers 503 on /identity in two completely different situations:

  1. while it runs database migrations at startup — transient, can take many minutes on a large library, and restarting makes it worse because the migration restarts from the beginning;
  2. when the server is wedged — for example the Butler database-backup deadlock reported on the forums, where PMS keeps its process alive and answers 503 to everything indefinitely.

A liveness probe is the only thing that catches (2), because the process never exits. But with only a liveness probe available, its budget has to be stretched to cover the worst case of (1) — otherwise it turns a slow migration into a restart loop. So users end up either with no probe at all, or with a probe deliberately detuned to the point of being slow to help.

startupProbe is the Kubernetes primitive that separates the two: it suspends the liveness probe until the container answers once, then hands over. With it, the same deployment can tolerate a long migration and keep a tight liveness probe afterwards.

I hit this in production: PMS deadlocked on its nightly database backup and served 503 to every request for over 13 hours while the container stayed Running. Adding a liveness probe fixes the outage, but without a startupProbe I had to widen it to ~32 minutes to stay safe across version upgrades, which gives up most of the benefit.

What this changes

Four files, and no behaviour change for anyone who does not set the new value:

Verification

  • helm template with no values: output is byte-identical to the unpatched chart apart from the helm.sh/chart version label. No probe key is injected by default.
  • helm template with pms.startupProbe and pms.livenessProbe both set: both render, correctly nested under the container, readinessProbe correctly absent.
  • The rendered manifest passes kubectl apply --dry-run=server against a live cluster.

Note on the example values

The commented example uses periodSeconds: 10 / failureThreshold: 180, i.e. a 30-minute startup budget. That is deliberately generous: the cost of an over-long startup probe is a slow first start, while the cost of one that is too short is a migration restarted from scratch, repeatedly. Happy to change the numbers if you would rather the example be less conservative.

The chart supports livenessProbe and readinessProbe (plexinc#117) but not
startupProbe, which forces a bad trade-off for anyone using probes.

PMS answers 503 on /identity in two very different situations:

  1. while it runs database migrations at startup — transient, can take
     many minutes on a large library, and restarting makes it worse
     because the migration starts over;
  2. when the server is wedged and will never recover on its own.

Only a liveness probe catches (2), because the process stays alive. But
with no startupProbe available, the liveness budget has to be stretched
to cover the worst case of (1), or it turns a slow migration into a
restart loop — the exact risk raised in the review of plexinc#117:

  "if it's a particularly long db migration, the restart will just
   result in it being run again from the beginning and will
   indefinitely until the migration is allowed enough time to
   complete."

startupProbe is the primitive that separates the two: it holds the
liveness probe off until the container answers once, then hands over.
With it, the same deployment can tolerate a long migration and still
keep a tight liveness probe afterwards.

Changes, with no behaviour change unless the new value is set:

  - templates/statefulset.yaml: one {{- with }} block mirroring the
    existing two, placed before livenessProbe.
  - values.yaml: startupProbe: {} with a commented-out example, in the
    same style as its neighbours.
  - README.md: helm-docs row.
  - Chart.yaml: 1.7.1 -> 1.8.0, matching how plexinc#168, plexinc#170 and plexinc#175 each
    bumped the minor version for an additive feature.

Verified: `helm template` with no values renders byte-identically to
before apart from the helm.sh/chart label; with startupProbe and
livenessProbe both set, both render correctly nested and readinessProbe
stays absent; the rendered manifest passes kubectl apply
--dry-run=server against a live cluster.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@alphayax
alphayax requested a review from a team as a code owner September 2, 2026 02:50
@alphayax
alphayax requested review from MarshallAsch and cookandy and removed request for a team September 2, 2026 02:50

@cilindrox cilindrox left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - thanks @alphayax - just some comments on the wording.

Comment thread charts/plex-media-server/values.yaml Outdated
@cilindrox cilindrox self-assigned this Sep 2, 2026
Co-authored-by: Gaston Festari <cilindrox@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants