[Feature Proposal] Native Breeze System Health — MSP diagnostics and opt-in customer status

[fabicarvano]

Summary

I'd like to propose Breeze System Health: a native, read-only page showing the operational health of the Breeze platform itself, distinct from the health of managed endpoints or customer infrastructure.

The goal is to reuse Breeze's existing observability data and provide two deliberately different experiences:

1. MSP / platform administrators: an easily accessible operational overview inside Breeze, without requiring Grafana for routine checks.
2. Customer Portal users: a small, customer-safe service status view, visible only when the MSP explicitly enables it for that organization.

This is a discussion to validate the direction and boundaries before writing an implementation spec.

Why this would be useful

MSP operations

Breeze already exposes health probes, metrics, and infrastructure alerts, but routine platform-health checks may still mean opening Grafana, querying Prometheus, or inspecting individual services.

A native page would help administrators answer, at a glance:

• Is the Breeze API available and responding normally?
• Are PostgreSQL and Redis healthy?
• Are required workers and background jobs running?
• Are requests slowing down, errors increasing, or queues backing up?
• Where available, are host capacity indicators (CPU, memory, disk) approaching limits?

This could also be a foundation for future, evidence-based AI-assisted performance recommendations. For example, if latency and sustained memory pressure correlate, an assistant could suggest investigating memory, swapping, database connection saturation, or worker capacity. This is a future direction, not part of the initial scope, and recommendations must not assert causation without supporting evidence or apply infrastructure changes automatically.

Customer transparency

Customers benefit from knowing whether the Breeze service they use is operational, degraded, or unavailable, without seeing the MSP's internal infrastructure.

The question answered in the Portal is "Is Breeze operating normally?", not "Are my monitored machines healthy?"

Existing foundations (to reuse, not duplicate)

Relevant current repository surfaces include:

• Optional observability overlay: docker-compose.monitoring.yml (Prometheus, Grafana, Alertmanager, Loki/Promtail, Redis/PostgreSQL exporters).
• Scrape targets and rules: monitoring/prometheus.yml and monitoring/rules/breeze-rules.yml.
• API liveness and dependency readiness: /health and /health/ready (the latter checks database, Redis, and required consumers). These are intentionally different signals.
• Existing platform-admin System page: /admin/system, currently with Connections and Deprecations tabs (issue System page — connection status + deprecations #6768 and its approved design). The Connections tab describes configuration, not live health.

The proposal is to reuse canonical signals, not create another monitoring engine or require Grafana for this UI. We should also account for optional worker-split and managed-database deployment modes, and not assume host metrics are available in every deployment.

Proposed initial experience

A. Platform administrator — System → Health

Potential new tab in the existing /admin/system page, rather than a separate admin section.

• Platform-wide summary: Operational / Degraded / Unavailable / Unknown.
• Component-level health for API, dependencies, background processing, and any supported host/capacity metrics.
• Last successful observation / freshness indication; do not show stale data as healthy.
• Optional technical drill-down for admins only, using existing signals. No customer device/fleet monitoring on this page.
• If the optional monitoring stack is disabled: show "System monitoring is disabled", not an empty dashboard.

B. Customer Portal — simple service status

A minimal read-only view backed by a customer-safe API projection, for example:

{
  "overall": "degraded",
  "components": [
    { "id": "api", "status": "operational" },
    { "id": "background-processing", "status": "degraded" }
  ],
  "checkedAt": "2026-10-30T12:00:00Z"
}

Illustrative only; component names and final status semantics require a spec. The Portal should not receive raw Prometheus queries, internal service URLs, hostnames, logs, metrics labels, credentials, database details, or unfiltered admin DTOs.

C. Per-organization MSP visibility control

The MSP should be able to enable/disable System Health in the Customer Portal per organization.

• An independent, explicit opt-in for the System Health visibility family, OFF by default.
• The customer Portal navigation and underlying API access must both be gated.
• Resolve organization scope from the authenticated Portal session, not from a client-supplied authoritative orgId.
• When off, the feature and its data are inaccessible to that organization.
• If monitoring is disabled globally, hide the feature for customers regardless of the organization's setting.
• Platform health data is generally deployment-wide; per-org configuration controls visibility, not a claim that each organization has a separate infrastructure health state.

This follows the maintainer's guidance in #8775 on one explicit opt-in per data family, default-off controls, field allowlists, and customer-safe projections. #8775 concerns visibility into the customer's environment; this proposal concerns availability of the Breeze platform itself, so the scopes should remain distinct.

Status semantics and failure handling

We should explicitly distinguish:

• Operational: fresh authoritative signals indicate the component is healthy.
• Degraded: it is available but meaningful performance/functional health criteria are failing.
• Unavailable: the component/service is confirmed unavailable.
• Unknown: monitoring is stale, unreachable, or insufficient to make a reliable claim.
• Disabled / not configured: an optional capability is intentionally not active; this is not an outage.

An alert about managed agents/devices must not automatically make the Breeze platform unhealthy. Also, the existing /health liveness probe must not be treated as equivalent to /health/ready dependency readiness.

One architectural question is how an API-hosted status endpoint behaves during an API-wide outage: it should not misleadingly indicate "operational" or imply guaranteed external status-page availability when the serving application itself is down.

Out of scope for the first version

• Monitoring customer endpoints or network assets (already separate Breeze capabilities).
• Exposing full Grafana/Prometheus or infrastructure diagnostics in the Customer Portal.
• Building a replacement observability stack.
• AI recommendations or automated remediation.
• SLA claims, historical uptime guarantees, or incident communication workflows unless separately specified.

Questions for the maintainer / community

1. Does System → Health within the existing platform-admin System page seem like the right home?
2. Should the customer-facing status live as a small Portal page/card, with a dedicated customer-safe API?
3. Does an independent, default-off, per-organization Portal visibility flag fit the existing Portal settings model?
4. Which platform components should be included in a useful MVP, and which existing signals should be authoritative for each?
5. Should the native view depend on the optional Prometheus stack, or should basic readiness be available even without it, while advanced diagnostics remain optional?
6. Are there existing plans, specs, or ongoing work we should extend instead of introducing overlapping mechanisms?

If the direction is useful, the next step would be a focused spec identifying source metrics/probes, permissions, the exact allowlisted Portal response, status/freshness rules, and tests for disabled monitoring, stale signals, and cross-organization access.

Related: #6768 (System page), #8775 (Portal visibility principles), docker-compose.monitoring.yml, docs/operations/health-probes.md.

[ToddHebebrand]

Thanks. Keeping platform health separate from the customer's environment is the right line, and your status semantics (especially Unknown and Disabled as distinct from an outage) are good as written. I'd take the admin tab to spec now and hold the portal half.

Q1, home: yes, a Health tab on /admin/system. apps/api/src/routes/admin/systemConnectionHealth.ts is the pattern to follow: read-only, platform-admin only through adminRoutes, no tenant data, and a Redis outage reported in the body rather than as a 500.

Q5, Prometheus: don't depend on it. The monitoring overlay is optional and the API has no Prometheus client today. Build the base on what the API can already see, such as the readiness evaluator behind /health/ready (apps/api/src/index.ts:597-605) and job heartbeats like the connection-health ones. Prometheus can be optional enrichment later. With several API replicas, an in-process reading only describes the replica that answered, so the spec should say which signals are shared and which are per process.

Q4, MVP: API readiness, PostgreSQL, Redis, queue depth and job heartbeats. I'd leave host CPU, memory and disk out of the first version, since the API can't see them in every deployment.

Q2 and Q3, portal: not decided yet, so please make it a separate later wave. Three things need answers first:

• Audience on hosted: /admin/system is platform-admin only, so an MSP on a hosted deployment wouldn't see the admin tab at all. They'd be enabling a customer status for a service they can't inspect or act on. The spec should say who this is for in each deployment mode.
• Usefulness: you named the outage problem. The portal uses the same API, so during the failure that matters most it can't report anything. It may end up only ever saying "operational".
• Exposure: even a small component list tells customers about the MSP's tooling, and the portal is white-labelled.

If it does go ahead, the flag model fits: its own enable* column on portal_branding, listed in PORTAL_VISIBILITY_FLAG_KEYS (apps/api/src/services/portal/portalFlags.ts:11) and kept outside "Enable all" through PORTAL_SENSITIVE_FLAG_KEYS.

Q6, overlap: extend the System page from #6768. I didn't find an open issue with this in its title.

Next step: a spec for the admin tab only, with the portal as an open question at the end.

Status: direction accepted for the admin tab, portal undecided.