chore: replace checked-in certificates with single-purpose generated ones - #4882
Merged
Conversation
github-actions
Bot
added
the
Sensitive
Removes the 37 certificate artifacts that were committed to the repository, together with the two generation configurations they were produced from (keystore/docker/generate_cert.sh and keystore/client_cert/openssl.conf). Several of these were not replaced by anything, because nothing used them: - keystore/selfsigned/localhost-untrusted.* had no references in the repository - keystore/local_ca/zowe-dev-ca.cer was byte-identical to keystore/localhost/localca.cer - keystore/localhost/Zowe_Service_..._Certificate_Authority_.cer duplicated the leaf certificate already exported as localhost.keystore.cer - keystore/selfsigned/localhost.truststore.p12 was the sole carrier of a certificate authority that signed nothing - zss-sample-service/cert.der was a real Broadcom-internal certificate for usilca32.lvn.broadcom.net, issued by DigiCert and expired on 2021-09-25 Also untracks the module-local copies that are generated from now on: common-service-core jwt-public-key.pub and the zaas-client keystore pair. This commit on its own leaves the build unable to resolve its certificates. The replacement follows in the next commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
richard-salac
force-pushed
the
reboot/chore/keystore_replacement_alt
branch
3 times, most recently
from
August 10, 2026 11:31
dbdfaeb to
382fe10
Compare
Signed-off-by: Richard Salac <richard.salac@broadcom.com>
richard-salac
force-pushed
the
reboot/chore/keystore_replacement_alt
branch
from
August 10, 2026 13:17
b4ff4ed to
77b46c9
Compare
Running the build from Git Bash or PowerShell on Windows failed with:
Task :generateCertificates
WSL ERROR: CreateProcessCommon:800: execvpe(/bin/bash) failed:
No such file or directory
The task invoked 'bash', and on Windows a bare bash on the PATH is normally
C:\Windows\System32\bash.exe - the WSL launcher, not a shell. With no WSL
distribution installed it cannot run anything. 'sh' is no better, because it is
usually absent from the PATH outside Git Bash.
Gradle now locates the bash that ships with Git for Windows, searching the PATH and
the usual install locations while skipping anything under System32 or Sysnative, and
fails with an actionable message if none is found. -Papiml.bash= or APIML_BASH
override the choice on any platform. Other platforms continue to use sh, which the
script is written for.
Verified on this platform: the default resolves to sh, an override is honoured, and
an override pointing at a missing shell fails the build rather than passing silently.
The Windows branch itself is unverified - it needs a run from Git Bash and from
PowerShell.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Salac <richard.salac@broadcom.com>
Generation failed after "Relaxing permissions for use inside containers" when the build was started from PowerShell, reporting that no *.p12 file was found. From Git Bash the same task succeeded. The permissions step used find(1). When Gradle launches the script from PowerShell, the shell inherits the Windows PATH, which has C:\Windows\System32 ahead of Git's usr/bin - and System32 contains an unrelated find.exe that searches files for a string. It rejects the findutils arguments with "File not found". Git Bash works because its own bin directories come first there. Replaced with shell globbing, so the step relies only on chmod. find was the only command in the script whose name collides with a different Windows tool; cat, chmod, cp, grep, keytool, mkdir, mktemp, openssl and rm all exist solely in Git's usr/bin and resolve correctly. Still POSIX: passes sh -n and bash --posix -n, and every generated artifact still ends up 0644. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
path: . relied on the reader knowing that download-artifact unpacks relative to the
workspace. Uses ${{ github.workspace }} instead, which says so.
Also records the coupling between the two halves, which was invisible from either
file: the upload lists paths across several top level directories, so the artifact is
rooted at the repository root and keeps its entries repository-relative, and the
restore depends on that when it unpacks. Narrowing the upload to a single directory
would move the artifact root and restore the files to the wrong place.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t_alt' into reboot/chore/keystore_replacement_alt
…t_alt' into reboot/chore/keystore_replacement_alt
richard-salac
marked this pull request as ready for review
August 11, 2026 13:11
Adds a section covering the two cases a developer onboarding their own service against a local API ML runs into. The first is that no new certificate is usually needed: the service certificate already covers localhost, the multi-instance names, the loopback address and every container hostname, with both clientAuth and serverAuth, so pointing at the existing pair is enough. The second is issuing one from the same authority when the hostname is not covered or a distinct identity is wanted. Declares the subject and the names inline rather than adding another .ext file, and notes the two things that are easy to get wrong: -copy_extensions copy, without which the names are dropped and API ML rejects the certificate on hostname verification, and 127.0.0.1 needing to be an IP entry rather than a DNS one. Every command was run as written to confirm the result chains to the service CA and keeps its extensions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Refers throughout to "the onboarding service" rather than "your service", so it is clear the subject is a service being onboarded to API ML rather than API ML itself, and states the first case as a fact about the existing certificate - it can be reused when the onboarding service runs on localhost - rather than as advice. Second person is gone from the section. Commands are unchanged and were run again as written. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three certificates carried OU=MSD and O=Broadcom in their subjects, and the README example and one integration test assertion repeated them. They now use the values the rest of the set already uses, OU=API Mediation Layer and O=Zowe Sample, so no certificate in the repository names a vendor: server-only CN=Zowe Component, OU=API Mediation Layer, O=Zowe Sample client-cert CN=Zowe Component Client, OU=API Mediation Layer, O=OMP hostname-mismatch CN=nonlocalhost.local, OU=API Mediation Layer, O=Zowe Sample X509SchemeTest asserts on the client certificate's distinguished name and is updated with it. Verified that the regenerated certificate still matches the assertion in the RFC 2253 form the test compares against. Remaining occurrences elsewhere in the repository are unrelated to certificates - maintainer addresses, support URLs, documentation links and contributor guidance - and are left alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
iansergeant42
requested changes
Aug 11, 2026
…cally
Validity drops from five years for certificates and ten for authorities to 90 days
for both. These are development certificates regenerated on demand rather than
committed, so a long validity buys nothing, and an authority outliving the
certificates it signs serves no purpose when both are minted in the same run.
That alone would trade one failure for another: the generation task is driven by its
outputs, so a working copy left alone for three months would keep a complete set that
no longer validates, and report itself up to date. The task now also considers how
long the certificates remain valid, and Gradle reissues them like any other stale
output:
> Task :generateCertificates
Certificates expire on Mon Nov 09 16:14:01 CET 2026 - regenerating
Renewal starts shortly before expiry rather than after it, so a certificate cannot
lapse midway through a long test run or under a running local instance. The margin
defaults to seven days and -Papiml.keystore.renewBeforeDays=0 renews only once they
have expired. The check reads the public certificates, so it needs no passwords.
A set restored from a CI artifact is never renewed, because every job in a workflow
run has to keep trusting the same authority.
Verified: a fresh set is valid for 90 days and stays up to date on the next
invocation; a set inside the renewal window is reissued; a planted expired
certificate is reissued; and a restored set is left alone even with a renewal window
wider than the validity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Richard Salac <richard.salac@broadcom.com>
richard-salac
force-pushed
the
reboot/chore/keystore_replacement_alt
branch
from
August 11, 2026 15:42
dec884f to
9b83f96
Compare
iansergeant42
approved these changes
Aug 12, 2026
Signed-off-by: Richard Salac <richard.salac@broadcom.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces every checked-in certificate and keystore with single-purpose ones generated on demand. No private key material remains in the repository.
The repository previously carried 37 certificate artifacts signed by 6 certificate authorities. Three of those authorities shared an identical subject DN, and the CA private key that shipped in the repository belonged to a different authority than the one that had signed the default service certificate — so nothing could actually be re-issued.
Now there are 12 certificates from 3 authorities, all produced by
keystore/generate-certificates.sh.Layout
Directories group artifacts by purpose rather than by deployment profile, since one service certificate now covers every hostname API ML is reached by in any profile:
ca/service-ca,client-ca,untrusted-caservice/serverAuth-only andclientAuth-only split-role identities, the default truststoreclient/APIMTST,USER,UNKNOWNUSERnegative/public_ca/Each certificate's DN, key usage and SANs live in one committed openssl
.extfile beside the keystore it produces: a CSR per certificate, signed withopenssl x509 -req, packaged into PKCS12, truststores assembled withkeytool.Nothing with a private key is tracked
Only two pieces of certificate material remain in git, both holding public entries only and neither derived from anything generated here:
keystore/public_ca/public-roots.p12— real third-party roots. The generator merges these intoservice/service.truststore.p12, because API ML validates live HTTPS endpoints with its own truststore rather than the JVM default one: the OIDC support fetches a JWKS over TLS throughHttpConfig.getSecureSslContextWithoutKeystore()(seeHttpsJwksProvider).public-roots.p12stays the maintained source, since those certificates expire on their own schedule.zaas-client/src/test/resources/mockserver-ca.cer— MockServer's own authority, needed by one test that starts MockServer over HTTPS. It was previously an invisible entry inside a committed binary truststore.Public certificates and truststores derived from our own keys are generated too, not committed: regenerating mints fresh CA keypairs, so committed anchors would be stale on arrival in a fresh clone and would not match its own keys.
How generation is wired
Locally, Gradle runs the generator before every test task and before the
jibimage builds, so a clone-and-test needs no manual step. The task is output-driven, so it runs once and then stays out of the way.In CI, the set is generated once per workflow run and shared. This is required rather than cosmetic: every run mints a fresh certificate authority, and everything taking part in a mutual TLS handshake has to trust the same one. A new
GenerateCertificatesjob produces the set and uploads it; the 32 other jobs restore it through.github/actions/restore-certificates. Without this, the container images carry one authority while the test runner presents certificates from another — same subject DN, so path building succeeds and validation then fails withPKIX path validation failed: signature check failed.A restored set also sets
APIML_CERTIFICATES_PREGENERATED, which the Gradle task honours: a fresh checkout has no task history, so it would otherwise regenerate and replace what was just restored.What was merged away
localhost,localhost-multianddocker/all-servicesdiffered only in their SAN list → one service certificate holding the unionCA Incanchors droppedzaas-clientkeystore pair were stale duplicates on a different authority, carrying a deadjwtsecretentryRemovals that replaced nothing, because nothing referenced them:
selfsigned/localhost-untrusted.*,local_ca/zowe-dev-ca.cer(byte-identical tolocalhost/localca.cer),Zowe_Service_..._Certificate_Authority_.cer(duplicate of an already-exported leaf), andzss-sample-service/cert.der— a real Broadcom-internal certificate forusilca32.lvn.broadcom.net, DigiCert-issued, expired 2021-09-25.Defects fixed
127.0.0.1was a DNS SAN rather than an IP SAN, which no TLS stack matches against the literal addressall-services.extsetCNtwice where it meantC, so those certificates had no countryserver-onlyandclient-certsubject DNs differed only by letter caseconfig/local/otel/config.yamlreferenced filenames that never existed in this repositoryEnvironment support
Generating at build time rather than committing means several things a git checkout used to provide for free now have to be arranged explicitly:
.github/actions/setup, because the jobs that run in a bareubuntu:latestcontainer do not have it. The generator also checks foropensslandkeytoolup front and names whichever is missing.0644. The OpenTelemetry collector bind-mountskeystore/and runs unprivileged, so every generated artifact is relaxed to0644..gitattributespins the script to LF, since bash cannot execute a CRLF script; Gradle locates the bash that ships with Git for Windows rather than the WSL launcher inSystem32; and the script avoidsfind, whose name collides with an unrelatedSystem32tool when the Windows PATH is inherited.Verification
The generator self-checks every chain before finishing: that each authority's keystore holds the certificate published beside it, that each leaf validates against its issuer, and that the untrusted leaf does not validate against the service CA.
Review notes
The first commit removes the checked-in certificates and does not build on its own; everything after it adds the generator, the
.extconfiguration, the Gradle and CI wiring, and repoints the paths across configs, tests and docs.Most of the diff is mechanical path repointing. The parts worth real attention are
keystore/generate-certificates.sh,gradle/certificates.gradle,.github/actions/restore-certificates/, and theGenerateCertificatesjob.TODO
🤖 Generated with Claude Code