20261110-linuxkm-stdrng-purge-etc by douzzer · Pull Request #12107 · wolfSSL/wolfssl

douzzer · 2026-11-12T03:31:27Z

linuxkm/linuxkm_wc_port.h linuxkm/lkcapi_sha_glue.c linuxkm/simd_hammer/simd_hammer.c .wolfssl_known_macro_extras: fixes for removal of stdrng alg from the LKCAPI.

gate crypto/internal/rng.h inclusion and "stdrng" alg registration on kernel
<7.5 (<linux-next of 20261110).
add WOLFSSL_LINUXKM_LKCAPI_HAVE_STDRNG with override
WOLFSSL_LINUXKM_LKCAPI_NO_STDRNG.
add dummy macro variant of wc_linux_kernel_rng_is_wolfcrypt().
add gating on WOLFSSL_LINUXKM_LKCAPI_HAVE_STDRNG as needed throughout
linuxkm/lkcapi_sha_glue.c.
add a HAVE_STDRNG indicator macro to linuxkm/simd_hammer/simd_hammer.c and
gate the "rng" probe on it. also add a runtime report of the instantiated RNG
driver name, and a warning if it isn't wolfCrypt.
note, on !WOLFSSL_LINUXKM_LKCAPI_HAVE_STDRNG builds, there’s no longer a
deinstall_algs prerequisite before rmmod.

wolfcrypt/src/asn.c wolfcrypt/src/asn_orig.c wolfcrypt/src/signature.c wolfcrypt/test/test.c doc/dox_comments/header_files/asn_public.h:

bound the digSz in wc_EncodeSignature(), and catch zero (error) return from it
in wc_SignatureDerEncode(). Add test coverage to rsa_sig_test().

linuxkm/lkcapi_rsa_glue.c: add digest length checks to the pkcs1 sign callbacks,
matching those in the verify callbacks since ff93e6d and 8fef82c, by
factoring the common argument validation into km_pkcs1_check_digest_len().

tests/api/test_tls.c: skip test_tls_unset_fd_no_stdin_io() if defined(HAVE_VALGRIND).

tested with

wolfssl-multi-test.sh ...
check-source-text
defaults-gcc-latest-valgrind
'.*linuxkm-next.*'
linuxkm-fips-dev-insmod-cust-kernel-1
linuxkm-fips-dev-insmod-wolfguard-cust-kernel-3 linuxkm-fips-dev-insmod-cust-kernel-1
linuxkm-fips-v5-vanilla-insmod-wolfguard-cust-kernel-6
linuxkm-fips-v5-strict-dist-insmod-cust-kernel-2-amdrdseed
linuxkm-6.15-all-cryptonly-quantum-safe-allintelasm-LKCAPI-insmod-crypto-fuzzer-kmemleak-no-stdrng
linuxkm-6.15-all-cryptonly-quantum-safe-noasm-LKCAPI-insmod-crypto-fuzzer-ksan-no-stdrng

…er/simd_hammer.c .wolfssl_known_macro_extras: fixes for removal of stdrng alg from the LKCAPI. * gate crypto/internal/rng.h inclusion and "stdrng" alg registration on kernel <7.5 (<linux-next of 20261110). * add WOLFSSL_LINUXKM_LKCAPI_HAVE_STDRNG with override WOLFSSL_LINUXKM_LKCAPI_NO_STDRNG. * add dummy macro variant of wc_linux_kernel_rng_is_wolfcrypt(). * add gating on WOLFSSL_LINUXKM_LKCAPI_HAVE_STDRNG as needed throughout linuxkm/lkcapi_sha_glue.c. * add a HAVE_STDRNG indicator macro to linuxkm/simd_hammer/simd_hammer.c and gate the "rng" probe on it. also add a runtime report of the instantiated RNG driver name, and a warning if it isn't wolfCrypt. * note, on !WOLFSSL_LINUXKM_LKCAPI_HAVE_STDRNG builds, there’s no longer a deinstall_algs prerequisite before rmmod.

…c wolfcrypt/test/test.c doc/dox_comments/header_files/asn_public.h: bound the digSz in wc_EncodeSignature(), and catch zero (error) return from it in wc_SignatureDerEncode(). Add test coverage to rsa_sig_test().

… callbacks, matching those in the verify callbacks since ff93e6d and 8fef82c, by factoring the common argument validation into km_pkcs1_check_digest_len().

…(HAVE_VALGRIND).

github-actions · 2026-11-12T03:35:50Z

MemBrowse Memory Report · Explore what changed →

gcc-arm-cortex-m0plus

FLASH: .text -20 B (-0.0%, 71,147 B / 262,144 B, total: 27% used)

gcc-arm-cortex-m4-dtls13

FLASH: .rodata +16 B, .text +192 B (+0.1%, 197,224 B / 1,048,576 B, total: 19% used)

gcc-arm-cortex-m4-openssl-compat

FLASH: .rodata +40 B, .text +1,216 B (+0.2%, 803,948 B / 1,048,576 B, total: 77% used)

gcc-arm-cortex-m4-pkcs7

FLASH: .text +192 B (+0.1%, 224,084 B / 262,144 B, total: 85% used)

gcc-arm-cortex-m4-pq

FLASH: .rodata +8 B, .text +192 B (+0.1%, 314,136 B / 1,048,576 B, total: 30% used)

gcc-arm-cortex-m4-rsa-only

FLASH: .text +128 B (+0.0%, 344,464 B / 1,048,576 B, total: 33% used)

gcc-arm-cortex-m4-tls12

FLASH: .text -64 B (-0.0%, 132,945 B / 262,144 B, total: 51% used)

gcc-arm-cortex-m4-tls13

FLASH: .text +192 B (+0.1%, 250,862 B / 262,144 B, total: 96% used)

gcc-arm-cortex-m7-pq

FLASH: .rodata +8 B, .text +192 B (+0.1%, 314,968 B / 1,048,576 B, total: 30% used)

gcc-arm-cortex-m7-tls13

FLASH: .text +192 B (+0.1%, 250,862 B / 262,144 B, total: 96% used)

stm32-sim-stm32h753

FLASH: .text +64 B (+0.0%, 205,484 B / 2,097,152 B, total: 10% used)

No memory changes detected for:

_{Catch memory regressions in CI with MemBrowse. Free for open source.}

Copilot

🟡 Changes recommended

Active issues include a compilation failure in the kprobe configuration without stdrng, phantom registration accounting that blocks deinstallation, and a header guard that prevents wolfCrypt RNG recognition.

3 open issues

What changed in this PR

This PR updates wolfSSL’s Linux kernel integration for removal of stdrng and strengthens RSA digest-length validation.

Changes:

Gate stdrng registration and probes by kernel support, with an explicit disable override.
Bound signature digest encoding, share RSA callback validation, and add regression tests.
Document encoding limits and skip the unset-file-descriptor TLS test under Valgrind.

File	Description
`wolfcrypt/test/test.c`	Test oversized digest rejection for in-place and separate-buffer encoding.
`wolfcrypt/src/signature.c`	Handle signature-encoding failure.
`wolfcrypt/src/asn.c`	Bound digest size in template-based encoding.
`wolfcrypt/src/asn_orig.c`	Bound digest size in legacy encoding.
`tests/api/test_tls.c`	Skip the unset-descriptor test under Valgrind.
`linuxkm/simd_hammer/simd_hammer.c`	Gate RNG probes and report the selected driver.
`linuxkm/lkcapi_sha_glue.c`	Adapt RNG registration and cleanup for builds without `stdrng`.
`linuxkm/lkcapi_rsa_glue.c`	Share digest validation, advertise digest sizes, and extend tests.
`linuxkm/linuxkm_wc_port.h`	Add `stdrng` feature gating and a fallback RNG detector macro.
`doc/dox_comments/header_files/asn_public.h`	Document digest-size limits and error returns.
`.wolfssl_known_macro_extras`	Register the new `stdrng` disable macro.

_{Review effort: Balanced}

wolfSSL-Fenrir-bot

Fenrir Automated Review — PR #12107

Scan targets checked: linuxkm-src, linuxkm-bugs, wolfcrypt-src, wolfcrypt-bugs, wolfssl-bugs
Not reviewed (no scan target covers them): .wolfssl_known_macro_extras, doc/dox_comments/header_files/asn_public.h
Coverage: 9 of 9 in-scope changed file(s) opened by the reviewer

Findings: 3
3 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Review tier: Max

linuxkm/linuxkm_wc_port.h and linuxkm/lkcapi_sha_glue.c: fix scoping of wc_linux_kernel_rng_is_wolfcrypt(); linuxkm/lkcapi_sha_glue.c: fix dummy call to wc_linuxkm_drbg_startup() to leave wc_linuxkm_drbg_loaded defined; wolfcrypt/src/asn_orig.c: check digSz against actual size of digArray.

wolfSSL-Fenrir-bot

Fenrir Automated Review — PR #12107

Scan targets checked: linuxkm-src, linuxkm-bugs, wolfcrypt-src, wolfcrypt-bugs
Unchanged since last review (not re-run): wolfssl-bugs
Not reviewed (no scan target covers them): .wolfssl_known_macro_extras, doc/dox_comments/header_files/asn_public.h
Coverage: 2 of 3 in-scope changed file(s) opened by the reviewer; not opened: wolfcrypt/src/asn_orig.c

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

Review tier: Lite

Fenrir's latest completed scan found no issues; clearing the prior automated change request.

Copilot

🔵 Needs a closer look

The unload notice is hidden for no-STDRNG builds with random-bytes callbacks, although those builds still require deinstall_algs to release the module reference and allow rmmod; the PR description also needs qualification.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Keep the unload notice for callback-backed no-STDRNG builds

linuxkm/lkcapi_sha_glue.c:5828

With WOLFSSL_LINUXKM_LKCAPI_NO_STDRNG and random-bytes callbacks enabled, rmmod still fails because the supplied kernel patches retain a module reference until the callbacks are unregistered. This guard hides the deinstall_algs instruction needed to release that reference. Retain the unload notice for callback-backed builds too, and qualify the PR description's claim that no-STDRNG builds no longer require this step.

3 resolved since last review

_{Review effort: Balanced}

philljj · 2026-11-12T14:31:48Z

@claude review

claude

Findings marked 🟡 are optional suggestions and need no follow-up push.

Additional findings (outside the current diff — GitHub can't attach inline comments there):

🟡 linuxkm/lkcapi_sha_glue.c — nit: On no-stdrng builds for kernels before 7.1, a failed default-bank checkout at startup leaves the bank and its entropy daemon running until a later cleanup. The guard at 5814-5815 still tests only LINUX_VERSION_CODE >= KERNEL_VERSION(7, 1, 0). Every sibling < 7.1 guard in this change gained && defined(WOLFSSL_LINUXKM_LKCAPI_HAVE_STDRNG). Fix: make the condition (LINUX_VERSION_CODE >= KERNEL_VERSION(7, 1, 0)) || !defined(WOLFSSL_LINUXKM_LKCAPI_HAVE_STDRNG), so it matches the #else branch at 5764 that initialized local_default_bank.

Why this was flagged

The trigger is a build with WOLFSSL_LINUXKM_LKCAPI_NO_STDRNG on a kernel older than 7.1 (for example the 6.15 no-stdrng targets in the PR description) and LINUXKM_LKCAPI_REGISTER_HASH_DRBG_DEFAULT. wc_linuxkm_drbg_startup takes the #else /* >= 7.1.0 || !WOLFSSL_LINUXKM_LKCAPI_HAVE_STDRNG */ branch at 5764 and initializes local_default_bank through wc_linuxkm_rng_bank_init at 5790. If wc_rng_bank_default_checkout or checkin then fails, the error path at 5814-5815 skips wc_linuxkm_rng_bank_fini(default_bank), because that guard still requires kernel >= 7.1. Startup returns -ECANCELED while the default bank and its entropy daemon are still live. Teardown is delayed until linuxkm_lkcapi_unregister runs wc_linuxkm_drbg_cleanup (wc_linuxkm_drbg_loaded was already set at 5797). The install_algs sysfs path at lkcapi_glue.c:284 does not unregister on failure. This is an inconsistency on an error path, not a permanent leak.

Verification: The checkout/checkin error path at lkcapi_sha_glue.c:5813-5821 was not updated. Its fini is still guarded only by (LINUX_VERSION_CODE >= KERNEL_VERSION(7, 1, 0)) (5815), so on these kernels it returns -ECANCELED without calling wc_linuxkm_rng_bank_fini(default_bank). On the base branch this could not happen, because kernels older than 7.1 always took the stdrng path.

…tory comment for !HAVE_VALGRIND dependency.

* in km_rsa_set_key(), return -EBADMSG for a key that fails to decode, matching the in-tree rsa-generic, and -ENOMEM on allocation failure; update the rejected-setkey probes in the self-testers to match. * fix the self-testers for WOLFKM_DEBUG_RSA_GENERIC: when testing the in-tree drivers, expect the kernel's default (key-size) crypto_sig_digestsize(), and skip the sign digest length probe on the akcipher pkcs1pad, which doesn't check it.

…ot review of the stdrng purge PR. * gate the definitions of wc_linuxkm_rng_bank_init(), wc_linuxkm_rng_bank_fini(), linuxkm_get_drbg(), linuxkm_put_drbg(), and wc_linuxkm_drbg_generate() on the disjunctions of their callers' gates, fixing -Werror=unused-function in !WOLFSSL_LINUXKM_LKCAPI_HAVE_STDRNG builds against kernels without the get-random-callbacks patch (reproduced on unpatched 5.11 with WOLFSSL_LINUXKM_LKCAPI_NO_STDRNG). * wc_linuxkm_drbg_startup(): rework the default-bank checkout/checkin error path -- key the unwind on a new local_default_bank_inited ownership flag rather than on kernel-version/config gates (the old gate skipped wc_linuxkm_rng_bank_fini() in no-stdrng pre-7.1 builds, leaving the bank and its entropy daemon running until delayed cleanup); log fini failures rather than ignoring them; null out default_bank so the delayed cleanup's fini arm degrades to a guarded no-op; and in !WOLFSSL_LINUXKM_LKCAPI_HAVE_STDRNG builds roll wc_linuxkm_drbg_loaded back to 0, mirroring the gate where it was set (in stdrng builds it stays set, so that delayed cleanup can unregister the still-registered alg). * add a comment on the CONFIG_CRYPTO_FIPS arm of the default-bank setup noting that on >=7.1 <7.5 kernels, crypto_stdrng_get_bytes() resolves to the top-priority stdrng iff fips_enabled.

wolfSSL-Fenrir-bot

Fenrir Automated Review — PR #12107

Scan targets checked: linuxkm-src, linuxkm-bugs, wolfssl-bugs
Unchanged since last review (not re-run): wolfcrypt-src, wolfcrypt-bugs
Not reviewed (no scan target covers them): .wolfssl_known_macro_extras, doc/dox_comments/header_files/asn_public.h
Coverage: 3 of 3 in-scope changed file(s) opened by the reviewer

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Review tier: Lite

Copilot

🔵 Needs a closer look

Callback-enabled builds hide required unload instructions despite remaining in use, while consuming kernel modules incorrectly report that the RNG is not wolfCrypt on affected pre-7.5 configurations.

No open issues 🎉

2 previously missed issues

In code that hasn't changed since last review

Expose the hash-DRBG availability check to consuming modules

linuxkm/linuxkm_wc_port.h:2336

WOLFSSL_LINUXKM_LKCAPI_HAVE_STDRNG is defined inside BUILDING_WOLFSSL, so consuming kernel modules select the constant-zero macro even on pre-7.5 kernels with hash-DRBG-default enabled. Their calls therefore report that the RNG is not wolfCrypt instead of invoking the implementation. Make this condition available to consumers too, using the same registration, container, kernel-version, and disable-override checks.

Retain unload instructions when random-byte callbacks pin the module

linuxkm/lkcapi_sha_glue.c:5843

With WOLFSSL_LINUXKM_LKCAPI_NO_STDRNG and the patched-kernel random-byte callbacks enabled, this guard hides the unload instructions even though rmmod still reports the module as in use. Callback registration pins THIS_MODULE until unregistration (linuxkm/patches/7.3/WOLFSSL_LINUXKM_HAVE_GET_RANDOM_CALLBACKS-7v3.patch:63,158). Retain the deinstall_algs notice for callback-enabled builds and qualify the PR description's no-prerequisite claim accordingly.

_{Review effort: Balanced}

…gate around declaration of local_default_bank_inited.

wolfSSL-Fenrir-bot

Fenrir Automated Review — PR #12107

Scan targets checked: linuxkm-src, linuxkm-bugs
Unchanged since last review (not re-run): wolfcrypt-src, wolfcrypt-bugs, wolfssl-bugs
Not reviewed (no scan target covers them): .wolfssl_known_macro_extras, doc/dox_comments/header_files/asn_public.h
Coverage: 1 of 1 in-scope changed file(s) opened by the reviewer

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

Review tier: Lite

Fenrir's latest completed scan found no issues; clearing the prior automated change request.

douzzer · 2026-11-13T01:18:34Z

retest this please

Running [fips-ready:v5.X.X] on boz-amd (attempt 1/3)
[...]
git clone --branch=master git@github.com:wolfssl/wolfssl
Cloning into 'wolfssl'...
Sending interrupt signal to process
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
script returned exit code 128

philljj

thank you, the km_pkcs1_check_digest_len is a nice refactor

douzzer added 5 commits November 11, 2026 21:23

linuxkm/lkcapi_rsa_glue.c: add digest length checks to the pkcs1 sign…

0d3c30a

… callbacks, matching those in the verify callbacks since ff93e6d and 8fef82c, by factoring the common argument validation into km_pkcs1_check_digest_len().

tests/api/test_tls.c: skip test_tls_unset_fd_no_stdin_io() if defined…

ede279e

…(HAVE_VALGRIND).

linuxkm/lkcapi_sha_glue.c: fix UTF-8 in comment.

c258ef3

douzzer requested review from wolfSSL-Bot and wolfSSL-Fenrir-bot November 12, 2026 03:31

douzzer assigned wolfSSL-Bot Nov 12, 2026

Copilot AI balanced review requested due to automatic review settings November 12, 2026 03:31

Copilot started reviewing on behalf of douzzer November 12, 2026 03:32

View session

Copilot AI reviewed Nov 12, 2026

View reviewed changes

Comment thread linuxkm/lkcapi_sha_glue.c Resolved

Comment thread linuxkm/lkcapi_sha_glue.c Resolved

Comment thread linuxkm/linuxkm_wc_port.h Resolved

wolfSSL-Fenrir-bot previously requested changes Nov 12, 2026

View reviewed changes

douzzer requested review from wolfSSL-Fenrir-bot and a balanced review from Copilot November 12, 2026 04:43

Copilot started reviewing on behalf of douzzer November 12, 2026 04:43

View session

wolfSSL-Fenrir-bot reviewed Nov 12, 2026

View reviewed changes

Copilot AI reviewed Nov 12, 2026

View reviewed changes

philljj self-assigned this Nov 12, 2026

claude Bot reviewed Nov 12, 2026

View reviewed changes

Comment thread linuxkm/lkcapi_sha_glue.c Resolved

Comment thread linuxkm/lkcapi_rsa_glue.c Resolved

philljj reviewed Nov 12, 2026

View reviewed changes

Comment thread tests/api/test_tls.c Resolved

philljj assigned douzzer Nov 12, 2026

douzzer added 3 commits November 12, 2026 14:19

tests/api/test_tls.c: in test_tls_unset_fd_no_stdin_io(), add explana…

fbe2fff

…tory comment for !HAVE_VALGRIND dependency.

douzzer requested review from wolfSSL-Fenrir-bot and a balanced review from Copilot November 12, 2026 22:10

Copilot started reviewing on behalf of douzzer November 12, 2026 22:11

View session

wolfSSL-Fenrir-bot previously requested changes Nov 12, 2026

View reviewed changes

Comment thread linuxkm/lkcapi_sha_glue.c Resolved

Copilot AI reviewed Nov 12, 2026

View reviewed changes

linuxkm/lkcapi_sha_glue.c: in wc_linuxkm_drbg_startup(), add missing …

52c00cf

…gate around declaration of local_default_bank_inited.

douzzer requested a review from wolfSSL-Fenrir-bot November 12, 2026 23:05

wolfSSL-Fenrir-bot reviewed Nov 12, 2026

View reviewed changes

philljj approved these changes Nov 13, 2026

View reviewed changes

philljj merged commit 803796e into wolfSSL:master Nov 13, 2026
447 checks passed

Conversation

douzzer commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MemBrowse Memory Report · Explore what changed →

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wolfSSL-Fenrir-bot left a comment

Choose a reason for hiding this comment

Fenrir Automated Review — PR #12107

Uh oh!

wolfSSL-Fenrir-bot left a comment

Choose a reason for hiding this comment

Fenrir Automated Review — PR #12107

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

philljj commented Nov 12, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wolfSSL-Fenrir-bot left a comment

Choose a reason for hiding this comment

Fenrir Automated Review — PR #12107

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

wolfSSL-Fenrir-bot left a comment

Choose a reason for hiding this comment

Fenrir Automated Review — PR #12107

Uh oh!

douzzer commented Nov 13, 2026

Uh oh!

philljj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

douzzer commented Nov 12, 2026 •

edited

Loading

github-actions Bot commented Nov 12, 2026 •

edited

Loading