[LinuxJedi]

Fixes the PRB-wolfpkcs11-fips-v5 Jenkins job (FIPS v5.2.1 with --enable-nss), which has been failing on every PR since mid-October. Its result had stopped reaching PRs; that is fixed on the Jenkins side, so the FIPSv5 (./fips-check.sh linuxv5) status shows again.

Build failures

Two static functions are defined in builds where nothing calls them, which breaks -Werror builds with unused-function.

• find_store_file() in tests/object_policy_integrity_test.c is only used by the legacy store test. That test is compiled out for FIPS with NSS, without PBKDF2 or scrypt, and with WOLFSSL_STM32U5_DHUK, so the FIPSv5 job stopped at compile time. The function now lives inside the legacy store test block.
• wp11_Rsa_CheckRange() in src/internal.c is only called from the CKM_RSA_X_509 verify path, which is built with WC_RSA_DIRECT. It is now built only with WC_RSA_DIRECT.

FIPS v5 test failures

With the build fixed, make check failed 4 tests under FIPS v5:

• pkcs11test, login_pin_state_test: since C_SetPIN re-encrypts token objects, the user path hashes the old PIN itself. An old PIN below the FIPS HMAC minimum key length (14 bytes) made C_SetPIN return CKR_FUNCTION_FAILED instead of CKR_PIN_INCORRECT. It is now reported as an incorrect PIN, as C_Login and the SO path already do, and like a login it does not count toward the lockout.
• login_pin_state_test: the wrong old PIN and the PIN replacing an empty one were 13 bytes, which FIPS cannot hash, so the lockout and empty PIN tests could not work. They are now 15 bytes.
• kdf_tls_derive_test: the HKDF length test used a 4 byte salt, which HKDF-Extract uses as the HMAC key. It is now 16 bytes.
• token_store_test: the init out of memory test fails every allocation during C_Initialize, which runs the FIPS self-tests. The self-tests that allocate fail, and a failed self-test disables its algorithm until the process exits, so the retry and every later initialization in the test fail. The test is skipped in FIPS builds.

Testing

• FIPS v5.2.1 built as the Jenkins job does (fips-check.sh linuxv5 on wolfSSL master, same configure flags), in debian:bookworm: before the fixes make check failed the same 4 tests as the job. After, 85 pass, 8 skip, 0 fail, run as a non-root user so the store failure tests run too.
• wolfSSL without scrypt (legacy store test compiled out), --enable-nss, -Werror: the unfixed find_store_file() fails to build with the same error as the job; with the fix it builds clean and make check passes.
• wolfSSL with scrypt and no WC_RSA_DIRECT in options.h, --enable-nss, -Werror: builds clean and make check passes.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.