Fix SHA3 first-use dispatch race

[Aryadeepta]

Fixes #2482

Summary

On current main, the first SHA3/SHAKE call can enter the XKCP callback table
while Keccak_Dispatch() installs the AVX512VL table by copying over the live
default callback struct. This creates a production data race for concurrent
first use. It also allows one incremental context to be initialized through
XKCP and then absorbed, finalized, or squeezed through AVX512VL.

PR #2426 (Douglas Stebila) previously identified both mechanisms and proposed
moving top-level backend selection into sha3.c/sha3x4.c. This change applies
that SHA3 dispatch design to current main; the reproduction and validation
below are the additional evidence for issue #2482.

The top-level callback table is now selected once, before public dispatch, with
pthread_once when pthread support is enabled and the existing first-use path
for non-pthread builds. The selected table is held through a callback-table
pointer. Keccak_Dispatch() and Keccak_X4_Dispatch() only select XKCP's
internal implementation pointers. OQS_SHA3_set_callbacks() and
OQS_SHA3_x4_set_callbacks() still accept a custom table after automatic
initialization; concurrent calls to those setter APIs are not made safe by this
change.

Deterministic actual-entry reproduction

Temporary diagnostic instrumentation of the actual XKCP function bodies and
linker wrappers around the AVX512VL assembly entry points was run in a fresh,
single-threaded process. On current main:

XKCP shake256_inc_init
AVX512VL shake256_inc_absorb
AVX512VL shake256_inc_finalize
AVX512VL shake256_inc_squeeze

XKCP shake128_x4_inc_init
AVX512VL shake128_x4_inc_absorb
AVX512VL shake128_x4_inc_finalize
AVX512VL shake128_x4_inc_squeeze

With this change:

AVX512VL shake256_inc_init
AVX512VL shake256_inc_absorb
AVX512VL shake256_inc_finalize
AVX512VL shake256_inc_squeeze

AVX512VL shake128_x4_inc_init
AVX512VL shake128_x4_inc_absorb
AVX512VL shake128_x4_inc_finalize
AVX512VL shake128_x4_inc_squeeze

The trace observes function entry rather than inferring backend identity from a
callback-table read. The temporary instrumentation and wrappers are not part
of this change.

ThreadSanitizer evidence

A temporary reproducer using only the public scalar SHAKE incremental API
(OQS_SHA3_shake256_inc_*) reports a race between Keccak_Dispatch() writing
sha3_default_callbacks and another thread reading it from
OQS_SHA3_shake256_inc_init on current main. An equivalent public API
SHAKE128 x4 workload reports the same race in Keccak_X4_Dispatch() against
sha3_x4_default_callbacks.

The same two workloads report no TSan warnings with this change. The
reproducers do not inspect callback tables or include private SHA3 headers.

Tests

The existing tests/test_sha3.c callback override checks were run for scalar
and x4 callbacks, along with the SHA3/SHAKE known-answer tests for:

• SHA3-256, SHA3-384, SHA3-512
• SHAKE-128 and SHAKE-256
• four-way SHAKE-128 and four-way SHAKE-256

The same test now snapshots representative scalar and x4 first-use callback
entries before first-use incremental dispatch and verifies that those entries
remain unchanged. With the current-main source and this test, the test
fails deterministically with:

Failure! SHA3 first-use dispatch modified the default callbacks
Failure! SHA3-x4 first-use dispatch modified the default callbacks

With this change it passes.

Commands used for the ordinary test builds included:

cmake -S . -B /tmp/liboqs-2482-min -GNinja \
  -DCMAKE_BUILD_TYPE=Release -DOQS_DIST_BUILD=ON \
  -DOQS_USE_SHA3_OPENSSL=OFF -DOQS_BUILD_ONLY_LIB=OFF \
  -DOQS_MINIMAL_BUILD=KEM_ml_kem_512
cmake --build /tmp/liboqs-2482-min --target test_sha3
/tmp/liboqs-2482-min/tests/test_sha3

cmake -S . -B /tmp/liboqs-2482-fix-nopthread -GNinja \
  -DCMAKE_BUILD_TYPE=Release -DOQS_DIST_BUILD=ON \
  -DOQS_USE_SHA3_OPENSSL=OFF -DOQS_BUILD_ONLY_LIB=OFF \
  -DOQS_MINIMAL_BUILD=KEM_ml_kem_512 -DOQS_EMBEDDED_BUILD=ON
cmake --build /tmp/liboqs-2482-fix-nopthread --target test_sha3
/tmp/liboqs-2482-fix-nopthread/tests/test_sha3

The tests passed in AVX512VL distribution, debug, and non-pthread embedded-style
configurations. No SHA3 output, KAT value, public API, or algorithm availability
changed.

The permanent regression test checks representative callback entries rather
than backend identity or every field of each table. The public API intentionally
does not expose the selected backend, so linker wrapping, private symbols, or
production-only tracing would be unnecessarily brittle in the normal suite.
The actual-entry and TSan harnesses are retained as investigation evidence
rather than committed test machinery.

Scope

This work reproduces and fixes a real C data race/undefined concurrent behavior
and a deterministic cross-backend incremental-context ownership violation. No
incorrect digest, production crash, memory corruption, secret leakage, or
security compromise was reproduced, and none is claimed here.

Generative-AI disclosure

I used Codex/ChatGPT to assist with repository investigation, reproducer
development, testing strategy, and drafting. I manually reviewed the resulting
changes and validation results before submission.