Pip install performance improvements on Windows

[khmyznikov]

What does this PR do?

Towards #12712.

On Windows, bytecode compilation for large wheels can be dominated by
filesystem and real-time antivirus scanning. This PR compiles Python files from
wheels containing at least eight such files with a bounded
ThreadPoolExecutor. The worker count respects the process-available CPU count
and is capped at eight. Smaller wheels and all non-Windows platforms retain the
existing serial path.

This approach avoids the interpreter startup and pip self-upgrade hazards of
process- or subinterpreter-based workers: no additional interpreter imports pip
while its installed files may be replaced.

The parallel path is designed to preserve existing behavior:

• Results, diagnostics, exceptions, and RECORD updates remain in source order.
• Distinct source paths that resolve to the same .pyc output path, including
  case-insensitive Windows aliases, are compiled serially.
• Workers suppress diagnostics and failed files are retried serially, preserving
  ordered diagnostics without replacing process-wide sys.stdout.
• Failure to create the thread pool or submit work because of resource
  exhaustion falls back to serial compilation.
• Worker threads are cleaned up after success, failure, or early generator
  close.
• Bytecode output remains deterministic.
• --no-parallel-compile retains bytecode compilation while forcing the serial
  path.

Supporting benchmark and free-threaded determinism artifacts are available in
this gist.
The benchmark creates deterministic synthetic wheels, interleaves baseline and
candidate runs, reports Defender state, and verifies installed-file hashes. The
reproducer supports python/cpython#156504.

A representative CPython 3.14 run with Microsoft Defender real-time, IOAV, and
behavior monitoring enabled measured:

Python files	Serial	Parallel	Speedup
128	1.260 s	0.968 s	1.30x
256	1.801 s	1.160 s	1.55x
1,200	7.389 s	2.588 s	2.85x

All compiled-install snapshots were byte-for-byte identical. A 1,200-file
--no-compile control was unchanged within run-to-run noise.

Representative-wheel measurements were also collected in a Windows 11 Hyper-V
VM using CPython 3.14.7 with four process-available CPUs. Microsoft Defender
antivirus, real-time protection, IOAV protection, and behavior monitoring were
enabled. Each value is the median of three balanced pip install --prefix
runs:

Volume	Wheel	Python files	Serial	Parallel	Speedup
NTFS	pip 26.2.1	404	3.209 s	1.918 s	1.67x
NTFS	Plotly 7.0.0	1,507	9.339 s	4.042 s	2.31x
ReFS Dev Drive	pip 26.2.1	404	1.397 s	1.328 s	1.05x
ReFS Dev Drive	Plotly 7.0.0	1,507	4.374 s	3.817 s	1.15x

The ReFS volume was a trusted Dev Drive, and fsutil reported the wesp and
WdFilter antivirus filters attached. Its smaller incremental gains are
expected because Defender's Dev Drive performance mode already removes much of
the filesystem-scanning bottleneck. All baseline, candidate-serial, and
candidate-parallel installed-file snapshots were byte-for-byte identical.

Validation includes:

• 112 passing focused wheel, install-command, and requirement-install tests on
  current upstream.
• Equivalent wheel tests on Python 3.10, Python 3.14, and Python 3.14
  free-threaded.
• Existing wheel compile and no-compile functional tests.
• Ordered diagnostics, output-path collisions, unchanged sys.stdout,
  serial opt-out, pool failure fallback, exception propagation, and executor
  lifecycle coverage.
• Context-aware warning suppression with worker context inheritance disabled.
• A successful pip self-reinstall while parallel compilation was active.
• Black, Ruff, and codespell.

This also adds news/12712.feature.rst.

PR Checklist:

• I agree to follow the PSF Code of Conduct.
• I have read and have followed the CONTRIBUTING.md file.
• I have added a news file fragment (or this PR does not need one).
• I have read and followed the AI_POLICY.md file, and if any AI tools were used, I have disclosed it below.

Assisted-by: GitHub Copilot CLI

[notatallshaw]

One of the downsides of this approach is non-deterministic byte code, e.g. python/cpython#129724, so you aren't guaranteed "byte-for-byte reproducibility" regardless.

[khmyznikov]

@notatallshaw yeah, that's why I've opened similar issue... Do you think that fixable in theory?

[notatallshaw]

@notatallshaw yeah, that's why I've opened similar issue... Do you think that fixable in theory?

No, the issue I linked isn't related to free threading, and if you read it (and the linking issues) I think it's unlikely going to be fixed unless someone figures out something clever: python/cpython#129724 (comment)

[khmyznikov]

@notatallshaw python/cpython#156862 is this might be right? I see both regressions and gains, I'm trying to understand is it valuable to continue this direction or it's likely no go?

[notatallshaw]

@khmyznikov I'm not a core Python developer so I can not advise you on python/cpython#156862

As for this PR, I don't know, I'm unlikely to have motivation to review this any time soon, this approach has been tried a few times before, and every time it suffered that fate of when digging into edge cases it always ended up with blocking issues.

If you want to leave it open in case another maintainer is interested please remove all the additional files you've added under tools, they are not something that would be accepted and make the PR gigantic. Feel free to add them as gist links in the PR description, if you think they are worth addition context.