fix(zipapp): reduce bundled interpreter size - #4165
Conversation
Bazel reports source symlinks as regular files, causing Python executable aliases to be copied into zipapps as full binaries. Hermetic runtimes also include shared libpython artifacts even when the interpreter links Python statically. Preserve source symlinks through sandbox indirection and exclude libpython shared objects from runtime files while retaining them for explicit native dependencies. Add regression coverage for archive structure and native extension loading.
There was a problem hiding this comment.
🟡 Changes recommended
The new archive assertion also runs on macOS, where the runtime intentionally retains libpython dylibs.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
This PR reduces zipapp sizes by preserving Python executable symlinks and excluding unused Linux libpython shared objects.
Changes:
- Detects source symlinks hidden behind Bazel sandbox indirection.
- Excludes shared
libpythonfiles from hermetic runtime files. - Adds archive and native-extension regression coverage.
File summaries
| File | Description |
|---|---|
tools/zipapp/zipper.py |
Preserves sandboxed source symlinks. |
tests/tools/zipapp/zipper_test.py |
Tests symlink archive behavior. |
tests/py_zipapp/venv_zipapp_test.py |
Checks archive structure and runtime contents. |
tests/py_zipapp/main.py |
Exercises native _ssl loading. |
python/private/hermetic_runtime_repo_setup.bzl |
Excludes Linux shared libpython objects. |
news/py-zipapp-size.fixed.md |
Documents the fix. |
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Address PR review finding: the runtime-size assertion should mirror the Linux-only .so exclusion. macOS hermetic runtimes intentionally retain libpython dylibs, so only reject libpython shared-object entries. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
The two failures in buildkite seems to relate to using python 3.13.1. The transition where Astral switched python-build-standalone to statically link libpython directly into the bin/python executable occurred in May 2025. The custom A few options:
Thoughts? |
Custom hermetic distributions may use dynamically linked interpreters and require libpython.so.1.0 at runtime. The zipapp size optimization incorrectly removed that library from every Linux runtime. Restore the existing versioned-library inclusion and keep the zipapp optimization limited to preserving executable symlinks.
Astral Python Standalone builds from 20250604 onward include libpython statically in the interpreter, so packaging their shared libpython objects adds substantial unused size to self-contained zipapps. Recognize only official Astral release URLs at or after that build date and exclude libpython shared objects for those runtimes. Unknown, custom, and older distributions retain the existing versioned libraries for dynamically linked interpreters.
|
Implemented the Policy:
|
aignas
left a comment
aignas
left a comment
There was a problem hiding this comment.
LGTM, but it would be good to expose a python extension configuration to disable the auto-detection. Please let me know if you would like to do this as part of this PR.
I am OK to just do python_repository attribute with an "auto" default where the user can very easily do a patch for now to override this if needed.
The static libpython transition started with the 20250517 Astral build, not 20250604. Also allow users to force inclusion or exclusion when auto-detection does not match a mirrored or customized runtime. Expose the auto/include/exclude mode through the Python extension overrides and document the Starlark helper argument types.
Document the auto, include, and exclude modes exposed by the Python extension, including the 20250517 Astral cutoff and fallback behavior for unknown runtimes. Also standardize indentation in the generated hermetic runtime BUILD definition.
|
It seems one of the CI jobs hangs, how to re-trigger it? |
|
@aignas @rickeylev Any more comments? Or can this be merged? |
|
Thanks for the PR! Yeah, we've been debating about how to exclude libpython, since its often not needed. |
Excluding shared libpython.so files by default from recognized Astral Python Standalone runtimes affects all targets using the hermetic toolchain, not just py_zipapp. Add news/4165.changed.md to document the default behavior change and the libpython override attribute.
|
/backport |
|
Workflow failed for command See workflow run for logs. |
|
/backport |
Bazel reports source symlinks as regular files, causing Python executable aliases to be copied into zipapps as full binaries. Hermetic runtimes also include shared libpython artifacts even when the interpreter links Python statically. Preserve source symlinks through sandbox indirection and exclude libpython shared objects from runtime files while retaining them for explicit native dependencies. Add regression coverage for archive structure and native extension loading. Fixes #3534 Fixes #4163 --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Richard Levasseur <richardlev@gmail.com> (cherry picked from commit fd22cff) Work towards #4175
Updates CHANGELOG.md and removes news files for backports: - bazel-contrib#4165 Work towards bazel-contrib#4175 Release-Tracking-Issue: bazel-contrib#4175 Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
|
posterity
I was playing with this idea a bit a couple months ago. Its possible, but also somewhat of a pain. During the repo phase, we could inspect the binary and look at its linkage. That incurs having to use some host-compatible tool to do that. TBH I like this idea the best, but don't have time to set that up. Full compatibility would need A variation of the above was to look at PYTHON.json, which i think did tell the linkage, but the install_only archives don't contain python.json. The second idea I was toying with was to have a build action inspect things and then conditionally emit libpython in the depsets. At the time, this ran into the bug where generated files couldn't be put into the runtime (now fixed). However, I then ran into the problem of having a conditional file output in bazel. Best I could come up with was either (1) generate an empty .so file (valid, but feels hacky) or (2) directory artifacts (not sure how this would fit into things more broadly). |
Bazel reports source symlinks as regular files, causing Python executable aliases to be copied into zipapps as full binaries. Hermetic runtimes also include shared libpython artifacts even when the interpreter links Python statically.
Preserve source symlinks through sandbox indirection and exclude libpython shared objects from runtime files while retaining them for explicit native dependencies. Add regression coverage for archive structure and native extension loading.
Fixes #3534
Fixes #4163