Skip to content

fix(npm): extract each package under node_modules/<name> in its repo - #159

Open
mikn wants to merge 1 commit into
mikn/ts-test-untyped-packagesfrom
mikn/npm-node-modules-layout
Open

fix(npm): extract each package under node_modules/<name> in its repo#159
mikn wants to merge 1 commit into
mikn/ts-test-untyped-packagesfrom
mikn/npm-node-modules-layout

Conversation

@mikn

@mikn mikn commented Sep 6, 2026

Copy link
Copy Markdown
Owner

npm_import extracted a package at the root of its external repository (_npm_import_impl, npm/private/npm_import.bzl), so every path the ruleset wrote for it -- a tsconfig paths value, an action input, an exec path -- lacked a node_modules segment. TypeScript classifies a paths match by that segment alone: under one the file is a library file, type-checked and never emitted; under none it is project source, emit-eligible and checked against rootDir. The files the sandbox stages for an npm package are its .d.ts, its package.json and its .json (tsgo_inputs, ts/private/ts_compile.bzl), which are exempt either way, so nothing failed on the layout by itself; a .ts reached through paths is TS6059 the moment a package's module entry is one. That is the shape behind the trial's //packages/workers-wide-event-sink:workers-wide-event-sink, one TS2306 on import type { TraceItem, TraceLog } from "@cloudflare/workers-types" because the pin names the package's global-script index.d.ts (logs/trial-7/measure/scoped.log:17): pinning the module-form index.ts instead trades the TS2306 for TS6059 in a node_modules-free replica of the program (logs/inv7/wide-event-sink-ts2306/replica-r1-paths-index-ts-rootdir.log), and the same pin under a node_modules/@cloudflare/workers-types/ directory is 0 errors (replica-r1c-node-modules-segment-index-ts.log). This PR is the layout half; the entry half (J2) stages the module entry and pins paths to it.

The tarball is still extracted and patched at the repository root, where rctx.patch applies, and then moved under node_modules/<name>/ through a staging directory (_move_package), since a package can ship a node_modules of its own -- @parcel/watcher-wasm@2.3.0 and 2.6.0, each holding node_modules/napi-wasm, are the 2 of the 513 npm repositories fetched into the trial's output base 1de19487761e9cf1e53b4aca66ee44ff under the old layout that do (logs/fix9r/J1/01-parcel-nested-node-modules.log) -- and a directory cannot be renamed into one of the same name. A BUILD, BUILD.bazel, WORKSPACE, WORKSPACE.bazel, MODULE.bazel or REPO.bazel shipped at the package root is deleted: at the repository root the generated BUILD file overwrote it, and one level down Bazel reads it as a package or repository boundary and the glob returns nothing. The stanza writes package_dir = ":node_modules/<name>/package.json", package_files = glob(["node_modules/<name>/**/*"], ...), exports_types under the same directory, and exports_files(["node_modules/<name>/package.json"]); the manifest is read and every has_file/read predicate answers under that root (logs/fix8r/J1/10-layout.log shows the generated files for zod, @cloudflare/workers-types, @types/culori and nanoid). Every consumer already derives its paths from NpmPackageInfo.package_dir.dirname -- ts_npm_package's package_root, ts_compile's paths writer, tsconfig_aspect's installed tree, node_modules' copy manifest -- so none of them changes. The editor's .bazel/npm/<name>/ copies are laid out from the package root as before: bazel run //:refresh_tsconfig regenerates the checked-in tsconfig.json with no diff (09-refresh-tsconfig.log, git status --short after the run lists only the then-uncommitted edits of this PR and git diff --stat -- tsconfig.json prints nothing), and //:refresh_tsconfig_test passes without it. The node_modules tree is laid out from the package root too, and the scoped glob no longer matches each package's generated BUILD.bazel and REPO.bazel: bazel-bin/tests/js_entry/generator_node_modules/nanoid lists 14 entries on this output base against 16 on one built before the change, the two being those files (12-tree-and-markers.log).

Docs: docs/guides/npm.md § One Repository per Package gains the paragraph on where a package sits and what the segment means to TypeScript; docs/rules/ts-compile.md § Action Inputs gains the sentences on where a value naming an npm package points, and that the map's other values -- a module_name, a path_aliases prefix, a workspace member -- are workspace paths (on the skeptic's output base, 10401 of the 10505 values in 213 generated tsconfigs point into external/, every one with the segment, and the other 104 across 26 keys are such paths: logs/fix8r/fix-J1-skeptic/10-paths-scan.log, 10b-paths-nonexternal.log); AGENTS.md § npm Internals names the layout and that consumers derive from package_dir.dirname; the module docstring of npm/private/npm_bin.bzl shows the stanza as it is now written, and the comment at its suffix match the path it matches. changelog.d/npm-node-modules-layout.md records it under ### Changed, exec paths included: a test or script that matched an exec path by repository name alone has to expect node_modules/<name>/ after it.

Fixture

//tests/workers:library_paths_test (tests/workers/library_paths_tests.bzl) reads the tsconfig ts_compile writes for :experimental_types, whose one dep is @npm_workers//:cloudflare_workers-types, and asserts every value of the @cloudflare/workers-types and @cloudflare/workers-types/* keys carries /node_modules/@cloudflare/workers-types/. The written-form test in tests/npm/exports_types_tests.bzl asserts, over the 14 manifests in its table that designate a declaration, that the stanza writes exports_types as a file under node_modules/<name>/, package_dir as the manifest there, and package_files as the glob over that directory alone; the package_stanza calls in it and in type_references_tests.bzl pass package on the attrs struct, which the stanza now reads. The tests that matched an exec path by repository name alone are moved to the layout: tests/npm_types_barename/precedence_tests.bzl (culori and culori/* end at /node_modules/@types/culori), tests/npm_subpath/package_root_tests.bzl (the wildcard's first value ends at /node_modules/<name>), tests/npm_exports_pattern/exports_pattern_tests.bzl (unenv's second value is /node_modules/unenv/*), tests/npm_types_barename/config_agreement_test.mjs (the build route's split takes the package as node_modules/<name> inside the repository, one segment or two, and compares what is under it), and the four Go tests over runfiles (npm_patched_dependency_test.go, npm_platform_independent_graph_test.go, npm_tarball_prefix_test.go, npm_multi_version_test.go), whose FoundFile/FoundDir patterns name the two segments; tests/npm/link_depth_tests.bzl's three exec-path strings are rewritten for truth, as the skeptic noted, though that test reads only the package-relative half and passed on either layout. tests/npm_types_shim/shim_tests.bzl and tests/workers/shipped_subpath_tests.bzl match a repository name as a path segment or a file suffix and are unchanged. No behavioural fixture is possible in this PR alone: no .ts of an npm package reaches the sandbox until J2 stages one, so the TS6059 shape has nothing to fail on here; J2's ts_compile over the module import is that fixture.

Red on 2bcc23f with the tests in the tree and the .bzl untouched, tree 69a44bd (logs/fix8r/J1/01-red.log): 8 of the 17 tests fail, the nine controls in the three suites pass.

$ bazelisk --output_base=/var/tmp/fix8-J1-ob test --noshow_progress --nocache_test_results --local_test_jobs=2 --test_summary=short --test_output=errors --keep_going //tests/npm:exports_types_tests //tests/npm:type_references_tests //tests/npm:link_depth_tests //tests/workers:library_paths_test //tests/npm_types_barename:types_package_root_test //tests/npm_subpath:wildcard_root_test //tests/npm_exports_pattern:pattern_paths_test //tests/npm:npm_patched_dependency_test //tests/npm:npm_platform_independent_graph_test //tests/npm:npm_tarball_prefix_test
[... 5 lines ...]
In test _written_form_test from //tests/npm:exports_types_tests.bzl: vite@8.2.2: exports_types names a file under the package root, not a repository (Expected "node_modules/vite/dist/node/index.d.ts", but got "dist/node/index.d.ts")
[... 46 lines ...]
    npm_platform_independent_graph_test.go:20: no file in runfiles matches "*fsevents__2_3_3/node_modules/fsevents/package.json"
[... 145 lines ...]
In test _library_paths_impl from //tests/workers:library_paths_tests.bzl: @cloudflare/workers-types names a file outside node_modules, which TypeScript takes for project source: ../../../../../external/+npm+npm_workers__cloudflare_workers-types__5_20260825_1/index.d.ts
In test _library_paths_impl from //tests/workers:library_paths_tests.bzl: @cloudflare/workers-types/* names a file outside node_modules, which TypeScript takes for project source: ../../../../../external/+npm+npm_workers__cloudflare_workers-types__5_20260825_1/*
[... 4 lines ...]
In test _types_package_root_impl from //tests/npm_types_barename:precedence_tests.bzl: culori resolves inside the @types package, not to it: ["../../../../../external/+npm+npm__types_culori__2_1_1"]
[... 5 lines ...]
In test _wildcard_root_impl from //tests/npm_subpath:package_root_tests.bzl: postcss/* must look under the package root first, not ../../../../../external/+npm+npm__postcss__8_5_26/*
[... 33 lines ...]
Executed 17 out of 17 tests: 9 tests pass and 8 fail locally.
[... 2 lines ...]
# exit=3

Green on this tree, 6824a23 (logs/fix9r/J1/02-green.log), //tests/npm:npm_multi_version_test, //tests/npm_types_barename:test_config_agreement, //tools/changelog:changelog_test and //:refresh_tsconfig_test added to the set:

$ bazelisk --output_base=/var/tmp/fix9-J1-ob test --noshow_progress --nocache_test_results --local_test_jobs=2 --test_summary=short --test_output=errors --keep_going //tests/npm:exports_types_tests //tests/npm:type_references_tests //tests/npm:link_depth_tests //tests/workers:library_paths_test //tests/npm_types_barename:types_package_root_test //tests/npm_subpath:wildcard_root_test //tests/npm_exports_pattern:pattern_paths_test //tests/npm:npm_patched_dependency_test //tests/npm:npm_platform_independent_graph_test //tests/npm:npm_tarball_prefix_test //tests/npm:npm_multi_version_test //tests/npm_types_barename:test_config_agreement //tools/changelog:changelog_test //:refresh_tsconfig_test
[... 9 lines ...]
//:refresh_tsconfig_test                                                 PASSED in 0.0s
[... 18 lines ...]
//tests/workers:library_paths_test                                       PASSED in 0.0s
[... 2 lines ...]
Executed 21 out of 21 tests: 21 tests pass.
[... 2 lines ...]
# exit=0

The first full suite (11-suite.log, tree f78a6e9) found the two tests the targeted set had missed: //tests/npm:npm_multi_version_test (FoundDir("*rolldown_pluginutils__1_0_1") then package.json in it) and //tests/npm_types_barename:test_config_agreement (116 resolves to a different file in each config lines, every one the build's node_modules/<name>/ against the editor's bare path), Executed 371 out of 371 tests: 369 tests pass and 2 fail locally. Both moved to the layout, the two pass (13-green-two.log, Executed 2 out of 2 tests: 2 tests pass.) and the suite on tree b72d8a1, this change on 2bcc23f (14-suite.log):

$ bazelisk --output_base=/var/tmp/fix8-J1-ob test --noshow_progress --nocache_test_results --local_test_jobs=2 //...
[... 5 lines ...]
INFO: Build completed successfully, 372 total actions
[... 1 lines ...]
Executed 371 out of 371 tests: 371 tests pass.
[... 2 lines ...]
# exit=0

bazelisk build --config=ci //... --output_groups=+_validation completes on tree b72d8a1 (15-ci-build-validation.log, exit 0) and at d3c9520, the restacked stack's tip (logs/fix8r/restack/13-ci-build-validation.log, exit 0). On this tree, 6824a23: buildifier --mode=check -r ., gofmt -l . and go vet over CI's package list exit 0 (logs/fix9r/J1/03-lint-buildifier.log, 04-lint-gofmt.log, 05-lint-govet.log); uvx --from mkdocs-material mkdocs build --strict builds the site (06-lint-mkdocs.log, exit 0); tools/ci/check_test_sources.sh (134 test source files, all claimed by a test target; 6 manual-only, allowlisted) and tools/ci/check_integration_shards.sh (npm 5, core 10) exit 0 (07-check_test_sources.log, 08-check_integration_shards.log). bazel run //gazelle -- -mode=diff on tree b72d8a1 prints 595 lines, as on the stack tip (logs/fix8r/P/05-modediff.stdout); the two lines that differ are a context line and a hunk header inside the asset_library proposal Gazelle already makes for tests/workers/BUILD.bazel, moved down by the six lines this PR adds to that file (17-modediff.log). Of the 531 npm repositories the suite fetched into output base fix8-J1-ob, none ships a node_modules of its own and none a boundary file at its package root (12-tree-and-markers.log), so the staging move and the deletion run on every fetch and the cases they exist for are pinned by no fixture here.

This head, ad8a66c, is 94c641f -- 870658d rebased onto eb947cd with the patch unchanged, git range-diff prints = (logs/fix8r/restack/08-rebase-J1.log) -- amended with comment, docstring and docs hunks in six files, 14 insertions and 18 deletions (git diff --stat 94c641f ad8a66c, logs/fix9r/J1/09-diffstat.log; the seven hunks are in 11-polish-diff.log), and the subject shortened to 68 characters. The full suite at d3c9520, J2 on 94c641f, is in #160's body (logs/fix8r/restack/12-suite.log, 380 of 380).

On the trial

Nothing in lov-paraglide is edited by this PR. Measured at cae299e with the override at 2bcc23f, //packages/workers-wide-event-sink:workers-wide-event-sink reports one error TS line, the TS2306 above (logs/trial-7/measure/scoped.log:17). With the override at ad8a66c alone that line is expected to stay: the pin still names index.d.ts, and the segment changes what TypeScript does with a paths hit that is a .ts, of which the sandbox holds none until J2. No diagnostic count is expected to move from this half. What the trial will see is every paths value in every generated tsconfig carrying node_modules/<name>/, exec paths two segments longer (external/+npm+npm__zod__4_1_5/node_modules/zod/index.d.ts), node_modules trees without each package's BUILD.bazel and REPO.bazel copy, and @parcel/watcher-wasm's shipped node_modules going through the staging move; whether any script in the monorepo matched an exec path by repository name is the trial's to find. With J2 on top, the index.ts pin for @cloudflare/workers-types is expected to resolve as a library file, replica-r1c's 0 errors rather than replica-r1's TS6059.

Base branch: mikn/ts-test-untyped-packages (1edaa40); head a1fb2e7, the polished commit ad8a66c rebased onto it with its patch unchanged (logs/fix9r/restack/restack-mechanical.log).

npm_import extracted a package at the root of its external repository, so
every path the ruleset wrote for it -- a tsconfig `paths` value, an action
input, an exec path -- lacked a node_modules segment. TypeScript classifies a
`paths` match by that segment alone: under one the file is a library file,
type-checked and never emitted; under none it is project source,
emit-eligible and checked against rootDir. The .d.ts files the sandbox stages
today are exempt either way, which is why nothing failed on it; a .ts reached
through `paths` is TS6059 the moment a package's module entry is one
(rt-evidence/logs/inv7/wide-event-sink-ts2306, replica-r1 against
replica-r1c).

The tarball is still extracted and patched at the repository root, where
rctx.patch applies, and then moved under node_modules/<name>/ through a
staging directory, since a package can ship a node_modules of its own
(@parcel/watcher-wasm). A BUILD, WORKSPACE, MODULE.bazel or REPO.bazel shipped
at the package root is deleted: at the repository root the generated BUILD
file overwrote it, and one level down Bazel would read it as a boundary and
the glob would return nothing. The stanza writes package_dir, package_files,
exports_types and exports_files under that directory. Every consumer derives
its paths from package_dir.dirname, so ts_npm_package, ts_compile,
tsconfig_aspect and node_modules are unchanged; the node_modules tree and the
editor's .bazel/npm/<name>/ copies are laid out from the package root as
before, and `bazel run //:refresh_tsconfig` regenerates the checked-in
tsconfig.json with no diff. The tree no longer carries each package's
generated BUILD.bazel and REPO.bazel, which the root glob had matched.

//tests/workers:library_paths_test asserts every `paths` value written for
@cloudflare/workers-types carries /node_modules/@cloudflare/workers-types/;
the written-form test in tests/npm/exports_types_tests.bzl asserts the three
labels the stanza writes; the tests that matched an exec path by repository
name alone -- link_depth, precedence, package_root, exports_pattern,
config_agreement and the Go tests over runfiles -- expect node_modules/<name>/
after it.

Red on 2bcc23f with the tests in the tree and the .bzl untouched
(rt-evidence/logs/fix8r/J1/01-red.log):

    $ bazelisk --output_base=/var/tmp/fix8-J1-ob test --noshow_progress --nocache_test_results --local_test_jobs=2 --test_summary=short --test_output=errors --keep_going //tests/npm:exports_types_tests //tests/npm:type_references_tests //tests/npm:link_depth_tests //tests/workers:library_paths_test //tests/npm_types_barename:types_package_root_test //tests/npm_subpath:wildcard_root_test //tests/npm_exports_pattern:pattern_paths_test //tests/npm:npm_patched_dependency_test //tests/npm:npm_platform_independent_graph_test //tests/npm:npm_tarball_prefix_test
    [... 5 lines ...]
    In test _written_form_test from //tests/npm:exports_types_tests.bzl: vite@8.2.2: exports_types names a file under the package root, not a repository (Expected "node_modules/vite/dist/node/index.d.ts", but got "dist/node/index.d.ts")
    [... 192 lines ...]
    In test _library_paths_impl from //tests/workers:library_paths_tests.bzl: @cloudflare/workers-types names a file outside node_modules, which TypeScript takes for project source: ../../../../../external/+npm+npm_workers__cloudflare_workers-types__5_20260825_1/index.d.ts
    In test _library_paths_impl from //tests/workers:library_paths_tests.bzl: @cloudflare/workers-types/* names a file outside node_modules, which TypeScript takes for project source: ../../../../../external/+npm+npm_workers__cloudflare_workers-types__5_20260825_1/*
    [... 44 lines ...]
    Executed 17 out of 17 tests: 9 tests pass and 8 fail locally.
    [... 2 lines ...]
    # exit=3

Green on this tree (rt-evidence/logs/fix9r/J1/02-green.log):

    $ bazelisk --output_base=/var/tmp/fix9-J1-ob test --noshow_progress --nocache_test_results --local_test_jobs=2 --test_summary=short --test_output=errors --keep_going //tests/npm:exports_types_tests //tests/npm:type_references_tests //tests/npm:link_depth_tests //tests/workers:library_paths_test //tests/npm_types_barename:types_package_root_test //tests/npm_subpath:wildcard_root_test //tests/npm_exports_pattern:pattern_paths_test //tests/npm:npm_patched_dependency_test //tests/npm:npm_platform_independent_graph_test //tests/npm:npm_tarball_prefix_test //tests/npm:npm_multi_version_test //tests/npm_types_barename:test_config_agreement //tools/changelog:changelog_test //:refresh_tsconfig_test
    [... 9 lines ...]
    //:refresh_tsconfig_test                                                 PASSED in 0.0s
    [... 18 lines ...]
    //tests/workers:library_paths_test                                       PASSED in 0.0s
    [... 2 lines ...]
    Executed 21 out of 21 tests: 21 tests pass.
    [... 2 lines ...]
    # exit=0

Full suite on tree b72d8a1, this change on 2bcc23f (fix8r/J1/14-suite.log).
The tree here is that one rebased onto eb947cd, `git range-diff` `=`
(fix8r/restack/08-rebase-J1.log), plus comment, docstring and docs hunks in
six files (fix9r/J1/11-polish-diff.log):

    $ bazelisk --output_base=/var/tmp/fix8-J1-ob test --noshow_progress --nocache_test_results --local_test_jobs=2 //...
    [... 5 lines ...]
    INFO: Build completed successfully, 372 total actions
    [... 1 lines ...]
    Executed 371 out of 371 tests: 371 tests pass.
    [... 2 lines ...]
    # exit=0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant