Modularizing stdexec

[ispeters]

I've been chatting with Claude 5 Sonnet to work out a plan for how to modularize stdexec. My initial foray into the problem space led to #2138. I've got a follow-up diff pending that extends the module beyond the proof of concept but I've run into problems with the stdexec tests using notionally-private metaprogramming utilities. I had a fairly long conversation with Claude to figure out a way forward. The result is rest of this issue, which Claude created for me here.

Plan: layered module architecture for stdexec (stdexec / stdexec.meta / stdexec.authoring)

Problem

As we extend C++20 module support past the minimal smoke test (#2138) into the
real test suite, test headers like test/test_common/type_helpers.hpp reach
into implementation-detail metaprogramming (__mand, __mapply,
__mcontains, __msize, __mset, __mset_eq, __result_of, from
__meta.hpp) that isn't exported from the stdexec module interface. This
raises a broader question than "how do we unblock this one test": what should
the module boundary between "public API," "internal implementation," and "the
tools an out-of-tree library needs to build a stdexec-like sender library"
actually be, once we're free to design it cleanly instead of inheriting the
informal __-prefix convention from the header-only build?

Two consumer classes need to be handled differently:

1. Regular consumers — write senders/receivers against stdexec's public
   API. Have no business with __meta.hpp or __sexpr_defaults.
2. Sender-library authors — write something like exec, in-tree or
   out-of-tree, and legitimately need stdexec's metaprogramming vocabulary and
   basic-sender internals (__sexpr_defaults etc.) to implement their own
   algorithms.

Options considered and rejected

• Export __-prefixed internals directly from stdexec. Works, but
  permanently pollutes the one public module surface for all consumers, not
  just authors, with implementation detail that was never meant to be
  read as stable API.
• A second, independently-compiled stdexec.internal module that
  #includes the same private headers stdexec does, exporting more than the
  public build. This looked promising for tests (a closed program that only
  ever imports one variant is internally consistent), and we verified the
  export-gating mechanics work as expected (macro-toggled export, macro
  -substituted module name, tested against Clang 18). It breaks down for
  anything that needs to interoperate with regular stdexec consumers
  (e.g. a future stdexec.exec), because a second independent compilation of
  the same headers produces distinct module-linkage entities with the same
  name — __mset<...> built inside stdexec and __mset<...> rebuilt inside
  stdexec.internal are not the same type, even though they look identical.
  Any code that needs to pass values across that boundary (e.g. a sender built
  with plain stdexec handed into an stdexec.exec-based algorithm) would
  hit real type-identity mismatches, not just inconvenience. This ruled out
  "internal-only module as a general strategy" for anything beyond a
  self-contained test binary.

Chosen design: shared modules via import, never duplicate compilation

The fix is to make sure any internals shared across module boundaries are
compiled exactly once and reached everywhere else via import /
export import — never re-declared by #include-ing the same headers into a
second module's purview. export import re-exports an already-compiled
entity; a second #include recompiles it as a distinct entity. This
distinction is the load-bearing rule for the whole plan below.

Module layout

• stdexec.meta — the general-purpose metaprogramming layer currently
  living in __meta.hpp (plus its dependency closure — __concepts.hpp,
  __type_traits.hpp, __typeinfo.hpp, roughly the same header set
  __execution_fwd.hpp already treats as foundational). No stdexec-specific
  business logic — analogous in spirit to a standalone library like
  Boost.MP11.
• stdexec.authoring — an SPI module for sender-library authors. Exports
  __sexpr_defaults and the rest of the basic-sender implementation, and
  re-exports what an author needs to have in one place:
  export import stdexec; + export import stdexec.meta;.
• stdexec — the public module, unchanged in spirit from today. Uses
  stdexec.meta and the basic-sender internals to build itself, but links
  them PRIVATE — it does not re-export them.
• stdexec.exec (future) — becomes an ordinary (if privileged) consumer
  of stdexec.authoring, the same as any third-party algorithm-author
  library would be, rather than a special case wired directly into stdexec's
  internals. If exec can be built entirely against stdexec.authoring's
  exported surface, that's the validation that the authoring module's surface
  is sufficient for third parties too.

Why mixing stdexec and an stdexec.authoring-based library is safe

export import stdexec; inside stdexec.authoring doesn't redeclare
anything — it re-exports the literal entities from the single compiled
stdexec BMI. just_t reached via a consumer's own import stdexec; and
just_t reached transitively through a third-party library's
import stdexec.authoring; are the same entity, not two lookalike ones. This
only holds as long as every layer consistently imports/re-exports rather than
re-including headers into a new module's purview — worth documenting
prominently near the top of any new .cppm so a future contributor doesn't
accidentally reopen the type-identity trap described above.

Shipping model / visibility

C++20 modules don't currently have a portable, cross-toolchain binary
distribution story — BMIs are compiler/version/flag-specific (we've already
hit this with clangd needing to be pinned to the same LLVM release as the
compiler, due to PCM format incompatibility across major versions). In
practice, "shipping" a modularized library means shipping the module
interface source, which the consumer's own toolchain compiles locally as
part of their build — the same vendored-from-source model Sendosio already
uses for its stdexec dependency.

Given that, the actual lever for keeping stdexec.meta and
stdexec.authoring out of the ordinary consumer's hands is CMake's
PUBLIC/PRIVATE FILE_SET CXX_MODULES distinction, not a language-level
mechanism:

target_sources(stdexec
  PUBLIC
    FILE_SET stdexec_modules TYPE CXX_MODULES FILES modules/stdexec.cppm
  PRIVATE
    FILE_SET meta_modules TYPE CXX_MODULES FILES modules/stdexec_meta.cppm
)

• stdexec.meta: PRIVATE everywhere. Not installed, not exported, no
  stdexec::meta target — pure build-time dependency of stdexec and
  stdexec.authoring.
• stdexec.authoring: installed and exported, but as a separately
  requested CMake component (find_package(stdexec COMPONENTS authoring)),
  gated behind an explicit, defaulted-OFF build option (e.g.
  STDEXEC_ENABLE_AUTHORING_API) whose help text states plainly that it's an
  unstable SPI with a weaker compatibility policy than the rest of stdexec.
  None of this is compiler-enforced — a motivated regular user could still
  flip the option — but that's the right amount of friction for this
  audience: the goal is preventing accidental use, not hard enclosure.

This is a packaging/build-graph boundary, not a language-enforced one. In a
from-source/vendored consumption model, the source for stdexec.meta is
still physically present in the checkout, same as any __detail header is
today; what PRIVATE buys us is that it's excluded from the sanctioned
path (no leakage via find_package, no accidental import), not literal
unreachability.

Immediate next step (this issue is about groundwork, not the full split)

For now, to unblock the growing test suite, we're going to take the direct
route and export the needed __-prefixed symbols straight from stdexec,
deferring the actual stdexec.meta / stdexec.authoring extraction. To keep
that shortcut cheap to undo later:

• Introduce two macros now, both currently equal to STDEXEC_MODULE_EXPORT:

  // Placeholder until stdexec.meta / stdexec.authoring exist as separate
  // modules; these currently just export into stdexec itself. See #<this
  // issue>.
  #define STDEXEC_MODULE_EXPORT_META      STDEXEC_MODULE_EXPORT
  #define STDEXEC_MODULE_EXPORT_AUTHORING STDEXEC_MODULE_EXPORT

• Tag __meta.hpp exports with STDEXEC_MODULE_EXPORT_META and
  __sexpr_defaults/basic-sender-internal exports with
  STDEXEC_MODULE_EXPORT_AUTHORING, instead of the plain
  STDEXEC_MODULE_EXPORT used for genuinely public API. Classify reactively,
  driven by build errors as today's test-porting work already does — no need
  to front-load a full audit.

• When the split eventually happens, grep -rn STDEXEC_MODULE_EXPORT_META
  and grep -rn STDEXEC_MODULE_EXPORT_AUTHORING are the manifests of what
  moves where.

• Avoid defining new metaprogramming helpers inline outside __meta.hpp's
  existing dependency closure for convenience — that closure is what makes
  the eventual extraction boundary tractable; ad hoc helpers placed outside
  it become extraction bugs later.

[ericniebler]

Generally, I like this direction. I'm luke warm on stdexec.authoring, though. Ultimately, I'd like to replace all the __sexpr machinery with ordinary sender types. basic-sender is a specification tool. It makes algorithm specifications smaller. It isn't necessarily the best way to build a real sender library though. But replacing __sexpr with custom senders everywhere is a big job. A smaller job would be to discontinue its use in exec::. Then I think we can do without stdexec.authoring.