[Feature] Sub-agent Resource for Flink Agents - Framework part

[pltbkd]

[Feature] Sub-agent Resource for Flink Agents

Motivation

As discussed in #660, there are scenarios where a Flink Agent needs to call external agents — for example, delegating a subtask to a remote LLM agent service or a third-party agent platform. Currently, users can only do this by manually writing remote calls inside actions, which means they miss out on durable execution, checkpoint recovery, and tool call integration.

Beyond external agent calls, several related scenarios have emerged:

• Parallel exploration: Multiple agents working in different directions simultaneously, each with its own reasoning loop and tools.
• Independent review: An agent providing an independent perspective — e.g., a reviewer agent that evaluates code without being influenced by the main agent's context.
• Reusing agents as sub-links: Composing an agent defined by another team as a building block in a larger agent pipeline.

These scenarios also create demand for internal sub-agents — the ability to register a Flink Agents Agent as a sub-agent of another Agent, with proper context isolation and execution semantics.

This proposal introduces AGENT as a first-class resource type in Flink Agents, supporting both internal and external sub-agents with a unified API.

Goals

• Introduce AGENT resource type: Sub-agents can be registered via addResource(name, AGENT, ...) and accessed via ctx.getResource(name, AGENT), usable by both Actions and ChatModel.
• Support for external, internal, and Remote Agents: Supports custom external agents, supports qualified internal agents, and will later implement remote parallel execution of internal agents based on RpcOperator.
• Context isolation, DurableExecute, and failover recovery: Both internal and external sub-agents benefit from durable execution with automatic failover recovery. Internal agents introduce scope-level isolation.

The proposed changes have been initially validated with a POC (mainly on java api). The overall design still has room for discussion — feedback and suggestions are welcome.

---

Proposed Interface

1. Resource Type

public enum ResourceType {
    CHAT_MODEL, CHAT_MODEL_CONNECTION, TOOL, EMBEDDING_MODEL,
    VECTOR_STORE, PROMPT, MCP_SERVER, SKILLS,
    AGENT  // Sub-agent resource
}

2. Subagent Interface

Subagent is the common interface for all sub-agents. Both external (SubagentSetup) and internal sub-agents implement it.

public class Result {
    boolean success;        // success or failure
    Object result;       // type depends on agent implementation; built-in agents return List<Object>
    Exception exception; // present when success=false
}

public interface Subagent {
    default Result call(RunnerContext ctx, Object prompt) {
        return call(ctx, prompt, ctx.nextSessionId());
    }

    default Result call(RunnerContext ctx, Object prompt, String sessionId) {
        return ctx.durableExecuteAsync(asAsyncCallable(ctx, prompt, sessionId));
    }

    default DurableCallable<Result> asAsyncCallable(RunnerContext ctx, Object prompt) {
        return asAsyncCallable(ctx, prompt, ctx.nextSessionId());
    }

    DurableCallable<Result> asAsyncCallable(RunnerContext ctx, Object prompt, String sessionId);
}

class Result:
    success: bool
    result: Any 
    exception: Optional[Exception]

class Subagent(Protocol):
    def call(self, ctx: RunnerContext, prompt: Any, session_id: Optional[str] = None) -> Result:
        if session_id is None:
            session_id = ctx.next_session_id()
        return ctx.durable_execute_async(
            self.as_async_callable(ctx, prompt, session_id))

    def as_async_callable(self, ctx: RunnerContext, prompt: Any, session_id: Optional[str] = None) -> DurableCallable[Result]:
        if session_id is None:
            session_id = ctx.next_session_id()
        ...

SubagentSetup is the abstract base for external sub-agents, implementing Subagent:

public abstract class SubagentSetup extends SerializableResource implements Subagent {

    @Override
    @JsonIgnore
    public ResourceType getResourceType() { return ResourceType.AGENT; }
}

class SubagentSetup(SerializableResource, Subagent):
    def get_resource_type(self) -> ResourceType:
        return ResourceType.AGENT

3. Registration

Internal and external sub-agents are registered identically, only via addResource, not via annotation:

// Internal sub-agent
addResource("reviewer", ResourceType.AGENT, new ReviewerAgent());

// External sub-agent
addResource("summarizer", ResourceType.AGENT, new HttpSubagentSetup(endpoint));

# Internal
self.add_resource("reviewer", ResourceType.AGENT, ReviewerAgent())

# External
self.add_resource("summarizer", ResourceType.AGENT, HttpSubagentSetup(endpoint))

agents:
  - name: reviewer-agent
    module: com.example
    clazz: ReviewerAgent

  - name: root-agent
    module: com.example
    clazz: RootAgent
    subagents:
      - name: reviewer
        agent_ref: reviewer-agent          # internal: reference to another agent definition
      - name: summarizer
        target_module: com.example          # external: reference to SubagentSetup subclass
        target_clazz: HttpSubagentSetup
        arguments:
          endpoint: "http://summarizer:8080"

4. Calling

// Synchronous
Subagent subagent = ctx.getResource("reviewer", ResourceType.AGENT);
Result result = subagent.call(ctx, prompt);

// Parallel
DurableCallable<Result> c1 = subagent1.asAsyncCallable(ctx, prompt1);
DurableCallable<Result> c2 = subagent2.asAsyncCallable(ctx, prompt2);
List<Result> results = ctx.durableExecuteAllAsync(c1, c2);

# Synchronous
subagent: Subagent = ctx.get_resource("reviewer", ResourceType.AGENT)
result: Result = subagent.call(ctx, prompt)

# Parallel
c1 = subagent1.as_async_callable(ctx, prompt1)
c2 = subagent2.as_async_callable(ctx, prompt2)
results: list[Result] = await ctx.durable_execute_all_async(c1, c2)

5. Implementing External Sub-agent

public class HttpSubagentSetup extends SubagentSetup {
    private final String endpointUrl;

    public HttpSubagentSetup(String endpointUrl) {
        this.endpointUrl = endpointUrl;
    }

    @Override
    public DurableCallable<Result> asAsyncCallable(RunnerContext ctx, Object prompt, String sessionId) {
        return () -> {
            List<Object> output = httpClient.post(endpointUrl,
                    Map.of("prompt", prompt, "session", sessionId));
            return new Result(true, output, null);
        };
    }
}

class HttpSubagentSetup(SubagentSetup):
    def __init__(self, endpoint: str):
        self._endpoint = endpoint

    def as_async_callable(self, ctx: RunnerContext, prompt: Any, session_id: str) -> DurableCallable[Result]:
        def _call() -> Result:
            output = httpx.post(self._endpoint, json={"prompt": prompt, "session": session_id})
            return Result(success=True, result=output, exception=None)
        return DurableCallable(_call)

---

Execution Process

1. Compilation

Both internal and external sub-agents compile to SerializableResourceProvider in the plan JSON, ensuring cross-language interoperability:

AGENT resource processing
  │
  ├── SubagentSetup (external)
  │     → SerializableResourceProvider
  │
  └── Agent (internal)
        → Compiled into child AgentPlan
          → InternalSubagentSetup → SerializableResourceProvider

Both paths produce SerializableResourceProvider, resulting in a uniform plan JSON representation with no runtime difference.

Additionally, the compiler registers built-in sub-agents: if CHAT_MODEL is available, ReActAgent is automatically registered as "general-purpose". This registration runs after all user-declared resources are extracted, ensuring decorator-declared CHAT_MODEL is ready.

2. Execution Model

The default call() implementation executes the DurableCallable returned by asAsyncCallable via durableExecuteAsync, leveraging the durable execution mechanism for state recording and recovery.

Parallel calls: users manually split — call asAsyncCallable for each sub-agent, then pass all DurableCallables to durableExecuteAllAsync (to be introduced in an upcoming discussion).

call() returns Result; sub-agent implementations should intercept internal exceptions and populate Result, without directly exposing them to the caller. tool_call_action constructs success/error ToolResponseEvent based on Result, without try/catch.

3. ChatModel Integration

Sub-agents are exposed to the LLM as tools (Subagent as Tool pattern).

• Prompt injection: Available AGENT resources are listed in the system prompt alongside tool descriptions, letting the LLM know which sub-agents it can delegate to.
• Dispatch: tool_call_action checks resource type — TOOL resources go to tool.call(), AGENT resources go to subagent.call(ctx, prompt).
• Result: Returned via existing ToolResponseEvent.

LLM response → tool_calls → ToolRequestEvent → tool_call_action
  │
  ├── name matches TOOL resource → tool.call(**kwargs)
  └── name matches AGENT resource → subagent.call(ctx, prompt)
  │
  └── ToolResponseEvent → back to LLM

LLM APIs (OpenAI, Ollama, etc.) only have tool/function call concepts — there is no "agent call". Sub-agents are exposed as tools with a single prompt parameter. The tool_call_action already holds ctx as an action parameter, which is passed directly to subagent.call(ctx, prompt).

---

Internal Sub-agent Design Preview

Internal sub-agent compilation — registering Agent instances as sub-agents — will be discussed in a future proposal. The resource model and calling API are already designed to accommodate it.

1. Compilation

The sub-agent's Agent is compiled into an independent AgentPlan (childPlan), wrapped as InternalSubagentSetup(childPlan, scope). The child plan inherits the parent plan's AgentConfiguration. Circular references are detected at compile time. scope is the sub-agent's resource name, used for event routing.

2. Invocation

Calling a sub-agent emits a SubagentCallEvent. During async execution, the coroutine yields, and the AEO processes the event in a subsequent mailbox cycle, scheduling the sub-agent's internal actions. Only the event sending timing is modified for SubagentCallEvent to support this execution model — asAsyncCallable sends SubagentCallEvent on the mailbox thread, and durableExecuteAsync blocks on the worker thread waiting for the result.

3. Scope-based Execution

SubagentCallEvent is converted to InputEvent within the sub-agent's scheduling scope. Intermediate events are dispatched only within that scope — they do not leak to the parent agent or other sub-agents. The result OutputEvent is intercepted and returned to the caller. When all pending events are processed and no actions are running (quiesce), the call completes automatically.

4. Isolation

SubagentRunnerContext provides isolated context:

• Actions and events: Sub-agent uses its own AgentPlan's definitions, fully independent from parent.
• Memory: Read-only access to parent memory; Sensory memory is allowed but not written back to parent scope memory.
• Resources: Uses own registered resources first, falls back to parent (parent resource cache inheritance).
• Config: Inherits parent plan's AgentConfiguration.

Nested sub-agents are supported recursively via parent resource cache fallback.

5. Fault Tolerance

• Completed calls: Intercepted directly by DurableExecute mechanism; failover hits cached result.
• Incomplete calls: Supports ActionState; after failover, deterministic SessionId associates the new sub-agent call with recovered Actions, enabling intermediate state recovery rather than full re-execution.

---

Design Discussion

1. Internal vs External Capability Parity

External and internal sub-agents provide identical user-facing API (registration, call(), asAsyncCallable()). However, the underlying implementations differ significantly — external sub-agents are black-box calls (HTTP/gRPC), while internal sub-agents are framework-internal plan nesting execution.

Capability	External Sub-agent	Internal Sub-agent
Context isolation	Managed by external system via deterministic SessionId	Framework creates isolated SubagentRunnerContext
Memory	Maintained by external system	Read-only parent access, write-isolated, not persisted
State recovery	DurableExecute persists call records	Same as external; additionally, deterministic SessionId links call chain to sub-agent internal action ActionState, supporting intermediate state recovery
Parallel execution	Multiple agents in parallel via durableExecuteAllAsync	One action at a time per ActionExecutionOperator, but coroutine-based async partial parallelism
Backpressure	Via durableExecuteAsync worker pool	Same as external; needs to consider subagent nested call deadlock issues

2. Introducing SubagentCompatible Marker

When an Agent is registered as a sub-agent, it runs in an isolated SubagentRunnerContext, meaning certain capabilities are restricted, such as the inability to write to memory. Not all Agents can correctly accommodate these constraints.

To ensure registration correctness, we propose a SubagentCompatible marker (interface / annotation / yaml field) that an Agent declares to indicate it is safe to run as a sub-agent. The framework validates this marker at compile time: when an unmarked Agent is registered as a sub-agent, compilation fails fast.

This provides an explicit contract: Agent authors must evaluate and declare compatibility with sub-agent constraints (e.g., no dependency on writing to parent memory, no reliance on cross-scope events), avoiding hard-to-debug behavioral anomalies at runtime.

We need to evaluate what capability restrictions sub-agents have, to confirm whether this marker is necessary.

---

To Be Refined

• Timeout / cancellation semantics: call() currently does not provide timeout or cancellation. Further investigation of requirements is needed to refine the design.
• Sub-agent prompt injection and dispatch chain for ChatModel: The prompt injection format for exposing sub-agents as tools to the LLM, and the tool_call_action dispatch chain, need further clarification.
• External agent support framework: Consider providing SubagentSetup implementation helpers for gRPC, A2A (Agent-to-Agent) protocol, etc., to reduce the cost of integrating external agents.
• RpcOperator-based RemoteAgent design: The remote execution approach via RpcOperator (FLIP-577) on a separate ActionExecutionOperator is yet to be designed.

[joeyutong]

Thanks for the proposal. I understand the motivation for providing a unified invocation API for internal and external agents. However, I think the current Subagent abstraction mixes the agent being registered with the capability used to invoke it.

From the user's perspective, an external agent is a black-box invocation target, so call() naturally matches its execution boundary. An internal Agent, however, is already presented to users through actions, resources, and event-driven execution. Exposing the same Agent as a callable sub-agent makes users reason about two public execution models for the same abstraction: it is event-driven when used as a regular Agent, but request/response-oriented when used as a sub-agent.

This is not only an internal implementation difference. To understand the behavior of call(), users also need to know how the request becomes an input event, which output events form the result, and what marks the call as complete, such as quiescence. In other words, the callable view is exposed to users, while the semantic adaptation that creates this view remains implicit.

Therefore, the proposal unifies the invocation shape, but does not fully unify the user-facing semantics. From this perspective, the shared abstraction seems closer to a SubagentInvoker:

• for an external agent, it delegates to the invocation protocol provided by the external integration;
• for an internal agent, it invokes the Agent through a CallProtocol that defines request, result, and completion mappings.

Under this model, the authoring experience remains different where it should be. For an external agent, its integration can provide a custom SubagentInvoker that encapsulates the agent endpoint and its invocation protocol. For an internal agent, users continue to define and register the Agent itself as ResourceType.AGENT; the framework provides a built-in SubagentInvoker, invisible to users, that drives the Agent according to the internal call protocol. Users therefore do not construct or register this invoker as another kind of Agent: SubagentInvoker is the invocation mechanism, while Agent remains the user-facing authoring abstraction.

This would preserve a unified API for callers without exposing call() as a second intrinsic execution model of every Agent.

[pltbkd]

Hi joeyutong,

Thanks for sharing your thoughts. I understand the confusion. The InternalSubagent preview in the proposal intentionally simplified many of the details, so let me briefly clarify a few points here.

First, I'd like to clarify that Subagent is a separate interface intended for callers and is independent of the concrete Agent implementation. Callers always use subagent.call(), regardless of whether the target is an internal or external subagent. However, only external subagents need to implement this method themselves; for internal subagents, the framework handles it transparently. In that sense, its role is essentially the same as the SubagentInvoker abstraction you described.

Any Agent that satisfies the required constraints can be registered as an internal subagent without any modification. InternalSubagent is a framework-provided abstraction that translates a call() request into an InputEvent within the subagent scope and returns the subagent's OutputEvent as the call result. Once the event is emitted, the calling Action yields, and the remainder of the execution continues under the existing event-driven execution model. There will be some framework changes to ensure that SubagentCallEvent can be dispatched and processed at that point.

In short, callers only need to understand subagent.call() and do not need to care whether the target is internal or external. Implementers only need to develop a standard Agent—they do not need to modify it or understand or implement call(). From the framework's perspective, there is still only a single event-driven execution model; subagent.call() is simply a resource-oriented invocation interface built on top of that model.

I hope this addresses your concerns. If anything is still unclear, I'd be happy to discuss it further here, or we can dive into the details in the dedicated InternalSubagent discussion.

[joeyutong]

Thanks for the clarification. I think we are aligned on the implementation architecture. What you describe as InternalSubagent is essentially the framework-provided implementation of the internal call protocol that I mentioned above. It also clarifies the distinction between the underlying Agent, which users author, and the caller-side invocation capability, which the framework provides for an internal agent.

Subagent is the common interface for all sub-agents. Both external (SubagentSetup) and internal sub-agents implement it.

My remaining concern is how this distinction is represented in the public API. While call() is a natural contract for an external integration, the proposal applies the same public Subagent abstraction to internal agents. An internal Agent author does not need to implement call(), but call() still becomes part of how the internal subagent is publicly represented, even though it is a framework-provided capability rather than part of the Agent authoring contract.

Therefore, my concern is not with the InternalSubagent implementation. This is why I characterized the shared interface as being closer to SubagentInvoker: not necessarily to prescribe a particular solution, but to emphasize that, for an internal sub-agent, the interface describes a framework-provided invocation capability rather than part of the Agent authoring contract.

[weiqingy]

Hi @pltbkd, Thanks for putting together this comprehensive proposal! The introduction of ResourceType.AGENT and the Subagent abstraction is a helpful step forward. It addresses the capability-agnostic orchestration vision discussed in #660 by treating internal agents, external setups, and tools as peer capabilities, while eliminating the need for users to split logical operations into inverted send/receive actions (Action-A / Action-A').

A few thoughts and questions regarding design details, open items, and alignment with earlier roadmap discussions:

• SubagentCompatible Marker vs. Runtime Enforcement:
  Just to confirm my understanding: when registering an internal agent directly via addResource(name, ResourceType.AGENT, new TargetAgent()), does compilation fail if TargetAgent wasn't originally declared with @SubagentCompatible? If so, this might create friction when users want to reuse existing or third-party agents whose source code they don't own or cannot easily modify. Since SubagentRunnerContext already enforces read-only parent memory access and scope-bounded events at runtime, do you think strict runtime enforcement (throwing a clear exception if forbidden operations like parent memory mutation are attempted) could be sufficient to catch anomalies without requiring compile-time marker declarations?

• External Idempotency Framework:
  Building on the discussion above regarding side-effect recovery during crash windows, you mentioned the potential for a shared framework for external sub-agents (SubagentSetup). How do you envision this framework enforcing sessionId-based idempotency automatically so that individual integration authors don't have to write custom deduplication/reconciliation logic manually?

• Nested Backpressure & Deadlocks:
  The proposal highlights potential deadlock issues in nested sub-agent calls under backpressure. Are you leaning toward addressing this via non-blocking yield mechanics in the execution operator, or by enforcing explicit call-stack depth / concurrency limits on worker pools?

• Timeout & Cancellation Semantics (Refining "To Be Refined"):
  Regarding the timeout/cancellation gap noted in the proposal, here are a few scenario requirements that might be worth considering:

  1. Resource & Token Control: Supporting a timeout context on call() to unblock the parent pipeline if a sub-agent stalls or hallucinates.
  2. Parallel Short-Circuiting: When running durableExecuteAllAsync(c1, c2), if one branch returns a conclusive answer early or fails, having a mechanism to pass a cancellation signal to the remaining in-flight callables so we don't waste LLM tokens or external compute.
  3. Propagation: Emitting a scope cancellation event within SubagentRunnerContext for internal agents, or triggering an HTTP abort/gRPC cancel for external SubagentSetup instances.

• Framework Primitives vs. Application Templates:
  Looking at this proposal alongside @xintongsong's suggestions in #660 which discussed higher-level user templates (e.g., introducing the sub-agent concept on top of the built-in ReActAgent, or a built-in SupervisorAgent handling routing, critic steps, and termination budgets): with these new primitives, do you think an out-of-the-box template layer remains valuable so users don't have to write custom orchestration logic? Is the main goal here to establish the primitive layer for templates to build upon, or does your POC also explore a built-in SupervisorAgent template?

Looking forward to hearing your thoughts!

[pltbkd]

Hi, thanks for sharing the thoughtful ideas!

SubagentCompatible Marker vs. Runtime Enforcement:

The marker is introduced mainly for safety, to prevent unexpected behaviors from Agents that do not satisfy the required constraints. Therefore, under the current design, such Agents cannot be directly used, which may reduce usability.

For InternalSubagent, we can detect invalid usage and fail at runtime, which may satisfy part of the requirement. However, @xintongsong pointed out that compile-time detection, especially when issues can be directly surfaced from YAML configuration, is also valuable for usability and agent friendliness. WDYT?

External Idempotency Framework:

The current sessionId/callId design follows the durable call idea. The framework derives a scope from the invocation chain and generates a deterministic ID by adding the call ordinal within that scope. The ID is part of the durable call identifier, so after failover we can short-circuited repeated calls and reuse previous results.

For external sessions, if specifying session IDs is supported, the repeated calls are skipped and they can use the ID to continue previous conversations (with possible reconciliation for the latest message). Otherwise, the conversation history should be included in the result and passed back by callers in subsequent requests, so the request itself is deterministic.

Nested Backpressure & Deadlocks:

Since subagent execution relies on durableExecuteAsync yielding and waiting, some async threads may be occupied during the wait. If the subagent also requires async execution internally, thread pool exhaustion could cause deadlocks.

Because Actions cannot execute in parallel, the caller has to yield while waiting for InternalSubagent, so returning an unfinished Future and continuing execution is not possible under the current model.

For solutions, since durableExecuteAsync mainly performs waiting, we leverage it only to reuse the durable execute and yielding mechanism, excluding it from the worker thread count or using a dedicated thread pool may be simple options. A solution that avoids additional threads while preserving these capabilities would be preferable and worth exploring.

Could you clarify what execution model you mean by "non-blocking yield mechanics"?

Timeout & Cancellation Semantics:

These requirements can be divided into timeout configuration, cancellation initiation/propagation, and cancellation handling.

Timeout is relatively straightforward: callers can provide timeout parameters, and subagents can set timers or propagate them to clients such as gRPC.

Cancellation is more complicated:

• From the caller side, the current API returns a Result containing only the final result. Due to Action serialization constraints, callers cannot naturally continue execution and actively trigger cancellation. However, cancellation caused by timeout or failures in parallel execution can be supported.
• External subagents can expose a non-blocking Future and allow callers to invoke cancel(). However, InternalSubagent cannot provide the same semantics because the caller cannot cancel an ongoing Action. Do you prefer the Future style interface to the current one?

For cancellation propagation, we may need a cancel method or Cancellable trait for DurableCallable, allowing the caller or framework to propagate cancellation signals. How to interrupt an executing DurableCallable still requires further design.

Ultimately, subagents need to handle cancellation themselves. External subagents need to send cancellation messages through their own mechanisms. For InternalSubagent, since Actions generally cannot be interrupted, cancellation/timeout checks can be performed between Actions via SubagentRunnerContext. Interrupting an ongoing Action (e.g., interrupting durable execution) is a separate issue.

Framework Primitives vs. Application Templates:

This proposal mainly focuses on framework primitives and does not include a SupervisorAgent design.

It would be helpful to discuss how a SupervisorAgent works, and how it could leverage subagents, which may also help identify further subagent requirements. Currently, subagents seem more valuable for InternalSubagent, while the distinction from tools/MCP is less obvious for ExternalSubagent. The unique value and requirements of subagents still need further exploration.

[weiqingy]

Hi @pltbkd, thanks for the detailed follow-up. Focusing strictly on the framework primitives for the initial PR sounds like a good approach.

SubagentCompatible marker: Sounds good. We can use the compile-time @SubagentCompatible marker for now and reconsider runtime enforcement if it creates friction.

Non-blocking yield mechanics: What I meant is that the wait itself should not require a thread. On the continuation path the Action already suspends without holding the mailbox thread: the continuation yields, and the operator re-queues the task, allowing other keys to interleave.

What still consumes a thread is the wait, because the executor's current resume condition is a Future produced by its own submission. A wait-only callable therefore parks an async-pool thread purely as a placeholder. Generalizing that resume condition to a completion handle that the mailbox or another thread may complete would let a wait-only invocation consume a continuation but no worker thread.

If a dedicated pool is the pragmatic solution for this iteration, I have no objection, provided nested calls cannot exhaust it.

Timeout and cancellation: Agreed on avoiding a caller-controlled Future interface in this iteration. Since an InternalSubagent Action generally cannot be interrupted mid-flight, framework-managed deadlines and cooperative checks between Actions through SubagentRunnerContext seem sufficient for the initial scope.

External idempotency: Deterministic session and call IDs make sense as the invocation identity. Reading @alnzng's example, the step that actually closes the completed-but-not-checkpointed window is reconciliation before resending, rather than the IDs alone. It therefore seems worth naming reconcile-before-resend explicitly as a requirement of the external-subagent support framework.

Async interface: Given the Python and JDK < 21 constraints, unifying on asynchronous result semantics sounds reasonable. I also agree that surfacing the durable-callable factory as the caller-facing method exposes an execution mechanism, and is not an ideal name for that surface. If possible, the public API could keep an invocation-oriented operation on the sub-agent surface, with DurableCallable staying the durability and batch-execution mechanism underneath rather than something a caller reaches for.

Templates: Agreed on establishing the framework primitives before higher-level templates such as SupervisorAgent.

[alnzng]

Thanks for this proposal @pltbkd . I think treating agents as a first-class resource is a good direction.

One thing I would like to understand better: for the external agent case specifically, what is the unique value of this design, compared with doing the same thing using the existing Tool or Action APIs? Today I can wrap a remote agent as a Tool. It registers, the LLM can call it, it runs under durable execution, and it takes a structured input. Or I can write an Action and call the remote directly. I just want to learn more here, not to question why we need this.

For extenal agent proposal, let's use a concrete example to discuss. Say I call LangGraph agents owned by another team. I give one a metric dashboard link, it pulls logs, runs its own tools, and returns a diagnosis. Their protocol is not one single HTTP request, it is three steps (check below). You POST a message to a thread, it starts a background run and returns right away. Then you open an SSE stream to watch that thread, and the run keeps going even if your client disconnects. And you can GET the thread status any time, which returns running, completed, failed, or not_started. The thread id is given by the client, and it gives multi-turn continuity.

1. Send a message (triggers background execution):
POST /api/agents/{agent_id}/threads/{thread_id}/messages
curl -X POST 'http://localhost:5110/api/agents/demo_agent_async/threads/my_thread_123/messages' \
  -H 'Content-Type: application/json' \
  -d '{"message": {"content": "Hello, how are you?"}}'
# → {"thread_id": "my_thread_123", "status": "started"}

2. Stream the response (SSE):
GET /api/agents/{agent_id}/threads/{thread_id}/stream
curl -N 'http://localhost:5110/api/agents/demo_agent_async/threads/my_thread_123/stream' \
  -H 'Accept: text/event-stream'

3. Check status:
GET /api/agents/{agent_id}/threads/{thread_id}
# → status: running | completed | failed | not_started

So this remote is not a stateless function, it is a durable async job. It already exposes, on the server side, a similar lifecycle that @ofekron described. The thread id is a stable invocation id. The status endpoint gives real states. Completion is explicit, it is status completed, not the SSE socket closing. This matters for recovery. If Flink crashes after the call started, I do not need to run the remote agent again. I can use the same thread id to ask the server what happened. If it is still running, I reconnect to the stream. If it already finished, I read the saved result. So after a crash, recovery can be a query, not a re-run.

This is where I am not sure the current design fits. The external agent endpoint splits submit and stream on purpose. The current design merges them back into one callable. So, on failover there is only one move, run the whole callable again. The server actually handles the easy case well. If I re-post while the run is still active, it short-circuits and just returns the status, and re-opening the stream re-attaches. The problem is the narrow window where the run already completed but Flink crashed before saving the result. On recovery the thread is no longer active, so a replay post is accepted and starts a new run, which the checkpointer appends as a new turn. That is duplicate work, and it corrupts a multi-turn thread. Deterministic SessionId does not help me here, because it maps to the thread id, and re-posting to that thread is not idempotent in this window.

On recovery, the right behavior could be like this. Do not send the message again. First ask the server for the thread status. If it is still running, re-attach to the stream. If it already finished, just read the result. If it failed, report the error. This needs the call to be split into two parts, submit and probe. But the current interface is a single callable, so there is no place to express this split. To make it work, I would have to bypass the abstraction and handle the low level durable call myself. At that point I am basically writing it by hand with a Tool plus an Action, which is what I can already do today. And that brings me back to my first question.

[pltbkd]

Hi @alnzng, thanks for sharing! Your questions are very valuable for thinking through this design.

My thought is that the interface should be designed from the caller’s perspective, while the implementation side can provide framework capabilities through a thicker common layer, which is what I referred to as the “External subagent support framework” in the proposal. I think this case is a good example to discuss the capability and value of such a framework, and we can further improve this part.

---

From the caller perspective, the current assumption is that the caller expects to obtain a result (or at least an acknowledgment) before continuing with subsequent work. Therefore, the interface is designed around obtaining a result.

We also considered another approach during the design process: callAsync/await.

• callAsync eagerly sends the request.
• await is introduced into the context to wait for the result. This is very similar to the submit/query model, except that await internally polls the query until the result is available.

The problem with this approach is:

• It cannot naturally reuse the durable callable mechanism, including possible future batch execution support.
• Internal subagents and Python subagents also cannot truly eager fire requests. They would only actually start execution and yield when await is reached, which does not achieve the expected behavior.

In both approaches, the assumption is that the result is eventually needed inside the caller action. I think this interface model matches the caller’s mental model better, and subagent should avoid exposing execution management and polling responsibilities to the caller.

Due to the execution model limitations of internal subagents, we also cannot expose a similar polling interface there. Otherwise, the common subagent interface would become different between internal and external implementations.

I also raised a related question in a previous sync meeting: whether subagent needs a pub/sub style interface. This model would break the assumption that callers expect a result or acknowledgment, but I am not sure whether there are many such scenarios. At least from my perspective, an acknowledgment is likely still necessary. What do you think about the interface design?

---

From the implementation perspective, I agree that the current subagent abstraction is almost no different from implementing the same thing as a Tool. The main additional semantics of subagent are independent context and lifecycle.

Since external agents are naturally isolated from the main agent, the design process focused more on internal agent scenarios and did not provide many external-agent-specific capabilities. My view is:

• The interface should focus on caller usage.
• The implementation side can provide framework capabilities through a thicker common layer, i.e. the External subagent support framework. This could be a future work item, and we can further discuss and refine it based on this feedback and the preferred interface design.

Regarding the continuity issue, I think using deterministic session ID (+ call ID) together with the durable execute mechanism can satisfy this requirement.

• The session ID can represent the remote thread/session identity.
• The call ID can represent an individual invocation and become part of the durable execute identity.
• During recovery, an in-flight request can use reconciliation to query the remote state and determine whether the request needs to be resent.

Even without relying on ActionState, the callable provided by Subagent can check the state before sending the request and decide whether to proceed. This logic belongs to the subagent implementation rather than the caller side.

Therefore, I think the re-post issue you mentioned can be avoided with the existing mechanisms. However, I agree that these capabilities are not currently provided directly as framework features and still depend on the subagent implementation. Tools and MCP can implement the same approach, but they require the caller to provide deterministic IDs and handle the lifecycle management themselves.

---

I think the current discussion mainly focuses on two aspects:

1. The interface design: whether we should introduce submit/query, or an asyncCall/await model.
2. How the External subagent support framework should be implemented and what capabilities it should provide.

My current preference is still not to expose the lifecycle protocol of external agents directly in the caller API, but rather let the framework layer absorb these differences. What do you think?

[pltbkd]

I'd like to add one more point: Python and internal subagents running on JDK < 21 cannot provide synchronous interfaces (due to the limitations of Python coroutines and the lack of coroutine support in JDK < 21). Currently, we are throwing an NYI (Not Yet Implemented) exception for these cases. However, I'm reconsidering whether this is the right approach, and wondering if it would actually be sufficient to just unify all subagent interfaces to be purely asynchronous?

Besides, the interface was changed from callAsync to asDurableCallable to leverage durableExecuteAll. I don't really like this method or its name, while introducing another way to support batch execution is also not suitable. WDYT?

[alnzng]

What do you think about the interface design?

• For the first version I think your assumption is reasonable: the caller (e.g. a supervisor) should expect a full result back from the sub-agents it manages. Request/response is the right default.
• For the future, we might want a "talker / thinker"-style model: the supervisor (thinker) reacts to a continuous stream of output from its sub-agents (talkers) rather than blocking for one final answer per call. That's the similar idea to the pub/sub interface you mentioned. So I'd keep request/response for v1 and just keep the door open for a streaming result later.

Therefore, I think the re-post issue you mentioned can be avoided with the existing mechanisms.

Agreed, I think you're right. It looks like reconciler() is exactly the submit/probe split, I didn't realize that DurableCallable already offers this capability. We can use the codes like below:

return new BaseSubagentCallable(sessionId, callId) {                                                                                                           
      @Override                                                                                                                                                  
      protected Object callInternal() throws Exception {          // normal path: submit + consume                                                               
          String threadId = sessionId;                                                                                                                           
          httpPost(base + "/threads/" + threadId + "/messages",                                                                                                  
                   Map.of("message", Map.of("content", prompt),                                                                                                  
                          "idempotency_key", sessionId + "#" + callId));                                                                                         
          return drainSSE(base + "/threads/" + threadId + "/stream");                                                                                            
      }                                                                                                                                                          
      @Override                                                                                                                                                  
      public Callable<Result> reconciler() {                     // recovery path: probe, don't re-submit                                                        
          return () -> {                                                                                                                                         
              var st = httpGet(base + "/threads/" + sessionId);                                                                                                  
              switch (st.status) {                                                                                                                               
                  case "completed":   return Result.ok(st.savedOutput);                                                                                          
                  case "running":     return Result.ok(drainSSE(...));   // re-attach                                                                            
                  case "failed":      throw new RuntimeException(st.err);                                                                                        
                  case "not_started": return Result.ok(callInternal()); // safe iff submit is idempotent                                                         
              }                                                                                                                                                  
          };                                                                                                                                                     
      }                                                                                                                                                          
  };            

So the mechanism is already there, that part is clear to me now.

---

But one thing I am still not sure: should we provide some built-in abstraction for this pattern, instead of let every integration write it by hand?

My worry is, a big group of external agents (LangGraph, OpenAI Assistants runs, A2A long-running task, etc.) are actually the same shape: an async job with three steps — submit → observe → status. If we don't give an abstraction, then every author needs to write the same callInternal + reconciler split again and again. More important, they also need to understand why there are two methods, when reconciler will be called, and how to handle the crash window. This is framework recovery knowledge, but it is not really their domain. They only want to describe their remote agent.

So maybe we can add an additive specialization, for example AsyncSubagentSetup, where the author only provides the primitives they already understand:

• submit(ctx, id, callId, prompt) — start the job;
• awaitResult(ctx, id) — block / stream until the final result (happy path);
• status(ctx, id) — a read-only probe, return {not_started, running, completed, failed} (and the result if it is inline);

Then the framework composes callInternal and reconciler from these primitives, propagates sessionId#callId as the remote idempotency key automatically, and owns the crash-window switch. The author never writes reconciler, and does not need to reason about recovery.

WDYT, is this the right place for that framework layer?

[pltbkd]

Thanks for the feedback! I totally agree that we should provide some built-in abstraction. That's exactly what I mean by the "external subagent support framework", though at that time I wasn't sure whether it should be an abstraction or a set of tools.
I think the primitives you proposed are good starting points, which also echo the thoughts from weiqingy and ofekron in this thread.

While the external subagent framework can be a follow-up to the common framework, I plan to open a separate Draft PR to propose its shape early on. The goal of opening this framework PR is to let the two work in synergy, which will help us better clarify the shapes and the collaboration patterns between the common framework and the external framework.

[pltbkd]

Hi @alnzng @weiqingy @ofekron @joeyutong

I have created a PR for this framework: #938. I think it will help us ground the design discussion in concrete code, and I can land any conclusions we reach directly on the PR.

The recent discussion made me think through the interface design once more. I'd like to share the reasoning behind the current asDurableCallable interface and propose two alternatives for feedback.

Background

A key constraint is that durableExecuteAsync is inherently a combined "execute + yield" operation — it submits the callable for off-mailbox execution and suspends the current action until the callable completes. There is no "submit-only" mode that returns a pending handle without yielding.

For external sub-agents, the actual request (e.g., an HTTP call) must reside inside DurableCallable.call() (and its reconcile path) so that the durable execution mechanism can record, checkpoint, and replay it for recovery. This means issuing the request requires invoking durableExecuteAsync, which in turn requires a yield. The consequence: if submit encompasses the full durable execution flow, the caller always suspends at that point — making it impossible to submit multiple requests before waiting, or to perform additional work between submit and result retrieval.

This led to the current interface: asDurableCallable is an explicit declaration — "I only prepare the callable; I do not execute it." The actual execution (and yield) happens only when the caller passes it to durableExecuteAsync / durableExecuteAllAsync. This preserves access to parallel batch execution while guaranteeing durable execution is used.

The concern

I do find this interface unintuitive. Using DurableCallable as the caller-facing method forces callers to reason about the execution process (prepare → submit → yield) rather than focusing on their own semantics. A submit + Future/awaitable model would better match caller expectations. (This also echoes @weiqingy's point: "the public API could keep an invocation-oriented operation on the sub-agent surface, with DurableCallable staying the mechanism underneath.")

Two alternatives

Option A — submit = durableExecuteAsync (eager, serial)

submit() internally calls durableExecuteAsync(asDurableCallable(...)) and returns an already-completed result (or a Future that is resolved by the time the action resumes). Simple and straightforward, but the caller always yields at each call — only serial execution is possible. If a sub-agent implementation needs fire-and-forget parallelism, it would have to manage the lifecycle and reconciliation entirely on its own, outside the durable execution framework.

Option B — submit records only; get/await triggers execution (lazy, parallel-capable)

submit() only records the invocation parameters and returns an unresolved Future. The actual durableExecuteAsync is triggered on the first get()/await. A utility like Futures.of(f1, f2).get() maps to durableExecuteAllAsync for batch parallel execution.

• Pro: supports parallel execution; guarantees durable execution; natural submit → Future mental model.
• Con: lazy semantics — between submit() and get(), the request has not been sent; isDone() is always false until resolved. This may surprise callers who expect submit to have side effects.

My POC originally implemented Option B. I moved away from it to the current asDurableCallable design because I was concerned the lazy semantics could confuse users. However, given the feedback that the current surface is not ideal, I'm reconsidering.

Question

Do you prefer Option A (simple, serial), Option B (lazy, parallel-capable), or keeping the current asDurableCallable interface?

[alnzng]

@pltbkd small thing to make sure I'm reading the right code: in the comment you mention asDurableCallable, however I didn't see it anywhere in the existing PR: https://github.com/apache/flink-agents/pull/938/changes . Do you mean asAsyncCallable, or is asDurableCallable something from an earlier version that got renamed?

[pltbkd]

Sorry for the confusion. I actually meant asAsyncCallable, which was renamed from asDurableCallable.

Besides, I realize there might be a third option:

• A quick submit as 'pub' (still a durable execution): its result is a recipe/handle rather than the actual result, potentially returning a Future.
• The future.isDone() & future.get() or another query method works as 'sub': internally, they also perform durable execution, and their results are the running status or the actual result/exception.

In this way, the subagent works asynchronously if it supports it, or falls back to either of the previous two models if it cannot (for example, internal subagents). The downside is that a single logical call is split into multiple durable executions, and the number of executions depends on how many times the user calls query.
Would you prefer this approach, or stick to one of the previous options? And if fallback, which one do you think is better?

[alnzng]

Thanks for the detailed write-up and for laying out the options so clearly @pltbkd . Before going into the concrete call API, I'd like to step back and align on the high level scope first.

One thing I noticed: the A/B/C options mainly matter in one kind of design, where the user calls a sub-agent directly in the action code and the method on Subagent is the public API. If instead the framework orchestrates the sub-agents and the user never writes these calls directly, then Subagent and asAsyncCallable are internal, and their naming or shape matters much less. So I feel it might help to confirm the user facing model first.

I can see two possible models, please correct me if I'm misunderstanding:

• Composite orchestration: the user declares "run A then B" or "run A, B, C in parallel and merge", and the framework does the fan out and fan in. Here Subagent is internal plumbing.
• Imperative invocation: inside an action the user writes ctx.getResource("reviewer", AGENT).call(prompt) directly. Here the method on Subagent is the public API, so its shape matters more.

From PR #938 my impression is that the target is the imperative model, but I may be wrong, so it would be great if you could confirm.

If we are in the imperative model, I think the v1 scope may already answer most of A/B/C. For v1 we want to support two scenarios, and both always return the full result:

• Sequential: call sub-agents one by one, each call blocks until it returns. This looks like a blocking call, basically Option A at N=1.
• Parallel: fan out several sub-agents, then fan in the results. This looks like a callAll(...) over durableExecuteAllAsync, where the framework builds the callables and yields once.

In both cases, I don't think we ever need a single call to stay in flight while the action keeps doing other work. Option C (submit, get a handle, then poll) seems mainly useful for that in flight case, so it may be out of v1 scope for now.

So my feeling is that we might not need to choose between A/B/C yet. call plus callAll could cover both scenarios, while asAsyncCallable / DurableCallable stay as the internal mechanism rather than the caller facing API. I think this also aligns with the "unintuitive" concern you raised, since the caller only expresses intent (call or callAll). Please let me know if I'm missing something here.

Would you mind sharing the high level concept first, and maybe pinning it here or in the PR description, mainly which model v1 is committing to? Once that's settled, I think the method level discussion will become much more concrete.

[pltbkd]

The sub-agent design is currently being discussed mainly on the imperative invocation model. This lets the main agent post-process results more easily and simplifies main-action development.

For composite orchestration, my understanding is that the native workflow orchestration already covers it — a sub-agent resource could act as a subgraph there, but expanding it (if internal) and writing it together with the whole workflow is essentially equivalent. The ChatModel-invokes-subagent work we are doing in parallel reuses ToolCallAction: internally it still uses the imperative sub-agent invocation, while the outside is ReAct-loop action orchestration. A similar approach can reduce composite orchestration to imperative invocation + action orchestration. The fan-in/fan-out action orchestration pattern is generic and orthogonal to sub-agents — the existing ParallelChatRequestExample already does something similar.

I've updated the PR code based on the recent discussion. The interface is now in submit + await form; call can be seen as the composed submit + await action, so it does not conflict with the current model. Two new base classes are added: BaseDeferredSubagentSetup and BaseAsyncSubagentSetup. The former is similar to Option B: execution is deferred until triggered at await, which preserves the ability to batch-execute — the batch support is planned to build on #926. The latter is similar to Option C: the caller-facing surface is still submit/isDone/await, while developers get primitives that match the external sub-agent lifecycle: submitRequest, reconcileSubmitRequest, queryStatus, fetchResult. Since submitRequest has side effects while the other primitives are typically idempotent reads, only submitRequest is wired to a reconcile hook. This version also provides an initial implementation of cancel.

The code may still need polishing, but it should be readable and discussable in its current state — please take a look when convenient. Looking forward to your feedback. @alnzng @weiqingy @joeyutong @ofekron

[QvxSphere]

For an external sub-agent protocol, would it be useful for the reconciliation primitive to return the original persisted result artifact plus its canonical request hash and a verifier-discoverable signature/key reference—so recovery can prove ‘this is the same completed invocation’ before any resend?

I’m especially curious whether you would treat that as an optional transport-neutral artifact profile rather than make it part of the core callable API. We have explored this only as a non-production deterministic fixture pattern, not as a Flink integration or production assurance.

[pltbkd]

Thanks for the detailed input. If I got it right, you're suggesting we double-check the result we get back from reconcile, and have the framework (or something it ships) verify that the result really matches the request.

Right now durable execute only checks a completed result against the call identity it persisted itself. Reconcile has nothing like that: the framework can't see the arguments the reconciler captured or the request it built, so it just trusts whatever the reconciler returns.

Sub-agent execution is complex enough that I agree this kind of check is useful in some cases. But I'd rather keep it inside the reconciler than put it in the framework, which also matches how the framework works today. If this turns out to be a common need and we can find a general enough interface, we could add a component later for reconcilers to reuse. And if you have a real production scenario in mind, I'd be happy if you could share.

[QvxSphere]

Thanks—yes, that is the boundary I was trying to isolate, and keeping it inside the reconciler makes sense.

Our current implementation is not a Flink production integration, so I do not want to overstate it as one. The adjacent operational scenario is an external provider that may complete an invocation before the caller receives or persists the response. On recovery, the reconciler must determine whether the returned completed artifact belongs to the original invocation rather than merely trusting a matching call identity.

The pattern we are testing binds a stable invocation ID and canonical request hash to the original persisted result artifact, then makes that binding independently checkable through a signature and discoverable key reference. A mismatch, missing original artifact, or ambiguous reconciliation fails closed before any resend.

I agree this is probably best expressed first as a reusable reconciler component or profile. If useful, I can share a small synthetic example focused specifically on that boundary—without presenting it as a Flink integration or production assurance.