Extensibility requests triage + implementation by AleksandricMarko · Pull Request #762 · microsoft/BC-Bench

AleksandricMarko · 2026-07-29T20:19:49Z

No description provided.

Copilot

Pull request overview

Adds benchmark support for implementing and triaging Business Central extensibility requests.

Changes:

Adds ext-implement and ext-triage datasets, pipelines, results, and category wiring.
Adds extensibility agent skills, triage phases, rules, and prompts.
Updates tests, documentation, workflows, and agent configuration.

Reviewed changes

Copilot reviewed 60 out of 60 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
`tests/test_type_exhaustiveness.py`	Covers new entry types
`tests/test_agent_skills.py`	Uses explicit UTF-8
`tests/conftest.py`	Adds extensibility fixtures
`src/bcbench/types.py`	Registers new categories
`src/bcbench/results/extriage.py`	Adds triage results
`src/bcbench/results/__init__.py`	Exports triage result
`src/bcbench/evaluate/extriage.py`	Implements triage pipeline
`src/bcbench/evaluate/extimplement.py`	Implements request pipeline
`src/bcbench/evaluate/__init__.py`	Exports new pipelines
`src/bcbench/dataset/dataset_entry.py`	Adds entry schemas
`src/bcbench/dataset/__init__.py`	Exports entry schemas
`src/bcbench/commands/evaluate.py`	Adds mock scenarios
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/templates/result-contract.md`	Defines triage output
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/SKILL.md`	Defines triage skill
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/shared-rules.md`	Defines shared rules
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/7-finalize.md`	Defines finalization
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/6-team-assignment.md`	Defines team assignment
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/5-codebase-analysis.md`	Defines code analysis
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/4-requirements.md`	Defines requirements checks
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/3-classify.md`	Defines classification
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/2b-bcapps-check.md`	Defines BCApps check
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/2-eligibility.md`	Defines eligibility
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/1-preprocess.md`	Defines preprocessing
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/0-startup.md`	Defines startup checks
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/orchestrator.md`	Orchestrates triage phases
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/team-configuration/team_namespace_mapping.yaml`	Maps namespaces to teams
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/request_for_external_requirements.yaml`	Adds external requirements
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/general_requirements.yaml`	Adds general requirements
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/extensibility_enhancement_requirements.yaml`	Adds enhancement requirements
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml`	Adds event requirements
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/enum_request_requirements.yaml`	Adds enum requirements
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/comment-templates/comment_templates.yaml`	Adds response templates
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/request_for_external_implementation.yaml`	Adds accessibility rules
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/general_blockers.yaml`	Adds general blockers
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_warnings.yaml`	Adds event warnings
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_ishandled_warnings.yaml`	Adds IsHandled warnings
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_ishandled_implementation.yaml`	Adds IsHandled guidance
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_ishandled_blockers.yaml`	Adds IsHandled blockers
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_ishandled_alternative_suggestions.yaml`	Adds IsHandled alternatives
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_implementation.yaml`	Adds event guidance
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_alternative_suggestions.yaml`	Adds event alternatives
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/compatibility.md`	Maps host capabilities
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/ai-ext-fix/SKILL.md`	Adds implementation skill
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/ai-ext-fix/guidelines.md`	Adds AL event guidelines
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/agents/argus-triage.agent.md`	Adds triage agent
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/agents/argus-codebase-analysis.agent.md`	Adds analysis sub-agent
`src/bcbench/agent/shared/instructions/microsoftInternal-NAV/agents/ai-ext-fix.agent.md`	Adds implementation agent
`src/bcbench/agent/shared/config.yaml`	Adds category prompts
`src/bcbench/agent/copilot/agent.py`	Expands logging
`dataset/problemstatement/microsoftInternal__NAV-Ext_Impl-30377/README.md`	Adds report request
`dataset/problemstatement/microsoftInternal__NAV-Ext_Impl-30362/README.md`	Adds Service-Post request
`dataset/problemstatement/microsoftInternal__NAV-Ext_Impl-30361/README.md`	Adds archive request
`dataset/problemstatement/microsoftInternal__NAV-Ext_Impl-30346/README.md`	Adds accessibility request
`dataset/problemstatement/microsoftInternal__NAV-Ext_Impl-30336/README.md`	Adds routing-line request
`dataset/problemstatement/microsoftInternal__NAV-Ext_Impl-30223/README.md`	Adds Purchase Line request
`dataset/extriage.jsonl`	Adds triage benchmark entries
`CATEGORIES.md`	Documents new categories
`.github/workflows/copilot-evaluation.yml`	Exposes implementation category
`.github/workflows/claude-evaluation.yml`	Exposes implementation category

Comments suppressed due to low confidence (3)

src/bcbench/evaluate/extriage.py:145

These fields come from unconstrained agent JSON, but the code assumes labels_to_set is list[str] and coerces arbitrary comment/state values with str(). For example, labels_to_set: 1 raises TypeError in _normalize_labels and aborts the entry. Validate the full Final_Output contract before grading and save a failed result for invalid field types.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:103
flexible_validation and validation_notes are over-indented relative to the requirement item, making this YAML invalid rather than attaching the relaxed policy to data_sensitivity_review.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:118
flexible_validation and validation_notes are over-indented relative to the requirement item, making this YAML invalid rather than attaching the relaxed policy to multi_extension_interaction.

Copilot

Pull request overview

Copilot reviewed 60 out of 60 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (8)

src/bcbench/agent/shared/config.yaml:84

This prompt requires argus-triage, but the same agent config has skills.enabled: false, so setup_agent_skills() never installs that skill into the target checkout. Both Copilot and Claude will therefore be asked to run a skill that is unavailable. Enable/install the skill for this category, or make the prompt self-contained.

    Task: Run the `argus-triage` skill end to end on the extensibility request below and produce its
    triage decision. Follow the skill's phases (preprocess, eligibility, classify, requirements,

src/bcbench/agent/shared/config.yaml:70

The NAV checkout and every new problem statement use App/Layers/..., not src/Layers/.... This instruction prevents agents from recognizing W1 files and propagating changes to localization counterparts.

    - W1-first layering: when the file to change exists under `src/Layers/W1/...`, change W1 first, then apply
      the same change to every same-named counterpart file in the other country/region layers. If the file
      exists only in a specific country layer, edit that layer directly.

src/bcbench/dataset/dataset_entry.py:224

The offline task omits metadata required by the mandated eligibility phases: current issue state/type, author, updatedAt, and dated comment authorship. In particular, entries already labeled missing-info cannot evaluate the “bot last commenter” or 30-day rules, so their expected open/closed decision is not derivable from the prompt. Model these fields and render them here, or remove those checks in offline mode.
src/bcbench/evaluate/extriage.py:108
Malformed judge output currently escapes as JSONDecodeError/AttributeError and fails the whole evaluation, while string values such as "false" are truthy and can incorrectly pass. Convert unreadable or schema-invalid verdicts to LLMJudgeError and require an actual boolean.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/5-codebase-analysis.md:36
Not every rule stored in a blockers file means auto-reject. For example, potential_security_risk has action: include_human, entity_not_found has action: request_clarification, and ishandled_unsafe_code_block has action: suggest_alternative. Mapping every match to auto-reject closes requests that should instead be routed for human review or clarification.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/enum_request_requirements.yaml:30
Phase 3 emits subtype extend_existing_enum, but this requirements file names the section extend_enum. The requirements for target enum and compatibility therefore cannot be selected for requests that extend an existing enum.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:88
These keys are over-indented beneath the scalar validation_hints, making this YAML invalid; the same defect is repeated for data_sensitivity_review and multi_extension_interaction below. Any YAML-aware reader will fail to load the event requirements, blocking Phase 4 for event requests.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/4-requirements.md:18
The requirements data includes action: auto_reject (for example reject_ishandled_only_for_locktable), but this dispatcher recognizes only reject. Such a matched rule falls through and becomes missing-info rather than closing the rejected request.

haoranpb

Great work, it is close. Left some comments about styling => let's spell the name out.

Regarding the evaluation method, take a look at how nl2al is using LMChecklist, sounds like a good fit for your senarios

…ensibility/implementation # Conflicts: # src/bcbench/dataset/__init__.py # src/bcbench/dataset/dataset_entry.py

Copilot

Pull request overview

Copilot reviewed 60 out of 60 changed files in this pull request and generated 2 comments.

Suppressed comments (8)

src/bcbench/agent/shared/config.yaml:86

The triage prompt requires argus-triage, but this configuration still has skills.enabled: false, and setup_agent_skills() only copies profile skills when that flag is true. The evaluated NAV checkout therefore has no such skill or knowledge files, so the agent cannot execute the workflow this prompt requires. Enable/install the skill for these categories (ideally category-selectively) before advertising it in the prompt.

    Task: Run the `argus-triage` skill end to end on the extensibility request below and produce its
    triage decision. Follow the skill's phases (preprocess, eligibility, classify, requirements,
    codebase analysis, team assignment, finalize) and its knowledge base to decide the managed labels,
    the advisory comment, and whether the request should stay open or be closed.

src/bcbench/agent/shared/config.yaml:70

The NAV dataset and every new problem statement use App/Layers/W1/..., not src/Layers/W1/.... With this path, an agent following the benchmark prompt will miss W1 and skip required country-layer propagation. Use App/Layers/W1/... here.

    - W1-first layering: when the file to change exists under `src/Layers/W1/...`, change W1 first, then apply
      the same change to every same-named counterpart file in the other country/region layers. If the file
      exists only in a specific country layer, edit that layer directly.

src/bcbench/evaluate/extriage.py:108

A malformed judge artifact currently escapes as JSONDecodeError (or AttributeError when the JSON is not an object), while evaluate() only catches LLMJudgeError. Since this file is LLM-generated, one invalid response crashes the entire evaluation instead of recording comment_ok=False. Validate and wrap parse/type failures as LLMJudgeError.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/5-codebase-analysis.md:36
The Phase 5 algorithm maps every rule in the blockers batch to auto-reject, but that batch contains other required outcomes: ishandled_unsafe_code_block uses suggest_alternative, potential_security_risk uses include_human, and entity_not_found uses request_clarification. Those cases will therefore be closed as rejections instead of becoming missing-info/manual-review outcomes. Dispatch on each matched rule's action rather than its file category.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/enum_request_requirements.yaml:32
Phase 3 emits the subtype extend_existing_enum, and the orchestrator uses that same name when loading subtype rules, but this requirements file defines extend_enum. Existing-enum requests therefore do not match their subtype-specific requirements. Rename this key to the canonical subtype.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:184
Phase 4 only recognizes requirement actions reject and agent-not-processable; auto_reject is never handled there. As a result, a request solely trying to bypass LockTable can miss the required rejection path. Use the Phase 4 action vocabulary (reject) for this requirement.
src/bcbench/evaluate/extriage.py:126
This new evaluator has no focused tests, although the analogous judge-based pipeline is covered in tests/test_nl2al_pipeline.py. Add tests for missing/unparsable output, wrapped versus flat Final_Output, invalid field types, judge failure, and pass/fail aggregation so contract regressions do not crash evaluation jobs.
src/bcbench/evaluate/extimplement.py:57
The new judge-based pipeline is untested, while NL2ALPipeline.evaluate() has dedicated empty/non-empty-diff coverage in tests/test_nl2al_pipeline.py. Add equivalent tests for ExtImplementPipeline.evaluate() and its workspace setup, especially that only AL edits are captured and empty output persists the expected result.

…ding Applies Haoran's review feedback for the extensibility-request categories: - Rename to explicit `extensibility-request-implement` / `extensibility-request-triage` across category ids, enum members, classes, dataset files and module files (#1/#2/#4). - Slim both prompt templates to reflect real-world usage; category-specific rules now live in the ai-ext-fix / argus-triage skills (#3). - Merge duplicate `bcbench.results` import in commands/evaluate.py (#5). - Uptake #761: both ext entries subclass `RepoGroundedEntry` (#6/#7). - Triage `patch` defaults to None; stripped from the dataset (#8). - Constrain triage label fields with a `ManagedLabel` Literal to catch dataset typos (#9). - Move the ext dataset classes into dataset/extensibility_request.py, like code review (#10). - Drop `fetch_commit_if_missing` from both ext pipelines to match the other pipelines (#11). - Keep ext-implement judge-only for now; container build/publish noted as future work (#12). - Grade triage with the NL2AL LMChecklist instead of the code-review-style hybrid: `ExtRequestTriageResult` removed in favour of `JudgeBasedEvaluationResult`, expected_* fields replaced by an `expected` checklist, evaluators `lm_checklist`, core score `test_passed`; 17 triage entries migrated (#13/#14/#15/#16). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fb712366-b026-470c-9d09-1bd9e9f52a31

Copilot

Pull request overview

Copilot reviewed 59 out of 59 changed files in this pull request and generated 1 comment.

Suppressed comments (13)

src/bcbench/agent/shared/config.yaml:75

The triage prompt depends on “the skill's Final_Output object,” but the default config still has both skills.enabled: false and agents.enabled: false, and the runners read that config directly. Consequently, normal workflow runs never copy argus-triage or select its agent, so the model is asked to follow a JSON contract and triage procedure that are unavailable. Make skill/agent selection category-aware, or make this prompt self-contained.

    - Write your final decision as valid JSON to `triage_result.json` in {{repo_path}}, containing exactly the skill's `Final_Output` object. Do not apply it through any host tool.

src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/general_requirements.yaml:112

This key is indented deeper than its sibling rationale; the subsequent dedent makes the newly added knowledge file invalid YAML, so a YAML loader cannot consume the general requirements. Align it with the other quality_standards keys.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:88
These mapping keys are over-indented beneath the inline validation_hints value, which is invalid YAML. Align them with the other fields of this requirement so the IsHandled requirements file can be loaded.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:103
These mapping keys are over-indented beneath the inline validation_hints value, making this YAML invalid. Align them with the other fields of the requirement.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:118
These mapping keys are over-indented beneath the inline validation_hints value, making this YAML invalid. Align them with the other fields of the requirement.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/enum_request_requirements.yaml:30
The classifier and orchestrator call this subtype extend_existing_enum, but the requirements file names it extend_enum. A subtype lookup based on the emitted SUBTYPE therefore misses these mandatory requirements. Use the same identifier throughout the workflow.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/SKILL.md:31
These hard-coded repository assumptions do not match this category's checkout. Entries target microsoftInternal/NAV and their AL projects are under App/Layers/..., whereas the skill checks microsoft/BCAppsTest and ./src; the startup source check will therefore fail before triage. Derive repository and code roots from the evaluation context/project paths.

Everything is local in **`microsoft/BCAppsTest`**:

- The **issue** is in this repo → operated on with `gh`.
- The **AL source** is checked out at **`CODE_ROOT = ./src`** → Phase 5 reads local files
  (no remote GitHub API, no codebase token).

src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/codebase-rules/event_request_ishandled_alternative_suggestions.yaml:6

Every other subtype rule identifies this target as event-request/ishandled; this hyphenated value does not match the classifier's type/subtype contract, so action filtering can skip the only rule in this file. Use the established subtype identifier.

      applies_to: ["event-request-ishandled"]

src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/phases/5-codebase-analysis.md:33

This blanket mapping ignores each rule's declared action. For example, potential_security_risk uses include_human and explicitly requires agent-not-processable, while entity_not_found uses request_clarification; both are in the blocker batch and will be converted to auto-reject/closed here. Dispatch matched blockers according to their action instead of closing every match.
src/bcbench/dataset/extensibility_request.py:48
The triage skill classifies and emits extensibility-enhancement as a managed request-type label, but this public input type rejects that label in current_labels. A previously triaged enhancement entry therefore cannot be loaded. Keep the schema aligned with the skill's managed-label set.
src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/knowledge/input-requirements/event_request_requirements.yaml:184
Phase 4 only recognizes action: reject and action: agent-not-processable; auto_reject is not handled there. A request whose IsHandled block only skips LockTable will therefore fall through to the generic missing-info path instead of being closed as this rule requires. Use the action value defined by the requirements workflow.
src/bcbench/evaluate/ext_request_implement.py:59
The new pipeline's raw-diff and empty-diff result paths are untested, while analogous judge-based pipeline behavior is covered in tests/test_nl2al_pipeline.py. Add tests that mock stage_and_get_diff for both outcomes and verify the saved JudgeBasedEvaluationResult; otherwise regressions in this category can pass the exhaustiveness test unnoticed.
src/bcbench/evaluate/ext_request_triage.py:47
No test exercises the triage artifact contract introduced here. Add coverage for a missing file, a whitespace-only file, and a non-empty UTF-8 JSON file, asserting the empty/raw result saved in each case; existing judge-based pipelines have dedicated tests, so the current exhaustiveness fixture does not cover this behavior.

Rename the copied argus-triage skill/agents to extensibility-request-triage (and ai-ext-fix -> extensibility-request-implement) and rework the triage skill to run offline for evals: request comes from prompt text, source under caller-supplied CODE_ROOT, decision written to triage_result.json instead of applied via gh. Drop Argus branding, ISSUE_NUMBER, BCAppsTest and ./src hard-coding, and author/timestamp eligibility checks. Fix knowledge/rule bugs found in review: invalid YAML indentation in general_requirements.yaml and event_request_requirements.yaml; align enum subtype extend_enum -> extend_existing_enum; correct IsHandled applicability id to event-request/ishandled; add missing approved_extensibility_enhancement comment template; dispatch Phase 5 blockers by each rule's own action instead of blanket auto-reject. Update types.py comment and ext_request_triage.py docstring to drop stale names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: fb712366-b026-470c-9d09-1bd9e9f52a31

Copilot

Pull request overview

Copilot reviewed 59 out of 59 changed files in this pull request and generated 2 comments.

Suppressed comments (3)

src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/extensibility-request-triage/phases/4-requirements.md:14

The requirement files embed subtype requirements under subtypes (for example, event_request_requirements.yaml contains both ishandled and regular), and no separate subtype files exist. Telling the agent to evaluate every requirement in the type file makes regular events fail IsHandled-only checks and makes one enum subtype fail the other subtype's checks. Select only common requirements plus the active subtype.
src/bcbench/dataset/extensibility_request.py:51
extensibility-enhancement is a managed request-type label in shared-rules.md:10-14 and is emitted by the classifier, but it is absent from this literal. Any future triage entry already carrying that managed label will fail Pydantic validation before evaluation starts.
tests/test_type_exhaustiveness.py:45
This signature exceeds the repository's 200-character Ruff line length and will be reformatted by the required ruff-format hook. Apply the formatter (or split the parameters) so the formatting check does not leave this file modified.

+  "Final_Output": {
+    "labels_to_set": ["Finance", "event-request"],
+    "comment_to_post": "Hi, ...",
+    "request_state": "open",


+    Important constraints:
+    - This runs offline: read the request from the text below, do NOT call `gh` or touch any live issue.
+    - Do NOT build or run tests; base codebase analysis on the local `.al` files under {{repo_path}}.
+    - Write your final decision as valid JSON to `triage_result.json` in {{repo_path}}, containing exactly the skill's `Final_Output` object. Do not apply it through any host tool.


haoranpb

Great progress, you should be able to trigger some runs and how/if things works.

Start with test run (with few entries) and cheaper models.

With LMChecklist, you should be able to view results in Kuisto, link shared in private chat.

haoranpb · 2026-07-31T12:30:47Z

+        return {"assertions": self.expected}
+
+
+ManagedLabel = Literal[


[nit] For consistancy with the rest of codebase, more explicit typing

Suggested change

ManagedLabel = Literal[

type ManagedLabel = Literal[

haoranpb · 2026-07-31T12:34:13Z

+__all__ = ["ExtRequestImplementPipeline"]
+
+
+def _copy_problem_statement(entry: ExtRequestImplementEntry, repo_path: Path) -> None:


Can you take a look at copy_problem_statement_folder in instruction_operations.py?

Probably need to reflex the type of entry to RepoGroundedEntry, but then you could re-use that function instead of creating/maintaining your own

haoranpb · 2026-07-31T12:35:56Z

+        if not raw:
+            result = JudgeBasedEvaluationResult.create_empty_output(context)
+            logger.warning(f"Agent produced no {TRIAGE_RESULT_FILE} for {context.entry.instance_id}")


Consider just let it fail if not raw, meaning we probaly want to investigate what is wrong, why no file is generated

haoranpb · 2026-07-31T12:37:00Z

            case EvaluationCategory.BUG_FIX | EvaluationCategory.TEST_GENERATION:
                return "GitHub-BCBench"
-            case EvaluationCategory.CODE_REVIEW:
+            case EvaluationCategory.CODE_REVIEW | EvaluationCategory.EXT_REQUEST_IMPLEMENT | EvaluationCategory.EXT_REQUEST_TRIAGE:


Can't remember if I mentioned this, but do remember to check if this matches your production workflow

AleksandricMarko added 2 commits July 29, 2026 16:32

ext implementation

ac3c1ff

ext triage

976a94b

Copilot AI review requested due to automatic review settings July 29, 2026 20:19

Copilot started reviewing on behalf of AleksandricMarko July 29, 2026 20:20 View session

Copilot AI reviewed Jul 29, 2026

View reviewed changes

missing evaluation categories

c59108e

Copilot AI review requested due to automatic review settings July 29, 2026 20:27

Copilot started reviewing on behalf of AleksandricMarko July 29, 2026 20:28 View session

Copilot AI reviewed Jul 29, 2026

View reviewed changes

Comment thread ...bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/compatibility.md Outdated

Comment thread src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/orchestrator.md Outdated

haoranpb reviewed Jul 30, 2026

View reviewed changes

Merge branch 'main' of https://github.com/microsoft/BC-Bench into ext…

1b32570

…ensibility/implementation # Conflicts: # src/bcbench/dataset/__init__.py # src/bcbench/dataset/dataset_entry.py

Copilot AI review requested due to automatic review settings July 31, 2026 08:24

Copilot started reviewing on behalf of AleksandricMarko July 31, 2026 08:25 View session

Copilot AI reviewed Jul 31, 2026

View reviewed changes

Comment thread src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/SKILL.md Outdated

Comment thread src/bcbench/evaluate/extriage.py Outdated

Copilot AI review requested due to automatic review settings July 31, 2026 09:59

Copilot started reviewing on behalf of AleksandricMarko July 31, 2026 10:00 View session

Copilot AI reviewed Jul 31, 2026

View reviewed changes

Comment thread src/bcbench/agent/shared/instructions/microsoftInternal-NAV/skills/argus-triage/SKILL.md Outdated

Copilot AI review requested due to automatic review settings July 31, 2026 10:42

Copilot started reviewing on behalf of AleksandricMarko July 31, 2026 10:43 View session

Copilot AI reviewed Jul 31, 2026

View reviewed changes

haoranpb reviewed Jul 31, 2026

View reviewed changes

		__all__ = ["ExtRequestImplementPipeline"]


		def _copy_problem_statement(entry: ExtRequestImplementEntry, repo_path: Path) -> None:

Uh oh!

Conversation

AleksandricMarko commented Jul 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

haoranpb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

haoranpb left a comment

Choose a reason for hiding this comment

Uh oh!

haoranpb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

haoranpb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

haoranpb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

haoranpb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants