Skip to content

[Experiment] Code Review with BCQuality plugin - #759

Draft
haoranpb wants to merge 2 commits into
mainfrom
experiment/bcquality
Draft

[Experiment] Code Review with BCQuality plugin#759
haoranpb wants to merge 2 commits into
mainfrom
experiment/bcquality

Conversation

@haoranpb

Copy link
Copy Markdown
Collaborator

Try experimenting with BCQuality plugin, for code review skill

Copilot AI review requested due to automatic review settings July 29, 2026 05:58
@haoranpb haoranpb added the Sample Sample PRs for demonstration label Jul 29, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Enables BCQuality for AL code-review experiments.

Changes:

  • Replaces /review with the BCQuality review skill.
  • Enables the pinned BCQuality plugin.
Comments suppressed due to low confidence (2)

src/bcbench/agent/shared/config.yaml:116

  • This toggle is global, not code-review-specific: both agent runners call resolve_config_plugins(...) unconditionally for every category, and the evaluation workflows default to bug-fix. As a result, bug-fix and test-generation runs will also clone/load BCQuality and be recorded as plugin experiments, contrary to this PR's code-review-only experiment scope. Gate this plugin by EvaluationCategory.CODE_REVIEW or supply a category-specific experiment configuration.
    enabled: true

src/bcbench/agent/shared/config.yaml:54

  • Invoking this skill introduces an incompatible output contract. The pinned BCQuality skill mandates a findings-report object whose entries use message plus nested location.file/location.line, while parse_review_output() only recognizes top-level file/line_start and body fields. If the skill follows its own contract, every finding is silently skipped and the evaluation records an empty review. Add an explicit BCQuality-to-benchmark adapter (or parser support) rather than relying on the model to violate the skill contract and rewrite the shape.
    /bcquality-al-review

path: "."
- name: "BCQuality"
enabled: false
enabled: true

code-review-template: |
/review
/bcquality-al-review
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Sample Sample PRs for demonstration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants