Uptake AI Red Teaming Agent

[haoranpb]

Official Documentation: https://learn.microsoft.com/en-us/azure/foundry/how-to/develop/run-scans-ai-red-teaming-agent

Setup

Setup required:

1. Run .\scripts\Download-BCSymbols.ps1 -Category nl2al -InstanceId nl2al__move-name-customer-card-1
2. Create .env file like below
3. see also for local python setup https://github.com/microsoft/BC-Bench/commits/fix/bcal-pythonhome-under-uv-run/

# .env file

# Foundry Hub project (red teaming)
AZURE_SUBSCRIPTION_ID=<TODO>
AZURE_RESOURCE_GROUP=<TODO>
AZURE_PROJECT_NAME=<TODO>

# BCal agent
AZURE_OPENAI_ENDPOINT=<TODO>
AZURE_OPENAI_DEPLOYMENT=<TODO>

How to run:

# Build-in ones
uv run bcbench redteam scan --language en --risk-category violence

# Custom attack objectives
uv run bcbench redteam scan --language en --seeds C:\depot\BC-Bench\dataset\redteam\attack_objectives.sample.json

[Copilot]

Pull request overview

This PR adds a proof-of-concept AI Red Teaming capability to BC-Bench, wiring Azure AI Evaluation's RedTeam agent against the existing NL2AL (BCal) target. It introduces a new bcbench redteam CLI group (scan and report), an orchestration module that builds a BCal-backed target callback and runs the scan, and a one-shot run_bcal_prompt runner that feeds adversarial prompts to bcal and returns its combined output for a safety judge. Supporting changes add config paths, a sample attack-objectives file, .gitignore entries for private seeds, and the Azure dependencies.

Changes:

• New redteam scan/report CLI commands plus a redteam.py orchestration module (target builder, symbol-cache priming, scan runner, scorecard rendering).
• New run_bcal_prompt in the bcal agent that runs bcal once for a raw prompt and surfaces both generated .al files and stdout/diagnostics.
• Dependency, config, dataset-sample, and .gitignore additions to support the red-team workflow.

Reviewed changes

Copilot reviewed 11 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/bcbench/redteam.py	New scan orchestration: builds the BCal target, primes the symbol cache, runs RedTeam.scan.
src/bcbench/commands/redteam.py	New scan/report CLI commands and terminal scorecard rendering.
src/bcbench/agent/bcal/agent.py	Adds run_bcal_prompt red-team runner; minor comment typo (double space).
src/bcbench/agent/bcal/__init__.py	Exports run_bcal_prompt.
src/bcbench/config.py	Adds redteam_scorecard path.
src/bcbench/commands/__init__.py, src/bcbench/cli.py	Register the new redteam Typer app.
pyproject.toml	Adds azure-ai-evaluation[redteam] and azure-identity.
dataset/redteam/attack_objectives.sample.json	Sample custom attack-objective seed file.
.gitignore	Ignores private red-team seed files.
tests/conftest.py	Adds (currently unused) create_nl2al_result/sample_nl2al_result helpers.
tests/test_nl2al_pipeline.py	Removes an obsolete ty: ignore comment.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 12 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (5)

src/bcbench/agent/bcal/agent.py:184

• A nonzero BCal exit is converted into assistant text and then scored as though it were the model's answer. Because an error/traceback contains no harmful content, failed invocations can be reported as “resisted,” invalidating the attack-success results; execution errors must be surfaced outside the target response.

    except subprocess.CalledProcessError as exc:
        # Surface bcal's own output instead of letting the opaque CalledProcessError propagate (the red-team framework would otherwise report only "Something went wrong Command [...]").
        details = "\n".join(s.strip() for s in (exc.stdout, exc.stderr) if s and s.strip())
        return f"(bcal exited with status {exc.returncode})\n{details}".strip()

src/bcbench/commands/redteam.py:143

• attack_success is optional in the SDK result, but every missing/unevaluated value is rendered as a green “resisted.” Evaluation failures can therefore look like successful defenses; distinguish True, False, and None explicitly.

        result = "[red]\u2717 broke[/]" if row.get("attack_success") else "[green]\u2713 resisted[/]"

src/bcbench/agent/bcal/agent.py:137

• This new subprocess adapter has no automated coverage, although the sibling run_bcal_agent command construction is covered in tests/test_bcal_agent_provider.py. Add tests for argument/env plumbing and for generated-file, stdout, timeout, and nonzero-exit outcomes so scan-result integrity does not depend on untested process behavior.

def run_bcal_prompt(

src/bcbench/commands/redteam.py:45

• The PR's setup example provides only AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_DEPLOYMENT, but this default selects external-command, whose cli_args() requires BCAL_LLM_COMMAND. Following the documented setup therefore fails on the first target call; either default to azure-openai or update the setup/run example with the required backend and variables.

This issue also appears on line 143 of the same file.

    backend: Annotated[BCalLLMBackend, typer.Option(envvar="BCAL_LLM_BACKEND", help="BCal LLM backend used by the bcal target.")] = BCalLLMBackend.EXTERNAL_COMMAND,

src/bcbench/redteam.py:33

• Treating any .app as a complete cache can silently reuse a partial cache or symbols from a different BC version, because this fixed cache path records no version/completion marker. A subsequent scan then skips population and runs BCal against incomplete or stale symbols.

    if package_cache_path.exists() and any(package_cache_path.glob("*.app")):
        return

[Copilot]

Pull request overview

Copilot reviewed 10 out of 12 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

src/bcbench/agent/bcal/agent.py:184

• Do not turn BCal execution failures into target responses. The red-team judge receives these strings as ordinary assistant output, so a crashed or timed-out target can be scored as having “resisted” the attack and produce false-negative security results. Propagate the exception (or otherwise mark the row as an execution error) instead.

    except subprocess.TimeoutExpired as exc:
        return f"(bcal timed out after {_config.timeout.bcal_execution}s)\n{exc.stdout or ''}".strip()
    except subprocess.CalledProcessError as exc:
        # Surface bcal's own output instead of letting the opaque CalledProcessError propagate (the red-team framework would otherwise report only "Something went wrong Command [...]").
        details = "\n".join(s.strip() for s in (exc.stdout, exc.stderr) if s and s.strip())
        return f"(bcal exited with status {exc.returncode})\n{details}".strip()

src/bcbench/agent/bcal/agent.py:176

• The default external-command path inherits PYTHONHOME, PYTHONPATH, and VIRTUAL_ENV from uv run. With the documented separate bridge virtualenv, those variables can make its Python load uv’s incompatible stdlib and BCal fails before answering any prompt. Pass a sanitized environment to this subprocess (and reuse it for the existing BCal invocation) so the documented red-team command can run.

        result = subprocess.run(
            cmd_args,
            timeout=_config.timeout.bcal_execution,
            capture_output=True,
            text=True,
            encoding="utf-8",
            errors="replace",
            check=True,

src/bcbench/agent/bcal/agent.py:141

• Add unit coverage for this new subprocess adapter. Nearby run_bcal_agent behavior is covered in tests/test_bcal_agent_provider.py, but no test invokes run_bcal_prompt, leaving command construction, generated-file/stdout collection, subprocess environment, timeout, and nonzero-exit behavior unchecked; the latter paths directly determine whether scan results are trustworthy.

def run_bcal_prompt(
    entry: NL2ALEntry,
    query: str,
    package_cache_path: Path,
    export_folder: Path,

src/bcbench/commands/redteam.py:50

• RedTeam.scan 1.18.2 treats output_path as a directory (and writes evaluation_results.json inside it), but this option advertises a JSON file and defaults to scorecard.json. Consequently --output result.json creates a directory named result.json, so callers cannot consume the file at the path the CLI promises. Model this option/default as an output directory, or translate the requested file path to the SDK directory and expose the actual inner file.

    output: Annotated[Path, typer.Option(help="Where to write the upstream scorecard JSON.")] = _config.paths.redteam_scorecard,