Bedrock Summarize Buildkite Plugin

AI-powered build analysis and error diagnosis using Large Language Models (LLMs) provided by Amazon Bedrock. You can use any LLM offered by Amazon Bedrock, provided it is enabled on your AWS account. This plugin automatically analyzes build failures, provides root cause analysis, and suggests actionable fixes through Buildkite annotations.

Features

• 🤖 Intelligent Build Analysis: AI analyzes build logs to identify root causes of failures
• 📋 Buildkite Annotations: Creates rich annotations with analysis results and suggested fixes
• ⚡ Smart Triggering: Configurable triggers (on-failure, always, manual)
• 🔧 Actionable Insights: Provides specific steps to resolve issues and prevent future failures
• 🎯 Context-Aware: Understands build context including branch, commit, and job information

Requirements

• curl: For API requests
• jq: For JSON processing
• AWS CLI: Must be installed on agents and set up with access to AWS Bedrock (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables set for your account)
• Bedrock: Your AWS account needs access to Amazon Bedrock, and your desired model(s) must be enabled

Quick Start

Add the plugin to your pipeline like this:

steps:
  - label: "🧪 Run tests"
    command: "npm test"
    plugins:
      - bedrock-summarize#v1.0.0: ~

If your Buildkite agents are running in AWS, you could consider using the OIDC Assume Role plugin in conjunction with Bedrock Summarize. After creating an IAM role in AWS that has permission to use Bedrock, a configuration like this will allow your agent to assume that role when it uses the plugin:

steps:
  - label: "🧪 Run tests"
    command: "npm test"
    plugins:
      - bedrock-summarize#v1.0.0: ~
      - aws-assume-role-with-web-identity#v1.6.0:
          role-arn: arn:aws:iam::12345:role/bedrock-access

Configuration Options

Optional

buildkite_api_token (string)

Buildkite API token for fetching job logs directly from the Buildkite API. This improves analysis by providing the exact failing job logs. If not specified, the plugin will look for BUILDKITE_API_TOKEN in the environment.

model (string)

Bedrock model to use for analysis. Default: anthropic.claude-3-7-sonnet-20250219-v1:0

inference_profile (string)

Bedrock inference profile to use for analysis. Default: us.anthropic.claude-3-7-sonnet-20250219-v1:0

trigger (string)

When to trigger AI analysis. Options: on-failure, always, manual. Default: on-failure

• on-failure: Only analyze when the build step fails
• always: Analyze every build (success or failure)
• manual: Only when BEDROCK_ANALYZE=true environment variable is set or commit message contains [bedrock-analyze]

analysis_level (string)

Level at which to analyze logs. Options: step, build. These require buildkite_api_token to be set in order to fetch job logs, else we default to available environment variables. Default: step

• step: Analyze only the current step's logs
• build: Analyze logs from all jobs in the entire build

max_log_lines (integer)

Maximum number of log lines to send to the LLM for analysis. Default: 1000

custom_prompt (string)

Additional context or instructions to include in the analysis prompt.

timeout (integer)

Timeout in seconds for Bedrock API requests. Default: 60

annotate (boolean)

Whether to create Buildkite annotations with the analysis results. Default: true

annotation_scope (string)

Where the annotation is attached, matching the --scope option of buildkite-agent annotate. Options: build, job. Default: build

• build: Every job running the plugin shares one annotation at the top of the build, so the last job to finish replaces the analysis from the jobs before it
• job: Each job gets its own annotation, shown against that job, so a build running the plugin on several steps keeps an analysis for each of them

The annotation context follows the scope, keyed on the build id at build scope and the job id at job scope, so the analysis from one run replaces only the analysis it supersedes.

Job scope requires Buildkite agent v3.112.0 or newer. The plugin sets the scope through the BUILDKITE_ANNOTATION_SCOPE environment variable rather than the --scope flag, so older agents keep working and fall back to build scope instead of failing.

allow_multiple_annotations (boolean)

Create a new annotation on each run instead of replacing the previous one. Default: false

Annotations replace each other when they share a context. Setting this to true adds a random suffix to the context, so every run gets an annotation of its own and nothing is overwritten. Useful when a job runs the plugin more than once, or when you want to keep the analysis from each retry.

agent_file (boolean or string)

Include project context from an agent file in the analysis. Default: false

• true: Include AGENT.md from the repository root
• false: Don't include any agent context
• "path/to/file.md": Include the specified file

The agent file should contain project-specific context like architecture details, common issues, coding standards, or troubleshooting guides that help LLMs provide more relevant analysis.

compare_builds (boolean)

Enable build time comparison analysis. When enabled, the LLM will analyze build time trends by comparing the current build duration against recent builds. Default: false

comparison_range (integer)

Number of previous builds to compare against for build time analysis. Only used when compare_builds is true. Default: 5

Examples

Basic Usage - Analyze Failed Tests

steps:
  - label: "🧪 Run tests"
    command: "npm test"
    plugins:
      - bedrock-summarize#v1.0.0: ~

When tests fail, the LLM will analyze the output and create an annotation with:

• Root cause analysis
• Key error explanations
• Suggested fixes
• Prevention strategies

Build-Level Analysis

steps:
  - label: "🔍 Analyze entire build"
    command: "npm test"
    plugins:
      - bedrock-summarize#v1.0.0:
          buildkite_api_token: "$$BUILDKITE_API_TOKEN"
          analysis_level: "build"
          trigger: "always"

With analysis_level: "build", the LLM will analyze logs from all jobs in the build, providing insights across the entire pipeline.

An Annotation per Step

steps:
  - label: "🧪 Run tests"
    command: "npm test"
    plugins:
      - bedrock-summarize#v1.0.0:
          annotation_scope: "job"

  - label: "🏗️ Build application"
    command: "npm run build"
    plugins:
      - bedrock-summarize#v1.0.0:
          annotation_scope: "job"

Both steps keep their own analysis instead of the second one replacing the first. Use allow_multiple_annotations: true as well if a single step runs the plugin more than once and you want to keep every result.

Always Analyze Builds

steps:
  - label: "🏗️ Build application"
    command: "npm run build"
    plugins:
      - bedrock-summarize#v1.0.0:
          trigger: "always"
          custom_prompt: "Focus on build performance and optimization opportunities"

Manual Analysis with Custom Context

steps:
  - label: "🚀 Deploy to staging"
    command: "./deploy.sh staging"
    env:
      BEDROCK_ANALYZE: "true"  # Trigger manual analysis
    plugins:
      - bedrock-summarize#v1.0.0:
          trigger: "manual"
          custom_prompt: "This is a deployment script. Focus on infrastructure and configuration issues."
          max_log_lines: 2000

Build Time Analysis

steps:
  - label: "🏗️ Build with performance tracking"
    command: "npm run build"
    plugins:
      - bedrock-summarize#v1.0.0:
          compare_builds: true
          comparison_range: 10
          custom_prompt: "Focus on build performance trends and identify any performance regressions"

When compare_builds is enabled, the LLM will:

• Compare current build time against the last N builds (configurable via comparison_range)
• Identify performance trends and anomalies
• Suggest optimizations for slow builds
• Highlight significant performance changes

Multiple Steps with Different Configurations

steps:
  - label: "🔍 Lint code"
    command: "npm run lint"
    plugins:
      - bedrock-summarize#v1.0.0:
          custom_prompt: "Focus on code quality and style issues"

  - label: "🧪 Run tests"
    command: "npm test"
    plugins:
      - bedrock-summarize#v1.0.0:
          custom_prompt: "Focus on test failures and coverage issues"

  - label: "🏗️ Build production"
    command: "npm run build:prod"
    plugins:
      - bedrock-summarize#v1.0.0:
          trigger: "always"
          custom_prompt: "Focus on build optimization and bundle analysis"

Compatibility

Elastic Stack	Agent Stack K8s	Hosted (Mac)	Hosted (Linux)	Notes
✅	✅	✅	✅

• ✅ Fully compatible assuming requirements are met

⚒ Developing

Run tests with

docker compose run --rm tests

👩‍💻 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Ensure all tests pass
6. Submit a pull request

Please follow the existing code style and include tests for any new features.

📜 License

The package is available as open source under the terms of the MIT License.