STAC-25457 Publish and sign the agent and cluster-agent images from GitHub Actions

[LouisParkin]

Ports the publishing half of pre_release_main_agent_image / pre_release_cluster_agent_image and both merge_docker_manifest_* jobs from GitLab. #446 landed the build-and-verify halves; this adds the push.

Stacked on STAC-25142-agent-lint-unit (#444), like #446 and #448-#452.

What lands

New job	Workflow	Replaces
publish-agent-image (amd64, arm64)	build-deb.yml	pre_release_main_agent_image (push half)
merge-agent-manifest	build-deb.yml	merge_docker_manifest_main_agent
publish-cluster-agent-image (amd64, arm64)	build-binaries.yml	pre_release_cluster_agent_image (push half)
merge-cluster-agent-manifest	build-binaries.yml	merge_docker_manifest_cluster_agent

Each publish job needs: the existing image-build job; each merge job assembles the multi-arch tag from the two single-arch ones. All four are added to their workflow's cerberus-notify needs list, so a failed publish on the release branch still reaches Slack.

Gating

if: github.event_name == 'push'. The workflow's push: filter only carries the release branch, so the event check is the whole gate — and a workflow_dispatch on an arbitrary branch cannot publish. Same reasoning the existing cerberus-notify gate uses. PRs build and smoke-test images but never push, which is the point on a public repo.

image-pipeline instead of publish_image.sh

Uses the StackVista/image-pipeline composite actions, matching stackstate-process-agent and the STAC-24837 direction for product repos. Over publish_image.sh that buys:

• cosign signatures in both v2 and v3 bundle formats, with post-sign verification on the merged manifest
• SBOM + max-mode provenance attestations
• canonical SUSE Observability OCI labels
• an entrypoint ELF-architecture check that fails arch-mismatched images before signing rather than with exec format error on a node
• refusal to overwrite an existing tag

Neither Dockerfile declares ARG BASE_IMAGE, so base-name is passed explicitly — both final stages are registry.suse.com/bci/bci-micro. The agent build also needs --build-arg ARCH; the cluster-agent takes none.

Dropped: the <branch>-<arch> tag

publish_image.sh pushed both <branch>-<arch> and <short-sha>-<arch>. Only the short SHA is kept, because nothing consumes the branch tag:

• helm-charts-internal pins both images to an 8-character SHA (currently 9516cb41 — stackstate-7.78.2 HEAD)
• beest receives the tag as AGENT_HASH_UNDER_TEST, which GitLab set from CI_COMMIT_SHORT_SHA
• an org-wide code search for stackstate-k8s-agent:master, :stackstate-7*, :$CI_COMMIT_REF_SLUG and the cluster-agent equivalents returns 0 hits, run against a positive control so an empty result is not a false negative

Tag is cut -c1-8 of the commit SHA, matching GitLab's CI_COMMIT_SHORT_SHA and the chart's pin width.

⚠️ Blocked on STAC-25541

These jobs will fail until StackVista/terraform-infra#75 applies. The stackstate+agent robot behind QUAY_USER has no write grant on stackstate-k8s-agent or stackstate-k8s-cluster-agent — neither repo appears in any team block in quay/locals.tf. GitLab publishes them with the separate legacy gitlabci robot, whose access predates Terraform, which is why this gap is invisible until the GitHub lane runs. It is the same failure that blocked process-agent in STAC-25510.

Still out of scope

pre_release_deb (S3 install.sh publish) and sign_deb (GPG key material) remain unported — both are release-lane concerns needing credentials that do not exist on this repo yet.

Validation

• actionlint: no new findings (only the pre-existing self-hosted runner-label noise, which is unavoidable without an actionlint.yaml)
• zizmor: No findings to report
• YAML parses; every needs: reference resolves against a real job in both workflows
• Not yet executed end-to-end — the publish path cannot go green before Revert "STAC-3401 blacklisted processes (#71)" #75 applies

Jira: https://stackstate.atlassian.net/browse/STAC-25457

[LouisParkin]

Scan gate results, and what the gate found

The gate is now wired on both images with mode: gate and severity: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL.

Secret scanning is clean on both images.

Vulnerability scanning found a real backlog, which I have declared explicitly rather than hidden by weakening the severity filter:

Image	Findings	Cause
stackstate-k8s-agent	24	19 in the embedded CPython 3.13.13, 3 in cryptography 48.0.1, 2 Go advisories
stackstate-k8s-cluster-agent	2	Go advisories only

Every finding has a file under exceptions/, each with a 14-day expiry, so the gate starts failing again if the underlying bumps stall. Breakdown of the CPython set: 10 are fixed by 3.13.14, 7 have no fix on the 3.13 line (only pre-release 3.15 builds), and 2 have no published fix at all.

The runtime bumps are omnibus software-definition changes rather than anything a workflow in this repo can do, so they are tracked separately in STAC-25556.

LOCAL_IMAGE is renamed to the quay path the publish jobs already use. This is required, not cosmetic: the evaluator keys exceptions on the normalised image name, and normalisation strips only the tag or digest, not the registry and namespace. I verified this by building the evaluator at the pinned SHA and replaying the findings through it. Under the quay name all 26 exceptions match and the gate exits 0; under the old bare stackstate-agent name all 24 findings come back unmanaged.

Known gap: arm64 image scans

The arm64 scan jobs fail with "VEX repositories downloaded, but no OpenVEX documents were found for Grype". That message is misleading. The real cause is that jq is not installed in our self-hosted arm64 runner image, while it is present in the amd64 one. scan-image classifies VEX documents with a jq predicate whose errors are redirected to /dev/null, so a missing jq silently discards all 1105 documents and is then reported as missing VEX data.

Evidence: arm64 skipped all 1105 documents in 830 ms (~0.75 ms each, too fast for a process spawn) while amd64 skipped only 3 at ~32 ms each; and extracting both runner images shows /usr/bin/jq present on amd64 and absent on arm64. jq is also used later for the secret count, so no self-hosted arm64 job can run scan-image at all today.

The fix belongs in the arm64 runner image (stackstate-ci-images), followed by a runner tag bump. I have kept it out of this PR since it is infrastructure rather than agent CI.

YAML comments

Per house convention I have removed the whole-line comments from the workflow YAML across this stack. Shell comments inside run: blocks are script content and stay. I verified the removal is behaviour-neutral by parsing each file before and after and confirming the loaded documents are structurally identical.